RGB-d salient object detection, semantic segmentation method and system

Abstract

The present application relates to the technical field of image processing, and provides an RGB-D salient object detection and semantic segmentation method and system. The RGB-D segmentation model module comprises an encoder and a decoder; the encoder comprises a plurality of RGB-D blocks; in each RGB-D block, based on an RGB feature map and a depth feature map, a cross-modal attention mechanism is used to obtain a cross-modal attention feature map, and after depth separable convolution is performed on the depth feature map, logical operation is performed on the RGB feature map to obtain a local enhancement feature map; the cross-modal attention feature map, the local enhancement feature map and a shortcut feature map of the RGB feature map are linearly processed to obtain an RGB output result and a depth output result; and the RGB output result and the depth output result are taken as the input of the next RGB-D block.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an RGB-D salient object detection and semantic segmentation method and system. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] Salient object detection aims to simulate the human visual system to locate the most attractive objects in the image. Salient detection can be applied to various visual tasks, such as video, image retrieval, compression, editing, target segmentation, tracking, etc. Semantic segmentation is a very important direction in computer vision. Unlike object detection and recognition, semantic segmentation realizes pixel-level classification of images. It can divide a picture or video (if the video is extracted in frames, it is actually a picture) into multiple blocks according to the similarities and differences of the categories, and is generally applied in the fields of automatic driving and medical image diagnosis.

[0004] At present, the salient object detection based on deep learning or the semantic segmentation based on deep learning adopts an RGB image to train a neural network; in application, two trained neural networks are respectively adopted based on the RGB image and the depth image, and the output results of the two trained neural networks are interacted in the process of neural network detection or segmentation, so as to fully interact the RGB-D features of different levels; after a decoder, the salient object or the segmented target object is obtained.

[0005] It can be seen that the existing method uses an RGB image pre-trained or randomly initialized encoder to encode depth information, but this encoding method cannot fully utilize the depth information. SUMMARY

[0006] In order to solve the technical problems existing in the background art, the present application provides an RGB-D salient object detection and semantic segmentation method and system, which can fully extract and interact RGB-D information, improve the accuracy of RGB-D scene understanding, and improve the salient object detection accuracy and semantic segmentation accuracy.

[0007] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0008] The first aspect of the present application provides an RGB-D salient object detection method.

[0009] An RGB-D salient object detection method comprises:

[0010] An RGB image and a depth image of a picture to be detected are acquired;

[0011] Based on the RGB picture and the depth map of the picture to be detected, an RGB-D detection model is adopted to obtain an RGB-D salient object detection result.

[0012] The RGB-D detection model comprises an encoder and a decoder, and the encoder comprises a plurality of RGB-D blocks.

[0013] The shortcut feature map of the cross-modal attention feature map, the local enhancement feature map and the RGB feature map is linearly processed to obtain an RGB output result and a depth output result.

[0014] Further, the process of performing depth separable convolution on the depth feature map and performing logical operation with the RGB feature map to obtain the local enhancement feature map is realized by a local feature enhancement module.

[0015] Further, the shortcut feature map of the RGB feature map is obtained by shortcut processing of the RGB feature map.

[0016] The second aspect of the present application provides an RGB-D salient object detection system.

[0017] The second aspect of the present application provides an RGB-D salient object detection system.

[0018] An RGB-D salient object detection system comprises:

[0019] A data acquisition module configured to acquire an RGB picture and a depth map of a picture to be detected;

[0020] A detection module configured to adopt a trained RGB-D detection model based on the RGB picture and the depth map of the picture to be detected to obtain an RGB-D salient object detection result.

[0021] The RGB-D detection model module is configured to include an encoder and a decoder; the encoder includes a plurality of RGB-D blocks; in each RGB-D block, based on an RGB feature map and a depth feature map, a cross-modal attention mechanism is adopted to obtain a cross-modal attention feature map, and after depth separable convolution is performed on the depth feature map, the depth feature map is logically operated with the RGB feature map to obtain a local enhancement feature map; the cross-modal attention feature map, the local enhancement feature map and a shortcut feature map of the RGB feature map are linearly processed to obtain an RGB output result and a depth output result; and the RGB output result and the depth output result are taken as inputs of a next RGB-D block to repeat the above process, obtain multi-level RGB output results and depth output results, perform merging processing, and obtain an RGB-D salient object detection result through the decoder.

[0022] The third aspect of the application provides an RGB-D semantic segmentation method.

[0023] An RGB-D semantic segmentation method comprises:

[0024] An RGB image and a depth map of a picture to be segmented are obtained.

[0025] Based on the RGB image and the depth map of the picture to be segmented, an RGB-D segmentation model trained is adopted to obtain an RGB-D segmentation result.

[0026] The RGB-D segmentation model includes an encoder and a decoder; the encoder includes a plurality of RGB-D blocks; in each RGB-D block, based on an RGB feature map and a depth feature map, a cross-modal attention mechanism is adopted to obtain a cross-modal attention feature map, and after depth separable convolution is performed on the depth feature map, the depth feature map is logically operated with the RGB feature map to obtain a local enhancement feature map.

[0027] The cross-modal attention feature map, the local enhancement feature map and a shortcut feature map of the RGB feature map are linearly processed to obtain an RGB output result and a depth output result; and the RGB output result and the depth output result are taken as inputs of a next RGB-D block to repeat the above process, obtain multi-level RGB output results and depth output results, perform merging processing, and obtain an RGB-D segmentation result through the decoder.

[0028] Further, the process of logically operating the depth feature map with the RGB feature map after depth separable convolution is performed on the depth feature map to obtain a local enhancement feature map is implemented by a local feature enhancement module; each RGB-D block includes a cross-modal attention mechanism and a local feature enhancement module.

[0029] Further, the shortcut feature map of the RGB feature map is obtained by shortcut processing of the RGB feature map.

[0030] The fourth aspect of the present application provides an RGB-D semantic segmentation method.

[0031] An RGB-D semantic segmentation system comprises:

[0032] A data acquisition module configured to acquire an RGB image and a depth image of a picture to be segmented;

[0033] A segmentation module configured to, based on the RGB image and the depth image of the picture to be segmented, adopt a trained RGB-D segmentation model to obtain an RGB-D segmentation result;

[0034] An RGB-D segmentation model module configured to comprise an encoder and a decoder; the encoder comprises a plurality of RGB-D blocks; in each RGB-D block, based on an RGB feature map and a depth feature map, an inter-modal attention mechanism is adopted to obtain an inter-modal attention feature map, and after depth separable convolution is performed on the depth feature map, a logical operation is performed on the RGB feature map to obtain a local enhancement feature map; the inter-modal attention feature map, the local enhancement feature map and a shortcut feature map of the RGB feature map are linearly processed to obtain an RGB output result and a depth output result; and the RGB output result and the depth output result are taken as inputs of a next RGB-D block, the above process is repeated to obtain multi-level RGB output results and depth output results, the multi-level RGB output results and the depth output results are merged and processed, and the RGB-D salient object detection result is obtained through the decoder.

[0035] The fifth aspect of the present application provides a computer readable storage medium.

[0036] A computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps in the RGB-D salient object detection method according to the first aspect, or to implement the steps in the RGB-D semantic segmentation method according to the third aspect.

[0037] The sixth aspect of the present application provides a computer device.

[0038] A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps in the RGB-D salient object detection method according to the first aspect, or implements the steps in the RGB-D semantic segmentation method according to the third aspect when executing the program.

[0039] Compared with the prior art, the present application has the following beneficial effects:

[0040] The RGB-D classification pre-training framework is designed, so that the model can fully extract the RGB-D representation in the pre-training stage, and the corresponding RGB-D backbone network is designed, the RGB-D module containing the cross-modal attention mechanism and the local feature enhancement mechanism, the cross-modal attention mechanism uses the information of two modes to perform global semantic extraction, and the local feature enhancement mechanism enhances the detail representation in the local range, and then generates accurate RGB-D scene understanding, and improves the precision of salient object detection and semantic segmentation.

[0041] The RGB-D module is designed, the clues in the image and the depth map are deeply explored, the RGB-D information is captured in the global and local, the model has strong distinguishing ability for the scene, and the proposed module can play a role in the multi-modal tasks such as salient target detection and semantic segmentation. BRIEF DESCRIPTION OF DRAWINGS

[0042] The drawings constituting a part of the present application are used to provide further understanding of the present application, the illustrative embodiments of the present application and the description thereof are used to explain the present application, and do not constitute improper limitation on the present application.

[0043] Figure 1 The RGB-D classification pre-training and downstream task flowchart of the prior mainstream method;

[0044] Figure 2 The RGB-D classification pre-training and downstream task flowchart of the present application;

[0045] Figure 3 The structure diagram of the RGB-D backbone network for downstream tasks;

[0046] Figure 4 The RGB-D module schematic diagram;

[0047] Figure 5 The performance calculation curve diagram of the present application and the prior art. DETAILED DESCRIPTION

[0048] The present application will be further described below in combination with the drawings and embodiments.

[0049] It should be pointed out that the following detailed description is exemplary, and is intended to provide further description of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the present application belongs.

[0050] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present disclosure. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0051] It should be noted that the flow diagrams and block diagrams in the drawings are representative of the architecture, functionality, and operation of possible implementations of methods and systems according to various embodiments of the present disclosure. It should also be noted that each block in the flow diagrams and block diagrams can represent a module, a segment, or a portion of code, which includes one or more executable instructions for implementing the specified logical functions ("instructions"). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the flowchart illustrations and / or block diagrams, and combinations thereof, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or combinations of hardware and software.

[0052] Embodiment one

[0053] The embodiment provides an RGB-D salient object detection method. The embodiment takes the method applied to a server as an example. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server and is realized through interaction of the terminal and the server. The server can be a physical server, a server cluster composed of multiple physical servers or a distributed system, and can also be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communication, middleware services, domain name services, security services CDN, and basic cloud computing services such as big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, and the like, but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in the present application. In the embodiment, the method comprises the following steps.

[0054] obtaining an RGB image and a depth image of a to-be-detected image;

[0055] Based on the RGB image and depth map of the image to be detected, a trained RGB-D detection model is used to obtain RGB-D salient object detection results;

[0056] The RGB-D detection model includes an encoder and a decoder; the encoder includes multiple RGB-D blocks; in each RGB-D block, a cross-modal attention mechanism is used based on the RGB feature map and the depth feature map to obtain a cross-modal attention feature map, and the depth feature map is subjected to depth-separable convolution and then logically operated with the RGB feature map to obtain a local enhancement feature map;

[0057] The cross-modal attention feature map, local enhancement feature map, and shortcut feature map of the RGB feature map are linearly processed to obtain RGB output and depth output results. The RGB output and depth output results are then used as input for the next RGB-D block. The above process is repeated to obtain multi-level RGB output and depth output results. These are then merged and decoded to obtain the RGB-D salient object detection results.

[0058] The technical solution of this embodiment will be described in detail below with reference to the accompanying drawings:

[0059] Step S101: Utilize a depth estimation model to perform depth estimation on the pre-training dataset ImageNet, obtaining the corresponding depth map, RGB image, and class pairs for RGB-D classification pre-training. This differs from existing methods (such as...). Figure 1 As shown, two image backbone networks are used to extract information from the RGB image and the depth map. This invention uses an RGB-D backbone network to fully extract and fuse the two.

[0060] Step S102: This invention designs a module for RGB-D data, including a cross-modal attention mechanism and a local feature enhancement module, such as... Figure 2 As shown, for the input RGB features and depth features Processing is performed. In the cross-modal attention mechanism of this invention, the Query(Q) variable is composed of information from two modalities. and The result is a fusion, calculated as follows:

[0061]

[0062] Where [] indicates feature merging, Pool k×k This means that features are adaptively pooled to k×k, thereby achieving efficient computation and saving computational overhead. `Linear()` represents a linear mapping layer. The Key (K) and Value (V) variables are only... Get:

[0063]

[0064]

[0065] Then, the obtained Q, K, V are used for efficient cross-modal attention:

[0066] X CMA = UP(Q x K T x V),

[0067] where UP represents up-sampling in the spatial dimension to ensure that the feature dimension matches. As a supplement to global semantic extraction, a local feature enhancement module is designed:

[0068]

[0069] where DConv represents a depth separable convolution, and k represents the size of the convolution kernel. Then, the two features and the shortcut X res are combined through two linear layers respectively to obtain the output:

[0070]

[0071]

[0072] The above output is used as the input of the next module. The RGB-D backbone network includes 4 stages, each of which is composed of different numbers of RGB-D modules. In addition, four model variants are designed according to the model parameter size, which are Tiny (T), Small (S), Base (B) and Large (L).

[0073] Step S103: Migrate the pre-trained RGB-D backbone network to the downstream task

[0074] The structure of the RGB-D backbone network is shown in Figure 3 , and the task-specific decoder is added to the pre-trained RGB-D backbone network to migrate to any RGB-D downstream task. The decoder uses the multi-level features generated by the RGB-D backbone network for prediction.

[0075] The image I ∈ R H×W×3 and the depth map D ∈ R H×W are input into the encoder, as shown in Figure 4 , two groups of four multi-level features are generated through four stages (containing different numbers of RGB-D modules), denoted as Then the decoder predicts the combined result of the features to obtain the final result.

[0076] Figure 5 Take the semantic segmentation task as an example, the advantages of the present application are shown, mIoU is an evaluation index of semantic segmentation, the higher the better, and the calculation amount is the calculation overhead required by the calculation model, the smaller the better, so the closer to the upper left corner in the broken line graph, the more advantageous the performance-efficiency is, and the four sizes of the model of the present application all obviously surpass the existing method, fully showing the advantages of the present application.

[0077] Embodiment two

[0078] The embodiment provides an RGB-D salient object detection system.

[0079] An RGB-D salient object detection system comprises:

[0080] A data acquisition module configured to acquire an RGB picture and a depth map of a picture to be detected;

[0081] A detection module configured to, based on the RGB picture and the depth map of the picture to be detected, adopt a trained RGB-D detection model to obtain an RGB-D salient object detection result;

[0082] An RGB-D detection model module configured to comprise an encoder and a decoder; the encoder comprises a plurality of RGB-D blocks; in each RGB-D block, based on an RGB feature map and a depth feature map, a cross-modal attention mechanism is adopted to obtain a cross-modal attention feature map, and at the same time, the depth feature map is subjected to depth separable convolution and then logically operated with the RGB feature map to obtain a local enhancement feature map; the cross-modal attention feature map, the local enhancement feature map and a shortcut feature map of the RGB feature map are subjected to linear processing to obtain an RGB output result and a depth output result; and the RGB output result and the depth output result are taken as inputs of a next RGB-D block, the above process is repeated to obtain multi-level RGB output results and depth output results, which are subjected to merging processing, and then subjected to the decoder to obtain an RGB-D salient object detection result.

[0083] It should be noted that the above data acquisition module, detection module and RGB-D detection model module have the same examples and application scenarios as the steps in embodiment one, but are not limited to the content disclosed in the above embodiment one. It should be noted that the above modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0084] Embodiment three

[0085] The embodiment provides an RGB-D semantic segmentation method.

[0086] An RGB-D semantic segmentation method comprises:

[0087] obtain an RGB image and a depth image of the picture to be segmented;

[0088] based on the RGB image and the depth image of the picture to be segmented, an RGB-D segmentation model is trained to obtain an RGB-D segmentation result;

[0089] The RGB-D segmentation model comprises an encoder and a decoder; the encoder comprises a plurality of RGB-D blocks; in each RGB-D block, based on an RGB feature map and a depth feature map, a cross-modal attention mechanism is used to obtain a cross-modal attention feature map, and after depth separable convolution is performed on the depth feature map, the depth feature map is logically operated with the RGB feature map to obtain a local enhanced feature map;

[0090] The cross-modal attention feature map, the local enhanced feature map and a shortcut feature map of the RGB feature map are linearly processed to obtain an RGB output result and a depth output result; and the RGB output result and the depth output result are taken as inputs of a next RGB-D block to repeat the above process, thereby obtaining multi-level RGB output results and depth output results, which are combined and processed through the decoder to obtain the RGB-D segmentation result.

[0091] The embodiment will be described in detail below with reference to the drawings:

[0092] Step S101: A depth estimation model is used to perform depth estimation on a pre-training data set ImageNet to obtain corresponding depth images, RGB images and class pairs for RGB-D classification pre-training. Unlike existing methods (such as Figure 1 ), which use two image backbone networks to extract information in the RGB image and the depth image, the present application uses an RGB-D backbone network to fully extract and fuse the two.

[0093] Step S102: The present application designs a module for RGB-D data, which comprises a cross-modal attention mechanism and a local feature enhancement module, as shown in Figure 2 The input RGB feature and the depth feature are processed. In the cross-modal attention mechanism of the present application, the Query (Q) variable is formed by fusing the information X i rgb and X i d The calculation method is as follows:

[0094]

[0095] wherein [] represents feature merging, Pool k×kwhich means that the feature is adaptively pooled to k x k to achieve efficient computation. The Key (K) variable and the Value (V) variable are only passed through X i rgb Obtain:

[0096]

[0097]

[0098] Then, the obtained Q, K, and V are used to perform efficient cross-modal attention:

[0099] X CMA = UP(Q x K T x V),

[0100] where UP represents up-sampling in the spatial dimension to ensure that the feature dimension matches. As a supplement to global semantic extraction, a local feature enhancement module is designed:

[0101]

[0102] where DConv represents a depth separable convolution, and k represents the size of the convolution kernel. After that, the two features and the shortcut X res are combined through two linear layers respectively to obtain the output:

[0103]

[0104]

[0105] The above output is used as the input of the next module. The RGB-D backbone network includes 4 stages, each of which is composed of different numbers of RGB-D modules. In addition, four model variants are designed according to the model parameter size, which are Tiny (T), Small (S), Base (B), and Large (L).

[0106] Step S103: Migrate the pre-trained RGB-D backbone network to a downstream task

[0107] The structure of the RGB-D backbone network is shown in Figure 3 The task-specific decoder is added to the pre-trained RGB-D backbone network to migrate it to any RGB-D downstream task. The decoder uses the multi-level features generated by the RGB-D backbone network to make predictions.

[0108] An image I e R H×W×3 and a depth map D e R H×W are input into the encoder, as shown in Figure 4As shown, two groups of four multi-level features are first generated by four stages (containing different numbers of RGB-D modules), denoted as Then the decoder decodes the features The combined results are predicted to obtain the final results.

[0109] Embodiment Four

[0110] The embodiment provides an RGB-D semantic segmentation method.

[0111] An RGB-D semantic segmentation system comprises:

[0112] A data acquisition module configured to acquire an RGB image and a depth map of a picture to be segmented;

[0113] A segmentation module configured to, based on the RGB image and the depth map of the picture to be segmented, adopt a trained RGB-D segmentation model to obtain an RGB-D segmentation result;

[0114] An RGB-D segmentation model module configured to comprise an encoder and a decoder; the encoder comprises a plurality of RGB-D blocks; in each RGB-D block, based on an RGB feature map and a depth feature map, an inter-modal attention mechanism is adopted to obtain an inter-modal attention feature map, and at the same time, a depth separable convolution is performed on the depth feature map, and a logical operation is performed on the RGB feature map to obtain a local enhancement feature map; the inter-modal attention feature map, the local enhancement feature map and a shortcut feature map of the RGB feature map are linearly processed to obtain an RGB output result and a depth output result; and the RGB output result and the depth output result are taken as inputs of the next RGB-D block, the above process is repeated to obtain multi-level RGB output results and depth output results, which are combined and processed, and the decoder is used to obtain an RGB-D salient object detection result.

[0115] It should be noted that the above data acquisition module, segmentation module and RGB-D segmentation model module have the same examples and application scenarios as the steps in Embodiment Three, but are not limited to the contents disclosed in Embodiment Three. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a group of computer executable instructions.

[0116] Embodiment Five

[0117] A computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps of the RGB-D salient object detection method according to Embodiment One, or to implement the steps of the RGB-D semantic segmentation method according to Embodiment Three.

[0118] Embodiment Six

[0119] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the RGB-D salient object detection method as described in Embodiment One or implements the steps in the RGB-D semantic segmentation method as described in Embodiment Three when executing the program.

[0120] The above description is merely the preferred embodiments of the present application, and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, and the like made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for RGB-D salient object detection, characterized in that, The method comprises the following steps: obtaining an RGB image and a depth image of a picture to be detected; obtaining an RGB-D salient object detection result by using a trained RGB-D detection model based on the RGB image and the depth image of the picture to be detected; wherein the RGB-D detection model comprises an encoder and a decoder; the encoder comprises a plurality of RGB-D blocks; in each RGB-D block, a cross-modal attention mechanism is used to obtain a cross-modal attention feature map based on an RGB feature map and a depth feature map; the depth feature map needs to be processed by a linear mapping layer Linear () first and then subjected to depth separable convolution, and then subjected to Hadamard product operation with the RGB feature map processed by the linear mapping layer Linear () to obtain a local enhanced feature map; shortcut feature maps of the cross-modal attention feature map, the local enhanced feature map and the RGB feature map are linearly processed to obtain an RGB output result and a depth output result; and the RGB output result and the depth output result are taken as inputs of a next RGB-D block to repeat the above process to obtain multi-level RGB output results and depth output results, which are subjected to merging processing, and then output by the decoder to obtain the RGB-D salient object detection result. 2.The RGB-D salient object detection method of claim 1, wherein, The process of obtaining the local enhanced feature map is implemented by using a local feature enhancement module; each RGB-D block comprises a cross-modal attention mechanism and a local feature enhancement module. 3.The RGB-D salient object detection method of claim 1, wherein, The shortcut feature map of the RGB feature map is obtained by shortcut processing of the RGB feature map.

4. An RGB-D salient object detection system, comprising: The method comprises the following steps: a data acquisition module configured to obtain an RGB image and a depth image of a picture to be detected; a detection module configured to obtain an RGB-D salient object detection result by using a trained RGB-D detection model based on the RGB image and the depth image of the picture to be detected; an RGB-D detection model module configured to comprise an encoder and a decoder; the encoder comprises a plurality of RGB-D blocks; in each RGB-D block, a cross-modal attention mechanism is used to obtain a cross-modal attention feature map based on an RGB feature map and a depth feature map; the depth feature map needs to be processed by a linear mapping layer Linear () first and then subjected to depth separable convolution, and then subjected to Hadamard product operation with the RGB feature map processed by the linear mapping layer Linear () to obtain a local enhanced feature map; shortcut feature maps of the cross-modal attention feature map, the local enhanced feature map and the RGB feature map are linearly processed to obtain an RGB output result and a depth output result; and the RGB output result and the depth output result are taken as inputs of a next RGB-D block to repeat the above process to obtain multi-level RGB output results and depth output results, which are subjected to merging processing, and then output by the decoder to obtain the RGB-D salient object detection result. 5.A method of RGB-D semantic segmentation, characterized in that, The method comprises the following steps: obtaining an RGB image and a depth image of a picture to be segmented; obtaining an RGB-D segmentation result by using a trained RGB-D segmentation model based on the RGB image and the depth image of the picture to be segmented; The RGB-D segmentation model comprises an encoder and a decoder; the encoder comprises a plurality of RGB-D blocks; in each RGB-D block, based on an RGB feature map and a depth feature map, a cross-modal attention mechanism is adopted to obtain a cross-modal attention feature map; the depth feature map needs to be processed by a linear mapping layer Linear () first and then subjected to depth separable convolution, and then subjected to Hadamard product operation with the RGB feature map processed by the linear mapping layer Linear () to obtain a local enhanced feature map; The shortcut feature map of the cross-modal attention feature map, the local enhanced feature map and the RGB feature map is linearly processed to obtain an RGB output result and a depth output result; and the RGB output result and the depth output result are taken as the input of the next RGB-D block to repeat the above process to obtain multi-level RGB output results and depth output results, which are subjected to merging processing, and then output the RGB-D segmentation result through the decoder. 6.The RGB-D semantic segmentation method of claim 5, wherein, The process of obtaining the local enhanced feature map is implemented by using a local feature enhancement module; each RGB-D block comprises a cross-modal attention mechanism and a local feature enhancement module. 7.The RGB-D semantic segmentation method of claim 5, wherein, The shortcut feature map of the RGB feature map is obtained by shortcut processing of the RGB feature map.

8. An RGB-D semantic segmentation system, comprising: The method comprises the following steps: The data acquisition module is configured to acquire an RGB image and a depth map of a picture to be segmented; The segmentation module is configured to acquire an RGB-D segmentation result by using a trained RGB-D segmentation model based on the RGB image and the depth map of the picture to be segmented; The RGB-D segmentation model module is configured to comprise an encoder and a decoder; the encoder comprises a plurality of RGB-D blocks; in each RGB-D block, based on an RGB feature map and a depth feature map, a cross-modal attention mechanism is adopted to obtain a cross-modal attention feature map; the depth feature map needs to be processed by a linear mapping layer Linear () first and then subjected to depth separable convolution, and then subjected to Hadamard product operation with the RGB feature map processed by the linear mapping layer Linear () to obtain a local enhanced feature map; the shortcut feature map of the cross-modal attention feature map, the local enhanced feature map and the RGB feature map is linearly processed to obtain an RGB output result and a depth output result; and the RGB output result and the depth output result are taken as the input of the next RGB-D block to repeat the above process to obtain multi-level RGB output results and depth output results, which are subjected to merging processing, and then output the RGB-D segmentation result through the decoder.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps in the RGB-D salient object detection method of any one of claims 1-3, or implement the steps in the RGB-D semantic segmentation method of any one of claims 5-7.

10. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the steps in the RGB-D salient object detection method of any one of claims 1-3, or implement the steps in the RGB-D semantic segmentation method of any one of claims 5-7.

Images