RGB-D saliency detection method and device, electronic equipment and medium

Abstract

The invention discloses an RGB-D saliency detection method and device, electronic equipment and a medium. The method comprises the following steps: preprocessing an input image; generating a superpixel segmentation image and extracting a boundary; inputting the preprocessed RGB image and depth image into an encoder to obtain a multi-modal feature; fusing the RGB features and the depth features; and inputting the fused features into a decoder for decoding, and outputting a predicted saliency map. Inputting the RGB features into a boundary sensing module, and outputting a predicted RGB image boundary diagram; inputting the fusion features into a boundary sensing module, and outputting a predicted saliency map boundary; and supervising a saliency target prediction result by using a finely marked segmentation image, predicting an RGB image boundary by using a boundary image, and guiding learning of the model by using a loss function. According to the method, boundary guidance is provided for the network through the superpixel generation algorithm, the perception of the network on the edge is greatly improved, the segmentation quality is effectively improved, and the method can be widely applied to the field of image processing.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an RGB-D saliency detection method, device, electronic equipment and medium. Background Art

[0002] Saliency detection refers to simulating human visual characteristics through intelligent algorithms to extract salient areas (i.e. areas of human interest) in images or videos. Its purpose is to enable computers to quickly and accurately identify key information in a scene like humans, thereby improving the efficiency and accuracy of visual information processing.

[0003] Saliency detection has a wide range of applications in multiple computer vision fields. Due to its strong versatility and scalability, it can be used in the following fields: 1) Target recognition: Through saliency detection, the target area in the image can be quickly located, thereby improving the accuracy and efficiency of target recognition. 2) Image and video compression: Saliency detection can help determine the key information in an image or video, thereby achieving a more efficient compression algorithm. 3) Image retrieval: In a massive image library, saliency detection can quickly find images related to the user's query. 4) Image retargeting: When the image is resized, the salient area is kept unchanged to improve the user experience.

[0004] In traditional algorithms, saliency detection tasks rely on artificially constructed features to calculate saliency. In recent years, with the development of deep learning methods, visual feature extraction networks based on convolutional neural networks and Transformers can extract image features very well. However, when encountering complex environments such as similar textures and dark backgrounds, RGB images often cannot provide more discriminative information, and saliency detection methods based on RGB images seem to be unable to cope with the situation.

[0005] In recent years, the extensive application of depth sensors has made it easier to obtain depth maps. People have tried to perform saliency detection on RGB images supplemented with spatial information. In addition, with the popularization of thermal imaging cameras, people have found that thermal infrared information is very effective for recognition blur caused by poor lighting conditions. For example, it has achieved good results in the semantic segmentation task of urban street scenes, so the thermal infrared images generated by thermal imagers can also be used as important information supplements. Therefore, people have proposed RGB-D saliency detection combined with depth features and RGB-T saliency detection respectively. Existing RGB-D networks based on deep learning, such as BBS-Net, HAINet, and LSNet, generally have mediocre detection results and blurred edges under complex backgrounds. Therefore, a saliency detection algorithm with high precision and high-quality segmentation has a wide range of application scenarios. Summary of the invention

[0006] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the present invention aims to provide an RGB-D saliency detection method, device, electronic device and medium.

[0007] The first technical solution adopted by the present invention is:

[0008] A RGB-D saliency detection method comprises the following steps:

[0009] Input an RGB image and a depth image corresponding to the RGB image, and preprocess the RGB image and the depth image;

[0010] Perform segmentation processing on the input RGB image to obtain a superpixel segmentation map;

[0011] The preprocessed RGB image and depth image are input into the encoder for feature extraction to obtain RGB features and depth features at different levels from texture to semantics;

[0012] Fuse RGB features and depth features at different levels to obtain fused features;

[0013] Input the fused features into the decoder for decoding and output the predicted saliency map;

[0014] The RGB features are input into the boundary perception module, and the predicted RGB image superpixel segmentation map is output; the depth features and fusion features are input into the boundary perception module, and the predicted saliency map boundary is output; the saliency target prediction results are supervised according to the predicted RGB image superpixel segmentation map and saliency map boundary, and the model is trained using a preset loss function so that the trained model can be used for RGB-D saliency detection. The model includes an encoder and a decoder.

[0015] Furthermore, the preprocessing of the RGB image and the depth image includes:

[0016] The RGB image and the depth image are enhanced; the enhancement operations include rotation and flipping operations, cropping and scaling operations, and normalization operations.

[0017] Furthermore, the step of segmenting the input RGB image to obtain a superpixel segmentation map includes:

[0018] The NAMLAB superpixel segmentation algorithm is used to segment the input RGB image and generate a superpixel segmentation map;

[0019] The boundaries between superpixel segmentation images are extracted to generate binary images as boundary guidance to optimize the boundary perception ability of the model.

[0020] Furthermore, the encoder is an encoder based on Swin-Transformer;

[0021] The encoder includes an RGB feature extractor and a depth feature extractor; the RGB feature extractor is used to extract RGB features, and the feature extractor selects the swin-base architecture in the timm library as a basis; the depth feature extractor is used to extract depth features, and the feature extractor selects the swin-small architecture in the timm library as a basis;

[0022] Among them, the weight of the RGB feature extractor is frozen at the beginning of model training, or the LoRA technology is used to fine-tune the RGB feature extractor to prevent the knowledge learned in pre-training from being forgotten; the two feature extractors are set to feature_only mode to extract multi-level features.

[0023] Furthermore, the fusion of RGB features and depth features at different levels to obtain fusion features includes:

[0024] Since the feature dimensions between RGB features and depth features do not match, a feature alignment module composed of convolutional neural networks is used to align the features;

[0025] The convolutional neural network is used to splice the aligned features to obtain fused features.

[0026] Furthermore, the step of inputting the fused features into a decoder for decoding and outputting a predicted saliency map includes:

[0027] The fused features are input into the decoder of the U-Net structure to perform multi-scale fusion of the features, and the fused results are input into the classification head composed of the convolutional neural network to obtain the predicted saliency map.

[0028] Furthermore, the boundary of the superpixel segmentation map contains boundary information of image semantics;

[0029] The boundary information is extracted as a binary image, and the multi-level RGB features are input into the boundary perception module. The boundary binary image obtained by the NAMLAB superpixel segmentation algorithm is used to supervise the input of the boundary perception module to improve the model's perception of the object boundary features; at the same time, the fused features are input into the boundary perception module to improve the boundary perception of significant objects.

[0030] The second technical solution adopted by the present invention is:

[0031] An RGB-D saliency detection device, comprising:

[0032] A preprocessing module, used for inputting an RGB image and a depth image corresponding to the RGB image, and preprocessing the RGB image and the depth image;

[0033] The image segmentation module is used to segment the input RGB image to obtain a super-pixel segmentation map;

[0034] The feature extraction module is used to input the preprocessed RGB image and depth image into the encoder for feature extraction, and obtain RGB features and depth features at different levels from texture to semantics;

[0035] Feature fusion module, used to fuse RGB features and depth features at different levels to obtain fused features;

[0036] The image prediction module is used to input the fused features into the decoder for decoding and output the predicted saliency map;

[0037] Among them, the RGB features are input into the boundary perception module, and the predicted RGB image superpixel segmentation map is output; the deep features and fusion features are input into the boundary perception module, and the predicted saliency map boundary is output; the salient target prediction results are supervised according to the predicted RGB image superpixel segmentation map and saliency map boundary, and the model is trained using a preset loss function so that the trained model can be used for RGB-D saliency detection.

[0038] The third technical solution adopted by the present invention is:

[0039] An electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement an RGB-D saliency detection method as described above.

[0040] The fourth technical solution adopted by the present invention is:

[0041] A computer-readable storage medium, wherein at least one instruction, at least one program, a code set or an instruction set is stored in the storage medium, wherein the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement an RGB-D saliency detection method as described above.

[0042] The fifth technical solution adopted by the present invention is:

[0043] A computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned RGB-D saliency detection method.

[0044] Beneficial effects of the present invention:

[0045] (1) The present invention introduces depth images and integrates depth information into features from RGB images, thereby further improving the image segmentation effect and enabling the model to adapt to more complex scenes.

[0046] (2) The present invention combines the superpixel segmentation boundary information of the NAMLAB algorithm, making the feature extraction network more sensitive to object boundaries, thereby improving the edge refinement of the final result.

[0047] (3) The present invention avoids the disadvantage of traditional algorithms that require a large number of manually designed features by using an algorithm based on deep learning. In addition, methods based on traditional algorithms can often only mark rough areas, while deep learning methods only need to input data into the network for multiple rounds of training to obtain high-quality saliency detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 This is a workflow diagram of an RGB-D saliency detection method in an embodiment of the present invention;

[0050] Figure 2 Schematic diagram of the operation of the NAMLAB superpixel segmentation algorithm in an embodiment of the present invention;

[0051] Figure 3 is a schematic diagram of the structure of an encoder in an embodiment of the present invention;

[0052] Figure 4 is a schematic diagram of feature fusion in an embodiment of the present invention;

[0053] Figure 5 is a schematic diagram of the structure of a boundary perception module in an embodiment of the present invention;

[0054] Figure 6 Schematic diagram of the structure of a decoder in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0056] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0057] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0058] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0059] Terminology explanation:

[0060] NAM: Non-symmetry and Anti-packing pattern representation Model, is an asymmetric inverse layout pattern representation model and a method of image representation.

[0061] LoRA: Low-Rank Adaptation, is a fine-tuning technique that adjusts the behavior of the model by inserting low-rank matrices in specific parts of the model (such as the Q, K, V matrices in the attention mechanism) instead of fine-tuning the entire model.

[0062] In order to overcome the shortcomings and deficiencies of the prior art, the present invention provides a saliency detection method based on NAM and deep learning. The method extracts and fuses features of RGB images and depth images through a deep learning algorithm, decodes and obtains segmentation results, and provides boundary guidance for the network through a superpixel generation algorithm, which greatly improves the network's perception of edges and effectively improves the quality of segmentation.

[0063] Example 1

[0064] like Figure 1 As shown, this embodiment provides an RGB-D saliency detection method, comprising the following steps:

[0065] S1. Input an RGB image and a depth image corresponding to the RGB image, and preprocess the RGB image and the depth image.

[0066] Input an RGB image and a depth image, wherein the contents of the RGB image and the depth image correspond to each other. Preprocess the input image to obtain a preprocessed image. Exemplarily, the preprocessing operation includes: rotating, flipping, cropping, scaling, normalizing, etc. the image to facilitate network processing; image preprocessing can enhance image features and reduce noise to improve processing speed and algorithm accuracy.

[0067] S2. Segment the input RGB image to obtain a superpixel segmentation map.

[0068] Specifically, the NAMLAB superpixel segmentation algorithm is introduced to segment the input RGB image, and superpixel segmentation maps of different levels are obtained by setting thresholds. The boundaries between superpixel blocks are extracted to generate binary maps, which will be used as boundary guidance during model training.

[0069] The following combination Figure 2 Introduce the NAMLAB superpixel segmentation algorithm.

[0070] A1. Perform asymmetric inverse matrix pattern matching on the input image to obtain segmentation rectangles, that is, obtain rectangles that meet the matching pattern by grid scanning, so as to obtain preliminary superpixels of the image. Figure 2 (b) is the obtained superpixel, and each rectangle represents a superpixel.

[0071] A2. Calculate the color mean and color variance of each rectangle as features, and merge similar rectangles by comparing the features of the four neighborhoods of the superpixel.

[0072] A3. Difference merging: First, set a threshold and merge each superpixel with a pixel count less than the threshold with the most similar superpixel in its four neighborhoods. For region merging, the union-find algorithm can be used to reduce complexity. Figure 2 (c) is the region obtained after differential merging. Compared with the original rectangular superpixel, the number of regions is greatly reduced. By setting the threshold of the number of regions, different output results can be obtained.

[0073] The use of deep learning-based algorithms avoids the need for a large amount of hand-crafted feature extraction, while the use of NAMLAB superpixel segmentation results for supervision can greatly improve the model's ability to perceive edges.

[0074] To understand the algorithm intuitively, Figure 2 The original image on the BSDS500 dataset and the intermediate result of hierarchical image segmentation based on the NAMLab algorithm are shown, and the edges of the segmented area are marked with red lines on the blank image.

[0075] Figure 2 (a) shows the original test image of size 321×481×3 on the BSDS500 dataset. Figure 2 (b) shows the output of the color image representation algorithm based on NAMLab. It is also the input of the homogeneous block merging algorithm based on NAMLab, and the number of NAMLab blocks is 27195. Figure 2 (c) shows the output result of the NAMLab-based homogeneous block merging algorithm; similarly, this is also the input result of the NAMLab-based residual region removal algorithm. After the NAMLab blocks are merged, the number of original NAMLab regions is 9483. These original NAMLab regions usually contain a lot of redundant information that needs to be removed. Figure 2 (d) shows the output of the residual region removal algorithm based on NAMLab. It is also the input of the adjacent region merging algorithm based on NAMLab; the number of intermediate NAMLab regions after removing the redundant information of the original NAMLab region is 829. Figure 2 (e) to (n) show the output results of the NAMLab-based neighboring region merging algorithm, where the number of NAMLab regions is 800, 700, 600, 500, 400, 300, 200, 100, 50 and 30, respectively.

[0076] S3. Input the preprocessed RGB image and depth image into the encoder for feature extraction to obtain RGB features and depth features at different levels from texture to semantics.

[0077] In this embodiment, the preprocessed RGB image and depth image are input into an encoder based on Swin-Transformer to obtain RGB features and depth features at different levels from texture to semantics.

[0078] In some embodiments, see Figure 3 The encoder consists of two Swin-Transformers, including an RGB feature extractor and a deep feature extractor. The input of the RGB feature extractor is a 384*384 RGB three-channel image, which is based on the swin-base architecture in the timm library and pre-trained on the ImageNet dataset. The input of the deep feature extractor is a 384*384 single-channel depth map, which is based on the swin-small architecture in the timm library.

[0079] Freeze the weights of the RGB feature extractor at the beginning of training or use LoRA technology to fine-tune the RGB feature extractor to prevent the knowledge learned in pre-training from being forgotten. The two feature extractors are set to feature_only mode to extract multi-level features. The RGB feature dimensions are [128, 256, 512, 1024] and the depth feature dimensions are [96, 192, 384, 768].

[0080] S4. Fuse RGB features and depth features at different levels to obtain fused features.

[0081] For example, see Figure 4 ,Due to the mismatch of feature dimensions, a feature alignment module composed of ,convolutional neural networks is needed to align the features, and then the aligned features are ,concatenated and fused using convolutional neural networks.

[0082] S5. Input the fused features into the decoder for decoding and output the predicted saliency map.

[0083] See also Figure 6 The obtained fusion features are input into the decoder of the U-Net structure to perform multi-scale fusion of the features, and the fusion results are input into the classification head composed of the convolutional neural network to obtain the prediction map of the salient objects.

[0084] S6, input the RGB features into the boundary perception module, output the predicted RGB image superpixel segmentation map; input the deep features and fusion features into the boundary perception module, output the predicted saliency map boundary; supervise the salient target prediction results based on the predicted RGB image superpixel segmentation map and saliency map boundary, and train the model using a preset loss function to use the trained model for RGB-D saliency detection. The model includes an encoder and a decoder.

[0085] The boundary of the superpixel segmentation map obtained by the NAMLAB algorithm contains the boundary information of the image semantics. We extract the boundary as a binary map, input the multi-level RGB features into the boundary perception module, and use the boundary binary map obtained by NAMLAB to supervise the input of the boundary perception module to improve the model's perception of the object boundary features. At the same time, a salient object often contains multiple object structures. In order to improve the boundary perception of salient objects, the features output by the feature fusion module are also input into the feature perception module.

[0086] Specifically, see Figure 5 , input the RGB features into the boundary perception module, and output the predicted RGB image superpixel segmentation map; input the deep features and fused features into the boundary perception module, and output the predicted saliency map boundary. The prediction results are supervised by the finely labeled segmentation map, and the regional loss function and boundary loss function are used to guide the learning of the model.

[0087] Example 2

[0088] This embodiment provides an RGB-D saliency detection device, including:

[0089] A preprocessing module, used for inputting an RGB image and a depth image corresponding to the RGB image, and preprocessing the RGB image and the depth image;

[0090] The image segmentation module is used to segment the input RGB image to obtain a super-pixel segmentation map;

[0091] The feature extraction module is used to input the preprocessed RGB image and depth image into the encoder for feature extraction, and obtain RGB features and depth features at different levels from texture to semantics;

[0092] Feature fusion module, used to fuse RGB features and depth features at different levels to obtain fused features;

[0093] The image prediction module is used to input the fused features into the decoder for decoding and output the predicted saliency map;

[0094] Among them, the RGB features are input into the boundary perception module, and the predicted RGB image superpixel segmentation map is output; the deep features and fusion features are input into the boundary perception module, and the predicted saliency map boundary is output; the salient target prediction results are supervised according to the predicted RGB image superpixel segmentation map and saliency map boundary, and the model is trained using a preset loss function so that the trained model can be used for RGB-D saliency detection.

[0095] Since the device is an RGB-D saliency detection device of an embodiment of the present invention, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation process of the above method embodiment, and the repeated parts will not be repeated.

[0096] Example 3

[0097] An embodiment of the present invention further provides an electronic device, the electronic device comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the following Figure 1 A RGB-D saliency detection method is shown.

[0098] It is understood that the memory may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data created according to the use of the server, etc.

[0099] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect the various parts of the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor can integrate one or a combination of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes the operating system and application programs; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor, but implemented separately through a chip.

[0100] Since the electronic device is an electronic device corresponding to an RGB-D saliency detection method of an embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation process of the above method embodiment, and the repeated parts will not be repeated.

[0101] Example 4

[0102] The embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the following Figure 1 A RGB-D saliency detection method is shown.

[0103] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0104] Since the storage medium is a storage medium corresponding to an RGB-D saliency detection method in an embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above method embodiment, and the repeated parts will not be repeated.

[0105] Example 5

[0106] In some possible implementations, various aspects of the method of the embodiments of the present invention may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to perform the steps of a RGB-D saliency detection method according to various exemplary embodiments of the present application described above in this specification. Among them, the executable computer program code or "code" for executing various embodiments may be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, structured query language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0107] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0108] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0109] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable ordinary technicians in the field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made based on the essence of the content of the present invention should be included in the protection scope of the present invention.

Claims

1. A RGB-D saliency detection method, characterized in that: The following steps are involved: Input an RGB image and a depth image corresponding to the RGB image, and preprocess the RGB image and the depth image; Perform segmentation processing on the input RGB image to obtain a superpixel segmentation map; The preprocessed RGB image and depth image are input into the encoder for feature extraction to obtain RGB features and depth features at different levels; Fuse RGB features and depth features at different levels to obtain fused features; Input the fused features into the decoder for decoding and output the predicted saliency map; Among them, the RGB features are input into the boundary perception module, and the predicted RGB image superpixel segmentation map is output; the deep features and fusion features are input into the boundary perception module, and the predicted saliency map boundary is output; the salient target prediction results are supervised according to the predicted RGB image superpixel segmentation map and saliency map boundary, and the model is trained using a preset loss function so that the trained model can be used for RGB-D saliency detection.

2. The RGB-D saliency detection method according to claim 1, characterized in that: The preprocessing of the RGB image and the depth image includes: The RGB image and the depth image are enhanced; the enhancement operations include rotation and flipping operations, cropping and scaling operations, and normalization operations.

3. The RGB-D saliency detection method according to claim 1, characterized in that: The step of segmenting the input RGB image to obtain a superpixel segmentation map includes: The NAMLAB superpixel segmentation algorithm is used to segment the input RGB image and generate a superpixel segmentation map. The boundaries between the superpixel segmentation maps are extracted to generate a binary map as a boundary guide to optimize the boundary perception ability of the model.

4. The RGB-D saliency detection method according to claim 1, characterized in that: The encoder is a Swin-Transformer based encoder; The encoder includes an RGB feature extractor and a depth feature extractor; the RGB feature extractor is used to extract RGB features, and the feature extractor selects the swin-base architecture in the timm library as a basis; the depth feature extractor is used to extract depth features, and the feature extractor selects the swin-small architecture in the timm library as a basis; Among them, the weight of the RGB feature extractor is frozen at the beginning of model training, or the LoRA technology is used to fine-tune the RGB feature extractor to prevent the knowledge learned in pre-training from being forgotten; the two feature extractors are set to feature_only mode to extract multi-level features.

5. The RGB-D saliency detection method according to claim 1, characterized in that: The fusion of RGB features and depth features at different levels to obtain fusion features includes: Since the feature dimensions between RGB features and depth features do not match, a feature alignment module composed of convolutional neural networks is used to align the features; The convolutional neural network is used to splice the aligned features to obtain fused features.

6. The RGB-D saliency detection method according to claim 1, characterized in that: The step of inputting the fused features into a decoder for decoding and outputting a predicted saliency map comprises: The fused features are input into the decoder of the U-Net structure to perform multi-scale fusion of the features, and the fused results are input into the classification head composed of the convolutional neural network to obtain the predicted saliency map.

7. The RGB-D saliency detection method according to claim 1, characterized in that: The boundary of the superpixel segmentation map contains the boundary information of the image semantics; The boundary information is extracted as a binary image, and the multi-level RGB features are input into the boundary perception module. The boundary binary image obtained by the NAMLAB superpixel segmentation algorithm is used to supervise the input of the boundary perception module to improve the model's perception of the object boundary features; at the same time, the fused features are input into the boundary perception module to improve the boundary perception of significant objects.

8. An RGB-D saliency detection device, characterized in that: include: A preprocessing module, used for inputting an RGB image and a depth image corresponding to the RGB image, and preprocessing the RGB image and the depth image; The image segmentation module is used to segment the input RGB image to obtain a super-pixel segmentation map; The feature extraction module is used to input the preprocessed RGB image and depth image into the encoder for feature extraction, and obtain RGB features and depth features at different levels from texture to semantics; Feature fusion module, used to fuse RGB features and depth features at different levels to obtain fused features; The image prediction module is used to input the fused features into the decoder for decoding and output the predicted saliency map; Among them, the RGB features are input into the boundary perception module, and the predicted RGB image superpixel segmentation map is output; the deep features and fusion features are input into the boundary perception module, and the predicted saliency map boundary is output; the salient target prediction results are supervised according to the predicted RGB image superpixel segmentation map and saliency map boundary, and the model is trained using a preset loss function so that the trained model can be used for RGB-D saliency detection.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

Images