An RGB-D saliency detection method, device, electronic equipment and medium

Abstract

The application discloses an RGB-D saliency detection method and device, electronic equipment and medium, wherein the method comprises: preprocessing an input image; generating a superpixel segmentation map and extracting a boundary; inputting the preprocessed RGB image and depth image into an encoder to obtain multimodal features; fusing the RGB features and depth features; inputting the fused features into a decoder for decoding to output a predicted saliency map. The RGB features are input into a boundary perception module to output a predicted RGB image boundary map; the fused features are input into the boundary perception module to output a predicted saliency map boundary; the saliency target prediction result is supervised by using a fine-labeled segmentation map, the RGB image boundary is predicted by using the boundary map, and the learning of the model is guided by using a loss function. The application provides boundary guidance for the network through a superpixel generation algorithm, greatly improves the perception of the network to the edge, effectively improves the segmentation quality, and can be widely applied to the field of image processing.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more particularly to an RGB-D saliency detection method, apparatus, electronic device, and medium. Background Technology

[0002] Saliency detection refers to the use of intelligent algorithms to simulate human visual characteristics and extract salient regions (i.e., regions of interest to humans) from images or videos. Its purpose is to enable computers to quickly and accurately identify key information in a scene, just like humans do, thereby improving the efficiency and accuracy of visual information processing.

[0003] Saliency detection has wide applications in multiple computer vision fields. Due to its strong versatility and scalability, it can be used in the following areas: 1) Object recognition: Saliency detection can quickly locate target regions in images, thereby improving the accuracy and efficiency of object recognition. 2) Image and video compression: Saliency detection can help identify key information in images or videos, thus enabling more efficient compression algorithms. 3) Image retrieval: In massive image databases, saliency detection can quickly find images relevant to user queries. 4) Image resizing: When resizing images, salient regions are preserved from distortion, improving the user experience.

[0004] In traditional algorithms, saliency detection relies on manually constructed features to calculate saliency. In recent years, with the development of deep learning methods, visual feature extraction networks based on convolutional neural networks and Transformers can effectively extract image features. However, when encountering complex environments such as similar textures and dark backgrounds, RGB images often fail to provide more discriminative information, making saliency detection methods based on RGB images inadequate.

[0005] In recent years, the widespread application of depth sensors has made depth maps easier to acquire. Researchers have attempted to perform saliency detection on RGB images supplemented with spatial information. Furthermore, with the proliferation of thermal imaging cameras, it has been found that thermal infrared information is highly effective in addressing recognition ambiguity caused by poor lighting conditions, achieving excellent results, for example, in semantic segmentation tasks of urban street scenes. Therefore, thermal infrared images generated by thermal imagers can also serve as important supplementary information. Consequently, researchers have proposed RGB-D saliency detection and RGB-T saliency detection that combine depth features. Existing deep learning-based RGB-D networks such as BBS-Net, HAINet, and LSNet generally suffer from mediocre detection quality and blurred edges against complex backgrounds. Therefore, a saliency detection algorithm with high accuracy and high-quality segmentation has significant application potential. Summary of the Invention

[0006] In order to at least partially solve one of the technical problems existing in the prior art, the present invention aims to provide an RGB-D saliency detection method, device, electronic device and medium.

[0007] The first technical solution adopted in this invention is:

[0008] An RGB-D saliency detection method includes the following steps:

[0009] Input an RGB image and its corresponding depth image, and preprocess the RGB image and depth image;

[0010] The input RGB image is segmented to obtain a superpixel segmentation map;

[0011] The preprocessed RGB and depth images are input into the encoder for feature extraction to obtain RGB and depth features at different levels from texture to semantics.

[0012] By fusing RGB features and depth features at different levels, a fused feature is obtained;

[0013] The fused features are input into the decoder for decoding, and the predicted saliency map is output.

[0014] The model employs a three-stage process: RGB features are input into a boundary awareness module, which outputs a predicted RGB image superpixel segmentation map; depth and fused features are input into the boundary awareness module, which outputs a predicted saliency map boundary; the prediction results of salient targets are supervised based on the predicted RGB image superpixel segmentation map and saliency map boundary, and the model is trained using a preset loss function for RGB-D saliency detection. The model includes an encoder and a decoder.

[0015] Furthermore, the preprocessing of the RGB image and depth image includes:

[0016] Enhancement operations are performed on RGB and depth images; these enhancement operations include rotation and flipping, cropping and scaling, and normalization.

[0017] Further, the step of segmenting the input RGB image to obtain a superpixel segmentation map includes:

[0018] The NAMLAB superpixel segmentation algorithm is used to segment the input RGB image to generate a superpixel segmentation map;

[0019] The boundaries between superpixel segmentation maps are extracted to generate binary maps, which serve as boundary guides to optimize the model's boundary awareness capabilities.

[0020] Furthermore, the encoder is a Swin-Transformer based encoder;

[0021] The encoder includes an RGB feature extractor and a depth feature extractor; the RGB feature extractor is used to extract RGB features, and this feature extractor selects the swin-base architecture in the timm library as its base; the depth feature extractor is used to extract depth features, and this feature extractor selects the swin-small architecture in the timm library as its base.

[0022] In the early stages of model training, the weights of the RGB feature extractor are frozen, or LoRA technology is used to fine-tune the RGB feature extractor to prevent the knowledge learned during pre-training from being forgotten; both feature extractors are set to feature_only mode to extract multi-level features.

[0023] Furthermore, the fusion of RGB features and depth features at different levels to obtain fused features includes:

[0024] Because of the mismatch in feature dimensions between RGB features and depth features, a feature alignment module composed of convolutional neural networks is used to align the features.

[0025] A convolutional neural network is used to concatenate the aligned features to obtain fused features.

[0026] Further, the step of inputting the fused features into the decoder for decoding and outputting the predicted saliency map includes:

[0027] The fused features are input into the decoder of the U-Net structure to perform multi-scale fusion of the features. The fused result is then input into the classification head composed of a convolutional neural network to obtain the predicted saliency map.

[0028] Furthermore, the boundaries of the superpixel segmentation map contain boundary information of image semantics;

[0029] Boundary information is extracted into a binary image, and multi-level RGB features are input into the boundary perception module. The boundary binary image obtained by the NAMLAB superpixel segmentation algorithm is used to supervise the input of the boundary perception module to improve the model's perception of object boundary features. At the same time, fused features are input into the boundary perception module to improve the boundary perception of salient objects.

[0030] The second technical solution adopted in this invention is:

[0031] An RGB-D saliency detection device, comprising:

[0032] The preprocessing module is used to input RGB images and corresponding depth images, and to preprocess the RGB images and depth images.

[0033] The image segmentation module is used to segment the input RGB image to obtain a superpixel segmentation map;

[0034] The feature extraction module is used to input the preprocessed RGB image and depth image into the encoder for feature extraction, and obtain RGB features and depth features at different levels from texture to semantics;

[0035] The feature fusion module is used to fuse RGB features and depth features at different levels to obtain fused features;

[0036] The image prediction module is used to input fused features into the decoder for decoding and output the predicted saliency map;

[0037] Specifically, RGB features are input into the boundary awareness module, which outputs the predicted RGB image superpixel segmentation map; depth features and fusion features are input into the boundary awareness module, which outputs the predicted saliency map boundary; the prediction results of saliency targets are supervised based on the predicted RGB image superpixel segmentation map and saliency map boundary, and the model is trained using a preset loss function so that the trained model can be used for RGB-D saliency detection.

[0038] The third technical solution adopted in this invention is:

[0039] An electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement an RGB-D saliency detection method as described above.

[0040] The fourth technical solution adopted in this invention is:

[0041] A computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement an RGB-D saliency detection method as described above.

[0042] The fifth technical solution adopted in this invention is:

[0043] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned RGB-D saliency detection method.

[0044] The beneficial effects of this invention are:

[0045] (1) By introducing depth images, the present invention integrates depth information into features from RGB images, thereby further improving the image segmentation effect and enabling the model to adapt to more complex scenarios.

[0046] (2) This invention combines the superpixel segmentation boundary information of the NAMLAB algorithm, making the feature extraction network more sensitive to the object boundary, thereby improving the edge fineness of the final result.

[0047] (3) By using a deep learning-based algorithm, this invention avoids the disadvantage of traditional algorithms requiring a lot of manual feature design. In addition, traditional algorithm-based methods can often only mark rough areas, while deep learning methods only need to input data into the network for multiple rounds of training to obtain high-quality saliency detection results. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart of an RGB-D saliency detection method according to an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of the NAMLAB superpixel segmentation algorithm in an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of the encoder structure in an embodiment of the present invention;

[0052] Figure 4 This is a schematic diagram of feature fusion in an embodiment of the present invention;

[0053] Figure 5 This is a schematic diagram of the boundary sensing module in an embodiment of the present invention;

[0054] Figure 6 This is a schematic diagram of the decoder structure in an embodiment of the present invention. Detailed Implementation

[0055] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0056] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0057] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0058] In the description of this invention, unless otherwise explicitly defined, terms such as "setting," "installing," and "connecting" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0059] Terminology Explanation:

[0060] NAM: Short for Non-symmetry and Anti-packing pattern representation Model, it is an asymmetric inverse layout pattern representation model and a method for representing images.

[0061] LoRA: Low-Rank Adaptation is a fine-tuning technique that adjusts the behavior of a model by inserting low-rank matrices into specific parts of the model (such as the Q, K, and V matrices in the attention mechanism), rather than fine-tuning the entire model.

[0062] To overcome the shortcomings and deficiencies of existing technologies, this invention provides a saliency detection method based on NAM and deep learning. This method uses deep learning algorithms to extract and fuse features from RGB and depth images, decodes them to obtain segmentation results, and provides boundary guidance to the network through a superpixel generation algorithm, which greatly improves the network's perception of edges and effectively improves the quality of segmentation.

[0063] Example 1

[0064] like Figure 1 As shown, this embodiment provides an RGB-D saliency detection method, including the following steps:

[0065] S1. Input an RGB image and a corresponding depth image, and preprocess the RGB image and depth image.

[0066] The input consists of an RGB image and a depth image, where the contents of the RGB and depth images correspond. The input images are preprocessed to obtain a preprocessed image. For example, preprocessing operations include rotating, flipping, cropping, scaling, and normalizing the image to facilitate network processing. Image preprocessing enhances image features and reduces noise, thereby improving processing speed and algorithm accuracy.

[0067] S2. Perform segmentation processing on the input RGB image to obtain a superpixel segmentation map.

[0068] Specifically, the NAMLAB superpixel segmentation algorithm is used to segment the input RGB image, and different levels of superpixel segmentation maps are obtained by setting thresholds. The boundaries between the superpixel blocks are extracted to generate binary maps, which will be used for boundary guidance during model training.

[0069] The following combination Figure 2 This paper introduces the NAMLAB superpixel segmentation algorithm.

[0070] A1. Perform asymmetric inverse matrix pattern matching on the input image to obtain segmentation rectangles, that is, obtain rectangles that match the matching pattern by grid scanning, thus obtaining the initial superpixels of the image. Figure 2 In the middle (b), the superpixels are obtained, and each rectangle represents a superpixel.

[0071] A2. Calculate the color mean and color variance of each rectangle as features, and merge similar rectangles by comparing the four-neighbor features of the superpixel.

[0072] A3. Differential merging, specifically: First, set a threshold, then merge each superpixel with fewer than the threshold with its most similar superpixel in its four-neighborhood. For region merging, a disjoint-set data structure algorithm can be used to reduce complexity. Figure 2 In the diagram (c), the region obtained after differential merging is shown. Compared to the original rectangular superpixels, the number of regions is greatly reduced. Different output results can be obtained by setting a threshold for the number of regions.

[0073] Using deep learning-based algorithms avoids the need for extensive manual feature extraction, while using NAMLAB superpixel segmentation results for supervision can greatly improve the model's ability to perceive edges.

[0074] To intuitively understand the algorithm, Figure 2 The original images from the BSDS500 dataset are shown, along with intermediate results of hierarchical image segmentation based on the NAMLab algorithm. The edges of the segmented regions are outlined in red on the blank image.

[0075] Figure 2 In the middle (a), the original test image of size 321×481×3 is shown on the BSDS500 dataset. Figure 2 Figure (b) shows the output of the NAMLab-based color image representation algorithm. It is also the input to the NAMLab-based homogeneous block merging algorithm, with 27195 NAMLab blocks. Figure 2 Figure (c) shows the output of the NAMLab-based homogeneous block merging algorithm; similarly, it is also the input of the NAMLab-based residual region removal algorithm. After merging the NAMLab blocks, the number of original NAMLab regions is 9483, and these original NAMLab regions usually contain a lot of redundant information that needs to be removed. Figure 2 Figure (d) shows the output of the NAMLab-based residual region removal algorithm. It is also the input to the NAMLab-based neighbor region merging algorithm; the number of intermediate NAMLab regions after removing redundant information from the original NAMLab regions is 829. Figure 2 Tables (e) to (n) show the output results of the neighboring region merging algorithm based on NAMLab, where the number of NAMLab regions are 800, 700, 600, 500, 400, 300, 200, 100, 50 and 30, respectively.

[0076] S3. Input the preprocessed RGB image and depth image into the encoder for feature extraction to obtain RGB features and depth features at different levels from texture to semantics.

[0077] In this embodiment, the preprocessed RGB image and depth image are input into a Swing-Transformer-based encoder to obtain RGB features and depth features at different levels from texture to semantics.

[0078] In some embodiments, see Figure 3 The encoder consists of two Swin-Transformers: an RGB feature extractor and a depth feature extractor. The RGB feature extractor takes a 384x384 pixel RGB three-channel image as input and is pre-trained on the ImageNet dataset using the swin-base architecture from the timm library. The depth feature extractor takes a 384x384 pixel single-channel depth map as input and is based on the swin-small architecture from the timm library.

[0079] During the initial training phase, the weights of the RGB feature extractor are frozen or LoRA techniques are used to fine-tune the RGB feature extractor to prevent the knowledge learned during pre-training from being forgotten. The two feature extractors are set to feature_only mode to extract multi-level features, with RGB feature dimensions of [128, 256, 512, 1024] and depth feature dimensions of [96, 192, 384, 768].

[0080] S4. Fuse RGB features and depth features at different levels to obtain fused features.

[0081] For example, see Figure 4 Because of the mismatch in feature dimensions, a feature alignment module composed of convolutional neural networks is needed to align the features. Then, the aligned features are concatenated and fused using a convolutional neural network.

[0082] S5. Input the fused features into the decoder for decoding and output the predicted saliency map.

[0083] See Figure 6 The obtained fused features are input into the decoder of the U-Net structure to perform multi-scale fusion of the features, and the fused result is input into the classification head composed of a convolutional neural network to obtain the prediction map of salient objects.

[0084] S6. Input RGB features into the boundary awareness module and output the predicted RGB image superpixel segmentation map; input depth features and fusion features into the boundary awareness module and output the predicted saliency map boundary; supervise the prediction results of salient targets based on the predicted RGB image superpixel segmentation map and saliency map boundary, and train the model using a preset loss function so that the trained model can be used for RGB-D saliency detection. The model includes an encoder and a decoder.

[0085] The boundaries of the superpixel segmentation map obtained by the NAMLAB algorithm contain semantic boundary information of the image. We extract the boundaries as binary maps and input multi-level RGB features into the boundary perception module. The boundary binary maps obtained by NAMLAB are used to supervise the input of the boundary perception module to improve the model's perception of object boundary features. Meanwhile, a salient object often contains multiple object structures. To improve the boundary perception of salient objects, the features output by the feature fusion module are also input into the feature perception module.

[0086] Specifically, see Figure 5 The RGB features are input into the boundary awareness module, which outputs the predicted RGB image superpixel segmentation map. The depth features and fused features are also input into the boundary awareness module, which outputs the predicted saliency map boundary. The prediction results are supervised using finely labeled segmentation maps, and the model's learning is guided by region loss functions and boundary loss functions.

[0087] Example 2

[0088] This embodiment provides an RGB-D saliency detection device, including:

[0089] The preprocessing module is used to input RGB images and corresponding depth images, and to preprocess the RGB images and depth images.

[0090] The image segmentation module is used to segment the input RGB image to obtain a superpixel segmentation map;

[0091] The feature extraction module is used to input the preprocessed RGB image and depth image into the encoder for feature extraction, and obtain RGB features and depth features at different levels from texture to semantics;

[0092] The feature fusion module is used to fuse RGB features and depth features at different levels to obtain fused features;

[0093] The image prediction module is used to input fused features into the decoder for decoding and output the predicted saliency map;

[0094] Specifically, RGB features are input into the boundary awareness module, which outputs the predicted RGB image superpixel segmentation map; depth features and fusion features are input into the boundary awareness module, which outputs the predicted saliency map boundary; the prediction results of saliency targets are supervised based on the predicted RGB image superpixel segmentation map and saliency map boundary, and the model is trained using a preset loss function so that the trained model can be used for RGB-D saliency detection.

[0095] Since this device is an RGB-D saliency detection device according to an embodiment of the present invention, and the principle of the device in solving the problem is similar to that of the method, the implementation of this device can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.

[0096] Example 3

[0097] This invention also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to achieve the following: Figure 1 This illustrates an RGB-D saliency detection method.

[0098] It is understood that the memory may include random access memory (RAM) or read-only memory. Optionally, the memory may include non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, etc.; the stored data area may store data created according to the use of the server, etc.

[0099] A processor may include one or more processing cores. The processor connects to various parts of the server via various interfaces and lines, executing instructions, programs, code sets, or instruction sets stored in memory, and accessing data stored in memory to perform various server functions and process data. Optionally, the processor may be implemented using at least one of the following hardware forms: Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor may integrate one or more of the following: Central Processing Unit (CPU) and Modem. The CPU primarily handles the operating system and applications; the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor.

[0100] Since this electronic device is the electronic device corresponding to the RGB-D saliency detection method of the present invention, and the principle of solving the problem by this electronic device is similar to that of the method, the implementation of this electronic device can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.

[0101] Example 4

[0102] This invention also provides a computer-readable storage medium storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to achieve the following: Figure 1 This illustrates an RGB-D saliency detection method.

[0103] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0104] Since this storage medium is the storage medium corresponding to the RGB-D saliency detection method of the present invention, and the principle of the storage medium in solving the problem is similar to that of the method, the implementation of this storage medium can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.

[0105] Example 5

[0106] In some possible implementations, various aspects of the methods of the embodiments of the present invention can also be implemented as a program product comprising program code that, when run on a computer device, causes the computer device to perform the steps of an RGB-D saliency detection method according to various exemplary embodiments of the present application described above. The executable computer program code or "code" for performing the various embodiments can be written in high-level programming languages ​​such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0107] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0108] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0109] The above embodiments are merely illustrative of the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made based on the essence of the content of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for detecting RGB-D saliency, characterized in that, Includes the following steps: Input an RGB image and its corresponding depth image, and preprocess the RGB image and depth image; The input RGB image is segmented to obtain a superpixel segmentation map; The preprocessed RGB and depth images are input into the encoder for feature extraction to obtain RGB and depth features at different levels. By fusing RGB features and depth features at different levels, a fused feature is obtained; The fused features are input into the decoder for decoding, and the predicted saliency map is output. Specifically, RGB features are input into the boundary awareness module, which outputs the predicted RGB image superpixel segmentation map; depth features and fusion features are input into the boundary awareness module, which outputs the predicted saliency map boundary; the prediction results of salient targets are supervised based on the predicted RGB image superpixel segmentation map and saliency map boundary, and the model is trained using a preset loss function so that the trained model can be used for RGB-D saliency detection, wherein the model includes an encoder and a decoder; The step of segmenting the input RGB image to obtain a superpixel segmentation map includes: The NAMLAB superpixel segmentation algorithm is used to segment the input RGB image to generate a superpixel segmentation map; The boundaries between superpixel segmentation maps are extracted to generate binary maps, which serve as boundary guides to optimize the model's boundary awareness capabilities. The encoder is a Swing-Transformer based encoder; The encoder includes an RGB feature extractor and a depth feature extractor; the RGB feature extractor is used to extract RGB features, and this feature extractor is based on the swin-base architecture in the timm library; the depth feature extractor is used to extract depth features, and this feature extractor is based on the swin-small architecture in the timm library. In the early stages of model training, the weights of the RGB feature extractor are frozen, or LoRA technology is used to fine-tune the RGB feature extractor to prevent the knowledge learned during pre-training from being forgotten; both feature extractors are set to feature_only mode to extract multi-level features.

2. The RGB-D saliency detection method according to claim 1, characterized in that, The preprocessing of the RGB image and depth image includes: Enhancement operations are performed on RGB and depth images; these enhancement operations include rotation and flipping, cropping and scaling, and normalization.

3. The RGB-D saliency detection method according to claim 1, characterized in that, The process of fusing RGB features and depth features at different levels to obtain fused features includes: Because of the mismatch in feature dimensions between RGB features and depth features, a feature alignment module composed of convolutional neural networks is used to align the features. A convolutional neural network is used to concatenate the aligned features to obtain fused features.

4. The RGB-D saliency detection method according to claim 1, characterized in that, The step of inputting the fused features into the decoder for decoding and outputting the predicted saliency map includes: The fused features are input into the decoder of the U-Net structure to perform multi-scale fusion of the features. The fused result is then input into the classification head composed of a convolutional neural network to obtain the predicted saliency map.

5. The RGB-D saliency detection method according to claim 1, characterized in that, The boundaries of the superpixel segmentation map contain boundary information of image semantics; Boundary information is extracted into a binary image, and multi-level RGB features are input into the boundary perception module. The boundary binary image obtained by the NAMLAB superpixel segmentation algorithm is used to supervise the input of the boundary perception module to improve the model's perception of object boundary features. At the same time, fused features are input into the boundary perception module to improve the boundary perception of salient objects.

6. An RGB-D saliency detection device, used to implement the method as described in any one of claims 1 to 5, characterized in that, include: The preprocessing module is used to input RGB images and corresponding depth images, and to preprocess the RGB images and depth images. The image segmentation module is used to segment the input RGB image to obtain a superpixel segmentation map; The feature extraction module is used to input the preprocessed RGB image and depth image into the encoder for feature extraction, and obtain RGB features and depth features at different levels from texture to semantics; The feature fusion module is used to fuse RGB features and depth features at different levels to obtain fused features; The image prediction module is used to input fused features into the decoder for decoding and output the predicted saliency map; Specifically, RGB features are input into the boundary awareness module, which outputs the predicted RGB image superpixel segmentation map; depth features and fusion features are input into the boundary awareness module, which outputs the predicted saliency map boundary; the prediction results of saliency targets are supervised based on the predicted RGB image superpixel segmentation map and saliency map boundary, and the model is trained using a preset loss function so that the trained model can be used for RGB-D saliency detection.

7. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the method as described in any one of claims 1 to 5.

Images