Frequency domain perception RGBD image saliency detection system and method based on diffusion model
By adopting a frequency domain perception method based on the diffusion model in RGB-D significance detection, multimodal and frequency domain perception fusion and diffusion denoising processing, the problem of depth map quality problems and insufficient feature fusion in the prior art is solved, and the detection accuracy is significantly improved.
Patent Information
- Application Number
- CN202411979767.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-31
- Publication Date
- 2025-04-11
AI Technical Summary
The existing RGB-D significance detection methods do not fully pay attention to the quality of depth maps and do not fully integrate depth features and color features.
Using a frequency domain perception method based on the diffusion model, the color and depth significance feature maps are extracted through the feature extraction module, the significance-related modules perform multimodal and frequency-domain perception fusion, and the diffusion and noise-denoising module performs denoising processing to obtain the final significance detection result.
It significantly improves the accuracy of significance detection, enhances the quality of depth maps, makes full use of depth information, and reduces the interference of low-quality depth maps on detection.
Smart Images
Figure CN119810461A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an image saliency detection method, and in particular to a frequency domain perception RGBD image saliency detection system and method based on a diffusion model. Background Art
[0002] Image salient object detection is one of the important tasks in image processing. Its purpose is to capture and highlight the areas related to human visual attention from complex images. Traditional methods face challenges in dealing with complex backgrounds, illumination changes, and diverse targets when dealing with image salient object detection. However, in recent years, the rapid development of deep learning technology has brought new progress to image salient object detection. Deep neural networks improve the accuracy and robustness of salient object detection by learning complex image features and relationships. As salient detection has received more and more attention, people have begun to use depth information as auxiliary information for salient detection.
[0003] Existing RGB-D saliency detection methods can be divided into two categories according to whether the methods use deep learning technology. Traditional RGB-D saliency detection methods mainly rely on manually designed features and classic image processing techniques. These methods usually do not require a large amount of data for training, but use low-level features of the image (such as color, texture, and depth information) to perform saliency detection. Specifically, they include the following low-level feature-based methods and fusion feature methods: Low-level feature-based methods extract low-level features such as color, brightness, and texture in the image, and use statistical analysis and image processing techniques to judge saliency. For example, edge detection and color contrast are used to identify salient areas. Fusion feature methods use feature fusion technology of color images and depth images to combine RGB and depth information, and generate saliency maps through weighted averaging or other fusion strategies. This method can overcome the limitations of single features to a certain extent. Although traditional methods are relatively low in computational complexity and perform acceptable on small-scale datasets, they have limited processing capabilities for complex scenes and diverse objects, and generally lack adaptability to environmental changes.
[0004] With the rapid development of deep learning, more and more RGB-D saliency detection methods have begun to adopt deep learning frameworks. Deep learning methods can automatically extract high-level features through training on large-scale data sets, thereby achieving higher detection accuracy. For example, in 2017, Qu et al. were the first researchers to apply deep learning to salient object detection in RGB-D images. They proposed a framework that combines convolutional neural networks with traditional manual feature extraction to automatically learn the interaction mechanism of RGB-D images. Specifically, first, the framework performs superpixel segmentation on RGB-D image pairs to obtain saliency feature vectors. Then, the saliency feature vectors are input into a simple convolutional neural network to generate saliency confidence values. Finally, the superpixel-based Laplacian propagation framework is combined with the saliency confidence value to extract a spatially consistent final saliency map using the intrinsic structure of the input image. Unlike the previous model, in 2019, Liu et al. directly used four-channel RGB-D image pairs as the input of the model and proposed a single-stream recurrent convolutional neural network model. Although these deep learning methods have performed well in many visual tasks, they still have the following shortcomings:
[0005] 1. Although the above methods use depth information to generate saliency maps, they do not pay attention to the quality of depth maps;
[0006] 2. The depth features and color features are not fully integrated. Summary of the invention
[0007] In view of the defects in the prior art, the object of the present invention is to provide a frequency domain perception RGBD image saliency detection system and method based on diffusion model.
[0008] According to one aspect of the present invention, a frequency domain perception RGBD image saliency detection system based on a diffusion model is provided, comprising:
[0009] A feature extraction module extracts a color saliency feature map and a depth saliency feature map of an input image;
[0010] A saliency correlation module performs multimodal fusion and frequency domain perception fusion on the color saliency feature map and the depth saliency feature map to obtain an overall feature;
[0011] The diffusion denoising module performs diffusion denoising based on the overall feature as a conditional driver to obtain a final saliency detection result.
[0012] Preferably, the feature extraction module uses Transformer as the main network, including N stages, each stage generates a feature map of one scale, and multiple stages obtain feature maps of multiple scales;
[0013] The structure of each stage is the same, including:
[0014] Convolution block: The input of the convolution block of the first stage is the input image, and the input of the convolution block of the next N-1 stages is the output of the previous stage; all the convolution blocks extract the feature information of their inputs and modify the image size, and output feature images that meet the size requirements of subsequent processing;
[0015] Linear layer: The input is the output of the corresponding convolutional layer; the linear layer performs a linear transformation on its input and outputs a feature image after the linear transformation;
[0016] Multi-head attention layer: The input is the output of the linear layer; the multi-head attention layer focuses on different parts of its input and captures different features as the feature map of the output of each stage;
[0017] The final output of the feature extraction module is a multi-scale color salient feature map and a depth salient feature map.
[0018] Preferably, the saliency correlation module comprises a multimodal feature fusion submodule and a frequency domain perception feature fusion submodule connected in series, wherein:
[0019] The multimodal feature fusion submodule performs multimodal feature fusion on the multi-scale color saliency feature map and the depth saliency feature map output by the feature extraction module, and enhances the features of the depth saliency feature map to obtain a multi-scale fusion feature;
[0020] The frequency domain perception feature fusion submodule fuses the multi-scale fusion features output by the multimodal feature fusion submodule through frequency domain perception to obtain an overall feature.
[0021] Preferably, the multimodal feature fusion submodule includes:
[0022] A spatial attention module, which uses a spatial attention mechanism to extract the feature positions of the input multi-scale color saliency feature map and the depth saliency feature map;
[0023] A channel attention module, which uses a channel attention mechanism to extract feature content of the input multi-scale color saliency feature map and the depth saliency feature map;
[0024] A first addition fusion unit is configured to fuse the color salient feature map and the depth salient feature map that have passed through the spatial attention module and the channel attention module for multiple times to obtain first fused features of multiple scales;
[0025] The first size restoration unit restores the first fused features of multiple scales to the feature size of the original input image to enhance the feature representation of the image.
[0026] Preferably, the frequency domain perception feature fusion submodule includes:
[0027] A discrete cosine transform unit, performing discrete cosine transform and filtering on the features output by the first size restoration unit to obtain frequency domain features;
[0028] A spatial domain transformation unit, which obtains spatial domain features by performing spatial attention on the features output by the first size restoration unit;
[0029] A second addition and fusion unit, adding and fusing the frequency domain features and the space domain features;
[0030] The second size recovery unit: restores the fusion features output by the addition fusion unit to the feature size of the original input image to obtain an overall feature.
[0031] Preferably, the diffusion denoising module is a model driven by the features output by the saliency correlation module, and is capable of gradually denoising a noise sampled from a normal distribution to generate salient object detection results for any image group;
[0032] The diffusion denoising module comprises:
[0033] The forward diffusion unit concatenates the input features with noise sampled from a normal distribution;
[0034] The reverse diffusion unit denoises the concatenated features through the decoder to obtain the final salient object detection results.
[0035] According to a second aspect of the present invention, a frequency domain perception RGBD image saliency detection method based on a diffusion model is provided, comprising:
[0036] Extracting color saliency feature map and depth saliency feature map of the input image;
[0037] Performing multimodal fusion and frequency domain perception fusion on the color saliency feature map and the depth saliency feature map to obtain an overall feature;
[0038] The overall feature is used as a condition to drive diffusion denoising and obtain the final saliency detection result.
[0039] Preferably, the performing multimodal fusion and frequency domain perceptual fusion on the color saliency feature map and the depth saliency feature map to obtain the overall feature includes:
[0040] Performing multimodal feature fusion on the color saliency feature map and the depth saliency feature map, and enhancing features of the color saliency feature map to obtain multi-scale fusion features;
[0041] The multi-scale fusion features are fused into a whole through frequency domain perception to obtain an overall feature.
[0042] Preferably, the multimodal feature fusion includes:
[0043] Using a spatial attention mechanism, extracting feature positions of the input multi-scale color saliency feature map and the depth saliency feature map;
[0044] Adopting a channel attention mechanism to extract the feature content of the input multi-scale color saliency feature map and the depth saliency feature map;
[0045] Fusing the color salient feature map and the depth salient feature map that have passed through the spatial attention mechanism and the channel attention mechanism multiple times to obtain first fused features of multiple scales;
[0046] The first fused features of multiple scales are restored to the feature size of the original input image to enhance the feature representation of the image.
[0047] Preferably, the frequency domain perception feature fusion includes:
[0048] Performing discrete cosine transform and filtering on the features output by the first size recovery unit to obtain frequency domain features;
[0049] Obtaining spatial domain features by performing spatial attention on the features output by the first size restoration unit;
[0050] Adding and fusing the frequency domain feature and the space domain feature to obtain a second fused feature;
[0051] The second fused feature is restored to the feature size of the original input image to obtain an overall feature.
[0052] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:
[0053] The frequency domain perception RGBD image saliency detection system and method based on the diffusion model in the embodiment of the present invention uses a feature extraction module that has a stronger feature extraction capability than the prior art; the saliency correlation module can better eliminate the interference of low-quality depth saliency maps on saliency detection, and better fuse the depth saliency map and the color saliency map.
[0054] The frequency domain perception RGBD image saliency detection system and method based on the diffusion model in the embodiment of the present invention uses a depth map to generate a saliency feature map, and processes the color feature map and the depth feature map through multiple spatial attention mechanisms and channel attention mechanisms, and then enhances the depth feature map through the processed color feature map (i.e., fuses it with the color feature map), improves the quality of the depth map, makes full use of depth information, pays attention to depth quality, and significantly improves the accuracy of saliency detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:
[0056] Figure 1 is a framework diagram of a frequency domain perception RGBD image saliency detection system based on a diffusion model in one embodiment of the present invention;
[0057] Figure 2 1 is a practical example of a specific embodiment of the present invention, (a) is an input color image, (b) is an original depth image corresponding to the input color image;
[0058] Figure 3 These are some subjective experimental results obtained on a public dataset in the field of collaborative saliency detection in a specific embodiment of the present invention. DETAILED DESCRIPTION
[0059] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.
[0060] In one embodiment of the present invention, a frequency domain perception RGBD image saliency detection system based on a diffusion model is provided. Figure 1 As shown in the figure, it mainly includes feature extraction module, saliency correlation module and diffusion denoising module.
[0061] The feature extraction module extracts a color saliency feature map and a depth saliency feature map of the input image;
[0062] The saliency correlation module performs multimodal fusion and frequency domain perception fusion on the color saliency feature map and the depth saliency feature map to obtain the overall features;
[0063] The diffusion denoising module uses the overall features as a conditional driver to perform diffusion denoising and obtain the final saliency detection result.
[0064] In the aforementioned embodiment, the three modules work together to detect the saliency of the image, showing excellent feature extraction capabilities, thereby achieving accurate saliency detection results.
[0065] Conventionally, the input images are color images and depth images, and the respective saliency information is further extracted based on the input images. In a preferred embodiment of the present invention, the feature extraction module uses a pyramid-type visual Transformer as the backbone network, which is composed of 4 stage structures. Each stage generates feature maps of different scales, and each stage has a similar structure. Specifically, each stage includes a convolution block, a linear layer, and a multi-head attention layer. Among them:
[0066] Convolutional blocks: The input of the convolutional blocks of the first stage is the input image, and the input of the convolutional blocks of the next three stages is the output of the previous stage; all convolutional blocks extract the feature information of their input and modify the image size, and output feature images that meet the size of subsequent processing.
[0067] Linear layer: The input is the output of the convolutional layer of the corresponding layer; the linear layer performs a linear transformation on its input and outputs the feature image after the linear transformation;
[0068] Multi-head attention layer: The input is the output of the linear layer; the multi-head attention layer focuses on different parts of its input, captures different features, and outputs feature maps of different scales at each stage.
[0069] Therefore, the original color image and depth image are input into the above Transformer as the backbone network, and then after passing through the 4 stage structure, 4 color feature maps of different scales are obtained. And 4 kinds of deep feature maps
[0070] In another embodiment of the present invention, the saliency correlation module is composed of a multimodal feature fusion submodule and a frequency domain feature fusion submodule; the feature maps of the color image and the depth image generated by the Transformer backbone network are used as input of the backbone network, and the deep features of the image group are further extracted. Based on these features, the color image and the depth image are subjected to multimodal feature fusion, and then the features of the color image are enhanced, and the multi-scale fused features are fused into a whole through the frequency domain perception method.
[0071] In a preferred embodiment, the multimodal feature fusion submodule includes multiple spatial attentions and channel attentions connected in series. The role of spatial attention is to focus on where the valid information is on the feature map; the role of channel features is to focus on what is the useful information in the image. This module performs multiple spatial attention and channel attention operations on the color features and depth features output by the backbone network, and then passes the features processed by the two attention mechanisms through the first addition fusion unit for multimodal fusion. Finally, the fused features are restored to the feature size of the original input image through the first size recovery unit, and 4 aggregated features fused by color features and depth features of corresponding scales are obtained. This fusion method can enhance the feature representation of the image and improve the detection accuracy.
[0072] In another preferred embodiment, the frequency domain perception feature fusion submodule includes:
[0073] Discrete cosine transform unit: features output by the multimodal feature fusion submodule Perform cosine transform and filtering to reduce high-frequency features and obtain frequency domain features;
[0074] Spatial domain feature unit: features output by the multimodal feature fusion submodule Through the spatial attention mechanism, spatial domain features are obtained;
[0075] The second addition fusion unit: performs addition fusion operation on the frequency domain features and space domain features of the discrete cosine transform part.
[0076] The second size recovery unit: restores the fused features output by the addition fusion unit to the feature size of the original input image.
[0077] Therefore, the input of the frequency domain perception feature fusion submodule in the above embodiment is the aggregated feature The output is the final fusion feature F RD .
[0078] The output features of the frequency domain perception feature fusion submodule of the above embodiment are used as the conditional drive of the diffusion denoising module to complete denoising. In a preferred embodiment, in the diffusion denoising module, the following operations are performed:
[0079] First, the input features are concatenated and embedded with the noise sampled from the normal distribution;
[0080] Finally, the concatenated features and noise are passed through the decoder to denoise and output the salient object detection results.
[0081] In some preferred embodiments, the decoder is divided into unet and denoising parts. unet is responsible for performing simple processing on the two features after splicing, such as aligning the size. The denoising part uses a diffusion model to denoise the features after the size alignment to obtain the final output detection result. The denoising process of the diffusion model is to obtain the detection result from the noise. The input feature is actually a conditional feature of the denoising process, which is responsible for guiding the denoising process to generate the required saliency detection result from the noise.
[0082] Based on the same inventive concept, in another embodiment of the present invention, a frequency domain perception RGBD image saliency detection method based on a diffusion model is provided, comprising:
[0083] Step 1, extracting the color saliency feature map and the depth saliency feature map of the input image;
[0084] Step 2: Perform multimodal fusion and frequency domain perception fusion on the color saliency feature map and the depth saliency feature map to obtain the overall feature;
[0085] Step 3: Use the overall features as a condition to perform diffusion denoising and obtain the final saliency detection result.
[0086] In a preferred embodiment, a preferred solution of step 2 is provided, specifically:
[0087] Perform multi-modal feature fusion on the color saliency feature map and the depth saliency feature map, and enhance the features of the color saliency feature map to obtain multi-scale fusion features;
[0088] The multi-scale fusion features are fused into a whole through frequency domain perception to obtain an overall feature.
[0089] Furthermore, in a preferred embodiment, multimodal feature fusion includes:
[0090] The spatial attention mechanism is used to extract the feature positions of the input multi-scale color saliency feature map and the depth saliency feature map;
[0091] Adopt channel attention mechanism to extract the feature content of input multi-scale color saliency feature map and depth saliency feature map;
[0092] The color salient feature maps and the depth salient feature maps that have passed through the spatial attention mechanism and the channel attention mechanism for multiple times are fused to obtain the first fused features of multiple scales;
[0093] The first fused features of multiple scales are restored to the feature size of the original input image to enhance the feature representation of the image.
[0094] In another preferred embodiment, the frequency domain perception feature fusion includes:
[0095] Perform discrete cosine transform and filtering on the restored first fusion feature to obtain frequency domain features;
[0096] The spatial domain features are obtained by applying spatial attention to the first fusion features that are restored;
[0097] The frequency domain features and the spatial domain features are added and fused to obtain the second fused features;
[0098] The second fused feature is restored to the feature size of the original input image to obtain an overall feature.
[0099] The implementation schemes of other steps in the above embodiment are consistent with the corresponding modules / units of the frequency domain perception RGBD image saliency detection system based on the diffusion model in the above embodiment, and are not described in detail here.
[0100] In order to better extract features from the input image, in a preferred embodiment of the present invention, the track dataset NEU RSDSS-AUG is used as a training set. The combination of the image group and the noise of the diffusion model is directly used as input, and the frequency domain perception RGBD image saliency detection model based on the diffusion model (i.e., the initial detection model mentioned above) is trained end-to-end to automatically learn the mapping relationship between the original image and the true value map.
[0101] Furthermore, the loss function used in model training is IoU loss and BCE loss function, and the learning rate is 10 -6 The batch size is fixed to 12, and training until the model converges generally requires 150 epochs. The trained diffusion denoising module uses the output features of the frequency domain perception feature fusion module As the conditional features of the diffusion model, generate any set of images Results of salient object detection
[0102] In order to verify the feasibility and performance of the frequency domain perception RGBD image saliency detection system and method based on the diffusion model in the above embodiment, in a specific embodiment of the present invention, the track dataset NEU RSDSS-AUG is used as a test set; during the test, the color image and depth image (such as Figure 2 The trained frequency domain perception RGBD image saliency detection model based on diffusion model is input for detection, and the experimental results are finally obtained. Figure 3 From Figure a, Figure b, and Figure c, it can be seen that the system and method in the embodiments of the present invention can more effectively and completely highlight the salient objects in the two images and suppress irrelevant background areas.
[0103] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A frequency domain-aware RGBD image saliency detection system based on a diffusion model, characterized in that: include: A feature extraction module extracts a color saliency feature map and a depth saliency feature map of an input image; A saliency correlation module performs multimodal fusion and frequency domain perception fusion on the color saliency feature map and the depth saliency feature map to obtain an overall feature; The diffusion denoising module performs diffusion denoising based on the overall feature as a conditional driver to obtain a final saliency detection result.
2. According to claim 1, a frequency domain perception RGBD image saliency detection system based on diffusion model is characterized in that: The feature extraction module uses Transformer as the main network, including N stages, each stage generates a feature map of one scale, and multiple stages obtain feature maps of multiple scales; The structure of each stage is the same, including: Convolution block: The input of the convolution block of the first stage is the input image, and the input of the convolution block of the next N-1 stages is the output of the previous stage; all the convolution blocks extract the feature information of their inputs and modify the image size, and output feature images that meet the size requirements of subsequent processing; Linear layer: The input is the output of the corresponding convolutional layer; the linear layer performs a linear transformation on its input and outputs a feature image after the linear transformation; Multi-head attention layer: The input is the output of the linear layer; the multi-head attention layer focuses on different parts of its input and captures different features as the feature map of the output of each stage; The final output of the feature extraction module is a multi-scale color salient feature map and a depth salient feature map.
3. According to the frequency domain perception RGBD image saliency detection system based on diffusion model in claim 1, it is characterized in that: The saliency-related module includes a multimodal feature fusion submodule and a frequency-domain perception feature fusion submodule connected in series, wherein: The multimodal feature fusion submodule performs multimodal feature fusion on the multi-scale color saliency feature map and the depth saliency feature map output by the feature extraction module, and enhances the features of the depth saliency feature map to obtain a multi-scale fusion feature; The frequency domain perception feature fusion submodule fuses the multi-scale fusion features output by the multimodal feature fusion submodule through frequency domain perception to obtain an overall feature.
4. The frequency domain perception RGBD image saliency detection system based on diffusion model according to claim 3, characterized in that: The multimodal feature fusion submodule includes: A spatial attention module, which uses a spatial attention mechanism to extract the feature positions of the input multi-scale color saliency feature map and the depth saliency feature map; A channel attention module, which uses a channel attention mechanism to extract feature content of the input multi-scale color saliency feature map and the depth saliency feature map; A first addition fusion unit is configured to fuse the color salient feature map and the depth salient feature map that have passed through the spatial attention module and the channel attention module for multiple times to obtain first fused features of multiple scales; The first size restoration unit restores the first fused features of multiple scales to the feature size of the original input image to enhance the feature representation of the image.
5. The frequency domain perception RGBD image saliency detection system based on diffusion model according to claim 4, characterized in that: The frequency domain perception feature fusion submodule includes: A discrete cosine transform unit, performing discrete cosine transform and filtering on the features output by the first size restoration unit to obtain frequency domain features; A spatial domain transformation unit, which obtains spatial domain features by applying a spatial attention mechanism to the features output by the first size restoration unit; A second addition and fusion unit, adding and fusing the frequency domain features and the space domain features; The second size recovery unit: restores the fusion features output by the addition fusion unit to the feature size of the original input image to obtain an overall feature.
6. The frequency domain perception RGBD image saliency detection system based on diffusion model according to claim 1, characterized in that: The diffusion denoising module is a model driven by the features output by the saliency correlation module, which can gradually denoise a noise sampled from a normal distribution to generate salient object detection results for any image group; The diffusion denoising module comprises: The forward diffusion unit concatenates the input features with noise sampled from a normal distribution; The reverse diffusion unit denoises the concatenated features through the decoder to obtain the final salient object detection results.
7. A frequency domain perception RGBD image saliency detection method based on diffusion model, characterized in that: include: Extracting color saliency feature map and depth saliency feature map of the input image; Performing multimodal fusion and frequency domain perception fusion on the color saliency feature map and the depth saliency feature map to obtain an overall feature; The overall feature is used as a condition to drive diffusion denoising and obtain the final saliency detection result.
8. The frequency domain perception RGBD image saliency detection method based on diffusion model according to claim 7, characterized in that: The multimodal fusion and frequency domain perception fusion of the color saliency feature map and the depth saliency feature map are performed to obtain the overall feature, including: Performing multimodal feature fusion on the color saliency feature map and the depth saliency feature map, and enhancing features of the color saliency feature map to obtain multi-scale fusion features; The multi-scale fusion features are fused into a whole through frequency domain perception to obtain an overall feature.
9. The method for frequency domain perception RGBD image saliency detection based on diffusion model according to claim 8, characterized in that: The multimodal feature fusion includes: Using a spatial attention mechanism, extracting feature positions of the input multi-scale color saliency feature map and the depth saliency feature map; Adopting a channel attention mechanism to extract the feature content of the input multi-scale color saliency feature map and the depth saliency feature map; Fusing the color salient feature map and the depth salient feature map that have passed through the spatial attention mechanism and the channel attention mechanism multiple times to obtain first fused features of multiple scales; The first fused features of multiple scales are restored to the feature size of the original input image to enhance the feature representation of the image.
10. The frequency domain perception RGBD image saliency detection method based on diffusion model according to claim 9, characterized in that: The frequency domain perception fusion includes: Performing discrete cosine transform and filtering on the first fusion features of the restored size to obtain frequency domain features; The first fusion feature of the restored size is subjected to spatial attention to obtain a spatial domain feature; Adding and fusing the frequency domain feature and the space domain feature to obtain a second fused feature; The second fused feature is restored to the feature size of the original input image to obtain an overall feature.