Multi-modal variation fusion method based on RGB-D salient target detection
Through the MGA module, RGB and depth data are fused, and the generation feature and attention mechanism are used to solve the problem of inaccurate prediction of multimodal data in the prior art, achieving more accurate feature representation and more efficient data processing.
Patent Information
- Application Number
- CN202411148334.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-26
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing RGB-D significant object detection methods are limited in modeling the joint distribution of multimodal data, resulting in inaccurate prediction of significance objects.
The multimodal variational fusion method based on the MGA module is adopted to generate multimodal fusion features by fusion of shared latent features and private latent features and generating attention fusion, and the significance features are calculated through the significance decoder.
It improves the integration and expression ability of features, enhances robustness and generalization ability, reduces the risks of information loss and error accumulation, and improves the processing and analysis efficiency of complex multimodal data.
Smart Images

Figure CN119027771A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to a multimodal variational fusion method based on RGB-D salient object detection. Background Art
[0002] To simulate the human visual attention mechanism, i.e., selecting eye-catching regions in a scene, the task of salient object detection (SOD) is to automatically identify the most attractive objects in an input image. RGB-D SOD has attracted increasing attention due to the importance of depth images in depth perception and has been applied to many fields such as image relocalization, image caption generation, and image quality assessment.
[0003] The core of SOD for RGB and depth images lies in developing fusion strategies to integrate information cues from both modalities. Current methods usually adopt the classical fusion framework to learn discriminant functions that map multimodal images to corresponding saliency maps. However, these discriminant functions are often limited in modeling the joint distribution of multimodal data, resulting in inaccuracies in the prediction of salient objects. In contrast, generative methods excel at modeling distributions, providing a more comprehensive description of multimodal data, resulting in more accurate and reliable saliency predictions. Therefore, exploring the application of generative methods in multimodal distribution modeling is crucial for developing saliency detection models that can effectively handle complex multimodal data. Summary of the invention
[0004] To solve the above technical problems, the present invention proposes a multimodal variational fusion method based on RGB-D salient object detection, which effectively fuses RGB and depth data through the MGA module, thereby improving the integration and expression capabilities of features.
[0005] To achieve the above object, the present invention provides a multimodal variational fusion method based on RGB-D salient object detection, comprising:
[0006] Input an image to be detected, and perform feature extraction on the image to be detected;
[0007] Establishing a latent space model, respectively calculating shared latent features and private latent features based on the extracted features, performing multimodal fusion and generating attention fusion through the shared latent features and the private latent features, and obtaining multimodal fusion features;
[0008] The multimodal fusion features are decoded and calculated by a saliency decoder to obtain saliency features of the image to be detected.
[0009] Preferably, extracting features from the image to be detected includes:
[0010] A dual-branch encoder based on VGG16 as the backbone network performs feature extraction on the image to be detected to obtain RGB image features and depth image features respectively.
[0011] Preferably, establishing a latent space model comprises:
[0012] The latent space model is constructed by a multimodal generative integration (MGA) module, wherein the latent space model includes a shared latent subspace and a private latent subspace.
[0013] Preferably, the shared potential features are calculated based on the extracted features, including:
[0014] By generating a fusion module, given the high-level feature f r 5 and f d 5. Calculate the shared subspace feature z through the fully connected layer s ;
[0015] Based on the high-level features f d 5 and the shared subspace z s Calculate the latent subspace feature z pd .
[0016] Preferably, calculating the private latent features comprises:
[0017] Specify the private subspace of RGB data as z pr , absorbing the high-level features f through the fully connected layer r 5 and the shared subspace z s , calculate the private subspace features;
[0018] Specify the private subspace of Depth data as z pd, , based on the high-level features f d 5 and the shared subspace z s Calculate the latent subspace feature z pd ;
[0019] The shared subspace feature z is transformed into s , private subspace features and the latent subspace features z pd Combined together, we get the fused latent feature z f .
[0020] Preferably, generating the attention fusion comprises:
[0021] Based on the adaptive multimodal fusion AMF unit, given the RGB feature f r and the deep feature f d , and integrate to get f rd ;
[0022] Through the generated attention GA module, based on the feature f rd 、z f and their common spatial attention to construct the attention weight a, and f rd and z f Combined with the attention weight a, the combined feature f of attention fusion is obtained p rd .
[0023] Preferably, decoding and calculating the multimodal fusion features by a saliency decoder includes:
[0024] The upper path calculates saliency based on the fused features of the generated method, and the lower path calculates saliency using the features fused by the generated attention.
[0025] Preferably, the loss function of the upper path is:
[0026] £ p =-β×∑ j∈Y+ log(S p )-(1-β)×∑ j∈Y -log(1-S p )+∑ k KL(q(z k |x)||p(z k ))+TC+SM
[0027] In the formula, p is the total loss of the upper branch, β is the proportion of similar points whose true value is 1, S p is the predicted saliency map, j is the pixel index, Y is the true value map, KL is the Kullback-Leibler Divergence loss, q(z k |x) is the posterior distribution probability of the latent variable, p(z k ) is the prior Gaussian distribution of the latent variable, TC is the overall correlation value, and SM is the similarity value;
[0028] The loss function of the lower layer path is:
[0029]
[0030] In the formula, fp is the total loss of the lower branch, S fp is the predicted saliency map, is the posterior distribution of attention, and p(a) is the prior Gaussian distribution of attention.
[0031] Preferably, the method for obtaining the salient features of the image to be detected is:
[0032] Based on the saliency decoder ASD module, the final saliency feature is generated by connecting the saliency maps on each side, and the overall loss is:
[0033] £ t i =-β×∑ j∈Y+ log(S t i )-(1-β)×∑ j∈Y- log(1-S t i )
[0034] £ b i =-β×∑ j∈Y+ log(S b i )-(1-β)×∑ j∈Y- log(1-S b i )
[0035] £=∑ 4 i (£ t i +£ b i )+£ p +£ fp
[0036] In the formula, t i is the cross entropy loss of the i-th deconvolution layer in the upper branch, S t i is the salient map of the i-th deconvolution layer of the upper branch, b i is the cross entropy loss of the i-th deconvolution layer in the lower branch, S b i is the saliency map of the i-th deconvolution layer of the lower branch, £ is the overall loss, p The cross entropy loss calculated to generate the saliency map of the fused features, fp is the cross entropy loss calculated based on the saliency map after the generative and discriminative features are fused.
[0037] Compared with the prior art, the present invention has the following advantages and technical effects:
[0038] The present invention effectively fuses RGB and depth data through the MGA module, improving the integration and expression capabilities of features. Compared with traditional methods, the present invention can more accurately capture the complementarity between different modalities, thereby improving the efficiency of data processing and analysis.
[0039] By introducing generative features and attention mechanisms, the present invention can generate richer and more discriminative feature representations. This approach not only improves the robustness and generalization ability of features, but also effectively reduces the risk of information loss and error accumulation.
[0040] The present invention can better optimize the performance and learning ability of the model when processing complex multimodal data. Through more sophisticated feature analysis and integration, the present invention can improve recognition accuracy and efficiency in various application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0042] Figure 1 Schematic diagram of a salient object detection network according to an embodiment of the present invention;
[0043] Figure 2 1 is a diagram showing a variational fusion theory graphical model according to an embodiment of the present invention, wherein (a) is VED; (b) is VED+CoVAttn;
[0044] Figure 3 Schematic diagram of MVF saliency detection network according to an embodiment of the present invention;
[0045] Figure 4 This is a structural diagram of an adaptive multi-mode fusion AMF unit in an embodiment of the present invention;
[0046] Figure 5 FIG. 4 is a structural diagram of an adaptive saliency decoder ASD unit according to an embodiment of the present invention. DETAILED DESCRIPTION
[0047] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0048] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0049] The present invention proposes a multimodal variational fusion method based on RGB-D salient object detection, which specifically includes:
[0050] Input an image to be detected and perform feature extraction on the image to be detected, specifically:
[0051] Input image for feature extraction
[0052] like Figure 1-Figure 5 As shown in Figure 1, the MVF saliency detection network is based on an encoder-decoder framework. A dual-branch encoder based on VGG16 as the backbone network is used to learn features from RGB and depth data; Fr and Fd are used to represent RGB and depth features respectively.
[0053] A latent space model is established, and shared latent features and private latent features are calculated based on the extracted features. Multimodal fusion and attention fusion are performed through the shared latent features and the private latent features to obtain multimodal fusion features, specifically:
[0054] Multimodal fusion stage
[0055] In the multimodal fusion stage, there are two main steps: fusion-based generative aggregation and fusion-based generative attention.
[0056] 1) Generation and fusion of multimodal features:
[0057] After feature extraction, the goal is to construct a latent space to generate fused features. According to the shared and complementary relationship between RGB and depth data, the latent space is divided into subspaces: shared and private representations. The shared latent features are first calculated, and then the private latent features are further constructed based on these shared features.
[0058] The Generative Fusion (GF) module is designed according to the following theory to construct a shared and complementary latent space. Given a high-level feature f r 5 and f d 5. Shared subspace z s . It is calculated through a fully connected layer. Then, the private subspace of RGB data is designated as z pr , absorbed by the fully connected layer r 5 and z s Calculated. Similarly, z pd Based on f d 5 and z s Finally, these three potential subspaces are combined through the fully connected layer to obtain z f =cat(z s , z pr , z pd ).
[0059] A generative aggregation theory for multimodal data is used to decompose the data into shared and private latent spaces, and then integrate by aggregating features sampled from these spaces.
[0060] Assume that the latent space of each modality can be represented as a private space z p and shared spaces Given multimodal data (x1, x2), for example, a pair of color and depth images, we can infer the latent space p(z s 1|x1) and p(z s 2|x2). Ideally, the shared space should be consistent. s =z s 1=z s 2p(z s |x1,x2) is defined based on the product of experts (PoE). The assumption of this embodiment is that the shared latent space is a condition of the private space. p 1~q(z1|x1,z s )q(z s |x1,x2)and z p 2~q(z2|x2,z s )q(z s |x1,x2) can be in z s Then, the connections of all private and shared latent spaces can be regarded as the aggregated features of multimodal data.
[0061] 2) Fusion based on generative attention
[0062] In this fusion, the above-mentioned generative features are used to enhance the multimodal features. First, in the classical fusion approach, features from different modalities are calculated in the Adaptive Multimodal Fusion (AMF) unit. Then, the Generative Attention (GA) theory is used to generate conditional variational attention. After that, the fusion features based on the generative features are combined with the classical approach and fused according to the weights of the generative attention.
[0063] Given RGB and depth features f r and f d , these multimodal features are integrated through the AMF module to obtain frd. In this process, f r+b Represents the common part of multimodal features, using f r -f r+b and f d -f r+b represent the complementary features of RGB and depth modalities respectively. In addition, the common features and complementary features are connected as frd. Subsequently, through the GA module, based on f rd 、z f and their common spatial attention construct to generate the attention weight a. Then f rd and z f The features of are combined with the attention weight a, and the combined features are expressed as f p rd .
[0064] Generative attention fusion mechanism theory: Assume that the important i information of x and z is correlated. The mechanism design establishes a joint attention vector a as a random variable, conditioned on x and z. Then, the joint attention is represented as a=ax*az, and each method is calculated using the traditional attention method. Given x and z, assuming that p(a) is a standard Gaussian distribution, the generation of attention is then modeled as a CVAE q(a|x, z, ˙a).
[0065] When approximating the posterior attention from a Gaussian distribution with a variational loss, the formula is as follows:
[0066] J2=Ez~q(z),a~q(x,z,a˙)[log p(y|z,x,a)]-KL[q(a|x,z,a˙)∥p(a)]
[0067] This method cannot be directly applied to multimodal data because different modalities carry different information. To solve this problem, the following embodiment describes how to process multimodal data.
[0068] The multimodal fusion feature is decoded and calculated by a saliency decoder to obtain the saliency feature of the image to be detected, specifically:
[0069] Saliency Decoder
[0070] The saliency decoder consists of two paths. The upper path calculates saliency based on the fused features of the generated fusion, and the lower path calculates saliency using the features of the generated attention fusion. rd ), the saliency value Sp is regressed. The loss function is defined based on the binary cross entropy between Sp and the true value pixels (Y), salient pixels (Y+), and non-salient pixels (Y-):
[0071] £ p =-β×∑ j∈Y+ log(S p )-(1-β)×∑ j∈Y -log(1-S p )+∑ k KL(q(z k |x)||p(z k ))+TC+SM
[0072] For the decoder in the bottom path, the input (f prd ) is used to generate a rough significance estimate S fp .
[0073] The loss function is specified as:
[0074]
[0075] High-level features contain global semantic context, and low-level features are important for capturing local details. Therefore, an adaptive saliency decoder (ASD) module is designed as a unit to combine multi-level features, where the outputs of the previous layers guide the feature fusion of the bottom layers. For each decoder, its input includes RGB and depth features from the current layer and the output of the previous higher layer. The former is first fused by the AMF module and then further connected with the latter.
[0076] For the i-th decoder, multi-level and multi-modal features need to be fused to obtain f i Given f r i and f d i The multimodal features of f are obtained by using the AMF module. rd i . Move the previous layer Up(f i+1 ) and f rd i Connect them and further input them into the convolutional layer to generate a network with C i The characteristic f of the channel i , to guide the feature fusion of the next layer. i In the case of i Compute the saliency map S i During training, the decoder is supervised through two paths. The loss function is defined based on the binary cross entropy between each saliency and its corresponding ground truth. t i , S b i , £ t i , £ b i Represents the saliency map and the loss of different paths. Finally, the final saliency map is generated by connecting the saliency maps on each side, and the entire loss £ is as follows:
[0077] £ t i =-β×∑ j∈Y+ log(S t i )-(1-β)×∑ j∈Y- log(1-S t i )
[0078] £ b i =-β×∑ j∈Y+ log(S bi )-(1-β)×∑ j∈Y- log(1-S b i )
[0079] £=∑ 4 i (£ t i +£ b i )+£ p +£ fp
[0080] The present invention effectively fuses RGB and depth data through the MGA module, improving the integration and expression capabilities of features. Compared with traditional methods, this method can more accurately capture the complementarity between different modalities, thereby improving the efficiency of data processing and analysis.
[0081] Strong feature representation: By introducing generative features and attention mechanisms, the present invention can generate richer and more discriminative feature representations. This approach not only improves the robustness and generalization ability of features, but also effectively reduces the risk of information loss and error accumulation.
[0082] Optimized model performance: This invention can better optimize the performance and learning ability of the model when processing complex multimodal data. Through more sophisticated feature analysis and integration, it can improve recognition accuracy and efficiency in various application scenarios.
[0083] Wide applicability: The proposed method and module are not only applicable to the current RGB and depth data fusion, but can also be extended to other multimodal data fusion problems, such as the combination of speech and image or the joint processing of text and image.
[0084] The present invention aims to obtain multimodal features in a generative way and integrate them with traditional discriminative features to obtain accurate SOD final features. First, existing generative models such as variational autoencoders (VAE) are suitable for modeling rich clues in RGB-D data, but generative models that rely on shared representations or are specific to a single modality often cannot fully capture the inherent distribution of multimodal data. In response to the limitations of existing VAE methods, the present invention proposes a mechanism for multimodal generative aggregation, which generates multimodal feature aggregation from generative models to enhance the ability of saliency estimation. Specifically, this mechanism uses the inspiration of brain embedding information to decompose multimodal data into common and complementary parts, respectively capturing shared and specific distribution patterns, and constructing variables in the latent space of VAE. Subsequently, features are generated from these latent spaces and aggregated to capture more complex relationships and a more comprehensive representation of multimodal data. Secondly, since the traditional simple connection method fails to fully pay attention to the importance of latent variables in generative integration, another method that introduces variational attention mechanism ignores the differential impact of generative features. In this case, in view of the limitations of existing methods, this paper proposes a new mechanism, namely the generative attention fusion mechanism, to fully utilize the feature advantages of generative and deterministic models. This mechanism emphasizes the importance of generalization features and discrimination features by calculating their joint importance and using variational autoencoders (VAEs) to model these features themselves and their importance as latent variables. By comprehensively calculating the joint importance, this attention mechanism effectively enhances the key representation of these two features, thereby promoting a more balanced and more informative comprehensive feature expression.
[0085] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A multimodal variational fusion method for RGB-D salient object detection, characterized in that: include: Input an image to be detected, and perform feature extraction on the image to be detected; Establishing a latent space model, respectively calculating shared latent features and private latent features based on the extracted features, performing multimodal fusion and generating attention fusion through the shared latent features and the private latent features, and obtaining multimodal fusion features; The multimodal fusion features are decoded and calculated by a saliency decoder to obtain saliency features of the image to be detected.
2. The multimodal variational fusion method based on RGB-D salient object detection according to claim 1, characterized in that: Extracting features from the image to be detected includes: A dual-branch encoder based on VGG16 as the backbone network performs feature extraction on the image to be detected to obtain RGB image features and depth image features respectively.
3. The multimodal variational fusion method based on RGB-D salient object detection according to claim 1, characterized in that: Build a latent space model, including: The latent space model is constructed by a multimodal generative integration (MGA) module, wherein the latent space model includes a shared latent subspace and a private latent subspace.
4. The multimodal variational fusion method based on RGB-D salient object detection according to claim 1, characterized in that: Shared latent features are calculated based on the extracted features, including: By generating a fusion module, given the high-level feature f r 5 and f d 5. Calculate the shared subspace feature z through the fully connected layer s ; Based on the high-level features f d 5 and the shared subspace z s Calculate the latent subspace feature z pd .
5. The multimodal variational fusion method based on RGB-D salient object detection according to claim 4, characterized in that: Calculating the private latent features includes: Specify the private subspace of RGB data as z pr , absorbing the high-level features f through the fully connected layer r 5 and the shared subspace z s , calculate the private subspace features; Specify the private subspace of Depth data as z pd , based on the high-level feature f d 5 and the shared subspace z s Calculate the latent subspace feature z pd ; The shared subspace feature z is transformed into s , private subspace features and the latent subspace features z pd Combined together, we get the fused latent feature z f .
6. The multimodal variational fusion method based on RGB-D salient object detection according to claim 5, characterized in that: Generating the attention fusion, comprising: Based on the adaptive multimodal fusion AMF unit, given the RGB feature f r and the deep feature f d , and integrate to get f rd ; Through the generated attention GA module, based on the feature f rd 、z f and their common spatial attention to construct the attention weight a, and f rd and z f Combined with the attention weight a, the combined feature f of attention fusion is obtained p rd .
7. The multimodal variational fusion method based on RGB-D salient object detection according to claim 1, characterized in that: The multimodal fusion feature is decoded and calculated by a saliency decoder, including: The upper path calculates saliency based on the fused features of the generated method, and the lower path calculates saliency using the features fused by the generated attention.
8. The multimodal variational fusion method based on RGB-D salient object detection according to claim 7, characterized in that: The loss function of the upper path is: £ p =-β×∑ j∈Y+ log(S p )-(1-β)×∑ j∈Y -log(1-S p )+∑ k KL(q(z k |x)||p(z k ))+TC+SM In the formula, p is the total loss of the upper branch, β is the proportion of similar points whose true value is 1, S p is the predicted saliency map, j is the pixel index, Y is the true value map, KL is the Kullback-Leibler Divergence loss, q(z k |x) is the posterior distribution probability of the latent variable, p(z k ) is the prior Gaussian distribution of the latent variable, TC is the overall correlation value, and SM is the similarity value; The loss function of the lower layer path is: In the formula, fp is the total loss of the lower branch, S fp is the predicted saliency map, is the posterior distribution of attention, and p(a) is the prior Gaussian distribution of attention.
9. The multimodal variational fusion method based on RGB-D salient object detection according to claim 8, characterized in that: The method for obtaining the salient features of the image to be detected is: Based on the saliency decoder ASD module, the final saliency feature is generated by connecting the saliency maps on each side, and the overall loss is: £ t i =-β×∑ j∈Y+ log(S t i )-(1-β)×∑ j∈Y -log(1-S t i ) £ b i =-β×∑ j∈Y+ log(S b i )-(1-β)×∑ j∈Y -log(1-S b i ) £=∑ 4 i (£ t i +£ b i )+£ p +£ fp In the formula, t i is the cross entropy loss of the i-th deconvolution layer in the upper branch, S t i is the salient map of the i-th deconvolution layer of the upper branch, b i is the cross entropy loss of the i-th deconvolution layer in the lower branch, S b i is the saliency map of the i-th deconvolution layer of the lower branch, £ is the overall loss, p The cross entropy loss calculated to generate the saliency map of the fused features, fp is the cross entropy loss calculated based on the saliency map after the generative and discriminative features are fused.
Citation Information
Cited By
RGB-D image saliency target detection method based on progressive hierarchical fusion
CN120107620A
