Saliency detection method, saliency detection device and storage medium
By using multimodal feature extraction and fusion methods in RGB-D and RGB-T significance target detection, the problem of low robustness of detection results in the prior art is solved, and more stable and accurate significant detection results are achieved.
Patent Information
- Application Number
- CN202111507286.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-10
- Publication Date
- 2025-04-25
- Estimated Expiration
- 2041-12-10
AI Technical Summary
In the prior art, the RGB-D and RGB-T significance object detection methods have low robustness, mainly because the depth map and heat map quality are easily affected by noise, resulting in unstable detection results.
By determining the target convolutional layer output results of the first backbone network and the second backbone network, the global descriptor is calculated and the importance of each modal channel is determined, thereby extracting multimodal features and fusing to improve the robustness of the detection results.
This method enhances the accuracy of feature expression by fully utilizing the complementarity between multimodal features and improves the robustness of the significance detection results.
Smart Images
Figure CN114359162B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a saliency detection method, a saliency detection device and a computer-readable storage medium. Background Art
[0002] In the related art, multimodal RGB-D (RGB-Depth, color-depth) and RGB-T (RGB-Thermal, color-temperature) salient object detection mainly relies on the fusion of low-level clues. This type of method uses depth saliency or temperature saliency as additional information. That is, the former extracts depth features from the depth map and combines them with features extracted from the RGB (color system) image to obtain a saliency map, and the latter extracts temperature features from the thermal map and combines them with features extracted from the RGB image to obtain a saliency map. For example, in RGB-D salient object detection, the process is generally to calculate the priors in the RGB image and the Depth image, such as background prior, center prior and depth prior, and then compare the color, brightness, texture and depth features of different regions, and fuse them through multiplication or addition and some post-processing techniques to detect salient objects.
[0003] Since the quality of depth maps is easily affected by noise such as sensor temperature, background lighting, and the distance and reflectivity of the observed object, the depth map may have erroneous or missing areas. Similarly, the quality of thermal maps is easily affected by camera noise and sensor noise, resulting in thermal cross-talk or blurred areas in thermal maps. In addition, due to the different lighting conditions in the real environment, similar foreground and background, complex background and other challenging scenes, the acquired RGB images may not provide accurate information. The saliency detection methods described in the above-mentioned related technologies cannot handle the defects of the original image, resulting in low robustness of the saliency detection results.
[0004] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention
[0005] The main purpose of the present invention is to provide a saliency detection method, a saliency detection device and a computer-readable storage medium, aiming to solve the technical problem that the robustness of the saliency detection results in the related art is low.
[0006] To achieve the above object, the present invention provides a saliency detection method, which comprises the following steps:
[0007] Determine a first output result of a first target convolutional layer of the first backbone network and a second output result of a second target convolutional layer of the second backbone network;
[0008] determining the first global descriptor according to the first output result, and determining the second global descriptor according to the second output result;
[0009] Determine the importance of different channels of each modality according to the first global descriptor and the second global descriptor;
[0010] determining a first modal feature according to the importance and the first output result, and determining a second modal feature according to the importance and the second output result;
[0011] A saliency detection result is determined based on the first modal feature and the second modal feature.
[0012] Optionally, before the steps of determining the first global descriptor according to the first output result and determining the second global descriptor according to the second output result, the method further includes:
[0013] Obtaining the length, width, and channel information of the first target convolutional layer and the length, width, and channel information of the second target convolutional layer;
[0014] The steps of determining the first global descriptor according to the first output result, and determining the second global descriptor according to the second output result include:
[0015] Determine the first global descriptor according to the first output result and the length, width, and channel information of the first target convolutional layer;
[0016] The second global descriptor is determined according to the second output result and the length, width, and channel information of the second target convolutional layer.
[0017] Optionally, the step of determining the importance of different channels of each modality according to the first global descriptor and the second global descriptor includes:
[0018] Determine a first cascade result of the first global descriptor and the second global descriptor;
[0019] The importance is determined based on a preset activation function and the first cascade result.
[0020] Optionally, the importance includes a first importance of each channel of the first modality corresponding to the first backbone network, and a second importance of each channel of the second modality corresponding to the second backbone network, and the steps of determining the first modality feature according to the importance and the first output result, and determining the second modality feature according to the importance and the second output result include:
[0021] The first modal feature is determined based on the element-wise product of the first importance and the first output result, and the second modal feature is determined based on the element-wise product of the second importance and the second output result.
[0022] Optionally, before the steps of determining the first modal feature according to the element-wise product of the first importance and the first output result, and determining the second modal feature according to the element-wise product of the second importance and the second output result, the method further includes:
[0023] a second concatenation result between an element-wise product of the first importance and the first output result and an element-wise product of the second importance and the second output result;
[0024] Determining a convolution output of the second cascade result;
[0025] The steps of determining the first modal feature according to the element-wise product of the first importance and the first output result, and determining the second modal feature according to the element-wise product of the second importance and the second output result include:
[0026] determining the first modal feature according to the element-wise product of the first importance and the first output result, and the convolution output; and
[0027] The second modal feature is determined based on the element-wise product of the second importance and the second output result, and the convolution output.
[0028] Optionally, the step of determining a saliency detection result based on the first modal feature and the second modal feature includes:
[0029] When the first target convolution network and the second target convolution network are the last layer of convolution networks, fusing the first modality feature and the second modality feature, and determining the saliency detection result according to the fusion result;
[0030] When the first target convolutional network and the second target convolutional network are non-last convolutional networks, the first modal feature is used as an input of a next convolutional layer of the first target convolutional layer, and the second modal feature is used as an input of a next convolutional layer of the second target convolutional layer, so as to obtain a first output feature of the first backbone network and a second output feature of the second backbone network; and
[0031] The first output feature and the second output feature are fused, and the saliency detection result is determined according to the fusion result.
[0032] Optionally, the step of determining a saliency detection result based on the first modal feature and the second modal feature includes:
[0033] Determine a first decision output of a modality corresponding to a first backbone network according to the first modality feature, and determine a second decision output of a modality corresponding to a second backbone network according to the second modality feature;
[0034] The saliency detection result is determined according to the first decision output and the second decision output.
[0035] Optionally, the step of determining the saliency detection result according to the first decision output and the second decision output includes:
[0036] taking the sum of the elements of the first decision output and the second decision output as a preliminary fusion feature;
[0037] Upsampling the preliminary fusion features to obtain a sampling result;
[0038] Convolution and global pooling are performed on the sampling results, and the saliency detection result is determined according to the convolution and pooling results.
[0039] The present invention also provides a saliency detection device, which includes: a memory, a processor, and a saliency detection program stored in the memory and executable on the processor, wherein the saliency detection program implements the steps of the saliency detection method described above when executed by the processor.
[0040] The embodiment of the present invention proposes a saliency detection method, a saliency detection device and a computer-readable storage medium, first determining the first output result of the first target convolution layer of the first backbone network and the second output result of the second target convolution layer of the second backbone network, then determining the first global descriptor according to the first output result, and determining the second global descriptor according to the second output result, determining the importance of different channels of each modality according to the first global descriptor and the second global descriptor, and then determining the first modality feature according to the importance and the first output result, and determining the second modality feature according to the importance and the second output result, and determining the saliency detection result based on the first modality feature and the second modality feature. The scheme proposed in this embodiment is different from the method of directly using an independent backbone network to extract RGB features and auxiliary modality features, which indiscriminately introduces noise or redundant information. The scheme proposed in this embodiment makes full use of the complementarity between multimodal features and the quality defects of different modal inputs, and extracts RGB features and auxiliary modality features simultaneously through the backbone network feature extraction method with weight sharing, and fully utilizes the complementarity between multimodal features to enhance accurate feature expression, thereby achieving the effect of improving the robustness of saliency detection results. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 It is a schematic diagram of the terminal structure of the hardware operating environment involved in the embodiment of the present invention;
[0042] Figure 2 It is a flowchart of an embodiment of a saliency detection method of the present invention;
[0043] Figure 3 Schematic diagram of the overall architecture of a saliency detection network involved in an embodiment of the present invention;
[0044] Figure 4 A schematic diagram of the data processing flow of FCE involved in an embodiment of the present invention;
[0045] Figure 5 This is a schematic diagram of a detailed flow chart of step S50 in one embodiment of the saliency detection method of the present invention;
[0046] Figure 6 The figure is a schematic diagram of the data processing flow of the detection module involved in the embodiment of the present invention.
[0047] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0048] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0049] like Figure 1 As shown, Figure 1 It is a schematic diagram of the terminal structure of the hardware operating environment involved in the embodiment of the present invention.
[0050] The terminal in the embodiment of the present invention may be a saliency detection device, such as a computer, a PC or a server.
[0051] like Figure 1 As shown, the terminal may include: a processor 1001, an interface 1003, a memory 1004, and a communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The interface 1003 is configured to communicate with other devices or other components. The memory 1004 may be a NAND. The memory 1004 may also be a storage device independent of the aforementioned processor 1001.
[0052] Those skilled in the art will understand that Figure 1 The terminal structure shown in the figure does not constitute a limitation on the terminal, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.
[0053] like Figure 1 As shown, the memory 1004 as a computer storage medium may include a control system, an interface module, and a saliency detection program.
[0054] exist Figure 1 In the terminal shown, the processor 1001 can be used to call the saliency detection program stored in the memory 1004 and perform the following operations:
[0055] Determine a first output result of a first target convolutional layer of the first backbone network and a second output result of a second target convolutional layer of the second backbone network;
[0056] determining the first global descriptor according to the first output result, and determining the second global descriptor according to the second output result;
[0057] Determine the importance of different channels of each modality according to the first global descriptor and the second global descriptor;
[0058] determining a first modal feature according to the importance and the first output result, and determining a second modal feature according to the importance and the second output result;
[0059] A saliency detection result is determined based on the first modal feature and the second modal feature.
[0060] Optionally, in some embodiments, the processor 1001 may also be used to call a saliency detection program stored in the memory 1004 and perform the following operations:
[0061] Obtaining the length, width, and channel information of the first target convolutional layer and the length, width, and channel information of the second target convolutional layer;
[0062] The steps of determining the first global descriptor according to the first output result, and determining the second global descriptor according to the second output result include:
[0063] Determine the first global descriptor according to the first output result and the length, width, and channel information of the first target convolutional layer;
[0064] The second global descriptor is determined according to the second output result and the length, width, and channel information of the second target convolutional layer.
[0065] Optionally, in some embodiments, the processor 1001 may also be used to call a saliency detection program stored in the memory 1004 and perform the following operations:
[0066] Determine a first cascade result of the first global descriptor and the second global descriptor;
[0067] The importance is determined based on a preset activation function and the first cascade result.
[0068] Optionally, in some embodiments, the processor 1001 may also be used to call a saliency detection program stored in the memory 1004 and perform the following operations:
[0069] The first modal feature is determined based on the element-wise product of the first importance and the first output result, and the second modal feature is determined based on the element-wise product of the second importance and the second output result.
[0070] Optionally, in some embodiments, the processor 1001 may also be used to call a saliency detection program stored in the memory 1004 and perform the following operations:
[0071] a second concatenation result between an element-wise product of the first importance and the first output result and an element-wise product of the second importance and the second output result;
[0072] Determining a convolution output of the second cascade result;
[0073] The steps of determining the first modal feature according to the element-wise product of the first importance and the first output result, and determining the second modal feature according to the element-wise product of the second importance and the second output result include:
[0074] determining the first modal feature according to the element-wise product of the first importance and the first output result, and the convolution output; and
[0075] The second modal feature is determined based on the element-wise product of the second importance and the second output result, and the convolution output.
[0076] Optionally, in some embodiments, the processor 1001 may also be used to call a saliency detection program stored in the memory 1004 and perform the following operations:
[0077] When the first target convolution network and the second target convolution network are the last layer of convolution networks, fusing the first modality feature and the second modality feature, and determining the saliency detection result according to the fusion result;
[0078] When the first target convolutional network and the second target convolutional network are non-last convolutional networks, the first modal feature is used as an input of a next convolutional layer of the first target convolutional layer, and the second modal feature is used as an input of a next convolutional layer of the second target convolutional layer, so as to obtain a first output feature of the first backbone network and a second output feature of the second backbone network; and
[0079] The first output feature and the second output feature are fused, and the saliency detection result is determined according to the fusion result.
[0080] Optionally, in some embodiments, the processor 1001 may also be used to call a saliency detection program stored in the memory 1004 and perform the following operations:
[0081] Determine a first decision output of a modality corresponding to a first backbone network according to the first modality feature, and determine a second decision output of a modality corresponding to a second backbone network according to the second modality feature;
[0082] The saliency detection result is determined according to the first decision output and the second decision output.
[0083] Optionally, in some embodiments, the processor 1001 may also be used to call a saliency detection program stored in the memory 1004 and perform the following operations:
[0084] taking the sum of the elements of the first decision output and the second decision output as a preliminary fusion feature;
[0085] Upsampling the preliminary fusion features to obtain a sampling result;
[0086] Convolution and global pooling are performed on the sampling results, and the saliency detection result is determined according to the convolution and pooling results.
[0087] In the related art, multimodal RGB-D (RGB-Depth, color-depth) and RGB-T (RGB-Thermal, color-temperature) salient object detection mainly relies on the fusion of low-level clues. This type of method uses depth saliency or temperature saliency as additional information. That is, the former extracts depth features from the depth map and combines them with features extracted from the RGB (color system) image to obtain a saliency map, and the latter extracts temperature features from the thermal map and combines them with features extracted from the RGB image to obtain a saliency map. For example, in RGB-D salient object detection, the process is generally to calculate the priors in the RGB image and the Depth image, such as background prior, center prior and depth prior, and then compare the color, brightness, texture and depth features of different regions, and fuse them through multiplication or addition and some post-processing techniques to detect salient objects.
[0088] Since the quality of depth maps is easily affected by noise such as sensor temperature, background lighting, and the distance and reflectivity of the observed object, the depth map may have erroneous or missing areas. Similarly, the quality of thermal maps is easily affected by camera noise and sensor noise, resulting in thermal cross-talk or blurred areas in thermal maps. In addition, due to the different lighting conditions in the real environment, similar foreground and background, complex background and other challenging scenes, the acquired RGB images may not provide accurate information. The saliency detection methods described in the above-mentioned related technologies cannot handle the defects of the original image, resulting in low robustness of the saliency detection results.
[0089] In order to solve the above defects, an embodiment of the present invention proposes a saliency detection method, aiming to achieve the effect of improving the robustness of the saliency detection result. For ease of understanding, the saliency detection method proposed by the present invention is explained below through specific implementation methods.
[0090] Reference Figure 2 The present invention provides a first embodiment of a saliency detection method, the method comprising:
[0091] Step S10: determining a first output result of a first target convolutional layer of the first backbone network and a second output result of a second target convolutional layer of the second backbone network;
[0092] Step S20: determining the first global descriptor according to the first output result, and determining the second global descriptor according to the second output result;
[0093] Step S30: determining the importance of different channels of each modality according to the first global descriptor and the second global descriptor;
[0094] Step S40: determining a first modal feature according to the importance and the first output result, and determining a second modal feature according to the importance and the second output result;
[0095] Step S50: determining a saliency detection result based on the first modal feature and the second modal feature.
[0096] In this embodiment, the RGB image to be processed and the auxiliary image may be acquired first, wherein the auxiliary image may be a thermal map or a depth map, etc.
[0097] After obtaining the RGB image and the auxiliary image, if the first backbone network is set as the backbone network for processing RGB images, the RGB image is used as the input of the first backbone network, and the auxiliary image is used as the input of the second backbone network. If the second backbone network is set as the backbone network for processing RGB images, the RGB image is used as the input of the second backbone network, and the auxiliary image is used as the input of the first backbone network.
[0098] For example, please refer to Figure 3 This embodiment provides a saliency detection network, wherein the saliency detection network includes a feature extraction network and a fusion network. The feature extraction network includes a first backbone network and a second backbone network.
[0099] The first backbone network and the second backbone network may include multiple convolutional layers, and the number of convolutional layers included in the first backbone network is the same as the number of convolutional layers included in the second backbone network. For example, the first backbone network and the second backbone network may both include 5 convolutional layers.
[0100] The feature extraction network also includes at least one FCE (Flow collaboration enhancement module). The FCE can be set after any convolutional layer. Figure 3 As shown, in an optional implementation, FCE is set after the second convolutional layer, the third convolutional layer, the fourth convolutional layer and the fifth convolutional layer. It can be understood that this does not limit the feature extraction network given in this embodiment to only set FCE after the second convolutional layer, the third convolutional layer, the fourth convolutional layer and the fifth convolutional layer. In some other implementations, only one FCE can be set after any convolutional layer. Or one FCE is set after each convolutional layer.
[0101] Based on the saliency detection network provided in this embodiment, the first output result of the first target convolutional layer of the first backbone network and the second output result of the second target convolutional layer of the second backbone network can be determined first. The first target convolutional layer and the second target convolutional layer are the corresponding convolutional layers in the first backbone network and the second backbone network, respectively. For example, it can be the second convolutional layer Conv2 of the first backbone network and the second convolutional layer Conv2 of the second backbone network.
[0102] The first output result and the second output result can be input into FCE for processing to obtain the first modal feature and the second modal feature. The processing process of FCE includes first determining the first global descriptor according to the first output result, and determining the second global descriptor according to the second output result, and then determining the importance of different channels of each modality according to the first global descriptor and the second global descriptor, and then determining the first modal feature according to the importance and the first output result, and determining the second modal feature according to the importance and the second output result.
[0103] For example, please refer to Figure 4 , Figure 4 This is a schematic diagram of the FCE architecture. When FCE obtains the first output result X ri And the second output result X ti Afterwards, the first output result X ri Determine the first global descriptor, and according to the second output result X ti A second global descriptor is determined.
[0104] Optionally, when determining the first global descriptor and the second global descriptor, the length, width, and channel information of the first target convolutional layer and the length, width, and channel information of the second target convolutional layer can be first obtained, and then the first global descriptor is determined according to the first output result and the length, width, and channel information of the first target convolutional layer, and the second global descriptor is determined according to the second output result and the length, width, and channel information of the second target convolutional layer.
[0105] In a specific implementation case, the backbone network layer l outputs Xr i , Where H, W, and C represent the length, width, and channel of the lth layer, respectively. First, the global descriptors of the original RGB features and the auxiliary modal features can be calculated, that is, the first global descriptor and the second global descriptor can be calculated. The following takes the first backbone network set to process RGB images and the second backbone network to process auxiliary images as an example for explanation, and the first global descriptor and the second global descriptor are the global descriptors of the original RGB features and the auxiliary modal features, respectively. The calculation method of the global descriptor is as follows:
[0106]
[0107] Where (i, j) represents the position of the coordinates, Represents the spatial statistics of the Cth channel.
[0108] It can be understood that the calculation method of the first global descriptor and the second global descriptor is the same, and the first global descriptor and the second global descriptor can be calculated respectively by changing the parameters corresponding to different networks.
[0109] After calculating the first global descriptor and the second global descriptor, a first concatenation result of the first global descriptor and the second global descriptor may be determined, and then the importance may be determined based on a preset activation function and the first concatenation result.
[0110] For example, in order to enhance the interaction of channel dimensions and make full use of the complementarity between multi-modalities to suppress the noise feature response, the multi-modal global descriptor and Aggregation is achieved through collaborative enhancement layers adaptively learning from each other as follows:
[0111]
[0112] Where ReLU represents the ReLU activation function. represents the cascade operation of the channel dimension. FC1 and FC2 represent the learnable fully connected layers. Therefore, through the above, we can obtain and They represent the importance of different channels of each mode.
[0113] Further, the first modal feature may be determined based on the element-wise product of the first importance and the first output result, and the second modal feature may be determined based on the element-wise product of the second importance and the second output result.
[0114] Exemplarily, the convolution output of the second cascade result can be determined based on the element product of the first importance and the first output result, and the second cascade result between the element product of the second importance and the second output result. The first modal feature is determined based on the element product of the first importance and the first output result, and the convolution output; and the second modal feature is determined based on the element product of the second importance and the second output result, and the convolution output.
[0115] For example, in an optional implementation case, in order to obtain a robust multi-modal feature expression, the original modal feature input is de-calibrated based on the channel importance calculated above, and then further fused, as shown below:
[0116]
[0117]
[0118]
[0119] Among them, ⊙ represents the element-wise multiplication operation, Conv is the convolutional layer, Represents the cascade operation in the channel dimension.
[0120] Optionally, in some implementations, a residual connection is used to make the network easier to train and optimize, as shown below:
[0121]
[0122]
[0123] Enhanced and That is, the first modal feature and the second modal feature mentioned above.
[0124] Optionally, considering that context information is helpful for detecting salient objects of different sizes and positions, a multi-scale flow collaboration enhancement (MFCE) module may be added to the network.
[0125] Among them, MFCE can first calculate the feature embedding of the context, and then use the above FCE to generate enhanced multimodal feature expressions. Specifically, the multi-scale stream writing enhancement module can be expressed as follows:
[0126]
[0127] in is a 4x4 global average pooling operation, and U is a 4x upsampling operation. σ is a Sigmoid activation function. Similarly, each layer is enhanced The input to the next layer is continuously enhanced. As the network deepens, by using MFCE, effective contextual expressions will be highlighted in the encoding stage.
[0128] Optionally, in an optional embodiment, when the first target convolutional network and the second target convolutional network are the last convolutional networks of the feature extraction network, the first modal feature and the second modal feature can be input into a decision fusion network to obtain a saliency detection result through the decision fusion network.
[0129] When the first target convolution network and the second target convolution network are non-last convolution networks, the first modal feature is used as the input of the next convolution layer of the first target convolution layer, and the second modal feature is used as the input of the next convolution layer of the second target convolution layer to obtain a first output feature of the first backbone network and a second output feature of the second backbone network, and then the first output feature and the second output feature are fused, and the saliency detection result is determined according to the fusion result.
[0130] It is understandable that the convolutional layers after the first target convolutional layer and the second target convolutional layer are also provided with FCE, and the above method can be used to process again until the first output feature and the second output feature are obtained. The specific processing method is similar to the above, and will not be repeated here.
[0131] In the technical solution disclosed in this embodiment, the first output result of the first target convolution layer of the first backbone network and the second output result of the second target convolution layer of the second backbone network are first determined, and then the first global descriptor is determined according to the first output result, and the second global descriptor is determined according to the second output result, and the importance of different channels of each modality is determined according to the first global descriptor and the second global descriptor, and then the first modal feature is determined according to the importance and the first output result, and the second modal feature is determined according to the importance and the second output result, and the saliency detection result is determined based on the first modal feature and the second modal feature. The solution proposed in this embodiment is different from the method of directly using an independent backbone network to extract RGB features and auxiliary modal features, which indiscriminately introduces noise or redundant information. The solution proposed in this embodiment makes full use of the complementarity between multimodal features and the quality defects of different modal inputs, and extracts RGB features and auxiliary modal features simultaneously through the backbone network feature extraction method with weight sharing, and fully utilizes the complementarity between multimodal features to enhance accurate feature expression, thereby achieving the effect of improving the robustness of saliency detection results.
[0132] Optionally, see Figure 5 Based on the above embodiment, step S50 includes:
[0133] Step S51: determining a first decision output of a modality corresponding to a first backbone network according to the first modal feature, and determining a second decision output of a modality corresponding to a second backbone network according to the second modal feature;
[0134] Step S52: determining the saliency detection result according to the first decision output and the second decision output.
[0135] In this embodiment, the first decision output is the output of the first backbone network, and the second decision output is the output of the second backbone network. After the first decision output and the second decision output are obtained, the element sum of the first decision output and the second decision output can be used as a preliminary fusion feature, and then the preliminary fusion feature is upsampled to obtain a sampling result, and the sampling result is convolved and globally pooled, and the saliency detection result is determined according to the convolution and pooling results.
[0136] For example, the traditional feature-based fusion strategy depends on the accuracy of the extracted features and is sensitive to noise. If the extracted features have low reliability, directly fusing the noisy features will result in suboptimal results. Different from the feature-based fusion strategy proposed in the above embodiment, a decision-based fusion strategy (DFS) can be designed from the perspective of the reliability of the modal output, aiming to generate more robust and reliable salient object detection results based on the accuracy of the modal output.
[0137] like Figure 3 and Figure 6 As shown, first the RGB features from the RGB backbone network They are sent to a 3x3 convolutional layer (Conv) to obtain features Then is fed into the modality-specific RGB decoder G R Generate decision output S in RGB mode R , as shown below:
[0138] Among them, Conv is a 3x3 convolutional layer. represents the cascade operation of the channel dimension, and U is the upsampling operation. Similarly, the decision output S of the auxiliary mode can be obtained by the following formula: T :
[0139]
[0140] Then we feed the decision outputs of different modalities into a lightweight detection module (DetectionModule, Det) to obtain the final multimodal salient object detection output S f :
[0141] S f =Det(S R , S T )
[0142] like Figure 6 As shown, we can first R and S T Perform element addition operation to obtain preliminary fusion features. Then use bilinear interpolation and deconvolution to upsample the fusion features by 4 times, and then perform 1x1, 3x3, 5x5 convolution and global pooling operations to capture multi-scale multi-modal information. Finally, concatenate the above multi-scale information in the channel dimension, and use a deconvolution to transform the size and channel to the original size of the label to generate the final salient object detection output S f This achieves the effect of further improving the robustness of the saliency detection results.
[0143] The present invention also provides a saliency detection device, which includes: a memory, a processor, and a saliency detection program stored in the memory and executable on the processor, wherein the saliency detection program implements the steps of the saliency detection method described above when executed by the processor.
[0144] The present invention also provides a computer-readable storage medium, on which a saliency detection program is stored. When the saliency detection program is executed by a processor, each step of the saliency detection method described above is implemented.
[0145] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.
[0146] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0147] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a device to execute the methods described in each embodiment of the present invention.
[0148] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A saliency detection method, characterized in that: The saliency detection method comprises the following steps: Using the RGB image as the input of the first backbone network and the auxiliary image as the input of the second backbone network, or using the auxiliary image as the input of the first backbone network and the RGB image as the input of the second backbone network, determining a first output result of the first target convolution layer of the first backbone network and a second output result of the second target convolution layer of the second backbone network, wherein the auxiliary image is a thermal image or a depth image; Obtaining the length, width, and channel information of the first target convolutional layer and the length, width, and channel information of the second target convolutional layer; Determine a first global descriptor according to the first output result and the length, width, and channel information of the first target convolutional layer; Determine a second global descriptor according to the second output result and the length, width, and channel information of the second target convolutional layer; Determine a first cascade result of the first global descriptor and the second global descriptor; Determine the importance of different channels of each modality based on a preset activation function and the first cascade result, wherein the importance includes a first importance of each channel of the first modality corresponding to the first backbone network and a second importance of each channel of the second modality corresponding to the second backbone network; determining a second concatenation result between an element-wise product of the first importance and the first output result, and an element-wise product of the second importance and the second output result; Determining a convolution output of the second cascade result; determining a first modal feature based on an element-wise product of the first importance and the first output result, and the convolution output; determining a second modal feature according to an element-wise product of the second importance and the second output result, and the convolution output; A saliency detection result is determined based on the first modal feature and the second modal feature.
2. The saliency detection method according to claim 1, characterized in that: The step of determining a saliency detection result based on the first modal feature and the second modal feature comprises: When the first target convolution network and the second target convolution network are the last layer of convolution networks, fusing the first modality feature and the second modality feature, and determining the saliency detection result according to the fusion result; When the first target convolution network and the second target convolution network are non-last convolution networks, the first modal feature is used as the input of the next convolution layer of the first target convolution layer, and the second modal feature is used as the input of the next convolution layer of the second target convolution layer to obtain a first output feature of the first backbone network and a second output feature of the second backbone network; and the first output feature and the second output feature are fused, and the saliency detection result is determined according to the fusion result.
3. The saliency detection method according to claim 1, characterized in that: The step of determining a saliency detection result based on the first modal feature and the second modal feature comprises: Determine a first decision output of a modality corresponding to a first backbone network according to the first modality feature, and determine a second decision output of a modality corresponding to a second backbone network according to the second modality feature; The saliency detection result is determined according to the first decision output and the second decision output.
4. The saliency detection method according to claim 3, characterized in that: The step of determining the saliency detection result according to the first decision output and the second decision output comprises: taking the sum of the elements of the first decision output and the second decision output as a preliminary fusion feature; Upsampling the preliminary fusion features to obtain a sampling result; Convolution and global pooling are performed on the sampling results, and the saliency detection result is determined according to the convolution and pooling results.
5. A saliency detection device, characterized in that: The saliency detection device comprises: a memory, a processor, and a saliency detection program stored in the memory and executable on the processor, wherein the saliency detection program implements the steps of the saliency detection method according to any one of claims 1 to 4 when executed by the processor.
6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a saliency detection program, and when the saliency detection program is executed by a processor, the steps of the saliency detection method according to any one of claims 1 to 4 are implemented.