A significant target detection method, system, storage medium and product

Abstract

The present invention relates to the field of image data processing, and in particular to a method, system, storage medium and product for salient target detection, comprising the following steps performed in sequence: S1: acquiring RGB images and depth images for training; S2: establishing backbone networks for extracting features of RGB images and depth images respectively; S3: establishing feature fine-grained enhancement modules for enhancing the feature fine-grainedness of image features processed by step S2 respectively; S4: establishing a cross-modal dual-attention feature fusion module for cross-modal feature enhancement and fusion in the encoder stage; S5: constructing a multi-scale feature aggregation module in the decoder stage to efficiently fuse cross-modal features, enhance feature representation using the correlation between modalities, and decode multi-level features; a cross-modal dual-attention feature fusion module and a multi-scale feature aggregation module are designed to make full use of the complementary information between modalities, thereby significantly improving the detection accuracy in complex scenarios.

Description

Technical Field

[0001] The present invention relates to the field of image data processing, and in particular to a significant object detection method, system, storage medium and product. Background Art

[0002] The current mainstream RGB-D salient object detection methods all directly use the feature information extracted by the backbone network without any processing of the extracted rough features. They directly fuse multimodal information, ignoring the negative impact of low-quality RGB images and depth images on salient object detection, affecting the detection accuracy. Summary of the invention

[0003] The object of the present invention is to provide a significant target detection method, system, storage medium and product with improved detection accuracy.

[0004] In order to achieve the above object, the present invention adopts such technical solution:

[0005] A method for detecting a salient object comprises the following steps performed in sequence:

[0006] S1: Obtain an RGB-D dataset for training and testing. The RGB-D dataset includes the RGB image, depth image, and GT image of each image.

[0007] S2: Establish a first backbone network and a second backbone network, input the RGB images and depth images of the training samples in the RGB-D dataset into the first backbone network and the second backbone network one by one to extract features in multiple dimensions, and use the extracted feature images to extract and Indicates that Represents the dimension,

[0008] S3: Establish feature images for enhancing each dimension one by one A plurality of first feature fine-grained enhancement modules (FFGE1) of feature fine-grainedness are respectively established for one-to-one corresponding enhancement of feature images A plurality of second feature fine-grained enhancement modules (FFGE2) of feature fine-grainedness;

[0009] use Represents each feature image ,use Represents each feature image , each first feature fine-grained enhancement module Perform a global average pooling operation to generate a vector that contains the global information of each channel. Use a one-dimensional convolution to convolve the vector to calculate the attention weight of each channel. The attention weight is scaled by the Sigmoid activation function and Each channel of is multiplied by the corresponding attention weight to adjust the attention weight of the channel to obtain the enhanced feature , the operation of each first feature fine-grained enhancement module is expressed by the following formula (1):

[0010] (1);

[0011] in, is the Sigmoid activation function, is the global average pooling operation, is the convolution operation, is the number of the first feature fine-grained enhancement modules;

[0012] Each second feature fine-grained enhancement module Do the same operation to get the enhanced features , the operation of each second feature fine-grained enhancement module is expressed by the following formula (2):

[0013] (2);

[0014] S4: Establishment A cross-modal dual attention feature fusion module (CMDAFF) for cross-modal feature enhancement and fusion in the encoder stage. And the enhanced features The data are input into each cross-modal dual attention feature fusion module one by one for feature enhancement and feature fusion;

[0015] The specific operation steps of the cross-modal dual attention feature fusion module are as follows:

[0016] S4-1: Enhanced features And the enhanced features Feature reduction is performed through a 1×1 convolutional layer to obtain a preliminary feature representation and ;

[0017] S4-2: Using a Sigmoid activation function The convolutional layer processes the preliminary feature representation to obtain a normalized feature map:

[0018] ;

[0019] in, is the Sigmoid activation function, for Convolutional layer operation;

[0020] S4-3: Feature fusion is completed through element-by-element multiplication operation. The initial feature representation is multiplied with the normalized feature map to generate weighted features, and the weighted features are added to the initial feature representation using residual connection to generate fused features. and :

[0021] ;

[0022] S4-4: Fusion features and Use the channel attention mechanism to determine which complementary information to obtain, highlight key complementary features, suppress irrelevant features, and then use the spatial attention mechanism to determine which part of the feature map to focus on, reduce ambiguity in specific domain features, and obtain cross-enhanced features. and The formula for this step is shown in the following formula (3):

[0023] (3);

[0024] in, For spatial attention, For channel attention;

[0025] S4-5: Cross-enhanced features and Input into the connection layer and convolution layer to form a shared representation and obtain shared features with higher robustness and rich information , this step is shown by the following formula (4):

[0026] (4);

[0027] S5: In the decoder stage, n multi-scale feature aggregation modules (MSFA) are constructed to efficiently fuse cross-modal features, enhance feature representation by using the correlation between modalities, and decode multi-level features separately;

[0028] The specific operations of the multi-scale feature aggregation module are as follows:

[0029] S5-1: Shared features are combined through feature fusion module Respectively and enhanced features And the enhanced features Perform feature fusion:

[0030] ;

[0031] S5-2: In order to reduce the number of channels, the fused features are respectively processed through 1×1 convolution modules and Perform convolution operation to obtain features and , the feature and Perform element-by-element multiplication with the shared features and the corresponding enhanced features and add the multiplication results to obtain the corresponding fused features:

[0032] ;

[0033] S5-3: Fusion features and Perform 3×3 convolution operation, batch normalization operation and Activate the function and combine the processed features with the shared features Add, through Convolution obtains the output features. The formula of this step is as follows:

[0034] ;

[0035] S5-4: Output features and Connect along the channel dimension and pass convolution and batch normalization Processing to obtain the fused feature representation :

[0036] ;

[0037] S5-5: The fused features Shared features Added through residual connections and Activate to get the final output feature :

[0038] .

[0039] Preferably, in step S1, the RGB-D dataset contains a total of 2185 training samples and 2864 test samples, wherein the training samples include 1485 samples from NJU2K and 700 samples from NLPR; the test samples include the remaining 500 samples of NJU2K, the remaining 300 samples of NLPR, 135 samples of DES, 1000 samples of STERE and 929 samples of SIP.

[0040] Preferably, in step S2, the backbone network performs feature extraction in five dimensions, and a Res2Net50 network pre-trained on ImageNet1K is used as the first backbone network and the second backbone network.

[0041] Preferably, a hybrid loss function is constructed to effectively supervise the RGB image, the depth image, and the output of each fusion branch to generate the final saliency map. The binary cross entropy (BCE) loss is used for pixel-level restriction, and the intersection over union (IoU) loss is used for global-level restriction. The following loss function is constructed:

[0042] ;

[0043] in, Indicates the loss of RGB image during processing, Represents the loss of the depth image during processing, Represents the loss of RGB image and depth image in the fusion process,

[0044] use and Represent the output features and ,use Represents output features , Representing the GT image, the overall loss function can be expressed as follows:

[0045] .

[0046] A salient object detection system comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, any of the above method steps is implemented.

[0047] A storage medium comprises a memory, wherein a computer program is stored in the memory, and when the computer program is executed by a processor, any of the method steps described above is implemented.

[0048] A salient object detection product includes a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above method steps.

[0049] By adopting the aforementioned design scheme, the beneficial effects of the present invention are as follows: the present invention designs a feature fine-grained enhancement module to enhance the coarse features of each level extracted by the backbone network, effectively solving the problem that the previous method has poor detection effect when inputting low-quality images. In terms of feature fusion, a cross-modal dual-attention feature fusion module and a multi-scale feature aggregation module are designed to make full use of the complementary information between modalities, which significantly improves the detection accuracy in complex scenarios. These characteristics make the present invention have significant functional advantages and better meet the needs of actual applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flow chart of the method of the present invention;

[0051] Figure 2 It is a structural schematic diagram of a feature fine-grained enhancement module of the present invention;

[0052] Figure 3 Schematic diagram of the structure of the cross-modal dual-attention feature fusion module of the present invention;

[0053] Figure 4 It is a structural schematic diagram of the multi-scale feature aggregation module of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only partial embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] A salient object detection method, such as Figure 1 As shown, the steps include the following steps performed in sequence:

[0056] S1: Obtain an RGB-D dataset for training and testing. The RGB-D dataset includes the RGB image, depth image, and GT image of each image.

[0057] In this embodiment, the RGB-D dataset contains a total of 2185 training samples and 2864 test samples.

[0058] The training samples include 1485 samples from NJU2K and 700 samples from NLPR; the test samples include the remaining 500 samples from NJU2K, the remaining 300 samples from NLPR, 135 samples from DES, 1000 samples from STERE and 929 samples from SIP. Each sample contains RGB images, depth images and GT images.

[0059] S2: Establish a first backbone network and a second backbone network, input the RGB images and depth images of the training samples in the RGB-D dataset into the first backbone network and the second backbone network one by one to extract features in multiple dimensions, and use the extracted feature images to extract and Indicates that Represents dimension.

[0060] In this embodiment, the Res2Net50 network pre-trained on ImageNet1K is used as the first backbone network and the second backbone network.

[0061] S3: Establish feature images for enhancing each dimension one by one The first feature fine-grained enhancement module (FFGE1) of the feature fine-grained Figure 2 As shown, the enhanced feature images are established one by one A plurality of second feature fine-grained enhancement modules (FFGE2) of feature fine-grainedness;

[0062] In this embodiment, each backbone network extracts features in five dimensions, so five first feature fine-grained enhancement modules and five second feature fine-grained enhancement modules are established in step S3 to process feature images of each dimension, such as Figure 1 shown.

[0063] In this embodiment, the Represents each feature image ,use Represents each feature image , each first feature fine-grained enhancement module Perform a global average pooling operation to generate a vector that contains the global information of each channel; use a one-dimensional convolution to convolve the vector to calculate the attention weight of each channel, scale the attention weight through the Sigmoid activation function, and Each channel of is multiplied by the corresponding attention weight to adjust the attention weight of the channel to obtain the enhanced feature , the operation of each first feature fine-grained enhancement module is expressed by the following formula (1):

[0064] (1);

[0065] in, is the Sigmoid activation function, is the global average pooling operation, is the convolution operation, is the number of the first feature fine-grained enhancement modules;

[0066] Each second feature fine-grained enhancement module Do the same operation to get the enhanced features , the operation of each second feature fine-grained enhancement module is expressed by the following formula (2):

[0067] (2);

[0068] S4: Establishment A cross-modal dual attention feature fusion module (CMDAFF) for cross-modal feature enhancement and fusion in the encoder stage, such as Figure 4 As shown, the enhanced features And the enhanced features The data are input into each cross-modal dual attention feature fusion module one by one for feature enhancement and feature fusion;

[0069] The specific operation steps of the cross-modal dual attention feature fusion module are as follows:

[0070] S4-1: Enhanced features And the enhanced features Feature reduction is performed through a 1×1 convolutional layer to obtain a preliminary feature representation and ; In order to capture local contextual information across levels, cross-modal feature enhancement is used to learn the complementarity and correlation between the two modalities.

[0071] S4-2: Using a Sigmoid activation function The convolutional layer processes the preliminary feature representation to obtain a normalized feature map:

[0072] ;

[0073] in, is the Sigmoid activation function, for Convolutional layer operation;

[0074] S4-3: Feature fusion is completed through element-by-element multiplication operation. The initial feature representation is multiplied with the normalized feature map to generate weighted features, and the weighted features are added to the initial feature representation using residual connection to generate fused features. and :

[0075] ;

[0076] Through the operation of this step, the correlation between the two modes can be utilized to effectively enhance the robustness of feature expression and improve the effect of feature fusion.

[0077] S4-4: Fusion features and Use the channel attention mechanism to determine which complementary information to obtain, highlight key complementary features, suppress irrelevant features, and then use the spatial attention mechanism to determine which part of the feature map to focus on, reduce ambiguity in specific domain features, and obtain cross-enhanced features. and The formula for this step is shown in the following formula (3):

[0078] (3);

[0079] in, For spatial attention, For channel attention.

[0080] S4-5: Cross-enhanced features and Input into the connection layer and convolution layer to form a shared representation and obtain shared features with higher robustness and rich information , this step is shown by the following formula (4):

[0081] (4);

[0082] S5: In the decoder stage, n multi-scale feature aggregation modules (MSFA) are constructed to efficiently fuse cross-modal features, such as Figure 3 As shown, the correlation between modalities is used to enhance feature representation, and multi-level features are decoded separately;

[0083] The specific operations of the multi-scale feature aggregation module are as follows:

[0084] S5-1: Shared features are combined through feature fusion module Respectively and enhanced features And the enhanced features Perform feature fusion:

[0085] ;

[0086] S5-2: In order to reduce the number of channels, the fused features are respectively processed through 1×1 convolution modules and Perform convolution operation to obtain features and , the feature and Perform element-by-element multiplication with the shared features and the corresponding enhanced features and add the multiplication results to obtain the corresponding fused features:

[0087] ;

[0088] S5-3: Fusion features and Perform 3×3 convolution operation, batch normalization operation and Activate the function and combine the processed features with the shared features Add, through Convolution obtains the output features. The formula of this step is as follows:

[0089] ;

[0090] S5-4: Output features and Connect along the channel dimension and pass convolution and batch normalization Processing to obtain the fused feature representation :

[0091] ;

[0092] S5-5: The fused features Shared features Added through residual connections and Activate to get the final output feature :

[0093] ;

[0094] Through the above method, the present invention can efficiently fuse cross-model features and enhance feature representation by using the correlation between modalities. In addition, the modality-specific decoder can guide the learning of features that preserve the attributes of the specific modality, and when they are integrated into the fusion decoder, the final prediction results can be better generated. Such a method can improve the performance of the multimodal model, making it more robust and generalizable when handling complex tasks.

[0095] As a preferred method of this embodiment, a hybrid loss function is constructed to effectively supervise the RGB image, the depth image and the output of each fusion branch to generate the final saliency map, and the binary cross entropy (BCE) loss is used for pixel-level restriction, and the intersection over union (IoU) loss is used for global-level restriction to construct the following loss function:

[0096] ;

[0097] in, Indicates the loss of RGB image during processing, Represents the loss of the depth image during processing, Represents the loss of RGB image and depth image in the fusion process,

[0098] use and Represent the output features and ,use Represents output features , Representing the GT image, the overall loss function can be expressed as follows:

[0099] ;

[0100] By calculating the overall loss function, the network model can be better supervised and optimized, making the network model's salient target detection more accurate.

[0101] In this embodiment, a system for implementing the above method steps is also provided.

[0102] A salient object detection system comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, any of the above method steps is implemented.

[0103] In this embodiment, a storage medium storing a computer program capable of implementing the above method steps is also provided.

[0104] A storage medium comprises a memory, wherein a computer program is stored in the memory, and when the computer program is executed by a processor, any of the method steps described above is implemented.

[0105] In this embodiment, a computer product capable of implementing the above method steps is also provided.

[0106] A salient object detection product includes a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above method steps.

[0107] In summary, the present invention designs a feature fine-grained enhancement module to enhance the coarse features at all levels extracted by the backbone network, effectively solving the problem that the previous method has poor detection effect when inputting low-quality images. In terms of feature fusion, a cross-modal dual-attention feature fusion module and a multi-scale feature aggregation module are designed to make full use of the complementary information between modalities, which significantly improves the detection accuracy in complex scenes. These characteristics make the present invention have significant functional advantages and better meet the needs of actual applications.

[0108] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for detecting a salient object, characterized in that: The steps include the following steps: S1: Obtain an RGB-D dataset for training and testing. The RGB-D dataset includes the RGB image, depth image, and GT image of each image. S2: Establish a first backbone network and a second backbone network, input the RGB images and depth images of the training samples in the RGB-D dataset into the first backbone network and the second backbone network one by one to extract features in multiple dimensions, and use the extracted feature images to extract and Indicates that Represents the dimension, S3: Establish feature images for enhancing each dimension one by one A plurality of first feature fine-grained enhancement modules of feature fine-grainedness are respectively established for one-to-one corresponding enhancement of feature images A plurality of second feature fine-grained enhancement modules of feature fine-grainedness; use Represents each feature image ,use Represents each feature image , each first feature fine-grained enhancement module Perform a global average pooling operation to generate a vector that contains the global information of each channel. Use a one-dimensional convolution to convolve the vector to calculate the attention weight of each channel. The attention weight is scaled by the Sigmoid activation function and Each channel of is multiplied by the corresponding attention weight to adjust the attention weight of the channel to obtain the enhanced feature , the operation of each first feature fine-grained enhancement module is expressed by the following formula (1): （1）； in, is the Sigmoid activation function, is the global average pooling operation, is the convolution operation, is the number of the first feature fine-grained enhancement modules; Each second feature fine-grained enhancement module Do the same operation to get the enhanced features , the operation of each second feature fine-grained enhancement module is expressed by the following formula (2): （2）； S4: Establishment A cross-modal dual-attention feature fusion module for cross-modal feature enhancement and fusion in the encoder stage. And the enhanced features The data are input into each cross-modal dual attention feature fusion module one by one for feature enhancement and feature fusion; The specific operation steps of the cross-modal dual attention feature fusion module are as follows: S4-1: Enhanced features And the enhanced features Feature reduction is performed through a 1×1 convolutional layer to obtain a preliminary feature representation and ; S4-2: Using a Sigmoid activation function The convolutional layer processes the preliminary feature representation to obtain a normalized feature map: ； in, is the Sigmoid activation function, for Convolutional layer operation; S4-3: Feature fusion is completed through element-by-element multiplication operation. The initial feature representation is multiplied with the normalized feature map to generate weighted features, and the weighted features are added to the initial feature representation using residual connection to generate fused features. and : ； S4-4: Fusion features and Use the channel attention mechanism to determine which complementary information to obtain, highlight key complementary features, suppress irrelevant features, and then use the spatial attention mechanism to determine which part of the feature map to focus on, reduce ambiguity, and obtain cross-enhanced features. and The formula for this step is shown in the following formula (3): （3）； in, For spatial attention, For channel attention; S4-5: Cross-enhanced features and Input into the connection layer and convolution layer to form a shared representation and obtain shared features with higher robustness and rich information , this step is shown by the following formula (4): （4）； S5: In the decoder stage, n multi-scale feature aggregation modules are constructed to efficiently fuse cross-modal features, enhance feature representation using the correlation between modalities, and decode multi-level features separately; The specific operations of the multi-scale feature aggregation module are as follows: S5-1: Shared features are combined through feature fusion module Respectively and enhanced features And the enhanced features Perform feature fusion: ； S5-2: In order to reduce the number of channels, the fused features are respectively processed through 1×1 convolution modules and Perform convolution operation to obtain features and , the feature and Multiply the shared features and the corresponding enhanced features element by element and add the multiplication results to obtain the corresponding fused features: ； S5-3: Fusion features and Perform 3×3 convolution operation, batch normalization operation and Activate the function and combine the processed features with the shared features Add, through Convolution obtains the output features. The formula of this step is as follows: ； S5-4: Output features and Connect along the channel dimension and pass convolution and batch normalization Processing to obtain the fused feature representation : ； S5-5: The fused features Shared features Added through residual connections and Activate to get the final output feature : 。 2. The salient object detection method according to claim 1, wherein: In step S1, The RGB-D dataset contains a total of 2185 training samples and 2864 test samples, of which the training samples include 1485 samples from NJU2K and 700 samples from NLPR; the test samples include the remaining 500 samples from NJU2K, the remaining 300 samples from NLPR, 135 samples from DES, 1000 samples from STERE, and 929 samples from SIP.

3. The salient object detection method according to claim 1, wherein: In step S2, the backbone network performs feature extraction in five dimensions, and the Res2Net50 network pre-trained on ImageNet1K is used as the first backbone network and the second backbone network.

4. The salient object detection method according to claim 1, wherein: A hybrid loss function is constructed to effectively supervise the RGB image, depth image, and the output of each fusion branch to generate the final saliency map. Binary cross entropy loss is used for pixel-level restriction, and intersection-over-union loss is used for global-level restriction. The following loss function is constructed: ； in, Indicates the loss of RGB image during processing, Represents the loss of the depth image during processing, Represents the loss of RGB image and depth image in the fusion process, use and Represent the output features and ,use Represents output features , Represents the GT image, and the overall loss function is expressed as follows: 。 5. A salient object detection system, comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the method steps described in any one of claims 1 to 4 are implemented.

6. A storage medium, comprising a memory, wherein a computer program is stored in the memory, wherein: When the computer program is executed by a processor, the method steps described in any one of claims 1 to 4 are implemented.

7. A salient object detection product, comprising a computer program, characterized in that: When the computer program is executed by a processor, the method steps described in any one of claims 1 to 4 are implemented.

Images