A method for cooperative salient object detection and storage medium

Abstract

The application provides a kind of synergistic salient object detection method and storage medium, the method is realized by salient feature enhancement and global information guidance, constructs synergistic salient object detection model, in down-sampling network, image feature is extracted by VGG16 backbone network, and the saliency of image feature is enhanced using coordination attention module, and dynamic convolution collaborative search module is used to search common salient object feature as synergistic feature, in up-sampling network, receptive field inflation technology is used to increase receptive field, and long-distance dependence information of image is obtained by non-local module, to optimize synergistic feature, and as the input of global information guidance fusion module, to reduce non-salient background interference. Finally, the whole synergistic salient object detection model is optimized by loss function. The method is fast in operation, and the final synergistic salient object prediction result is complete in structure and accurate in target.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and specifically to a collaborative salient target detection method and storage medium. Background Technology

[0002] Co-saliency object detection is a method that utilizes saliency information to aid object detection. It is widely used in fields such as robot navigation, intelligent video surveillance, industrial inspection, and aerospace. By combining saliency detection and object detection, co-saliency object detection can reduce the search space for object detection, improving efficiency and accuracy. Co-saliency object detection is an important branch of salient object detection and also a crucial preprocessing step for many tasks, such as image segmentation, object recognition, object localization, and tracking. Due to its ability to reduce the consumption of human capital, co-saliency object detection has significant practical implications.

[0003] To date, collaborative salient object detection based on long-distance dependence and feature enhancement still faces several challenges. For example, effectively suppressing background while highlighting collaborative salient objects, effectively fusing salient features at various scales, and improving the completeness of the final prediction results are pressing technical problems that need to be addressed in this field. Summary of the Invention

[0004] The main problems solved by this invention are: how to effectively suppress background and highlight synergistic salient targets, effectively integrate salient features at various scales, and improve the completeness of the final prediction results.

[0005] To achieve the above objectives, this invention proposes a collaborative salient object detection method. It employs a coordinated attention module to enhance the saliency of image features and a dynamic convolutional collaborative search module to search for common salient object features as collaborative features, thereby effectively suppressing background and highlighting collaborative salient objects. The receptive field is increased through receptive field dilation, and long-range dependency information of the image is obtained through a non-local module to optimize collaborative features. This optimized collaborative features serve as global information to guide the input of the fusion module, effectively fusing salient features at various scales. Furthermore, to better optimize the entire neural network and bring it to a minimum convergence point, this invention proposes a multi-level loss function fusion method. Final model testing results show that the proposed model has fast computation speed, and the final salient object prediction results are accurate and structurally complete for collaborative targets.

[0006] According to a first aspect of the present invention, a collaborative salient target detection method includes the following steps:

[0007] Construct a collaborative salient target detection model;

[0008] In the downsampling network of the collaborative salient target detection model, the VGG backbone network branch is used to extract features from the image to be detected, thereby obtaining the image features of each layer;

[0009] The fifth layer image features are processed by a fully connected layer to obtain the fully connected data of the top layer features.

[0010] The fully connected data is sent to the receptive field expansion module to obtain the large receptive field features after receptive field expansion.

[0011] The large receptive field features are fed into the nonlocal module to obtain long-range dependency information, which is then fed into the upsampling network as global information.

[0012] The image features from the third, fourth, and fifth layers of the backbone network are fed into the coordinated attention module to obtain enhanced image features.

[0013] The enhanced image features are fed into the dynamic convolutional collaborative search module to search for common salient object features, which are then fed into the upsampling network as collaborative features.

[0014] The image features of the first and second layers of the backbone network are respectively fed into the position attention module to obtain positional salient features, which are then fed into the upsampling network.

[0015] In the upsampling network of the cooperative salient target detection model, the global information and the cooperative features are fed into the decoding network to obtain prediction maps for each layer;

[0016] The prediction maps, collaborative features, and location saliency features of each layer are fused layer by layer, and guided by global information, to generate a collaborative saliency map prediction result with complete structure and accurate target.

[0017] Furthermore, after the step of fusing the prediction maps, collaborative features, and location saliency features layer by layer, and generating a structurally complete and target-accurate collaborative saliency map prediction result guided by global information, the method further includes:

[0018] Based on the collaborative saliency map prediction results and the ground truth map, six side losses are calculated using the loss function, and the final loss of the fusion layer is used to obtain the final loss function;

[0019] The network parameters of the collaborative salient object detection model are optimized based on the final loss function, and the optimized collaborative saliency map results are obtained.

[0020] Furthermore, the specific expression for the step of performing fully connected processing on the fifth layer image features to obtain the fully connected data of the top layer features is as follows:

[0021] {F1,F2,F3,F4,F5} = VGG(Input)

[0022] F5 c =FC(F5)

[0023] Where VGG represents the VGG backbone network, Input represents the input image data, FC represents the fully connected processing step, {F1,F2,F3,F4,F5} represents the five layers of image features extracted through the VGG backbone network, and F5 represents the last layer of image features extracted through the VGG backbone network. c This represents the fully connected data of the top-level features obtained after fully connected processing of the fifth-layer image features.

[0024] Furthermore, the specific expression for the step of transmitting the fully connected data to the receptive field expansion module to obtain the large receptive field features after receptive field expansion is as follows:

[0025] F5 1 =Relu(BatchNorm2d(conv(F5)) c )))

[0026] F5 2 =DASPP(F5) 1 )

[0027] Among them, F5 1 This indicates F5 for fully connected data. c Preprocessing, DASPP represents the receptive field expansion module, F5 2 This represents the output of the receptive field expansion module, and the specific steps of the receptive field expansion module are as follows:

[0028] f1=Upsample(Relu(BatchNorm2d(conv(AvgPool(f)))))

[0029] f2=Relu(BatchNorm2d(conv2(Relu(BatchNorm2d(conv(f1))))))

[0030] f3=Relu(BatchNorm2d(conv4(Relu(BatchNorm2d(conv(f1))))))

[0031] f4=Relu(BatchNorm2d(conv8(Relu(BatchNorm2d(conv(f1))))))

[0032] f5=conv(torch.cat(f2,f3,f4))

[0033] Where Upsample represents upsampling, conv represents ordinary 3×3 convolution, AvgPool represents average pooling, BatchNorm2d represents normalization, ReLU represents activation function, conv2 represents dilated convolution with stride of 2, conv4 represents dilated convolution with stride of 4, conv8 represents dilated convolution with stride of 8, f represents the input of the receptive field dilation module, f1 represents the intermediate process of f after average pooling, ordinary 3×3 convolution, regularization, ReLU activation function and upsampling, f2, f3 and f4 represent the intermediate results of f1 after ordinary 3×3 convolution, regularization and ReLU activation function, after dilated convolution with stride of 2, dilated convolution with stride of 4 and dilated convolution with stride of 8 respectively, and after regularization and ReLU activation function, f5 represents the final output of the receptive field dilation module after concatenating f2, f3 and f4 and then performing ordinary 3×3 convolution.

[0034] Furthermore, the specific representation of the step of feeding the large receptive field features to the nonlocal module to obtain long-distance dependency information is as follows:

[0035]

[0036] f2 = conv(F5) 2 )

[0037] f3 = softmax(f1) × f2

[0038] F5 3 =W×f3+F5 2

[0039] Where f1, f2, f3 represent the intermediate processing results of the nonlocal module, softmax represents the classification prediction, W represents the fusion weights of the residual module, and F5 represents the fusion weights of the residual module. 3 This indicates the output of long-distance dependency information for nonlocal modules.

[0040] Furthermore, the image features from the third, fourth, and fifth layers of the backbone network are respectively fed into the coordinated attention module to obtain the enhanced image features. The specific expression for this step is as follows:

[0041]

[0042]

[0043] f = δ(BatchNorm([z h ,z w ]))

[0044] g h =σ(conv) h(f h ))

[0045] g w =σ(conv) w (f w ))

[0046]

[0047] Where, x c (h,i) represents the input combination x with height dimension h and different width dimensions i under channel c. c , This represents the weighted sum of these different inputs, i.e., the encoding result along the height dimension h-channel direction; x c (j,w) represents the input combination x with width dimension w and different height dimensions j under channel c. c , This represents the weighted sum of these different inputs, i.e., the encoding result in the width dimension w along the channel direction; z h z represents the encoding result in the height dimension h. w Let z represent the encoding result in the width dimension w; BatchNorm represents the regularization process applied to the combination of these encoding results; δ() represents the non-linear activation function, and f represents the intermediate feature map after regularization and activation; f h and f w conv represents the two components of f along the spatial dimensions in the height and width dimensions. h Tables and conv w They represent the values ​​for f respectively h and f w The convolution operation, where σ() represents the nonlinear activation function; g h and g w f h and f w The result after convolution and non-linear activation function; x c (i,j) represents the channel encoding result under the horizontal coordinate i and the vertical coordinate j. and These represent the attention weights in the height dimension h and width dimension w, respectively, at the horizontal coordinate i and the vertical coordinate j; c (i,j) represents x c (i,j) and The product of these three factors is the final coordinated attention value. Further, the enhanced image features are fed into the dynamic convolutional collaborative search module to search for common salient object features, including:

[0048] The encoder is used to extract the encoded feature map X∈R of each of the enhanced image features. N×H×W×C Where H, W and C represent the height, width and channel number of the encoded feature map X, respectively;

[0049] An adaptive max pooling layer is applied to X with three target scales to obtain output features with spatial sizes of 1×1, 3×3 and 6×6, respectively.

[0050] Flatten and concatenate the output features to generate feature F∈R N×46×C For each image to be detected, the obtained features aggregate the dominant object features at multiple scales, and the common salient object features are searched.

[0051] Furthermore, the loss function consists of the cross-entropy loss function, and the specific calculation formula is as follows:

[0052]

[0053] Where N is the total number of samples, y i This refers to the category of the i-th sample. For the co-saliency target detection task, there are two categories, p. i It is the predicted value of the i-th sample.

[0054] Furthermore, the total loss function is the result of fusing six sides, and the specific calculation formula is as follows:

[0055] Loss=α1×L1+α2×L2+α3×L3+α4×L4+α5×L5+α6×L6

[0056] Where L1, L2, L3, L4, L5, and L6 represent the six side losses, α1, α2, α3, α4, α5, and α6 represent the weight values ​​of different side losses, and Loss represents the final total loss function after fusion.

[0057] According to another aspect of the present invention, a storage medium is provided, the storage medium being a computer-readable storage medium, wherein the aforementioned cooperative salient target detection method is stored in the computer-readable storage medium.

[0058] The technical solution provided by this invention has the following beneficial effects:

[0059] This invention employs a coordinated attention module to enhance the saliency of image features and a dynamic convolutional collaborative search module to search for common salient object features as collaborative features, thereby effectively suppressing background and highlighting collaboratively salient targets.

[0060] This invention increases the receptive field through receptive field dilation technology and obtains long-range dependency information of the image through a non-local module to optimize collaborative features, which are then used as input to the fusion module to guide global information, thereby effectively fusing salient features at various scales.

[0061] Furthermore, in order to better optimize the entire neural network and make it converge to the minimum point, this invention proposes a multi-level loss function fusion method.

[0062] The final model test results show that the model proposed in this invention has a fast computation speed, and the final salient target prediction results are accurate and structurally complete. Attached Figure Description

[0063] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0064] Figure 1 The following is an execution flowchart of a collaborative salient target detection method provided in an embodiment of the present invention;

[0065] Figure 2 This is a schematic diagram of a collaborative salient target detection model structure provided in an embodiment of the present invention;

[0066] Figure 3 This is a schematic diagram of the receptive field expansion module structure provided in an embodiment of the present invention;

[0067] Figure 4 This is a schematic diagram of a non-local module provided in an embodiment of the present invention;

[0068] Figure 5 A schematic diagram of the coordinated attention module provided in an embodiment of the present invention;

[0069] Figure 6 This is a schematic diagram of the dynamic convolutional collaborative search module structure provided in an embodiment of the present invention;

[0070] Figure 7 The figure shows the test results of the significant target detection method provided in the embodiments of the present invention.

[0071] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0072] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0073] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0074] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0075] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0076] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0077] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0078] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0079] See Figure 1 and Figure 2 This invention provides a collaborative salient target detection method based on long-distance dependency and feature enhancement. The method is implemented through salient feature enhancement and global information guidance, and specifically includes the following steps:

[0080] S1: Construct a collaborative salient target detection model;

[0081] S2: In the downsampling network for constructing the collaborative salient target detection model, the VGG backbone network branch is used to extract features from the image to be detected, and the image features of each layer are obtained.

[0082] S3: Perform fully connected processing on the fifth layer image features to obtain the fully connected data of the top layer features;

[0083] Specifically, refer to Figure 2 The image features extracted from each layer of the VGG backbone network in the collaborative salient object detection model all play a role in collaborative salient object detection. Therefore, the features of each layer are extracted to facilitate subsequent processing. The specific calculation process is as follows:

[0084] {F1,F2,F3,F4,F5} = VGG(Input)

[0085] F5 c =FC(F5)

[0086] Where VGG represents the VGG backbone network, Input represents the input image data, FC represents the fully connected processing step, {F1,F2,F3,F4,F5} represents the five layers of image features extracted through the VGG backbone network, and F5 represents the last layer of image features extracted through the VGG backbone network. c This represents the fully connected data of the top-level features obtained after fully connected processing of the fifth-layer image features.

[0087] As a preferred embodiment, VGG16 is used as the backbone network for salient object detection.

[0088] S4: The fully connected data is sent to the receptive field expansion module to obtain the large receptive field features after receptive field expansion;

[0089] Specifically, refer to Figure 3 Dense, hollow spatial pyramid pooling is used to increase the receptive field of the top-level fully connected data to obtain more information and improve the accuracy of network recognition. The resulting output is a sample of inputs with receptive fields of different proportions. The specific expression for the step of feeding the fully connected data to the receptive field dilation module to obtain the large receptive field features after dilation is as follows:

[0090] F5 1 =Relu(BatchNorm2d(conv(F5)) c )))

[0091] F5 2 =DASPP(F5) 1 )

[0092] Among them, F5 1 This indicates F5 for fully connected data. c Preprocessing, DASPP represents the receptive field expansion module, F5 2 This represents the output of the receptive field expansion module. The specific steps of the receptive field expansion module are as follows:

[0093] f1=Upsample(Relu(BatchNorm2d(conv(AvgPool(f)))))

[0094] f2=Relu(BatchNorm2d(conv2(Relu(BatchNorm2d(conv(f1))))))

[0095] f3=Relu(BatchNorm2d(conv4(Relu(BatchNorm2d(conv(f1))))))

[0096] f4=Relu(BatchNorm2d(conv8(Relu(BatchNorm2d(conv(f1))))))

[0097] f5=conv(torch.cat(f2,f3,f4))

[0098] Where Upsample represents upsampling, conv represents ordinary 3×3 convolution, AvgPool represents average pooling, BatchNorm2d represents normalization, ReLU represents activation function, conv2 represents dilated convolution with stride of 2, conv4 represents dilated convolution with stride of 4, conv8 represents dilated convolution with stride of 8, f represents the input of the receptive field dilation module, f1 represents the intermediate process of f after average pooling, ordinary 3×3 convolution, regularization, ReLU activation function and upsampling, f2, f3 and f4 represent the intermediate results of f1 after ordinary 3×3 convolution, regularization and ReLU activation function, after dilated convolution with stride of 2, dilated convolution with stride of 4 and dilated convolution with stride of 8 respectively, and after regularization and ReLU activation function, f5 represents the final output of the receptive field dilation module after concatenating f2, f3 and f4 and then performing ordinary 3×3 convolution.

[0099] S5: The large receptive field features are fed into the nonlocal module to obtain long-distance dependency information, and then fed into the upsampling network as global information;

[0100] Specifically, refer to Figure 4 This invention also proposes a method for removing the spatiotemporal dimension from three-dimensional nonlocal modules, making it applicable to 2D image convolutional networks. This method can improve the efficiency and accuracy of the model while maintaining the original basic idea. The specific calculation process is as follows:

[0101]

[0102] f2 = conv(F5) 2 )

[0103] f3 = softmax(f1) × f2

[0104] F5 3 =W×f3+F5 2

[0105] Where f1, f2, and f3 represent the intermediate processing results of the nonlocal module, softmax represents the classification prediction, and W represents the fusion weights of the residual module. F5 3 This indicates the output of long-distance dependency information for nonlocal modules.

[0106] S6: The image features of the third, fourth and fifth layers of the backbone network are respectively fed into the coordinated attention module to obtain enhanced image features;

[0107] Specifically, refer to Figure 5 The coordinated attention processing flow consists of two steps: coordinated information embedding and coordinated attention generation. In the coordinated information embedding stage, the attention encodes channel relationships to obtain information about the correlations between channels. In the coordinated attention generation stage, the attention enhances the expressive power of features by encoding long-range dependencies. In this way, coordinated attention can help improve the learning and expressive power of features in the network, thereby more accurately capturing the key features of the input data. The specific calculation process is as follows:

[0108]

[0109]

[0110] f = δ(BatchNorm([z h ,z w ]))

[0111] g h =σ(conv) h (f h ))

[0112] g w =σ(conv) w (f w ))

[0113]

[0114] Where, x c (h,i) represents the input combination x with height dimension w and different width dimensions i under channel c. c , This represents the weighted sum of these different inputs, i.e., the encoding result along the height dimension h-channel direction; x c (j,w) represents the input combination x with width dimension w and different height dimensions j under channel c. c , This represents the weighted sum of these different inputs, i.e., the encoding result in the width dimension w along the channel direction; z h z represents the encoding result in the height dimension w. w Let z represent the encoding result in the width dimension w; BatchNorm represents the regularization process applied to the combination of these encoding results; δ() represents the non-linear activation function, and f represents the intermediate feature map after regularization and activation; f h and f wconv represents the two components of f along the spatial dimensions in the height and width dimensions. h Tables and conv w They represent the values ​​for f respectively h and f w The convolution operation, where σ() represents the nonlinear activation function; g h and g w f h and f w The result after convolution and non-linear activation function; x c (i,j) represents the channel encoding result under the horizontal coordinate i and the vertical coordinate j. and These represent the attention weights in the height dimension h and width dimension w, respectively, at the horizontal coordinate i and the vertical coordinate j; c (i,j) represents x c (i,j) and The product of these three factors is the final value for coordinated attention.

[0115] S7: The enhanced image features are fed into the dynamic convolutional collaborative search module to search for common salient object features, which are then fed into the upsampling network as collaborative features.

[0116] Specifically, refer to Figure 6 The encoder is used to extract their encoded feature maps X∈R N×H×W×C Here, H, W, and C represent its height, width, and channel number, respectively. Adaptive max-pooling layers are applied to X with three target scales, yielding output features of spatial sizes of 1×1, 3×3, and 6×6, respectively. These output features are then flattened and concatenated to generate features F∈R. N×46×C For each image, the obtained features aggregate dominant object features across multiple scales, thus exhibiting robustness to changes in the location and scale of common salient objects.

[0117] S8: The image features of the first and second layers of the backbone network are respectively fed into the position attention module to obtain positional salient features, and then fed into the upsampling network;

[0118] S9: In the upsampling network of the collaborative salient target detection model, the global information and the collaborative features are fed into the decoding network to obtain prediction maps for each layer;

[0119] S10: The prediction maps, collaborative features, and location saliency features of each layer are fused layer by layer, and guided by global information, a collaborative saliency map prediction result with complete structure and accurate target is generated.

[0120] To better train the entire network, the loss function of this invention consists of the cross-entropy loss function, and the specific calculation formula is as follows:

[0121]

[0122] Where N is the total number of samples, y i is the category to which the i-th sample belongs. For the co-saliency target detection task, it is divided into two categories. p i It is the predicted value of the i-th sample.

[0123] Furthermore, the total loss function is the result of fusing six sides, and the specific calculation formula is as follows:

[0124] Loss=α1×L1+α2×L2+α3×L3+α4×L4+α5×L5+α6×L6

[0125] Where L1, L2, L3, L4, L5, and L6 represent the six side losses, α1, α2, α3, α4, α5, and α6 represent the weight values ​​of different side losses, and Loss represents the final fused loss function.

[0126] Reference Figure 7 The figure shows the test results of the collaborative salient target detection method of the present invention. It can be intuitively seen from the test results that the final collaborative salient target prediction results of the model proposed in this invention are accurate and structurally complete.

[0127] Reference Figure 8This paper illustrates a schematic diagram of the physical structure of an electronic device, which may include a processor 610, a communication interface 620, a memory 630, and a communication bus 640. The processor 610, communication interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute the steps of the aforementioned cooperative salient object detection method, specifically including: constructing a cooperative salient object detection model; in the downsampling network of the cooperative salient object detection model, using a VGG backbone network branch to extract features from the image to be detected, obtaining image features at each layer; performing fully connected processing on the fifth layer image features to obtain fully connected data of the top layer features; sending the fully connected data to the receptive field dilation module to obtain the large receptive field features after receptive field dilation; sending the large receptive field features to the nonlocal module to obtain long-distance dependency information, and sending it as global information to the upsampling network; and processing the third, fourth, and fifth layer image features of the backbone network. The enhanced image features are fed into the coordinated attention module to obtain enhanced image features. These enhanced image features are then fed into the dynamic convolutional collaborative search module to search for common salient object features, which are then fed into the upsampling network as collaborative features. The first and second layer image features of the backbone network are fed into the positional attention module to obtain positional salient features, which are then fed into the upsampling network. In the upsampling network of the collaborative salient object detection model, the global information and the collaborative features are fed into the decoding network to obtain prediction maps for each layer. The prediction maps, collaborative features, and positional salient features are then fused layer by layer, and guided by global information, a structurally complete and target-accurate collaborative salient map prediction result is generated.

[0128] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0129] In another aspect, embodiments of the present invention also provide a storage medium storing a computer program, which, when executed by a processor, implements the steps of the aforementioned cooperative salient target detection method, specifically including: constructing a cooperative salient target detection model; in the downsampling network of the cooperative salient target detection model, using a VGG backbone network branch to extract features from the image to be detected, obtaining image features of each layer; performing fully connected processing on the fifth layer image features to obtain fully connected data of the top layer features; sending the fully connected data to a receptive field dilation module to obtain large receptive field features after receptive field dilation; sending the large receptive field features to a nonlocal module to obtain long-distance dependency information, and sending it as global information to the upsampling network; and sending the backbone network... The third, fourth, and fifth layer image features are fed into the coordinating attention module to obtain enhanced image features. These enhanced image features are then fed into the dynamic convolutional collaborative search module to search for common salient object features, which are then fed into the upsampling network as collaborative features. The first and second layer image features of the backbone network are fed into the positional attention module to obtain positional salient features, which are then fed into the upsampling network. In the upsampling network of the collaborative salient object detection model, the global information and the collaborative features are fed into the decoding network to obtain prediction maps for each layer. The prediction maps, collaborative features, and positional salient features are then fused layer by layer, and guided by global information, a structurally complete and target-accurate collaborative salient map prediction result is generated.

[0130] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0131] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

Claims

1. A method of synergistic salient object detection, characterized in that, The synergistic salient object detection method comprises the following steps: constructing a synergistic salient object detection model; in the down-sampling network of the synergistic salient object detection model, a VGG backbone network branch is used to extract features of the image to be detected, and image features of each layer are obtained; full connection processing is performed on the fifth layer image features to obtain full connection data of the top layer features; the full connection data is sent to a receptive field inflation module to obtain large receptive field features after receptive field inflation; the large receptive field features are sent to a non-local module to obtain long-distance dependency information and are sent to the up-sampling network as global information; the third, fourth and fifth layer image features of the backbone network are sent to a coordination attention module respectively to obtain enhanced image features; the enhanced image features are sent to a dynamic convolution synergistic search module respectively to search for common salient object features and are sent to the up-sampling network as synergistic features; the first and second layer image features of the backbone network are sent to a position attention module respectively to obtain position salient features and are sent to the up-sampling network; in the up-sampling network of the synergistic salient object detection model, the global information and the synergistic features are sent to a decoding network to obtain prediction maps of each layer; each layer prediction map, each layer synergistic feature and position salient feature are fused layer by layer, and a structure complete and accurate salient map prediction result is generated through global information guidance; the specific expression of the step of sending the third, fourth and fifth layer image features of the backbone network to the coordination attention module respectively to obtain enhanced image features is as follows: wherein, represents the combination of inputs in the channel direction with different height dimensions and different width dimensions , , represents the weighted combination of these different inputs, i.e. the encoding result in the height dimension channel direction; represents the combination of inputs in the channel direction with different width dimensions and different height dimensions , , represents the weighted combination of these different inputs, i.e. the encoding result in the width dimension channel direction; represents the encoding result in the height dimension , , represents the encoding result in the width dimension ; ; represents the regularization of the combination of these encoding results; represents a nonlinear activation function, represents the intermediate feature map after regularization and activation function; and are two components along the spatial dimension in the height and width dimensions, and respectively represent the convolution operation on and , represents a nonlinear activation function; and represent and the results after convolution and nonlinear activation function; represents the channel encoding result in the horizontal coordinate and vertical coordinate , and respectively represent the attention weight in the height dimension and width dimension direction in the horizontal coordinate and vertical coordinate ; represents , and the product of the three, i.e. the final coordinated attention value.

2. The method of claim 1, wherein, after the step of fusing each layer prediction map, each layer synergistic feature and position salient feature layer by layer and generating a structure complete and accurate salient map prediction result through global information guidance, the following steps are further included: six side loss is calculated through a loss function according to the salient map prediction result and a true value map, and a final loss function is obtained by fusing layer final loss; the network parameters of the synergistic salient object detection model are optimized according to the final loss function, and an optimized salient map result is finally obtained.

3. The method of claim 1, wherein, the specific expression of the step of performing full connection processing on the fifth layer image features to obtain full connection data of the top layer features is as follows: wherein, represents a VGG backbone network, represents input image data, represents a fully connected processing step, represents five layers of image features extracted by the VGG backbone network, represents the last layer of image features extracted by the VGG backbone network, represents fully connected data of the top layer features obtained after fully connecting the fifth layer of image features.

4. The method of claim 1, wherein, the specific expression of the step of sending the full connection data to the receptive field inflation module to obtain large receptive field features after receptive field inflation is as follows: wherein, denotes a pre-processing of fully connected data , denotes the receptive field inflation module, denotes the output of the receptive field inflation module, the specific representation of the step of the receptive field inflation module being: wherein, denotes an up-sampling operation, denotes a normal convolution, denotes a pooling, denotes a normalization, denotes an activation function, denotes a dilated convolution with a stride of 2, denotes a dilated convolution with a stride of 4, denotes a dilated convolution with a stride of 8, denotes the input of the receptive field expansion module, denotes the intermediate result after the average pooling, the normal convolution, the regularization, the activation function and the up-sampling, , and denote the intermediate result after the normal convolution, the regularization and the activation function, respectively, followed by the dilated convolution with a stride of 2, the dilated convolution with a stride of 4 and the dilated convolution with a stride of 8, respectively, followed by the regularization and the activation function, denotes the output of the final receptive field expansion module after concatenating , and and then performing the normal convolution.

5. The method of claim 1, wherein, the specific expression of the step of sending the large receptive field features to the non-local module to obtain long-distance dependency information is as follows: wherein, represents an output of the receptive field expansion module, represents an intermediate processing result of the non-local module, represents a classification prediction, represents a fusion weight of the residual module, represents a long-range dependency information output of the non-local module.

6. The method of claim 1, wherein, the step of sending the enhanced image features to the dynamic convolution synergistic search module respectively to search for common salient object features comprises: extracting encoded feature maps of the respective enhanced image features using an encoder wherein H , W and C respectively represent the height, width and channel number of the encoded feature maps ​ In the case of three target scales An adaptive max-pooling layer is used above and output features with spatial sizes of 1x1, 3x3 and 6x6 are obtained, respectively; each output feature is flattened and concatenated to generate a feature For each image to be detected, the obtained features aggregate dominant object features at multiple scales, and the search obtains common salient object features.

7. The method of claim 2, wherein, the loss function is composed of a cross entropy loss function, and the specific calculation formula is as follows: wherein, N is the total number of samples, is the class to which the th sample belongs, for the co-saliency object detection task, is divided into two classes, is the predicted value of the th sample.

8. The method of claim 2, wherein, the final loss function is the six side fusion results, and the specific calculation formula is as follows: wherein, respectively represent six side loss, respectively represent weight values of different side loss, represents the final fused total loss function.

9. A storage medium, characterized by the storage medium is a computer readable storage medium, and the computer readable storage medium stores the synergistic salient object detection method of any one of claims 1-8.

Images