A salient target detection method based on a multi-flow feedback network framework

Abstract

This invention discloses a salient object detection method based on a multi-stream feedback network framework, comprising: acquiring the original image to be detected; inputting the original image into a trained multi-stream feedback saliency prediction model to obtain a saliency prediction result; wherein the prediction model includes a prediction network with shared parameters across multiple streams, and the prediction method is as follows: except for the first-stream prediction network, each subsequent stream prediction network enhances the saliency features of the original image based on the prediction map of the previous stream prediction network to obtain an enhanced image; after feature extraction, aggregation, convolution, and upsampling processing of the enhanced image, prediction maps of each stream prediction network are obtained; the prediction map of the last stream is used as the final saliency prediction result. This method utilizes the features of the enhanced original image from the prediction maps of previous streams as input to the current stream prediction network, and the parameters of each stream network are shared, thus improving the accuracy of salient object detection while maintaining high detection efficiency.

Description

Technical Field

[0001] This invention relates to a salient target detection method based on a multi-flow feedback network framework, belonging to the field of image salient target detection technology. Background Technology

[0002] Saliency target detection aims to detect the most eye-catching complete target or region in an image. As an important preprocessing step, it is widely used in many image processing and computer vision tasks, and its research has important theoretical significance and practical value.

[0003] Saliency detection methods can be broadly categorized into two types: traditional handcrafted feature-based methods and deep learning-based methods. Traditional handcrafted feature-based methods utilize features such as brightness, color, and texture contrast in images to predict saliency, but their accuracy is low for complex scene images. Deep learning-based methods automatically extract features through deep neural networks, achieving higher detection accuracy. However, when faced with complex scene images, such as when the target and background have similar brightness, color, or other appearance features, or when the background texture is complex, the saliency detection results are still unsatisfactory. This is mainly due to insufficient understanding or utilization of the high-level semantic information of complex images. Some methods attempt to improve detection accuracy by designing deeper or wider complex network structures, but this leads to more network parameters and computational load, thus reducing detection efficiency.

[0004] Therefore, it is necessary to design a suitable deep network model for salient target detection that has high detection accuracy while having small network parameters and computational cost, so as to be better applied to downstream tasks.

[0005] The information disclosed in this background section is intended only to enhance the understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a salient target detection method based on a multi-stream feedback network framework. Compared with general detection methods based on single-stream, non-feedback network structures, the multi-stream feedback saliency prediction model of this invention makes better use of high-level semantic information, thereby improving the accuracy of saliency detection. The shared parameters of the multi-stream prediction network keep the overall model relatively small in terms of parameters and computational cost, making it easy to deploy in practical application systems.

[0007] To achieve the above objectives, the present invention is implemented using the following technical solution:

[0008] This invention discloses a salient target detection method based on a multi-flow feedback network framework, comprising the following steps:

[0009] Obtain the original image to be detected;

[0010] The original image is input into the trained multi-stream feedback saliency prediction model to obtain the saliency prediction result corresponding to the original image;

[0011] The multi-stream feedback saliency prediction model includes a prediction network with shared multi-stream parameters, and the specific prediction method is as follows:

[0012] The first-level prediction network performs feature extraction, aggregation, convolution, and upsampling on the original image to obtain a prediction map.

[0013] In addition to the first-stream prediction network, each subsequent-stream prediction network enhances the original image with saliency features based on the prediction map of the previous-stream prediction network to obtain an enhanced image; after feature extraction, aggregation, convolution and upsampling processing of the enhanced image, the prediction map of each-stream prediction network is obtained.

[0014] The prediction graph from the last-order prediction network is used as the final saliency prediction result corresponding to the original image.

[0015] Furthermore, each stream prediction network includes a feature extraction module, a channel attention module, a feature aggregation module, a convolutional layer, and an upsampling module, and the parameters of each stream prediction network are shared; in addition to the first stream prediction network, subsequent stream prediction networks also include a saliency enhancement module.

[0016] Furthermore, the feature extraction module adopts a ResNet-50 network structure, including five sequentially connected convolutional blocks, to obtain multi-level feature extraction of the original image or the enhanced image. The expression for the extracted features is as follows:

[0017] {F n |n=1,2,3,4,5}

[0018] In the formula, n represents the sequential index of the convolutional block; F n This represents the extracted features of the nth convolutional block, with a corresponding feature resolution of . H and W represent the height and width of the image, respectively, and C n Indicates the extraction of features F n The number of channels.

[0019] Furthermore, the channel attention module is used to obtain a channel-weighted feature F5' based on the extracted feature F5 from the 5th convolutional block by capturing the important responses of the feature in the channel dimension. The expression of the channel-weighted feature F5' is as follows:

[0020]

[0021] In the formula, F'5 represents the channel-weighted feature; F5 represents the extracted feature of the 5th convolutional block; δ represents element-wise addition; Conv_1 represents 1×1 convolution; δ represents ReLU activation; GAP represents global average pooling; ⊙ represents element-wise multiplication.

[0022] Furthermore, the feature aggregation module includes three structurally identical feature aggregation units connected in sequence. The feature aggregation unit is used to aggregate channel-dimensional weighted features or the aggregated features of the previous feature aggregation unit, and combine them with the extracted features of the corresponding convolutional block to obtain the aggregated features of the current feature aggregation unit.

[0023] Furthermore, the aggregation steps of the feature aggregation unit are as follows:

[0024] The aggregated channel dimension weighted features or the aggregated features of the previous feature aggregation unit are used as decoding features, and the extracted features of the corresponding convolutional block are used as encoding features.

[0025] After applying 3×3 convolutions to the decoded features and encoded features respectively, element-wise additive fusion and spatial attention operations are performed to obtain a feature space weight map;

[0026] Based on the feature space weight map, the decoded features and encoded features after two 3×3 convolutions are weighted by spatial dimension using element-wise multiplication to obtain spatially weighted decoded features and encoded features.

[0027] The spatially weighted decoded and encoded features are concatenated by channels, and then refined again using 3×3 convolution to obtain the aggregated features of the current feature aggregation unit.

[0028] Furthermore, the expression for the enhanced image is as follows:

[0029]

[0030] In the formula, I sal_enh Indicates an enhanced image; I input Represents the original image; S i-1 This represents the prediction graph corresponding to the (i-1)th flow prediction network; ⊙ indicates element-wise addition; ⊙ indicates element-wise multiplication.

[0031] Furthermore, the training method for the multi-stream feedback saliency prediction model is as follows:

[0032] Obtain the training images and their corresponding ground truth images;

[0033] The training images are input into a pre-built multi-stream feedback saliency prediction model to obtain the prediction map of each stream prediction network;

[0034] Based on the ground truth image, the prediction graphs of each flow prediction network are jointly supervised. Based on the binary cross-entropy loss, the overall loss function of the multi-flow feedback saliency prediction model is constructed. The network parameters of the multi-flow feedback saliency prediction model are adjusted through gradient backpropagation, and finally the trained multi-flow feedback saliency prediction model is obtained.

[0035] Furthermore, the expression for the overall loss function is as follows:

[0036]

[0037] In the formula, L total Represents the overall loss function; i represents the order of the prediction network; N represents the total number of prediction networks; L bce (S i G) represents the prediction graph S corresponding to the i-th flow prediction network. i The binary cross-entropy loss; G represents the ground truth image.

[0038] Furthermore, the expression for the binary cross-entropy loss function is as follows:

[0039]

[0040] In the formula, L bce (S i G) represents the prediction graph S corresponding to the i-th flow prediction network. i Binary cross-entropy loss; H 真 W represents the height of the truth image; 真 The width of the ground truth image is represented by p; the first coordinate of the pixel is represented by q; and G represents the second coordinate of the pixel. pq S represents the value of the true image at pixel (p, q); pq This represents the corresponding value of the predicted image at pixel (p, q).

[0041] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0042] The salient target detection method based on a multi-stream feedback network framework of this invention, compared with general detection methods based on a single-stream, non-feedback network structure, provides a multi-stream feedback saliency prediction model that better utilizes high-level semantic information, thereby improving saliency detection accuracy. The shared parameters of the multi-stream prediction network keep the overall model relatively small in terms of parameters and computational cost, making it easy to deploy in practical application systems.

[0043] This invention employs a multi-stream joint supervision approach during training, adjusting the overall multi-stream prediction network parameters through gradient backpropagation, thereby optimizing the overall network's learning efficiency and making the overall model easier to train. Attached Figure Description

[0044] Figure 1 This is a flowchart of the salient target detection method based on a multi-stream feedback network framework provided in the embodiment;

[0045] Figure 2 This is a schematic diagram of the prediction network structure provided in the embodiment;

[0046] Figure 3 This is a schematic diagram of the channel attention module provided in the embodiment;

[0047] Figure 4 This is a schematic diagram of the structure of the feature aggregation unit provided in the embodiment. Detailed Implementation

[0048] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0049] This embodiment provides a salient target detection method based on a multi-stream feedback network framework, including the following steps:

[0050] Obtain the original image to be detected;

[0051] The original image is input into the trained multi-stream feedback saliency prediction model to obtain the saliency prediction result corresponding to the original image;

[0052] The multi-stream feedback significance prediction model includes a prediction network with shared multi-stream parameters, and the specific prediction method is as follows:

[0053] The first-class prediction network performs feature extraction, aggregation, convolution, and upsampling on the original image to obtain the prediction map;

[0054] In addition to the first-level prediction network, subsequent prediction networks enhance the original image with saliency features based on the prediction map of the previous prediction network to obtain an enhanced image; after feature extraction, aggregation, convolution and upsampling of the enhanced image, the prediction map of each prediction network is obtained.

[0055] The prediction graph from the last-order prediction network is used as the final saliency prediction result corresponding to the original image.

[0056] The technical concept of this invention is as follows: Compared with general detection methods based on single-stream, feedback-free network structures, the provided multi-stream feedback saliency prediction model makes better use of high-level semantic information, thereby improving the accuracy of saliency detection. The shared parameters of the multi-stream prediction network keep the overall model relatively small in terms of parameters and computational cost, making it easy to deploy in practical application systems.

[0057] like Figure 1 and Figure 2 As shown, the multi-stream feedback saliency prediction model includes a prediction network with shared parameters for each stream. Each stream prediction network has the same structure, including a feature extraction module, a channel attention module, a feature aggregation module, a convolutional layer, and an upsampling module. The parameters of each stream prediction network are shared. In addition to the first stream prediction network, each subsequent stream prediction network also includes a saliency enhancement module.

[0058] The first-level prediction network takes the original image with 3 channels as input, performs multi-level feature extraction and aggregation, and obtains a saliency prediction map S1 with 1 channel through 1×1 convolution and upsampling;

[0059] Subsequent flow prediction networks first use the prediction maps obtained from the previous flow prediction network to enhance the saliency features of the original image, then perform multi-level feature extraction and aggregation, and obtain the predicted saliency map through 1×1 convolution and upsampling. Let N be the number of flows in the prediction network, then the prediction map S of the Nth flow is... N This represents the final saliency prediction result corresponding to the original image.

[0060] like Figure 2 As shown, the feature extraction module adopts a ResNet-50 network structure, including 5 sequentially connected convolutional blocks, used to obtain multi-level feature extraction of the original image or the enhanced image. The expression for feature extraction is as follows:

[0061] {F n |n=1,2,3,4,5}

[0062] In the formula, n represents the sequential index of the convolutional block; F n This represents the extracted features of the nth convolutional block, with a corresponding feature resolution of . H and W represent the height and width of the image, respectively, and C n Indicates the extraction of features F n The number of channels, with values ​​of {C} n |64,256,512,1024,2048}.

[0063] like Figure 2 and Figure 3 As shown, the channel attention module is located after the ResNet-50-based feature extraction module. It is used to obtain the channel-weighted feature F5' by capturing the important responses of the features in the channel dimension based on the extracted feature F5 from the 5th convolutional block. Specifically, the extracted feature F5 is first subjected to global average pooling and a 1×1 convolution, followed by ReLU activation and another 1×1 convolution to obtain channel-level weights. These weights are then multiplied by the extracted feature F5, and the result is added back to the extracted feature F5 to obtain the channel-weighted feature F5', expressed by the formula:

[0064]

[0065] In the formula, F'5 represents the channel-weighted feature; F5 represents the extracted feature of the 5th convolutional block; δ represents element-wise addition; Conv_1 represents 1×1 convolution; δ represents ReLU activation; GAP represents global average pooling; ⊙ represents element-wise multiplication.

[0066] The channel-dimensional weighted feature F5' is then fed into the feature aggregation module as one of the inputs to the first feature aggregation unit.

[0067] like Figure 2 and Figure 4 As shown, the feature aggregation module includes three structurally identical feature aggregation units connected in sequence. Each feature aggregation unit aggregates channel-weighted features or the aggregated features of the previous feature aggregation unit, and combines these with the extracted features from the corresponding convolutional block to obtain the aggregated features of the current feature aggregation unit. The specific aggregation steps are as follows:

[0068] The aggregated channel dimension weighted features or the aggregated features of the previous feature aggregation unit are used as decoding features, and the extracted features of the corresponding convolutional block are used as encoding features.

[0069] After applying 3×3 convolutions to the decoded and encoded features respectively, so that the two features have the same spatial and channel dimensions, element-wise addition fusion and spatial attention operations are performed to obtain the feature space weight map.

[0070] The specific process of spatial attention operation is as follows: average pooling and max pooling operations are performed along the channel axis on the input additive fusion features respectively, element-wise addition is performed on the results, and then 3×3 convolution and sigmoid activation function are used to generate the corresponding feature space weight map.

[0071] Based on the feature space weight map, the decoded features and encoded features after two 3×3 convolutions are weighted by spatial dimension using element-wise multiplication to obtain spatially weighted decoded features and encoded features.

[0072] The spatially weighted decoded and encoded features are concatenated by channels, and then refined again using 3×3 convolution to obtain the aggregated features of the current feature aggregation unit.

[0073] The aggregated features of the current feature aggregation unit are sent to the next feature aggregation unit. The same process is used to achieve top-down decoding and transmission of multi-layer features.

[0074] like Figure 2As shown, the aggregated features of the last feature aggregation unit are convolved with 1×1 and upsampled to obtain the prediction map of the current flow prediction network.

[0075] The principle of the saliency enhancement module is as follows: Let the current flow be the i-th flow prediction network. First, the prediction graph S obtained from the previous flow prediction network is used. i-1 As saliency feedback information, the original image is enhanced with saliency features. Specifically, S... i-1 As weights and the original image I input Perform a multiplication operation, and then multiply the result by the original image I. input Adding them together yields the feature I with enhanced significance. sal_enh This can be expressed as a formula:

[0076]

[0077] In the formula, I sal_enh This indicates an enhanced image, with the number of channels remaining at 3; I input Represents the original image; S i-1 This represents the prediction graph corresponding to the (i-1)th flow prediction network; ⊙ indicates element-wise addition; ⊙ indicates element-wise multiplication.

[0078] Multi-level feature extraction and aggregation are performed on the enhanced image, and the prediction map S of the current flow prediction network is obtained through 1×1 convolution and upsampling. i .

[0079] like Figure 1 As shown, the training method for the multi-stream feedback saliency prediction model is as follows:

[0080] Obtain the training images and their corresponding ground truth images;

[0081] The training images are input into a pre-built multi-stream feedback saliency prediction model to obtain the prediction map of each stream prediction network;

[0082] Based on the ground truth image, the prediction graph {S} of each stream prediction network is calculated. i Joint supervision is performed on |i=1,...,N}. Based on binary cross-entropy loss, the overall loss function of the multi-flow feedback saliency prediction model is constructed. The network parameters of the multi-flow feedback saliency prediction model are adjusted through gradient backpropagation, and finally the trained multi-flow feedback saliency prediction model is obtained.

[0083] Furthermore, the expression for the overall loss function is as follows:

[0084]

[0085] In the formula, L totalRepresents the overall loss function; i represents the order of the prediction network; N represents the total number of prediction networks; L bce (S i G) represents the prediction graph S corresponding to the i-th flow prediction network. i The binary cross-entropy loss; G represents the ground truth image.

[0086] Furthermore, the expression for the binary cross-entropy loss function is as follows:

[0087]

[0088] In the formula, L bce (S i G) represents the prediction graph S corresponding to the i-th flow prediction network. i Binary cross-entropy loss; H 真 W represents the height of the truth image; 真 The width of the ground truth image is represented by p; the first coordinate of the pixel is represented by q; and G represents the second coordinate of the pixel. pq S represents the value of the true image at pixel (p, q); pq This represents the corresponding value of the predicted image at pixel (p, q).

[0089] In summary, existing salient object detection models utilizing deep networks do not fully utilize high-level semantic information, resulting in low detection accuracy for complex scene images. Furthermore, the large number of parameters in these models hinders practical deployment. This application proposes a salient object detection method based on a multi-stream feedback network framework, leveraging multi-stream feedback mechanisms and parameter sharing across multiple streams to address these issues. By using a multi-stream feedback network structure, compared to general detection methods based on single-stream, non-feedback network structures, higher-level semantic information is better utilized, thereby improving salient object detection accuracy. The identical structure and shared parameters of the multi-stream prediction networks maintain a small overall model size and computational cost, facilitating deployment in practical application systems. During training, ground-value images are used to jointly supervise the prediction maps of each stream using binary cross-entropy loss, optimizing the network training process. This method utilizes the features of the original image enhanced by the prediction maps of previous streams as input to the current network stream, and the shared network parameters across streams improve both salient object detection accuracy and detection efficiency.

[0090] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0091] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0092] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0093] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0094] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A salient object detection method based on a multi-stream feedback network framework, characterized in that, The method comprises the following steps: obtaining an original image to be detected; inputting the original image into a trained multi-stream feedback saliency prediction model to obtain a saliency prediction result corresponding to the original image; The multi-stream feedback saliency prediction model comprises a multi-stream parameter shared prediction network, and the specific prediction method is as follows: The first stream prediction network performs feature extraction, aggregation, convolution and up-sampling processing on the original image to obtain a prediction map; In addition to the first stream prediction network, each subsequent stream prediction network performs saliency feature enhancement on the original image based on the prediction map of the previous stream prediction network to obtain an enhanced image; and performs feature extraction, aggregation, convolution and up-sampling processing on the enhanced image to obtain a prediction map of each stream prediction network; The prediction map of the last stream prediction network is taken as the final saliency prediction result corresponding to the original image; Each stream prediction network comprises a feature extraction module, a channel attention module, a feature aggregation module, a convolution layer and an up-sampling module, and the parameters of each stream prediction network are shared; in addition to the first stream prediction network, each subsequent stream prediction network further comprises a saliency enhancement module; The feature aggregation module comprises three feature aggregation units connected in sequence and having the same structure, which are used to aggregate channel dimension weighted features or aggregated features of a previous feature aggregation unit, and combine extracted features of a corresponding convolution block to obtain aggregated features of a current feature aggregation unit; The aggregation step of the feature aggregation unit is as follows: The aggregated channel dimension weighted features or the aggregated features of the previous feature aggregation unit are taken as decoding features, and the extracted features of the corresponding convolution block are taken as encoding features; After 3*3 convolution is respectively performed on the decoding features and the encoding features, element-level addition fusion and spatial attention operation are performed to obtain a feature spatial weight map; According to the feature spatial weight map, spatial dimension weighting is performed on the decoding features and the encoding features after two times of 3*3 convolution, and the weighting mode is element-level multiplication to obtain spatially weighted decoding features and encoding features; The spatially weighted decoding features and the encoding features are channel spliced, and 3*3 convolution is used again for feature refinement to obtain the aggregated features of the current feature aggregation unit.

2. The salient object detection method based on multi-stream feedback network framework according to claim 1, characterized in that, The feature extraction module adopts a ResNet-50 network structure, comprises five convolution blocks connected in sequence, and is used to obtain multi-level extracted features of the original image or the enhanced image, and the expression of the extracted features is as follows: ； In the formula, n represents the sequence of the convolution block; represents the extracted feature of the nth convolution block, and the corresponding feature resolution size is , H and W represent the height and width of the image, respectively, represents the number of channels of the extracted feature .

3. The salient object detection method based on multi-stream feedback network framework according to claim 2, characterized in that, The channel attention module is used to obtain channel dimension weighted features F5' by capturing important responses of features in the channel dimension according to the extracted features F5 of the fifth convolution block, and the expression of the channel dimension weighted features F5' is as follows: ； wherein, represents channel-wise weighting features; represents extracted features of the 5th convolutional block; represents element-wise addition; represents 1x1 convolution; represents ReLU activation; represents global average pooling; represents element-wise multiplication.

4. The salient object detection method based on multi-stream feedback network framework according to claim 1, characterized in that, The expression of the enhanced image is as follows: ； In the formula, denotes an enhanced image; denotes an original image; denotes a prediction image corresponding to the i-1th flow prediction network; denotes element-wise addition; denotes element-wise multiplication.

5. The salient object detection method based on multi-stream feedback network framework according to claim 1, characterized in that, The training method of the multi-stream feedback saliency prediction model is as follows: obtaining training images and corresponding true value images; inputting the training images into a pre-constructed multi-stream feedback saliency prediction model to obtain a prediction map of each stream prediction network; According to the true value image, the prediction image of each flow prediction network is jointly supervised, a total loss function of the multi-flow feedback saliency prediction model is constructed based on a binary cross entropy loss, network parameters of the multi-flow feedback saliency prediction model are adjusted through gradient back propagation, and finally a trained multi-flow feedback saliency prediction model is obtained.

6. The salient object detection method based on multi-stream feedback network framework according to claim 5, characterized in that, An expression of the total loss function is as follows: ； wherein, represents the overall loss function; i represents the index of the prediction network; N represents the total number of prediction networks; represents the binary cross-entropy loss of the i-th flow prediction network corresponding to the predicted map ; represents the ground truth image.

7. The salient object detection method based on multi-stream feedback network framework according to claim 6, characterized in that, An expression of the function of the binary cross entropy loss is as follows: An expression of the function of the binary cross entropy loss is as follows: ； In the formula, This represents the prediction graph corresponding to the i-th flow prediction network. Binary cross-entropy loss; Indicates the height of the truth image; Indicates the width of the truth image; This represents the first coordinate of the pixel; This represents the second coordinate of the pixel. This represents the value of the true image at pixel (p, q); This represents the corresponding value of the predicted image at pixel (p, q).

Images