An image saliency detection method based on information complementary double stream decoder

Abstract

An image saliency detection method based on a double-flow decoder of information complementarity, comprising the following steps: S1, decomposing a label image to obtain a corresponding main label image and a contour detail label image; S2, performing random cropping, random rotation, normalization and grayscale processing on a training data set image to enhance the diversity of samples; S3, inputting an image, pre-processing the image by using a VGG16 framework, and collecting image features of different sizes by using a set of encoding blocks of different dimensions; S4, inputting five-layer output feature maps obtained through an encoder into an Embedding layer to unify the dimensions; S5, inputting the features of the encoded target main image and the features of the encoded contour detail image into a saliency branch and a contour image branch respectively, and simultaneously performing supervision on the respective branches by using the obtained main feature map and contour feature map; and S6, adding and fusing the target main features and the contour detail features obtained on the two main branches to obtain a final prediction image.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to an image saliency detection method based on a dual-stream decoder with complementary information. Background Technology

[0002] Traditional saliency detection works by acquiring specific features of the target. While this method clearly identifies the object's class, it inevitably falls into the limitations of the algorithm. Saliency detection is the task of segmenting the most visually distinctive targets or regions in an image; in other words, these studies aim to identify the subject of an image. Unlike other detailed techniques such as semantic segmentation or instance segmentation, saliency detection always focuses on a few key regions. Therefore, it is often used as the first step in many studies, such as object tracking, object recognition, and action classification.

[0003] The main challenge in saliency detection lies in distinguishing between salient objects and object edges in an image. Inspired by the correlation between salient objects and object contours, this study employs a complementary approach compared to methods that rely solely on edge information or overall image progress learning. This approach obtains both the object's main image and its contour detail image, allowing both to jointly supervise the model's training. This reduces the impact of extremely unbalanced edge pixel distribution of salient objects, thus improving performance. Summary of the Invention

[0004] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provides a saliency detection method based on information complementarity dual-stream decoder.

[0005] This invention combines the advantages of using the VGG model, which contains significantly fewer parameters than other VGG-based methods, with the feature fusion module and loss function designed in this method, achieving relatively good salient target detection results.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] An image saliency detection method based on information complementarity dual-stream decoder includes the following steps:

[0008] Step S1: Generate a contour detail map by expanding and eroding the highlight and calculating the difference between the label images, and use the seed fill algorithm to find the region within the closed contour to obtain the target subject image;

[0009] Step S2: Perform random cropping, random rotation, normalization, and grayscale processing on the training dataset images to enhance the diversity of the samples;

[0010] Step S3: Input an image with a scale of H×W, preprocess it using the VGG16 framework, and collect image features of different sizes using a set of encoding blocks of different dimensions.

[0011] Step S4: Input the five-layer output feature map obtained by the encoder into the Embedding layer and unify the dimensions;

[0012] Step S5: Input the features encoded by the target subject map and the features encoded by the contour detail map into the saliency branch and the contour map branch respectively. At the same time, supervise the obtained subject feature map and contour feature map on their respective branches. On the one hand, the information of the two are interacted and fused to achieve information complementarity. On the other hand, the two are input into their respective image decoders to enrich the features.

[0013] Step S6: After adding and fusing the target subject features and contour detail features obtained from the two main branches, the final predicted image is obtained through upsampling and the final connection operation.

[0014] This invention provides an image saliency detection method based on a dual-stream decoder with complementary information. It utilizes the image's ground truth label map to obtain a target subject map and a contour detail map. The subject map primarily contains the image's theme information, while the contour map mainly contains the image's edge detail information. The subject map and contour map are used together to supervise the model's training. In the feature fusion module, the continuously optimized subject map and contour map are added and fused. Since the two images contain complementary information, iterative training of their features yields better salient target prediction results.

[0015] The advantages of this invention are: it emphasizes the correlation between salient targets and target contours, segmenting the real image into two complementary information segments, each focusing on different regions. The salient target image and the target contour map jointly supervise the iterative training of the model, and the two features are added and fused together, allowing more information from the main body and contour edges of the salient target to be utilized, thereby achieving better prediction results. Attached Figure Description

[0016] Figure 1 This is a flowchart of the method of the present invention.

[0017] Figure 2 right Figure 1 The flowchart shows the schematic diagram of the feature interaction module of the decoder. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application.

[0020] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0021] An image saliency detection method based on information complementarity dual-stream decoder includes the following steps:

[0022] Step S1: Generate a contour detail map by expanding and eroding the highlight and calculating the difference between the label images, and use the seed fill algorithm to find the region within the closed contour to obtain the target subject image;

[0023] Step S2: Perform random cropping, random rotation, normalization, and grayscale processing on the training dataset images to enhance the diversity of the samples;

[0024] Step S3: Input an image with a scale of H×W, preprocess it using the VGG16 framework, and collect image features of different sizes using a set of encoding blocks of different dimensions.

[0025] Step S4: Input the five-layer output feature map obtained by the encoder into the Embedding layer and unify the dimensions;

[0026] Step S5: Input the encoded features of the target subject image and the encoded features of the contour detail image into the saliency branch and the contour branch respectively. At the same time, use the obtained subject feature image and contour feature image to supervise each branch. On the one hand, the information of the two are interacted and fused to achieve information complementarity. On the other hand, the two are input into their respective image decoders to enrich the features.

[0027] Step S6: After adding and fusing the target subject features and contour detail features obtained from the two main branches, the final predicted image is obtained through upsampling and the final connection operation.

[0028] Step S1 involves using dilation and erosion to highlight and calculate the difference between label images to generate a contour detail map, and employing a seed fill algorithm to find regions within closed contours to obtain the target subject image. Specifically, this includes:

[0029] S11. Input a label image, with the salient target area being white and the background being black. The size is not required. Set the pixel value of the background part to 0 and the pixel value of the foreground part to 1.

[0030] S12. A contour detail map is generated by using dilation and erosion to highlight and calculate the difference between the label images. A seed fill algorithm is then used to find regions within closed contours, thus obtaining the target subject image. The calculation formula is as follows:

[0031]

[0032]

[0033] Here, X and Y represent erosion and expansion operations, respectively, and B(x) represents the structuring element, which performs the corresponding operation on each point x in the workspace E.

[0034] Step S3 specifically includes: using the pre-trained model obtained from the VGG16 model, loading parameters, inputting data, obtaining 5 layers of feature output except for the fully connected layers, and using a set of encoding blocks of different dimensions to collect image features of different sizes, denoted as F = {F...} i |i=1,2,3,4,5};

[0035] To reduce feature channels and computational cost, a channel pooling layer is added at the top of each feature map to allow information to be passed through different channels. Its specific definition is:

[0036] Emebdding i =cp(E i (3)

[0037]

[0038] Where j and K are integers, i∈[1,5] represents the index value of each feature map, and p(X) represents the channel pooling operation. This represents the first feature map X. The channel, while the pooling layer collects each The maximum value of the channel, where N and M represent the input channel and the output channel, respectively.

[0039] Step S5 specifically includes: Each feature fusion module comprises two branches: a main branch and a contour branch. Each branch also includes a corresponding new branch, which is supervised using both the main and contour images. The resulting features are then superimposed and fused. The new features from different new branches are input into the opposite main branch for feature fusion. Since the final output focuses on saliency, the main branch is prioritized, and its result is input into the next feature fusion module. Modules can be superimposed, as detailed below:

[0040] S51. For each feature fusion module, two data points need to be added together. The two input sources are the feature output layer corresponding to the encoder and the output of the module in the previous layer. For the first feature fusion module of the main image, since there are no input features from the module in the previous layer, the output of the 5th layer in the encoder is used as the input of the decoder in the previous layer. The intermediate connections of the branches are implemented as follows:

[0041]

[0042]

[0043] Among them, A i and B i These represent the main branch and the contour branch, respectively. `conv` represents the convolution operation, and the subscript of `conv` indicates the corresponding branch. `P` i A and P i B This represents a prediction of the relevant task, with additional supervision added.

[0044] S52. Each feature fusion module includes two branches: a main branch and a contour branch. Each branch also includes a corresponding new branch, which is supervised by both the main image and the contour image. The resulting features are then superimposed and fused. Different new branches input their new features into the opposite main branch for feature fusion. This process is repeated five times. Finally, the fused input features are used to output a topic prediction map. The specific branch operations and the final prediction are as follows:

[0045]

[0046]

[0047] Here, `upsample` and `concat` represent upsampling and concatenation operations, respectively. For the final prediction, all features in the main branch are concatenated to balance hierarchical information, specifically as follows:

[0048] Final = conv Final(concat([upsample(A i ),i=1,2,3,4,5])) (9)

[0049]

[0050] Among them, all A i Before connection, all data are upsampled to the input size, and then the final prediction is made by aggregating the connection features.

[0051] Step S6 specifically includes: comparing the three prediction maps obtained from each module—including the topic prediction map, the contour detail prediction map, and the salient target prediction map—with the real data labels, calculating the loss value, and backpropagating to update the weights in the model. The loss value calculation is as follows:

[0052] For the obtained main body prediction map and contour detail prediction map, the loss value is calculated by comparing them with the corresponding label map. The loss value is calculated using the Binary Cross Entropy function.

[0053]

[0054] Where g(x,y) is the value of the pixel in the label image located at (x,y), and the value ranges from [0,1]; p(x,y) is the value of the pixel in the prediction image located at (x,y); l bce (x,y) represents the loss value.

[0055] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0056] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An image saliency detection method based on a double-flow decoder with information complementarity, comprising the following steps: Step S1, generating a contour detail map by dilating and eroding the saliency and calculating the difference between the label images, and using a seed filling algorithm to find the area within the closed contour to obtain a target subject map; Step S2, randomly cropping, randomly rotating, normalizing and greying the training data set images to enhance the diversity of the samples; Step S3, inputting an image with a size of HxW, pre-processing using the VGG16 framework, and using a set of encoding blocks with different dimensions to collect image features of different sizes respectively; Step S4, inputting the five-layer output feature maps obtained by the encoder into the Embedding layer for uniform dimension; Step S5, inputting the encoded features of the target subject map and the contour detail map into the saliency branch and the contour branch respectively, and using the obtained subject feature map and contour feature map to supervise the respective branches, on the one hand, the information of the two is interactively fused to achieve information complementarity, and on the other hand, the two are respectively input into the image decoder to enrich the features; specifically including: Each feature fusion module includes two branches, namely the subject branch and the contour branch, and each branch will also be attached to the corresponding new branch and supervised by the subject map and the contour map, then the obtained features are superimposed and fused, and the new features obtained by different new branches are input into the opposite main branch for feature fusion; since the final output is mainly the saliency target, the subject branch is mainly used, and the result is input into the next feature fusion module, and the modules can be superimposed, the specific steps are as follows: S51. For each feature fusion module, two data need to be added, the two input sources are from the corresponding feature output layer of the encoder and the output of the module from the previous layer; for the first feature fusion module of the subject map, since there is no input feature from the module of the previous layer, the 5th layer output in the encoder is used as the input of the decoder of the previous layer, wherein the implementation of the branch connection is as follows: （5） （6） wherein, and represent the main branch and the profile branch, respectively, denotes a convolution operation, the subscript of and represent the prediction of the related task, with additional supervision added. S52. Each feature fusion module includes two branches, namely the subject branch and the contour branch, and each branch will also be attached to the corresponding new branch and supervised by the subject map and the contour map, then the obtained features are superimposed and fused, and the new features obtained by different new branches are input into the opposite main branch for feature fusion, and the final input feature is fused to output the theme prediction map, the branch specific operation and the final prediction are as follows: （7） （8） where, and denote upsampling and concatenation operations, respectively; for the final prediction, all features in the main branch are concatenated to balance the hierarchical information, which is specifically denoted as: （9） （10） where all All are upsampled to the input size before the connection and aggregate the final prediction on the concatenated features. Step S6, adding and fusing the target subject features and contour detail features obtained on the two main branches, and then performing upsampling operation and final connection operation to obtain the final prediction image.

2. The image saliency detection method based on information complementarity dual-stream decoder as described in claim 1, characterized in that: The dilating and eroding saliency and calculating the difference between the label images to generate the contour detail map, and using the seed filling algorithm to find the area within the closed contour to obtain the target subject map in step S1, specifically including: S11. Input a label image, the salient object region is white part, the background is black part, the size is not required, the background part pixel value is 0, and the foreground part pixel value is 1; S12. The contour detail image is generated by using expansion and erosion and calculating the difference value between the label images, and the region in the closed contour is found by using a seed filling algorithm, so that the target main body image is obtained, and the calculation formula is as follows: （1） （2） wherein, and respectively denote erosion and dilation operations, with representing the structuring element, the corresponding operation being performed on each point in the workspace .

3. The image saliency detection method based on information complementarity dual-stream decoder as described in claim 1, characterized in that: The step S3 specifically comprises: using the pre-trained model obtained by training the VGG16 model, loading parameters, inputting data, obtaining 5 layers of feature outputs except the full connection layer, and using a set of encoding blocks with different dimensions to respectively collect image features of different sizes, which are respectively denoted as ; In order to reduce the feature channel and the amount of calculation, a channel pooling layer is added at the top of each feature map, so that information is transmitted through different channels, and the specific definition is as follows: （3） （4） wherein, and are integers, denotes an index value of each feature map, denotes a channel pooling operation, denotes a feature map of the th channel, while the pooling layer collects the maximum value of each th channel, wherein and denote the input channel and the output channel, respectively.

4. The method of claim 1, wherein the information complementary based dual stream decoder is a JEM decoder. Step S6 specifically comprises: comparing the three prediction maps obtained by each module, including the theme prediction map, the contour detail prediction map and the salient object prediction map, with the real data label, calculating the loss value, and updating the weight value in the model by reverse propagation, and the loss value calculation content is as follows: For the obtained main body prediction map and contour detail prediction map, the loss value is calculated with the corresponding label image, and the loss value calculation adopts a Binary Cross Entropy function: （11） wherein, is the value of the pixel where the label map is located ; ; is the value of the pixel where the prediction map is located ; is the loss value.

Images