A method for detecting salient targets in images

Abstract

This application provides a method for detecting salient objects in images. It constructs a detection model based on a lightweight Mobilenetv2 backbone network and introduces fusion side connections into this backbone network to progressively fuse features from each layer. The method predicts salient objects at multiple scales and performs supervised learning, effectively avoiding overfitting. The proposed method constructs a lightweight detection model, and by introducing fusion side connections, it fully integrates features from each layer, making the model's performance comparable to larger existing models. The lightweight and high-performance detection model constructed in this application is suitable for deployment on television terminals and can be applied to television application scenarios such as visual object tracking and intelligent picture quality settings.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method for detecting salient targets in images. Background Technology

[0002] Salient object detection mimics the focusing effect of human vision to detect regions of salient objects in an image. Currently, salient object detection has wide applications in image retrieval, visual object tracking, scene classification, content-based video compression, and weakly supervised learning.

[0003] Currently, commonly used salient object detection methods are all based on fully convolutional networks (WCNNs) from deep learning. The backbone of a WCNN consists of stacked convolutional modules; modules closer to the input are called low-level modules, and modules farther from the input are called high-level modules. The high-level modules of the neural network contain semantically relevant information, while the low-level modules contain more detailed information. In the process of salient object detection, fusing features from different levels (including high-level and low-level modules) can effectively improve the salient object detection results.

[0004] However, existing salient object detection models based on deep fully convolutional networks involve complex methods for fusing high-level and low-level features, resulting in high computational costs. Furthermore, current salient object detection models employ large backbone networks, which consume significant resources and are difficult to deploy on small devices. Summary of the Invention

[0005] This application provides a method for detecting salient targets in images, which addresses the problem that existing methods for detecting salient targets in images use large backbone networks and employ complex feature fusion methods, resulting in high computational overhead and resource consumption of the detection model, making it difficult to deploy on small devices.

[0006] This application provides a method for detecting salient targets in an image, including:

[0007] Construct an "encoder-decoder" type convolutional neural network model, wherein the "encoding" part adopts a lightweight Mobilenetv2 backbone network. The "encoding" part is used to extract features from the input image and generate a set of feature maps, which contain feature maps from high level to low level. Each level of feature map has semantic information at different scales.

[0008] In the "decoding" part, several fusion side connections are introduced. In order from high layer to low layer, each level of fusion side connection fuses the feature map of the corresponding level with the feature map of the previous level, and uses the fused feature map to obtain the predicted saliency map and calculate the loss function.

[0009] The training process utilizes the loss functions of each layer to obtain the trained "encoder-decoder" type convolutional neural network model.

[0010] Image data is input into a trained "encoder-decoder" type convolutional neural network model for salient object detection in the image.

[0011] This application provides a salient target detection method for images. It constructs a salient target detection model based on a lightweight Mobilenetv2 backbone network and introduces fusion side connections into the backbone network to gradually fuse features from each level of the backbone network. It then predicts salient targets at multiple scales and performs supervised learning, effectively avoiding overfitting.

[0012] The proposed image salient object detection method constructs a lightweight detection model. Furthermore, by introducing fusion side connections, it fully integrates features from various levels, making the detection model's performance comparable to existing, larger models. The lightweight and high-performance detection model constructed in this application is suitable for deployment on television terminals and can be applied to television application scenarios such as visual object tracking and intelligent picture quality settings. Attached Figure Description

[0013] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart of the salient target detection method in this application;

[0015] Figure 2 This is a structural diagram of an embodiment of the "encoder-decoder" type convolutional neural network of this application;

[0016] Figure 3 A flowchart illustrating an embodiment of constructing an "encoder-decoder" type convolutional neural network model for this application;

[0017] Figure 4 This is a flowchart of the feature fusion module of this application;

[0018] Figure 5 This is a flowchart of the segmentation header module of this application;

[0019] Figure 6 This is a structural diagram of another embodiment of the "encode-decode" type convolutional neural network of this application;

[0020] Figure 7 A structural diagram illustrating another embodiment of the "encoder-decoder" type convolutional neural network model for this application;

[0021] Figure 8 A flowchart showing the connections to the details of this application;

[0022] Figure 9 A schematic diagram of the input image provided in the embodiments of this application;

[0023] Figure 10 This is a schematic diagram of the salient target detection results in an embodiment of this application. Detailed Implementation

[0024] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of this application.

[0025] This application provides a method for detecting salient targets in images, which addresses the problem that existing methods for detecting salient targets in images use large backbone networks and employ complex feature fusion methods, resulting in high computational overhead and resource consumption of the detection model, making it difficult to deploy on small devices.

[0026] Figure 1 This is a flowchart of one embodiment of the image salient target detection method of this application. Figure 2 This is a structural diagram of an embodiment of the "encoder-decoder" type convolutional neural network of this application, combined with... Figure 1 and Figure 2 As shown, the image salient target detection method includes:

[0027] Step S100: Construct an "encoder-decoder" type convolutional neural network model. In this example, the "encoder-decoder" type convolutional neural network has a U-shaped connection structure, which means... Figure 2 The input is processed layer by layer through the "encoding" part to extract the highest level features. Then, the lower-level "encoding" features are fused layer by layer from the highest level features to form the "decoding" part. The flow of the entire data in the "encoding-decoding" structure forms a left-falling letter U.

[0028] In some embodiments, such as Figure 3As shown, the process of constructing an "encoder-decoder" type convolutional neural network model includes the following steps. Step S110: The "encoder" part of this model uses a lightweight Mobilenetv2 backbone network, and the "decoder" part is used to extract features from the input image, generating a set of feature maps, including feature maps from high-level to low-level. Each level of feature map has different scales of semantic information. Specifically, high-level feature maps contain global semantic information, while low-level feature maps mainly contain fine-grained local features. In practical applications, feature maps of all levels of the lightweight Mobilenetv2 backbone network are obtained, i.e., from the lowest level to the highest level. For example, a lightweight Mobilenetv2 backbone network contains 18 levels. The "encoder" part obtains feature maps of levels 1, 2, ..., 18, which are respectively the feature maps of level 1, level 2, ..., level 18. The feature map of level 1 is the lowest level feature map, and the feature map of level 18 is the highest level feature map.

[0029] In some embodiments, the lightweighting process of the Mobilenetv2 backbone network is as follows: In the Mobilenetv2 network, the last convolutional layer, one adaptive average pooling layer, and one fully connected layer are removed, that is, all layers before the last bottleneck layer are retained, resulting in a lightweight Mobilenetv2 backbone network. For example, an input 224×224 3-channel color image, after passing through the Mobilenetv2 backbone network, that is, sequentially passing through one convolutional layer and 7 bottleneck modules (a total of 17 inverse residual layers), outputs a 7×7 feature map with 320 channels, as shown in Table 1 below.

[0030] Table 1

[0031]

[0032] In Table 1, t is the inflation factor of the number of channels in the intermediate convolutional layers of the inverted residual layer in the bottleneck module, c is the number of output channels, n is the number of repetitions of the inverted residual layer in this module, s is the stride of the convolutional layer of the first inverted residual layer in this module, and the stride of the convolutional layer of other repeated inverted residual layers is 1.

[0033] The "encode-decode" type convolutional neural network model constructed using the lightweight Mobilenetv2 backbone network is a lightweight model. Compared with traditional detection models that use large backbone networks, the "encode-decode" type convolutional neural network model of this application consumes fewer resources and is suitable for deployment on TV terminals.

[0034] Step S120 involves introducing several fusion-side connections in the "decoding" section. These connections sequentially fuse features at their respective levels, from high to low. The fused feature map is then used to obtain a predicted saliency map and calculate the loss function. It should be noted that the number of fusion-side connections can be adjusted by those skilled in the art according to actual needs, for example, by introducing 4, 6, or 7 levels. The position of the fusion-side connections can also be adjusted according to actual needs. For example, seven levels could be introduced, including levels 2, 5, 7, 10, 13, 17, and 22, where level 22 is the highest level, level 17 is the second highest level, level 13 is a mid-high level, level 10 is a middle level, level 7 is a mid-low level, level 4 is the second lowest level, and level 2 is the lowest level. Alternatively, seven levels could be 3, 5, 8, 12, 15, 18, or 24 levels, all of which fall within the scope of this application.

[0035] The specific implementation process of feature fusion, prediction saliency map acquisition, and loss function calculation in the fusion-side connections includes: a feature fusion module (FFM), a segmentation head module (Seg head), and an automatic loss calculation module (Seg loss). A concatenation module is set between the feature fusion modules of adjacent fusion-side connections. Specifically, the feature map of the corresponding level is concatenated with the fused feature map of the previous level through the concatenation module. For example... Figure 2 As shown, the feature map of the corresponding level is concatenated with the fused feature map of the previous level in the Tao dimension, that is... Figure 2 In The concatenated feature maps are then processed by the feature fusion module to obtain the fused feature maps for the corresponding levels.

[0036] Figure 4 This is a flowchart of the feature fusion module of this application, as shown below. Figure 4 As shown, the concatenated feature maps enter the feature fusion module for feature fusion processing to obtain fused feature maps of the corresponding levels. This includes the concatenated feature maps undergoing three layers of processing: conv+bn+relu (convolutional layer, batch normalization layer, and activation layer), and then being divided into three branches. The first branch (with...) Figure 4 Taking the indicated position as a reference, the first branch (the topmost branch) is the direct branch in the residual structure. The second branch (based on...) Figure 4 Taking the direction shown as a reference, the second branch is the branch in the middle position. After passing through a 1x1 convolutional layer, it is divided into three sub-branches, corresponding to dilated convolutions with dilation coefficients of 1, 2, and 3 respectively. The outputs of the three sub-branches obtained after dilated convolution are added together, passed through a batch normalization layer and an activation layer, and then through another 1x1 convolutional layer before being added to the first sub-branch. The first and second sub-branches together form a residual structure, which is then fused through a 1x1 convolution. This fused output will be fused with the third sub-branch again. The third branch (with...) Figure 4 Taking the direction shown as a reference, the third branch is the bottommost branch) passes through the global pooling layer, 1x1 convolutional layer, activation layer, 1x1 convolutional layer and sigmoid layer in sequence to obtain the weight coefficients. These weight coefficients are multiplied by the results of the fusion output of the first and second branches to obtain the final feature fusion output.

[0037] The fused feature map is input into the segmentation head module, and after semantic segmentation, a predicted saliency map for the corresponding level is obtained. The fused features from each level are then processed by the segmentation head module ( Figure 2 After processing the Seg Head, the output can be a prediction of the salient target (this prediction is a binary map, with the salient target region being 1 and other background regions being 0). At different levels, salient target prediction binary saliency maps with different resolutions can be obtained. Figure 5 This is a flowchart of the segmentation header module of this application. Figure 5 As shown, the segmentation head module includes a 1x1 convolutional layer with one output channel and a sigmoid activation layer. In actual use, the fused feature map first passes through a 1x1 convolutional layer with one output channel, and then through a sigmoid layer to obtain the final predicted saliency map output.

[0038] Finally, using the predicted saliency map and the true saliency map, the automatic loss function corresponding to each level is calculated.

[0039] In some embodiments, the "decoding" part introduces several fusion-side connections from several layers of the MobileNetv2 backbone network. These fusion-side connections are sequentially activated from high to low levels. Specifically, the highest-level feature map enters the highest-level fusion-side connection. Specifically, the highest-level feature map enters the feature fusion module of the corresponding layer, where it is upsampled using bilinear interpolation to obtain the highest-level fused feature map. Then, it passes through the segmentation head module to obtain the highest-level predicted saliency map. The highest-level saliency map is then used for loss calculation. It is upsampled to the input image size using bilinear interpolation, and the loss function compared to the true saliency map is calculated; this is the highest-level automatic loss function prediction loss.

[0040] Then, the second-highest level feature map enters the second-highest level fusion side connection. Specifically, the second-highest level feature map and the highest level fusion feature map enter the stitching module and are stitched together in the channel dimension. After stitching, they enter the feature fusion module and the segmentation head module to obtain the second-highest level predicted saliency map. The image is then upsampled to the input image size and the loss function is calculated with the true saliency map.

[0041] Next, the process moves to the next level of fusion-side connections, which is similar to the process described above and will not be repeated here, until all levels of fusion-side connections have completed their work. Finally, for each fusion-side connection, a corresponding number of prediction saliency maps and losses will be obtained.

[0042] To better understand the above process, a specific example is provided below. In this example, the "decoding" part introduces five side connections, i.e., fusion side connections, from the feature maps of layers 2, 4, 7, 14, and 18 of the Mobilenetv2 backbone network. Among these five layers, layer 18 is the highest layer, layer 14 is the second highest layer, layer 7 is the middle layer, layer 4 is the second lowest layer, and layer 2 is the lowest layer.

[0043] The feature map of the highest level (level 18) is introduced into the corresponding fusion side connection, and then passes through the FFM module and the segmentation head module in sequence to obtain the highest level prediction saliency map, which is 7×7 in size. The prediction saliency map is upsampled to the size of the input image using bilinear interpolation, and the loss function between the prediction saliency map and the real saliency map is calculated. During the calculation, the binary cross-entropy loss is calculated pixel by pixel and the average is taken as the loss.

[0044] The 18-level feature map is upsampled by 2 times (to 14×14 size) using bilinear interpolation after passing through the FFM module. It is then concatenated with the feature map of the next higher level (14th level) (14×14 size) in the channel dimension. After being fused by the feature fusion module, it passes through the segmentation head module to output the predicted saliency map of this level. The predicted saliency map is then upsampled to the size of the input image and the loss function is calculated with the real saliency map.

[0045] In the next higher level, the fused features are upsampled by 2x and concatenated with the feature map of the next lower level (level 7) in the channel dimension. The concatenated feature map is then fused with the feature map of the next lower level, and the output is the same. Subsequent processes repeat the operations from the previous level, which will not be elaborated here. Finally, saliency map predictions at five scales are obtained, and five loss functions are calculated. By predicting at five scales and calculating losses, the generated features from each level are fully integrated, effectively preventing overfitting and enhancing robustness.

[0046] In this application, features from each level are fully integrated progressively from high to low, resulting in fused features that simultaneously contain global semantic information and fine-grained local features. Supervised learning for salient object prediction is then performed at multiple scales based on these fused features, thus achieving accurate salient object detection. Furthermore, in this application, the predicted saliency maps and loss functions at different levels fully integrate features from each level. Therefore, using loss functions at each level for model training effectively prevents overfitting and enhances the model's robustness.

[0047] Step S200: Train the model using the loss functions of each layer to obtain the trained "encoder-decoder" type convolutional neural network model.

[0048] Step S300: Input the image data into the trained "encoder-decoder" type convolutional neural network model to perform salient target detection in the image.

[0049] This application also provides another method for detecting salient targets in images, which is basically the same as the previous embodiment, except that detail-side connections are introduced in the lightweight Mobilenetv2 backbone network.

[0050] Figure 6 This is a structural diagram of another embodiment of the "encoder-decoder" type convolutional neural network of this application. Figure 7 A structural diagram illustrating another embodiment of the "encoder-decoder" type convolutional neural network model for this application, combined with... Figure 6 and Figure 7 As shown, the salient target detection method in this application includes:

[0051] Step T100: Construct an "encode-decode" type convolutional neural network model.

[0052] Step T110: The "encoding" part of this model uses a lightweight Mobilenetv2 backbone network. This "encoding" part extracts features from the input image, generating a set of feature maps, ranging from high-level to low-level features. Each level of feature map contains semantic information at different scales. Specifically, high-level feature maps contain global semantic information, while low-level feature maps mainly contain fine-grained local features. In practical applications, feature maps from all levels of the lightweight Mobilenetv2 backbone network are obtained, i.e., from the lowest level to the highest level.

[0053] In some embodiments, the lightweighting process of the Mobilenetv2 backbone network is as follows: in the Mobilenetv2 network, the last convolutional layer, an adaptive average pooling layer, and a fully connected layer are removed, that is, all layers before the last bottleneck layer are retained, resulting in a lightweight Mobilenetv2 backbone network. The "encoder-decoder" type convolutional neural network model constructed using the lightweight Mobilenetv2 backbone network is a lightweight model. Compared with traditional detection models using large backbone networks, the "encoder-decoder" type convolutional neural network model of this application consumes fewer resources and is suitable for deployment on television terminals.

[0054] In step T120, several fusion side connections are introduced in the "decoding" part. The several fusion side connections perform feature fusion of the corresponding level in sequence from high level to low level, and use the fused feature map to obtain the predicted saliency map and loss function.

[0055] The specific implementation process of feature fusion, obtaining predicted saliency maps, and calculating loss functions in the fusion-side connection includes: the fusion-side connection comprises a feature fusion module (FFM), a segmentation head module (Seg head), and an automatic loss calculation module (Seg loss), with a concatenation module between the feature fusion modules of adjacent fusion-side connections. Specifically, the feature map of the corresponding level is concatenated with the fused feature map of the previous level through the concatenation module; the feature map of the corresponding level is also concatenated with the fused feature map of the previous level along the channel dimension; the concatenated feature map enters the feature fusion module to obtain the fused feature map of the corresponding level; the fused feature map is input into the segmentation head module, and after semantic segmentation, the predicted saliency map of the corresponding level is obtained; using the predicted saliency map and the ground truth saliency map, the loss function corresponding to each level is calculated.

[0056] In some embodiments, the "decoding" part introduces several fusion-side connections from several layers of the MobileNetv2 backbone network. These fusion-side connections are sequentially activated from high to low levels. Specifically: First, the feature map of the highest level enters the highest-level fusion-side connection. Specifically, the highest-level feature map enters the feature fusion module of the corresponding layer to obtain the highest-level fused feature map. Then, it passes through the segmentation head module to obtain the highest-level predicted saliency map. The highest-level saliency map undergoes loss calculation, and is then upsampled to the input image size using bilinear interpolation. The loss function between the highest-level feature map and the ground truth saliency map is calculated, which is the highest-level prediction loss. Then, the next-highest level feature map enters the next-highest level fusion side connection. Specifically, the next-highest level feature map and the highest-level fusion feature map are concatenated in the concatenation module along the channel dimension. After concatenation, they enter the feature fusion module and the segmentation head module to obtain the next-highest level predicted saliency map. This map is then upsampled to the input image size and the loss function is calculated with the ground truth saliency map. Afterward, it enters the next level fusion side connection, and the process is similar to the above, so it will not be elaborated here, until all levels of fusion side connections are completed. Finally, for each fusion side connection, a corresponding number of predicted saliency maps and prediction losses will be obtained.

[0057] To include edge features in the feature map, this example further includes step T130 in constructing the "encoder-decoder" type convolutional neural network model, which introduces several detail-side connections into the lightweight Mobilenetv2 backbone network. It should be noted that the number of detail-side connections can be adjusted by those skilled in the art according to actual needs, for example, introducing 4, 6, or 7 layers. The position of the detail-side connections can also be adjusted according to actual needs, for example, introducing 7 layers, including layers 2, 4, 8, 13, 19, 22, and 32, where layer 22 is the highest layer, layer 17 is the second highest layer, layer 13 is a half-high layer, layer 10 is a middle layer, layer 7 is a half-low layer, layer 4 is the second lowest layer, and layer 2 is the lowest layer. Of course, introducing 7 layers can also be 3, 7, 10, 15, 20, 26, and 32 layers, all of which fall within the scope of this application.

[0058] like Figure 6 As shown, the detail-side connection includes a detail head module and a detail loss calculation module. The feature maps of each level pass through the detail head module to obtain the salient object edge predictions of the corresponding level, and the detail loss function is calculated with the edges of the real salient map at different scales. That is, Laplcian convolution with stride is performed on the real salient map. By predicting the edges of the salient map at different scales and supervising learning, the feature map contains the characteristics of salient object edges, which can effectively reduce the blurring of the boundary.

[0059] Figure 8 The flowchart for the header details module of this application is as follows: Figure 8 As shown, the detail head module includes a convolutional layer, a batch normalization layer, a ReLU activation layer, a 1x1 convolutional layer with 1 output channel, and a Sigmoid activation layer. In actual use, the feature map passes through the convolutional layer, the batch normalization layer, the ReLU layer, the 1x1 convolutional layer with 1 output channel, and the Sigmoid activation layer in sequence, outputting a prediction of the edge of the salient target.

[0060] The detailed lateral connections constructed in this application predict the edges of salient targets at multiple levels through the detailed head module based on the feature maps generated by the lightweight Mobilenetv2 backbone network and supervised learning, so that the feature maps contain edge characteristics, thereby improving the boundary clarity of the predicted salient targets.

[0061] Step T200: Use the loss functions of each layer to train the model and obtain the trained "encoder-decoder" type convolutional neural network model.

[0062] Step T300: Input the image data into the trained "encoder-decoder" type convolutional neural network model to perform salient target detection in the image.

[0063] To better understand the above process, a specific example is provided below. In this example, the "encoding" part of the "encoder-decoder" type convolutional neural network model encodes the input image (such as...). Figure 9 Feature extraction is performed (as shown) to generate a set of feature maps, which include feature maps from high level to low level. Each level of feature map has semantic information at different scales. Among them, the high-level feature maps contain global semantic information, while the low-level feature maps mainly contain fine-grained local features.

[0064] The "decoding" part introduces five side connections, or fusion side connections, from the feature maps of layers 2, 4, 7, 14, and 18 of the Mobilenetv2 backbone network. Among these five layers, layer 18 is the highest layer, layer 14 is the second highest layer, layer 7 is the middle layer, layer 4 is the second lowest layer, and layer 2 is the lowest layer.

[0065] The feature map of the highest level (level 18) is introduced into the corresponding fusion side connection, and then passes through the FFM module and the segmentation head module in sequence to obtain the highest level prediction saliency map, which is 7×7 in size. The prediction saliency map is upsampled to the size of the input image using bilinear interpolation, and the loss function between the prediction saliency map and the real saliency map is calculated. During the calculation, the binary cross-entropy loss is calculated pixel by pixel and the average is taken as the loss.

[0066] The 18-level feature map is upsampled by 2 times (to 14×14 size) using bilinear interpolation after passing through the FFM module. It is then concatenated with the feature map of the next higher level (14th level) (14×14 size) in the channel dimension. After being fused by the feature fusion module, it passes through the segmentation head module to output the predicted saliency map of this level. The predicted saliency map is then upsampled to the size of the input image and the loss function is calculated with the real saliency map.

[0067] In the next higher level, the fused features are upsampled by 2x and concatenated with the feature map of the next lower level (level 7) in the channel dimension. The concatenated feature map is then fused with the feature map of the next lower level, and the output is the same. Subsequent processes repeat the operations from the previous level, which will not be elaborated here. Finally, saliency map predictions at five scales are obtained, and five loss functions are calculated. By predicting at five scales and calculating losses, the generated features from each level are fully integrated, effectively preventing overfitting and enhancing robustness.

[0068] Simultaneously, the feature maps at levels 2, 4, 7, 14, and 18 of the lightweight Mobilenetv2 backbone network generate five side connections, which are then processed by five detail head modules (convolutional layer, batch normalization layer, ReLU activation layer, 1x1 convolutional layer with 1 output channel, and Sigmoid activation layer) to output salient target edge predictions. These predictions are then compared with the edge predictions of the ground truth saliency maps at different scales to derive five detail loss functions. In this example, a Laplcian convolutional kernel is used. By predicting the edges of salient maps at different scales and supervising learning, the feature maps contain salient target edge characteristics, effectively reducing the blurring of boundaries.

[0069] The 10 losses (segmentation loss and detail loss) obtained above are summed to form the total loss function for training. During testing, the predicted saliency map of the lowest layer is taken, and after bilinear interpolation and upsampling by a factor of 2, it is used as the final prediction output. The image salient object detection results are as follows: Figure 10 As shown.

[0070] To address the computational complexity and difficulty in deploying existing image salient target detection models on terminals, this application provides a lightweight image salient target detection method suitable for television terminal deployment. When a salient target is detected, its category is simultaneously identified, allowing for adjustments to television picture quality based on the corresponding target category. This highlights the salient target, responds to the human eye's focusing effect, and enhances the viewing experience. By utilizing salient targets detected in the current frame and multiple historical frames, combined with the target's category information, visual tracking of salient targets can be established, enabling real-time monitoring of key targets.

[0071] In the image salient target detection method of this application, a convolutional neural network model based on a lightweight Mobilenetv2 backbone network is constructed, including "encoding" and "decoding" parts. The "encoding" part uses the lightweight Mobilenetv2 backbone network to gradually generate feature maps of various levels from low to high from the input image. The "decoding" part fully integrates the features of each level from high to low, so that the fused features contain both global semantic information and fine-grained local features. Supervised learning for salient target prediction is performed on the fused features at multiple scales, thereby achieving accurate detection of salient targets. That is, when an image is input, salient targets in the image can be detected efficiently.

[0072] Specifically, the "decoding" part introduces several fusion-side connections, each including a feature fusion module, a segmentation head module, and a loss calculation module. A concatenation module is set between the feature fusion modules of adjacent fusion-side connections. In practical applications, the fusion-side connections "decode" from high to low levels, using the feature fusion module to fuse the fused features from the previous layer and the corresponding feature maps from the "encoding" part layer by layer. This ensures that the features generated by the lightweight backbone network are fully integrated across layers. Furthermore, based on the fused features, the segmentation head module performs saliency map prediction and supervised learning at multiple levels, effectively avoiding overfitting. Additionally, several detail-side connections are introduced into the lightweight Mobilenetv2 backbone network. Each detail-side connection includes a detail head module and a detail loss calculation module. Based on the feature maps generated by the lightweight backbone network, the detail head module predicts salient target edges at multiple levels and performs supervised learning, ensuring that the feature maps contain edge characteristics and improving the clarity of the predicted salient target boundaries.

[0073] Since the above embodiments are all described in conjunction with other methods, and different embodiments have the same parts, the same or similar parts between the various embodiments in this specification can be referred to mutually. They will not be described in detail here.

[0074] It should be noted that in this specification, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "equipped with," or any other variations thereof are intended to cover a non-exclusive arrangement, such that a circuit structure, article, or device comprising a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a circuit structure, article, or device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the circuit structure, article, or device that includes said element.

[0075] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the content of the claims.

Claims

1. A method for image salient object detection, characterized in that, The application relates to a method for constructing an "encoding-decoding" type convolutional neural network model. The "encoding" part adopts a lightweight mobilenetv2 backbone network, and is used for feature extraction of an input picture to generate a group of feature maps containing high-level to low-level feature maps with different scale semantic information. A plurality of fusion side connections are introduced in the "decoding" part, and each level fusion side connection fuses the feature maps of the corresponding level with the feature maps of the previous level in sequence from high level to low level, and uses the fused feature maps to obtain a predicted saliency map and calculate a loss function. The loss function of each level is used for training to obtain a trained "encoding-decoding" type convolutional neural network model. Image data is input into the trained "encoding-decoding" type convolutional neural network model for image salient object detection. Each fusion side connection comprises a feature fusion module, a segmentation head module and an automatic loss calculation module, and a splicing module is arranged between the feature fusion modules of adjacent fusion side connections. The feature maps of the corresponding level and the fused feature maps of the previous level are spliced through the splicing module. The spliced feature maps enter the feature fusion module for feature fusion processing to obtain fused feature maps of the corresponding level. The fused feature maps are input into the segmentation head module to obtain a predicted saliency map of the corresponding level through semantic segmentation. According to the predicted saliency map and a real saliency map, a loss function corresponding to each level is calculated. The spliced feature maps enter the feature fusion module for feature fusion processing to obtain fused feature maps of the corresponding level. After the spliced feature maps pass through three layers of convolution layers, batch normalization layers and activation layers, the spliced feature maps are divided into three branches. The first branch is a straight branch in a residual structure. The second branch is divided into three sub-branches after passing through a 1x1 convolution layer, and the three sub-branches correspond to dilated convolutions with dilated coefficients of 1, 2 and 3 respectively. The outputs of the three sub-branches obtained after the dilated convolutions are added, and then pass through a batch normalization layer and an activation layer, and then pass through a 1x1 convolution layer and are added to the first branch.

2. The method of claim 1, wherein, The first branch and the second branch form a residual structure, and then pass through a 1x1 convolution layer to realize fusion.

3. The method of claim 2, wherein, The third branch sequentially passes through a global pooling layer, a 1x1 convolution layer, an activation layer, a 1x1 convolution layer and a sigmoid layer to obtain a weight coefficient. The weight coefficient is multiplied by the result of the fusion output of the first and second branches to obtain the final feature fusion output. A plurality of detail side connections are introduced in the lightweight mobilenetv2 backbone network, and the detail side connections are used for predicting salient object edges at multiple levels. Each detail side connection comprises a detail head module and a detail loss calculation module. The feature maps of each level pass through the detail head module to obtain salient object edge prediction of the corresponding level, and a detail loss function is calculated based on the edge of the real saliency map at different scales.

4. The method of claim 1, wherein, The lightening process of the mobilenetv2 backbone network is as follows: in the mobilenetv2 network, a last convolution layer, an adaptive average pooling layer and a full connection layer are removed to obtain a light mobilenetv2 backbone network.

5. The method of claim 1, wherein, The segmentation head module comprises a 1x1 convolution layer with an output channel number of 1 and a Sigmoid activation layer.

Images