Method, device, equipment and storage medium for extracting salient regions of images

Abstract

The present invention provides a method, apparatus, device and storage medium for extracting a salient region of an image, comprising: extracting N feature maps of different levels of a target image based on a pre-trained salient region extraction model; i ; For M feature maps f i Each feature map f in i , the feature map f i Fuse it with its high-frequency information to generate the feature map f i The corresponding high-frequency enhanced feature map h j ; Based on each high frequency enhanced feature map h j The boundary information of the high-frequency enhanced feature map h1 is enhanced, and the initial recognition result of the salient area of ​​the target image is generated based on the high-frequency enhanced feature map h1 after the boundary information is enhanced; based on the feature map f N and the initial recognition result of the salient region to generate a first fused feature map; the first fused feature map, the initial recognition result of the salient region and at least one feature map f i The salient regions of the target are extracted based on the second fused feature map. The above method can more accurately extract the salient regions of the target.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and storage medium for extracting a salient region of an image. Background Art

[0002] Humans are able to rapidly allocate attention to important regions in a visual scene. Understanding and simulating this amazing ability, known as visual attention or visual saliency, is a fundamental research question in psychology, neurobiology, cognitive science, and computer vision.

[0003] There are many salient object detection algorithms currently. However, the image information used by the existing salient object detection algorithms is not sufficient and detailed enough, which will lead to inaccurate results of the salient regions extracted by them. Since the confidence of the loss function used in training the salient object detection model is low, the boundaries of the salient regions output by the salient object detection model are blurred, which will also lead to inaccurate results of the salient regions extracted by them, and the consistency of the salient regions cannot be guaranteed. Summary of the invention

[0004] The present disclosure provides a method, device, equipment and storage medium for extracting a salient region of an image.

[0005] According to a first aspect of the present disclosure, a method for extracting a salient region of an image is provided, comprising:

[0006] Based on the pre-trained salient region extraction model, N feature maps of different levels of the target image are extracted. i , where i is a positive integer not greater than N, and feature graphs f1 to f N The levels are raised successively;

[0007] Feature maps f at N different levels i Determine M feature maps f i , for M feature maps f i Each feature map f in i , the feature map f i Fuse it with its high-frequency information to generate the feature map f i The corresponding high-frequency enhanced feature map h j , where j is a positive integer not greater than M;

[0008] Based on each high-frequency enhanced feature map h j The boundary information of the high-frequency enhanced feature map h1 is enhanced, and an initial recognition result of a salient region of the target image is generated based on the high-frequency enhanced feature map h1 after the boundary information is enhanced;

[0009] Based on the feature map f Nand the initial recognition result of the salient region to generate a first fused feature map;

[0010] The first fused feature map, the initial recognition result of the salient region and at least one feature map f i Merge to obtain a second fusion feature map;

[0011] A salient region target recognition result of the target image is generated based on the second fused feature map, and a target salient region is extracted based on the salient region target recognition result.

[0012] In the embodiment of the present disclosure, for M feature graphs f i Each feature map f in i , the feature map f i Fuse it with its high-frequency information to generate the feature map f i The corresponding high-frequency enhanced feature map h i ,include:

[0013] For M feature maps f i Each feature map f in i , get the feature map f i The average value of the features after pooling;

[0014] Calculate the feature map f i The difference with the feature average value is used as the feature map f i High-frequency information;

[0015] Calculate the feature map f i The sum of the high-frequency information is used as the feature map f i The corresponding high-frequency enhanced feature map h j .

[0016] In the embodiment of the present disclosure, the salient region extraction model includes (M-1) boundary enhancement units k j , boundary enhancement unit k j With the high frequency enhancement feature map h j+1 Correspondingly, it is used to strengthen the boundary information of the high-frequency enhanced feature map h1;

[0017] Based on each high-frequency enhanced feature map h j The boundary information of the high-frequency enhanced feature map h1 is enhanced, and the initial recognition result of the salient area of ​​the target image is generated based on the high-frequency enhanced feature map h1 after the boundary information is enhanced, including:

[0018] When j=1, the high-frequency enhancement feature map h1 and the high-frequency enhancement feature map h2 are input to the boundary enhancement unit k1, and the high-frequency enhancement feature map h1 with the first boundary enhancement is output based on the boundary enhancement unit k1;

[0019] When 1 < j < M, the high-frequency enhanced feature map h j+1 and the high-frequency enhanced feature map h1 after the (j - 1)-th boundary enhancement are input into the boundary enhancement unit k j , and based on the boundary enhancement unit k j , the high-frequency enhanced feature map h1 after the j-th boundary enhancement is output;

[0020] Based on the high-frequency enhanced feature map h1 after the (M - 1)-th boundary enhancement, an initial recognition result of the salient region of the target image is generated.

[0021] In the embodiments of the present disclosure, generating a first fusion feature map based on the feature map f N and the initial recognition result of the salient region includes:

[0022] Extracting multi-scale information from the feature map f N ;

[0023] Adjusting the size and number of channels of the initial recognition result of the salient region to be the same as those of the feature map f N ;

[0024] Concatenating the multi-scale information and the adjusted initial recognition result of the salient region to obtain a first fusion feature map.

[0025] In the embodiments of the present disclosure, merging the first fusion feature map, the initial recognition result of the salient region, and at least one feature map f i to obtain a second fusion feature map includes:

[0026] Doubling the size of the first fusion feature map;

[0027] Adjusting the size and number of channels of the initial recognition result of the salient region to be the same as those of the first fusion feature map with the doubled size;

[0028] Fusing the first fusion feature map with the doubled size and the adjusted initial recognition result of the salient region to obtain an intermediate fusion feature map;

[0029] Merging the intermediate fusion feature map and at least one feature map f i to obtain a second fusion feature map.

[0030] In the embodiments of the present disclosure, the total loss for training the salient region extraction model includes a segmentation loss and a boundary loss; the segmentation loss is used to characterize the accuracy of the classification result of the pixel points in the salient region, and the boundary loss is used to characterize the accuracy of the position of the boundary pixel points in the salient region.

[0031] In the embodiments of the present disclosure, the calculation formula for the segmentation loss is:

[0032] Loss seg =BCE+λ2CL, Loss seg is the segmentation loss, Loss seg is the segmentation loss, BCE is the cross entropy loss, CL is the consistency enhancement loss function, and λ2 is the trade-off parameter between the cross entropy loss and the consistency enhancement loss function.

[0033] In the embodiment of the present disclosure, the calculation formula of the boundary loss is:

[0034] β is the percentage of non-edge pixels in the true image; p i and g i are the predicted value and true value of the i-th pixel, respectively, and K is the number of pixels.

[0035] According to a second aspect of the present disclosure, there is provided a device for extracting a salient region of an image, comprising:

[0036] The feature extraction module is used to extract N different levels of feature maps f of the target image based on the pre-trained salient region extraction model. i , where i is a positive integer not greater than N, and feature graphs f1 to f N The levels are raised successively;

[0037] High-frequency enhancement module, used for feature maps f at N different levels i Determine M feature maps f i , for M feature maps f i Each feature map f in i , the feature map f i Fuse it with its high-frequency information to generate the feature map f i The corresponding high-frequency enhanced feature map h j , where j is a positive integer not greater than M;

[0038] Boundary enhancement module, used to enhance the feature map h based on each high frequency j The boundary information of the high-frequency enhanced feature map h1 is enhanced, and an initial recognition result of a salient region of the target image is generated based on the high-frequency enhanced feature map h1 after the boundary information is enhanced;

[0039] The first fusion feature generation module is used to generate the feature map f N and the initial recognition result of the salient region to generate a first fused feature map;

[0040] The second fusion feature generation module is used to combine the first fusion feature map, the initial recognition result of the salient region and at least one feature map f i Merge to obtain a second fusion feature map;

[0041] The target saliency region extraction module is used to generate the saliency region target recognition result of the target image based on the second fused feature map, and extract the target saliency region based on the saliency region target recognition result.

[0042] In the embodiment of the present disclosure, when the high-frequency enhancement module is used to fuse each feature map f i in M feature maps f i with its high-frequency information to generate the high-frequency enhanced feature map h i corresponding to the feature map f i , it is specifically used for: i Specifically, for each feature map f

[0043] in M feature maps f i , obtain the average value of the features after pooling the feature map f i ; i Calculate the difference between the feature map f

[0044] and the average value of the features, and use the difference as the high-frequency information of the feature map f i ; i Calculate the sum value of the feature map f

[0045] and the high-frequency information, and use the sum value as the high-frequency enhanced feature map h i corresponding to the feature map f i , that is, the high-frequency enhanced feature map h j .

[0046] In the embodiment of the present disclosure, the saliency region extraction model includes (M - 1) boundary enhancement units k j , and the boundary enhancement unit k j corresponds to the high-frequency enhanced feature map h j+1 , and is used to strengthen the boundary information of the high-frequency enhanced feature map h1;

[0047] When the boundary strengthening module is used to strengthen the boundary information of the high-frequency enhanced feature map h1 based on each high-frequency enhanced feature map h j and generate the initial saliency region recognition result of the target image based on the high-frequency enhanced feature map h1 with strengthened boundary information, it is specifically used for:

[0048] When j = 1, input the high-frequency enhanced feature map h1 and the high-frequency enhanced feature map h2 into the boundary enhancement unit k1, and output the high-frequency enhanced feature map h1 with the first boundary strengthening based on the boundary enhancement unit k1;

[0049] When 1 < j < M, input the high-frequency enhanced feature map h j+1 and the high-frequency enhanced feature map h1 with the (j - 1)th boundary strengthening into the boundary enhancement unit k j, based on the boundary enhancement unit k j Output the high-frequency enhanced feature map h1 of the j-th boundary enhancement and input the high-frequency enhanced feature map h1 of the j-th boundary enhancement;

[0050] The initial recognition result of the salient area of ​​the target image is generated based on the high-frequency enhanced feature map h1 with the (M-1)th boundary enhancement.

[0051] In the embodiment of the present disclosure, the first fusion feature generation module is used to generate a feature map based on the feature map. N When generating the first fused feature map with the initial recognition result of the salient region, it is specifically used for:

[0052] From the feature map f N Extract multi-scale information from

[0053] Adjust the size and number of channels of the initial recognition result of the salient region to match the feature map f N The size and number of channels are consistent;

[0054] The multi-scale information and the adjusted initial recognition results of the salient regions are concatenated to obtain the first fused feature map.

[0055] In the embodiment of the present disclosure, the second fusion feature generation module is used to combine the first fusion feature map, the initial recognition result of the salient region and at least one feature map f i When merging to obtain the second fused feature map, it is specifically used for:

[0056] Double the size of the first fused feature map;

[0057] Adjust the size and number of channels of the initial recognition result of the salient region to be consistent with the size and number of channels of the first fused feature map after the size is enlarged;

[0058] The first fused feature map after the size increase is fused with the adjusted initial recognition result of the salient region to obtain an intermediate fused feature map;

[0059] The intermediate fusion feature map and at least one feature map f i Merge to obtain the second fused feature map.

[0060] In the embodiment of the present disclosure, the total loss of training the salient region extraction model includes segmentation loss and boundary loss; the segmentation loss is used to characterize the accuracy of the classification results of the pixel points of the salient region, and the boundary loss is used to characterize the accuracy of the position of the boundary pixel points of the salient region.

[0061] In the embodiment of the present disclosure, the calculation formula of the segmentation loss is:

[0062] Loss seg=BCE+λ2CL, Loss seg is the segmentation loss, Loss seg is the segmentation loss, BCE is the cross entropy loss, CL is the consistency enhancement loss function, and λ2 is the trade-off parameter between the cross entropy loss and the consistency enhancement loss function;

[0063] The calculation formula of boundary loss is:

[0064] β is the percentage of non-edge pixels in the true image; p i and g i are the predicted value and true value of the i-th pixel, respectively, and K is the number of pixels.

[0065] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0066] at least one processor; and a memory communicatively coupled to the at least one processor;

[0067] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor so that the at least one processor can execute the method for extracting a significant area of ​​an image provided by the first aspect.

[0068] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the method for extracting a salient region of an image provided by the first aspect.

[0069] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description.

[0070] The beneficial effects of the technical solution provided by the present disclosure are:

[0071] The method for extracting a salient region of an image provided by the present invention uses a salient region extraction model to extract multi-level deep features of a target image, making it easier to understand the overall meaning of the entire image and the detailed structure of the target, and to obtain more detailed information of the target image. In addition, a process for detecting the boundaries of salient regions by enhancing high-frequency features is established, making it easier to more accurately extract the salient regions of the target by combining boundary detection and salient region detection in the subsequent step. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0073] Figure 1A schematic diagram showing a flow chart of a method for extracting a salient region of an image provided by an embodiment of the present disclosure;

[0074] Figure 2 A flowchart of a specific example of a method for extracting a salient region of an image provided by an embodiment of the present disclosure is shown;

[0075] Figure 3 A schematic diagram of a process for obtaining a high-frequency enhanced feature map h1 with enhanced boundaries provided by an embodiment of the present disclosure is shown;

[0076] Figure 4 A schematic diagram of a process for obtaining a first fusion feature map provided in an embodiment of the present disclosure is shown;

[0077] Figure 5 A schematic diagram of a process of extracting a salient region of an image provided by an embodiment of the present disclosure is shown;

[0078] Figure 6 A schematic block diagram of an example electronic device that can be used to implement the method for extracting a salient region of an image provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0079] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0080] Humans are able to rapidly allocate attention to important regions in a visual scene. Understanding and simulating this amazing ability, known as visual attention or visual saliency, is a fundamental research question in psychology, neurobiology, cognitive science, and computer vision.

[0081] Humans are able to quickly allocate attention to important regions in visual scenes. Understanding and simulating this amazing ability, namely visual attention or visual saliency, is a fundamental research problem in psychology, neurobiology, cognitive science, and computer vision. The rise of salient object detection is driven by a wide range of object-level computer vision applications. Most early non-deep salient object detections rely on low-level features and certain heuristics (e.g., color contrast, background priors).

[0082] With the rise of data-driven deep learning technology in recent years, salient object detection has developed rapidly with its help. Since its first introduction in 2015, deep learning-based salient object detection algorithms have quickly shown superior performance over traditional solutions and have remained at the top of various rankings.

[0083] Salient object detection has become a very popular research direction in the field of computer vision and has been widely used in many fields. In the field of computer vision, such as visual tracking, image retrieval, non-photorealistic rendering, 4D saliency detection, quality assessment of synthetic images without reference, semantic segmentation, etc. Exemplary applications in robotics, such as human-computer interaction and object discovery, also benefit from salient object detection to better understand scenes / objects. Although deep learning methods have made great progress compared to traditional methods, how to obtain more refined saliency predictions and how to improve the spatial consistency of predictions are still issues worthy of attention.

[0084] There are many salient object detection algorithms, but many of them are not ideal, mainly due to the following two aspects: On the one hand, saliency is mainly defined on the global contrast of the entire image, rather than local or pixel features. In order to obtain accurate results, the saliency detection method must understand the overall meaning of the entire image and the detailed structure of the target. However, the image information used by the existing salient object detection algorithms is not sufficient and detailed. On the other hand, most salient object detection methods use cross entropy as the training loss function, but the cross entropy loss has a low confidence when judging boundary pixels, resulting in very blurred boundaries and unable to guarantee the consistency of salient areas.

[0085] In summary, the image information used by the existing salient object detection algorithms is not sufficient and detailed enough, which will lead to inaccurate results of the salient regions extracted. Since the confidence of the loss function used in training the salient object detection model is low, the boundaries of the salient regions output by the salient object detection model are blurred, which also leads to inaccurate results of the extracted salient regions and the inability to guarantee the consistency of the salient regions.

[0086] The method, device, equipment and storage medium for extracting a salient region of an image provided by the embodiments of the present disclosure are intended to solve at least one of the above technical problems in the prior art.

[0087] The method for extracting a salient region of an image provided by the present invention uses a salient region extraction model to extract multi-level deep features of a target image, making it easier to understand the overall meaning of the entire image and the detailed structure of the target, and to obtain more detailed information of the target image. In addition, a process for detecting the boundaries of salient regions by enhancing high-frequency features is established, making it easier to more accurately extract the salient regions of the target by combining boundary detection and salient region detection in the subsequent step.

[0088] The disclosed embodiment provides a method for extracting salient regions of an image, which is implemented based on a pre-trained salient region extraction model. The salient region extraction model can be trained using several public commonly used salient image data sets. During the model training process, the model parameters can be continuously adjusted based on the loss using back propagation until the loss converges to a smaller value, and the model parameters are saved to obtain a trained salient region extraction model.

[0089] Figure 1 A schematic diagram of a process of extracting a salient region of an image provided by an embodiment of the present disclosure is shown. Figure 1 The method may mainly include the following steps:

[0090] S110: Extract N feature maps of different levels of the target image based on the pre-trained salient region extraction model i .

[0091] The target image may be an image acquired in real time using a camera, or an image acquired from a preset image library or storage device, or an image received from other devices. The embodiment of the present disclosure does not limit the source of the target image.

[0092] i is a positive integer not greater than N, and feature graphs f1 to f N The specific value of N can be determined according to the actual design needs. For example, N can be 5. The salient region extraction model can use ResNet-50 trained on ImageNet. Based on the salient region extraction model, the target image is feature extracted. The void rate of the last block of the salient region extraction model is set to 2 to expand the receptive field. Figure 2 As shown in the figure, the original target image is input and passed through five convolutional layers Conv1-Conv5 to gradually obtain feature information of different scales from low-level to high-level. Specifically, five different levels of feature maps with sizes of 1 / 2, 1 / 4, 1 / 8, 1 / 16 and 1 / 16 of the original target image size can be obtained. These five different levels of feature maps are feature map f1, feature map f2, feature map f3, feature map f4 and feature map f5 respectively. These feature maps of different scales represent rich feature information from low-level to high-level, respectively.

[0093] S120: Feature maps f at N different levels i Determine M feature maps f i , for M feature maps f i Each feature map f in i , the feature map f i Fuse it with its high-frequency information to generate the feature map f i The corresponding high-frequency enhanced feature map h j .

[0094] It can be understood that M is not greater than N, and j is a positive integer not greater than M. For example, M is set to 4, and the feature maps f at the above five different levels can be i Four feature maps, feature map f1, feature map f2, feature map f3 and feature map f5, are determined. Taking feature map f1 as an example, feature map f1 can be fused with the high-frequency information of feature map f1 to generate a high-frequency enhanced feature map h1 corresponding to feature map f1.

[0095] Optionally, in step S120, for M feature graphs f i Each feature map f in i , get the feature map f i The average value of the feature after pooling can be expressed as AvgPool(f i ). After obtaining the feature map f i After averaging the features after pooling, calculate the feature map f i The difference with the feature average value is used as the feature map f i That is, the high-frequency information can be expressed as (f i -AvgPool(f i )). Then calculate the feature map f i The sum of the high-frequency information is used as the feature map f i The corresponding high-frequency enhanced feature map h j It can be understood that the high-frequency enhancement feature map h j The calculation formula is as follows:

[0096] h i =2f i -AvgPool(f i )

[0097] f i is the i-th feature map f i, AvgPool represents the mean pooling operation. Through the above method, the high-frequency enhanced feature maps h1, high-frequency enhanced feature map h2, high-frequency enhanced feature map h3 and high-frequency enhanced feature map h4 corresponding to the feature map f1, feature map f2, feature map f3 and feature map f5 can be obtained respectively.

[0098] The disclosed embodiment fuses the feature map with its high-frequency information, thereby enhancing the high-frequency information of the feature map itself and obtaining a high-frequency enhanced feature map. Based on the high-frequency enhanced feature map, the boundary of the salient area can be detected more accurately.

[0099] S130: Based on each high frequency enhancement feature map h j The boundary information of the high-frequency enhanced feature map h1 is enhanced, and an initial recognition result of a salient area of ​​the target image is generated based on the high-frequency enhanced feature map h1 after the boundary information is enhanced.

[0100] In the embodiment of the present disclosure, the high frequency enhanced feature map h j Taking the high-frequency enhancement feature map h1, the high-frequency enhancement feature map h2, the high-frequency enhancement feature map h3 and the high-frequency enhancement feature map h4 as examples, the high-frequency enhancement feature map h2, the high-frequency enhancement feature map h3 and the high-frequency enhancement feature map h4 can be used to sequentially enhance the boundary information of the high-frequency enhancement feature map h1, and the salient area of ​​the target image can be identified based on the high-frequency enhancement feature map h1 containing the information of each high-frequency enhancement feature map, to obtain the initial recognition result of the salient area.

[0101] Optionally, the salient region extraction model includes (M-1) boundary enhancement units k j , boundary enhancement unit k j With the high frequency enhancement feature map h j+1 Correspondingly, it is used to strengthen the boundary information of the high-frequency enhancement feature map h1. For example, Figure 2 As shown, M can be set to 4, and the salient region extraction model includes a boundary enhancement unit k1 corresponding to the high-frequency enhancement feature map h2, a boundary enhancement unit k2 corresponding to the high-frequency enhancement feature map h3, and a boundary enhancement unit k3 corresponding to the high-frequency enhancement feature map h4.

[0102] Optionally, step S130 may include the following steps:

[0103] (a) When j=1, the high-frequency enhanced feature map h1 and the high-frequency enhanced feature map h2 are input to the boundary enhancement unit k1, and the high-frequency enhanced feature map h1 with the first boundary enhancement is output based on the boundary enhancement unit k1.

[0104] Specifically, Figure 3As shown in the figure, the high-frequency enhanced feature map h2 is passed through a 1×1 convolution to reduce the number of channels, so that the number of channels of the high-frequency enhanced feature map h2 is the same as that of the high-frequency enhanced feature map h1. Then, an upsampling operation is performed on the high-frequency enhanced feature map h2 to ensure that the size of the high-frequency enhanced feature map h2 is the same as that of the high-frequency enhanced feature map h1. After that, the processed high-frequency enhanced feature map h2 and the high-frequency enhanced feature map h1 are added element by element to obtain the high-frequency enhanced feature map h1 with the first boundary enhancement.

[0105] (b) When 1 < j < M, the high-frequency enhanced feature map h j+1 and the high-frequency enhanced feature map h1 with the (j - 1)th boundary enhancement are input into the boundary enhancement unit k j , and based on the boundary enhancement unit k j , the high-frequency enhanced feature map h1 with the jth boundary enhancement is output.

[0106] As Figure 2 shown, after obtaining the high-frequency enhanced feature map h1 with the first boundary enhancement, the high-frequency enhanced feature map h3 and the high-frequency enhanced feature map h1 with the first boundary enhancement can be input into the boundary enhancement unit k2, and based on the boundary enhancement unit k2, the high-frequency enhanced feature map h1 with the second boundary enhancement is output; then, the high-frequency enhanced feature map h4 and the high-frequency enhanced feature map h1 with the second boundary enhancement are input into the boundary enhancement unit k3, and based on the boundary enhancement unit k3, the high-frequency enhanced feature map h1 with the third boundary enhancement is output.

[0107] (c) Based on the high-frequency enhanced feature map h1 with the (M - 1)th boundary enhancement, an initial recognition result of the salient region of the target image is generated. The initial recognition result of the salient region includes information related to the boundary of the salient region.

[0108] It can be understood that when M is set to 4, an initial recognition result of the salient region of the target image can be generated based on the high-frequency enhanced feature map h1 with the third boundary enhancement described above.

[0109] The processing process of the boundary enhancement unit for the feature map can be represented by the following formula:

[0110] edge j+1 = Up(Trans(h j+1 )) + Res(edge j )

[0111] where edge j+1 is the output of the current (j + 1)th boundary enhancement unit, h j is the jth high-frequency enhanced feature map after high-frequency information enhancement; edge jis the output of the jth boundary enhancement unit; Trans is a combination of convolution, normalization, and nonlinear change operations, Up is an upsampling operation, and Res is a convolution of a residual structure. From low to high layers, the boundary information in the feature layer is enhanced in turn, and non-boundary information is filtered. After multiple enhancements and filtering, the boundary of the salient area is obtained.

[0112] The disclosed embodiment can significantly enhance the boundary information of the feature map by superimposing multiple high-frequency enhanced feature maps corresponding to the multi-level deep features of the target image, thereby ensuring that a more accurate initial recognition result of the salient area including the boundary of the salient area can be obtained.

[0113] S140: Based on feature map f N The first fused feature map is generated by combining the initial recognition result of the salient region.

[0114] In the embodiment of the present disclosure, the feature map f N At the same time, the size and number of channels of the initial recognition result of the salient region are adjusted to be consistent with the feature map f N The size and number of channels are consistent; the multi-scale information and the adjusted initial recognition result of the salient region are spliced ​​to obtain the first fused feature map.

[0115] It can be understood that the feature map f N contains more detailed information of the target image, and the feature map f N After being fused with the initial recognition results of the salient regions, the information in the feature map can be further enriched, which helps to improve the detection accuracy of the salient regions.

[0116] Specifically, the feature map f N That is, the top-level feature map f5 mentioned above. The feature map f5 is processed by the ASPP module to extract its multi-scale information. For example, the feature map f5 can be output to five modules in parallel. The first module is average pooled, and then upsampled to restore the resolution after 1x1 convolution channel transformation. The second to fifth modules are sampled in parallel with different ratios of 1, 6, 12, and 18 dilated convolutions, respectively. Multi-scale information is obtained through the above five modules. After that, the initial recognition result of the salient area can be resized to make the size of the initial recognition result of the salient area consistent with the size of the feature map f5; the initial recognition result of the salient area is subjected to 1×1 convolution to change the number of channels, so that the number of channels of the initial recognition result of the salient area is consistent with the number of channels of the feature map f5. The multi-scale information and the adjusted initial recognition result of the salient area are spliced ​​to obtain the first fused feature map.

[0117] S150: The first fused feature map, the initial recognition result of the salient region and at least one feature map f iMerge to obtain the second fused feature map.

[0118] Optionally, in step S150, the size of the first fused feature map may be doubled, and then the size and number of channels of the initial salient region recognition result may be adjusted to be consistent with the size and number of channels of the first fused feature map after the size increase. The first fused feature map after the size increase and the adjusted initial salient region recognition result are fused to obtain an intermediate fused feature map; the intermediate fused feature map and at least one feature map are fused. i The two features are merged to obtain the second fused feature map. More detailed information of the target image is obtained. In addition, a process of detecting the boundary of the salient area by enhancing high-frequency features is established, which facilitates the subsequent joint boundary detection and salient area detection to more accurately extract the target salient area. The feature map containing more detailed and accurate boundary information is fused with the original feature map for the salient area detection process, ensuring that the boundary of the target salient area is identified with high clarity and accuracy.

[0119] Specifically, step S150 is a process of reverse step-by-step feature fusion, such as Figure 2 As shown, the first fused feature map can be defined as the first fused feature map f6. Starting from the first fused feature map f6, the first fused feature map f6 is first upsampled to double its size, and the size and number of channels of the initial recognition result of the salient region are adjusted to be consistent with the size and number of channels of the first fused feature map f6 after the size is enlarged; the first fused feature map f6 after the size is enlarged and the adjusted initial recognition result of the salient region are element-wise added to obtain the intermediate fused feature map. Here, the size of the intermediate fused feature map is consistent with the size of the feature map f3, and the intermediate fused feature map and the feature map f3 can be directly merged along the channel dimension. After the merging is completed, 1×1 and 3×3 convolutions are also added to reduce the channel dimension and reduce the amount of parameter calculation; this is done in this way, and finally the first fused feature map f6, feature map f3 and feature map f2 of different scales are fused step by step. Here, for each fused feature map, two convolution operations of 3×3 and 1×1 are added to it to obtain a sub-output result, and the last sub-output result is used as the final output result, and the final output result is the above-mentioned second fused feature map.

[0120] S160: Generate a salient region target recognition result of the target image based on the second fused feature map, and extract a target salient region based on the salient region target recognition result.

[0121] In the disclosed embodiment, the total loss of training the salient region extraction model includes segmentation loss and boundary loss. The segmentation loss is used to characterize the accuracy of the classification results of the pixels of the salient region, and the boundary loss is used to characterize the accuracy of the positions of the boundary pixels of the salient region.

[0122] The total loss of training the salient region extraction model can be expressed as follows:

[0123] Loss=Loss seg +λ1Loss edge

[0124] Among them, Loss is the total loss of training the salient region extraction model, Loss seg is the segmentation loss, λ1Loss edge is the boundary loss, λ1 is the trade-off parameter between the two segmentation losses and the boundary loss, and its value can be determined according to actual needs. For example, λ1 can be 1.

[0125] The disclosed embodiment combines boundary detection and salient region detection to extract target salient regions. Therefore, during the model training process, boundary loss and segmentation loss are configured for boundary detection and salient region detection, and the two losses are used to adjust model parameters to ensure that the salient region extraction model has better boundary detection capability and salient region detection capability, so that the salient region extraction model can extract the target salient region more accurately.

[0126] In the embodiment of the present disclosure, the calculation formula of the segmentation loss is:

[0127] Loss seg =BCE+λ2CL, Loss seg is the segmentation loss, Loss seg is the segmentation loss, BCE is the cross entropy loss, CL is the consistency enhancement loss function, and λ2 is the trade-off parameter between the cross entropy loss and the consistency enhancement loss function.

[0128] Furthermore, the calculation formula of the cross entropy loss BCE is:

[0129]

[0130] p i and g i They represent the predicted value and true value of the i-th pixel respectively, and K represents the number of pixels.

[0131] Furthermore, the calculation formula of the consistency enhancement loss CL is:

[0132]

[0133]

[0134] C is the number of categories in the target image, K i is the number of pixels in the i-th category; is the average value of pixel prediction in the i-th category; ReLU is a nonlinear activation function; p i,j is the predicted value of the jth pixel in the ith category.

[0135] In the embodiment of the present disclosure, the calculation formula of the boundary loss is:

[0136]

[0137] β is the percentage of non-edge pixels in the true image; p i and g i are the predicted value and true value of the i-th pixel, respectively, and K is the number of pixels.

[0138] On the other hand, most salient object detection methods use cross entropy as the training loss function, but the cross entropy loss has low confidence when judging boundary pixels, resulting in very blurred boundaries and unable to guarantee the consistency of salient areas.

[0139] The segmentation loss of the disclosed embodiment includes two losses: cross entropy loss and consistency enhancement loss. The consistency enhancement loss considers the foreground salient area and the background area respectively, so that the prediction probability of pixels of the same class tends to be consistent, so that the salient area can be highlighted more evenly, thereby ensuring that the boundaries of the identified salient area are clearer.

[0140] It should be noted here that for images whose target areas need to be detected, the following methods are usually used to detect the target areas in the images:

[0141] Method 1: Extract multiple feature maps of different levels of the image step by step, and directly use the feature maps of different levels to identify the target area in the image. Method 2: Extract multiple feature maps of different levels of the image step by step, determine the top-level feature map, and identify the target area in the image based on the top-level feature map. Method 3: Extract multiple feature maps of different levels of the image step by step, fuse the top-level features with the lower-level features, and independently identify the target area in the image based on each fused feature map.

[0142] The present disclosure embodiment aims at the extraction process of the salient region of the target image, and aims to find the most salient target region in the target image, that is, to detect the specific target object in the target image. The feature maps of multiple different levels extracted in the present disclosure embodiment are used to characterize at least one of the color features, texture features, shape features and spatial relationship features of the image. After obtaining the feature maps of multiple different levels through the convolution operation, the present disclosure embodiment does not directly detect the salient region based on the top-level features, nor does it directly fuse the top-level features and the low-level features, but continues to extract the high-frequency information of the feature map of the feature map based on the pooling operation, and further re-fuse the extracted high-frequency information with the corresponding feature map to obtain a high-frequency enhanced feature map. It should be noted here that the high-frequency information corresponds to the part of the image that changes dramatically, that is, the edge (contour) or noise and detail part of the image. Generally speaking, the grayscale between adjacent areas is very different, and there is usually a significant difference. The grayscale of this part changes very quickly, that is, the part with high frequency of change. Therefore, the grayscale value of the edge of the image changes quickly, which corresponds to high frequency, that is, high frequency display of the edge of the image. Since the high-frequency enhanced feature map is obtained by further superimposing its own high-frequency information on the original feature map, the high-frequency enhanced feature map has richer high-frequency information. The initial recognition results of the salient areas based on these high-frequency enhanced feature maps contain more accurate boundary information, which facilitates the identification of clearer boundaries. Afterwards, the first fused feature map of the upper layer is obtained based on the fusion of the top-level information and the boundary information. The first fused feature map is fused with the original low-level feature map in the reverse direction to obtain the second fused feature map. It can be understood that the second fused feature map contains rich boundary information and is compatible with information of different scales in feature maps of each level. The salient area is detected based on the second fused feature, which can not only detect the salient area of ​​target detection more accurately, but also ensure that the boundary of the salient area of ​​target detection has a high clarity.

[0143] Based on the same principle as the above-mentioned method for extracting a salient region of an image, the embodiment of the present disclosure provides a device for extracting a salient region of an image. Figure 5 FIG. 1 is a schematic diagram of a salient region extraction device for an image provided by an embodiment of the present disclosure. Figure 5 As shown, the image salient region extraction device includes a feature extraction module, a high frequency enhancement module, a boundary enhancement module, a first fusion feature generation module, a second fusion feature generation module and a target salient region extraction module.

[0144] The feature extraction module is used to extract N different levels of feature maps f of the target image based on the pre-trained salient region extraction model. i , where i is a positive integer not greater than N, and feature graphs f1 to f N The levels increase in sequence.

[0145] The high-frequency enhancement module is used to enhance the feature maps f at N different levels. i Determine M feature maps f i , for M feature maps f i Each feature map f in i , the feature map f i Fuse it with its high-frequency information to generate the feature map f i The corresponding high-frequency enhanced feature map h j , where j is a positive integer not greater than M.

[0146] The boundary enhancement module is used to enhance the feature map h based on each high frequency j The boundary information of the high-frequency enhanced feature map h1 is enhanced, and an initial recognition result of a salient area of ​​the target image is generated based on the high-frequency enhanced feature map h1 after the boundary information is enhanced.

[0147] The first fusion feature generation module is used to generate a feature map based on the feature map f N The first fused feature map is generated by combining the initial recognition result of the salient region.

[0148] The second fusion feature generation module is used to combine the first fusion feature map, the initial recognition result of the salient region and at least one feature map f i Merge to obtain the second fused feature map.

[0149] The target salient region extraction module is used to generate a salient region target recognition result of the target image based on the second fusion feature map, and extract the target salient region based on the salient region target recognition result.

[0150] The image salient region extraction device provided by the present disclosure uses a salient region extraction model to extract multi-level deep features of a target image, making it easier to understand the overall meaning of the entire image and the detailed structure of the target, and to obtain more detailed information of the target image. In addition, a process for performing boundary detection of salient regions by enhancing high-frequency features is established, making it easier to more accurately extract target salient regions by combining boundary detection and salient region detection in the subsequent step.

[0151] In the embodiment of the present disclosure, the high frequency enhancement module is used for the M feature maps f i Each feature map f in i , the feature map f i Fuse it with its high-frequency information to generate the feature map f i The corresponding high-frequency enhanced feature map h i When, specifically used for:

[0152] For M feature maps f i Each feature map f in i, obtain the feature map f i The average value of the features after pooling;

[0153] Calculate the feature map f i The difference from the average feature value, and use the difference as the high-frequency information of the feature map f i ;

[0154] Calculate the sum of the feature map f i And the high-frequency information, and use the sum as the high-frequency enhanced feature map h corresponding to the feature map f i ; j .

[0155] In the embodiment of the present disclosure, the saliency region extraction model includes (M - 1) boundary enhancement units k j , and the boundary enhancement unit k j Corresponds to the high-frequency enhanced feature map h j+1 , and is used to strengthen the boundary information of the high-frequency enhanced feature map h1;

[0156] When the boundary strengthening module is used to strengthen the boundary information of the high-frequency enhanced feature map h1 based on each high-frequency enhanced feature map h j , and generate an initial recognition result of the saliency region of the target image based on the high-frequency enhanced feature map h1 with strengthened boundary information, it is specifically used for:

[0157] When j = 1, input the high-frequency enhanced feature map h1 and the high-frequency enhanced feature map h2 into the boundary enhancement unit k1, and output the high-frequency enhanced feature map h1 with the first boundary strengthening based on the boundary enhancement unit k1;

[0158] When 1 < j < M, input the high-frequency enhanced feature map h j+1 And the high-frequency enhanced feature map h1 with the (j - 1)th boundary strengthening into the boundary enhancement unit k j , and output the high-frequency enhanced feature map h1 with the jth boundary strengthening based on the boundary enhancement unit k j ;

[0159] Generate an initial recognition result of the saliency region of the target image based on the high-frequency enhanced feature map h1 with the (M - 1)th boundary strengthening.

[0160] In the embodiment of the present disclosure, when the first fusion feature generation module is used to generate the first fusion feature map based on the feature map f N And the initial recognition result of the saliency region, it is specifically used for:

[0161] Extract multi-scale information from the feature map f N ;

[0162] Adjust the size and number of channels of the initial recognition result of the salient region to match the feature map f N The size and number of channels are consistent;

[0163] The multi-scale information and the adjusted initial recognition results of the salient regions are concatenated to obtain the first fused feature map.

[0164] In the embodiment of the present disclosure, the second fusion feature generation module is used to combine the first fusion feature map, the initial recognition result of the salient region and at least one feature map f i When merging to obtain the second fused feature map, it is specifically used for:

[0165] Double the size of the first fused feature map;

[0166] Adjust the size and number of channels of the initial recognition result of the salient region to be consistent with the size and number of channels of the first fused feature map after the size is enlarged;

[0167] The first fused feature map after the size increase is fused with the adjusted initial recognition result of the salient region to obtain an intermediate fused feature map;

[0168] The intermediate fusion feature map and at least one feature map f i Merge to obtain the second fused feature map.

[0169] In the embodiment of the present disclosure, the total loss of training the salient region extraction model includes segmentation loss and boundary loss; the segmentation loss is used to characterize the accuracy of the classification results of the pixel points of the salient region, and the boundary loss is used to characterize the accuracy of the position of the boundary pixel points of the salient region.

[0170] In the embodiment of the present disclosure, the calculation formula of the segmentation loss is:

[0171] Loss seg =BCE+λ2CL, Loss seg is the segmentation loss, Loss seg is the segmentation loss, BCE is the cross entropy loss, CL is the consistency enhancement loss function, and λ2 is the trade-off parameter between the cross entropy loss and the consistency enhancement loss function;

[0172] The calculation formula of boundary loss is:

[0173] β is the percentage of non-edge pixels in the true image; p i and g i are the predicted value and true value of the i-th pixel, respectively, and K is the number of pixels.

[0174] It can be understood that the above-mentioned modules of the image salient region extraction device in the embodiment of the present disclosure have the function of implementing the corresponding steps of the above-mentioned image salient region extraction method. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions. The above-mentioned modules can be software and / or hardware, and the above-mentioned modules can be implemented separately or integrated with multiple modules. For the functional description of each module of the above-mentioned image salient region extraction device, please refer to the corresponding description of the above-mentioned image salient region extraction method, which will not be repeated here.

[0175] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0176] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0177] Figure 6 A schematic block diagram of an example electronic device that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0178] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0179] Multiple components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.;

[0180] Storage unit 608, such as a magnetic disk, optical disk, etc., and communication unit 609, such as a network card, modem, wireless communication transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0181] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the image salient region extraction method. For example, in some embodiments, the image salient region extraction method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the image salient region extraction method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the image salient region extraction method in any other appropriate manner (for example, by means of firmware).

[0182] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0183] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0184] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0185] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0186] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0187] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0188] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0189] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for extracting a salient region of an image, comprising: Based on the pre-trained salient region extraction model, N feature maps of different levels of the target image are extracted. i , where i is a positive integer not greater than N, and feature graphs f1 to f N The levels are raised successively; In the feature maps f at the N different levels i Determine M feature maps f i , for the M feature maps f i Each feature map f in i , the feature map f i Fuse it with its high frequency information to generate the feature map f i The corresponding high-frequency enhanced feature map h j , where j is a positive integer not greater than M; Based on each of the high frequency enhanced feature maps h j Strengthening the boundary information of the high-frequency enhanced feature map h1, and generating an initial recognition result of the salient region of the target image based on the high-frequency enhanced feature map h1 after the boundary information is strengthened; Based on the feature map f N and the initial recognition result of the salient region to generate a first fused feature map; The first fused feature map, the initial recognition result of the salient region and at least one of the feature maps f i Merge to obtain a second fusion feature map; A salient region target recognition result of the target image is generated based on the second fused feature map, and a target salient region is extracted based on the salient region target recognition result.

2. The method according to claim 1, wherein: The M feature maps f i Each feature map f in i , the feature map f i Fuse it with its high frequency information to generate the feature map f i The corresponding high-frequency enhanced feature map h i ,include: For the M feature maps f i Each feature map f in i , obtain the feature map f i The average value of the features after pooling; Calculate the feature map f i The difference between the feature average value and the feature map f i High-frequency information; Calculate the feature map f i and the sum of the high frequency information, and taking the sum as the feature map f i The corresponding high-frequency enhanced feature map h j .

3. The method according to claim 1, wherein: The salient region extraction model includes (M-1) boundary enhancement units k j , boundary enhancement unit k j With the high frequency enhancement feature map h j+1 Correspondingly, it is used to strengthen the boundary information of the high-frequency enhanced feature map h1; Based on each of the high frequency enhancement feature maps h j The boundary information of the high-frequency enhanced feature map h1 is enhanced, and an initial recognition result of a salient region of the target image is generated based on the high-frequency enhanced feature map h1 after the boundary information is enhanced, including: When j=1, the high-frequency enhancement feature map h1 and the high-frequency enhancement feature map h2 are input to the boundary enhancement unit k1, and the high-frequency enhancement feature map h1 with the first boundary enhancement is output based on the boundary enhancement unit k1; When 1 < j < M, the high-frequency enhanced feature map h j+1 and the high-frequency enhanced feature map h1 obtained by the (j - 1)-th boundary enhancement are input into the boundary enhancement unit k j , and based on the boundary enhancement unit k j , the high-frequency enhanced feature map h1 obtained by the j-th boundary enhancement is output; An initial recognition result of a salient region of the target image is generated based on the (M-1)th boundary-enhanced high-frequency enhanced feature map h1.

4. The method according to claim 1, wherein: Based on the feature map f N and the initial recognition result of the salient region to generate a first fused feature map, including: From the feature map f N Extract multi-scale information from The size and number of channels of the initial recognition result of the salient region are adjusted to be consistent with the feature map f N The size and number of channels are consistent; The multi-scale information and the adjusted initial recognition result of the salient region are spliced ​​to obtain a first fused feature map.

5. The method according to claim 1, characterized in that The first fusion feature map, the initial recognition result of the salient region and at least one of the feature maps f i Merge to obtain the second fused feature map, including: Double the size of the first fused feature map; Adjusting the size and the number of channels of the initial salient region recognition result to be consistent with the size and the number of channels of the first fused feature map after the size is enlarged; Fusing the first fused feature map after the size increase with the adjusted initial recognition result of the salient region to obtain an intermediate fused feature map; The intermediate fusion feature map and at least one of the feature maps f i Merge to obtain the second fused feature map.

6. The method according to claim 1, characterized in that The total loss of training the salient region extraction model includes segmentation loss and boundary loss; The segmentation loss is used to characterize the accuracy of the classification results of the pixels in the salient area, and the boundary loss is used to characterize the accuracy of the positions of the boundary pixels in the salient area.

7. The method according to claim 6, characterized in that include: The calculation formula of the segmentation loss is: Loss seg =BCE+λ2CL, Loss seg is the segmentation loss, Loss seg is the segmentation loss, BCE is the cross entropy loss, CL is the consistency enhancement loss function, and λ2 is the trade-off parameter between the cross entropy loss and the consistency enhancement loss function; The calculation formula of the boundary loss is: β is the percentage of non-edge pixels in the true image; p i and g i are the predicted value and true value of the i-th pixel respectively, and K is the number of pixels.

8. A device for extracting a salient region of an image, comprising: The feature extraction module is used to extract N different levels of feature maps f of the target image based on the pre-trained salient region extraction model. i , where i is a positive integer not greater than N, and feature graphs f1 to f N The levels are raised successively; The high-frequency enhancement module is used to generate the feature maps f at the N different levels. i Determine M feature maps f i , for the M feature maps f i Each feature map f in i , the feature map f i Fuse it with its high frequency information to generate the feature map f i The corresponding high-frequency enhanced feature map h j , where j is a positive integer not greater than M; A boundary enhancement module is used to enhance the high frequency feature map h based on each of the high frequency enhancement feature maps h j Strengthening the boundary information of the high-frequency enhanced feature map h1, and generating an initial recognition result of the salient region of the target image based on the high-frequency enhanced feature map h1 after the boundary information is strengthened; The first fusion feature generation module is used to generate a fusion feature based on the feature map f N and the initial recognition result of the salient region to generate a first fused feature map; The second fusion feature generation module is used to combine the first fusion feature map, the initial recognition result of the salient region and at least one of the feature maps f i Merge to obtain a second fusion feature map; The target salient region extraction module is used to generate a salient region target recognition result of the target image based on the second fused feature map, and extract a target salient region based on the salient region target recognition result.

9. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

Images