Image saliency object detection method, system, device and medium

Abstract

The application discloses an image saliency object detection method, system, device and medium; the method comprises the following steps: acquiring an image to be detected; inputting the image to be detected into a trained image saliency object detection model to obtain a saliency object detection result; wherein the trained saliency object detection model is used for: performing feature extraction on the image to be detected; performing multi-receptive field feature learning on the features extracted by each layer of a backbone network, and fusing the features of different receptive fields; performing context learning on the fused features, so that the output features of each layer are interactively aggregated with the output features of the upper and lower layers through a sampling operation, and the saliency object detection result is output. The application promotes the interaction between low-level and high-level features in the feature extraction process, learns context features, connects the independent features of each layer into a continuous whole, and enhances the continuity and integrity of feature extraction.

Description

Technical Field

[0001] This invention relates to the field of image saliency detection technology, and in particular to methods, systems, devices and media for image saliency target detection. Background Technology

[0002] The statements in this section merely refer to the background art related to this invention and do not necessarily constitute prior art.

[0003] The main purpose of salient object detection is to extract the most salient parts of an image, similar to the human attention mechanism. Essentially, it's a binary classification problem of pixels in an image, dividing each pixel into salient and non-salient parts. Salient object detection has applications in many fields, such as image segmentation, object tracking, autonomous driving, and 3D reconstruction. Especially with the development of intelligent transportation and intelligent manufacturing, extracting key information from thousands of images has become crucial. Due to the complex and varied backgrounds and varying object sizes in real-world images, salient object detection faces numerous challenges. Traditional salient object detection methods involve image preprocessing and manual feature extraction, relying too heavily on empirical methods and exhibiting poor robustness. With the development of deep learning, methods based on fully convolutional networks have been proposed, extracting features through deep networks and achieving better results. These methods can be further subdivided into several categories. Because salient objects in real images vary greatly in scale, many methods study multi-scale information learning; small scales can better extract detailed information, while large scales can obtain complete features. To capture finer-grained details, some methods have been proposed to enhance edge information, including supervised and unsupervised boundary learning. Completeness and continuity are crucial for salient object detection, so some methods learn contextual features, including feature fusion at different levels and extreme learning of high-level features.

[0004] The inventors discovered that while existing methods are effective for detecting salient targets, they are insufficient in real-world scenarios where targets often have complex backgrounds and insufficient image clarity and contrast. Furthermore, the methods often fail to adequately address the issues of multiple salient targets coexisting or occluded targets in the acquired images. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method, system, device, and medium for detecting salient objects in images. Based on a supervised contrastive learning encoder-decoder structure, the consistency of different image enhancements is learned by randomly occluding the image, followed by the fusion of multiple receptive field information and contextual learning to enhance the completeness of extracted features.

[0006] On the one hand, methods for detecting salient objects in images are provided, including:

[0007] Acquire the image to be detected;

[0008] The image to be detected is input into the trained image saliency target detection model to obtain the saliency target detection result;

[0009] The trained salient object detection model is used for: extracting features from the image to be detected; learning multi-receptive field features from the features extracted from each layer of the backbone network and fusing features from different receptive fields; learning the context of the fused features so that the fused features of each layer interact and aggregate with the fused features of its upper and lower layers through sampling operations, and outputting the salient object detection result.

[0010] On the other hand, an image salient object detection system is provided, including:

[0011] The acquisition module is configured to acquire the image to be detected.

[0012] The detection module is configured to input the image to be detected into the trained image salient object detection model to obtain the salient object detection result;

[0013] The trained salient object detection model is used for: extracting features from the image to be detected; learning multi-receptive field features from the features extracted from each layer of the backbone network and fusing features from different receptive fields; learning the context of the fused features so that the fused features of each layer interact and aggregate with the fused features of its upper and lower layers through sampling operations, and outputting the salient object detection result.

[0014] Furthermore, an electronic device is also provided, including:

[0015] Memory, used for non-transitory storage of computer-readable instructions; and

[0016] Processor, for executing the computer-readable instructions,

[0017] When the computer-readable instructions are executed by the processor, they perform the method described in the first aspect above.

[0018] In another aspect, a storage medium is also provided for non-transitory storage of computer-readable instructions, wherein when the non-transitory computer-readable instructions are executed by a computer, the instructions of the method described in the first aspect are executed.

[0019] In another aspect, a computer program product is also provided, including a computer program that, when run on one or more processors, is used to implement the method described in the first aspect above.

[0020] The above technical solution has the following advantages or beneficial effects:

[0021] This invention proposes a bi-branch supervised contrastive learning framework based on random occlusion. This framework enhances the model's detection capabilities in scenarios with target occlusion and complex backgrounds by learning the invariance of different data augmentations. The multi-receptive field fusion module designed in this invention considers the intrinsic connections and influences between features of different receptive field sizes, enabling the learning of multi-scale features. This invention promotes the interaction between low-level and high-level features during feature extraction, learns contextual features, and connects independent features from various layers into a continuous whole, enhancing the persistence and completeness of feature extraction. Attached Figure Description

[0022] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0023] Figure 1 This is a flowchart of the salient target detection method in an embodiment of the present invention;

[0024] Figure 2 This is a schematic diagram of the image preprocessing method in an embodiment of the present invention;

[0025] Figure 3 This is a schematic diagram of the trained image saliency target detection model framework in an embodiment of the present invention;

[0026] Figure 4 This is a schematic diagram of the multi-receptive field fusion module method in an embodiment of the present invention;

[0027] Figure 5 This is a schematic diagram of the context learning module method in an embodiment of the present invention;

[0028] Figure 6 This is an embodiment of the present invention. Figure 3 Internal structure diagram of Stage1, Stage2, Stage3 or Stage4 modules;

[0029] Figure 7 It is a visual comparison of the output results of a subset of images in the test set with the results of existing methods. Detailed Implementation

[0030] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0031] Terminology Explanation:

[0032] Feature map: The result of operating on an image by a series of convolutional kernels. It can be regarded as an abstract representation of the original image, with each pixel corresponding to a specific feature.

[0033] Receptive field: The size of the region in the original image that each point on the feature map maps to; that is, the size of the original image that one pixel in the feature map can see.

[0034] Example 1

[0035] This embodiment provides a method for detecting salient targets in images;

[0036] Image salient object detection methods include:

[0037] S101: Acquire the image to be detected;

[0038] S102: Input the image to be detected into the trained image saliency target detection model to obtain the saliency target detection result;

[0039] The trained salient object detection model is used for:

[0040] Feature extraction is performed on the image to be detected; multi-receptive field feature learning is performed on the features extracted from each layer of the backbone network, and features from different receptive fields are fused; context learning is performed on the fused features, so that the fused features of each layer interact and aggregate with the fused features of its upper and lower layers through sampling operations, and outputs the salient target detection results.

[0041] Furthermore, such as Figure 3 As shown, the trained image saliency target detection model has a network structure including: a first branch and a second branch in parallel; the first branch is implemented using a first backbone network; the first backbone network includes: a first-stage module, a second-stage module, a third-stage module, and a fourth-stage module connected in sequence; the second backbone network includes: a fifth-stage module, a sixth-stage module, a seventh-stage module, and an eighth-stage module connected in sequence.

[0042] The output of the third-stage module is connected to the input of the first multi-receptive-field feature fusion module; the output of the fourth-stage module is connected to the input of the second multi-receptive-field feature fusion module; the output of the fifth-stage module is connected to the input of the third multi-receptive-field feature fusion module; the output of the sixth-stage module is connected to the input of the fourth multi-receptive-field feature fusion module; the output of the seventh-stage module is connected to the input of the fifth multi-receptive-field feature fusion module; and the output of the eighth-stage module is connected to the input of the sixth multi-receptive-field feature fusion module.

[0043] The outputs of the first multi-receptive-field feature fusion module and the second multi-receptive-field feature fusion module are both connected to the input of the first context learning module; the output of the first context learning module is connected to the input of the first decoder, and the output of the first decoder is connected to the input of the loss function calculation module.

[0044] The trained image salient object detection model network structure also includes: the output of the third multi-receptive field feature fusion module and the output of the fourth multi-receptive field feature fusion module, both of which are connected to the input of the second context learning module;

[0045] The outputs of the third, fourth, and fifth multi-receptive-field feature fusion modules are all connected to the input of the third context learning module.

[0046] The outputs of the fourth, fifth, and sixth multi-receptive-field feature fusion modules are all connected to the input of the fourth context learning module.

[0047] The outputs of the fifth multi-receptive field feature fusion module and the sixth multi-receptive field feature fusion module are both connected to the input of the fifth context learning module.

[0048] The output of the second context learning module is connected to the input of the second decoder; the output of the third context learning module is connected to the input of the third decoder; the output of the fourth context learning module is connected to the input of the fourth decoder; and the output of the fifth context learning module is connected to the input of the fifth decoder.

[0049] The outputs of the second decoder, the third decoder, the fourth decoder, and the fifth decoder are all connected to the input of the loss function calculation module.

[0050] The output of the fifth decoder is used to output the salient target detection results in the image.

[0051] The decoder includes a convolutional layer and an upsampling layer connected in sequence, wherein the convolutional layer has 64 input channels and 1 output channel.

[0052] Furthermore, the feature extraction of the image to be detected is achieved through the second branch.

[0053] Furthermore, the multi-receptive field feature learning of the features extracted from each layer of the backbone network and the fusion of features from different receptive fields are achieved through the first, second, third, fourth, fifth and sixth multi-receptive field feature fusion modules.

[0054] Furthermore, the context learning of the fused features, which enables the output features of each layer to interact and aggregate with the output features of its upper and lower layers through sampling operations, is achieved through the first, second, third, fourth, and fifth context learning modules.

[0055] It should be understood that for N images {x1,x2,...,xN}, each image is copied into two copies, and a fixed-size, randomly positioned occlusion is added to one copy, resulting in a batch of 2N images, which divides the model into two branches.

[0056] like Figure 3 As shown, the first branch is the mask flow, and the second branch is the original flow. They are represented as... and These two branch roads form a straight line.

[0057] The backbone network outputs multiple features, each representing different scales. The features extracted from the two branches are input into the multi-receptive-field fusion module, whose structure is as follows: Figure 4 As shown.

[0058] Furthermore, the first and fifth stage modules are implemented using Stage 1 of the PVTv2 network; the second and sixth stage modules are implemented using Stage 2 of the PVTv2 network; the third and seventh stage modules are implemented using Stage 3 of the PVTv2 network; and the fourth and eighth stage modules are implemented using Stage 4 of the PVTv2 network.

[0059] Furthermore, such as Figure 6As shown, the internal structures of Stage 1, Stage 2, Stage 3, and Stage 4 of the PVTv2 (Pyramid Vision Transformer v2) network are the same. Stage 1 of the PVTv2 network includes: a patch embedding module, a first adder, and an encoder connected in sequence; the patch embedding module includes: a linear layer and a first normalization layer connected in sequence; the input of the first adder also receives position information.

[0060] The encoder includes: an encoder input, a second normalization layer, a spatial reduction layer, a multi-head attention mechanism layer, a second adder, a third normalization layer, a feedforward neural network layer, and an encoder output, connected in sequence; the input of the second adder is also connected to the input of the second normalization layer; the input of the third adder is also connected to the input of the third normalization layer; and the output of the second normalization layer is also connected to the input of the multi-head attention mechanism layer.

[0061] Furthermore, the spatial reduction layer is an improvement on the traditional multi-head attention module. It reduces the width and height of the keys and values ​​in the multi-head attention module by a set ratio, thereby reducing computational complexity and memory consumption.

[0062] Furthermore, such as Figure 4 As shown, the internal structures of the first, second, third, fourth, fifth, and sixth multi-receptive field feature fusion modules are the same.

[0063] The first multi-receptive-field feature fusion module includes:

[0064] The first convolutional layer has its input terminal used to input the corresponding feature map B;

[0065] The output of the first convolutional layer is connected to the inputs of the third branch, the fourth branch, the fifth branch, the sixth branch, and the seventh branch, respectively.

[0066] The third branch includes: the third branch input, the pooling layer, the second convolutional layer, the first upsampling layer, and the third branch output, which are connected in sequence.

[0067] The fourth branch includes: the fourth branch input, the third convolutional layer, and the fourth branch output connected in sequence;

[0068] The fifth branch consists of: the fifth branch input, the first Concat connection layer, the fourth convolutional layer, and the fifth branch output, which are connected in sequence.

[0069] The sixth branch consists of: the sixth branch input, the second Concat connection layer, the fifth convolutional layer, and the sixth branch output, which are connected in sequence.

[0070] The seventh branch consists of: the seventh branch input, the third Concat connection layer, the sixth convolutional layer, and the seventh branch output, which are connected in sequence.

[0071] Specifically, the input of the first Concat connection layer is also connected to the output of the third convolutional layer; the input of the second Concat connection layer is also connected to the output of the fourth convolutional layer; and the input of the third Concat connection layer is also connected to the output of the fifth convolutional layer.

[0072] The outputs of the third, fourth, fifth, sixth, and seventh branches are all connected to the input of the seventh convolutional layer, and the output of the seventh convolutional layer outputs the multi-receptive field feature fusion result.

[0073] The first Concat connection layer, the second Concat connection layer, and the third Concat connection layer are used to concatenate the input data.

[0074] Furthermore, the first multi-receptive-field feature fusion module acquires multi-scale information from each feature map layer and aggregates the multi-scale information together:

[0075] First, a convolutional layer compresses the features into B along the channel dimension, then divides it into five branches. The final output is the sum of the results obtained from operating on each of the five branches. The specific process is as follows:

[0076] B1 = B

[0077] B i =Concat(B i-1 (B) i = 2, 3, 4

[0078] Among them, B i Represents the input of each branch. Concat(·) means concatenating the input of the current branch with the output of the previous branch.

[0079] The final output is the sum of the results after performing convolution operations on each of the five branches. The specific operations are as follows:

[0080]

[0081] Among them, Conv 1×1 Represents a 1×1 convolution. The 3×3 dilated convolutions represent different dilation rates. Upsample(·) is the downsampling operation, and GAP(•) represents the global average pooling operation.

[0082] The multi-receptive field fusion module concatenates the features extracted from each branch with the original features before extracting features at a larger scale.

[0083] Furthermore, such as Figure 5 As shown, the internal structures of the first context learning module, the second context learning module, the third context learning module, the fourth context learning module, and the fifth context learning module are the same. The j-th context learning module includes:

[0084] Three input terminals: first input terminal, second input terminal, and third input terminal; the value of j can be one, two, three, four, or five.

[0085] The first input terminal is used to downsample the features output by the j-th multi-receptive field feature fusion module, and then input the downsampled data to the fourth adder;

[0086] The second input terminal is used to input the features output by the (j+1)th multi-receptive field feature fusion module into the fourth adder;

[0087] The third input terminal is used to upsample the features output by the (j+2)th multi-receptive field feature fusion module, and then input the upsampled data to the fourth adder.

[0088] The output of the fourth adder is connected to the input of the convolution module, the output of the convolution module is connected to the input of the channel attention module, and the output of the channel attention module is the output of the j-th context learning module.

[0089] Furthermore, the convolutional module includes: an eighth convolutional layer, a batch normalization layer, and an activation function layer connected in sequence. The convolutional module consists of 3×3 convolutions, followed by feature normalization and activation operations for feature extraction.

[0090] The channel attention module has the following formula:

[0091]

[0092] In this context, ⊙ represents element-wise multiplication. This represents a 1×1 convolution with feature normalization and activation operations.

[0093] Furthermore, the j-th context learning module regards the backbone network stage module corresponding to the j-th context learning module as the current stage module stage; the output features of the previous stage module stage of the backbone network current stage module stage are first subjected to multi-receptive field feature fusion processing, and then downsampled;

[0094] Similarly, the output features of the next stage module of the current stage module of the backbone network are first processed by multi-receptive field feature fusion, and then upsampled.

[0095] The result of fusing the output features of the current stage module after passing through multiple receptive field features is called the intermediate result.

[0096] The upsampling result, downsampling result, and intermediate result are simultaneously input into the convolution module, and then the output value of the convolution module is input into the channel attention module. The channel attention module outputs the complete contextual features.

[0097] It should be understood that the backbone network stage module corresponding to the j-th context learning module is regarded as the current stage module. Here, the first context learning module corresponds to stage4 of the first branch, the second context learning module corresponds to stage1 of the second branch, the third context learning module corresponds to stage2 of the second branch, the fourth context learning module corresponds to stage3 of the second branch, and the fifth context learning module corresponds to stage4 of the second branch.

[0098] Furthermore, among the features extracted in the above steps, shallow feature extraction can obtain information such as contours, while deep feature extraction focuses more on semantic information.

[0099] The context enhancement module enables the interaction of features across different layers. If the model simply performs chain propagation, the layers will not be interconnected, resulting in a loss of contextual integrity.

[0100] After passing through this module, the features of adjacent layers are effectively aggregated together, and the resulting features have complete contextual information.

[0101] Furthermore, such as Figure 1 As shown, the trained image saliency target detection model includes the following training process:

[0102] A training set is constructed, consisting of two images with known target detection results. The two images include a first enhanced image and a second enhanced image. The first enhanced image is the original image after adding a mask and enhancement. The second enhanced image is the original image after enhancement.

[0103] The first enhanced image is input to the input of the first branch; the first branch performs feature extraction on the first enhanced image;

[0104] The second enhanced image is input into the input of the second branch; the second branch extracts features from the second enhanced image.

[0105] The output features of the third and fourth stage modules are fused using multiple receptive field features, and then context learning is performed on the fused feature results to obtain the learning results of the first branch.

[0106] Multi-receptive field feature fusion is performed on the features output from the fifth, sixth, seventh, and eighth stage modules. Then, context learning is performed on the fused feature results to obtain the learning results of the second branch.

[0107] Calculate the loss function between the learning results of the first branch and the learning results of the second branch. When the total loss function value no longer decreases, or when the number of iterations exceeds the set number, stop training to obtain the trained image saliency target detection model.

[0108] Furthermore, the total loss function is calculated as follows:

[0109] L total =L bce +L iou ;

[0110] The total loss function is set inside the loss function calculation module;

[0111] The cross-entropy loss is as follows:

[0112]

[0113] Where W and H are the weight and height of the image, (x,y) represents the pixel coordinates, S(x,y) represents the predicted value, and G(x,y) represents the label.

[0114] The IoU loss function between predicted and actual values ​​is calculated as follows:

[0115]

[0116] Where W and H are the weight and height of the image, (x,y) represents the pixel coordinates, S(x,y) represents the predicted value, and G(x,y) represents the label.

[0117] The supervision strategy consists of two parts: loss calculation between the final output of the occluded branch and the ground truth value, and loss calculation for the original branch. The original branch employs deep supervision to extract features from each layer. To calculate the loss with the label values, the output channel count is compressed to one dimension, and the loss is calculated using the method described above. Finally, the final losses are weighted, and all losses calculated for both branches are summed.

[0118] For example, the construction of the training set includes: acquiring real scene videos from field devices and extracting key frames that need to be detected as RGB images to be detected.

[0119] Videos of specific road scenes are collected, and specific keyframes are extracted from them, including those with complex backgrounds and challenging structures. The images are then labeled to form training and testing sets.

[0120] Because the differences between consecutive frames in a video scene are small, to avoid overfitting the model training results, the keyframe extraction time interval is 1 second, thus obtaining many images with realistic scenes containing salient targets. The images are then binarized and labeled to obtain salient labels. Following the hold-out method, the dataset is proportionally divided into two disjoint parts: a training set and a test set.

[0121] The trained image salient object detection model is used to detect salient objects in scene images, obtain the most noteworthy regions in the image, and eliminate background and occlusion interference, so as to prepare for subsequent task decision-making and deployment.

[0122] like Figure 2 As shown, the enhanced original image is obtained by randomly occluding the input keyframe image for data augmentation. Augmentation methods include, but are not limited to: contrast enhancement, random cropping, and random flipping.

[0123] The enhanced input and the enhanced and masked original image are respectively input into the backbone network for feature extraction. Then, through the multi-receptive field information learning module, different receptive field features are learned and the intrinsic relationship between different receptive field features is found. After that, through the context learning module, the output of each layer is sampled and interacted with the output of its upper and lower layers to enhance the integrity and continuity of the features.

[0124] By using a trained image salient object detection model, keyframes are extracted from real-time video data for salient object detection. This allows subsequent tasks to focus more on the most important parts of the image, thereby removing many redundant parts and avoiding complicating the problem.

[0125] The input image is split into two branches after undergoing two different feature augmentations. Then, a randomly occluded image of a predetermined size is added to one of the branches. The two branches are treated as facing each other for comparison to learn the invariance of the different data augmentations.

[0126] Specifically, these two branches extract features through a backbone network (selecting PVTv2 as the backbone network). These branches extract multiple features from different receptive fields through a backbone network composed of convolutional layers, pooling layers, and normalization layers. Each feature is then learned through multi-scale learning via a multi-receptive field fusion module, revealing the relationships and influences between multiple receptive field features.

[0127] Features learned through the context enhancement module are used to achieve the integrity and continuity of the objective.

[0128] In the loss calculation, features from all original branches are used to achieve the purpose of deep supervision, while occluded branches only use the final output features to minimize the correct distance.

[0129] During network training, the input was scaled to [352, 352] using an augmentation method incorporating random pruning, with the augmented occlusion size set to [50, 50]. A stochastic gradient descent (SGD) optimizer with a momentum of 0.9 and weight decay of 5e-4 was employed. The initial learning rate was set to 0.01, and the batch size to 8. A warm-up strategy and a linear decay strategy were used to adjust the learning rate.

[0130] During the testing phase, images from the test set are input into the network model, and the output saliency map is saved. Finally, the mean absolute error (MAE), maximum F metric (maxF), augmented alignment metric (Em), and saliency metric (Sm) are calculated between the labeled labels and the saliency map output by the model to test the model's performance.

[0131] Specifically, during the testing phase, the images in the test set are augmented with the same data as during training and then input into the saliency detection network. It should be noted that the network in the testing phase is different from the two-branch network in the training phase. During the testing phase, the occlusion branch added during network training is removed, and the saliency map is generated end-to-end directly from the original images. Figure 7 This involves a visual comparison of the output results for a subset of images in the test set with the results of existing methods. Then, the similarity between different methods on the test set is calculated based on the original and labeled images, specifically using four metrics: MAE, maxF, Em, and Sm for comparison.

[0132] The saliency detection network was obtained through optimized training, and the parameters of each convolutional layer in the network were saved after optimization.

[0133] The network training process involves augmenting the training dataset with data augmentations, including contrast enhancement, random cropping, and random flipping. After two different feature augmentations, the input image is divided into two branches, and a fixed-size randomized mask is added to one branch. The two branches are then compared side-by-side to learn the invariance of different data augmentations.

[0134] Specifically, these two branches extract features through a backbone network (selecting PVTv2 as the backbone network) and learn each feature through multi-scale learning via a multi-receptive field fusion module. Features learned through a context enhancement module ensure the integrity and continuity of the objective.

[0135] Image annotation involves using a threshold segmentation algorithm to annotate each pixel, identifying salient target regions, and obtaining ground truth labels.

[0136] In loss calculation, features from all original streams are used to achieve deep supervision, while the masked stream only uses the final output features to minimize the correct distance. The loss function utilizes cross-entropy loss and IOU loss for network optimization.

[0137] The process of outputting a prediction map through supervised contrastive learning is as follows: the network prediction map of an unoccluded branch is upsampled to obtain the final prediction map.

[0138] This invention acquires real-world video footage, extracts keyframe images from the video, and utilizes a trained image salient object detection model to detect salient objects in each frame. The trained model is configured to: first, preprocess the image by dividing it into two branches for different image enhancement methods; then, obtain feature maps through channel transformation and feature refinement; finally, obtain prediction maps at various scales through dilated convolutions with different dilation rates; then, fuse and compress the features at different scales using a multi-receptive field fusion module; next, use a context learning module to upsample and sum the outputs of each stage of the network to obtain more complete and continuous features; and finally, upsample the features to obtain the final output prediction map. This method utilizes a detection network to detect salient objects, improving detection accuracy while maintaining processing speed.

[0139] Example 2

[0140] This embodiment provides an image salient target detection system, including:

[0141] The acquisition module is configured to acquire the image to be detected.

[0142] The detection module is configured to input the image to be detected into the trained image salient object detection model to obtain the salient object detection result;

[0143] The trained salient object detection model is used for: extracting features from the image to be detected; learning multi-receptive field features from the features extracted from each layer of the backbone network and fusing features from different receptive fields; learning the context of the fused features so that the fused features of each layer interact and aggregate with the fused features of its upper and lower layers through sampling operations, and outputting the salient object detection result.

[0144] It should be noted that the acquisition module and detection module described above correspond to steps S101 to S102 in Embodiment 1. The examples and application scenarios implemented by the modules and corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the modules described above, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0145] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0146] The proposed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of modules described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0147] Example 3

[0148] This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein, the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform the method described in Embodiment 1.

[0149] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0150] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0151] In the implementation process, each step of the above method can be completed by the integrated logic circuits in the processor hardware or by software instructions.

[0152] The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0153] Those skilled in the art will recognize that the units and algorithm steps described in connection with the various examples of this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0154] Example 4

[0155] This embodiment also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the method described in Embodiment 1.

[0156] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image salient target detection method, characterized by: include: Acquire the image to be detected; The image to be detected is input into the trained image saliency target detection model to obtain the saliency target detection result; The trained salient object detection model is used for: extracting features from the image to be detected; learning multi-receptive field features from the features extracted from each layer of the backbone network and fusing features from different receptive fields; learning the context of the fused features so that the fused features of each layer interact and aggregate with the fused features of its upper and lower layers through sampling operations, and outputting the salient object detection results. The trained image saliency object detection model includes the following training process: A training set is constructed, consisting of two images with known target detection results. The two images include a first enhanced image and a second enhanced image. The first enhanced image is the original image after adding a mask and enhancement. The second enhanced image is the original image after enhancement. The first enhanced image is input to the input of the first branch; the first branch performs feature extraction on the first enhanced image; The second enhanced image is input into the input of the second branch; the second branch extracts features from the second enhanced image. The output features of the third and fourth stage modules are fused using multiple receptive field features, and then context learning is performed on the fused feature results to obtain the learning results of the first branch. Multi-receptive field feature fusion is performed on the features output from the fifth, sixth, seventh, and eighth stage modules. Then, context learning is performed on the fused feature results to obtain the learning results of the second branch. Calculate the loss function between the learning results of the first branch and the learning results of the second branch. When the total loss function value no longer decreases, or when the number of iterations exceeds the set number, stop training to obtain the trained image saliency target detection model.

2. The image salient target detection method as described in claim 1, characterized in that, The trained image salient object detection model has a network structure including: a first branch and a second branch in parallel; the first branch is implemented using a first backbone network; the first backbone network includes: a first-stage module, a second-stage module, a third-stage module, and a fourth-stage module connected in sequence; the second backbone network includes: a fifth-stage module, a sixth-stage module, a seventh-stage module, and an eighth-stage module connected in sequence. The output of the third-stage module is connected to the input of the first multi-receptive-field feature fusion module; the output of the fourth-stage module is connected to the input of the second multi-receptive-field feature fusion module; the output of the fifth-stage module is connected to the input of the third multi-receptive-field feature fusion module; the output of the sixth-stage module is connected to the input of the fourth multi-receptive-field feature fusion module; the output of the seventh-stage module is connected to the input of the fifth multi-receptive-field feature fusion module; and the output of the eighth-stage module is connected to the input of the sixth multi-receptive-field feature fusion module. The outputs of the first multi-receptive-field feature fusion module and the second multi-receptive-field feature fusion module are both connected to the input of the first context learning module; the output of the first context learning module is connected to the input of the first decoder, and the output of the first decoder is connected to the input of the loss function calculation module.

3. The image salient target detection method as described in claim 2, characterized in that, The trained image saliency target detection model network structure also includes: the output of the third multi-receptive field feature fusion module and the output of the fourth multi-receptive field feature fusion module are both connected to the input of the second context learning module; the output of the third multi-receptive field feature fusion module, the output of the fourth multi-receptive field feature fusion module and the output of the fifth multi-receptive field feature fusion module are all connected to the input of the third context learning module. The outputs of the fourth, fifth, and sixth multi-receptive-field feature fusion modules are all connected to the input of the fourth context learning module; the outputs of the fifth and sixth multi-receptive-field feature fusion modules are all connected to the input of the fifth context learning module. The output of the second context learning module is connected to the input of the second decoder; the output of the third context learning module is connected to the input of the third decoder; the output of the fourth context learning module is connected to the input of the fourth decoder; and the output of the fifth context learning module is connected to the input of the fifth decoder. The outputs of the second decoder, the third decoder, the fourth decoder, and the fifth decoder are all connected to the input of the loss function calculation module. The output of the fifth decoder is used to output the salient target detection results in the image.

4. The image salient target detection method as described in claim 1, characterized in that, The first multi-receptive-field feature fusion module acquires multi-scale information from each feature map layer and aggregates the multi-scale information together: First, a convolutional layer compresses the features into B along the channel dimension, then divides it into five branches. The final output is the sum of the results obtained from operating on each of the five branches. The specific process is as follows: in, The input represents each branch. This represents connecting the input of the current branch with the output of the previous branch; The final output is the sum of the results after performing convolution operations on each of the five branches. The specific operations are as follows: in, represent convolution, Representing different void ratios Hollow convolution, It is a downsampling operation. This represents the global average pooling operation.

5. The image salient target detection method as described in claim 3, characterized in that, The internal structures of the first, second, third, fourth, and fifth context learning modules are the same. The context learning module includes: Three input terminals: first input terminal, second input terminal, and third input terminal; The value range is one, two, three, four, or five; The first input terminal is used for the first... The features output by the multi-receptive-field feature fusion module are downsampled, and then the downsampled data is input into the fourth adder. The second input terminal is used for the first... The features output by the multi-receptive-field feature fusion module are input into the fourth adder; The third input terminal is used for the first... The features output by the multi-receptive-field feature fusion module are upsampled, and then the upsampled data is input into the fourth adder. The output of the fourth adder is connected to the input of the convolution module, and the output of the convolution module is connected to the input of the channel attention module. The output of the channel attention module is the first... The output of the context learning module.

6. The image salient target detection method as described in claim 5, characterized in that, The first The context learning module will... The backbone network stage corresponding to the context learning module is regarded as the current stage module; the output features of the previous stage module of the backbone network are first processed by multi-receptive field feature fusion, and then downsampled. Similarly, the output features of the next stage module of the current stage module of the backbone network are first processed by multi-receptive field feature fusion, and then upsampled. The result of fusing the output features of the current stage module after passing through multiple receptive field features is called the intermediate result. The upsampling result, downsampling result, and intermediate result are simultaneously input into the convolution module, and then the output value of the convolution module is input into the channel attention module. The channel attention module outputs the complete contextual features.

7. An image salient target detection system, characterized in that, include: The acquisition module is configured to acquire the image to be detected. The detection module is configured to input the image to be detected into the trained image salient object detection model to obtain the salient object detection result; The trained salient object detection model is used for: extracting features from the image to be detected; learning multi-receptive field features from the features extracted from each layer of the backbone network and fusing features from different receptive fields; learning the context of the fused features so that the fused features of each layer interact and aggregate with the fused features of its upper and lower layers through sampling operations, and outputting the salient object detection results. The trained image saliency object detection model includes the following training process: A training set is constructed, consisting of two images with known target detection results. The two images include a first enhanced image and a second enhanced image. The first enhanced image is the original image after adding a mask and enhancement. The second enhanced image is the original image after enhancement. The first enhanced image is input to the input of the first branch; the first branch performs feature extraction on the first enhanced image; The second enhanced image is input into the input of the second branch; the second branch extracts features from the second enhanced image. The output features of the third and fourth stage modules are fused using multiple receptive field features, and then context learning is performed on the fused feature results to obtain the learning results of the first branch. Multi-receptive field feature fusion is performed on the features output from the fifth, sixth, seventh, and eighth stage modules. Then, context learning is performed on the fused feature results to obtain the learning results of the second branch. Calculate the loss function between the learning results of the first branch and the learning results of the second branch. When the total loss function value no longer decreases, or when the number of iterations exceeds the set number, stop training to obtain the trained image saliency target detection model.

8. An electronic device, characterized in that, include: A memory for non-transitory storage of computer-readable instructions; and a processor for executing the computer-readable instructions. When the computer-readable instructions are executed by the processor, they perform the method described in any one of claims 1-6.

9. A storage medium characterized by being non-transitory. The system stores computer-readable instructions, wherein, when the non-transitory computer-readable instructions are executed by a computer, the instructions of the method according to any one of claims 1-6 are executed.

Images