Salient object detection method and system based on local and global context fusion

Through the significance object detection method of local and global context fusion, the feature extraction network and Transformer encoder are used to solve the problem of poor performance of feature aggregation and complex scene detection in the prior art, and efficient and accurate significance object detection is achieved .

CN114581747BActive Publication Date: 2024-11-01XI AN JIAOTONG UNIV +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210211754.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-04
Publication Date
2024-11-01
Estimated Expiration
2042-03-04

AI Technical Summary

Technical Problem

The existing significance object detection method has poor significance detection performance in multi-level features and complex scenarios of different layers of the aggregated fully convolutional network, and designing complex fusion structures increases computational overhead.

Method used

A significant object detection method based on local and global context fusion was designed. Through feature extraction network, splicing and fusion network, local context extraction network and global context extraction network, combined with U-shaped autoencoder and Transformer encoder, extract and fuse Local and global features, a simple fusion strategy is used to generate significant object detection results.

Benefits of technology

It improves the efficiency and accuracy of significance detection in complex scenarios, reduces the amount of network parameters and calculations, can better position significant objects, and provide better edge information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114581747B_ABST
    Figure CN114581747B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and system for salient target detection based on local and global context fusion, the method comprising the following steps: obtaining an RGB picture to be detected as a salient target and inputting a pre-trained salient target detection model to obtain a salient target detection result; wherein the salient target detection model comprises: a feature extraction network for obtaining multi-scale features; a splicing fusion network for obtaining low-level detail features and high-level semantic features; a local context extraction network for outputting delicate local context features; a global context extraction network for outputting delicate global context features; and a fusion output network for outputting salient target detection results. The present invention can effectively extract local and global context information, and can improve the efficiency and accuracy of salient detection in complex scenes.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a salient object detection method and system based on local and global context fusion. Background Art

[0002] Salient object detection refers to accurately segmenting the most interesting areas in a given image that attract human attention. This type of task has extensive and meaningful real-world applications in areas such as information retrieval and public safety, such as: using salient content as a guide for image description; using salient detection to guide scene classification under unsupervised feature learning for high-resolution satellite images; converting unsupervised learning into multi-instance learning to achieve positioning and classification tasks; providing object-level cues to help unsupervised video object segmentation; providing image salient contour information for object detection tasks, etc.

[0003] At present, there are still three problems in the existing salient object detection methods, including:

[0004] (1) Existing work mainly focuses on how to aggregate multi-level features from different layers of fully convolutional networks, which is still an open problem in the deep learning community. At the same time, the existing methods do not perform well in saliency detection in complex situations such as when the image background and foreground are similar (low contrast).

[0005] (2) Although there are many networks that extract local and global information and fuse the two, it is still challenging to design an effective network to extract local and global features;

[0006] (3) Most of the previous work designed very complex fusion structures in order to better integrate local and global information.

[0007] In summary, it is meaningful to design useful local and global feature extraction networks as well as simple and computationally inexpensive fusion methods for the problem of salient object detection. Summary of the invention

[0008] The purpose of the present invention is to provide a method and system for salient object detection based on local and global context fusion to solve one or more of the above-mentioned technical problems. The present invention can effectively extract local and global context information, designs a simple and effective fusion strategy that can fully utilize the advantages of local and context information, and can improve the efficiency and accuracy of salient detection in complex scenes.

[0009] In order to achieve the above object, the present invention adopts the following technical solutions:

[0010] The present invention provides a salient object detection method based on local and global context fusion, comprising the following steps:

[0011] Obtain the RGB image to be detected for salient objects and input it into the pre-trained salient object detection model to obtain the salient object detection result;

[0012] The salient object detection model includes:

[0013] Feature extraction network, used to obtain multi-scale features of the input RGB image;

[0014] A splicing fusion network is used to fuse the preset first several layers of features in the multi-scale features to obtain low-level detail features; and is used to fuse the preset last several layers of features in the multi-scale features to obtain high-level semantic features;

[0015] A local context extraction network, used to input the low-level detail features and output delicate local context features;

[0016] A global context extraction network, used to input the high-level semantic features and output delicate global context features;

[0017] The fusion output network is used to input and fuse the delicate local context features and the delicate global context features to obtain context fusion features; based on the context fusion features, the salient target detection results are output.

[0018] A further improvement of the method of the present invention is that the local context extraction network comprises:

[0019] A low-level feature fusion network is used to input the low-level detail features, perform channel number uniform processing, and output fused low-level detail features;

[0020] The local feature refinement network is used to input the fused low-level detail features, perform refinement processing, and output delicate local context features; wherein the local feature refinement network is an autoencoder.

[0021] A further improvement of the method of the present invention is that the local feature refinement network is a U-type autoencoder;

[0022] In the U-shaped autoencoder, a short connection operation is added to each layer of upsampling decoding, and the features of the downsampling encoding are combined with the features of the upsampling decoding to compensate for the limited context aggregation caused by the downsampling operation on the features.

[0023] A further improvement of the method of the present invention is that the global context extraction network specifically includes:

[0024] A high-level feature fusion network is used to input the high-level semantic features, perform uniform channel number processing, and output fused high-level semantic features;

[0025] The global feature refinement network is used to input the fused high-level semantic features, perform refinement processing, and output delicate global context features; wherein the local feature refinement network is a Transformer-based autoencoder.

[0026] A further improvement of the method of the present invention is that the global feature refinement network comprises:

[0027] Continuous convolution operations and pooling operations are used to input the fused high-level semantic features and perform continuous downsampling encoding to obtain semantic features encoded by the convolutional neural network;

[0028] Transformer encoder, which is used to re-encode the semantic features encoded by the convolutional neural network and output the features encoded by the Transformer;

[0029] A decoder based on a convolutional neural network is used to input the features encoded by the Transformer and obtain global context features by upsampling with short connections;

[0030] Among them, each layer of the encoding block of the Transformer encoder is composed of Layer Norm, Multi-HeadAttention, Dropout and a multi-layer perceptron.

[0031] A further improvement of the method of the present invention is that the step of acquiring the pre-trained salient object detection model comprises:

[0032] Acquire a training sample set; wherein each sample in the training sample set includes an RGB sample image and a corresponding true saliency image;

[0033] The salient object detection model is trained based on the training sample set, and after the loss function value reaches a preset convergence condition, the pre-trained salient object detection model is obtained;

[0034] Among them, the loss function value is the loss function value between the predicted saliency image of the RGB sample picture of each sample and the true saliency image corresponding to the RGB sample picture; the loss function value is obtained by summing three different loss functions, and the three loss functions are BCE loss, SSIM loss and IOU loss respectively.

[0035] A further improvement of the method of the present invention is that when the salient object detection model is trained based on the training sample set, a stochastic gradient descent algorithm is used to optimize model parameters.

[0036] A further improvement of the method of the present invention is that in the fusion output network, delicate local context features and delicate global context features are input and fused to obtain context fusion features; based on the context fusion features, the step of outputting the salient object detection result specifically includes:

[0037] The fine local context features and fine global context features are input into a convolution layer with a convolution kernel of 1, and the number of channels is transformed to be consistent with the fused high-level semantic features; after splicing in the channel dimension, they are input into a ReLU layer to obtain enhanced local and global context features;

[0038] The enhanced local and global context features are concatenated in the channel dimension and input into a ReLU layer to obtain the context fusion feature;

[0039] The context fusion feature is input into a convolution layer with a convolution kernel of 1 and a Softmax operation is performed on it to obtain a saliency map with a channel number of 1.

[0040] A further improvement of the method of the present invention is that, in the splicing fusion network, the steps of fusing the preset first several layers of features in the multi-scale features to obtain low-level detail features; and fusing the preset last several layers of features in the multi-scale features to obtain high-level semantic features specifically include:

[0041] The preset first several layers of features in the multi-scale features are spliced ​​and fused in the channel dimension to obtain low-level detail features;

[0042] The preset last few layers of features in the multi-scale features are spliced ​​and fused in the channel dimension to obtain high-level semantic features.

[0043] The present invention provides a salient object detection system based on local and global context fusion, comprising:

[0044] The detection module is used to obtain the RGB image to be detected by the salient object and input it into the pre-trained salient object detection model to obtain the salient object detection result;

[0045] The salient object detection model includes:

[0046] Feature extraction network, used to obtain multi-scale features of the input RGB image;

[0047] A splicing fusion network is used to fuse the preset first several layers of features in the multi-scale features to obtain low-level detail features; and is used to fuse the preset last several layers of features in the multi-scale features to obtain high-level semantic features;

[0048] A local context extraction network, used to input the low-level detail features and output delicate local context features;

[0049] A global context extraction network, used to input the high-level semantic features and output delicate global context features;

[0050] The fusion output network is used to input and fuse the delicate local context features and the delicate global context features to obtain context fusion features; based on the context fusion features, the salient target detection results are output.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] The present invention specifically provides a method for processing salient target detection using local and global contexts. Since the local context contains rich detail information, it can provide delicate edge features for objects in the image. The global context has rich semantic features, which is conducive to locating salient areas. The present invention takes the local context into consideration and designs an effective deep network to extract more delicate local detail features. Therefore, the method of the present invention can provide better edge information than existing traditional methods. The present invention takes into account that the global context plays an important role in this task, and also takes into account that traditional convolution operations cannot expand the receptive field sufficiently to extract rich global features.

[0053] The present invention introduces a Transformer encoder to enhance the global context from a sequence-to-sequence perspective and highlight the positions of salient objects. Therefore, the method of the present invention can better locate salient objects in complex scenes than existing methods.

[0054] In order to better utilize the extracted local and global contexts, the present invention provides a simple fusion structure, which can effectively reduce the amount of network parameters and calculations and improve performance compared with existing methods. This is due to the two local and global feature extraction networks proposed above. Since the local network and the global network successfully extract delicate features, the present invention only needs a simple fusion strategy to obtain a well-performing saliency map. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art; obviously, the drawings described below are some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 It is a flowchart of a salient object detection method based on local and global context fusion according to an embodiment of the present invention;

[0057] Figure 2 It is a main network structure and analysis schematic diagram in an embodiment of the present invention;

[0058] Figure 3 is a schematic diagram of a network structure in an embodiment of the present invention;

[0059] Figure 4 It is a schematic diagram of the structure of the fusion network in an example of the present invention;

[0060] Figure 5 FIG. 4 is a schematic diagram comparing the visualization of the saliency map obtained by the method of the present invention and other existing methods in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0062] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0063] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0064] See also Figure 1 A salient object detection method based on local and global context fusion according to an embodiment of the present invention comprises the following steps:

[0065] Get the RGB image to be detected and input it into the pre-trained salient object detection model to obtain the salient object detection result.

[0066] See also Figures 2 to 4 , the salient object detection model specifically includes:

[0067] Feature extraction network, used to obtain multi-scale features of the input RGB image;

[0068] A splicing fusion network is used to fuse the preset first several layers of features in the multi-scale features to obtain low-level detail features; and to fuse the preset last several layers of features in the multi-scale features to obtain high-level semantic features; exemplarily explained, splicing is performed on the channel dimension to achieve fusion;

[0069] A local context extraction network, used to obtain delicate local context features based on the low-level detail features;

[0070] A global context extraction network, used to obtain delicate global context features based on the high-level semantic features;

[0071] The fusion output network is used to fuse the delicate local context features with the global context features to obtain context fusion features; based on the context fusion features, the salient object detection results are output.

[0072] In an exemplary embodiment of the present invention, the local context extraction network specifically includes:

[0073] A low-level feature fusion network is used to input the low-level detail features, perform channel number uniform processing, and output fused low-level detail features; exemplarily explained, the low-level feature fusion network includes a plurality of convolutional layers connected in series;

[0074] The local feature refinement network is used to input the fused low-level detail features, perform refinement processing, and output delicate local context features; exemplarily and optionally, the local feature refinement network is an autoencoder; more specifically, it can be a U-type autoencoder.

[0075] In an exemplary embodiment of the present invention, the global context extraction network specifically includes:

[0076] A high-level feature fusion network is used to input the high-level semantic features, perform channel number uniform processing, and output fused high-level semantic features; exemplarily explained, the layer feature fusion network includes a plurality of convolutional layers connected in series;

[0077] The global feature refinement network is used to input the fused high-level semantic features, perform refinement processing, and output delicate global context features; exemplarily and optionally, the local feature refinement network is a Transformer-based autoencoder.

[0078] In the embodiment of the present invention, the step of acquiring the pre-trained salient object detection model includes:

[0079] Step 1: Perform data enhancement processing on the input image data, including:

[0080] 1) First, compress the size of all images in each training set to 320×320;

[0081] 2) Use a variety of strategies to enhance the compressed image; exemplary optional, random flipping, rotation and boundary cropping, etc. Specifically, the color images in the training set are subjected to random flipping, rotation and boundary cropping operations in sequence, and then the color contrast of the color images is enhanced; for the real saliency images, random flipping, rotation and boundary cropping are performed in sequence, and then random Gaussian noise is added;

[0082] 3) Each original RGB image is combined with the corresponding true saliency image to form a training set.

[0083] Step 2: Build the FCN fully convolutional feature extraction network, including:

[0084] 1) ResNeXt101 is selected as the backbone feature extractor to obtain multi-level features. At the same time, the last pooling layer and fully connected layer in the ResNeXt101 model are removed, and the features obtained by the residual module in the last four layers are selected and represented as Res-2, Res-3, Res-4, and Res-5 respectively;

[0085] 2) The first three layers (Res-2, Res-3, Res-4) are concatenated in the channel dimension to obtain coarse low-level local detail features, and the last two layers (Res-4, Res-5) are concatenated in the same way to obtain coarse high-level semantic features;

[0086] Step 3: Construct local and global feature refinement networks respectively, including:

[0087] 1) First, multiple convolutional layers are used to process the rough bottom layer F l and high-level features F h , so that the number of channels of the two features is 16;

[0088] 2) For the local feature refinement network, a traditional autoencoder is used, which takes the low-level detail features after convolution as input and finally outputs the delicate local features F local ;

[0089] 3) For the global feature refinement network, since high-level semantic features contain rich global information, a deeper network with a larger receptive field is needed to extract it. According to research, Transformer performs well in natural language processing and image segmentation. It encodes features by converting images into patch sequences to further enhance the network's ability to model global context. Compared with convolutional neural networks, it can further expand the receptive field. Therefore, Transformer is embedded in the autoencoder as the backbone network for extracting global features, and finally the global feature F that is helpful for subsequent positioning is extracted. global .

[0090] Specifically, the embodiment of the present invention first refines the fused high-level semantic features again, that is, the features are continuously downsampled and encoded (preset continuous convolution operations and pooling operations) to obtain semantic features encoded by the convolutional neural network, and then the semantic features encoded by the convolutional neural network are re-encoded using the Transformer encoder, wherein the main operation before inputting into the Transformer encoder is to first flatten the features (assuming the size is c*w*h) into w*h patches in the spatial domain, and the size of the generated patch features is c*wh*1, and then flip it to wh*c*1, and then add the randomly generated trainable position encoding of the same size to the feature to obtain the implicit features input into the Transformer encoder, and at the same time, since saliency detection is category-agnostic, the classification token set in the traditional Vision Transformer is cancelled. For the Transformer encoder, its self-attention mechanism is used to enhance the accuracy of the positioning information of salient objects. It mainly uses 6 layers of encoding blocks, each layer is composed of Layer Norm, Multi-Head Attention (MHA), Dropout, Multi-layer Perceptron MLP (Fully Connected + GELU Activation Function + Dropout), to obtain the features after Transformer encoding, and finally use the decoder based on convolutional neural network to decode the features, which is consistent with the decoding of the local feature refinement network, and uses the upsampling method of short connection to obtain the global context features.

[0091] Step 4: Fusion of delicate local and global features, including:

[0092] 1) Use F h They are used to strengthen local and global features respectively, that is, they are input into a 1x1 convolutional layer after splicing in the channel dimension;

[0093] 2) The enhanced features are fused using an extremely simple strategy, that is, they are concatenated in the channel dimension and input into a 1x1 convolutional layer to obtain a fused saliency map.

[0094] Specifically, the embodiment of the present invention first inputs the global context extraction network and the local and global context features obtained by the global context extraction network into a convolution layer with a convolution kernel of 1, transforms the number of channels to be consistent with the fused high-level semantic features output by the high-level feature fusion network, and then strengthens the local and global context respectively, that is, after splicing in the channel dimension, inputs into a ReLU layer, and then splices the strengthened local and global context features in the channel dimension and inputs into a ReLU layer to obtain context fused features, and finally inputs the fused features into a convolution layer with a convolution kernel of 1 and performs a Softmax operation on them to obtain a saliency map with a channel number of 1.

[0095] Step 5: Train the network and include:

[0096] 1) FCN fully convolutional network: The parameters of the ResNeXt101 model trained on ImageNet are used for optimization, and the saliency annotation map with manually labeled salient objects in the image is used as the supervision information for training;

[0097] 2) Each original RGB image in the training set and its corresponding true value image are used as the original input images of the input layer and input into the convolutional neural network for training to obtain the predicted saliency image corresponding to each original RGB image in the training set; the loss function value between the predicted saliency image corresponding to each original RGB image in the training set and the corresponding true saliency image is calculated. The loss function value is obtained by summing three different loss functions. The three loss functions are BCE loss, SSIM loss, and IOU loss, respectively. The expressions are:

[0098] L bce =-∑ (i,j) [G i,j log P i,j +(1-G i,j )log(1-P i,j )];

[0099]

[0100]

[0101] Wherein, P is the prediction result of the salient object detection method of the present invention, G is the real salient image, μ x ,μ y and σ x ,σ y represent the mean and variance on P and G respectively.

[0102] 3) Optimize the pre-built convolutional neural network through the stochastic gradient descent algorithm.

[0103] The embodiment of the present invention preferably also includes step 6: testing using a test set, including: after the training is terminated, using the trained network to detect the test sample, performing a binary classification of significant or non-significant on each pixel point, and obtaining an end-to-end prediction as a global significance detection result.

[0104] In the embodiments of the present invention, a salient object detection method based on local and global context fusion is proposed to effectively extract local and global context information, and a simple and effective fusion strategy is designed to fully utilize the advantages of local and context information to improve the efficiency and accuracy of saliency detection in complex scenes.

[0105] Table 1(a) and Table 1(b) are the experimental results of the salient object detection method. Table 1(a) is the experimental result of the method under the DUTS-TE, DUTOMRON, and ECSSD datasets, and Table 1(b) is the experimental result of the method under the HKU-IS, PASCALS, and SOD datasets. Here, the present invention only selects four evaluation indicators, namely, the maximum Average F-measure (F β )、E-measure(E ε ) and mean absolute error (MAE). Here, except for MAE, the larger the value, the better the effect. From Table 1, we can see that our method has achieved good results on most test sets.

[0106] Figure 2 It is a schematic diagram of the main structural framework and analysis of the method of the embodiment of the present invention. From the left part of the figure, the local and global contexts are obtained by the local and global branches, namely the local context extraction network and the global context extraction network, respectively. The fusion branch makes good use of these contexts to generate the final fusion features for salient target detection. The present invention on the right side of the figure visualizes that different branch networks generate corresponding feature maps for different input images, where the preset area represents the attention of each branch network. It can be seen that no matter whether the salient object is small or large, the three branch models can help each other find the exact location of the salient object. Taking the dog as an example, the local context is the outline of the dog and the billboard, and the global context is the location of the dog; however, the boundary between the dog and the billboard is indistinguishable in the local branch; in addition, details such as the dog's legs are blurred in the global branch. However, the fusion features of the present invention alleviate these problems.

[0107] Table 1 (a) Experimental results of the method on DUTS-TE, DUTOMRON, and ECSSD datasets

[0108]

[0109] Table 1(b) Experimental results of this method on HKU-IS, PASCALS, and SOD datasets

[0110]

[0111] The following are device embodiments of the present invention, which can be used to perform method embodiments of the present invention. For details not disclosed in the device embodiments, please refer to the method embodiments of the present invention.

[0112] An embodiment of the present invention provides a salient object detection system based on local and global context fusion, comprising:

[0113] The detection module is used to obtain the RGB image to be detected by the salient object and input it into the pre-trained salient object detection model to obtain the salient object detection result;

[0114] The salient object detection model includes:

[0115] Feature extraction network, used to obtain multi-scale features of the input RGB image;

[0116] A splicing fusion network is used to fuse the preset first several layers of features in the multi-scale features to obtain low-level detail features; and is used to fuse the preset last several layers of features in the multi-scale features to obtain high-level semantic features;

[0117] A local context extraction network, used to input the low-level detail features and output delicate local context features;

[0118] A global context extraction network, used to input the high-level semantic features and output delicate global context features;

[0119] The fusion output network is used to input and fuse the delicate local context features and the delicate global context features to obtain context fusion features; based on the context fusion features, the salient target detection results are output.

[0120] In summary, the embodiment of the present invention discloses a method for detecting salient objects based on the fusion of local and global contexts, which belongs to the field of computer vision. The present invention identifies the content that the human eye pays most attention to for a given image. The image contains a lot of irrelevant background, which will affect the segmentation of salient objects. Existing work mainly focuses on how to aggregate multi-level features from different layers of a fully convolutional network, which is still an unresolved issue in deep learning tasks. The present invention segments detailed structures in a local view and simultaneously locates salient objects in a global view, uses traditional autoencoders to extract local edge features of objects, introduces a Transformer encoder and combines a U-shaped structure to construct a global branch network to extract global context, and obtains the location information of salient objects from the perspective of taking labeled feature patches as input sequences. Finally, the advantages of local and global information are taken separately, and a simple fusion strategy is adopted to achieve salient object detection and improve the accuracy of segmenting salient objects.

[0121] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0122] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0123] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0124] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A salient object detection method based on local and global context fusion, characterized in that: The following steps are involved: Obtain the RGB image to be detected for salient objects and input it into the pre-trained salient object detection model to obtain the salient object detection result; in, The salient object detection model includes: Feature extraction network, used to obtain multi-scale features of the input RGB image; A splicing fusion network is used to fuse the preset first several layers of features in the multi-scale features to obtain low-level detail features; and is used to fuse the preset last several layers of features in the multi-scale features to obtain high-level semantic features; A local context extraction network, used to input the low-level detail features and output delicate local context features; A global context extraction network, used to input the high-level semantic features and output delicate global context features; A fusion output network is used to input and fuse the delicate local context features and the delicate global context features to obtain context fusion features; based on the context fusion features, a salient object detection result is output; The local context extraction network comprises: A low-level feature fusion network is used to input the low-level detail features, perform channel number uniform processing, and output fused low-level detail features; A local feature refinement network is used to input the fused low-level detail features, perform refinement processing, and output delicate local context features; wherein the local feature refinement network is a U-type autoencoder; in the U-type autoencoder, a short connection operation is added to each layer of upsampling decoding, and the features of the downsampling encoding are combined with the features of the upsampling decoding to compensate for the limited context aggregation caused by the downsampling operation on the features; The global context extraction network specifically includes: A high-level feature fusion network is used to input the high-level semantic features, perform uniform channel number processing, and output fused high-level semantic features; A global feature refinement network, which is used to input the fused high-level semantic features, perform refinement processing, and output delicate global context features; wherein the global feature refinement network includes: continuous convolution operation and pooling operation, which is used to input the fused high-level semantic features, perform continuous downsampling encoding to obtain semantic features encoded by convolutional neural network; The Transformer encoder is used to re-encode the semantic features encoded by the convolutional neural network and output the features encoded by the Transformer. The decoder based on the convolutional neural network is used to input the features encoded by the Transformer and obtain the global context features by upsampling with short connections. Each layer of the encoding block of the Transformer encoder is composed of Layer Norm, Multi-Head Attention, Dropout and a multi-layer perceptron.

2. The method for detecting salient objects based on local and global context fusion according to claim 1, characterized in that: The steps of obtaining the pre-trained salient object detection model include: Obtaining a training sample set; wherein each sample in the training sample set includes an RGB sample image and a corresponding true saliency image; The salient object detection model is trained based on the training sample set, and after the loss function value reaches a preset convergence condition, the pre-trained salient object detection model is obtained; Among them, the loss function value is the loss function value between the predicted saliency image of the RGB sample picture of each sample and the true saliency image corresponding to the RGB sample picture; the loss function value is obtained by summing three different loss functions, and the three loss functions are BCE loss, SSIM loss and IOU loss respectively.

3. The method for detecting salient objects based on local and global context fusion according to claim 2, characterized in that: When the salient object detection model is trained based on the training sample set, a stochastic gradient descent algorithm is used to optimize model parameters.

4. The method for detecting salient objects based on local and global context fusion according to claim 1, characterized in that: In the fusion output network, the delicate local context features and the delicate global context features are input and fused to obtain context fusion features; Based on the context fusion feature, the step of outputting the salient object detection result specifically includes: The fine local context features and fine global context features are input into a convolution layer with a convolution kernel of 1, and the number of channels is transformed to be consistent with the fused high-level semantic features; after splicing in the channel dimension, they are input into a ReLU layer to obtain enhanced local and global context features; The enhanced local and global context features are concatenated in the channel dimension and input into a ReLU layer to obtain the context fusion feature; The context fusion feature is input into a convolution layer with a convolution kernel of 1 and a Softmax operation is performed on it to obtain a saliency map with a channel number of 1.

5. The method for salient object detection based on local and global context fusion according to claim 1, characterized in that: In the splicing fusion network, the preset first several layers of features in the multi-scale features are fused to obtain low-level detail features; The step of fusing the preset last several layers of features in the multi-scale features to obtain high-level semantic features specifically includes: The preset first several layers of features in the multi-scale features are spliced ​​and fused in the channel dimension to obtain low-level detail features; The preset last few layers of features in the multi-scale features are spliced ​​and fused in the channel dimension to obtain high-level semantic features.

6. A salient object detection system based on local and global context fusion, characterized in that: include: The detection module is used to obtain the RGB image to be detected by the salient object and input it into the pre-trained salient object detection model to obtain the salient object detection result; in, The salient object detection model includes: Feature extraction network, used to obtain multi-scale features of the input RGB image; A splicing fusion network is used to fuse the preset first several layers of features in the multi-scale features to obtain low-level detail features; and is used to fuse the preset last several layers of features in the multi-scale features to obtain high-level semantic features; A local context extraction network, used to input the low-level detail features and output delicate local context features; A global context extraction network, used to input the high-level semantic features and output delicate global context features; A fusion output network is used to input and fuse the delicate local context features and the delicate global context features to obtain context fusion features; based on the context fusion features, a salient object detection result is output; The local context extraction network comprises: A low-level feature fusion network is used to input the low-level detail features, perform channel number uniform processing, and output fused low-level detail features; A local feature refinement network is used to input the fused low-level detail features, perform refinement processing, and output delicate local context features; wherein the local feature refinement network is a U-type autoencoder; in the U-type autoencoder, a short connection operation is added to each layer of upsampling decoding, and the features of the downsampling encoding are combined with the features of the upsampling decoding to compensate for the limited context aggregation caused by the downsampling operation on the features; The global context extraction network specifically includes: A high-level feature fusion network is used to input the high-level semantic features, perform uniform channel number processing, and output fused high-level semantic features; A global feature refinement network is used to input the fused high-level semantic features, perform refinement processing, and output delicate global context features; wherein the global feature refinement network includes: continuous convolution operations and pooling operations, which are used to input the fused high-level semantic features and perform continuous downsampling encoding to obtain semantic features encoded by a convolutional neural network; a Transformer encoder is used to re-encode the semantic features encoded by the convolutional neural network and output Transformer encoded features; a decoder based on a convolutional neural network is used to input the Transformer encoded features and obtain global context features by using a short-connected upsampling method; wherein each layer of the encoding block of the Transformer encoder is composed of Layer Norm, Multi-Head Attention, Dropout, and a multi-layer perceptron.