A gaze-guided salient object detection method and system

Abstract

The application discloses a gaze-guided salient object detection method and system, and relates to the field of image processing. The method comprises the following steps: acquiring an original image and a gaze saliency map; constructing a salient object detection network model comprising an encoder, a convolution module, a diversified feature enhancement module, a gaze guidance module, an integrity enhancement module, a double-flow capsule module and a salient detector; inputting the original image and the gaze saliency map into the salient object detection network model to obtain a salient object detection result. The application can effectively suppress the interference of non-salient objects, and can avoid problems such as integrity overflow and inaccurate salient object positioning in the detection process, thereby improving the salient object detection positioning capability.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to a salient object detection method and system based on gaze guidance. BACKGROUND

[0002] Salient object detection aims to capture the most salient objects in an image scene, which is an important preprocessing step in computer vision tasks, usually containing two visual processing stages, first locating the most salient objects in the image, and second segmenting the region where the most salient objects are located. In order to accurately locate the salient region, researchers try to use eye gaze to detect objects in order to build a salient object detection model that is more consistent with human visual mechanisms. Human visual attention mechanism can help the human eye to quickly capture the region of interest, Fixed Prediction (FP) aims to predict the region of interest during free viewing, Salient Object Detection (SOD) aims to distinguish the most visually salient region, and the salient object detection task tends to identify and segment the salient object. Related works have confirmed that there is a strong correlation between gaze and salient object, however, known methods pay more attention to the interaction process between FP and SOD, such as the network proposed in the prior art combines FP and SOD datasets for training, and improves detection performance through training interaction process. Another way is to generate a saliency map in a top-down manner under the guidance of a gaze map, such as ASNet. These works try to treat FP and SOD as two task branches, and establish an interaction process between the information of the two.

[0003] The existing salient object detection method based on gaze guidance based on deep learning has room for improvement in suppressing the integrity overflow problem and the form of fusing the gaze saliency map. Based on the above description, the salient object detection model in the existing method is prone to problems such as detection integrity overflow and inaccurate salient object positioning when facing complex scene images. Therefore, it is necessary to explore how to avoid the integrity overflow problem in the detection process and how to use the gaze saliency map to improve the detection positioning ability of the model. SUMMARY

[0004] The purpose of the present application is to provide a salient object detection method and system based on gaze guidance, which can effectively suppress the interference of non-salient objects, and at the same time, avoid problems such as detection integrity overflow and inaccurate salient object positioning, thereby improving the salient object detection positioning ability.

[0005] To achieve the above purpose, the present application provides the following solutions:

[0006] In a first aspect, the present application provides a salient object detection method based on gaze guidance, comprising:

[0007] obtaining an original image and a gaze saliency map;

[0008] constructing a salient object detection network model; the constructed salient object detection network model comprises an encoder, a convolution module, a diversified feature enhancement module, a gaze guiding module, an integrity enhancement module, a dual-flow capsule module and a salient detector; the encoder is connected with the diversified feature enhancement module; the diversified feature enhancement module and the convolution module are both connected with the gaze guiding module; the gaze guiding module is connected with the integrity enhancement module; the integrity enhancement module, the dual-flow capsule module and the salient detector are sequentially connected;

[0009] inputting the original image and the gaze saliency map into the salient object detection network model to obtain a salient object detection result; wherein the encoder is used to extract features of the original image; the convolution module is used to perform convolution and blur processing on the original image and the gaze saliency map, and simultaneously perform fusion processing to obtain preliminary fusion features; the diversified feature enhancement module is used to enhance the diversity of the features extracted by the encoder to obtain convolution enhanced features; the gaze guiding module is used to obtain gaze fusion features based on the preliminary fusion features, and is used to realize alignment and enhancement of the convolution enhanced features and the gaze fusion features to obtain fusion features; the integrity enhancement module is used to learn and enhance the integrity of the fusion features to obtain integrity enhanced features; the dual-flow capsule module is used to generate a plurality of capsules based on the integrity enhanced features; each capsule comprises a pose matrix and an activation value; the pose matrix is used to represent the relationship between the integrity enhanced features; and the salient detector is used to perform fusion decoding on the plurality of capsules to obtain the salient object detection result.

[0010] Optionally, constructing the salient object detection network model comprises:

[0011] constructing a data set and dividing the data set into a training set and a test set; the data set comprises historical images, gaze saliency maps and object saliency maps; the object saliency maps are obtained by capturing gaze data of the historical images using an eye tracker;

[0012] constructing an initial model;

[0013] training and testing the initial model using the training set and the test set until the difference between the object saliency map and the predicted object saliency map meets a set value, and then taking the trained initial model as the salient object detection network model.

[0014] Optionally, the diversified feature enhancement module is composed of an asymmetric convolution, a hollow convolution and a convolution aggregation.

[0015] Optionally, the gaze guiding module comprises a gaze fusion unit and a gaze guiding cross-attention unit; the gaze fusion unit is configured to obtain gaze fusion features based on the preliminary fusion features; and the gaze guiding cross-attention unit is configured to align and enhance the convolution enhanced features and the gaze fusion features to obtain fusion features.

[0016] Optionally, the integrity enhancement module comprises a channel integrity enhancement sub-module and a spatial integrity enhancement sub-module; the channel integrity enhancement sub-module is configured to learn the integrity of the fusion features; and the spatial integrity enhancement sub-module is configured to enhance the integrity of the fusion features.

[0017] Optionally, the construction process of the dual-flow capsule module comprises capsule construction, capsule convolution and capsule connection.

[0018] Optionally, the process of the capsule construction comprises:

[0019] The integrity enhanced features are converted into first fusion features, and the first fusion features are converted into channel feature maps through a convolution layer;

[0020] The channel feature maps are reshaped into an initial pose matrix and initial activation values;

[0021] The initial pose matrix and the initial activation values are connected to obtain a plurality of initial capsules;

[0022] The process of the capsule convolution comprises:

[0023] The plurality of initial capsules are divided into two groups, and each group is passed to two capsule convolutions to explore the relationship between the object parts and the whole to obtain a pose matrix and activation values of each initial capsule;

[0024] The pose matrix of each initial capsule is converted into a pose matrix, and a voting matrix is obtained based on the pose matrix using a transformation matrix;

[0025] A routing algorithm is used to generate a final pose matrix and final activation values based on the voting matrix and the activation values of each initial capsule to obtain a capsule convolution result;

[0026] The process of the capsule connection comprises:

[0027] The two capsule convolution results are connected and merged into a first capsule, and are reshaped into a second capsule;

[0028] A convolution operation is performed on the second capsule to obtain a rich capsule, and the rich capsule is reshaped into a third capsule to obtain an intermediate pose matrix and intermediate activation values;

[0029] The routing algorithm is used to establish the initial connection between capsules based on the intermediate pose matrix and the intermediate activation value, and finally obtain the pose matrix and the activation value of each capsule.

[0030] In a second aspect, the application provides a gaze-guided salient object detection system, comprising:

[0031] An image acquisition module is configured to acquire an original image and a gaze saliency map.

[0032] A model construction module is configured to construct a salient object detection network model, wherein the constructed salient object detection network model comprises an encoder, a convolution module, a diversified feature enhancement module, a gaze guidance module, an integrity enhancement module, a double-flow capsule module, and a salient detector; the encoder is connected to the diversified feature enhancement module; the diversified feature enhancement module and the convolution module are both connected to the gaze guidance module; the gaze guidance module is connected to the integrity enhancement module; the integrity enhancement module, the double-flow capsule module, and the salient detector are sequentially connected.

[0033] A result detection module is configured to input the original image and the gaze saliency map into the salient object detection network model to obtain a salient object detection result; wherein the encoder is configured to extract features of the original image; the convolution module is configured to perform convolution and blur processing on the original image and the gaze saliency map, and simultaneously perform fusion processing to obtain preliminary fusion features; the diversified feature enhancement module is configured to enhance the diversity of the features extracted by the encoder to obtain convolution-enhanced features; the gaze guidance module is configured to obtain gaze fusion features based on the preliminary fusion features, and to realize alignment and enhancement of the convolution-enhanced features and the gaze fusion features to obtain fusion features; the integrity enhancement module is configured to learn and enhance the integrity of the fusion features to obtain integrity-enhanced features; the double-flow capsule module is configured to generate a plurality of capsules based on the integrity-enhanced features; each capsule comprises a pose matrix and an activation value; the pose matrix is configured to represent the relationship between the integrity-enhanced features; and the salient detector is configured to fuse and decode the plurality of capsules to obtain the salient object detection result.

[0034] According to the specific embodiments provided by the application, the application has the following technical effects:

[0035] This application provides a method and system for salient object detection based on gaze guidance. By constructing a salient object detection network model, the system obtains salient object detection results based on the input original image and gaze saliency map, thereby improving the detection and positioning capabilities of salient objects. Furthermore, by combining the gaze saliency map with salient object detection, interference from non-salient objects can be effectively suppressed. Furthermore, by providing an encoder, convolution module, diversified feature enhancement module, gaze guidance module, integrity enhancement module, dual-stream capsule module, and salient detector within the salient object detection network model, problems such as integrity overflow and inaccurate salient object positioning can be effectively avoided during the detection process. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 A flowchart of a salient object detection method based on gaze guidance provided in one embodiment of the present application;

[0038] Figure 2 A schematic diagram of an implementation architecture of a salient object detection method based on gaze guidance provided in one embodiment of the present application;

[0039] Figure 3 A schematic diagram of the network model structure for salient object detection provided in one embodiment of the present application;

[0040] Figure 4 Schematic diagram of original images of different scenes provided in an embodiment of the present application;

[0041] Figure 5 An embodiment of the present application provides Figure 4 The corresponding gaze saliency map;

[0042] Figure 6 An embodiment of the present application provides Figure 4 The corresponding real object saliency map;

[0043] Figure 7 Another embodiment of the present application provides Figure 4 Schematic diagram of the image detection result comparison array obtained by different salient object detection methods;

[0044] Figure 8 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0046] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0047] In an exemplary embodiment, a fixation-guided salient object detection method is provided, which is executed by a computer device, specifically, can be executed by a terminal or a server, or can be executed by the terminal and the server together. In the embodiments of the present application, the fixation-guided salient object detection method is taken as an example for illustration. Figure 1 As shown in the figure, the fixation-guided salient object detection method includes:

[0048] Step 100: obtaining an original image and a fixation saliency map.

[0049] Step 101: constructing a salient object detection network model. As shown in the figure, the constructed salient object detection network model includes an encoder, a convolution module, a diversity feature enhancement module (DFEM), a fixation guidance module (FGM), an integrity enhancement module (IEM), a two-stream capsule module (TSCM) and a salient detector. The encoder is connected with the diversity feature enhancement module. The diversity feature enhancement module and the convolution module are both connected with the fixation guidance module. The fixation guidance module is connected with the integrity enhancement module. The integrity enhancement module, the two-stream capsule module and the salient detector are connected in sequence. Figure 3

[0050] ​Step 102: input the original image and the gaze saliency map into the salient object detection network model to obtain a salient object detection result. The encoder is used to extract features of the original image. The convolution module is used to perform convolutional blur processing on the original image and the gaze saliency map, and simultaneously perform fusion processing to obtain preliminary fusion features. The diversified feature enhancement module is used to enhance the diversity of the features extracted by the encoder to obtain convolutional enhanced features. The gaze guiding module is used to obtain gaze fusion features based on the preliminary fusion features, and is used to realize alignment and enhancement of the convolutional enhanced features and the gaze fusion features to obtain fusion features. The integrity enhancement module is used to learn and enhance the integrity of the fusion features to obtain integrity enhanced features. The dual-flow capsule module is used to generate a plurality of capsules based on the integrity enhanced features. Each capsule includes a pose matrix and an activation value. The pose matrix is used to represent the relationship between the integrity enhanced features. The salient detector is used to fuse and decode the plurality of capsules to obtain the salient object detection result.

[0051] By implementing the above steps 100 to 102, the present application proposes a salient object detection method based on gaze guidance from the perspective of fusing the gaze saliency map. The gaze saliency map is directly input into the gaze guiding cross-attention module as prior information to realize the interaction between image features and gaze salient region information. Meanwhile, the integrity enhancement module and the dual-flow capsule module are used to further mine and refine the saliency map. Compared with existing methods, the present application not only performs better in various evaluation indicators, but also effectively suppresses the interference of non-salient objects in visual effects, and has significant improvement in improving detection positioning ability and solving the problem of integrity overflow.

[0052] In another exemplary embodiment of the present application, the encoder can use ResNet-50 as a backbone network for extracting multi-level features, and the decoder generates more accurate salient object detection results by aggregating multi-level features and fusing the gaze saliency map. Based on this, the input of the salient object detection network model provided in the above application is an original image and a gaze saliency map The output is an object saliency map Wherein C is the number of image channels, H and W are the height and width of the image, respectively.

[0053] In another exemplary embodiment of the present application, the implementation process of step 101 of the above application is described by taking salient object detection for RGB images as an example. The construction process includes:

[0054] (1) Construct a dataset and divide it into a training set and a test set. The dataset includes historical images, gaze saliency maps, and object saliency maps. This dataset can also be a public dataset, such as COCOcapsal, DUT-OMRON, etc., or the gaze data of the original image can be captured by an eye tracker to obtain the gaze saliency map, and the salient objects in the original image can be manually annotated to obtain the object saliency map.

[0055] (2) Construct an initial model.

[0056] (3) The initial model is trained and tested using the training set and the test set until the difference between the object saliency map and the predicted object saliency map meets the set value. The trained initial model is used as the salient object detection network model.

[0057] At the beginning of training, images can undergo a series of transformations, such as flipping, cropping, rotation, and normalization, to generate more training samples and improve the generalization ability and performance of the model.

[0058] In another exemplary embodiment of the present application, a rich variety of convolutional layers can help the network learn to capture spatial features under different receptive fields and help the network learn to capture objects of different sizes and shapes. In order to further enhance the diversity of features extracted by the encoder, this embodiment uses asymmetric convolution, hole convolution and original convolution to form a diversified feature enhancement module to enrich the receptive field and enhance the representation ability of features. For example:

[0059] Assume that the output features of the last four layers of the ResNet-50 encoder are χ represents the output feature set of the encoder, They represent the output features of the last four layers of the encoder respectively. After the output features of the encoder are processed by the diversified feature enhancement module, the feature representation capability can be enhanced. Among them, the aggregation process of the diversified feature enhancement module can be expressed as:

[0060]

[0061] Where C ori (·), C Asy (·) and C Atr (·) represents original convolution, hole convolution and asymmetric convolution respectively, Cat[·] and Conv[·] are channel connection and convolution operations, It is the convolution enhanced feature. is the output feature of the i-th layer after the encoder.

[0062] Furthermore, in this embodiment, the present application can use a dilation rate r = 2 for the dilation convolution C Asy (·) Helps the network extract features under different receptive fields. Asymmetric convolution CAtr (·) contains three different shapes of cores, for example, 3x3 square core K 3 ×3 , 1x3 horizontal core K 1×3 and 3x1 vertical core K 3×1 . Based on this, the operation process of asymmetric convolution C Atr (·) can be represented as:

[0063]

[0064] In the formula, X represents the convolution object, * and respectively represent the 2D convolution operator and the element-level addition operation.

[0065] In another exemplary embodiment of the present application, in order to fully fuse the fixation saliency map and the original image, the saliency around the fixation area is highlighted. In this embodiment, as shown in Figure 3 , the fixation guidance module can include a fixation fusion unit and a fixation guidance cross-attention unit. The fixation fusion unit is used to obtain fixation fusion features based on preliminary fusion features. The fixation guidance cross-attention unit is used to realize the alignment and enhancement of the convolution enhanced features and the fixation fusion features, and obtain fusion features.

[0066] (1) Fixation Fusion Unit (FFU):

[0067] In order to fully fuse the fixation saliency map and the original image, the present application constructs a simple feature extraction unit (i.e. FFU) to extract multi-scale fusion features. Multi-scale fusion can help the network to learn fixation area features from different scales. For example, convolution blur is performed on the fixation saliency map and the original image respectively to adaptively adjust the image information, and the fusion features are obtained by initially using simple element-level multiplication. fusion The layer-by-layer convolution operation scales the fusion features x fusion to different sizes. Based on this, the multi-scale feature fusion process can be represented as:

[0068]

[0069] In the formula, BN(·) is normalization processing, is the fixation fusion feature.

[0070] (2) Fixation Guidance Cross-Attention Unit (FGCAU):

[0071] FGCAU is the key component of further fusion of gaze saliency map and original image features, aiming to highlight the saliency around the gaze region. Convolution enhanced features and gaze fusion features respectively as Q (query) and K / V (key, value) inputs, the cross-attention mechanism is used to realize the information interaction and feature alignment of the two, in order to focus on learning the feature information of the gaze region. At the same time, this fusion method makes the network more focused on learning the relationship between the gaze salient region and other regions, helping to locate the salient object.

[0072] Specifically, convolution enhanced features and gaze fusion features After matrix transformation, Q and K / V are obtained, which are normalized and passed to multi-head cross attention (MHCA) to realize the alignment and enhancement of gaze fusion features and original features. Add a residual connection between the original features and MHCA, and perform local enhancement through forward propagation operation (FFN), and finally add a residual connection to output the final result (i.e. get the fusion features). The process of gaze-guided cross-attention unit enhanced features can be represented as:

[0073]

[0074] In the formula, denotes the normalization operation on *, and MHCA(·) and FFN(·) are multi-head cross attention and forward propagation process respectively. fgcau denotes the final output of FGCAU, i.e. fusion features. The specific implementation process of MHCA(·) is:

[0075]

[0076] In the formula, Softmax[·] represents the activation function, which maps the calculated weight to the interval (0, 1), and h is the number of attention heads, set to 2. and respectively denote the query, key and value of a single head. d represents the dimension of feature stretching, d = hw. q ,W k and W v are the projection matrices corresponding to Q, K and V respectively. A h denotes the attention value, and Attn() denotes the attention calculation function.

[0077] In another exemplary embodiment of the present application, in order to learn the integrity of features. The integrity enhancement module adopted in the present application is divided into two sub-modules, respectively, Channel Integrity Enhancement Module (CIEM) and Spatial Integrity Enhancement Module (SIEM), and the two sub-modules fuse channel and spatial features through element-level addition.

[0078] wherein (1) Channel Integrity Enhancement Module (CIEM):

[0079] Considering the multi-scale information from the adjacent three feature levels, further highlighting the consistency of the significant region of the channel feature. Let the fusion feature output by FGCAU be Two adjacent features are If there is no adjacent feature, replace it with itself. Two adjacent features are adjusted to the same resolution as the fusion feature through downsampling and upsampling operations, and the multi-scale feature fusion can be expressed as:

[0080]

[0081] In the formula, indicates the multi-scale fusion feature, indicates the i-scale fusion feature.

[0082] The Channel Integrity Enhancement Module calculates the L2 norm of each channel of the multi-scale fusion feature , wherein the optimization process of the parameter can be expressed as:

[0083]

[0084] In the formula, ReLU[·] is an activation function, LN[·] is a normalization layer, L2(·) represents the L2 norm, indicates the channel integrity enhancement feature

[0085] (2) Spatial Integrity Enhancement Module (SIEM):

[0086] SIEM aims to explore the significant region globally. The local is a method to supplement the global information. The traditional spatial attention is to perform global max-pooling and average-pooling on all channel features, which may lead to insufficient spatial attention map. Spatial group-wise enhance (SGE) adopts a grouping attention strategy to group and enhance the channels, but it does not consider the consistency between different channels. Weighted spatial attention (WSA) generates a local attention map for each channel sub-feature and generates the final spatial attention map by weighted fusion, which ignores the consistency between different feature levels.

[0087] ShuffleNet and shuffle attention disrupt the information communication between channels. The present application utilizes the idea to fuse the multi-scale fusion features into four channel sub-features by channel shuffling and feature splitting. In order to facilitate the operation, the multi-scale fusion features are represented as:

[0088]

[0089] In the formula, are the results obtained after channel shuffling and feature splitting.

[0090] The channel shuffling operation is performed on the feature channel dimension direction of the multi-scale fusion features , and the following can be obtained:

[0091]

[0092] The multi-scale fusion features are evenly divided into four groups of sub-features in the feature channel dimension, which are and , respectively. Each channel sub-feature can be represented as:

[0093]

[0094] Further, the spatial attention map corresponding to each channel sub-feature is calculated, and the specific process can be represented as:

[0095]

[0096] In the formula, F1(·) and F2(·) represent average pooling and max-pooling, respectively, is the initial spatial attention map, g(·) represents the sigmoid activation function, and s m∈(0, 1) 1×h×w is the spatial attention map of the m-th channel sub-feature.

[0097] Each channel sub-feature is assigned a learnable weight w ∈ [0, 1], and the fusion weighting can be obtained as:

[0098]

[0099] where s ∈ (0, 1) 1×h×w is the spatial attention map of the m-th channel sub-feature, and the weight w m is initialized to 0.25 and satisfies Through training, it gradually converges to the appropriate weight.

[0100] The spatial attention map s and the multi-scale fusion feature Through fusion, the consistency between adjacent feature levels is enhanced, and the final enhancement process can be represented as:

[0101]

[0102] Finally, the spatial integrity enhancement feature and the channel integrity enhancement feature are element-level fused to obtain the integrity enhancement feature to further explore the consistency of different channel features and spatial features.

[0103] In another exemplary embodiment of the present application, in order to further explore integrity learning and explore the part-whole relationship of objects in the image, the construction process of the two-stream capsule module (TSCM) includes capsule construction, capsule convolution, and capsule connection, wherein:

[0104] (1) Capsule Construction:

[0105] The present application only uses the first three integrity enhancement features and Then, the integrity enhancement features and are respectively upsampled and downsampled to adjust to the same dimension as the integrity enhancement feature , denoted as 22 × 22. The above three integrity enhancement features are connected using the same fusion method as the channel integrity enhancement sub-module to obtain the first fusion feature The first fusion feature x iem-f is converted into 16 capsules through capsule construction. Each capsule contains a pose matrix (4 × 4) and an activation value, the pose matrix reflects the characteristics of the object part and the whole, and the activation value represents the probability of the existence of the object.

[0106] Specifically, first, the first fusion feature x iem-f is converted into a channel feature map of 256x20x20, and then reshaped into an initial pose matrix (16-capsule vectorized pose matrix). In addition, the first fusion feature x iem-f is also converted into a channel feature map of 16x20x20 through a convolutional layer (kernel size 5x5), and then reshaped into initial activation values (16-capsule activation values). Finally, the initial pose matrix and the initial activation values are connected to obtain 16 capsules of 16x17x20x20.

[0107] (2) Capsule convolution:

[0108] For example, the 16 capsules obtained by capsule construction are evenly divided into two groups, each group containing 8 capsules of 8x17x20x20, and each group is passed to two capsule convolutions to explore the part-whole relationship of the object, and each capsule convolution is composed of 4 capsules with the same structure. The low-level capsules (parts) are assigned to high-level capsules (objects) through a routing algorithm (EM Routing). For example, the specific process of a single capsule convolution is as follows:

[0109] Step 1: enrich capsules: first, reshape each group of input capsules into capsules of 136x20x20, then perform a convolution operation with a stride of 2 on each group of capsules of 136x20x20 using a convolutional layer (kernel size 3x3) to obtain more enriched capsules of 9x136x10x10, and reshape them into capsules of 72x17x100. The final pose matrix and activation values are and

[0110] Step 2: calculate votes: first, convert the initial pose matrix of each capsule into a 4x4 pose matrix M, and reshape the pose matrix of low-level capsule i to The low-level capsule i corresponds to a transformation matrix for each high-level capsule j to obtain a voting matrix The voting matrix V ij and the activation value a i The pose matrix and activation value of the high-level capsule are generated through the routing algorithm and

[0111]

[0112] (3) Capsule connection:

[0113] The pose matrix and activation value of two capsule convolutions are connected respectively and then transmitted to the capsule connection (also called capsule convolution, 4 capsules), which aims to explore the object part-whole relationship between the two flow capsules. The two capsule convolution connections are merged into the first capsule 8x17x100, and are reshaped into the second capsule 136x10x10, followed by a convolution operation with a kernel size of 3x3 and a stride of 1, to obtain a rich capsule 9x136x100, which is reshaped into a third capsule 72x17x100, to obtain an intermediate pose matrix and intermediate activation value The next step is to perform the operation of calculating the voting matrix in the capsule convolution, and the routing algorithm is used to establish the connection between the low-level capsule and the high-level capsule, and finally the pose matrix of the capsule is obtained and activation value

[0114]

[0115] In another exemplary embodiment of the present application, in order to generate a more fine-grained saliency map, the present application adopts an effective cascaded partial decoder framework (CPD) as a key component of the saliency detector. Since the low-level feature layer retains more detailed information, the high-level feature layer pays more attention to deep semantic information, and the pose matrix P focuses on exploring the object part-whole relationship, which has similar semantic characteristics as the high-level feature layer. Therefore, first, the pose matrix and the high-level feature layer are fused in turn, and the fusion process can be represented as:

[0116]

[0117] Secondly, the above features are fused step by step The process is represented as:

[0118]

[0119] In the formula, f(·) represents a convolution fusion function, which is composed of two convolution layers, a normalization layer and an activation function.

[0120] Then, the effective cascaded partial decoder component is used to integrate the features and This strategy can effectively suppress the interference terms in the features.

[0121] The output dimension of the traditional effective cascaded partial decoder does not match the feature , which may affect the final fusion effect. Therefore, the output convolution size of the CPD is adjusted to adapt to the dimension of the feature , and the final fusion process can be represented as:

[0122]

[0123] where CPD(·) denotes the effective cascaded partial decoders, for feature x c ′ pd The final saliency map is obtained by upsampling, and U(·) denotes upsampling.

[0124] In another exemplary embodiment of the present application, in order to further improve the accuracy of the salient object detection network model, a hybrid loss function is used to supervise the training of the salient object detection network model in this embodiment. The hybrid loss includes a BCE loss L BCE and an IoU loss L IoU Based on this, the total loss function L total can be expressed as:

[0125]

[0126] In the formula, S represents the predicted saliency map of the object, G represents the true saliency map of the object, and S(m, n) and G(m, n) are the saliency prediction value and the true value at position (m, n), respectively.

[0127] Further, the prepared network model is trained on the prepared data set, and the specific hyperparameter settings of the training process are: learning rate lr=0.05, momentum momen=0.9, weight decay weight_decay=5e-4, batch size batchsize=32, and iteration number epoch=60. The SGD (Stochastic Gradient Descent) optimizer is used to update the parameters of the model according to the gradient information calculated by the loss function, thereby minimizing the loss function.

[0128] In another exemplary embodiment of the present application, in order to quantitatively evaluate the performance of the salient object detection network model, a quantitative evaluation index function is established. Four commonly used quantitative evaluation indexes are used to evaluate the performance. The four commonly used quantitative evaluation indexes include mean absolute error (MAE), E-measure, S-measure, and weighted F-measure.

[0129] (1) MAE: MAE is used to evaluate the average pixel-level difference between the predicted result S and the true value G, and the expression is:

[0130]

[0131] (2) E-measure (E ξ ): The global mean and the local pixel value of the image are considered at the same time, and the specific expression can be written as:

[0132]

[0133] where φ ξ represents the alignment enhancement matrix, and the mean of E ξ is used as the final average index.

[0134] (3) S-measure (S m ): used to evaluate the structural similarity between the prediction result and the true value, which considers the structural similarity of object perception S o and region perception S r , and the calculation formula is S m = m x S o + (1-m) x S r . In the formula, m is set to 0.5.

[0135] (4) Weight WeightF-measure evaluates the overall performance, which intuitively summarizes F-measure by alternately calculating precision Precision and recall Recall, and gives different weights ω to different errors in different positions, which is expressed as:

[0136]

[0137] In order to emphasize the precision, β 2 is set to 0.3.

[0138] Further, the trained model parameters are imported to obtain a pre-trained model. The test image is subjected to a normalization operation and is uniformly adjusted to a resolution size of 352x352, and is input into the pre-trained network model to output a predicted salient object map.

[0139] The quantitative evaluation index function is used to calculate various indexes between the true salient object map and the predicted object salient map, and the performance of the model is further evaluated.

[0140] In another exemplary embodiment of the present application, the original image as shown in Figure 4 , the gaze salient map as shown in Figure 5 , and the true object salient image as shown in Figure 6 are taken as examples to illustrate the construction process and performance of the salient object detection network model provided by the present application. Among them, Figure 4 part (a) of which is the original image of scene one, Figure 4 part (b) of which is the original image of scene two, Figure 4 part (c) of which is the original image of scene three, Figure 4 part (d) of which is the original image of scene four, Figure 5 part (a) of which is the gaze salient map corresponding to the original image of scene one, Figure 5The (b) part of FIG. 1 is a gaze saliency map of the corresponding scene two original image, Figure 5 The (c) part of FIG. 1 is a gaze saliency map of the corresponding scene three original image, Figure 5 The (d) part of FIG. 1 is a gaze saliency map of the corresponding scene four original image, Figure 6 The (a) part of FIG. 2 is a real object saliency image of the corresponding scene one original image, Figure 6 The (b) part of FIG. 2 is a real object saliency image of the corresponding scene two original image, Figure 6 The (c) part of FIG. 2 is a real object saliency image of the corresponding scene three original image, Figure 6 The (d) part of FIG. 2 is a real object saliency image of the corresponding scene four original image. Figure 6 The real object saliency images shown in FIG. 2 include images with challenges in the test set. As shown in FIG. 3, Figure 2 The construction process of the salient object detection network model and the performance test process include:

[0141] Step 1: Prepare the data set.

[0142] The public data set is used as the training set and test set of the model. Before training, a series of transformations are performed on the images, such as flipping, cropping, rotating and standardization, to generate more training samples, so as to improve the generalization ability and performance of the model.

[0143] Step 2: Establish a salient object detection network model based on gaze guidance.

[0144] Step 2.1: Build a ResNet-50 pre-trained model.

[0145] The present application uses ResNet-50 as the encoder backbone network, and uses the official ResNet-50 pre-trained model. In the training process, the first layer of the encoder backbone network is frozen for transfer learning, and the learning rate of the other layers is reduced by 0.1 times. The four multi-scale feature representations output by the encoder are

[0146] Step 2.2: Establish a diversified feature enhancement module.

[0147] Step 2.3: Establish a gaze guidance module.

[0148] Step 2.4: Establish an integrity enhancement module.

[0149] Step 2.5: Establish a double-flow capsule module.

[0150] Step 2.6: Establish a salient detector.

[0151] Step 3: Establish a loss function to supervise the training.

[0152] Step 3.1: Set up parameter supervision training model.

[0153] Step 3.2: Given pre-training model, evaluate test image

[0154] The original image as shown in Figure 4 and the gaze saliency map as shown in Figure 5 are input into the network model for detection comparison. The detection comparison method adopted includes: salient object detection method leveraging captioning to boost semantics (CapSal) and salient object detection method of integrity cognition network (ICON), and the detection results of some challenging scene images are compared as shown in Figure 7 . Among them, Figure 7 the first behavior adopts the image detection result obtained by CapSal, Figure 7 the second behavior adopts the image detection result obtained by ICON, Figure 7 and the third behavior adopts the image detection result obtained by the salient object detection method provided by the present application. Figure 7 Each column of Figure 3 corresponds to the original image of a different scene in

[0155] Step 4: Test and evaluate model performance.

[0156] In order to quantitatively evaluate the performance of the above method, the present application selects four indexes: mean square error (MAE), S-measure (S m ), E-measure (E ξ ) and Weighted . The results are shown in Table 1. The results show that the method proposed in the present application is superior to the comparison method in salient object positioning and suppressing complete integrity detection overflow.

[0157] Table 1 Quantitative results of salient object detection

[0158]

[0159]

[0160] Based on the same inventive concept, the embodiments of the present application also provide a gaze-guided salient object detection system for implementing the above-mentioned gaze-guided salient object detection method. The problem-solving implementation scheme provided by the system is similar to the implementation scheme described in the above method, and therefore the specific limitations in one or more gaze-guided salient object detection system embodiments provided below can refer to the limitations of the gaze-guided salient object detection method described above, which will not be described here again.

[0161] In one exemplary embodiment, a gaze-guided salient object detection system is provided, comprising:

[0162] An image acquisition module is configured to acquire an original image and a gaze saliency map.

[0163] A model construction module is configured to construct a salient object detection network model. The constructed salient object detection network model comprises an encoder, a convolution module, a diversified feature enhancement module, a gaze guidance module, an integrity enhancement module, a dual-flow capsule module, and a salient detector. The encoder is connected to the diversified feature enhancement module. The diversified feature enhancement module and the convolution module are both connected to the gaze guidance module. The gaze guidance module is connected to the integrity enhancement module. The integrity enhancement module, the dual-flow capsule module, and the salient detector are sequentially connected.

[0164] A result detection module is configured to input the original image and the gaze saliency map into the salient object detection network model to obtain a salient object detection result. The encoder is configured to extract features of the original image. The convolution module is configured to perform convolutional blur processing and fusion processing on the original image and the gaze saliency map to obtain preliminary fusion features. The diversified feature enhancement module is configured to enhance the diversity of the features extracted by the encoder to obtain convolution-enhanced features. The gaze guidance module is configured to obtain gaze fusion features based on the preliminary fusion features and to realize alignment and enhancement of the convolution-enhanced features and the gaze fusion features to obtain fusion features. The integrity enhancement module is configured to learn and enhance the integrity of the fusion features to obtain integrity-enhanced features. The dual-flow capsule module is configured to generate a plurality of capsules based on the integrity-enhanced features. Each capsule comprises a pose matrix and an activation value. The pose matrix is configured to represent the relationship between the integrity-enhanced features. The salient detector is configured to fuse and decode the plurality of capsules to obtain the salient object detection result.

[0165] In an exemplary embodiment, a computer device is provided, which can be a server or a terminal, and the internal structure diagram thereof can be as shown in Figure 8As shown in the figure. The computer device includes a processor, a memory, an input / output interface (I / O for short) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store salient object detection data. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a salient object detection method based on gaze guidance.

[0166] Those skilled in the art can understand that, Figure 8 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement. In an exemplary embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in each of the method embodiments.

[0167] In an exemplary embodiment, a computer readable storage medium is provided, storing a computer program, which is executed by a processor to implement the steps in each of the method embodiments.

[0168] In an exemplary embodiment, a computer program product is provided, including a computer program, which is executed by a processor to implement the steps in each of the method embodiments.

[0169] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0170] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc.

[0171] The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a blockchain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0172] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described, but as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the present application.

[0173] The principles and implementation modes of the present application are described by applying specific examples in the present application. The above description of the embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed. In conclusion, the content of the present application should not be understood as a limitation.

Claims

1. A method for salient object detection based on gaze guidance, characterized in that: The salient object detection method based on gaze guidance includes: Get the original image and gaze saliency map; Constructing a salient object detection network model; the constructed salient object detection network model includes: an encoder, a convolution module, a diversified feature enhancement module, a gaze guidance module, an integrity enhancement module, a dual-stream capsule module, and a salient detector; the encoder is connected to the diversified feature enhancement module; the diversified feature enhancement module and the convolution module are both connected to the gaze guidance module; the gaze guidance module is connected to the integrity enhancement module; the integrity enhancement module, the dual-stream capsule module, and the salient detector are connected in sequence; The original image and the gaze saliency map are both input into the salient object detection network model to obtain a salient object detection result; wherein, the encoder is used to extract features of the original image; the convolution module is used to perform convolution blur processing on the original image and the gaze saliency map, and perform fusion processing at the same time to obtain preliminary fusion features; the diversified feature enhancement module is used to enhance the diversity of features extracted by the encoder to obtain convolution enhanced features; the gaze guidance module is used to obtain gaze fusion features based on the preliminary fusion features, and to align and enhance the convolution enhanced features and the gaze fusion features to obtain fusion features; the integrity enhancement module is used to learn and enhance the integrity of the fusion features to obtain integrity enhanced features; the dual-stream capsule module is used to generate multiple capsules based on the integrity enhanced features; each capsule includes a pose matrix and an activation value; the pose matrix is ​​used to characterize the relationship between integrity enhanced features; the salient detector is used to fuse and decode multiple capsules to obtain the salient object detection result.

2. The method for salient object detection based on gaze guidance according to claim 1, wherein: Build a salient object detection network model, including: Constructing a data set and dividing the data set into a training set and a test set; the data set includes historical images, a gaze saliency map, and an object saliency map; the object saliency map is obtained by capturing gaze data of the historical images using an eye tracker; Build an initial model; The initial model is trained and tested using the training set and the test set until a difference between the object saliency map and the predicted object saliency map meets a set value, and the trained initial model is used as the salient object detection network model.

3. The method for salient object detection based on gaze guidance according to claim 1, wherein: The diversified feature enhancement module is composed of asymmetric convolution, hole convolution and convolution aggregation.

4. The method for salient object detection based on gaze guidance according to claim 1, wherein: The gaze guidance module includes: a gaze fusion unit and a gaze guidance cross-attention unit; the gaze fusion unit is used to obtain a gaze fusion feature based on the preliminary fusion feature; the gaze guidance cross-attention unit is used to realize the alignment and enhancement of the convolution enhancement feature and the gaze fusion feature to obtain a fusion feature.

5. The method for salient object detection based on gaze guidance according to claim 1, wherein: The integrity enhancement module includes: a channel integrity enhancement submodule and a spatial integrity enhancement submodule; the channel integrity enhancement submodule is used to learn the integrity of the fusion feature; the spatial integrity enhancement submodule is used to enhance the integrity of the fusion feature.

6. The method for salient object detection based on gaze guidance according to claim 1, wherein: The construction process of the dual-stream capsule module includes: capsule construction, capsule convolution and capsule connection.

7. The method for salient object detection based on gaze guidance according to claim 6, wherein: The process of capsule construction includes: Converting the integrity enhancement feature into a first fusion feature, and converting the first fusion feature into a channel feature map through a convolutional layer; Reshape the channel feature map into an initial pose matrix and initial activation value; Connecting the initial pose matrix and the initial activation value to obtain a plurality of initial capsules; The capsule convolution process includes: Divide the plurality of initial capsules into two groups, and pass each group to two capsule convolutions to explore the relationship between the object part and the whole, thereby obtaining a pose matrix and activation value of each initial capsule; Converting the pose matrix of each initial capsule into a position matrix, and obtaining a voting matrix based on the position matrix using a transformation matrix; Using the routing algorithm, the final pose matrix and final activation value are generated based on the voting matrix and the activation value of each initial capsule to obtain the capsule convolution result; The capsule connection process includes: The concatenation results of the two capsules are merged into the first capsule and reshaped into the second capsule; Perform a convolution operation on the second capsule to obtain an enriched capsule, and reshape the enriched capsule into a third capsule to obtain an intermediate pose matrix and an intermediate activation value; The routing algorithm is used to establish the connection between the initial capsules based on the intermediate pose matrix and intermediate activation value, and finally the pose matrix and activation value of each capsule are obtained.

8. A salient object detection system based on gaze guidance, characterized in that: The gaze-guided salient object detection system comprises: An image acquisition module, used to acquire the original image and the gaze saliency map; A model construction module is configured to construct a salient object detection network model; the constructed salient object detection network model comprises: an encoder, a convolution module, a diversified feature enhancement module, a gaze guidance module, an integrity enhancement module, a dual-stream capsule module, and a salient detector; the encoder is connected to the diversified feature enhancement module; the diversified feature enhancement module and the convolution module are both connected to the gaze guidance module; the gaze guidance module is connected to the integrity enhancement module; the integrity enhancement module, the dual-stream capsule module, and the salient detector are sequentially connected; The result detection module is used to input the original image and the gaze saliency map into the salient object detection network model to obtain the salient object detection result; wherein, the encoder is used to extract the features of the original image; the convolution module is used to perform convolution blur processing on the original image and the gaze saliency map, and perform fusion processing at the same time to obtain preliminary fusion features; the diversified feature enhancement module is used to enhance the diversity of features extracted by the encoder to obtain convolution enhanced features; the gaze guidance module is used to obtain gaze fusion features based on the preliminary fusion features, and to align and enhance the convolution enhanced features and the gaze fusion features to obtain fusion features; the integrity enhancement module is used to learn and enhance the integrity of the fusion features to obtain integrity enhanced features; the dual-stream capsule module is used to generate multiple capsules based on the integrity enhanced features; each capsule includes a pose matrix and an activation value; the pose matrix is ​​used to characterize the relationship between integrity enhanced features; the salient detector is used to fuse and decode multiple capsules to obtain the salient object detection result.

Images