A visual saliency prediction method and system

Abstract

The present invention relates to the field of computer vision, and in particular to a method and system for predicting visual saliency. A first visual saliency prediction model is constructed, and the first visual saliency prediction model includes: a small cardinality feature extraction network, a global refinement module, and a decoder module. The image to be tested is input into the small cardinality feature extraction network, image features are extracted from the image to be tested, the image features are input into the global refinement module, contextual information of the image features in the global range is extracted, and the globally refined features are output. The globally refined features are input into the decoder module, and the final saliency prediction result is output. The first visual saliency prediction model is trained using a training method based on knowledge distillation to obtain a target visual saliency prediction model. The global perception ability of the target visual saliency prediction model is improved, the number of parameters of the target visual saliency prediction model is reduced, and the accuracy of the target visual saliency prediction model is further improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a visual saliency prediction method and system. Background Art

[0002] When the human brain processes visual information of a scene image, it does not treat all regional information equally, but pays more attention to certain local information, thereby greatly reducing the amount of computational processing in the human brain. At any given moment, the human visual perception system uses a selective processing strategy to perceive the environment, that is, attention is only focused on a limited spatial area or a small number of objects within the field of view. This mechanism is called the visual attention mechanism. Visual saliency prediction refers to imitating the human visual attention mechanism to predict the saliency of each pixel position in the image, which is applied to advertising, video surveillance and barrier-free technology.

[0003] Traditional saliency prediction models can be divided into bottom-up computational models and top-down computational models. The bottom-up computational model extracts three types of feature information from the input image, namely color, brightness and orientation. At multiple scales, the center-surround bias mechanism is used to imitate the human retina to obtain the saliency maps corresponding to different features, and a multi-feature fusion strategy is adopted to normalize the saliency maps of all features and fuse them into the final saliency map. In addition, there are models that design algorithms for prediction from the perspectives of information theory, graph theory, Bayesian theory and frequency domain. However, these bottom-up saliency prediction models can only capture low-level visual features, resulting in limited prediction performance. Therefore, top-down high-level visual features, auxiliary intermediate features and human visual cognitive priors are gradually integrated into saliency prediction models. However, traditional saliency prediction models rely on manually designed features. Due to the limited representation ability of manually designed features, it is often difficult to obtain accurate prediction results when processing complex scenes.

[0004] Nowadays, deep learning-based methods have become mainstream. These deep learning-based methods automatically extract richer and higher-level feature representations from images through deep neural networks, thus significantly surpassing traditional methods in saliency prediction performance. Existing deep learning-based saliency prediction models can be roughly divided into methods based on convolutional neural networks, methods incorporating long short-term memory networks, and methods based on transformers. Methods incorporating long short-term memory networks can simulate the sequence information of human visual attention transfer and improve the performance of saliency prediction models by modeling the correlation between spatial information. Transformer-based methods can aggregate the correlation information between global feature points to capture long-range feature information. Although methods incorporating long short-term memory networks and transformer-based methods can process long-range visual information and improve the accuracy of saliency prediction, they have problems such as high time complexity, slow training speed, and many weight parameters. According to the type of feature scale, methods based on convolutional neural networks can be further divided into single-scale models and multi-scale models. Single-scale models are models that use only one scale to extract features for saliency prediction. They usually use pre-trained convolutional neural networks to extract image features, and then input them into deconvolution networks, generative adversarial networks or other structures to generate the final saliency prediction results. The advantages of these models are high computational efficiency and strong real-time performance. However, the feature expression ability of single-scale models is limited, and they perform poorly when faced with salient objects with large scale differences. Multi-scale is a method that allows the model to learn different features at different scales. It uses information at different scales to enrich the representation ability of the model. This method has been proven to effectively improve the performance of the model. Multi-scale models can be further divided into two types: multi-scale input and multi-scale feature extraction. The multi-scale input work inputs pictures of different scales to obtain feature maps of different sizes, and splices the generated feature maps to obtain the final saliency map. However, the multi-scale input work requires multiple networks to extract multi-scale features, resulting in an increase in the amount of calculation. In addition, these multi-scale input models do not design more sophisticated convolution modules and components to enhance the perception ability of the network itself.

[0005] The work of multi-scale feature extraction is to use different receptive fields to capture feature information at different scales. However, these saliency prediction models based on multi-scale feature extraction still have some problems. First, these models lack effective long-distance dependency modules, the correlation information between global feature points is lost, and it is difficult to capture long-range feature information. Secondly, most of these models use fixed traditional convolution kernels to extract image features, which limits the diversity and richness of the receptive field, resulting in limited multi-scale feature expression capabilities. In addition, these models usually improve performance by increasing the depth of the network and the number of convolutional layers. As the network becomes deeper and more complex, the model can obtain more accurate results in saliency prediction tasks, but it is difficult to effectively balance the accuracy and real-time performance of visual saliency prediction. Summary of the invention

[0006] To this end, the technical problem to be solved by the present invention is to overcome the problems in the prior art that the saliency prediction model based on multi-scale feature extraction is difficult to effectively capture long-range feature information, the multi-scale feature expression ability is limited, and as the network becomes deeper and more complex, the model can obtain more accurate results in the saliency prediction task, but it is difficult to effectively balance the visual saliency prediction accuracy and real-time performance.

[0007] To solve the above technical problems, the present invention provides a visual saliency prediction method, comprising the following steps:

[0008] Constructing a first visual saliency prediction model, the first visual saliency prediction model comprising: a small cardinality feature extraction network, a global refinement module and a decoder module;

[0009] Inputting the image to be tested into the small cardinality feature extraction network to extract features from the image to be tested to obtain image features;

[0010] Inputting the image features into the global refinement module, extracting context information in the global range of the image features, and outputting the global refined features, including:

[0011] Perform two-dimensional global average pooling on the image features along the channel dimension, height dimension, and width dimension to obtain the channel dimension vector, height dimension vector, and width dimension vector;

[0012] Decompose the channel dimension vector, height dimension vector and width dimension vector into a channel dimension vector group, a height dimension vector group and a width dimension vector group composed of multiple sub-vectors through a one-dimensional convolution block respectively;

[0013] Calculate the global context refinement weight based on the channel dimension vector group, the height dimension vector group and the width dimension vector group;

[0014] Input the image features into the three-dimensional convolution block to obtain the original residual of the image features;

[0015] Multiply the original residual of the image feature by the global context refinement weight to obtain the global feature;

[0016] Perform residual connection between global features and image features to obtain global refined features;

[0017] Inputting the global refined features into the decoder module and outputting the final saliency prediction result;

[0018] A training method based on knowledge distillation is adopted to train the first visual saliency prediction model to obtain a target visual saliency prediction model. The saliency of the actual image to be tested is predicted by the target visual saliency prediction model.

[0019] Preferably, the global context refinement weight is calculated based on the channel dimension vector group, the height dimension vector group and the width dimension vector group, and the calculation formula is:

[0020] ,

[0021] Among them, A is the global context refinement weight, represents the vector outer product, r is the number of sub-vectors in the channel dimension vector group, the height dimension vector group, or the width dimension vector group;

[0022] represents the i-th sub-vector in the channel dimension vector group, represents the i-th sub-vector of the height-dimensional vector group, Represents the i-th subvector in the width dimension vector group.

[0023] Preferably, the decoder module comprises: N decoder layers and Sigmoid activation function layers, wherein the first N-1 decoder layers each comprise a two-dimensional convolution block and a multi-scale perception module connected in sequence, and the Nth decoder layer comprises a two-dimensional convolution block;

[0024] The multi-scale perception module includes: four parallel branches with different receptive fields and a second deformable convolution branch, wherein the four parallel branches with different receptive fields include three dilated convolution branches with different scales and different dilation rates and the first deformable convolution branch.

[0025] Preferably, inputting the input features of the multi-scale perception module into the multi-scale perception module, enriching the receptive field of the input features of the multi-scale perception module, and obtaining the output feature map of the multi-scale perception module comprises the following steps:

[0026] The input features of the multi-scale perception module are input into four parallel branches with different receptive fields respectively, and the features extracted by the four parallel branches are cascaded along the channel dimension to obtain the comprehensive features of the cascaded receptive fields;

[0027] The comprehensive features of the cascaded receptive fields are passed through the second deformable convolution branch for fusion and correction to obtain a mixed feature of the comprehensive multi-receptive fields;

[0028] The mixed features of the comprehensive multi-receptive fields are residually connected with the input features of the multi-scale perception module to obtain the output feature map of the multi-scale perception module.

[0029] Preferably, the small cardinality feature extraction network is an expanded residual network based on pre-trained ResNet18.

[0030] Preferably, the training method based on knowledge distillation is used to train the first visual saliency prediction model to obtain the target visual saliency prediction model, specifically:

[0031] Constructing a second visual saliency prediction model, wherein the second visual saliency prediction model includes: a large cardinality feature extraction network, a global refinement module, and a decoder module;

[0032] Inputting the saliency prediction data set into the second visual saliency prediction model to obtain a saliency prediction result of each sample, training the second visual saliency prediction model, and obtaining a trained second visual saliency prediction model;

[0033] The trained second visual saliency prediction model is used as the teacher model, and the first visual saliency prediction model is used as the student model. The teacher model guides the student model to learn and obtain the target visual saliency prediction model.

[0034] Preferably, the step of guiding the student model to learn by the teacher model to obtain the target visual saliency prediction model includes:

[0035] Input the current frame image into the trained second visual saliency prediction model, and obtain an output feature map of the last multi-scale perception module of the decoder module of the trained second visual saliency prediction model as the first output feature map;

[0036] Inputting the current frame image into the first visual saliency prediction model, obtaining an output feature map of the last multi-scale perception module of the decoder module of the first visual saliency prediction model as the second output feature map;

[0037] Calculate the similarity between the self-attention map of the first output feature map and the self-attention map of the second output feature map as the self-attention distillation loss;

[0038] Calculating the first-order statistical information and the second-order statistical information of each convolutional layer in the trained second visual saliency prediction model and the first visual saliency prediction model;

[0039] Calculate the loss between the first-order statistical information of each convolutional layer in the trained second visual saliency prediction model and the first visual saliency prediction model and the loss between the second-order statistical information;

[0040] The weighted sum of the loss between the first-order statistical information of each convolutional layer in the trained second visual saliency prediction model and the first visual saliency prediction model and the loss between the second-order statistical information is used as the feature consistency distillation loss;

[0041] Calculating a first total variation distance loss between a saliency prediction result output by the trained second visual saliency prediction model and a saliency prediction result output by the first visual saliency prediction model;

[0042] Calculating a second total variation distance loss between a saliency prediction result output by the first visual saliency prediction model and annotated images;

[0043] The weighted sum of the first total variation distance loss and the second total variation distance loss is used as the significant prediction value distillation loss;

[0044] Inputting the adjacent frame image of the current frame into the trained second visual saliency prediction model, obtaining the output feature map of the last multi-scale perception module of the decoder module of the trained second visual saliency prediction model as the third output feature map;

[0045] Inputting the adjacent frame image of the current frame into the first visual saliency prediction model, obtaining an output feature map of the last multi-scale perception module of the decoder module of the first visual saliency prediction model as a fourth output feature map;

[0046] Based on the first output feature map and the third output feature map, calculate the similarity between each position of the first output feature map and each position of the third output feature map to obtain a similarity map between the first output feature map and the third output feature map;

[0047] Based on the second output feature map and the fourth output feature map, calculating the similarity between each position of the second output feature map and each position of the fourth output feature map, and obtaining a similarity map between the second output feature map and the fourth output feature map;

[0048] Based on the similarity map between the first output feature map and the third output feature map and the similarity map between the second output feature map and the fourth output feature map, calculating the loss between the trained second visual saliency prediction model and the similarity map of the first visual saliency prediction model as the inter-frame dependency distillation loss;

[0049] Inputting the saliency prediction data set into the first visual saliency prediction model to obtain the saliency prediction result of each sample, training the first visual saliency prediction model, and using the total loss function during training to minimize the difference between the saliency prediction result output by the first visual saliency prediction model and the annotated image, thereby obtaining a target visual saliency prediction model;

[0050] Among them, the calculation formula of the total loss function is:

[0051] ,

[0052] in, represents the total loss function, is the feature consistency distillation loss, is the significant prediction value distillation loss, is the self-attention distillation loss, is the inter-frame dependency distillation loss, represents the weight corresponding to the self-attention distillation loss, Represents the weight corresponding to the inter-frame dependency distillation loss.

[0053] Preferably, the large cardinality feature extraction network is an expanded residual network based on the pre-trained ResNet50.

[0054] Preferably, the training of the second visual saliency prediction model includes: calculating a total variation distance loss function with linear regularization between a saliency prediction result output by the second visual saliency prediction model and annotated images during the training process, and performing fully supervised training on the second visual saliency prediction model.

[0055] The present invention also provides a visual saliency prediction system, comprising:

[0056] A model building module, used to build a first visual saliency prediction model, wherein the first visual saliency prediction model includes: a small cardinality feature extraction network, a global refinement module and a decoder module;

[0057] An image feature extraction module is used to input the image to be tested into the small cardinality feature extraction network, extract features from the image to be tested, and obtain image features;

[0058] The global refined feature acquisition module is used to input the image features into the global refinement module, extract the context information of the image features in the global range, and output the global refined features, including:

[0059] Perform two-dimensional global average pooling on the image features along the channel dimension, height dimension and width dimension to obtain the channel dimension vector, height dimension vector and width dimension vector;

[0060] Decompose the channel dimension vector, height dimension vector and width dimension vector into a channel dimension vector group, a height dimension vector group and a width dimension vector group composed of multiple sub-vectors through a one-dimensional convolution block respectively;

[0061] Calculate the global context refinement weight based on the channel dimension vector group, the height dimension vector group and the width dimension vector group;

[0062] Input the image features into the three-dimensional convolution block to obtain the original residual of the image features;

[0063] The original residual of the image feature is multiplied by the global context refinement weight, and the global feature obtained after the multiplication is residually connected with the image feature to obtain the global refined feature;

[0064] A saliency prediction result module, used for inputting the global refined features into the decoder module and outputting the final saliency prediction result;

[0065] The training module is used to train the first visual saliency prediction model by using a training method based on knowledge distillation to obtain a target visual saliency prediction model, and predict the saliency of the actual image to be tested by using the target visual saliency prediction model.

[0066] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0067] A visual saliency prediction method and system described in the present invention inputs a test image into a small cardinality feature extraction network, extracts features from the test image to obtain image features, inputs the image features into the global refinement module, extracts context information of the image features in a global range, and outputs global refined features. The global refinement module provides a better global context gain for the image features at a lower computational cost, effectively reduces the loss of information between global feature points, endows the network with excellent global perception capabilities, and improves the accuracy of visual saliency prediction.

[0068] In addition, the input features of the multi-scale perception module are input into the multi-scale perception module, and features are extracted respectively through four parallel branches with different receptive fields. The features extracted by the four parallel branches with different receptive fields are cascaded along the channel dimension to obtain the comprehensive features of the cascaded receptive fields. The comprehensive features of the cascaded receptive fields are fused and slightly corrected through the second deformable convolution branch to obtain mixed features of the comprehensive multi-receptive fields. The mixed features of the comprehensive multi-receptive fields are residually connected with the input features of the multi-scale perception module to obtain the output feature map of the multi-scale perception module. The multi-scale perception module provides a rich and diverse set of receptive fields for the scale perception module input features, which is conducive to the perception of feature signals of different ranges and intensities. The multi-scale perception module is used by stacking it multiple times to ensure the diversity of the receptive field sizes, which can better handle and cope with the multi-scale changes of objects.

[0069] A model lightweight training method based on knowledge distillation is adopted. The trained second visual saliency prediction model is used as the teacher model, and the first visual saliency prediction model is used as the student model. The teacher model guides the student model to learn and obtain the target visual saliency prediction model. The lightweight student model can learn more comprehensive and richer knowledge from the teacher model. The saliency prediction accuracy of the student model is further improved, the number of parameters of the target visual saliency prediction model is reduced, and a better balance is achieved in visual saliency prediction accuracy and speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0071] Figure 1 It is a structural diagram of a first visual saliency prediction model in a visual saliency prediction method and system of the present invention.

[0072] Figure 2 It is a structural diagram of a global refinement module in a visual saliency prediction method and system of the present invention.

[0073] Figure 3 It is a structural diagram of a multi-scale perception module in a visual saliency prediction method and system of the present invention.

[0074] Figure 4 It is a structural schematic diagram of a training method based on knowledge distillation in a visual saliency prediction method of the present invention.

[0075] Figure 5 It is a flowchart of the steps of a visual saliency prediction method of the present invention.

[0076] Figure 6It is a system structure diagram of a visual saliency prediction system of the present invention. DETAILED DESCRIPTION

[0077] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0078] Reference Figure 5 As shown, a visual saliency prediction method provided by an embodiment of the present invention comprises the following steps:

[0079] Step S1: Construct the first visual saliency prediction model, such as Figure 1 As shown, Figure 1 is a structural diagram of a first visual saliency prediction model, wherein the first visual saliency prediction model comprises: a small cardinality feature extraction network, a global refinement module and a decoder module;

[0080] Step S2: Input the image to be tested into the small cardinality feature extraction network to extract features from the image to be tested to obtain image features. ;

[0081] The image to be tested passes through the small cardinality feature extraction network and obtains a dimension size of The image features are: , and They represent the channel dimension, height and width of the image features respectively.

[0082] In an embodiment of the present invention, preferably, the small-cardinality feature extraction network is a dilated residual network based on pre-trained ResNet18, namely DRN18. DRN18 is a network structure that combines ResNet18 and dilated convolution, which can reduce spatial resolution loss and minimize detail information loss.

[0083] like Figure 2 As shown, Figure 2 This is a structural diagram of the global refinement module, which includes three one-dimensional convolution blocks and one three-dimensional convolution block.

[0084] Step S3: inputting the image features into the global refinement module, extracting context information of the image features in the global scope, and outputting the global refined features, including:

[0085] The image features Perform two-dimensional global average pooling along the channel dimension, height dimension, and width dimension to obtain the channel dimension vector , height dimension vector Vector with width dimension ;

[0086] Channel dimension vector , height dimension vector Vector with width dimension Decompose into a channel dimension vector group composed of multiple sub-vectors by corresponding to the one-dimensional convolution block respectively , height dimension vector group Vector group with width dimension ,

[0087] Channel dimension vector group , height dimension vector group Vector group with width dimension , calculate the global context refinement weight;

[0088] Preferably, the global context refinement weight is calculated based on the channel dimension vector group, the height dimension vector group and the width dimension vector group, and the calculation formula is: ,

[0089] Among them, A is the global context refinement weight, represents the vector outer product, r is the number of sub-vectors in the channel dimension vector group, the height dimension vector group, or the width dimension vector group;

[0090] represents the i-th sub-vector in the channel dimension vector group, represents the i-th sub-vector of the height-dimensional vector group, Represents the i-th subvector in the width dimension vector group.

[0091] Based on the CP matrix rank reduction theory, the channel dimension vector group, height dimension vector group and width dimension vector group are reversely synthesized into the global context refinement weight, which greatly saves the amount of calculation and makes the global context refinement weight have stronger feature nonlinear expression ability and detail information supplement ability.

[0092] The image features are input into the three-dimensional convolution block to obtain the original residual of the image features ;

[0093] The original residual of the image feature Multiply it with the global context refinement weight A to obtain the global feature;

[0094] Combine global features with image features Perform residual connection to obtain global refined features ;

[0095] The original residual of the image feature Multiply it with the global context refinement weight A to obtain the global feature, and then combine the global feature with the image feature Perform residual connection to obtain global refined features , the calculation formula is:

[0096] ,

[0097] in, is the global refined feature, Represents the Hadamard product.

[0098] In order to better obtain contextual information in a global scope, the present invention proposes a global refinement module based on the CP matrix rank reduction theory. The global refinement module provides better global context gain for image features at a lower computational cost, effectively reduces information loss, endows excellent global perception ability, and improves the accuracy of visual saliency prediction.

[0099] Step S4: inputting the global refined features into the decoder module and outputting the final saliency prediction result;

[0100] In this embodiment, preferably, the decoder module includes: N decoder layers and Sigmoid activation function layers, wherein the first N-1 decoder layers each include a two-dimensional convolution block and a multi-scale perception module connected in sequence, the Nth decoder layer includes a two-dimensional convolution block, and upsampling operations are performed between each decoder layer;

[0101] The globally refined features are input into the decoder module, which reduces the dimension and complexity of the globally refined features and makes the globally refined features have a richer and more diverse receptive field. The visual saliency prediction results are output through the Sigmoid activation function layer, and the prediction results that conform to the saliency distribution are generated through nonlinear mapping, which further enhances the perceptual diversity and richness of the model.

[0102] like Figure 3 As shown, Figure 3 It is a structural diagram of a multi-scale perception module, which includes: four parallel branches with different receptive fields and a second deformable convolution branch, wherein the four parallel branches with different receptive fields include three dilated convolution branches with different scales and different dilation rates and a first deformable convolution branch.

[0103] In this embodiment, preferably, inputting the input features of the multi-scale perception module into the multi-scale perception module, enriching the receptive field of the input features of the multi-scale perception module, and obtaining the output feature map of the multi-scale perception module includes the following steps:

[0104] The input features of the multi-scale perception module Four parallel branches with different receptive fields are input respectively, and the features extracted by the four parallel branches are cascaded along the channel dimension to obtain the comprehensive features of the cascaded receptive fields. ;

[0105] The calculation formula is:

[0106] ,

[0107] in, To cascade the comprehensive features of each receptive field, Indicates cascade, represents the first deformable convolution, represents all the parameters of the convolution kernel of the first deformable convolution, represents dilated convolution, Represents the convolution kernel of the dilated convolution, and the dilated convolution size is , a represents the dilation coefficient of the dilated convolution, represents the activation function, Represents the input features of the multi-scale perception module.

[0108] The comprehensive features of the cascaded receptive fields are passed through the second deformable convolution branch for fusion and correction to obtain a mixed feature of the comprehensive multi-receptive fields;

[0109] The calculation formula is: ,

[0110] in, To integrate the mixed features of multiple receptive fields, Represents all the parameters of the convolution kernel of the second deformable convolution, represents the second deformable convolution.

[0111] Perform residual connection between the mixed features of the integrated multi-receptive fields and the input features of the multi-scale perception module to obtain the output feature map of the multi-scale perception module;

[0112] In order to further enhance the perceptual diversity and richness of the saliency prediction model, the present invention innovatively proposes a multi-scale perception module, which has a very rich receptive field, ranging in size from small to large, from fixed to adaptive. By stacking this module in the saliency prediction model, the network has a rich and diverse receptive field, which can better process and cope with the multi-scale changes of objects.

[0113] Step S5: adopting a training method based on knowledge distillation to train the first visual saliency prediction model to obtain a target visual saliency prediction model, and predicting the saliency of the actual image to be tested by using the target visual saliency prediction model.

[0114] In this embodiment, preferably, the training method based on knowledge distillation is used to train the first visual saliency prediction model to obtain the target visual saliency prediction model, specifically:

[0115] Constructing a second visual saliency prediction model, wherein the second visual saliency prediction model includes: a large cardinality feature extraction network, a global refinement module, and a decoder module;

[0116] Inputting the saliency prediction data set into the second visual saliency prediction model to obtain a saliency prediction result of each sample, training the second visual saliency prediction model, and obtaining a trained second visual saliency prediction model;

[0117] The trained second visual saliency prediction model is used as the teacher model, and the first visual saliency prediction model is used as the student model. The teacher model guides the student model to learn and obtain the target visual saliency prediction model.

[0118] In this embodiment, preferably, Figure 4 As shown, Figure 4 Schematic diagram of the structure of the training method based on knowledge distillation, wherein the teacher model guides the student model to learn and obtain the target visual saliency prediction model, including:

[0119] The current frame image Input the trained second visual saliency prediction model, and obtain the output feature map of the last multi-scale perception module of the decoder module of the trained second visual saliency prediction model as the first output feature map ;

[0120] The current frame image Input the first visual saliency prediction model, and obtain the output feature map of the last multi-scale perception module of the decoder module of the first visual saliency prediction model as the second output feature map ;

[0121] Calculate the first output feature map The self-attention map and the second output feature map The similarity between the self-attention maps is used as the self-attention distillation loss ;

[0122] The calculation formula is:

[0123] ,

[0124] in, Represents the first output feature map of the last multi-scale perception module of the trained second visual saliency prediction model decoder module Middle Position and The similarity between the locations.

[0125] ,

[0126] in, Represents the second output feature map of the last multi-scale perception module of the first visual saliency prediction model decoder module Middle Position and The similarity between the locations.

[0127] ,

[0128] in, Represents the similarity between the self-attention map of the output feature map of the last multi-scale perception module of the decoder module of the trained second visual saliency prediction model and the first visual saliency prediction model, that is, the self-attention distillation loss .

[0129] Self-attention maps are used to measure the similarity between two features. This is because self-attention maps can encode high-order relationships between pixels, which are more robust than relationships between pixels.

[0130] Calculating the first-order statistical information and the second-order statistical information of each convolutional layer in the trained second visual saliency prediction model and the first visual saliency prediction model;

[0131] Calculate the loss between the first-order statistics of each convolutional layer in the trained second visual saliency prediction model and the first visual saliency prediction model The loss between the second-order statistics ;

[0132] The loss between the first-order statistics of each convolutional layer in the trained second visual saliency prediction model and the first visual saliency prediction model is The loss between the second-order statistics Weighted sum as feature consistency distillation loss;

[0133] The calculation formula for the loss between the first-order statistical information of each convolutional layer in the trained second visual saliency prediction model and the first visual saliency prediction model is: ,

[0134] in, represents the loss between the first-order statistical information of each convolutional layer in the trained second visual saliency prediction model and the first visual saliency prediction model, represents the calculation of the second norm, represents the L2 regularization operation, and E represents the total number of convolutional layers of the trained second visual saliency prediction model or the first visual saliency prediction model;

[0135] represents the feature map output by the e-th convolutional layer of the trained second visual saliency prediction model, represents the feature map output by the e-th convolutional layer of the first visual saliency prediction model, represents the number of channels of the feature map output by the e-th convolutional layer of the trained second visual saliency prediction model, Represents the number of channels of the feature map output by the e-th convolutional layer of the first visual saliency prediction model;

[0136] express Middle Channels The value at position, express Middle Channels The value at position, express The first two-dimensional matrix obtained by the global maximum pooling operation, express The second two-dimensional matrix obtained by the global maximum pooling operation.

[0137] The calculation formula for the loss between the second-order statistical information of each convolutional layer in the trained second visual saliency prediction model and the first visual saliency prediction model is: ,

[0138] in, Representation feature map and of high, Representation feature map and The width, represents the loss between the second-order statistical information of each convolutional layer in the trained second visual saliency prediction model and the first visual saliency prediction model, [ . ] transpose Represents a transpose operation.

[0139] ,

[0140] in, represents the feature consistency distillation loss, represents the corresponding weight of the loss between the first-order statistical information of each convolutional layer in the trained second visual saliency prediction model and the first visual saliency prediction model, Represents the corresponding weight of the loss between the trained second visual saliency prediction model and the second-order statistical information of each convolutional layer in the first visual saliency prediction model.

[0141] Calculate the saliency prediction results output by the trained second visual saliency prediction model The saliency prediction results output by the first visual saliency prediction model The first total variation distance loss between ;

[0142] Calculate the saliency prediction results output by the first visual saliency prediction model The second total variation distance loss between the labeled images ;

[0143] The first total variation distance loss With the second total variation distance loss Weighted sum as the distillation loss for significant prediction ;

[0144] The calculation formula of the first total variation distance loss is:

[0145] ,

[0146] in, is the first total variation distance loss, Num is the total number of pixels in the saliency prediction result image of the trained second visual saliency prediction model or the first visual saliency prediction model, Represents the saliency prediction result output by the trained second visual saliency prediction model The value of the qth pixel in Represents the saliency prediction result output by the first visual saliency prediction model The value of the qth pixel in .

[0147] The calculation formula of the second total variation distance loss is:

[0148] ,

[0149] in, is the second total variation distance loss, Represents the value of the qth pixel in the labeled image.

[0150] The calculation formula for the significant prediction value distillation loss is:

[0151] ,

[0152] in, is the significant prediction value distillation loss, represents the corresponding weight of the first total variation distance loss, Denotes the corresponding weight of the second total variation distance loss.

[0153] The adjacent frame images of the current frame Input the trained second visual saliency prediction model, and obtain the output feature map of the last multi-scale perception module of the decoder module of the trained second visual saliency prediction model as the third output feature map ;

[0154] The adjacent frame images of the current frame Input the first visual saliency prediction model to obtain the output feature map of the last multi-scale perception module of the decoder module of the first visual saliency prediction model as the fourth output feature map ;

[0155] Based on the first output feature map With the third output feature map , calculate the first output feature map Each position and the third output feature map The similarity of each position is obtained to obtain the first output feature map With the third output feature map Similarity graph between ;

[0156] Based on the second output feature map With the fourth output feature map , calculate the second output feature map Each position and the fourth output feature map The similarity of each position is used to obtain the second output feature map With the fourth output feature map Similarity graph between ;

[0157] Based on the first output feature map And the third output feature map The similarity graph between And the fourth output feature map The similarity graph between the trained second visual saliency prediction model and the first visual saliency prediction model is used as the inter-frame dependency distillation loss.

[0158] The calculation formula is:

[0159] ,

[0160] in, Represents the first output feature map Middle Positions and the third output feature map No. Similarity between locations;

[0161] According to the first output feature map Each position and the third output feature map The similarity of each position is obtained to obtain the first output feature map With the third output feature map Similarity graph between ; According to the above formula, the second output feature map can be obtained in the same way With the fourth output feature map Similarity graph between .

[0162] ,

[0163] in, represents the inter-frame dependency distillation loss.

[0164] The saliency prediction data set is input into the first visual saliency prediction model to obtain the saliency prediction result of each sample, and the first visual saliency prediction model is trained. During the training, a total loss function is used to minimize the difference between the saliency prediction result output by the first visual saliency prediction model and the annotated image, so as to obtain the target visual saliency prediction model.

[0165] The calculation formula of the total loss function is:

[0166] ,

[0167] in, represents the total loss function, is the feature consistency distillation loss, is the significant prediction value distillation loss, is the self-attention distillation loss, is the inter-frame dependency distillation loss, represents the self-attention distillation loss The corresponding weight, Represents the inter-frame dependency distillation loss The corresponding weight.

[0168] The loss function that comprehensively considers multiple aspects is helpful for learning the target visual saliency model and can optimize multiple objectives at the same time. On the basis of not increasing the number of parameters of the first visual saliency prediction model, its performance is improved and the performance is closer to the second visual saliency prediction model, making the visual saliency prediction results more accurate.

[0169] In this embodiment, preferably, the large cardinality feature extraction network is a dilated residual network based on pre-trained ResNet50, namely DRN50. DRN50 is a network structure that combines ResNet50 and dilated convolution, which can reduce spatial resolution loss and minimize detail information loss.

[0170] In this embodiment, preferably, the training of the second visual saliency prediction model includes: calculating a total variation distance loss function with linear regularization between the saliency prediction result output by the second visual saliency prediction model and the annotated image during the training process, and performing fully supervised training on the second visual saliency prediction model. The total variation distance loss function plays a key role in the fully supervised training of the visual saliency prediction model, and improves the consistency of saliency prediction by maintaining spatial continuity and smoothness, suppressing noise, and reducing the risk of overfitting, thereby improving the performance and stability of the second visual saliency prediction model.

[0171] After training, during the inference process of the model, the saliency prediction value distillation, feature consistency distillation, self-attention distillation, and inter-frame dependency distillation can all be removed, leaving only the target visual saliency prediction model. Therefore, without increasing the number of parameters and computational complexity, the target visual saliency prediction model can achieve faster inference with higher accuracy and higher temporal consistency.

[0172] Reference Figure 6 As shown, an embodiment of the present invention further provides a visual saliency prediction system, comprising:

[0173] A model building module 10, for building a first visual saliency prediction model, wherein the first visual saliency prediction model comprises: a small cardinality feature extraction network, a global refinement module and a decoder module;

[0174] An image feature extraction module 20 is used to input the image to be tested into the small cardinality feature extraction network, extract features from the image to be tested, and obtain image features;

[0175] The global refined feature acquisition module 30 is used to input the image features into the global refinement module, extract the context information of the image features in the global range, and output the global refined features, including:

[0176] Perform two-dimensional global average pooling on the image features along the channel dimension, height dimension, and width dimension to obtain the channel dimension vector, height dimension vector, and width dimension vector;

[0177] Decompose the channel dimension vector, height dimension vector and width dimension vector into a channel dimension vector group, a height dimension vector group and a width dimension vector group composed of multiple sub-vectors through a one-dimensional convolution block respectively;

[0178] Calculate the global context refinement weight based on the channel dimension vector group, the height dimension vector group and the width dimension vector group;

[0179] Input the image features into the three-dimensional convolution block to obtain the original residual of the image features;

[0180] The original residual of the image feature is multiplied by the global context refinement weight, and the global feature obtained after the multiplication is residually connected with the image feature to obtain the global refined feature;

[0181] A saliency prediction result module 40, used to input the global refined features into the decoder module and output the final saliency prediction result;

[0182] The training module 50 is used to train the first visual saliency prediction model by using a training method based on knowledge distillation to obtain a target visual saliency prediction model, and predict the saliency of the actual image to be tested by using the target visual saliency prediction model.

[0183] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0184] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0185] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0186] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0187] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A visual saliency prediction method, characterized in that: The following steps are involved: Constructing a first visual saliency prediction model, the first visual saliency prediction model comprising: a small cardinality feature extraction network, a global refinement module and a decoder module; The image to be tested is input into the small cardinality feature extraction network to extract features from the image to be tested to obtain image features, including: the image to be tested passes through the small cardinality feature extraction network to obtain a dimension size of The image features are: , and Respectively represent the channel dimension, height and width of the image features; the small cardinality feature extraction network is a dilated residual network DRN18 based on the pre-trained ResNet18; Inputting the image features into the global refinement module, extracting context information in the global range of the image features, and outputting the global refined features, including: Perform two-dimensional global average pooling on the image features along the channel dimension, height dimension, and width dimension to obtain the channel dimension vector, height dimension vector, and width dimension vector; Decompose the channel dimension vector, height dimension vector and width dimension vector into a channel dimension vector group, a height dimension vector group and a width dimension vector group composed of multiple sub-vectors through a one-dimensional convolution block respectively; Calculate the global context refinement weight based on the channel dimension vector group, the height dimension vector group and the width dimension vector group; Input the image features into the three-dimensional convolution block to obtain the original residual of the image features; Multiply the original residual of the image feature by the global context refinement weight to obtain the global feature; Perform residual connection between global features and image features to obtain global refined features; Inputting the global refined features into the decoder module and outputting the final saliency prediction result; A training method based on knowledge distillation is adopted to train the first visual saliency prediction model to obtain a target visual saliency prediction model. The saliency of the actual image to be tested is predicted by the target visual saliency prediction model.

2. A visual saliency prediction method according to claim 1, characterized in that: The global context refinement weight is calculated based on the channel dimension vector group, the height dimension vector group and the width dimension vector group. The calculation formula is: ， Among them, A is the global context refinement weight, represents the vector outer product, r is the number of sub-vectors in the channel dimension vector group, the height dimension vector group, or the width dimension vector group; represents the i-th sub-vector in the channel dimension vector group, represents the i-th sub-vector of the height-dimensional vector group, Represents the i-th subvector in the width dimension vector group.

3. A visual saliency prediction method according to claim 1, characterized in that: The decoder module comprises: N decoder layers and Sigmoid activation function layers, wherein the first N-1 decoder layers each comprise a two-dimensional convolution block and a multi-scale perception module connected in sequence, and the Nth decoder layer comprises a two-dimensional convolution block; The multi-scale perception module includes: four parallel branches with different receptive fields and a second deformable convolution branch, wherein the four parallel branches with different receptive fields include three dilated convolution branches with different scales and different dilation rates and the first deformable convolution branch.

4. A visual saliency prediction method according to claim 3, characterized in that: Inputting the input features of the multi-scale perception module into the multi-scale perception module, enriching the receptive field of the input features of the multi-scale perception module, and obtaining an output feature map of the multi-scale perception module, includes the following steps: The input features of the multi-scale perception module are input into four parallel branches with different receptive fields respectively, and the features extracted by the four parallel branches are cascaded along the channel dimension to obtain the comprehensive features of the cascaded receptive fields; The comprehensive features of the cascaded receptive fields are passed through the second deformable convolution branch for fusion and correction to obtain a mixed feature of the comprehensive multi-receptive fields; The mixed features of the comprehensive multi-receptive fields are residually connected with the input features of the multi-scale perception module to obtain the output feature map of the multi-scale perception module.

5. A visual saliency prediction method according to claim 1, characterized in that: The training method based on knowledge distillation is used to train the first visual saliency prediction model to obtain the target visual saliency prediction model, specifically: Constructing a second visual saliency prediction model, wherein the second visual saliency prediction model includes: a large cardinality feature extraction network, a global refinement module, and a decoder module; Inputting the saliency prediction data set into the second visual saliency prediction model to obtain a saliency prediction result of each sample, training the second visual saliency prediction model, and obtaining a trained second visual saliency prediction model; The trained second visual saliency prediction model is used as the teacher model, and the first visual saliency prediction model is used as the student model. The teacher model guides the student model to learn and obtain the target visual saliency prediction model.

6. A visual saliency prediction method according to claim 5, characterized in that: The method of guiding the student model to learn by the teacher model to obtain the target visual saliency prediction model includes: Input the current frame image into the trained second visual saliency prediction model, and obtain an output feature map of the last multi-scale perception module of the decoder module of the trained second visual saliency prediction model as the first output feature map; Inputting the current frame image into the first visual saliency prediction model, obtaining an output feature map of the last multi-scale perception module of the decoder module of the first visual saliency prediction model as the second output feature map; Calculate the similarity between the self-attention map of the first output feature map and the self-attention map of the second output feature map as the self-attention distillation loss; Calculating the first-order statistical information and the second-order statistical information of each convolutional layer in the trained second visual saliency prediction model and the first visual saliency prediction model; Calculate the loss between the first-order statistical information of each convolutional layer in the trained second visual saliency prediction model and the first visual saliency prediction model and the loss between the second-order statistical information; The weighted sum of the loss between the first-order statistical information of each convolutional layer in the trained second visual saliency prediction model and the first visual saliency prediction model and the loss between the second-order statistical information is used as the feature consistency distillation loss; Calculating a first total variation distance loss between a saliency prediction result output by the trained second visual saliency prediction model and a saliency prediction result output by the first visual saliency prediction model; Calculating a second total variation distance loss between a saliency prediction result output by the first visual saliency prediction model and annotated images; The weighted sum of the first total variation distance loss and the second total variation distance loss is used as the significant prediction value distillation loss; Inputting the adjacent frame image of the current frame into the trained second visual saliency prediction model, obtaining the output feature map of the last multi-scale perception module of the decoder module of the trained second visual saliency prediction model as the third output feature map; Inputting the adjacent frame image of the current frame into the first visual saliency prediction model, obtaining an output feature map of the last multi-scale perception module of the decoder module of the first visual saliency prediction model as a fourth output feature map; Based on the first output feature map and the third output feature map, calculate the similarity between each position of the first output feature map and each position of the third output feature map to obtain a similarity map between the first output feature map and the third output feature map; Based on the second output feature map and the fourth output feature map, calculating the similarity between each position of the second output feature map and each position of the fourth output feature map, and obtaining a similarity map between the second output feature map and the fourth output feature map; Based on the similarity map between the first output feature map and the third output feature map and the similarity map between the second output feature map and the fourth output feature map, calculating the loss between the trained second visual saliency prediction model and the similarity map of the first visual saliency prediction model as the inter-frame dependency distillation loss; Inputting the saliency prediction data set into the first visual saliency prediction model to obtain the saliency prediction result of each sample, training the first visual saliency prediction model, and using the total loss function during training to minimize the difference between the saliency prediction result output by the first visual saliency prediction model and the annotated image, thereby obtaining a target visual saliency prediction model; Among them, the calculation formula of the total loss function is: , in, represents the total loss function, is the feature consistency distillation loss, is the significant prediction value distillation loss, is the self-attention distillation loss, is the inter-frame dependency distillation loss, represents the weight corresponding to the self-attention distillation loss, Represents the weight corresponding to the inter-frame dependency distillation loss.

7. A visual saliency prediction method according to claim 5, characterized in that: The large cardinality feature extraction network is an expanded residual network based on the pre-trained ResNet50.

8. A visual saliency prediction method according to claim 5, characterized in that: The training of the second visual saliency prediction model includes: calculating a total variation distance loss function with linear regularization between a saliency prediction result output by the second visual saliency prediction model and an annotated image during the training process, and performing full-supervised training on the second visual saliency prediction model.

9. A visual saliency prediction system, characterized in that: include: A model building module, used to build a first visual saliency prediction model, wherein the first visual saliency prediction model includes: a small cardinality feature extraction network, a global refinement module and a decoder module; The image feature extraction module is used to input the image to be tested into the small cardinality feature extraction network, extract features from the image to be tested, and obtain image features, including: the image to be tested passes through the small cardinality feature extraction network to obtain a dimension size of The image features are: , and Respectively represent the channel dimension, height and width of the image features; the small cardinality feature extraction network is a dilated residual network DRN18 based on the pre-trained ResNet18; The global refined feature acquisition module is used to input the image features into the global refinement module, extract the context information of the image features in the global range, and output the global refined features, including: Perform two-dimensional global average pooling on the image features along the channel dimension, height dimension and width dimension to obtain the channel dimension vector, height dimension vector and width dimension vector; Decompose the channel dimension vector, height dimension vector and width dimension vector into a channel dimension vector group, a height dimension vector group and a width dimension vector group composed of multiple sub-vectors through a one-dimensional convolution block respectively; Calculate the global context refinement weight based on the channel dimension vector group, the height dimension vector group and the width dimension vector group; Input the image features into the three-dimensional convolution block to obtain the original residual of the image features; The original residual of the image feature is multiplied by the global context refinement weight, and the global feature obtained after the multiplication is residually connected with the image feature to obtain the global refined feature; A saliency prediction result module, used for inputting the global refined features into the decoder module and outputting the final saliency prediction result; The training module is used to train the first visual saliency prediction model by using a training method based on knowledge distillation to obtain a target visual saliency prediction model, and predict the saliency of the actual image to be tested by using the target visual saliency prediction model.

Images