Human eye fixation point prediction method based on deep convolutional network and frequency domain feature enhancement

Abstract

The invention discloses a human eye fixation point prediction method based on a deep convolutional network and frequency domain feature enhancement, and the method comprises the steps: employing the deep convolutional network (namely a residual network ResNet) as a backbone network to form an encoder branch, and employing five layers of coding blocks to extract the spatial features of five layers of an input image; the five layers of spatial features extracted by the encoder are respectively sent to a frequency domain feature enhancement module for frequency domain feature enhancement based on discrete cosine transform (DCT) so as to enhance the features of each layer of human eye fixation area and reduce interference features; sending the frequency-domain-enhanced characteristics of each level into a decoding block of a corresponding layer of a decoder branch, and carrying out decoding and spatial up-sampling operation in sequence from a high layer to a low layer; and the output of the last decoding block of the decoder branch is subjected to 1 * 1 convolution and double up-sampling to obtain a human eye fixation point prediction result. According to the method, frequency domain feature enhancement and sequential decoding from a high layer to a low layer are carried out on the spatial multi-layer convolution features extracted by the backbone network, so that the human eye fixation point prediction precision is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to a human eye gaze point prediction method based on a deep convolutional network and frequency domain feature enhancement, and belongs to the technical field of human eye gaze point prediction for images. BACKGROUND

[0002] Human eye gaze point prediction technology estimates the position or region that attracts the most human eye attention in a scene by simulating human visual selection mechanism, so as to help the human brain or machine quickly identify the most important target content in the scene and provide support for subsequent effective information processing. This technology has important applications in automatic driving, image editing, advertisement placement and other fields, and can effectively improve the efficiency and accuracy of information processing and improve the intelligent level of information processing.

[0003] Human eye gaze point prediction methods can be broadly divided into two categories: methods based on traditional handcrafted features and methods based on deep learning. Methods based on traditional handcrafted features mainly extract low-level visual features such as brightness, color, texture, etc. of an image, and predict the human eye gaze point position by analyzing the contrast of these features. The prediction effect of traditional methods on simple scenes is acceptable, but the prediction effect on complex scenes is usually poor. Methods based on deep learning have stronger feature extraction and discrimination ability by designing a deep network model and training on a large-scale dataset, so they have higher prediction accuracy for complex scenes. However, current human eye gaze point prediction models based on deep learning usually use convolutional networks or Transformer networks to extract spatial information of images, and the degree of utilization of frequency domain information is low, resulting in insufficient utilization of image information, and problems such as insufficient adaptability and low prediction accuracy when facing complex scenes. Some methods use deeper network structures to obtain a certain prediction accuracy improvement, but result in a large increase in computational complexity and are more difficult to train.

[0004] To solve the above problems, the present application adds a frequency domain feature enhancement module to the deep convolutional network, fully combines spatial information and frequency domain information, and effectively improves the accuracy of human eye gaze point prediction.

[0005] The information disclosed in this BACKGROUND section is only for the purpose of increasing the understanding of the overall background of the present application, and should not be considered as recognition or in any form as suggesting that this information forms prior art that is known to those of ordinary skill in the art. SUMMARY

[0006] The present application aims at the defects and deficiencies of the prior art, and provides a human eye fixation point prediction method based on a deep convolutional network and frequency domain feature enhancement, which adds a multi-layer frequency domain feature enhancement module on the basis of a deep convolutional network framework, better utilizes image spatial information and frequency domain information, and adopts sequential decoding from a high layer to a low layer in a decoder branch, thereby improving human eye fixation point prediction accuracy.

[0007] The technical scheme adopted by the present application to solve its technical problems is: a human eye fixation point prediction method based on a deep convolutional network and frequency domain feature enhancement, comprising:

[0008] Step 1: a deep convolutional network (such as a residual network ResNet) is used as a backbone network to constitute an encoder branch, and five layers of encoding blocks are used to extract five levels of spatial features of an input image;

[0009] Step 2: the five layers of spatial features extracted by the encoder are respectively sent to a frequency domain feature enhancement module for frequency domain feature enhancement based on a discrete cosine transform (DCT), so as to strengthen human eye fixation area features of each layer and reduce interference features;

[0010] Step 3: the frequency domain enhanced features of each level are sent to a decoding block of a corresponding layer of a decoder branch for sequential decoding and spatial upsampling operation from a high layer to a low layer;

[0011] Step 4: an output of the last decoding block of the decoder branch is subjected to 1x1 convolution and two times upsampling to obtain a human eye fixation point prediction result.

[0012] Further, step 1 of the present application uses a deep convolutional network (such as a residual network ResNet) as a backbone network to constitute an encoder branch, and five layers of encoding blocks are used to extract five levels of spatial features of an input image, and the process and expression of feature extraction include:

[0013] Firstly, the spatial size of the input image is uniformly scaled to 224x224 pixels;

[0014] Then, five layers of encoding blocks of a residual network ResNet34 architecture are used to extract five layers of spatial convolution features F n , {F n = EncBlock(I) | n = 0, 1, 2, 3, 4}

[0015] In the formula, n represents the index of the five layers of output features of the encoding block, and 3x3 convolution and residual connection are used inside the encoding block EncBlock(); the five layers of spatial convolution features {F nThe height, width and channel dimension of the five layers of spatial features extracted by the encoder are {112x112x64, 56x56x64, 28x28x128, 14x14x256, 7x7x512}, respectively.

[0016] Further, the step 2 of the present application sends the five layers of spatial features extracted by the encoder into a frequency domain feature enhancement module respectively for frequency domain feature enhancement based on discrete cosine transform (DCT) to strengthen the features of the eye fixation area of each layer and reduce interference features, and the processing flow and expression include:

[0017] The five layers of spatial features extracted by the encoder branch are respectively subjected to frequency domain feature enhancement processing, and the processing flow of the frequency domain feature enhancement module is the same. The output features after enhancement have the same dimension as the input features. The frequency domain enhancement flow of the nth layer of encoding features is taken as an example for description.

[0018] First, the multi-channel feature map F n extracted by the input encoding block is subjected to two-dimensional discrete cosine transform (DCT) respectively. n n The DCT frequency domain representation of each feature channel of F is obtained; for example, the DCT frequency domain representation of the ith feature channel of F is obtained. The calculation expression is: wherein is the DCT frequency domain representation of F

[0019] .

[0020]

[0021] In the formula, x represents the frequency domain coefficient of the DCT frequency domain representation of the feature map at each spatial position.

[0022] Then, the inverse discrete cosine transform (IDCT) is performed on the frequency domain representation of each channel processed by the sign function to obtain a spatial representation. The 3x3 convolution is performed on each channel of the spatial features after IDCT transformation. The Softmax processing is performed on the feature map after 3x3 convolution to obtain the spatial position saliency weight W n of each channel; for example, the spatial position saliency weight W of the ith feature channel processed by the sign function is obtained. The calculation expression of the spatial position saliency weight W is:

[0023]

[0024] In the formula, x represents the frequency domain coefficient of the DCT frequency domain representation of the feature map at each spatial position. For the input feature channels generated based on DCT frequency domain characteristics The saliency weight of two-dimensional spatial location, Conv 3×3 It is a 3×3 convolution. Indicates to Two-dimensional IDCT transform;

[0025] Finally, the spatial significance weights W of each channel are... n The multi-channel feature map F extracted from the input coded block n Multiply the corresponding parts, then multiply the result by F. n By adding the corresponding features, we obtain the frequency-enhanced multi-channel feature map. For example, regarding F n The i-th feature channel The calculation expression is: In the formula ⊙ and These represent element-wise multiplication and addition operations of two two-dimensional feature maps in spatial location, respectively.

[0026] Furthermore, in step 3 of this invention, the frequency-enhanced features at each level are fed into the decoding block of the corresponding layer of the decoder branch, and sequential decoding and spatial upsampling operations are performed from higher to lower layers. The processing flow and expression include:

[0027] The nth layer decoding block receives the output features of the previous layer (n+1th layer) decoding block and the frequency-domain enhanced features of the current layer (nth layer) coding block as input, and the decoded features are used as the decoding output features of the current layer (nth layer). The 4th layer decoding block only uses the frequency-domain enhanced features of the 4th layer coding block as input.

[0028] The specific processing flow for the nth layer decoding block is as follows:

[0029] First, the output feature F of the decoded block of the previous layer (the (n+1)th layer) n+1 Bilinear interpolation is used to perform a 2x spatial upsampling to obtain the frequency domain enhancement features of the current layer (nth layer) coding block. Same space size;

[0030] Next, the upsampled decoded block output features and the coded block frequency domain enhancement features are concatenated along the channel dimension and then subjected to a 1×1 convolution to transform the number of channels. The transformed feature map has 1 channel. Softmax processing is then applied to the feature map with 1 channel to obtain the spatial location importance weight W. n+1 ; Calculate the spatial location importance weight W n+1 The expression is:

[0031]

[0032] In the formula, W n+1Conv 1×1 is a 1x1 convolution, Concat is a channel-wise concatenation operation, Up(F n+1 ) is a 2x upsampling operation on the output feature F n+1 of the previous layer (n+1th layer);

[0033] Finally, the spatial position importance weight W n+1 is multiplied with the multi-channel feature map of the encoded block after the frequency domain enhancement, and the multiplication result is added with the output feature F of the n-th layer decoding block, to obtain the output feature F of the n-th layer decoding block; for the n-th layer decoding block, the calculation expression is: n wherein and respectively represent the element-wise multiplication and addition operations of two two-dimensional feature maps in the spatial position.

[0034] Further, the step 4 of the present application obtains the human eye fixation point prediction result by performing 1x1 convolution and two times upsampling on the output of the last decoding block of the decoder branch, and the processing flow and expression include:

[0035] The dimension of the output feature F0 of the last layer (0th layer) decoding block is 112x112x64, in order to obtain the human eye fixation point prediction map with the same spatial size as the network input, directly performing 1x1 convolution and two times upsampling operation on F0 to obtain the human eye fixation point prediction result map P with the channel number of 1, and the calculation expression is:

[0036] P = Up(Conv 1×1 (F0))

[0037] wherein, Up() represents the two times upsampling operation using bilinear interpolation, and Conv 1×1 is a 1x1 convolution.

[0038] Further, the method of the present application includes the human eye fixation point prediction model based on the deep convolutional network and the frequency domain feature enhancement, and the training method of the human eye fixation point prediction model based on the deep convolutional network and the frequency domain feature enhancement is as follows:

[0039] obtaining a training image and a corresponding human eye fixation point true value image;

[0040] inputting the training image into the human eye fixation point prediction model based on the deep convolutional network and the frequency domain feature enhancement to obtain a human eye fixation point prediction map;

[0041] ​The human eye fixation point prediction model is supervised according to the human eye fixation point true value image, a loss function of the feedback human eye fixation point prediction model is constructed, and finally a trained human eye fixation point prediction model based on a deep convolutional network and frequency domain feature enhancement is obtained.

[0042] The loss function Loss(P, G) is jointly composed of the Kullback-Leibler divergence KLD(P, G) and the Pearson correlation coefficient PCC(P, G) between the human eye fixation point prediction image P and the human eye fixation point true value image G, and the calculation expression is:

[0043] Loss(P,G)=KLD(P,G)-λ×PCC(P,G)

[0044]

[0045] In the formula, P and G represent the human eye fixation point prediction image and the human eye fixation point true value image respectively, P i and G i respectively represent the human eye fixation point prediction value and the true value at the spatial position i; λ is a positive constant (0.2), representing the weight coefficient between KLD(P, G) and PCC(P, G); ∈ is a small positive constant (0.001) to maintain the stability of the calculation value; cov(P, G) represents the covariance between P and G, and std(P) and std(G) represent the standard deviations of P and G respectively.

[0046] Advantages:

[0047] 1. The human eye fixation point prediction method based on the deep convolutional network and the frequency domain feature enhancement provided in the application better combines the image spatial information and the frequency domain information compared with the general prediction method based on the convolutional network structure, and the prediction method based on the deep convolutional network and the frequency domain feature enhancement effectively improves the human eye fixation point prediction accuracy by using the decoding and prediction from high to low in the decoder branch.

[0048] 2. The network model in the application adopts the Kullback-Leibler divergence and the Pearson correlation coefficient to jointly constitute the loss function during training, so that the training process is more robust, and the model training result is better. DETAILED DESCRIPTION

[0049] Figure 1 is a flowchart of the human eye fixation point prediction method based on the deep convolutional network and the frequency domain feature enhancement provided in the embodiment.

[0050] Figure 2is a structural schematic diagram of a frequency domain feature enhancement module provided by the embodiment.

[0051] Figure 3 is a structural schematic diagram of a decoder module provided by the embodiment, taking block 2 as an example. DETAILED DESCRIPTION

[0052] The application will be further described below with reference to the drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the application, and cannot be used to limit the protection scope of the application.

[0053] As shown in the figure, the embodiment provides a human eye fixation point prediction method based on a deep convolutional network and frequency domain feature enhancement, comprising the following steps: Figure 1

[0054] An original image to be predicted is acquired;

[0055] The original image is input into a trained human eye fixation point prediction model based on a deep convolutional network and frequency domain feature enhancement, to obtain a human eye fixation point prediction result corresponding to the original image;

[0056] The human eye fixation point prediction model based on a deep convolutional network and frequency domain feature enhancement comprises an encoder branch, a multi-layer frequency domain feature enhancement module, and a decoder branch, and the specific processing method is as follows:

[0057] The encoder branch scales the acquired original image to be predicted to a proper spatial size, uses a deep residual convolutional network (ResNet34) as a backbone network to constitute the encoder branch, and uses five encoding blocks to extract five levels of spatial features of the input image;

[0058] The five levels of spatial features extracted by the encoder branch are respectively sent into the frequency domain feature enhancement module for frequency domain feature enhancement based on a discrete cosine transform (DCT), to strengthen the human eye fixation area features of each layer and reduce interference features;

[0059] The decoder branch comprises five decoding blocks, each decoding block uses the output features of the previous decoding block and the current layer of frequency domain enhanced encoding features as input, and performs decoding and spatial upsampling operations from high layer to low layer in sequence;

[0060] The human eye fixation point prediction result is obtained by 1x1 convolution and two times upsampling of the output of the last decoding block of the decoder branch.

[0061] ​The technical concept of this invention is as follows: Compared with general prediction methods based on deep convolutional networks, the provided human gaze point prediction method based on deep convolutional networks and frequency domain feature enhancement better utilizes image spatial and frequency domain information by adding multiple layers of frequency domain feature enhancement modules, thereby improving the accuracy of human gaze point prediction. The frequency domain feature enhancement module has high computational efficiency and is plug-and-play, making it easy to deploy in practical application systems.

[0062] like Figure 1 As shown, the human gaze prediction model based on deep convolutional networks and frequency domain feature enhancement includes an encoder branch, a multi-layer frequency domain feature enhancement module, and a decoder branch. The encoder branch includes a five-layer encoding block based on a deep residual convolutional network (ResNet34); the multi-layer frequency domain feature enhancement module has the same computational process to enhance the frequency domain features of the five layers of features output by the encoder; the decoder branch includes a five-layer decoding block.

[0063] The encoder branch scales the acquired original image to be predicted to an appropriate spatial size. It uses a deep residual convolutional network (ResNet34) as the backbone network to form the encoder branch, and uses five coding blocks to extract spatial features of five levels from the input image. The feature extraction process and expression are as follows:

[0064] First, the spatial dimensions of the input images are uniformly scaled to 224×224 pixels;

[0065] Subsequently, the five-layer coding blocks of the ResNet34 deep residual network architecture are used to extract the five-layer spatial convolutional features F of image I. n ,

[0066] {F n =EncBlock(I)|n=0, 1, 2, 3, 4}

[0067] In the formula, n represents the index of the five-layer output features of the encoding block, and the encoding block EncBlock() is constructed using 3×3 convolutions and residual connections; the extracted five-layer spatial convolution features {F n The height, width, and channel dimensions of |n=0, 1, 2, 3, 4} are {112×112×64, 56×56×64, 28×28×128, 14×14×256, 7×7×512}, respectively.

[0068] like Figure 1 and Figure 2 As shown, the five spatial features extracted from the encoder branch are fed into the frequency domain feature enhancement module for frequency domain feature enhancement based on Discrete Cosine Transform (DCT) to strengthen the features of the human eye's gaze region in each layer and reduce interference features. The processing flow and expression are as follows:

[0069] The five spatial features extracted from the encoder branches are subjected to frequency domain feature enhancement processing. The processing flow of the frequency domain feature enhancement module is the same, and the enhanced output features have the same dimension as the input features. The following explanation uses the frequency domain enhancement process of the nth layer encoded features as an example.

[0070] First, input the multi-channel feature map F extracted from the coded block. n , for F n Perform a two-dimensional discrete cosine transform (DCT) on each channel to obtain the DCT frequency domain representation of each feature map; for example, for F... n The i-th feature channel The calculation expression is: In the formula To Frequency domain representation after two-dimensional DCT transform;

[0071] The DCT frequency domain coefficients of each feature map are processed using the sign function sign(), which is defined as follows:

[0072]

[0073] In the formula, x represents the frequency domain coefficients of the feature map DCT at each spatial location;

[0074] Then, the inverse discrete cosine transform (IDCT) is performed on the frequency domain representation of each channel after processing by the sign function sign() to obtain the spatial representation. A 3×3 convolution is then performed on the spatial feature maps of each channel after the IDCT transform, and the Softmax processing is applied to the feature maps after the 3×3 convolution to obtain the spatial position saliency weights W for each channel. n For example, the i-th feature channel after processing the sign function. Calculate the spatial location saliency weight The expression is:

[0075]

[0076] In the formula, For the input feature channels generated based on DCT frequency domain characteristics The saliency weight of two-dimensional spatial location, Conv 3×3 It is a 3×3 convolution. Indicates to Two-dimensional IDCT transform;

[0077] Finally, the spatial significance weights W of each channel are... n The multi-channel feature map F extracted from the input coded block n Multiply the corresponding parts, then multiply the result by F. nBy adding the corresponding features, we obtain the frequency-enhanced multi-channel feature map. For example, regarding F n The i-th feature channel The calculation expression is: In the formula ⊙ and These represent element-wise multiplication and addition operations of two two-dimensional feature maps in spatial location, respectively.

[0078] like Figure 1 and Figure 3 As shown, the decoder branch consists of five decoding blocks. Each decoding block uses the output features of the previous decoding block and the frequency-enhanced encoded features of the current layer as input, performing sequential decoding and spatial upsampling operations from the higher to the lower layers. The height, width, and channel dimensions of the output features of the five decoding blocks from the higher layer (layer 4) to the lower layer (layer 0) are {7×7×512, 14×14×256, 28×28×128, 56×56×64, 112×112×64}, respectively. The specific processing flow and expressions are as follows:

[0079] The nth layer decoding block receives the output features of the previous layer (n+1th layer) decoding block and the frequency-domain enhanced features of the current layer (nth layer) coding block as input, and the decoded features are used as the decoding output features of the current layer (nth layer). The 4th layer decoding block only uses the frequency-domain enhanced features of the 4th layer coding block as input.

[0080] The specific processing flow for the nth layer decoding block is as follows:

[0081] First, the output feature F of the decoded block of the previous layer (the (n+1)th layer) n+1 Bilinear interpolation is used to perform a 2x spatial upsampling to obtain the frequency domain enhancement features of the current layer (nth layer) coding block. Same space size;

[0082] Next, the upsampled decoded block output features and the coded block frequency domain enhancement features are concatenated along the channel dimension and then subjected to a 1×1 convolution to transform the number of channels. The transformed feature map has 1 channel. Softmax processing is then applied to the feature map with 1 channel to obtain the spatial location importance weight W. n+1 ; Calculate the spatial location importance weight W n+1 The expression is:

[0083]

[0084] In the formula, W n+1 To calculate the spatial importance weights, Conv 1×1 For 1×1 convolution, Concat is a channel-dimensional concatenation operation, Up(F n+1′ F represents the output feature F of the decoded block of the previous layer (the (n+1)th layer).n+1 Perform a 2x spatial upsampling operation;

[0085] Finally, the spatial location importance weight W n+1 Multi-channel feature map after frequency domain enhancement of the coded block Multiply corresponding items, then multiply the result by... By adding the corresponding features, we obtain the output feature map F of the nth layer decoding block. n For example, for the nth layer decoded block, the calculation expression is: In the formula ⊙ and These represent element-wise multiplication and addition operations of two two-dimensional feature maps in spatial location, respectively.

[0086] Furthermore, the output of the last decoded block in the decoder branch is convolved with 1×1 and upsampled twice to obtain the human eye gaze prediction result. The processing flow and expression are as follows:

[0087] The output feature F0 of the last layer (layer 0) decoder block has a dimension of 112×112×64. To obtain a human gaze prediction map with the same size as the network input space, F0 is directly convolved with 1×1 and upsampled by a factor of two to obtain a human gaze prediction result map P with 1 channel. The calculation expression is as follows:

[0088] P = Up(Conv) 1×1 (F0))

[0089] In the formula, Up() represents a double upsampling operation using bilinear interpolation, and Conv 1×1 It is a 1×1 convolution.

[0090] like Figure 1 As shown, the training method for the human eye gaze prediction model based on deep convolutional networks and frequency domain feature enhancement is as follows:

[0091] Obtain the training images and the corresponding ground truth images of human eye gaze points;

[0092] The training images are input into a human eye gaze prediction model based on deep convolutional networks and frequency domain feature enhancement to obtain a human eye gaze prediction map.

[0093] Based on the ground truth image of human eye gaze point, the prediction map of human eye gaze point is supervised, and a loss function for the human eye gaze point prediction model is constructed to finally obtain the trained human eye gaze point prediction model based on deep convolutional network and frequency domain feature enhancement.

[0094] The loss function Loss(P,G) is jointly constructed by the Kullback-Leibler divergence KLD(P,G) between the predicted gaze map P and the ground truth gaze image G, and the Pearson correlation coefficient PCC(P,G). The calculation expression is as follows:

[0095] Loss(P,G)=KLD(P,G)-λ×PCC(P,G)

[0096]

[0097] In the formula, P and G represent the predicted gaze point image and the ground truth gaze point image, respectively. i With G i denoted as the predicted and true values ​​of the human eye fixation point at spatial location i, respectively; λ is a positive constant (taken as 0.2) representing the weighting coefficient between KLD(P,G) and PCC(P,G); ∈ is a small positive constant (taken as 0.001) to maintain the stability of the calculated values; cov(P,G) represents the covariance between P and G, and std(P) and std(G) represent the standard deviations of P and G, respectively.

[0098] In summary, to address the problems of low prediction accuracy, high computational complexity, and practical deployment difficulties in existing deep convolutional network-based human gaze prediction models due to insufficient utilization of image frequency domain information, this application proposes a human gaze prediction method based on deep convolutional networks and frequency domain feature enhancement. This method adds a frequency domain feature enhancement module to the deep convolutional network, fully combining image spatial and frequency domain information to solve the aforementioned problems. Compared with general prediction methods based on deep convolutional network structures, this proposed method better combines image spatial and frequency domain information. Furthermore, the method employs sequential decoding and prediction from high to low layers in the decoder branch, effectively improving the accuracy of human gaze prediction. The proposed frequency domain feature enhancement module has high computational efficiency, is plug-and-play, and easy to deploy in practical application systems. During training, the loss function is jointly constructed using Kullback-Leibler divergence and Pearson correlation coefficient, making the training process more robust and resulting in better model training results.

[0099] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0100] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0101] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0102] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0103] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for predicting human eye gaze points based on deep convolutional networks and frequency domain feature enhancement, characterized in that... include: Step 1: The encoder branch is constructed using a deep convolutional network (such as ResNet) as the backbone network, and five layers of coding blocks are used to extract the spatial features of five levels of the input image. Step 2: The five spatial features extracted by the encoder are sent to the frequency domain feature enhancement module for frequency domain feature enhancement based on discrete cosine transform (DCT) to enhance the features of the human eye gaze region in each layer and reduce interference features. Step 3: The enhanced features of each level in the frequency domain are sent to the decoding block of the corresponding layer of the decoder branch, and sequential decoding and spatial upsampling operations are performed from the higher to the lower layers. Step 4: The human eye gaze prediction result is obtained by performing a 1×1 convolution and doubling upsampling on the output of the last decoder block of the decoder branch.

2. The method for predicting human eye gaze points based on deep convolutional networks and frequency domain feature enhancement according to claim 1, characterized in that, Step 1 uses a deep convolutional network (such as ResNet) as the backbone network to form the encoder branch, and uses five layers of coding blocks to extract five levels of spatial features from the input image. The feature extraction process and expression include: First, the spatial dimensions of the input images are uniformly scaled to 224×224 pixels; Subsequently, the five-layer coding blocks of the ResNet34 residual network architecture are used to extract the five-layer spatial convolutional features F of image I. n , {F n =EncBlock(I)|n=0, 1, 2, 3, 4} In the formula, n represents the index of the five-layer output features of the encoding block, and the encoding block EncBlock() is constructed using 3×3 convolutions and residual connections; the extracted five-layer spatial convolution features {F n The height, width, and channel dimensions of |n=0, 1, 2, 3, 4} are {112×112×64, 56×56×64, 28×28×128, 14×14×256, 7×7×512}, respectively.

3. The method for predicting human eye gaze points based on deep convolutional networks and frequency domain feature enhancement according to claim 1, characterized in that, Step 2 involves sending the five spatial features extracted by the encoder into the frequency domain feature enhancement module for frequency domain feature enhancement based on Discrete Cosine Transform (DCT) to strengthen the features of each layer of the human eye's gaze region and reduce interference features. The processing flow and expressions include: The five spatial features extracted from the encoder branches are subjected to frequency domain feature enhancement processing. The processing flow of the frequency domain feature enhancement module is the same, and the enhanced output features have the same dimension as the input features. The following explanation uses the frequency domain enhancement process of the nth layer encoded features as an example. First, input the multi-channel feature map F extracted from the coded block. n , for F n Perform a two-dimensional discrete cosine transform (DCT) on each channel to obtain the DCT frequency domain representation of each feature map; for example, for F... n The i-th feature channel The calculation expression is: In the formula To Frequency domain representation after two-dimensional DCT transform; The DCT frequency domain coefficients of each feature map are processed using the sign function sign(), which is defined as follows: In the formula, x represents the frequency domain coefficients of the feature map DCT at each spatial location; Then, the inverse discrete cosine transform (IDCT) is performed on the frequency domain representation of each channel after processing by the sign function sign() to obtain the spatial representation. A 3×3 convolution is then performed on the spatial feature maps of each channel after the IDCT transform, and the Softmax processing is applied to the feature maps after the 3×3 convolution to obtain the spatial position saliency weights W for each channel. n For example, the i-th feature channel after processing the sign function. Calculate the spatial location saliency weight The expression is: In the formula, For the input feature channels generated based on DCT frequency domain characteristics The saliency weight of two-dimensional spatial location, Conv 3×3 It is a 3×3 convolution. Indicates to Two-dimensional IDCT transform; Finally, the spatial significance weights W of each channel are... n The multi-channel feature map F extracted from the input coded block n Multiply the corresponding parts, then multiply the result by F. n By adding the corresponding features, we obtain the frequency-enhanced multi-channel feature map. For example, regarding F n The i-th feature channel The calculation expression is: In the formula ⊙ and These represent element-wise multiplication and addition operations of two two-dimensional feature maps in spatial location, respectively.

4. The method for predicting human eye gaze points based on deep convolutional networks and frequency domain feature enhancement according to claim 1, characterized in that, Step 3 involves sending the frequency-enhanced features of each layer into the decoding block of the corresponding layer of the decoder branch, performing sequential decoding and spatial upsampling operations from higher to lower layers. The processing flow and expressions include: The nth layer decoding block receives the output features of the previous layer (n+1th layer) decoding block and the frequency-domain enhanced features of the current layer (nth layer) coding block as input, and the decoded features are used as the decoding output features of the current layer (nth layer). The 4th layer decoding block only uses the frequency-domain enhanced features of the 4th layer coding block as input. The specific processing flow for the nth layer decoding block is as follows: First, the output feature F of the decoded block of the previous layer (the (n+1)th layer) n+1 Bilinear interpolation is used to perform a 2x spatial upsampling to obtain the frequency domain enhancement features of the current layer (nth layer) coding block. Same space size; Next, the upsampled decoded block output features and the coded block frequency domain enhancement features are concatenated along the channel dimension and then subjected to a 1×1 convolution to transform the number of channels. The transformed feature map has 1 channel. Softmax processing is then applied to the feature map with 1 channel to obtain the spatial location importance weight W. n+1 ; Calculate the spatial location importance weight W n+1 The expression is: In the formula, W n+1 To calculate the spatial importance weights, Conv 1×1 For 1×1 convolution, Concat is a channel-dimensional concatenation operation, Up(F n+1′ F represents the output feature F of the decoded block of the previous layer (the (n+1)th layer). n+1 Perform a 2x spatial upsampling operation; Finally, the spatial location importance weight W n+1 Multi-channel feature map after frequency domain enhancement of the coded block Multiply corresponding items, then multiply the result by... By adding the corresponding features, we obtain the output feature map F of the nth layer decoding block. n For example, for the nth layer decoded block, the calculation expression is: In the formula ⊙ and These represent element-wise multiplication and addition operations of two two-dimensional feature maps in spatial location, respectively.

5. The method for predicting human eye gaze points based on deep convolutional networks and frequency domain feature enhancement according to claim 1, characterized in that, Step 4 involves convolving the output of the last decoded block of the decoder branch with a 1×1 convolution and doubling upsampling to obtain the human eye gaze prediction result. The processing flow and expression include: The output feature F0 of the last layer (layer 0) decoder block has a dimension of 112×112×64. To obtain a human gaze prediction map with the same size as the network input space, F0 is directly convolved with 1×1 and upsampled by a factor of two to obtain a human gaze prediction result map P with 1 channel. The calculation expression is as follows: P=Up(Conv 1×1 (F0)) In the formula, Up() represents a double upsampling operation using bilinear interpolation, and Conv 1×1 It is a 1×1 convolution.

6. The method for predicting human eye gaze points based on deep convolutional networks and frequency domain feature enhancement according to claim 1, characterized in that, The method includes a human gaze point prediction model based on deep convolutional networks and frequency domain feature enhancement. The training method for the human gaze point prediction model based on deep convolutional networks and frequency domain feature enhancement is as follows: Obtain the training images and the corresponding ground truth images of human eye gaze points; The training images are input into a human eye gaze prediction model based on deep convolutional networks and frequency domain feature enhancement to obtain a human eye gaze prediction map. Based on the ground truth image of human eye gaze point, the prediction map of human eye gaze point is supervised, and a loss function for the human eye gaze point prediction model is constructed to finally obtain the trained human eye gaze point prediction model based on deep convolutional network and frequency domain feature enhancement. The loss function Loss(P,G) is jointly constructed by the Kullback-Leibler divergence KLD(P,G) between the predicted gaze map P and the ground truth gaze image G, and the Pearson correlation coefficient PCC(P,G). The calculation expression is as follows: Loss(P,G)=KLD(P,G)-λ×PCC(P,G) In the formula, P and G represent the predicted gaze point image and the ground truth gaze point image, respectively. i With G i Let i represent the predicted value and the true value of the human eye fixation point at spatial location i, respectively; λ is a positive constant (taken as 0.2) representing the weighting coefficient between KLD(P,G) and PCC(P,G); ∈ is a positive small constant (taken as 0.001) to maintain the stability of the calculated values; cov(P, G) represents the covariance between P and G, and std(P) and std(G) represent the standard deviations of P and G, respectively.

Images