Gaze estimation method and device based on multi-scale convolutional neural network and self-attention network, equipment and storage medium

Abstract

The invention discloses a staring estimation method, device and equipment based on a multi-scale convolutional neural network and a self-attention network and a storage medium, and relates to the technical field of image data processing, and the method comprises the steps: carrying out the parallel feature extraction of a target face image through a preset multi-scale convolutional network, and obtaining an initial face feature; performing feature cross fusion on the initial facial features to obtain multi-scale fusion features; performing feature enhancement on the multi-scale fusion features through a self-attention network to obtain a target feature map; and performing staring direction prediction based on the target feature map to obtain a target staring result. According to the method, the features of different sizes of the target face image are extracted in parallel, then the initial face features are subjected to feature cross fusion, and the features of different scales are effectively fused through different receptive fields; and finally, enhancing global information of the multi-scale fusion features through the self-attention network, and improving the precision of a final target staring result based on the accurately obtained target feature map.

Description

Technical Field

[0001] The present application relates to the technical field of image data processing, and in particular to a gaze estimation method, apparatus, device and storage medium based on a multi-scale convolutional neural network and a self-attention network. Background Art

[0002] Gaze is a kind of gazing behavior. As one of the non-verbal interaction cues, gaze direction reflects meaningful attention information. Gaze estimation technology is widely used in human-computer interaction, augmented reality or virtual reality, medical diagnosis, assisted driving and other fields.

[0003] Commonly used gaze estimation techniques can be divided into model-based methods and appearance-based methods. Model-based methods are mostly limited by strict user-camera distance or professional equipment, and have poor reliability in unconstrained environments. Although appearance-based methods have fewer restrictions, they are easily affected by environmental factors and cannot solve the accuracy problem under occlusion.

[0004] Therefore, how to effectively improve the accuracy of 3D gaze estimation has become a problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of this application is to provide a gaze estimation method, device, equipment and storage medium based on a multi-scale convolutional neural network and a self-attention network, aiming to solve the technical problem of how to effectively improve the accuracy of three-dimensional gaze estimation.

[0006] To achieve the above objectives, the present application proposes a gaze estimation method based on a multi-scale convolutional neural network and a self-attention network, the method comprising: Perform parallel feature extraction on the target facial image through a preset multi-scale convolutional network to obtain initial facial features; Performing feature cross-fusion on the initial facial features to obtain multi-scale fusion features; Performing feature enhancement on the multi-scale fusion features through a self-attention network to obtain a target feature map; A gaze direction prediction is performed based on the target feature map to obtain a target gaze result.

[0007] In one embodiment, the initial facial features include initial face features and initial eye features; The step of performing feature cross-fusion on the initial facial features to obtain multi-scale fusion features comprises: Performing feature cross-fusion on the initial eye features through a preset multi-scale feature fusion module to obtain a first fusion feature; Upsampling the initial eye feature to obtain a deformed eye feature; Performing feature cross-fusion on the deformed eye feature and the initial facial feature through the preset multi-scale feature fusion module to obtain a second fusion feature; Preset feature splicing is performed on the first fusion feature and the second fusion feature to obtain a multi-scale fusion feature.

[0008] In one embodiment, the preset multi-scale feature fusion module includes a channel feature module, a first position feature module, and a second position feature module; The step of performing feature cross-fusion on the initial eye features through a preset multi-scale feature fusion module to obtain a first fusion feature comprises: Performing channel feature analysis on the initial eye feature through the channel weight module to obtain channel weight information; Performing spatial feature analysis on the initial eye feature by the first position feature module to obtain first position weight information; Performing spatial feature analysis on the initial eye feature by the second position feature module to obtain second position weight information; The initial eye feature, the channel weight information, the first position weight information, and the second position weight information are multiplied item by item to obtain a first fusion feature.

[0009] In one embodiment, the step of performing feature enhancement on the multi-scale fusion features through a self-attention network to obtain a target feature map includes: Perform channel convolution stacking on the multi-scale fusion features to obtain a multi-channel feature body; Performing global attention analysis on the multi-channel feature body through a fusion attention module to obtain optimized fusion features; The optimized fusion features are enhanced through a self-attention network to obtain a target feature map.

[0010] In one embodiment, the fusion attention module includes: a channel attention module, a spatial attention module and an activation module; The step of performing global attention analysis on the multi-channel feature body through a fusion attention module to obtain optimized fusion features includes: Performing preset channel processing on the multi-channel feature body through the channel attention module to obtain global channel information; Performing preset spatial processing on the multi-channel feature body through the spatial attention module to obtain global spatial information; Multiplying the global channel information and the global spatial information element by element and inputting the resultant information into the activation module to obtain a unified attention weight; The multi-channel feature body is reorganized according to the unified attention weight to obtain optimized fusion features.

[0011] In one embodiment, the step of performing gaze direction prediction based on the target feature map to obtain a target gaze result includes: Performing preset feature dimensionality reduction on the target feature map to obtain a two-dimensional gaze feature; The two-dimensional gaze feature is transformed in angle dimension to obtain a target gaze result.

[0012] In one embodiment, the backbone network of the preset multi-scale convolutional network is an improved ConvNext-T network including at least two parallel convolution kernels.

[0013] In addition, to achieve the above purpose, the present application also proposes a gaze estimation device based on a multi-scale convolutional neural network and a self-attention network. The gaze estimation device based on a multi-scale convolutional neural network and a self-attention network includes: A multi-scale feature extraction module is used to perform parallel feature extraction on the target facial image through a preset multi-scale convolutional network to obtain initial facial features; A cross-feature fusion module, used for performing cross-feature fusion on the initial facial features to obtain multi-scale fusion features; A feature optimization module, used to enhance the multi-scale fusion features through a self-attention network to obtain a target feature map; The gaze analysis module is used to transform the target feature map at a preset angle to obtain a target gaze result.

[0014] In addition, to achieve the above-mentioned objectives, the present application also proposes a gaze estimation device based on a multi-scale convolutional neural network and a self-attention network, the device comprising: a memory, a processor, and a gaze estimation program based on a multi-scale convolutional neural network and a self-attention network stored in the memory and runnable on the processor, the gaze estimation program based on the multi-scale convolutional neural network and the self-attention network being configured to implement the steps of the gaze estimation method based on the multi-scale convolutional neural network and the self-attention network as described above.

[0015] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, which is a computer-readable storage medium, and the computer-readable storage medium stores a program for implementing a gaze estimation method based on a multi-scale convolutional neural network and a self-attention network. The program for implementing a gaze estimation method based on a multi-scale convolutional neural network and a self-attention network is executed by a processor to implement the steps of the gaze estimation method based on a multi-scale convolutional neural network and a self-attention network as described above.

[0016] The present application provides a gaze estimation method, device, equipment and storage medium based on a multi-scale convolutional neural network and a self-attention network, the method comprising: performing parallel feature extraction on a target facial image through a preset multi-scale convolutional network to obtain initial facial features; performing feature cross-fusion on the initial facial features to obtain multi-scale fusion features; performing feature enhancement on the multi-scale fusion features through a self-attention network to obtain a target feature map; performing gaze direction prediction based on the target feature map to obtain a target gaze result. The present application extracts initial facial features containing features of different sizes corresponding to the target facial image in parallel to better capture the local features and global features of the image and improve the understanding of the target image; then, the present application performs feature cross-fusion on the initial facial features, effectively fuses features of different scales through different receptive fields, and obtains multi-scale fusion features; finally, the global feature information is enhanced through a self-attention network to further improve the accuracy of feature analysis, and the accuracy of the target gaze result is finally obtained based on the accurately acquired target feature map. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 This is a flow chart of the first embodiment of the gaze estimation method based on a multi-scale convolutional neural network and a self-attention network of the present application; Figure 2 A schematic diagram of the structure of a multi-scale convolutional network preset for the gaze estimation method based on a multi-scale convolutional neural network and a self-attention network in this application; Figure 3 This is a schematic diagram of the process of enhancing the self-attention network features in the gaze estimation method based on a multi-scale convolutional neural network and a self-attention network in this application; Figure 4 This is a flow chart of the second embodiment of the gaze estimation method based on a multi-scale convolutional neural network and a self-attention network of the present application; Figure 5 This is a schematic diagram of the feature cross-fusion process of the second embodiment of the gaze estimation method based on a multi-scale convolutional neural network and a self-attention network of the present application; Figure 6This is a schematic diagram of the global attention analysis process of the fusion attention module of the second embodiment of the gaze estimation method based on a multi-scale convolutional neural network and a self-attention network of the present application; Figure 7 This is a schematic diagram of the process of the gaze estimation method based on a multi-scale convolutional neural network and a self-attention network in this application; Figure 8 This is a schematic diagram of the module structure of a gaze estimation device based on a multi-scale convolutional neural network and a self-attention network according to an embodiment of the present application; Fig. 9 This is a schematic diagram of the device structure of the hardware operating environment involved in the gaze estimation method based on a multi-scale convolutional neural network and a self-attention network in an embodiment of the present application.

[0020] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0021] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0022] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0023] The main solution of this application is: to perform parallel feature extraction on the target facial image through a preset multi-scale convolutional network to obtain initial facial features; to perform feature cross-fusion on the initial facial features to obtain multi-scale fusion features; to perform feature enhancement on the multi-scale fusion features through a self-attention network to obtain a target feature map; to predict the gaze direction based on the target feature map to obtain the target gaze result.

[0024] Commonly used gaze estimation techniques can be mainly divided into model-based methods and appearance-based methods. Among them, model-based methods, as traditional methods, mainly calculate the three-dimensional gaze direction by geometrically modeling the eyes and using the geometric relationship between different facial and eye features (facial landmarks, cornea, pupil, etc.). Rely on external light sources to detect eye features, or observe the center of the pupil and the edge of the iris to infer the gaze direction. Although model-based gaze estimation methods are accurate and adaptable to head movement, they usually require personalized calibration to ensure accuracy and rely heavily on professional equipment. Therefore, they are limited by strict user-camera distance or professional equipment. These limitations affect the reliability and user-friendliness of eye tracking systems in uncontrolled environments, making this method less reliable and less accurate in unconstrained environments, and also reducing the overall user experience.

[0025] Appearance-based methods can extract features directly from eye images and predict gaze direction. This method only shows reasonable performance in constrained environments (i.e., fixed head pose and specific subjects). Alternatively, the gaze estimation accuracy may be affected by factors such as complex head poses in the captured images, object occlusion or self-occlusion, and complex environments or ambient lighting.

[0026] Therefore, the accuracy of existing gaze estimation techniques needs to be improved.

[0027] To solve the above problems, the present application first extracts the initial facial features containing features of different sizes corresponding to the target facial image in parallel, so as to better capture the local and global features of the image and improve the understanding of the target image; then, the present application performs feature cross-fusion on the initial facial features, effectively fuses features of different scales through different receptive fields, and obtains multi-scale fusion features; finally, the global feature information is enhanced through the self-attention network to further improve the accuracy of feature analysis, and numerical regression is performed based on the accurately acquired target feature map to obtain the predicted two-dimensional gaze vector, and then the angle is changed to obtain the final three-dimensional gaze result.

[0028] Therefore, the present application can make the acquired features include the spatial information of two-dimensional plane features of different scales and the spatial information of three-dimensional stereo features at the same time, and predict the gaze direction based on the accurately acquired target feature map, thereby improving the accuracy of three-dimensional gaze estimation from two-dimensional image data, and finally obtaining accurate three-dimensional gaze results.

[0029] It should be noted that the execution subject of this embodiment can be a gaze estimation system based on a multi-scale convolutional neural network and a self-attention network, or a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a gaze estimation device based on a multi-scale convolutional neural network and a self-attention network that can achieve the above functions, etc. This embodiment does not specifically limit this. The following takes the gaze estimation device based on a multi-scale convolutional neural network and a self-attention network (referred to as the estimation device) as an example of the execution subject to illustrate this embodiment and the following embodiments.

[0030] Based on this, the embodiment of the present application provides a gaze estimation method based on a multi-scale convolutional neural network and a self-attention network, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the gaze estimation method based on a multi-scale convolutional neural network and a self-attention network of the present application.

[0031] In this embodiment, the gaze estimation method based on the multi-scale convolutional neural network and the self-attention network includes steps S10 to S40: Step S10, performing parallel feature extraction on the target facial image through a preset multi-scale convolutional network to obtain initial facial features; It should be understood that the target facial image includes the left eye image, the right eye image and the face image of the target user. In this embodiment, in the feature extraction stage, a convolution residual structure is introduced to improve feature accuracy to perform parallel feature extraction on the target facial image, thereby effectively integrating features of different scales and considering different receptive fields that can be obtained by convolution kernels of different sizes.

[0032] Therefore, in a feasible implementation manner, in this embodiment, the backbone network of the preset multi-scale convolutional network is an improved ConvNext-T network including at least two parallel convolution kernels.

[0033] It is easy to understand that in this embodiment, the backbone network of the preset multi-scale convolutional network adopts a parallel structure. After inputting the two-dimensional image data of both eyes and the face, this embodiment can use multi-scale convolution kernels in the backbone network ConvNext-T to obtain more levels of receptive fields in the process of feature extraction, thereby obtaining feature maps of different scales. For the facial feature map, deeper feature information is obtained through the residual module. For the eye image features, upsampling operations are required to reach a high scale, so as to perform feature splicing later to form a multi-channel feature map.

[0034] For ease of understanding, Figure 2 Take this as an example to illustrate: Figure 2 The schematic diagram of the structure of the multi-scale convolutional network is preset for the gaze estimation method based on the multi-scale convolutional neural network and the self-attention network in this application. Figure 2 As shown, in the network, an improved ConvNext-T including at least two convolution kernels can be used as the backbone network to obtain more receptive fields in the feature extraction process.

[0035] For example, Figure 2 As shown, the preset multi-scale sentence convolutional neural network may include a 2×2 convolution module, a layer normalization module, an improved ConvNext-T network (i.e. Figure 2 Among them, this embodiment can obtain receptive fields of more scales by adding 3×3 convolution kernels and 5×5 convolution kernels in the multi-scale convolution block ConvNext Block (the original ConvNext Block only contains 7×7 convolution kernels). Compared with the existing single-scale feature extraction method, the improved ConvNext-T network of this embodiment obtains features with different receptive fields at the same level by adopting 3×3, 5×5 and 7×7 convolution kernel branches. These convolution kernels of different scales can effectively aggregate the contextual information of each region, significantly improving the network's ability to obtain global information.

[0036] At the same time, after the parallel multi-branch structure, this embodiment can also add two 1×1 convolution kernels in the improved ConvNext-T network to focus on more local feature information, in order to capture the details in the image and enhance the model's ability to express features of different sizes, providing richer visual clues for gaze estimation. This multi-scale convolution feature extraction method enables the network to learn local and global information better, thereby achieving more ideal results. Figure 2 Drop Path is a regularized variant of Stochastic Depth.

[0037] Therefore, this embodiment can obtain the overall structure of a larger target or global features in a scene by presetting a multi-scale convolution kernel in a multi-scale convolution network, and at the same time focus on local detail information in the feature map through a 1×1 small-scale convolution kernel, thereby improving the accuracy of gaze estimation from two-dimensional image data.

[0038] Step S20, performing feature cross fusion on the initial facial features to obtain multi-scale fusion features; Step S30, performing feature enhancement on the multi-scale fusion feature through a self-attention network to obtain a target feature map; It can be understood that after the left eye image, right eye image and face image of the target user are input into the preset multi-scale convolutional network, the left eye image features and the right eye image features, i.e., the initial eye features, and the face image features, i.e., the initial face features, can be outputted accordingly. Based on the consistency between the left eye image features and the right eye image features and the association between the binocular image features and the face image features, this embodiment can also perform feature cross-fusion on the left eye image features, the right eye image features and the face image features to obtain more accurate multi-scale fusion features after cross-fusion.

[0039] In addition, this embodiment also introduces Transformer, that is, the self-attention network mentioned above, to enhance the global features of multi-scale fusion features to make up for the lack of globality in the convolution feature extraction process. The network consists of a multi-head self-attention mechanism, a feedforward neural network, layer normalization, a multi-layer perceptron and residual connections.

[0040] Specifically, refer to Figure 3 To illustrate with an example, Figure 3 This is a schematic diagram of the process of enhancing the self-attention network features in the gaze estimation method based on multi-scale convolutional neural network and self-attention network in this application. Figure 3 As shown, this embodiment can be used to input The feature map of the image is reshaped by the multi-scale fusion features mentioned above, and the left and right eye features and facial feature information are integrated as a whole so that the global relationship can be captured through overall learning through the self-attention network. The feature matrix is ​​then input into the self-attention network. Figure 3 The two normalization modules, the multi-head attention module and the multi-layer perceptron module can form a self-attention network. Furthermore, this embodiment can enhance the global feature relationship in the feature body after the left and right eye images and the face image are fused by taking advantage of the global feature capture of the self-attention network.

[0041] Therefore, this embodiment introduces the Transformer module in three-dimensional gaze estimation, and strengthens the global feature information through the multi-head attention mechanism in the Transformer module. Different from the existing gaze estimation, this embodiment strengthens the local feature extraction and global feature attention by combining the convolution operation and the Transformer module, so that the model can be better adjusted during the training process to obtain more accurate gaze estimation results.

[0042] Step S40: performing gaze direction prediction based on the target feature map to obtain a target gaze result.

[0043] In a feasible implementation manner, in this embodiment, step S40 may include steps A1 to A2: Step A1, performing preset feature dimensionality reduction on the target feature map to obtain a two-dimensional gaze feature; It should be noted that, in this embodiment, the means of presetting feature dimensionality reduction for the target feature map output by the self-attention network can be to flatten it into one dimension through the flatten function in pytorch. Then, the processed one-dimensional features are sent to two fully connected layers created by the nn.Linear function and activated by the activation function, and finally a two-dimensional gaze vector is obtained, that is, the above-mentioned two-dimensional gaze feature.

[0044] Specifically, the first fully connected layer can reduce the dimension of the input features to 128. This layer generates new feature representations by learning the linear relationship between the input features. Then the output of the first fully connected layer is normalized. Batch normalization can adjust the distribution of activations in the neural network by normalization (subtracting the mean and dividing by the standard deviation), which helps to speed up the training process and improve the stability of the model. After normalization, it is activated by the Relu function, which helps to introduce the nonlinear characteristics of the network and enable the network to learn complex patterns. Dropout regularization technology can also be applied in this process to randomly set the value of the input unit to 0 with a probability of 0.5. This is a technology to prevent overfitting. It forces the network to learn more robust features by randomly "discarding" the activation values ​​of some neurons during training.

[0045] Then, a second fully connected layer is used to further reduce the dimension of the input feature from 128 to 2, and the vectors in two directions are regressed in the gaze estimation task to obtain the two-dimensional angle of gaze estimation that represents the prediction, that is, the above-mentioned two-dimensional gaze feature.

[0046] Furthermore, based on experimental experience, since Dropout may lead to a loss of prediction accuracy in image regression problems, this embodiment may also discard Dropout in the last fully connected layer to ensure better image analysis accuracy.

[0047] Step A2: transform the two-dimensional gaze feature into an angle dimension to obtain a target gaze result.

[0048] It is easy to understand that in this embodiment, the above-mentioned angle dimension transformation can be performed through the general gaze to 3D (Gaze3D) three-dimensional gaze vector conversion function. The Gaze3D conversion can realize the conversion from the spherical coordinate system to the Cartesian coordinate system. Therefore, the two-dimensional gaze angle can be converted into a three-dimensional gaze angle to obtain the prediction result of the gaze estimation, that is, the target gaze result.

[0049] In summary, this embodiment designs a simple method for three-dimensional gaze estimation based on binocular images and facial images. The network has three key parts, namely, a preset multi-scale convolutional network for extracting features, a feature cross-fusion for learning binocular feature correlation and binocular feature guidance for facial related feature extraction, and a self-attention network for strengthening global feature correlation. Thus, multi-scale feature extraction, feature cross-fusion and global feature enhancement are effectively integrated, so that the model learns more comprehensive and deeper feature information, thereby enhancing the model's ability to capture the importance and correlation of features in both spatial and channel dimensions, helping the model to better handle occlusion problems, and is not affected by environmental factors, does not require complex professional equipment, and effectively improves the accuracy, reliability and scope of application of three-dimensional estimation.

[0050] This embodiment provides a gaze estimation method based on a multi-scale convolutional neural network and a self-attention network, the method comprising: performing parallel feature extraction on a target facial image through a preset multi-scale convolutional network to obtain initial facial features; performing feature cross-fusion on the initial facial features to obtain multi-scale fusion features; performing feature enhancement on the multi-scale fusion features through a self-attention network to obtain a target feature map; performing preset feature dimension reduction on the target feature map to obtain a two-dimensional gaze feature; performing angle dimension transformation on the two-dimensional gaze feature to obtain a target gaze result. This embodiment designs a simple method for three-dimensional gaze estimation based on binocular images and facial images, the network has three key parts, namely a preset multi-scale convolutional network for extracting features, feature cross-fusion for learning binocular feature correlation and binocular feature guidance for facial related feature extraction, and a self-attention network for strengthening global feature correlation. This effectively combines multi-scale feature extraction, feature cross-fusion and global feature enhancement, allowing the model to learn more comprehensive and in-depth feature information, thereby enhancing the model's ability to capture the importance and relevance of features in both spatial and channel dimensions. It helps the model to better handle occlusion problems, is not affected by environmental factors, and does not require complex professional equipment, thereby effectively improving the accuracy, reliability and scope of applicability of three-dimensional estimation.

[0051] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction and will not be repeated later.

[0052] Based on the first embodiment, please refer to Figure 4 , Figure 4 This is a flow chart of the second embodiment of the gaze estimation method based on a multi-scale convolutional neural network and a self-attention network of the present application. In this embodiment, the initial facial features include initial facial features and initial eye features; step S20 includes steps S21 to S24: Step S21, performing feature cross-fusion on the initial eye features through a preset multi-scale feature fusion module to obtain a first fusion feature; Step S22, upsampling the initial eye feature to obtain a deformed eye feature; Step S23, performing feature cross-fusion on the deformed eye feature and the initial facial feature through the preset multi-scale feature fusion module to obtain a second fusion feature; Step S24, performing preset feature splicing on the first fusion feature and the second fusion feature to obtain a multi-scale fusion feature.

[0053] It should be noted that the present embodiment can remove interference features in facial features by utilizing the similarity of binocular features and eye features to guide facial feature extraction. Therefore, the present embodiment can consider the correlation between left and right eye features in the cross-feature fusion process, choose to directly cross-join left and right eye feature maps, and perform feature cross-fusion on the initial eye features through a preset multi-scale feature fusion module to obtain a first fusion feature; at the same time, in order to utilize eye features to enhance the extraction effect of facial features, an upsampling operation is used to make the left and right eye feature maps consistent in size with the facial feature map for convenience of feature fusion operation, that is, the deformed eye features after upsampling and the initial facial features are cross-fused through a preset multi-scale feature fusion module to obtain a second fusion feature, and finally the first fusion feature and the second fusion feature are spliced ​​by preset features to obtain a multi-scale fusion feature.

[0054] It should be understood that in this embodiment, the initial eye features and the initial facial features can be cross-fused by using different preset multi-scale feature fusion modules, such as Figure 5 As shown, Figure 5 This is a schematic diagram of the feature cross-fusion process of the second embodiment of the gaze estimation method based on a multi-scale convolutional neural network and a self-attention network of the present application. Figure 5 The multi-scale feature fusion in the above-mentioned preset multi-scale feature fusion module can be used. Since the feature processing process inside each preset multi-scale feature fusion module is the same, although Figure 5 is the initial facial feature (such as Figure 5 The internal display diagram of the preset multi-scale feature fusion module corresponding to the facial feature map in FIG. Figure 5 , with initial eye features (such as Figure 5 The function of the preset multi-scale feature fusion module proposed in this embodiment is illustrated by the feature processing process corresponding to the left eye feature map and the right eye feature map in the figure.

[0055] In a feasible implementation, in this embodiment, the preset multi-scale feature fusion module includes a channel feature module, a first position feature module and a second position feature module; step S21 may include steps B1 to B4: Step B1, performing channel feature analysis on the initial eye feature through the channel weight module to obtain channel weight information; Step B2, performing spatial feature analysis on the initial eye feature by using the first position feature module to obtain first position weight information; Step B3, performing spatial feature analysis on the initial eye feature by the second position feature module to obtain second position weight information; Step B4, multiplying the initial eye feature, the channel weight information, the first position weight information, and the second position weight information item by item to obtain a first fusion feature.

[0056] It should be understood that the present embodiment presets a multi-scale feature fusion module, and in the process of fusing features, it mainly performs feature analysis and then feature fusion through three separate branches, namely, the channel feature module, the first position feature module, and the second position feature module, in the channel dimension. Among them, the channel feature module captures different feature information, such as edges, textures, colors, etc. The introduction of channel fusion can integrate this information and help the model understand the input data more comprehensively. The feature fusion of the x-axis and y-axis corresponding to the first position feature module and the second position feature module, respectively, is to capture the global and local information of the image at the same time by fusing features at different scales, which is particularly effective for processing multi-scale objects. Therefore, in the cross-fusion process of this embodiment, by using a separate channel weight module (such as Figure 5 The first feature analysis branch in ) and the separated x-axis and y-axis weight modules (such as Figure 5 The second and third feature analysis branches in the image are used to strengthen the feature fusion between the left and right eyes and the effective guidance of facial features by both eyes.

[0057] like Figure 5 As shown, the preset multi-scale feature fusion module in this embodiment can be a residual structure, which is divided into a channel weight path, two position weight paths and a jump connection. Specifically, the channel weight path corresponding to the channel feature module can perform global average pooling, convolution, GELU activation and Sigmoid activation on the input multi-channel two-dimensional data to obtain the channel weight and multiply it by the initial eye feature data of the main path; the spatial position path corresponding to the first position feature module and the second position feature module can perform global average pooling, convolution, layer normalization and GELU activation operations on the input feature map on the x-axis and y-axis respectively, obtain the spatial weights in the x-axis and y-axis directions and multiply it by the data of the main path, and finally perform a jump connection with the input original initial eye feature data to obtain the first fusion feature.

[0058] Specifically, in the channel branch corresponding to the channel feature module, the input feature map x (i.e., the initial eye feature) is first subjected to global average pooling to compress the spatial information (height and width) of each channel into a single value to obtain the global feature (C×1×1). The mathematical expression of this step can be: ; (1) in, and are the height and width of feature x, is the feature map at position The value of .

[0059] Then, a 1x1 convolution and an activation function GELU (Gaussian Error Linear Unit) are used to learn the correlation between channels and output the same weight value as the number of channels in the input feature map. The mathematical expression of this step can be: ; (2) in, Representation 1 1 convolution operation, is the weight of the convolutional layer, represents the Gaussian error linear unit activation function.

[0060] Finally, another 1x1 convolution and a Sigmoid activation function are used to generate the first channel weight information. The mathematical expression of this step is: ; (3) in, Representation 1 1 convolution operation, are the weights of another convolutional layer, Represents the Sigmoid activation function, ensuring that the weight value is between 0 and 1.

[0061] In the position branches corresponding to the first position feature module and the second position feature module, considering the effective feature distribution problem in the gaze estimation task, this embodiment chooses to process the features in the X-axis and Y-axis directions separately, so as to obtain the effective weight selection on the X-axis and Y-axis more reasonably. The global average pooling formula for the feature values ​​in the X-axis and Y-axis directions is as follows: ; (4) ; (5) in, and are the height and width of feature x, is the feature map at position The value of .

[0062] Then two separate 1x1 convolution and batch normalization layers are used to learn the correlation in the X-axis and Y-axis directions respectively, and output the same weight value as the input feature map X-axis and Y-axis. The mathematical expression of this step is: ; (6) ; (7) in, represents a 1x1 convolution operation, is the weight of the Y-axis convolutional layer, is the weight of the X-axis convolutional layer, Represents a batch normalization layer.

[0063] Then use the GELU activation function to introduce nonlinearity. The mathematical expression of this step is: ; (8) ; (9) Finally, another 1x1 convolution is used to generate weights in the height and width directions to obtain the weight information of the first position mentioned above. and second position weight information , the expression of this step is as follows: ; (10) ; (11) in, and Represent the weights of two separate convolution kernels respectively.

[0064] In summary, in this embodiment, after the channel branch and the two position branches, three weights of different scales can be obtained: , , , and then apply these three weights to each channel of the original feature map x to obtain the fused feature map F.

[0065] At this time, corresponding to the above-mentioned multiplication of the initial eye feature, the channel weight information, the first position weight information, and the second position weight information, the mathematical expression for obtaining the first fusion feature can be: ; (12) in, represents element-wise multiplication, The size is , The size is , The size is .

[0066] It is understandable that Figure 5It can be seen that this embodiment considers the correlation between the left and right eye features before cross-feature fusion, and chooses to directly cross-join the left and right eye feature maps; at the same time, in order to use the eye features to enhance the extraction effect of facial features, an upsampling operation is used to make the left and right eye feature maps and the facial feature maps consistent in size for easy feature fusion operation. Therefore, this embodiment uses a separate channel weight module and separate x-axis and y-axis weight modules to enhance the feature fusion between the left and right eyes and the effective guidance of the two eyes on the facial features.

[0067] In a feasible implementation manner, in this embodiment, step S30 may include steps C1 to C3: Step C1, performing channel convolution stacking on the multi-scale fusion features to obtain a multi-channel feature volume; Step C2, performing global attention analysis on the multi-channel feature body through a fusion attention module to obtain optimized fusion features; Step C3, performing feature enhancement on the optimized fusion features through a self-attention network to obtain a target feature map.

[0068] It is easy to understand that this embodiment can introduce a fusion attention mechanism to evaluate the importance difference between different features before performing global feature enhancement through the self-attention network, so as to reduce the influence of other irrelevant features or weakly related features on the recognition results. Before inputting the features into the fusion attention module, for the multi-scale fusion feature map output by the feature extraction part, this embodiment can first form a multi-channel feature body by stacking in the channel dimension, that is, performing the above-mentioned channel convolution stacking, the purpose of which is to make full use of the plane feature information in the feature map and the possible correlation between each plane feature information.

[0069] In a feasible implementation, in this embodiment, the fusion attention module includes: a channel attention module, a spatial attention module and an activation module; step C1 includes steps C11 to C14: Step C11, performing preset channel processing on the multi-channel feature body through the channel attention module to obtain global channel information; Step C12, performing preset spatial processing on the multi-channel feature body through the spatial attention module to obtain global spatial information; Step C13, multiplying the global channel information and the global spatial information element by element and inputting the resultant information into the activation module to obtain a unified attention weight; Step C14, reorganizing the multi-channel feature body according to the unified attention weight to obtain optimized fusion features.

[0070] It should be noted that, unlike the traditional channel space attention mechanism, this embodiment does not choose to individually multiply the channel attention mechanism by the original features and then use the spatial attention mechanism to process the channel-enhanced features, but multiplies the channel weight by the spatial weight first and then uses a Sigmoid function to obtain a unified attention weight on the entire space and channel, and uses the integrated unified attention weight to enhance the overall features of the eye and facial feature splicing data. This allows the model to adaptively learn which channels and spatial positions are more important. This adaptability allows the model to pay more attention to the key information in the image, thereby improving the distinguishability of the features.

[0071] Specifically, combined Figure 6 The improvement effect of the fusion attention module of this embodiment is illustrated by example. Figure 6 This is a schematic diagram of the global attention analysis process of the fusion attention module of the second embodiment of the gaze estimation method based on a multi-scale convolutional neural network and a self-attention network of the present application.

[0072] like Figure 6 As shown in Figure 1, for the input feature map F, i.e., the multi-channel feature body mentioned above, the channel attention module in this embodiment first performs global average pooling and global maximum pooling operations on each channel, and calculates the average eigenvalue and maximum eigenvalue on each channel respectively. This generates two vectors containing the number of channels, the global average feature of each channel and the maximum feature respectively It is expressed as follows: ; (12) ; (13) in, and are the height and width of feature f, respectively, is the feature map F at position The value of .

[0073] Then, the feature vector after global average pooling and maximum pooling is input into a shared multi-layer perceptron, which can be used to learn the attention weights of each channel. Through learning, the network can adaptively decide which channels are more important for the current task and obtain the final global channel information. , the process expression is as follows: ; (14) in, Represents a multilayer perceptron.

[0074] For the input feature map F, the spatial attention module in this embodiment first applies MaxPooling and AvgPooling on each channel, respectively, as shown below: ; (15) Then the two 1xHxW feature maps are obtained and Connect them by channel and send them to a standard convolution layer. After the activation function, the global spatial information is obtained. , the process is expressed as follows: ( ); (16) in, and They are the feature maps after average pooling and global maximum pooling, respectively, and “+” means the addition of values.

[0075] After the channel and spatial attention modules are processed separately, unlike the existing method, this embodiment multiplies the spatial and channel attention weights once to obtain the completed channel spatial weights, and activates the obtained channel spatial weights and multiplies them element-wise with the original features to obtain the feature map after strengthening the feature attention, that is, the unified attention weight. The process expression is as follows: ; (17) in, Indicates the attention weight on the channel dimension , Represents the attention weight on the spatial dimension as , Represents element-wise multiplication. is the commonly used sigmoid activation function. And the final is the final attention weight, which is .

[0076] It can be understood that this embodiment performs element-by-element multiplication of the global channel information and the global spatial information with the original feature map so that the model can suppress noise and irrelevant information while retaining key information, thereby obtaining channel and spatial weight information that satisfies the network data structure, achieving the effect of maintaining the spatial structure of the features while strengthening important features and suppressing unimportant features.

[0077] In summary, in the fusion attention module proposed in this embodiment, the channel attention module uses maximum pooling and average pooling to process data in the channel dimension respectively. The dimensional feature data is compressed into one real number, thus obtaining two dimensional channel description. Then a multi-layer perceptron is used to reduce the dimension of the channel description and then increase the dimension, so as to help the model extract the most important features from the data by reducing the dimension first, while reducing the amount of calculation of the model. After extracting the key features, the dimension of the data is restored by the dimension increase operation for further processing or analysis. Finally, the channel weight is obtained by weighted summation through k and b and Sigmoid activation, where k and b are learning parameters and can be set according to actual conditions.

[0078] The spatial attention compresses the features of all channels at each position of the two-dimensional feature through maximum pooling and average pooling, and obtains two dimensional space description and concatenation. Then a convolution operation is used to compress the number of channels of the feature data to 1, and finally a Sigmoid operation is performed to obtain the spatial attention weight. The weights of spatial and channel attention are then integrated and calculated to obtain the final weight.

[0079] Therefore, the fusion attention module in this embodiment cleverly combines channel attention and spatial attention. This can not only handle the correlation and importance between different depths of two-dimensional data, but also capture local structural information at different positions, thereby enhancing the model's ability to capture the importance and correlation of features in both spatial and channel dimensions, and helping the model to better handle occlusion problems.

[0080] In this embodiment, the attention mechanism is introduced to evaluate the importance difference between different features to reduce the influence of other irrelevant features or weakly related features on the recognition results. Furthermore, this embodiment can simultaneously focus on the important information of the feature map in the channel dimension and the spatial dimension through the parallel channel attention module and the spatial attention module. In this way, an effective fusion feature body of left and right eye features and facial features is obtained, which enhances the model's ability to capture the importance and correlation of features in both spatial and channel dimensions, and helps the model better deal with occlusion problems.

[0081] This embodiment discloses that a preset multi-scale feature fusion module includes a channel feature module, a first position feature module and a second position feature module; the initial eye feature is analyzed through a channel weight module to obtain channel weight information; the initial eye feature is analyzed through a first position feature module to obtain a first position weight information; the initial eye feature is analyzed through a second position feature module to obtain a second position weight information; the initial eye feature, the channel weight information, the first position weight information and the second position weight information are multiplied item by item to obtain a first fusion feature. The initial eye feature is upsampled to obtain a deformed eye feature; the deformed eye feature and the initial facial feature are cross-fused through a preset multi-scale feature fusion module to obtain a second fusion feature; the first fusion feature and the second fusion feature are spliced ​​by preset features to obtain a multi-scale fusion feature. This embodiment considers the correlation between left and right eye features before cross-feature fusion, and chooses to directly cross-splice left and right eye feature maps; at the same time, in order to use eye features to enhance the extraction effect of facial features, an upsampling operation is used to make the left and right eye feature maps consistent in size with the facial feature map to facilitate feature fusion operations. Therefore, this embodiment uses a separate channel weight module and a separate spatial weight module to enhance the feature fusion between the left and right eyes and the effective guidance of the facial features by both eyes.

[0082] This embodiment also discloses channel convolution stacking of multi-scale fusion features to obtain a multi-channel feature body; preset channel processing of the multi-channel feature body through a channel attention module to obtain global channel information; preset spatial processing of the multi-channel feature body through a spatial attention module to obtain global spatial information; element-by-element multiplication of the global channel information and the global spatial information and input into an activation module to obtain a unified attention weight; feature reorganization of the multi-channel feature body according to the unified attention weight to obtain an optimized fusion feature. Feature enhancement of the optimized fusion feature is performed through a self-attention network to obtain a target feature map. This embodiment introduces an attention mechanism to evaluate the importance difference between different features to reduce the influence of other irrelevant features or weakly related features on the recognition results. Furthermore, this embodiment can simultaneously pay attention to the important information of the feature map in the channel dimension and the spatial dimension through a parallel channel attention module and a spatial attention module. Thereby obtaining an effective fusion feature body of left and right eye features and facial features, enhancing the ability of the model to capture the importance and correlation of features in the two dimensions of space and channel, and helping the model to better handle occlusion problems.

[0083] For example, in order to help understand the technical concept or technical principle of the gaze estimation method based on a multi-scale convolutional neural network and a self-attention network after combining this embodiment with the above-mentioned embodiment 1 and embodiment 2, please refer to Figure 7 , Figure 7The process diagram of the gaze estimation method based on multi-scale convolutional neural network and self-attention network in this application is as follows: like Figure 7 As shown, the processing steps of this application mainly include: extracting multiple scale features through the backbone network; obtaining multi-scale fusion feature data through cross-feature fusion; enhancing the relevance and importance of features through fusion attention; strengthening global features through the Transformer model; and finally obtaining gaze estimation results through coordinate transformation and angle calculation. Therefore, it can be divided into the following four parts: 1) Feature extraction part, that is, Figure 7 The preset multi-scale convolutional network corresponding to the multi-scale feature extraction network in the middle, combined with the above analysis, can be seen that this part adopts a parallel structure. After inputting the two-dimensional image data of both eyes and face, the backbone network is used to improve ConvNext-T to obtain feature maps of different scales. For the facial feature map, the residual module is used to obtain deeper feature information. For the eye image features, upsampling operations are required to reach a high scale, so as to perform subsequent feature splicing to form a multi-channel feature map.

[0084] At the same time, the multi-scale convolution kernel is used in the feature extraction module to improve ConvNext-T, so as to obtain more levels of receptive fields in the process of feature extraction; 2) Feature cross fusion part, that is, Figure 7 In the cross-fusion feature module corresponding to the multi-scale feature fusion, the correlation between the left and right eye features is considered, and the left and right eye feature maps are directly cross-joined. At the same time, in order to utilize the eye features to enhance the extraction effect of facial features, an upsampling operation is used to make the left and right eye feature maps consistent in size with the facial feature map to facilitate feature fusion operations.

[0085] The cross-fusion feature module is a residual structure, which is internally divided into a channel weight path, a position weight path, and a jump connection. The multi-channel two-dimensional data input in the channel path is subjected to global average pooling, fully connected layers, ReLU activation, and Sigmoid activation through branches to obtain weights and multiply them by the data of the main path; the spatial position path performs average pooling, convolution, layer normalization, and Sigmoid activation on the input feature map on the x-axis and y-axis to obtain weights in both directions of the x-axis and y-axis and multiply them by the data of the main path, and finally jumps with the input original data. This achieves the use of binocular consistency to enhance binocular features, and uses binocular features to enhance the effective extraction of facial features.

[0086] 3) The feature optimization part, namely the fusion attention module, takes as input the multi-scale fusion feature map output by the feature cross-fusion part. It forms a multi-channel feature body by stacking it in the channel dimension and then inputs it into the fusion attention module to learn the feature body as a whole through the fusion attention module.

[0087] In the fusion attention module, the channel path uses the convolution residual structure to learn features, and directly compresses the C×W×H feature map containing global information into a C×1×1 feature vector Z. The weights of C feature maps are obtained through global average pooling and multiplied by each feature map, thereby achieving weight evaluation of each feature data in the two-dimensional feature data, optimizing the model's learning and capture of key features. At the same time, nonlinearity is introduced through the use of multiple activation functions, allowing the model to learn and represent more complex feature relationships.

[0088] The spatial position path uses global pooling in two directions to directly compress the C×W×H feature map into a feature vector in two directions, C×W×1 and C×1×H, and obtains the weights on the x-axis and y-axis through operations such as convolution layer normalization and multiplies the weights with each feature map, thereby achieving weight evaluation of features in two directions in the two-dimensional feature data, helping the model to obtain more key features on the x-axis and y-axis.

[0089] Through the parallel channel attention module and spatial attention module in this module, it is possible to simultaneously focus on the important information of the feature map in the channel dimension and the spatial dimension, thereby obtaining an effective fusion feature body of left and right eye features and facial features.

[0090] 4) In the feature enhancement part, the Transformer model, namely the self-attention network, is used here to enhance the extraction of global features and make up for the lack of globality in the convolutional feature extraction process.

[0091] Finally, two fully connected layers (i.e. Figure 7 The fully connected 128 and fully connected 2 in the image are used to complete the regression of the gaze task, and the two-dimensional vector is converted into a three-dimensional vector through angle transformation to obtain the predicted gaze direction.

[0092] Depend on Figure 7 It can be seen that compared with other gaze estimation methods, this embodiment combines multi-scale feature extraction and cross-feature fusion, effectively obtaining more receptive fields in the feature extraction stage to strengthen the extraction of local features. At the same time, cross-feature fusion can better utilize binocular feature correlation and eye image features to strengthen the extraction of effective facial features. At the same time, this embodiment introduces fusion attention and self-attention networks, cleverly combining channel attention and spatial attention, thereby enhancing the model's ability to capture the importance and correlation of features in both spatial and channel dimensions, helping the model to better handle occlusion problems.

[0093] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the gaze estimation method based on a multi-scale convolutional neural network and a self-attention network of the present application. More forms of simple transformations based on this technical concept are all within the scope of protection of the present application.

[0094] This application also provides a gaze estimation device based on a multi-scale convolutional neural network and a self-attention network, please refer to Figure 8 , Figure 8 This is a schematic diagram of the module structure of a gaze estimation device based on a multi-scale convolutional neural network and a self-attention network in an embodiment of the present application. In this embodiment, the gaze estimation device based on a multi-scale convolutional neural network and a self-attention network includes: The multi-scale feature extraction module T1 is used to perform parallel feature extraction on the target facial image through a preset multi-scale convolutional network to obtain initial facial features; A cross-feature fusion module T2 is used to perform cross-feature fusion on the initial facial features to obtain multi-scale fusion features; A feature optimization module T3 is used to enhance the multi-scale fusion features through a self-attention network to obtain a target feature map; The gaze analysis module T4 is used to transform the target feature map by a preset angle to obtain a target gaze result.

[0095] Optionally, in this embodiment, the initial facial features include initial face features and initial eye features; the cross-feature fusion module T2 is further used to perform feature cross-fusion on the initial eye features through a preset multi-scale feature fusion module to obtain a first fusion feature; The cross feature fusion module T2 is further used to upsample the initial eye feature to obtain a deformed eye feature; The cross-feature fusion module T2 is further used to perform feature cross-fusion on the deformed eye feature and the initial facial feature through the preset multi-scale feature fusion module to obtain a second fusion feature; The cross-feature fusion module T2 is further used to perform preset feature splicing on the first fusion feature and the second fusion feature to obtain a multi-scale fusion feature.

[0096] Optionally, in this embodiment, the preset multi-scale feature fusion module includes a channel feature module, a first position feature module and a second position feature module; The cross feature fusion module T2 is further used to perform channel feature analysis on the initial eye feature through the channel weight module to obtain channel weight information; The cross feature fusion module T2 is further used to perform spatial feature analysis on the initial eye feature through the first position feature module to obtain first position weight information; The cross feature fusion module T2 is further used to perform spatial feature analysis on the initial eye feature through the second position feature module to obtain second position weight information; The cross-feature fusion module T2 is also used to multiply the initial eye feature, the channel weight information, the first position weight information, and the second position weight information item by item to obtain a first fused feature.

[0097] Optionally, in this embodiment, the feature optimization module T3 is further used to perform channel convolution stacking on the multi-scale fusion features to obtain a multi-channel feature volume; The feature optimization module T3 is further used to perform global attention analysis on the multi-channel feature body through a fusion attention module to obtain optimized fusion features; The feature optimization module T3 is also used to enhance the optimized fusion features through a self-attention network to obtain a target feature map.

[0098] Optionally, in this embodiment, the fusion attention module includes: a channel attention module, a spatial attention module and an activation module; The feature optimization module T3 is further used to perform preset channel processing on the multi-channel feature body through the channel attention module to obtain global channel information; The feature optimization module T3 is further used to perform preset spatial processing on the multi-channel feature body through the spatial attention module to obtain global spatial information; The feature optimization module T3 is further used to multiply the global channel information and the global spatial information element by element and input them into the activation module to obtain a unified attention weight; The feature optimization module T3 is also used to reorganize the features of the multi-channel feature body according to the unified attention weight to obtain optimized fusion features.

[0099] Optionally, in this embodiment, the gaze analysis module T4 is further used to perform preset feature dimensionality reduction on the target feature map to obtain a two-dimensional gaze feature; The gaze analysis module T4 is further used to perform angle dimension transformation on the two-dimensional gaze feature to obtain a target gaze result.

[0100] Optionally, in this embodiment, the backbone network of the preset multi-scale convolutional network is an improved ConvNext-T network including at least two parallel convolution kernels.

[0101] The gaze estimation device based on a multi-scale convolutional neural network and a self-attention network provided in the present application adopts the gaze estimation method based on a multi-scale convolutional neural network and a self-attention network in the above-mentioned embodiment, which can solve the technical problem of gaze estimation based on a multi-scale convolutional neural network and a self-attention network. Compared with the prior art, the beneficial effects of the gaze estimation device based on a multi-scale convolutional neural network and a self-attention network provided in the present application are the same as the beneficial effects of the gaze estimation method based on a multi-scale convolutional neural network and a self-attention network provided in the above-mentioned embodiment, and the other technical features in the gaze estimation device based on a multi-scale convolutional neural network and a self-attention network are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0102] The present application provides a gaze estimation device based on a multi-scale convolutional neural network and a self-attention network. The gaze estimation device based on a multi-scale convolutional neural network and a self-attention network includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the gaze estimation method based on the multi-scale convolutional neural network and the self-attention network in the above-mentioned embodiment one.

[0103] Reference below Fig. 9 , which shows a schematic diagram of the structure of a gaze estimation device based on a multi-scale convolutional neural network and a self-attention network suitable for implementing an embodiment of the present application. The gaze estimation device based on a multi-scale convolutional neural network and a self-attention network in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The gaze estimation device based on the multi-scale convolutional neural network and the self-attention network shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0104] like Fig. 9As shown, the gaze estimation device based on the multi-scale convolutional neural network and the self-attention network may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM: ReadOnly Memory) 1002 or the program loaded from the storage device 1003 to the random access memory (RAM: RandomAccess Memory) 1004. In RAM1004, various programs and data required for the operation of the gaze estimation device based on the multi-scale convolutional neural network and the self-attention network are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other via a bus 1005. The input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the gaze estimation device based on the multi-scale convolutional neural network and the self-attention network to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a gaze estimation device based on a multi-scale convolutional neural network and a self-attention network with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0105] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a gaze estimation program product based on a multi-scale convolutional neural network and a self-attention network, which includes a gaze estimation program based on a multi-scale convolutional neural network and a self-attention network carried on a computer-readable medium, and the gaze estimation program based on the multi-scale convolutional neural network and the self-attention network contains a program code for executing the method shown in the flowchart. In such an embodiment, the gaze estimation program based on the multi-scale convolutional neural network and the self-attention network can be downloaded and installed from the network through a communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the gaze estimation program based on the multi-scale convolutional neural network and the self-attention network is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0106] The gaze estimation device based on a multi-scale convolutional neural network and a self-attention network provided in the present application adopts the gaze estimation method based on a multi-scale convolutional neural network and a self-attention network in the above-mentioned embodiment, which can solve the technical problem of gaze estimation based on a multi-scale convolutional neural network and a self-attention network. Compared with the prior art, the beneficial effects of the gaze estimation device based on a multi-scale convolutional neural network and a self-attention network provided in the present application are the same as the beneficial effects of the gaze estimation method based on a multi-scale convolutional neural network and a self-attention network provided in the above-mentioned embodiment, and the other technical features in the gaze estimation device based on a multi-scale convolutional neural network and a self-attention network are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0107] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0108] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0109] The present application provides a storage medium having computer-readable program instructions stored thereon (i.e., a gaze estimation program based on a multi-scale convolutional neural network and a self-attention network), and the computer-readable program instructions are used to execute the gaze estimation method based on the multi-scale convolutional neural network and the self-attention network in the above-mentioned embodiment.

[0110] The storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: RandomAccess Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0111] The above-mentioned storage medium may be included in a gaze estimation device based on a multi-scale convolutional neural network and a self-attention network; or it may exist independently without being assembled into a gaze estimation device based on a multi-scale convolutional neural network and a self-attention network.

[0112] The above-mentioned storage medium carries one or more programs. When the above-mentioned one or more programs are executed by a gaze estimation device based on a multi-scale convolutional neural network and a self-attention network, the gaze estimation device based on a multi-scale convolutional neural network and a self-attention network enables: gaze estimation based on a multi-scale convolutional neural network and a self-attention network.

[0113] The gaze estimation program code based on multi-scale convolutional neural network and self-attention network for performing the operation of the present application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0114] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the gaze estimation program products based on the systems, methods and multi-scale convolutional neural networks and self-attention networks according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram may represent a module, a program segment or a portion of a code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes may also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified functions or operations, or can be implemented with a combination of dedicated hardware and computer instructions.

[0115] The modules involved in the embodiments of the present application may be implemented by software or hardware, wherein the name of the module does not limit the unit itself in some cases.

[0116] The readable storage medium provided in the present application is a storage medium, which stores computer-readable program instructions for executing the above-mentioned gaze estimation method based on multi-scale convolutional neural networks and self-attention networks (i.e., gaze estimation program based on multi-scale convolutional neural networks and self-attention networks), which can solve the technical problems of gaze estimation based on multi-scale convolutional neural networks and self-attention networks. Compared with the prior art, the beneficial effects of the storage medium provided in the present application are the same as the beneficial effects of the gaze estimation method based on multi-scale convolutional neural networks and self-attention networks provided in the above-mentioned embodiments, and will not be repeated here.

[0117] The above are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A gaze estimation method based on multi-scale convolutional neural network and self-attention network, characterized in that: The method comprises: Perform parallel feature extraction on the target facial image through a preset multi-scale convolutional network to obtain initial facial features; Performing feature cross-fusion on the initial facial features to obtain multi-scale fusion features; Performing feature enhancement on the multi-scale fusion features through a self-attention network to obtain a target feature map; A gaze direction prediction is performed based on the target feature map to obtain a target gaze result.

2. The gaze estimation method according to claim 1, wherein: The initial facial features include initial face features and initial eye features; The step of performing feature cross-fusion on the initial facial features to obtain multi-scale fusion features comprises: Performing feature cross-fusion on the initial eye features through a preset multi-scale feature fusion module to obtain a first fusion feature; Upsampling the initial eye feature to obtain a deformed eye feature; Performing feature cross-fusion on the deformed eye feature and the initial facial feature through the preset multi-scale feature fusion module to obtain a second fusion feature; Preset feature splicing is performed on the first fusion feature and the second fusion feature to obtain a multi-scale fusion feature.

3. The gaze estimation method according to claim 2, wherein: The preset multi-scale feature fusion module includes a channel feature module, a first position feature module, and a second position feature module; The step of performing feature cross-fusion on the initial eye features through a preset multi-scale feature fusion module to obtain a first fusion feature comprises: Performing channel feature analysis on the initial eye feature through the channel weight module to obtain channel weight information; Performing spatial feature analysis on the initial eye feature by the first position feature module to obtain first position weight information; Performing spatial feature analysis on the initial eye feature by the second position feature module to obtain second position weight information; The initial eye feature, the channel weight information, the first position weight information, and the second position weight information are multiplied item by item to obtain a first fusion feature.

4. The gaze estimation method according to claim 1, wherein: The step of performing feature enhancement on the multi-scale fusion feature through a self-attention network to obtain a target feature map includes: Perform channel convolution stacking on the multi-scale fusion features to obtain a multi-channel feature body; Performing global attention analysis on the multi-channel feature body through a fusion attention module to obtain optimized fusion features; The optimized fusion features are enhanced through a self-attention network to obtain a target feature map.

5. The gaze estimation method according to claim 4, characterized in that: The fusion attention module includes: a channel attention module, a spatial attention module and an activation module; The step of performing global attention analysis on the multi-channel feature body through a fusion attention module to obtain optimized fusion features includes: Performing preset channel processing on the multi-channel feature body through the channel attention module to obtain global channel information; Performing preset spatial processing on the multi-channel feature body through the spatial attention module to obtain global spatial information; Multiplying the global channel information and the global spatial information element by element and inputting the resultant information into the activation module to obtain a unified attention weight; The multi-channel feature body is reorganized according to the unified attention weight to obtain optimized fusion features.

6. The gaze estimation method according to claim 1, wherein: The step of performing gaze direction prediction based on the target feature map to obtain a target gaze result comprises: Performing preset feature dimensionality reduction on the target feature map to obtain a two-dimensional gaze feature; The two-dimensional gaze feature is transformed in angle dimension to obtain a target gaze result.

7. The gaze estimation method according to claim 1, wherein: The backbone network of the preset multi-scale convolutional network is an improved ConvNext-T network including at least two parallel convolution kernels.

8. A gaze estimation device based on a multi-scale convolutional neural network and a self-attention network, characterized in that: The device comprises: A multi-scale feature extraction module is used to perform parallel feature extraction on the target facial image through a preset multi-scale convolutional network to obtain initial facial features; A cross-feature fusion module, used for performing cross-feature fusion on the initial facial features to obtain multi-scale fusion features; A feature optimization module, used to enhance the multi-scale fusion features through a self-attention network to obtain a target feature map; The gaze analysis module is used to transform the target feature map at a preset angle to obtain a target gaze result.

9. A gaze estimation device based on a multi-scale convolutional neural network and a self-attention network, characterized in that: The device includes: a memory, a processor, and a gaze estimation program based on a multi-scale convolutional neural network and a self-attention network stored in the memory and executable on the processor, wherein the gaze estimation program based on the multi-scale convolutional neural network and the self-attention network is configured to implement the steps of the gaze estimation method based on a multi-scale convolutional neural network and a self-attention network as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a gaze estimation program based on a multi-scale convolutional neural network and a self-attention network. When the gaze estimation program based on a multi-scale convolutional neural network and a self-attention network is executed by a processor, the steps of the gaze estimation method based on a multi-scale convolutional neural network and a self-attention network as described in any one of claims 1 to 7 are implemented.

Images