Line of sight estimation method

Abstract

The present application relates to the field of computer vision technology, and provides a line of sight estimation method, which obtains an image to be detected; reconstructs a line of sight estimation network, and the line of sight estimation network includes a first module, a second module, a third module and an attention module; performs downsampling and feature extraction on the image to be detected through the first module to output a first feature map; performs feature extraction on the first feature map through the second module to output a second feature map; then performs downsampling and feature extraction on the second feature map through the third module to output a second feature map; iterates the second feature map through the second module and the third module until the second module outputs a third feature map with a third number of channels; inputs the third feature map into the attention module to output a fourth feature map; and inputs the fourth feature map into the classification head to output a line of sight direction category that characterizes the pitch angle category and the yaw angle category. The present application improves the perception of spatial information and thus improves the accuracy of line of sight estimation by reconstructing the line of sight estimation network.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a line of sight estimation method. Background Art

[0002] Gaze estimation can understand the user's intention by estimating the user's gaze direction for application in different scenarios. Gaze estimation can be implemented in a variety of ways, such as geometry-based methods, regression-based methods, classification-based methods, appearance-based deep convolutional network algorithms, etc.

[0003] The appearance-based deep convolutional network algorithm for line of sight estimation uses a deep learning model to extract features from the appearance image of the eye and learn the mapping relationship between these features and the line of sight direction. The appearance-based deep convolutional network algorithm is more convenient than other methods, has low hardware cost, and is applicable to a wider range of environments.

[0004] However, the deep convolutional network model has high computational complexity and requires more computing resources, as well as factors such as head posture, lighting conditions, and individual differences, resulting in low line of sight estimation accuracy. Summary of the invention

[0005] The present application provides a line of sight estimation method to solve the problem of low line of sight estimation accuracy.

[0006] The present application provides a line of sight estimation method, comprising:

[0007] Acquire an image to be detected, wherein the image to be detected at least includes an image of a monocular area;

[0008] Constructing a sight line estimation network, wherein the sight line estimation network includes a first module, a second module, a third module and an attention module;

[0009] Downsampling the image to be detected by the first module to output a first feature map with a first number of channels;

[0010] Performing feature extraction on the first feature map by the second module to output a second feature map;

[0011] Downsampling the second feature map by the third module to output a second feature map with a second number of channels;

[0012] Iterate the second feature map through the second module and the third module until the second module outputs a third feature map with a third number of channels;

[0013] Inputting the third feature map into an attention module to output a fourth feature map;

[0014] The fourth feature map is input to a classification head to output a gaze direction category, where the gaze direction category is a pitch angle category and a yaw angle category.

[0015] In some feasible embodiments, the first module includes a first convolution kernel and a first normalization layer;

[0016] The downsampling of the image to be detected by the first module to output a first feature map with a first number of channels includes:

[0017] Performing feature extraction and block downsampling on the image to be detected by using the first convolution kernel to output a first extracted feature map, wherein the number of pixels moved when the convolution kernel slides on the image to be detected is 4, and the number of pixel rows and columns added at the boundary of the image to be detected is 0;

[0018] The first extracted feature map is input into the first normalization layer to output a first feature map with a first number of channels.

[0019] In some feasible embodiments, the second module includes a partial convolution layer and a convolution block;

[0020] The partial convolution layer includes a second convolution kernel;

[0021] The performing feature extraction on the first feature map by the second module to output a second feature map includes:

[0022] Inputting the first feature map into a partial convolutional layer to obtain a first intermediate feature map;

[0023] Inputting the first intermediate feature map into a convolution block in sequence to obtain a second intermediate feature map, the convolution block comprising a first convolution layer, a second normalization layer, an activation function, and a second convolution layer, wherein the second normalization layer and the activation function are between the first convolution layer and the second convolution layer;

[0024] Perform an element-by-element addition operation on the first feature map and the second intermediate feature map at corresponding positions to output a second feature map.

[0025] In some feasible embodiments, the step of inputting the first feature map into a partial convolutional layer to obtain a first intermediate feature map includes:

[0026] Performing a splitting operation on the first feature map to obtain a first group of feature maps and a second group of feature maps, wherein the number of channels of the first group of feature maps is one quarter of the number of channels of the first feature map, and the number of channels of the second group of feature maps is three quarters of the number of channels of the first feature map;

[0027] Input the first set of feature maps to the second convolution kernel to output a first convolution result, wherein the number of pixels moved by the second convolution kernel when sliding on the first set of feature maps is 1, and 1 row and 1 column of pixels are added to the four boundaries of the first set of feature maps;

[0028] The first convolution result is concatenated with the second set of feature maps in the channel dimension to obtain a first intermediate feature map.

[0029] In some feasible embodiments, the third module includes a third convolution kernel and a third normalization layer;

[0030] The downsampling of the second feature map by the third module to output a second feature map with a second number of channels includes:

[0031] Performing feature extraction and block downsampling on the second feature map through the third convolution kernel to output a second extracted feature map, wherein the number of pixels moved when the convolution kernel slides on the second extracted feature map is 2, and the number of pixel rows and columns added at the boundary of the second extracted feature map is 0;

[0032] The second extracted feature map is input into the third normalization layer to output a second feature map with a second number of channels.

[0033] In some feasible embodiments, inputting the third feature map into an attention module to output a fourth feature map includes:

[0034] Based on the number of channels, performing a grouping operation on the third feature map having a third number of channels to output a third intermediate feature map, wherein the number of channels of the third intermediate feature map is the product of the number of channels of the third feature map and the inverse of the number of groups, and the batch size of the third intermediate feature map is a batch size times the number of groups of the third feature map;

[0035] Compressing the horizontal spatial direction features of the third intermediate feature map using a first global average pooling layer to generate a first feature vector, where the first feature vector is a feature vector in a vertical direction;

[0036] Compressing the vertical spatial direction features of the third intermediate feature map using a second global average pooling layer to generate a second feature vector, where the second feature vector is a feature vector in the horizontal direction;

[0037] concatenating the first feature vector and the second feature vector to output a two-dimensional feature vector;

[0038] Using the first convolution layer to perform a channel-wise convolution operation on the two-dimensional feature vector to output a third extracted feature map and a fourth extracted feature map, where the height of the third extracted feature map is 1 and the width of the fourth extracted feature map is 1;

[0039] Inputting the third extracted feature map and the fourth extracted feature map into a logic function to output a first weighted representation and a second weighted representation through the logic function;

[0040] Based on the first weight representation, the second weight representation and the third intermediate feature map, a reweighting strategy is used to reweight to output a weighted feature map.

[0041] In some feasible embodiments, inputting the third feature map into an attention module to output a fourth feature map includes:

[0042] Extracting features of the third intermediate feature map using a second convolution kernel to output a fifth extracted feature map, wherein the fifth extracted feature map includes local context information;

[0043] Performing an average pooling operation on the fifth extracted feature map to output a pooled feature map, wherein the height and width of the pooled feature map are 1;

[0044] Based on the pooled feature map, a first channel descriptor is obtained through a normalization operation;

[0045] Performing a group normalization operation on the weighted feature map, and outputting a fourth intermediate feature map;

[0046] The fourth intermediate feature map is multiplied by the first channel descriptor to output a first relationship vector.

[0047] In some feasible embodiments, the step of inputting the third feature map into an attention module to output a fourth feature map further includes:

[0048] Performing an average pooling operation and a normalization operation on the fourth intermediate feature map to obtain a second channel descriptor;

[0049] multiplying the fifth extracted feature map by the second channel descriptor to output a second relationship vector;

[0050] Performing element-by-element addition of the first relationship vector and the second relationship vector at corresponding positions to output a fifth intermediate feature map;

[0051] Inputting the fifth intermediate feature map into a logic function to output a third weight representation through the logic function;

[0052] Based on the third weight representation and the third intermediate feature map, a reweighting strategy and merging and grouping are utilized to output a fourth feature map, where the number of channels and the batch size of the fourth feature map are the same as those of the third feature map.

[0053] In some feasible embodiments, the classification head includes a global pooling layer, a first convolutional layer, an activation function, and two fully connected layers;

[0054] Inputting the fourth feature map into a classification head to output a gaze direction category includes:

[0055] After inputting the fourth feature map into the global pooling layer, the fourth feature map is input into the first convolutional layer to output a sixth intermediate feature map;

[0056] After the sixth intermediate feature map is input into the activation function, it is input into the first fully connected layer and the second fully connected layer respectively, so as to output the yaw angle category through the first fully connected layer and the pitch angle category through the second fully connected layer.

[0057] In some feasible embodiments, the method further includes:

[0058] Using a normalization layer to convert the gaze direction category into an expected value of a probability distribution;

[0059] Constructing a composite loss function through the expected value, cross entropy loss and smoothing loss;

[0060] The line of sight estimation network is trained using the composite loss function.

[0061] It can be seen from the above technical solutions that the line of sight estimation method provided by the present application is to obtain an image to be detected, wherein the image to be detected is a face image; then construct a line of sight estimation network, wherein the line of sight estimation network includes a first module, a second module, a third module and an attention module; the first module performs downsampling on the image to be detected to output a first feature map of the first channel quantity; the second module performs feature extraction on the first feature map to output a second feature map; then the third module performs downsampling on the second feature map to output a second feature map of the second channel quantity; iterates the second feature map through the second module and the third module until the second module outputs a third feature map of the third channel quantity; inputs the third feature map into the attention module to output a fourth feature map; and inputs the fourth feature map into the classification head to output a line of sight direction category, wherein the line of sight direction category is a pitch angle category and a yaw angle category. The method provided by the present application improves the network's perception of spatial information and improves the accuracy of line of sight estimation by reconstructing the line of sight estimation network and the attention module. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0063] Figure 1 A flowchart of a line of sight estimation method provided in an embodiment of the present application;

[0064] Figure 2A schematic diagram of a line of sight estimation network structure provided in an embodiment of the present application;

[0065] Figure 3 A schematic diagram of the structure of the first module provided in an embodiment of the present application;

[0066] Figure 4 A schematic diagram of the structure of the second module provided in an embodiment of the present application;

[0067] Figure 5 A schematic diagram of the structure of a partial convolutional layer provided in an embodiment of the present application;

[0068] Figure 6 A schematic diagram of the structure of the third module provided in the embodiment of the present application;

[0069] Figure 7 A schematic diagram of the structure of the attention module provided in the embodiment of the present application;

[0070] Figure 8 A schematic diagram of the structure of a classification head provided in an embodiment of the present application;

[0071] Fig. 9 A schematic diagram of the execution flow of the attention module provided in an embodiment of the present application. DETAILED DESCRIPTION

[0072] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following embodiments do not represent all embodiments consistent with the present application.

[0073] 3D line of sight estimation can be divided into two categories: the first category is the line of sight tracking algorithm based on three-dimensional models (Model-based), and the second category is the line of sight tracking algorithm based on two-dimensional appearance (Appearance-based).

[0074] 3D line of sight estimation technology can be applied to a variety of application scenarios, such as medical treatment, human-computer interaction, VR / AR, assisted driving, etc. For example, in the medical field, it can be used to observe the user's eye movements; in the field of human-computer interaction, it can replace the keyboard and mouse input method; in the field of VR / AR, it can optimize the rendering effect according to the user's line of sight; in the field of assisted driving, it can monitor the driver's attention state to make the driving process safer.

[0075] 3D gaze estimation can accurately estimate the user's gaze direction, thereby providing a more accurate interactive experience. Compared with 2D gaze estimation, 3D gaze estimation considers the three-dimensional structure and physical properties of the eyeball, and can reduce errors caused by changes in head posture.

[0076] The appearance-based deep convolutional network algorithm for line of sight estimation uses a deep learning model to extract features from the appearance image of the eye and learn the mapping relationship between these features and the line of sight direction. The appearance-based deep convolutional network algorithm is more convenient than other methods, has low hardware cost, and is applicable to a wider range of environments. However, the deep convolutional network model has a high computational complexity and requires more computing resources, as well as factors such as head posture, lighting conditions, and individual differences, resulting in low line of sight estimation accuracy.

[0077] In some embodiments, to solve the above problems, the accuracy of line of sight estimation can be improved by designing a lightweight network structure, utilizing transfer learning and pre-training, expanding the data set, fusing head posture information, transfer learning and domain adaptation, etc.

[0078] Among them, adopting a shallower network structure or a streamlined convolution kernel size to reduce the number of parameters and calculations, and introducing compression techniques such as pruning and quantization can reduce the complexity of the model. Fine-tuning the network model pre-trained on a large amount of data can reduce the training requirements of the target dataset and accelerate the convergence process. Collecting more diverse training data, including face images under different head postures, lighting conditions, and individual differences, can enhance the generalization ability of the model. Introducing head posture information as an auxiliary input in the model, the accuracy of line of sight estimation can be improved through multimodal fusion technology. Use transfer learning technology to migrate the model from one dataset to another to improve the performance of the model in the new environment. Use domain adaptation technology to reduce the impact of distribution differences between different datasets on model performance.

[0079] In an embodiment of the present application, a line of sight estimation method is provided, which reconstructs the line of sight estimation network, reduces network parameters, improves parameter reuse rate and feature extraction capability of the network, and adds an attention module between the second module and the classification head to further enhance the network's perception of spatial information, improve network accuracy, and thus improve line of sight estimation accuracy.

[0080] like Figure 1 As shown, the line of sight estimation method provided by the present application comprises the following steps:

[0081] S100: Acquire an image to be detected.

[0082] The image to be detected may be an eye image, such as a monocular image or a binocular image. The monocular image estimates the line of sight by analyzing the features of a single eye, such as the pupil position, the eye corner position, etc. The binocular image can provide more information, such as the coordination of the two eyes, parallax, etc., which helps to improve the accuracy and robustness of line of sight estimation.

[0083] The image to be detected may also be a face image, i.e., a facial image. In addition to the eye image, the face image also contains other facial parts, such as the relative positions and postures of the eyes, eyebrows, nose, mouth, etc., and the relationship between them and the line of sight direction, to more accurately estimate the line of sight direction.

[0084] Among them, the eye image or the face image can come from continuous frames in the video stream. For example, in the process of virtual reality, human-computer interaction, etc., by tracking the user's line of sight, the video frame when interactive response is performed as needed.

[0085] The image to be detected can also be a synthetic image, which is generated using computer graphics technology to train the gaze estimation model. These synthetic images can simulate the gaze direction of different human eyes, different lighting conditions, and different head postures.

[0086] In order to improve the accuracy of line of sight estimation, the image to be detected can be processed in a variety of ways. In some embodiments, after acquiring the image to be detected, the image to be detected is preprocessed and data enhanced, and the preprocessing includes normalization, cropping, and flipping. Among them, normalization can process the image to be detected on a uniform scale, reducing the impact of differences in image size, lighting, etc. on the estimation results. For example, the camera matrix and annotation information can be used to calculate the normalization parameters, and the image to be detected can be transformed accordingly. Cropping can remove the background part of the image to be detected that is not related to the line of sight estimation, reduce the amount of calculation and improve the pertinence of the estimation. For example, according to the coordinate information of the key points of the eyes, the image areas of the left eye and the right eye are cropped.

[0087] The data enhancement includes transforming the original image by flipping, rotating, scaling, adding noise, etc. Among them, transforming the image to be detected by flipping, rotating, scaling, adding noise, etc., generating more training samples, helps the model learn more feature representations and improves generalization ability.

[0088] The image to be detected can be a color image or a grayscale image, wherein the number of channels of a color image is 3, namely, three color channels of red (R), green (G), and blue (B); the number of channels of a grayscale image is 1, namely, a grayscale value. Since color images contain richer information, they are helpful for more accurate estimation of the line of sight direction. In this embodiment, the image to be detected is a color image as an example.

[0089] S200: constructing a line of sight estimation network.

[0090] like Figure 2As shown in the figure, the sight estimation network includes a first module, a second module, a third module and an attention module. The image to be detected first passes through the first module for downsampling to obtain a feature map of the first channel number, then passes through the second module for feature extraction, and then passes through the third module for downsampling again to obtain a feature map of the second channel number. The process of the second and third modules is iterated until a feature map of the third channel number is obtained, and then passes through the second module for feature extraction, and then passes through the attention module (EMA), which can effectively establish the dependency between short-range and long-range features, thereby improving the performance of the network, and finally passes through the classification head to realize the classification estimation of 3D sight lines.

[0091] Figure 2 In the figure, the number 0 in the upper right box of the first module indicates that the first step passes through the first module, and the number 4 in the lower right box indicates 4-fold downsampling; the number 1 in the upper right box of the second module indicates that the second step passes through the second module, and the number 1 in the lower right box indicates that the number of second modules is 1; the number 2 in the upper right box of the third module indicates that the third step passes through the third module, and the number 8 in the lower right box indicates 8-fold downsampling; the number 3 in the upper right box of the second module indicates that the fourth step passes through the second module, and the number 2 in the lower right box indicates that the number of second modules is 2; the number 4 in the upper right box of the third module indicates that the fifth step passes through the third module, and the number 4 in the lower ... The number 16 in the box indicates 16-fold downsampling; the number 5 in the box in the upper right corner of the second module indicates that the sixth step passes through the second module, and the number 13 in the box in the lower right corner indicates that the number of second modules is 13; the number 6 in the box in the upper right corner of the third module indicates that the seventh step passes through the third module, and the number 32 in the box in the lower right corner indicates 32-fold downsampling; the number 7 in the box in the upper right corner of the second module indicates that the eighth step passes through the second module, and the number 2 in the box in the lower right corner indicates that the number of second modules is 2; the number 8 in the box of the EMA attention module indicates that the ninth step passes through the EMA attention module; the number 9 in the box of the classification head indicates that the tenth step passes through the classification head.

[0092] Among them, Figure 3 As shown, the first module serves as the beginning of the line of sight estimation network. The batch size of the image to be detected is represented by B, the height is represented by H, and the width is represented by W. Taking the image to be detected as a color image as an example, the number of channels is 3, and the size of the input to the first module is [B, 3, H, W].

[0093] In some embodiments, the first module includes a first convolution kernel and a first normalization layer, the size of the first convolution kernel is 4×4, the step length of the first convolution kernel in performing the convolution operation is 4, and the padding is 0, wherein the step length is the number of pixels moved each time when the convolution kernel slides on the image to be detected; the padding is to add additional pixel rows and columns around the boundary of the image to be detected so that the size of the output feature map can be maintained or its size can be changed in a specific way when performing the convolution operation. The larger the step length, the smaller the size of the output feature map. In this embodiment, the convolution kernel skips 4 pixels each time it moves.

[0094] In this embodiment, the number of pixel rows and columns added in the width and height directions of the image is 0, that is, no additional pixel rows and columns are added around the boundary of the image to be detected.

[0095] The normalization layer can accelerate convergence, improve model stability, prevent gradient disappearance or explosion, and regularize. The normalization layer normalizes the distribution of feature maps so that the input of each layer is kept within a relatively stable distribution range, which helps to speed up the convergence of the network. It can also reduce the sensitivity of the model to initialization parameters and learning rate, making the model more stable, that is, under different initialization conditions, improve the consistency of the model's performance. In the line of sight estimation network, due to the effect of the chain rule, the gradient may gradually disappear or explode during the back propagation process. The normalization layer normalizes the distribution of feature maps to make the gradient more stable. The normalization layer can reduce the model's dependence on specific samples, thereby preventing overfitting to a certain extent. In this embodiment, the first normalization layer, the second normalization layer, and the third normalization layer all play the above role, which will not be repeated in the subsequent description.

[0096] like Figure 4 , Figure 5 As shown, in some embodiments, the second module includes a partial convolution layer and a convolution block, wherein the partial convolution layer includes a second convolution kernel, the convolution block includes a first convolution layer, a second normalization layer, an activation function and a second convolution layer, and the second normalization layer and the activation function are between the first convolution layer and the second convolution layer.

[0097] The size of the second convolution kernel is 3×3. The stride of the second convolution kernel in performing the convolution operation is 1 and the padding is 1. In this embodiment, the convolution kernel moves one pixel position to the right or downward after each operation, and adds 1 row and 1 column of pixels around the four boundaries of the input data.

[0098] like Figure 4As shown, in some embodiments, the first convolution layer and the second convolution layer are both convolution kernels of size 1×1, and the step size is 1, and the padding is 0. A second normalization layer and a ReLU activation function are provided in the first convolution layer and the second convolution layer, and the second normalization layer and the ReLU activation function can maintain the diversity of features and the nonlinear fitting ability of the network, and can obtain lower latency.

[0099] like Figure 6 As shown, in some embodiments, the third module is used to perform downsampling on the output, including a third convolution kernel and a third normalization layer; wherein the third convolution kernel is a convolution kernel of size 2×2, and the step length is 2, and the padding is 0, and the convolution kernel moves 2 pixel positions to the right or downward after each operation, and no additional pixel rows and columns are added around the boundary of the input feature map. The third normalization layer is similar to the first normalization layer, and is used to standardize the output of the third convolution kernel.

[0100] S300: Downsampling the image to be detected by a first module to output a first feature map with a first number of channels.

[0101] In some embodiments, feature extraction and block downsampling are performed on the image to be detected through a first convolution kernel to output a first extracted feature map, the number of pixels moved when the convolution kernel slides on the image to be detected is 4, and the number of pixel rows and columns added at the boundary of the first extracted feature map is 0; the first extracted feature map passes through a first normalization layer to output a first feature map with a first number of channels.

[0102] The first convolution kernel generates an output feature map by sliding a 4×4 convolution kernel on the image to be detected and calculating the dot product of the pixels at the corresponding position. The first extracted feature map can capture different features of the image to be detected. In a monocular image, it can capture the boundaries of the eye, the texture features of the eye, the light and dark changes of the eye, etc.; in a face image, it can also capture the boundaries of facial features such as eyebrows, nose, and mouth; and the texture features of skin, hair, and other parts.

[0103] Pooling layers can be set in the first convolution kernel. Pooling layers are used to downsample feature maps. Pooling layers reduce the dimensions (height and width) of feature maps by selecting the maximum value (max pooling), average value (average pooling), or other statistics in a local area, such as a 2×2 or 3×3 window, while keeping the depth (i.e. the number of channels) unchanged. Downsampling operations help reduce the amount of computation, control overfitting, and make the model more robust to small changes in input data.

[0104] After the first convolution kernel with a step size of 4 and padding of 0, 4 times downsampling is performed to expand the number of channels of the image to be detected to 128. The execution process of the first module can facilitate the line of sight estimation network to deeply learn the feature representation of the data.

[0105] S400: Perform feature extraction on the first feature map through the second module to output a second feature map.

[0106] In some embodiments, the first feature map is input into a partial convolution layer to obtain a first intermediate feature map; the first intermediate feature map is then sequentially input into the convolution block to obtain a second intermediate feature map, and an element-by-element addition (add) operation is performed on the first feature map and the second intermediate feature map at corresponding positions to output the second feature map.

[0107] The processing of the partial convolution layer first performs a split operation on the first feature map to obtain the first set of feature maps and the second set of feature maps, and then inputs the first set of feature maps into the second convolution kernel to output the first convolution result. The number of pixels moved by the second convolution kernel when sliding on the first set of feature maps is 1, and 1 row and 1 column of pixels are added to each of the four boundaries of the first set of feature maps. The first convolution result and the second feature map are concatenated in the channel dimension to obtain the first intermediate feature map.

[0108] Specifically, the input first feature map is split into two groups, wherein the number of channels of the first group of feature maps is one-fourth of the number of channels of the first feature map, and the number of channels of the second group of feature maps is three-fourths of the number of channels of the first feature map. For example, the number of channels of the first feature map is C, and the number of channels of the first group of feature maps is C / 4, 3C / 4; then the second convolution kernel is used to perform a convolution operation on the first group of feature maps. The second convolution kernel slides on the first group of feature maps with a step size of 1. After the convolution operation is completed, the first convolution result, i.e., the feature map processed by the convolution, is obtained. The feature map processed by the convolution is spliced ​​with the feature map that has not been directly processed by the convolution, i.e., the second group of feature maps, so that the information from these two different sources can be used in the subsequent layers.

[0109] concat is used to connect two or more tensors together in a certain dimension. In this embodiment, it is used to increase the number of channels of the feature. For example, in the line of sight estimation network, the feature map processed by convolution and the feature map that has not been directly processed by the convolution need to be merged. The concat operation is used to splice them together. The first intermediate feature map after splicing will have more channels, which helps the network capture richer feature information.

[0110] The partial convolution layer applies convolution operations to some channels of the first feature map while keeping other channels unchanged. This operation can effectively reduce computational redundancy and improve computational efficiency.

[0111] S500: Down-sampling the second feature map through a third module to output a second feature map with a second number of channels.

[0112] In some embodiments, feature extraction and block downsampling are performed on the second feature map through a third convolution kernel to output a second extracted feature map, the number of pixels moved when the convolution kernel slides on the second extracted feature map is 2, and the number of pixel rows and columns added at the boundary of the second extracted feature map is 0; the second extracted feature map passes through a third normalization layer to output a second feature map with a second number of channels.

[0113] The third convolution kernel slides a 2×2 convolution kernel, and the second feature map further abstracts and combines low-level features through feature extraction to form a higher-level feature representation. High-level features include the shape, size, relative position of facial features, and the spatial relationship between them. In addition, during the downsampling process, the redundant information between adjacent pixels or features is reduced, and the information density of the feature map is improved, which can reduce the amount of calculation and avoid overfitting. Due to the effects of downsampling and feature extraction, the second feature map is more robust to noise and interference in the image, such as illumination changes, occlusion, expression changes, etc., making the model more stable and reliable when performing line of sight estimation in complex environments.

[0114] For the second feature map, features related to the line of sight are more highlighted, such as the geometric shape of the eyes, the position and size of the pupils, the degree of opening and closing of the eyelids, etc.

[0115] After the third convolution kernel with a step size of 2 and padding of 0, 8 times downsampling is performed to expand the number of channels of the image to be detected to 256.

[0116] S600: Iterate the second feature map through the second module and the third module until the second module outputs a third feature map with a third number of channels.

[0117] like Figure 2As shown, for the first module, the second module, the third module, the EMA attention module and the classification head, 0-9 represent the order of steps. First, the first module performs 4-fold downsampling with a channel number of 128, wherein 4 after the first module represents 4-fold downsampling, and passes through a second module, wherein 1 after the second module represents that the number of second modules is 1, the third module performs 8-fold downsampling with a channel number of 256, and 8 after the third module represents 8-fold downsampling, and passes through two second modules, wherein 2 after the second module represents that the number of second modules is 2, and then passes through a third module to perform 16-fold downsampling with a channel number of 512, and 16 after the third module represents 16-fold downsampling, and then passes through thirteen second modules, wherein 13 after the second module represents that the number of second modules is 13, and then passes through a third module to perform 32-fold downsampling, and 32 after the third module represents 32-fold downsampling, and the third module outputs a feature map with a channel number of 1024, and then passes through two second modules to perform feature extraction, and 2 after the second module represents that the number of second modules is 2, and the second module also outputs a feature map with a channel number of 1024, i.e., the third feature map.

[0118] The increase in the number of channels indicates that the gaze estimation network can capture more types of features. The change in the number of channels from 256 to 512 and then to 1024 indicates that the gaze estimation network is gradually building more complex feature representations. By using the second module multiple times, the gaze estimation network can gradually extract higher-level and more abstract features from the image to be detected. High-level features are more helpful in understanding the overall content and context of the image to be detected.

[0119] S700: Input the third feature map into the attention module to output a fourth feature map.

[0120] The attention module can retain the information on each channel while reducing the computational overhead, reconstruct part of the channels into batch dimensions, and group the channel dimensions into multiple sub-features so that the spatial semantic features are evenly distributed within each feature group. In addition to encoding global information to recalibrate the channel weights in each parallel branch, the output features of the two parallel branches are further aggregated through cross-spatial interactions to capture pixel-level pairwise relationships and effectively establish dependencies between short-range and long-range features, thereby improving the performance of the network.

[0121] like Figure 7 , Fig. 9 As shown, the execution process of the attention module is as follows:

[0122] S910: Based on the number of channels, perform a grouping operation on the third feature map of the third number of channels to output a third intermediate feature map.

[0123] The number of channels of the third intermediate feature map is the product of the number of channels of the third feature map and the inverse of the number of groups, and the batch size of the third intermediate feature map is the batch size times the number of groups of the third feature map. For example, the batch size of the third feature map is B, the number of channels is C, the height is H, and the width is W. After grouping, the number of channels is changed to C / / G, and the batch size is changed to B×G, wherein the batch size of the third intermediate feature map is B×G, the number of channels is C / / G, the height is H, and the width is W.

[0124] S920: compress the horizontal spatial direction features of the third intermediate feature map using a first global average pooling layer (X Avg pool) to generate a first feature vector with a height of 1.

[0125] The first global average pooling layer compresses the horizontal spatial direction features, divides the third intermediate feature map into multiple horizontal strips in the vertical direction (Y axis), and averages all elements in each horizontal strip to obtain a value representing the global information of the strip. That is, the third intermediate feature map is compressed into a one-dimensional vector (feature vector) in the vertical direction, in which each element represents the global information of a horizontal strip. In other words, the output first feature vector is a feature vector in the horizontal direction.

[0126] S930: compress the vertical spatial direction features of the third intermediate feature map using a second global average pooling layer (Y Avg pool) to generate a second feature vector with a width of 1.

[0127] The second global average pooling layer compresses the vertical spatial features and divides the feature map into multiple vertical strips in the horizontal direction (X axis). All elements in each vertical strip are averaged to obtain a value representing the global information of the strip. That is, the third intermediate feature map is compressed into a one-dimensional vector in the horizontal direction, where each element represents the global information of a vertical strip, that is, the output second feature vector is a feature vector in the vertical direction.

[0128] S940: Concatenate the first feature vector and the second feature vector to output a two-dimensional feature vector.

[0129] After the two one-dimensional feature vectors obtained by Y Avg pool and X Avg pool are concatenated, the concatenated feature vector contains the global information of the feature map in both horizontal and vertical directions.

[0130] S950: Use the first convolutional layer to perform a channel-wise convolution operation on the two-dimensional feature vector to output a third extracted feature map and a fourth extracted feature map.

[0131] A convolution operation is performed in the width direction, and a third extracted feature map with a height of 1 is generated using the first convolution layer, which helps to combine multiple feature information. A convolution operation is performed in the height direction, and a fourth extracted feature map with a width of 1 is generated using the first convolution layer.

[0132] S960: Input the third extracted feature map and the fourth extracted feature map into a sigmoid function respectively, so as to output a first weight representation and a second weight representation through the sigmoid function.

[0133] The sigmoid function is applied to perform nonlinear transformation on the third extracted feature map and the fourth extracted feature map, so that the network can learn complex mapping relationships.

[0134] S970: Based on the first weight representation, the second weight representation and the third intermediate feature map, re-weight (re-weight) using a re-weighting strategy to output a weighted feature map.

[0135] By re-weighting, important features can be enhanced, such as pupil and iris features, eye corner features, facial posture features, lighting and reflection features, eye movement features, etc., which helps to estimate the gaze direction more accurately.

[0136] Group normalization is a normalization technique used to stabilize deep network training. In line of sight estimation, since the image to be detected may contain different lighting conditions, facial postures, and occlusions, group normalization can help the network better adapt to these changes and improve the generalization ability of the model. To improve the accuracy of line of sight estimation, in some embodiments, a group normalization operation is performed on the weighted feature map to output a fourth intermediate feature map; the features that are more important for line of sight estimation are selected through the above-mentioned re-weighting strategy; and the training process of these features is stabilized through group normalization to reduce the impact of data distribution changes on model performance. The model can more accurately extract information related to the line of sight direction, thereby improving the accuracy of line of sight estimation.

[0137] For the third intermediate feature map, the second convolution kernel is used to extract the features of the third intermediate feature map, and a fifth extracted feature map is output, wherein the fifth extracted feature map includes local context information; then an average pooling operation is performed on the fifth extracted feature map, and a pooled feature map is output, wherein the height and width of the pooled feature map are 1; based on the pooled feature map, a first channel descriptor (Channel Descriptor) is obtained through a normalization operation, and the channel descriptor is used to convert multiple output feature maps (or called "scores" or "logits") into a probability distribution, wherein the probability value of each channel is between 0 and 1, and the sum of the probabilities of all channels is 1.

[0138] The fourth intermediate feature map is then multiplied by the first channel descriptor to output a first relationship vector. The first relationship vector is a fusion of the feature information of the fourth intermediate feature map and the feature information of the first channel descriptor. The model can more accurately extract information related to the line of sight direction from the image to be detected, so that the line of sight estimation network can more accurately predict the user's line of sight direction.

[0139] In the field of computer vision, features of different scales contain information at different levels. The shallower feature maps contain more low-level features and local information, while the deeper feature maps contain richer semantics and global information. In order to aggregate the spatial position information in two scale spaces, in some embodiments, the fourth intermediate feature map is subjected to average pooling and normalization operations to obtain a second channel descriptor; the fifth extracted feature map is multiplied by the second channel descriptor to output a second relationship vector.

[0140] By performing an average pooling operation on the fourth intermediate feature map, its resolution can be effectively reduced while retaining important spatial position information; average pooling can reduce the redundancy of feature maps and enhance the robustness of the model to local feature changes; normalization operations help adjust the numerical range of feature maps to the same scale, so that features at different levels and scales can be better fused and compared. The second channel descriptor contains spatial position information and semantic information, and interacts and fuses with the fifth extracted feature map to enhance the representation ability of the feature map to better capture feature information related to the line of sight direction.

[0141] In some embodiments, the first relationship vector and the second relationship vector are added element by element at corresponding positions to output a fifth intermediate feature map; the fifth intermediate feature map is input into a logic function to output a third weight representation through the logic function; based on the third weight representation and the third intermediate feature map, a reweighting strategy and merging groups are used to output a fourth feature map, wherein the number of channels and batch size of the fourth feature map are the same as those of the third feature map. Fusion of the first relationship vector and the second relationship vector through the add operation is information integration at the feature level, and this fusion method enables the network to simultaneously utilize useful information in the two relationship vectors in subsequent processing, thereby improving the performance of the network.

[0142] Among them, the add operation can enhance the useful information in the feature map to a certain extent, while suppressing noise and irrelevant information, so that the output fifth intermediate feature map has higher robustness. In other words, even if there is some interference or noise in the image to be detected, the line of sight estimation network provided by the present application can accurately estimate the user's line of sight direction.

[0143] In some embodiments, the first relationship vector and the second relationship vector may be concatenated using a concatenation operation. However, the add operation is more efficient in computation and does not require additional parameters and computational resources for concatenating or fusing features, so the add operation can improve the efficiency of network estimation.

[0144] Finally, the weight representation is generated through sigmoid, the input is calibrated again through spatial weights, and the fourth feature map is output.

[0145] S800: Input the fourth feature map to a classification head to output a gaze direction category.

[0146] like Figure 8 As shown, the classification head includes a global pooling layer, a first convolutional layer, an activation (SiLU) function and two fully connected layers, for example, the two fully connected layers are a first fully connected layer and a second fully connected layer; in some embodiments, after the fourth feature map is input into the global pooling layer, it is input into the first convolutional layer to output a sixth intermediate feature map; after the sixth intermediate feature map is input into the activation function, it is input into the first fully connected layer and the second fully connected layer, respectively, to output the yaw angle category through the first fully connected layer and to output the pitch angle category through the second fully connected layer.

[0147] The global pooling layer is used to perform a global average pooling operation to globally average the fourth feature map of each channel, thereby reducing the spatial dimension while retaining the channel information. The first convolutional layer extracts more features through convolution operations, and the SiLU function helps introduce nonlinearity, enabling the network to learn more complex patterns; the predictions of the yaw and pitch angles are output through two fully connected layers, that is, the direction category results.

[0148] Considering the difficulty of directly regressing the 3D gaze direction of the human eye, the 3D line of sight is estimated from coarse to fine. In some embodiments, according to the direction category results of the yaw angle and the pitch angle, the network output is converted into the expected value of the probability distribution using softmax to obtain fine-grained line of sight prediction, and then a composite loss function Loss is constructed through the expected value, cross entropy loss (CE) and smooth L1 loss (Smooth L1 Loss), see the following formula:

[0149] ;

[0150] ;

[0151] ;

[0152] in, is the actual value, is the predicted value, is the number of data sets, is the probability of predicting the positive class, is the true label (0 or 1), To smooth the weight of L1 loss in the entire composite loss, is the difference between the predicted value and the true value.

[0153] The compound loss is back-propagated through the network, fine-tuning the network weights from coarse to fine and improving the network generalization ability, thereby improving the performance and reliability of the model in practical applications.

[0154] The line of sight estimation method provided by the present application is obtained by obtaining an image to be detected, wherein the image to be detected is a face image; then constructing a line of sight estimation network, wherein the line of sight estimation network includes a first module, a second module, a third module and an attention module; performing downsampling on the image to be detected by the first module to output a first feature map of the first channel quantity; performing feature extraction on the first feature map by the second module to output a second feature map; then performing downsampling on the second feature map by the third module to output a second feature map of the second channel quantity; iterating the second feature map through the second module and the third module until the second module outputs a third feature map of the third channel quantity; inputting the third feature map into the attention module to output a fourth feature map; inputting the fourth feature map into the classification head to output a line of sight direction category, wherein the line of sight direction category is a pitch angle category and a yaw angle category. The method provided by the present application improves the network's perception of spatial information and improves the accuracy of line of sight estimation by reconstructing the line of sight estimation network and the attention module.

[0155] Similar parts between the embodiments provided in this application can be referenced to each other. The specific implementation methods provided above are only a few examples under the general concept of this application and do not constitute a limitation on the protection scope of this application. For those skilled in the art, any other implementation methods expanded based on the scheme of this application without creative work belong to the protection scope of this application.

Claims

1. A line of sight estimation method, characterized in that: include: Acquire an image to be detected, wherein the image to be detected is an image including at least a monocular area; Constructing a sight estimation network, the sight estimation network comprising a first module, a second module, a third module and an attention module, the first module comprising a first convolution kernel and a first normalization layer, the second module comprising a partial convolution layer and a convolution block, the third module comprising a third convolution kernel and a third normalization layer, and the attention module comprising a first global average pooling layer, a second global average pooling layer, a first convolution layer and a second convolution kernel; Downsampling the image to be detected by the first module to output a first feature map with a first number of channels; Performing feature extraction on the first feature map by the second module to output a second feature map; Downsampling the second feature map by the third module to output a second feature map with a second number of channels; Iterate the second feature map through the second module and the third module until the second module outputs a third feature map with a third number of channels; Inputting the third feature map into an attention module to output a fourth feature map; The fourth feature map is input into a classification head to output a gaze direction category, where the gaze direction category is a pitch angle category and a yaw angle category. The classification head includes a global pooling layer, a first convolutional layer, an activation function, and a fully connected layer.

2. The line of sight estimation method according to claim 1, characterized in that: The downsampling of the image to be detected by the first module to output a first feature map with a first number of channels includes: Performing feature extraction and block downsampling on the image to be detected by using the first convolution kernel to output a first extracted feature map, wherein the number of pixels moved when the convolution kernel slides on the image to be detected is 4, and the number of pixel rows and columns added at the boundary of the image to be detected is 0; The first extracted feature map is input into the first normalization layer to output a first feature map with a first number of channels.

3. The line of sight estimation method according to claim 1, characterized in that: The partial convolution layer includes a second convolution kernel; The performing feature extraction on the first feature map by the second module to output a second feature map includes: Inputting the first feature map into a partial convolutional layer to obtain a first intermediate feature map; Inputting the first intermediate feature map into a convolution block in sequence to obtain a second intermediate feature map, the convolution block comprising a first convolution layer, a second normalization layer, an activation function, and a second convolution layer, wherein the second normalization layer and the activation function are between the first convolution layer and the second convolution layer; Perform an element-by-element addition operation on the first feature map and the second intermediate feature map at corresponding positions to output a second feature map.

4. The line of sight estimation method according to claim 3, characterized in that: The step of inputting the first feature map into a partial convolutional layer to obtain a first intermediate feature map includes: Performing a splitting operation on the first feature map to obtain a first group of feature maps and a second group of feature maps, wherein the number of channels of the first group of feature maps is one quarter of the number of channels of the first feature map, and the number of channels of the second group of feature maps is three quarters of the number of channels of the first feature map; Input the first set of feature maps to the second convolution kernel to output a first convolution result, wherein the number of pixels moved by the second convolution kernel when sliding on the first set of feature maps is 1, and 1 row and 1 column of pixels are added to the four boundaries of the first set of feature maps; The first convolution result is concatenated with the second set of feature maps in the channel dimension to obtain a first intermediate feature map.

5. The line of sight estimation method according to claim 1, characterized in that: The downsampling of the second feature map by the third module to output a second feature map with a second number of channels includes: Performing feature extraction and block downsampling on the second feature map through the third convolution kernel to output a second extracted feature map, wherein the number of pixels moved when the convolution kernel slides on the second extracted feature map is 2, and the number of pixel rows and columns added at the boundary of the second extracted feature map is 0; The second extracted feature map is input into the third normalization layer, and a second feature map with a second number of channels is output.

6. The line of sight estimation method according to claim 1, characterized in that: The step of inputting the third feature map into an attention module to output a fourth feature map comprises: Based on the number of channels, performing a grouping operation on the third feature map having a third number of channels to output a third intermediate feature map, wherein the number of channels of the third intermediate feature map is the product of the number of channels of the third feature map and the inverse of the number of groups, and the batch size of the third intermediate feature map is a batch size times the number of groups of the third feature map; Compressing the horizontal spatial direction features of the third intermediate feature map using a first global average pooling layer to generate a first feature vector, where the first feature vector is a feature vector in a vertical direction; Compressing the vertical spatial direction features of the third intermediate feature map using a second global average pooling layer to generate a second feature vector, where the second feature vector is a feature vector in the horizontal direction; concatenating the first feature vector and the second feature vector to output a two-dimensional feature vector; Using the first convolution layer to perform a channel-wise convolution operation on the two-dimensional feature vector to output a third extracted feature map and a fourth extracted feature map, where the height of the third extracted feature map is 1 and the width of the fourth extracted feature map is 1; Inputting the third extracted feature map and the fourth extracted feature map into a logic function to output a first weighted representation and a second weighted representation through the logic function; Based on the first weight representation, the second weight representation and the third intermediate feature map, a reweighting strategy is used to reweight to output a weighted feature map.

7. The sight line estimation method according to claim 6, characterized in that: The step of inputting the third feature map into an attention module to output a fourth feature map comprises: Extracting features of the third intermediate feature map using a second convolution kernel to output a fifth extracted feature map, wherein the fifth extracted feature map includes local context information; Performing an average pooling operation on the fifth extracted feature map to output a pooled feature map, wherein the height and width of the pooled feature map are 1; Based on the pooled feature map, a first channel descriptor is obtained through a normalization operation; Performing a group normalization operation on the weighted feature map, and outputting a fourth intermediate feature map; The fourth intermediate feature map is multiplied by the first channel descriptor to output a first relationship vector.

8. The sight line estimation method according to claim 7, characterized in that: The step of inputting the third feature map into an attention module to output a fourth feature map further includes: Performing an average pooling operation and a normalization operation on the fourth intermediate feature map to obtain a second channel descriptor; multiplying the fifth extracted feature map by the second channel descriptor to output a second relationship vector; performing an element-by-element addition operation on the first relationship vector and the second relationship vector at corresponding positions to output a fifth intermediate feature map; Inputting the fifth intermediate feature map into a logic function to output a third weight representation through the logic function; Based on the third weight representation and the third intermediate feature map, a reweighting strategy and merging and grouping are utilized to output a fourth feature map, where the number of channels and the batch size of the fourth feature map are the same as those of the third feature map.

9. The sight line estimation method according to claim 1, characterized in that: Inputting the fourth feature map into a classification head to output a gaze direction category includes: After inputting the fourth feature map into the global pooling layer, the fourth feature map is input into the first convolutional layer to output a sixth intermediate feature map; After the sixth intermediate feature map is input into the activation function, it is respectively input into the first fully connected layer and the second fully connected layer to output the yaw angle category through the first fully connected layer, and to output the pitch angle category through the second fully connected layer.

10. The sight line estimation method according to claim 1, characterized in that: The method further comprises: Converting the gaze direction category into an expected value of a probability distribution using normalization; Constructing a composite loss function through the expected value, cross entropy loss and smoothing loss; The line of sight estimation network is trained using the composite loss function.

Images