Multi-task sight estimation model construction method and system, terminal and medium

Abstract

The invention provides a construction method of a multi-task sight line estimation model. And constructing a multi-task sight estimation model based on frequency domain and airspace interaction. According to the construction method of the multi-task sight line estimation model, convolution of a receptive field with an image size is effectively realized in a frequency domain through Fourier transform, and global dependency can be captured so as to better utilize small and complex local features in an image; a multi-task learning strategy is adopted, and a plurality of related tasks are trained at the same time, so that the model can share useful information among different tasks, and the performance of each task is improved; end-to-end training of the multi-task sight line estimation model based on frequency domain and spatial domain interaction is completed by adopting a known sight line estimation data set and based on a saliency map representing a human eye region generated by left and right eye corner labels in the known sight line estimation data set; and testing the trained multi-task line-of-sight estimation model based on frequency domain and spatial domain interaction by adopting a test image in a known line-of-sight estimation data set.

Description

Technical Field

[0001] The present invention relates to the field of line of sight estimation technology, and in particular to a multi-task line of sight estimation model construction method, system, terminal and medium. Background Art

[0002] The human visual attention mechanism guides people to selectively observe scenes and ignore areas with less information, enabling humans to quickly analyze complex and diverse scenes. Changes in attention can be explicitly converted into eye movements that can be captured by instruments. The analysis of eye movement data helps researchers understand human mental states and behavioral intentions. Gaze estimation aims to predict a person's three-dimensional gaze direction or two-dimensional gaze point from a face or eye image. Currently, gaze estimation has various applications. For example, in assisted driving systems, it can predict the driver's attention distribution, thereby improving road safety; in the medical field, it can help analyze the patient's visual and cognitive conditions.

[0003] Early gaze estimation was mainly based on mechanical and optical principles, with eye movements recorded by the rotation of mechanical devices or light reflection. These methods are usually invasive and require the device to be directly attached to the subject's eyeball. Modern gaze estimation devices have greatly expanded their application range by using computer vision-based methods. The proposed methods can be mainly divided into three categories: methods based on three-dimensional human eye models, methods based on two-dimensional human eye feature regression, and methods based on appearance. Although the first two methods provide high accuracy, most of them require specialized hardware equipment. And usually a personal calibration stage is required to estimate personal eye parameters, which will be limited by many factors such as light intensity, head posture, and camera resolution in actual application scenarios.

[0004] The appearance-based gaze estimation method only needs to use a webcam to capture the appearance image of the subject. Compared with the above two methods, webcams have been integrated into many devices, such as mobile phones, tablets, and laptops, and are relatively cheap. Therefore, appearance-based gaze estimation is easier to popularize. Traditional appearance-based methods rely on manually extracted features and it is difficult to maintain good generalization ability under unconstrained conditions. In recent years, gaze estimation methods based on convolutional neural networks have become a research hotspot. Compared with traditional appearance-based methods, convolutional neural network-based methods have shown many advantages: higher-level abstract features can be extracted from images; higher-dimensional nonlinear mapping functions from images to gaze can be learned. However, the actual receptive field of convolutional neural networks is limited, and global features are ignored, resulting in the inability to effectively utilize small and complex local features in the image in the process of extracting facial features. Summary of the invention

[0005] 1. Technical issues to be resolved In view of the shortcomings of the prior art, the present invention provides a multi-task line of sight estimation model construction method, system, terminal and medium, which have the advantages of comprehensiveness and solve the problem that the process of extracting facial features cannot effectively utilize small and complex local features in the image; (II) Technical solution To achieve the above comprehensive purpose, the present invention provides the following technical solution: a method for constructing a multi-task line of sight estimation model, comprising the following steps: S1, build a multi-task line of sight estimation model based on the interaction between frequency domain and spatial domain; S2 adopts a multi-task learning strategy. By training multiple related tasks simultaneously, the model can share useful information between different tasks, thereby improving the performance of each task. S3, using a known gaze estimation dataset and generating a saliency map representing the human eye area based on the annotations of the left and right eye corners, to complete end-to-end training of the multi-task gaze estimation model based on the interaction between the frequency domain and the spatial domain; S4, using a test image in a known line of sight estimation dataset to test the trained multi-task line of sight estimation model based on the interaction between the frequency domain and the spatial domain, and constructing a final line of sight estimation model; Preferably, the multi-task line of sight estimation model based on the interaction between the frequency domain and the spatial domain is constructed. By introducing the interaction between the frequency domain and the spatial domain into the line of sight estimation model, the global dependency in the frequency domain and the local dependency in the spatial domain can be fully obtained and effectively integrated. Frequency domain information can capture the global features of the image, while spatial domain information can reflect the local details of the image. By combining these two types of information, the model can perform line of sight estimation more accurately, specifically including: S101, constructing a spatial domain module for extracting local features of the image; this module can capture detailed information in the image, such as the edge of the eye and the position of the pupil; S102, constructing a frequency domain module for extracting global dependencies of the image; this module helps the model better understand the contextual information in the image; S103, constructing a frequency domain and spatial domain interactive fusion module, and adaptively fusing global frequency information and local spatial information based on the difference and complementarity between frequency information and spatial information; S104, constructing a sight line prediction module, regressing high-level features to sight line directions, and generating sight line estimation results; S105, constructing a saliency map reading module, which is used to compress and normalize the feature map along the channel dimension to obtain a prediction result of a salient region of the image for human eyes; Preferably, the spatial domain module is composed of cascaded residual convolution blocks; the residual convolution blocks retain input information by adding skip connections, so as to improve the feature extraction capability and training efficiency of the model; The frequency domain module uses fast Fourier transform (FFT) to transform the latent features into the frequency domain to obtain its frequency domain representation; in the frequency domain, the global features are extracted by a global average pooling layer, and a convolution layer and a sigmoid activation layer are used to generate an instance adaptive mask, which is multiplied by the frequency domain representation to enhance the frequency components that are beneficial to line of sight estimation; finally, the enhanced frequency domain features are transformed back to the original latent space by inverse fast Fourier transform (IFFT) to obtain the global features; The frequency domain and spatial domain interactive fusion module first combines the frequency domain and spatial domain features by element-by-element multiplication and summation; then, the combined features are input into a parallel structure consisting of a convolutional layer, a ReLU activation layer, and a batch normalization layer to generate different spatial attention maps for fusion; the fused features are used as guidance information to further fuse the frequency domain and spatial domain features in the channel dimension to improve the accuracy of the model's line of sight estimation; The sight line prediction module includes a global average pooling layer and two fully connected layers; the global average pooling layer is used to reduce the feature dimension, and the fully connected layer converts the fused feature vector into a predicted value of the sight line direction to generate the final sight line estimation result; The saliency map readout module compresses and normalizes multiple feature maps along the channel dimension to obtain the human eye saliency area prediction result of the image; the network layer is composed of multiple convolutional layers and ReLU activation functions.

[0006] Preferably, in S2, a multi-task learning strategy is adopted to train multiple related tasks simultaneously, so that the model can share useful information between different tasks, thereby improving the performance of each task, including: Combining the two tasks of gaze estimation and eye region detection; the gaze estimation task is responsible for predicting the gaze direction of the subject, while the eye region detection task is used to identify the specific position of the eye in the image. By combining these two tasks, the model can not only obtain more accurate gaze estimation results, but also maintain high robustness in different scenarios.

[0007] Preferably, S3 uses a known gaze estimation dataset and generates a saliency map representing the eye area based on the annotations of the left and right eye corners to complete the end-to-end training of the multi-task gaze estimation model based on the interaction between the frequency domain and the spatial domain, specifically including: The multi-task line of sight estimation model based on the interaction between frequency domain and spatial domain is pre-trained using the public line of sight estimation dataset ETH-XGaze, and the line of sight estimation datasets MPIIFaceGace and EYEDIAP are used to correct the multi-task line of sight estimation model based on the interaction between frequency domain and spatial domain; Setting initialization parameters of the multi-task line of sight estimation model based on frequency domain and spatial domain interaction; Determining a loss function of the multi-task line of sight estimation model based on frequency domain and spatial domain interaction; Determine relevant hyperparameters in the multi-task line of sight estimation model based on frequency domain and spatial domain interaction; The end-to-end training of the multi-task line of sight estimation model based on the interaction between the frequency domain and the spatial domain is completed through the above steps; The method of using a test image in a known line of sight estimation dataset to test the trained multi-task line of sight estimation model based on the interaction between the frequency domain and the spatial domain comprises: In the line of sight estimation datasets MPIIFaceGace and EYEDIAP, the cross-validation method is used to test the trained multi-task line of sight estimation model based on the interaction between frequency domain and spatial domain, so as to evaluate the performance of the model.

[0008] A method for constructing a multi-task line of sight estimation model, characterized in that it also includes any one or more of the following: using a public line of sight estimation dataset ETH-XGaze to pre-train the multi-task line of sight estimation model based on frequency domain and spatial domain interaction, and using line of sight estimation datasets MPIIFaceGace and EYEDIAP to fine-tune the multi-task line of sight estimation model based on frequency domain and spatial domain interaction, including: Obtain the public gaze estimation datasets ETH-XGaze, MPIIFaceGace, and EYEDIAP, average the annotations of the left and right eye corners in the gaze estimation datasets to obtain an estimate of the pupil position; for the estimate of the pupil position, use a mixed Gaussian distribution to generate a gaze point density map representing the salient area of ​​the human eye as a true value map; Using the image of the line of sight estimation dataset ETH-XGaze as a model input, using the line of sight direction annotation and the true value map corresponding to the line of sight estimation dataset ETH-XGaze as labels, training the multi-task line of sight estimation model based on the interaction between the frequency domain and the spatial domain in an end-to-end manner, so that the model automatically learns the mapping relationship between the original image and the line of sight direction annotation and the true value map; Using the images of the line of sight estimation datasets MPIIFaceGace and EYEDIAP as model inputs, using the line of sight direction annotations and truth maps corresponding to the images of the line of sight estimation datasets MPIIFaceGace and EYEDIAP as labels, fine-tuning the multi-task line of sight estimation model based on the interaction between the frequency domain and the spatial domain in an end-to-end manner, so that the model automatically learns the mapping relationship between the original image and the line direction annotations and the truth map; The setting of initialization parameters of the multi-task line of sight estimation model based on the interaction between the frequency domain and the spatial domain includes: The initial parameters of the sight prediction module and the saliency map readout module are randomly initialized, and the parameters of the remaining modules are pre-trained on the ETH-XGaze dataset as initialization parameters; Determining the loss function of the multi-task sight line estimation model based on the interaction between the frequency domain and the spatial domain includes: The loss function adopts a weighted linear combination of mean square error and two significant performance evaluation indicators KL and CC; The determining of relevant hyper parameters in the multi-task sight line estimation model based on the interaction between the frequency domain and the spatial domain includes: The stochastic gradient descent algorithm is used in the pre-training process. The initial learning rate is 0.0004, and it decreases by 10 times every 10 epochs. The batch size is 32, and the pre-training process needs to iterate 25 epochs until the model converges.

[0009] According to another aspect of the present invention, a system for constructing a multi-task sight line estimation model based on frequency domain and spatial domain interaction includes: A face image acquisition module, which is used to acquire a face image of a subject; Prediction model building module, which is used to build a multi-task line of sight estimation model based on the interaction between frequency domain and spatial domain; A model training module, which is used to pre-train the multi-task sight line estimation model based on the interaction between the frequency domain and the spatial domain using the sight line estimation dataset ETH-XGaze, and to fine-tune the multi-task sight line estimation model based on the interaction between the frequency domain and the spatial domain using the sight line estimation datasets MPIIFaceGace and EYEDIAP, thereby completing the end-to-end training of the multi-task sight line estimation model based on the interaction between the frequency domain and the spatial domain; The model testing module is used to test the trained multi-task line of sight estimation model based on the interaction between frequency domain and spatial domain using known test images to evaluate the performance of the constructed model.

[0010] According to the third aspect of the present invention, a multi-task line of sight estimation method based on the interaction between frequency domain and spatial domain is provided, which adopts the line of sight estimation obtained by constructing the line of sight estimation model by any of the construction methods or construction systems described above, takes any image as the input of the model, and outputs the prediction result of the line of sight direction.

[0011] According to a fourth aspect of the present invention, there is provided a computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute any of the above methods when executing the program.

[0012] According to a fifth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to execute any of the methods described above.

[0013] (III) Beneficial effects Compared with the prior art, the present invention provides a multi-task line of sight estimation model construction method, system, terminal and medium, which have the following beneficial effects: 1. The multi-task line of sight estimation model construction method, system, terminal and medium, by adopting a multi-task line of sight estimation model based on the interaction between frequency domain and spatial domain, fully considers the difference and complementarity between spatial domain information and frequency domain information, and obtains excellent performance on the line of sight estimation datasets MPIIFaceGace and EYEDIAP. The realization of this performance mainly depends on the exploration of spatial and frequency domain information. The frequency domain module adopted by it efficiently captures the global dependency of the image by Fourier transform, and through adaptive feature fusion and multi-task learning strategies, it can more fully learn the information related to the line of sight under the supervision of line of sight direction annotation and truth map, and achieve excellent performance.

[0014] 2. The multi-task line of sight estimation model construction method, system, terminal and medium use a spatial domain module composed of cascaded residual blocks and a frequency domain module realized by Fourier transform to effectively extract local detail information and global context information of the image respectively, and explore the complementarity between dual-domain information and multiple tasks, thereby improving the accuracy and robustness of the line of sight estimation model. It is efficient, low-cost, easy to implement and very flexible, and can be deployed on a backbone network with fewer parameters or a backbone network with better performance according to actual needs in terms of efficiency and performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 The figure is a workflow diagram of a sight line estimation method in a preferred embodiment of the present invention.

[0016] Figure 2 Schematic diagram of component modules of a sight line estimation system in a preferred embodiment of the present invention.

[0017] Figure 3 These are some subjective experimental results obtained by using a line of sight estimation method and system provided in a preferred embodiment of the present invention on a public dataset in the field of line of sight estimation. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention. Embodiment 1: An embodiment of the present invention provides a method for constructing a line of sight estimation model. The method aims at the characteristics of frequency domain information and realizes the construction of a multi-task line of sight estimation model based on the interaction between frequency domain and spatial domain.

[0019] like Figure 1 As shown, the method for constructing the sight line estimation model provided in this embodiment may include: S1, construct a line of sight estimation model based on the interaction between frequency domain and spatial domain; S2 adopts a multi-task learning strategy. By training multiple related tasks simultaneously, the model can share useful information between different tasks, thereby improving the performance of each task. S3, using a known gaze estimation dataset and generating a saliency map representing the human eye area based on the annotations of the left and right eye corners, to complete end-to-end training of the multi-task gaze estimation model based on the interaction between the frequency domain and the spatial domain; S4, using a test image in a known line of sight estimation data set to test the trained multi-task line of sight estimation model based on the interaction between the frequency domain and the spatial domain, and constructing a final line of sight estimation model.

[0020] In a preferred embodiment of S1, constructing a multi-task sight line estimation model based on the interaction between frequency domain and spatial domain may include: A multi-task line of sight estimation model based on the interaction between frequency domain and spatial domain is constructed. The model mainly includes a spatial domain module, a frequency domain module, a frequency domain and spatial domain interaction fusion module, a line of sight prediction module and a saliency map readout module to perform end-to-end line of sight estimation. Among them: S101, build a spatial domain module. The spatial domain module consists of cascaded residual convolution blocks. Specifically, each residual convolution block contains 2 convolution layers and a batch normalization layer. The convolution layer extracts features, and the batch normalization layer is used to accelerate training and stabilize the network. The jump connection directly passes the input to the output end and fuses it with the features processed by convolution and batch normalization, making the feature representation richer and more robust.

[0021] S102, construct frequency domain module. Given the input spatial domain features , using the Fast Fourier Transform (FFT) Convert to the frequency domain and obtain its frequency domain representation ( Each position of corresponds to a different frequency component). In the frequency domain, an instance adaptive mask is generated by a frequency domain filter Specifically, for the frequency domain filter, each frequency component is compressed using the maximum pooling layer and the average pooling layer along the channel dimension, and then the two compressed representations are concatenated and filtered using a large kernel 7×7 convolution layer, followed by a sigmoid function to learn the instance adaptive mask. With frequency domain representation Multiply them to enhance the frequency components that are beneficial to line of sight estimation. Finally, the enhanced frequency domain features are transformed back to the original latent space through inverse fast Fourier transform (IFFT) to obtain the global features.

[0022] S103, construct a frequency domain and spatial domain interactive fusion module. The frequency domain and spatial domain interactive fusion module consists of a feature enhancement unit (FEU) and a cross-domain attention unit (CAU). It first generates different attention maps for the two types of information through FEU to fuse them. Then the fused features are As the input of CAU, it guides the fusion of semantic information represented by different channels.

[0023] In FEU, first, we combine by element-wise multiplication and summation. and ,generate , . can reveal important information coexisting in the two features, and It can capture the information contained in the two features comprehensively and without omission. and The sum is added and input into a parallel structure consisting of average pooling, convolution layer and sigmoid function to encode the position information of the image in the vertical and horizontal directions respectively. , . and are broadcasted to the same dimension Then multiply element by element to obtain the spatial attention weight .

[0024]

[0025] Given the uniqueness and complementarity between frequency information and spatial information, the channel dimension is divided , to generate adaptive attention and , used to adaptively reweight them,

[0026] in is the final fusion result.

[0027] In CAU, in order to reduce the computational cost of CAU, a permutation operation is used to transfer part of the spatial information to the channel dimension and compress the channel dimension of the query and key. Specifically, CAU has two inputs, including guidance information and source information . Given a scaling factor , first and Split into The non-overlapping size is The query is generated after channel compression and rearrangement. With key Directly to Perform linear operations and rearrange to generate values , then, the cross attention between source information and guidance information can be obtained by the following formula:

[0028]

[0029] in, represents a normalized scalar, . Let the frequency information and spatial information As source information,

[0030] The two parameters correspond to guidance information and source information respectively.

[0031] Based on the spatial domain module, frequency domain module and frequency domain spatial domain interactive fusion module, a feature extraction network is constructed. The feature extraction network is divided into four stages, each stage is represented as Stage i ( ), the corresponding extracted features are expressed as Two cascaded spatial domain modules are used to form Stage 1 and Stage 2. From Stage 3 to Stage 4, the spatial domain module is replaced by a dual-branch structure consisting of a spatial domain module, a frequency domain module, and a frequency-spatial domain interactive fusion module. It is sent to the sight line prediction module to get the sight line direction. , and It is sent to the saliency map readout module to generate the saliency map of the human eye area of ​​the input image.

[0032] S104, constructing a sight prediction module. The sight prediction module consists of a global average pooling layer and two fully connected layers. First, the input feature map is reduced in dimension through the global average pooling layer, converting the high-dimensional feature map into a low-dimensional feature vector, reducing the computational complexity and retaining important information. Then, the feature vector is input into the first fully connected layer to further fuse the features, and the ReLU activation layer is used to enhance the nonlinearity. Then, the second fully connected layer generates the final sight estimation result. S105, construct a saliency map readout network layer. Two 1×1 convolutional layers and one 7×7 depthwise separable convolutional layer are used to reduce the number of channels and perform feature aggregation. Subsequently, two 4×4 transposed convolutions are used to gradually increase the resolution of the feature map and restore it to the size of the original input image. Finally, a saliency map is generated by applying a sigmoid function to the output of the convolutional layer, thereby predicting the human eye salient area in the image.

[0033] In a preferred embodiment of S2, a multi-task learning strategy is adopted to enable the model to share useful information between different tasks by training multiple related tasks simultaneously, thereby improving the performance of each task, which may include: Combine the two tasks of gaze estimation and eye region detection. The gaze estimation task is responsible for predicting the gaze direction of the subject, while the eye region detection task is used to identify the specific position of the eye in the image. By combining these two tasks, the model can not only obtain more accurate gaze estimation results, but also maintain high robustness in different scenarios.

[0034] In a preferred embodiment of S3, a known gaze estimation dataset is used, and a saliency map representing the human eye area is generated based on the annotations of the left and right eye corners, to complete the end-to-end training of the multi-task gaze estimation model based on the interaction between the frequency domain and the spatial domain, which may include: S301, using the public sight estimation dataset ETH-XGaze to pre-train the multi-task sight estimation model based on the interaction between the frequency domain and the spatial domain, and using the sight estimation datasets MPIIFaceGace and EYEDIAP to fine-tune the multi-task sight estimation model based on the interaction between the frequency domain and the spatial domain. Wherein: Obtain the public gaze estimation datasets ETH-XGaze, MPIIFaceGace, and EYEDIAP, average the annotations of the left and right eye corners in the gaze estimation datasets to obtain an estimate of the pupil position; for the estimate of the pupil position, use a mixed Gaussian distribution to generate a gaze point density map representing the salient area of ​​the human eye as a true value map and train the constructed model in subsequent steps; Taking the image of the line of sight estimation dataset ETH-XGaze as the model input, taking the line of sight direction annotation and the truth map corresponding to the line of sight estimation dataset ETH-XGaze as labels, training the multi-task line of sight estimation model based on the interaction between frequency domain and spatial domain in an end-to-end manner, so that the model automatically learns the mapping relationship between the original image and the line of sight direction annotation and the truth map; taking the image of the line of sight estimation dataset MPIIFaceGace and EYEDIAP as the model input, taking the line of sight direction annotation and the truth map corresponding to the image of the line of sight estimation dataset MPIIFaceGace and EYEDIAP as labels, fine-tuning the multi-task line of sight estimation model based on the interaction between frequency domain and spatial domain in an end-to-end manner, so that the model automatically learns the mapping relationship between the original image and the line direction annotation and the truth map, and fine-tuning the model; specifically: The ETH-XGaze dataset is collected by using 18 Canon 250D digital SLR cameras with a resolution of 6000×4000 pixels in 3D space to collect ground truth data, capturing a wide range of head poses. It contains 756K training images of 80 participants and test images of 15 participants, so the proposed gaze estimation model is first pre-trained on this dataset to provide good initialization parameters for subsequent training. MPIIFaceGaze contains a total of 213,659 images collected from 15 subjects. These images were collected by showing participants random points on a laptop screen in their daily lives over a period of several months. During the data collection process, the participants' head poses were not restricted. The EYEDIAP dataset contains 94 video clips from 16 participants. The data was collected in a laboratory environment and includes three visual target tasks: continuous moving targets, discrete moving targets, and floating balls. Two cameras were used for data collection: an RGBD camera and a high-definition camera; S302, setting initialization parameters for the model. The initial parameters of the sight prediction module and the saliency map readout module are randomly initialized, and the parameters of the remaining modules are pre-trained on the ETH-XGaze dataset as initialization parameters; S303, determine the loss function. The loss function used in model training is a weighted linear combination of mean square error and two significant performance evaluation indicators KL and CC. The weight of each indicator is determined based on the experimental results to better balance the impact of each indicator on the model performance; for the CC indicator, the better the performance of the model, the larger the value of the CC indicator, so the coefficient of the CC indicator in the loss function is negative, and the coefficients of the other indicators are positive; S304, determine the relevant hyperparameters in the model. The initial learning rate is 0.0004, and it decreases by 10 times every 10 epochs. The batch size is 32, and the pre-training process needs to iterate 25 epochs until the model converges. In a preferred embodiment of S4, testing the trained multi-task line of sight estimation model based on frequency domain and spatial domain interaction using a test image in a known line of sight estimation dataset to construct a final line of sight estimation model may include: S401, using two commonly used public datasets in the field of line of sight estimation to test the performance of the model proposed in the present invention, namely MPIIFaceGaz and EYEDIAP. During the test, following the previous work on the two datasets, for MPIIFaceGaze, the leave-one-out method was used to evaluate the performance of the proposed method, and for EYEDIAP, four-fold cross validation was used for evaluation to construct the final line of sight estimation model.

[0035] An embodiment of the present invention provides a system for constructing a multi-task sight line estimation model based on frequency domain and spatial domain interaction, which may include: A face image acquisition module, which is used to acquire a face image of a subject; Prediction model building module, which is used to build a multi-task line of sight estimation model based on the interaction between frequency domain and spatial domain; A model training module, which is used to pre-train the multi-task sight line estimation model based on the interaction between the frequency domain and the spatial domain using the sight line estimation dataset ETH-XGaze, and to fine-tune the multi-task sight line estimation model based on the interaction between the frequency domain and the spatial domain using the sight line estimation datasets MPIIFaceGace and EYEDIAP, thereby completing the end-to-end training of the multi-task sight line estimation model based on the interaction between the frequency domain and the spatial domain; The model testing module is used to test the trained multi-task line of sight estimation model based on the interaction between frequency domain and spatial domain using known test images to evaluate the performance of the constructed model.

[0036] In a preferred embodiment, the multi-task line of sight estimation model based on the interaction between frequency domain and spatial domain may include: The spatial domain module is used to extract local features of the image. This module can capture detailed information in the image, such as the edge of the eye and the position of the pupil.

[0037] The frequency domain module is used to extract the global dependencies of the image. This module helps the model better understand the contextual information in the image.

[0038] The frequency-domain and spatial-domain interactive fusion module adaptively fuses global frequency information and local spatial information based on the difference and complementarity between frequency information and spatial information.

[0039] The line of sight prediction module regresses high-level features to the line of sight direction and generates line of sight estimation results.

[0040] The saliency map readout module is used to compress and normalize the feature map along the channel dimension to obtain the prediction result of the salient area of ​​the image for human eyes.

[0041] It should be noted that the steps in the method provided by the present invention can be implemented by using corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the method to realize the composition of the system, that is, the embodiments in the method can be understood as preferred examples for constructing the system, which will not be elaborated here.

[0042] An embodiment of the present invention provides a multi-task line of sight estimation method based on the interaction between frequency domain and spatial domain. A line of sight estimation model is constructed using the line of sight estimation model construction method or construction system of any one of the above embodiments. Any image is used as the input of the model, and a prediction result of the line of sight direction is obtained as output.

[0043] An embodiment of the present invention provides a computer terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to execute any method or system of the above embodiments of the present invention.

[0044] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random access memory (English: random-access memory, abbreviated: RAM), such as static random access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above computer programs, computer instructions, data, etc. can be called by the processor.

[0045] The above-mentioned computer programs, computer instructions, etc. may be stored in one or more memories in partitions, and the above-mentioned computer programs, computer instructions, data, etc. may be called by a processor.

[0046] The processor is used to execute the computer program stored in the memory to implement the various steps of the method or various modules of the system involved in the above embodiments. For details, please refer to the relevant descriptions in the above method and system embodiments.

[0047] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.

[0048] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the program can be used to execute the method or system of any one of the above embodiments of the present invention.

[0049] The method, system, terminal and medium for constructing the line of sight estimation model provided by the above-mentioned embodiments of the present invention adopt a multi-task line of sight estimation model based on the interaction between frequency domain and spatial domain, fully consider the difference and complementarity between spatial domain information and frequency domain information, and obtain excellent performance in the line of sight estimation data sets MPIIFaceGace and EYEDIAP. The realization of this performance mainly depends on the exploration of spatial and frequency domain information. The frequency domain module adopted by it efficiently captures the global dependency of the image by Fourier transform, and through adaptive feature fusion and multi-task learning strategies, it can more fully learn the information related to the line of sight under the supervision of the line of sight direction annotation and the truth map, and achieve excellent performance.

[0050] The method, system, terminal and medium for constructing a line of sight estimation model provided by the above-mentioned embodiments of the present invention utilize a spatial domain module composed of cascaded residual convolution blocks and a frequency domain module implemented by Fourier transform to effectively extract local detail information and global context information of the image, respectively, and explore the complementarity between dual-domain information and multiple tasks, thereby improving the accuracy and robustness of the line of sight estimation model.

[0051] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for constructing a multi-task line of sight estimation model, characterized in that: The following steps are involved: S1, build a multi-task line of sight estimation model based on the interaction between frequency domain and spatial domain; S2 adopts a multi-task learning strategy. By training multiple related tasks simultaneously, the model can share useful information between different tasks, thereby improving the performance of each task. S3, using a known gaze estimation dataset and generating a saliency map representing the human eye area based on the annotations of the left and right eye corners, to complete end-to-end training of the multi-task gaze estimation model based on the interaction between the frequency domain and the spatial domain; S4, using a test image in a known line of sight estimation data set to test the trained multi-task line of sight estimation model based on the interaction between the frequency domain and the spatial domain, and constructing a final line of sight estimation model.

2. A multi-task sight line estimation model construction method according to claim 1, characterized in that: The multi-task line of sight estimation model based on the interaction between frequency domain and spatial domain is constructed. By introducing the interaction between frequency domain and spatial domain into the line of sight estimation model, the global dependency in the frequency domain and the local dependency in the spatial domain can be fully obtained and effectively integrated; the frequency domain information can capture the global features of the image, while the spatial domain information can reflect the local details of the image; by combining these two kinds of information, the model can perform line of sight estimation more accurately, specifically including: S101, constructing a spatial domain module for extracting local features of the image; this module can capture detailed information in the image, such as the edge of the eye and the position of the pupil; S102, constructing a frequency domain module for extracting global dependencies of the image; this module helps the model better understand the contextual information in the image; S103, constructing a frequency domain and spatial domain interactive fusion module, and adaptively fusing global frequency information and local spatial information based on the difference and complementarity between frequency information and spatial information; S104, constructing a sight line prediction module, regressing high-level features to sight line directions, and generating sight line estimation results; S105, constructing a saliency map reading module, which is used to compress and normalize the feature map along the channel dimension to obtain a prediction result of the human eye saliency region of the image.

3. A multi-task sight line estimation model construction method according to claim 2, characterized in that: Also includes any one or more of the following: The spatial domain module is composed of cascaded residual convolution blocks; the residual convolution blocks retain input information by adding skip connections to improve the feature extraction capability and training efficiency of the model; The frequency domain module uses fast Fourier transform (FFT) to transform the latent features into the frequency domain to obtain its frequency domain representation; in the frequency domain, the global features are extracted by a global average pooling layer, and a convolution layer and a sigmoid activation layer are used to generate an instance adaptive mask, which is multiplied by the frequency domain representation to enhance the frequency components that are beneficial to line of sight estimation; finally, the enhanced frequency domain features are transformed back to the original latent space by inverse fast Fourier transform (IFFT) to obtain the global features; The frequency domain and spatial domain interactive fusion module first combines the frequency domain and spatial domain features by element-by-element multiplication and summation; Then, the combined features are input into a parallel structure consisting of a convolutional layer, a ReLU activation layer, and a batch normalization layer to generate different spatial attention maps for fusion. The fused features are used as guidance information to further fuse the frequency domain and spatial domain features in the channel dimension to improve the accuracy of the model's line of sight estimation. The sight line prediction module includes a global average pooling layer and two fully connected layers; the global average pooling layer is used to reduce the feature dimension, and the fully connected layer converts the fused feature vector into a predicted value of the sight line direction to generate the final sight line estimation result; The saliency map readout module compresses and normalizes multiple feature maps along the channel dimension to obtain the human eye saliency area prediction result of the image; the network layer is composed of multiple convolutional layers and ReLU activation functions.

4. The method, system, terminal and medium for constructing a multi-task line of sight estimation model according to claim 1, characterized in that: In S2, a multi-task learning strategy is adopted to train multiple related tasks simultaneously, so that the model can share useful information between different tasks, thereby improving the performance of each task, including; Combine the two tasks of gaze estimation and eye region detection; The gaze estimation task is responsible for predicting the subject's gaze direction, while the eye region detection task is used to identify the specific position of the eye in the image; through the combination of these two tasks, the model can not only obtain more accurate gaze estimation results, but also maintain high robustness in different scenarios.

5. The method for constructing a multi-task line of sight estimation model according to claim 1, characterized in that: The S3 uses a known gaze estimation dataset and generates a saliency map representing the human eye area based on the annotations of the left and right eye corners to complete the end-to-end training of the multi-task gaze estimation model based on the interaction between the frequency domain and the spatial domain. The end-to-end training of the multi-task gaze estimation model based on the interaction between the frequency domain and the spatial domain is completed by the following steps, specifically including: S301, using the public line of sight estimation dataset ETH-XGaze to pre-train the multi-task line of sight estimation model based on the interaction between the frequency domain and the spatial domain, and using the line of sight estimation datasets MPIIFaceGace and EYEDIAP to correct the multi-task line of sight estimation model based on the interaction between the frequency domain and the spatial domain; S302, setting initialization parameters of the multi-task line of sight estimation model based on frequency domain and spatial domain interaction; S303, determining a loss function of the multi-task sight line estimation model based on frequency domain and spatial domain interaction; S304, determining relevant hyperparameters in the multi-task sight line estimation model based on frequency domain and spatial domain interaction; The method of using test images in a known line of sight estimation dataset to test the trained multi-task line of sight estimation model based on interaction between frequency domain and spatial domain comprises: in the line of sight estimation datasets MPIIFaceGace and EYEDIAP, using a cross-validation method to test the trained multi-task line of sight estimation model based on interaction between frequency domain and spatial domain, so as to evaluate the performance of the model.

6. A multi-task sight line estimation model construction method according to claim 1, characterized in that: Also includes any one or more of the following: using the public line of sight estimation data set ETH-XGaze to pre-train the multi-task line of sight estimation model based on the interaction between the frequency domain and the spatial domain, and using the line of sight estimation data sets MPIIFaceGace and EYEDIAP to fine-tune the multi-task line of sight estimation model based on the interaction between the frequency domain and the spatial domain, including: Obtain the public gaze estimation datasets ETH-XGaze, MPIIFaceGace, and EYEDIAP, average the annotations of the left and right eye corners in the gaze estimation datasets to obtain an estimate of the pupil position; for the estimate of the pupil position, use a mixed Gaussian distribution to generate a gaze point density map representing the salient area of ​​the human eye as a true value map; Using the image of the line of sight estimation dataset ETH-XGaze as a model input, using the line of sight direction annotation and the true value map corresponding to the line of sight estimation dataset ETH-XGaze as labels, training the multi-task line of sight estimation model based on the interaction between the frequency domain and the spatial domain in an end-to-end manner, so that the model automatically learns the mapping relationship between the original image and the line of sight direction annotation and the true value map; Using the images of the line of sight estimation datasets MPIIFaceGace and EYEDIAP as model inputs, using the line of sight direction annotations and truth maps corresponding to the images of the line of sight estimation datasets MPIIFaceGace and EYEDIAP as labels, fine-tuning the multi-task line of sight estimation model based on the interaction between the frequency domain and the spatial domain in an end-to-end manner, so that the model automatically learns the mapping relationship between the original image and the line direction annotations and the truth map; The setting of initialization parameters of the multi-task line of sight estimation model based on the interaction between the frequency domain and the spatial domain includes: The initial parameters of the sight prediction module and the saliency map readout module are randomly initialized, and the parameters of the remaining modules are pre-trained on the ETH-XGaze dataset as initialization parameters; Determining the loss function of the multi-task sight line estimation model based on the interaction between the frequency domain and the spatial domain includes: The loss function adopts a weighted linear combination of mean square error and two significant performance evaluation indicators KL and CC; The determining of relevant hyper parameters in the multi-task sight line estimation model based on the interaction between the frequency domain and the spatial domain includes: The stochastic gradient descent algorithm is used in the pre-training process. The initial learning rate is 0.0004, and it decreases by 10 times every 10 epochs. The batch size is 32, and the pre-training process needs to iterate 25 epochs until the model converges.

7. A system for constructing a multi-task line of sight estimation model based on the interaction between frequency domain and spatial domain, comprising: A face image acquisition module, which is used to acquire a face image of a subject; Prediction model building module, which is used to build a multi-task line of sight estimation model based on the interaction between frequency domain and spatial domain; A model training module, which is used to pre-train the multi-task sight line estimation model based on the interaction between the frequency domain and the spatial domain using the sight line estimation dataset ETH-XGaze, and to fine-tune the multi-task sight line estimation model based on the interaction between the frequency domain and the spatial domain using the sight line estimation datasets MPIIFaceGace and EYEDIAP, thereby completing the end-to-end training of the multi-task sight line estimation model based on the interaction between the frequency domain and the spatial domain; The model testing module is used to test the trained multi-task line of sight estimation model based on the interaction between frequency domain and spatial domain using known test images to evaluate the performance of the constructed model.

8. A multi-task line of sight estimation method based on frequency domain and spatial domain interaction, characterized in that: A line of sight estimation model constructed by the line of sight estimation model construction method described in any one of claims 1 to 6 or the line of sight estimation model construction system described in claim 7 is used, and any image is used as input of the model to output a prediction result of the line of sight direction.

9. A computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it can be used to perform the method described in any one of claims 1 to 6 or 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it can be used to execute the method described in any one of claims 1 to 7 or 9.

Images