A line-of-sight estimation model training method, apparatus and system

Abstract

The line-of-sight estimation model training method, device and system provided by the embodiment of the application are applied to the technical field of model training, feature extraction is performed on the eye region and the face region in a sample face image to obtain sample eye features and sample face features, a preset self-supervision loss algorithm is used to obtain sample eye self-supervision loss and sample face self-supervision loss, a supervision loss is calculated according to a pre-labeled line-of-sight direction and an estimated line-of-sight direction of the sample face image, the parameters of a feature extraction network to be trained are adjusted through the sample eye self-supervision loss and the sample face self-supervision loss, the parameters of a line-of-sight direction estimation network to be trained are adjusted through the supervision loss, and the training is continued until a trained feature extraction network and a trained line-of-sight direction estimation network are obtained. Through the scheme of the embodiment of the application, the training efficiency of the line-of-sight estimation model can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of model training, in particular to a gaze estimation model training method, device and system. BACKGROUND

[0002] In recent years, the gaze estimation method has been widely applied in the fields of human-computer interaction, augmented / virtual / mixed reality, eye movement behavior analysis, and assisted driving (intelligent cockpit). Through gaze estimation, the display can be performed according to the user's gaze, thereby improving the user experience.

[0003] However, when training the gaze estimation model at present, the sample image is usually obtained, the features are extracted through the feature extraction network, then the gaze is estimated through the gaze estimation network to be trained according to the extracted features, finally the loss is calculated through the pre-labeled gaze direction and the estimated gaze, and the parameters of the feature extraction network and the gaze estimation network are adjusted, so that the model training efficiency is low. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a gaze estimation model training method, device and system, to solve the problem of low training efficiency when training the current gaze estimation model by calculating the loss and adjusting the parameters of the feature extraction network and the gaze estimation network at the same time. The specific technical solutions are as follows:

[0005] The first aspect of the embodiments of the present application provides a gaze estimation model training method, comprising:

[0006] Obtaining a sample face image, wherein the sample face image is a face image pre-labeled with a gaze direction;

[0007] Inputting the sample face image into a feature extraction network to be trained, and performing feature extraction on the eye region and the face region in the sample face image respectively to obtain sample eye features and sample face features;

[0008] Calculating the self-supervised loss according to the sample eye features and the sample face features respectively through a pre-set self-supervised loss algorithm to obtain sample eye self-supervised loss and sample face self-supervised loss;

[0009] Inputting the sample eye features and the sample face features into a gaze direction estimation network to be trained to calculate the gaze direction to obtain an estimated gaze direction;

[0010] Calculating the supervised loss according to the pre-labeled gaze direction of the sample face image and the estimated gaze direction;

[0011] adjusting parameters of the to-be-trained feature extraction network through the sample eye self-supervision loss and the sample face self-supervision loss, adjusting parameters of the to-be-trained gaze direction estimation network through the supervision loss, and returning the step of inputting the sample face image into the to-be-trained feature extraction network to extract features of an eye region and a face region in the sample face image respectively to obtain a sample eye feature and a sample face feature, is continuously executed until a first preset iteration stop condition is met, and a trained feature extraction network and a trained gaze direction estimation network are obtained.

[0012] Optionally, the sample eye feature includes a sample left eye feature and a sample right eye feature, and the step of inputting the sample face image into the to-be-trained feature extraction network to extract features of an eye region and a face region in the sample face image respectively to obtain a sample eye feature and a sample face feature includes:

[0013] normalizing the sample face image to determine a corresponding left eye region, a right eye region, and a face region;

[0014] extracting features of the left eye region and the right eye region through the to-be-trained eye feature extraction network to obtain the sample left eye feature and the sample right eye feature;

[0015] extracting features of the face region through the to-be-trained face feature extraction network to obtain the sample face feature.

[0016] Optionally, the sample eye self-supervision loss includes a sample left eye self-supervision loss and a sample right eye self-supervision loss.

[0017] The step of calculating self-supervision losses respectively according to the sample eye feature and the sample face feature through a preset self-supervision loss algorithm to obtain a sample eye self-supervision loss and a sample face self-supervision loss includes:

[0018] calculating self-supervision losses respectively according to the sample left eye feature and the sample right eye feature through a preset self-supervision loss algorithm to obtain the sample left eye self-supervision loss and the sample right eye self-supervision loss;

[0019] calculating a self-supervision loss according to the sample face feature through a preset self-supervision loss algorithm to obtain the sample face self-supervision loss;

[0020] The step of adjusting parameters of the to-be-trained feature extraction network through the sample eye self-supervision loss and the sample face self-supervision loss includes:

[0021] adjust parameters of the eye feature extraction network to be trained through the sample eye self-supervision loss;

[0022] adjust parameters of the face feature extraction network to be trained through the sample face self-supervision loss.

[0023] Optionally, the inputting the sample face image into the feature extraction network to be trained, and performing feature extraction on the eye region and the face region in the sample face image respectively to obtain sample eye features and sample face features, comprises:

[0024] performing data enhancement on the eye region and the face region in the sample face image respectively to generate a self-supervision sample pair, wherein the self-supervision sample pair comprises an eye sample pair and a face sample pair, the eye sample pair comprises the eye region before data enhancement and the eye region after data enhancement, and the face sample pair comprises the face region before data enhancement and the face region after data enhancement;

[0025] inputting the eye sample pair into the eye feature extraction network to be trained to perform feature extraction to obtain eye features corresponding to the eye region before data enhancement and the eye region after data enhancement respectively;

[0026] inputting the face sample pair into the face feature extraction network to be trained to perform feature extraction to obtain face features corresponding to the face region before data enhancement and the face region after data enhancement respectively;

[0027] the calculating self-supervision loss according to the sample eye features and the sample face features respectively through a preset self-supervision loss algorithm to obtain a sample eye self-supervision loss and a sample face self-supervision loss, comprises:

[0028] taking the eye features corresponding to the eye region before data enhancement as positive samples, taking the eye features corresponding to the eye region after data enhancement as negative samples, and calculating the sample eye self-supervision loss through a preset self-supervision loss algorithm;

[0029] taking the face features corresponding to the face region before data enhancement as positive samples, taking the face features corresponding to the face region after data enhancement as negative samples, and calculating the sample face self-supervision loss through a preset self-supervision loss algorithm.

[0030] Optionally, the inputting the sample face image into the feature extraction network to be trained, and performing feature extraction on the eye region and the face region in the sample face image respectively to obtain sample eye features and sample face features, comprises:

[0031] For any sample character, the image reconstruction is performed on the plurality of sample face images corresponding to the sample character in the sample face image, to obtain a plurality of reconstructed images, wherein the sample face image includes a plurality of images corresponding to the same character;

[0032] The feature extraction is performed on the eye region and the face region in the reconstructed image, to obtain a sample eye feature and a sample face feature;

[0033] The preset self-supervised loss algorithm is used to calculate the self-supervised loss according to the sample eye feature and the sample face feature respectively, to obtain a sample eye self-supervised loss and a sample face self-supervised loss, including:

[0034] The optical flow recognition is performed according to the sample eye feature and the sample face feature, and an image is constructed according to the recognized optical flow, to obtain a constructed eye feature and a constructed face feature;

[0035] The sample eye self-supervised loss and the sample face self-supervised loss are calculated according to the constructed eye feature and the constructed face feature.

[0036] Optionally, the parameters of the to-be-trained feature extraction network are adjusted by the sample eye self-supervised loss and the sample face self-supervised loss, the parameters of the to-be-trained gaze direction estimation network are adjusted by the supervision loss, and the step of inputting the sample face image into the to-be-trained feature extraction network, performing the feature extraction on the eye region and the face region in the sample face image respectively to obtain the sample eye feature and the sample face feature is continued to be executed until a first preset iteration stop condition is met, and after the trained feature extraction network and the trained gaze direction estimation network are obtained, the method further includes:

[0037] Obtaining a target user face image;

[0038] The target user face image is input into the trained feature extraction network, and the feature extraction is performed on the eye region and the face region in the target user face image respectively to obtain a user eye feature and a user face feature;

[0039] The preset self-supervised loss algorithm is used to calculate the self-supervised loss according to the user eye feature and the user face feature respectively, to obtain a user eye self-supervised loss and a user face self-supervised loss;

[0040] adjust parameters of the trained feature extraction network according to the user eye self-supervision loss and the user face self-supervision loss, and return the step of inputting the target user face image into the trained feature extraction network to perform feature extraction on the eye region and the face region in the target user face image respectively to obtain the user eye feature and the user face feature, until a second preset iteration stop condition is met, to obtain the user feature extraction network.

[0041] In a second aspect, the embodiment of the application provides a gaze estimation model training device, which comprises:

[0042] a sample acquisition module, configured to acquire a sample face image, wherein the sample face image is a face image pre-labeled with a gaze direction;

[0043] a feature extraction module, configured to input the sample face image into a feature extraction network to be trained to perform feature extraction on an eye region and a face region in the sample face image respectively, to obtain a sample eye feature and a sample face feature;

[0044] a self-supervision loss calculation module, configured to calculate a self-supervision loss according to the sample eye feature and the sample face feature respectively by using a preset self-supervision loss algorithm, to obtain a sample eye self-supervision loss and a sample face self-supervision loss;

[0045] a gaze estimation module, configured to input the sample eye feature and the sample face feature into a gaze direction estimation network to be trained to calculate a gaze direction, to obtain an estimated gaze direction;

[0046] a supervised loss calculation module, configured to calculate a supervised loss according to the gaze direction pre-labeled in the sample face image and the estimated gaze direction;

[0047] a model output module, configured to adjust parameters of the feature extraction network to be trained according to the sample eye self-supervision loss and the sample face self-supervision loss, adjust parameters of the gaze direction estimation network to be trained according to the supervised loss, and return the step of inputting the sample face image into the feature extraction network to be trained to perform feature extraction on the eye region and the face region in the sample face image respectively to obtain the sample eye feature and the sample face feature, until a first preset iteration stop condition is met, to obtain a trained feature extraction network and a trained gaze direction estimation network.

[0048] Optionally, the feature extraction module comprises:

[0049] a normalization submodule, configured to normalize the sample face image, and determine a left eye region, a right eye region and a face region corresponding to the sample face image.

[0050] The left and right eye feature extraction sub-module is configured to perform feature extraction on the left eye region and the right eye region by using the eye feature extraction network to be trained, to obtain the sample left eye feature and the sample right eye feature.

[0051] The face feature extraction sub-module is configured to perform feature extraction on the face region by using the face feature extraction network to be trained, to obtain the sample face feature.

[0052] Optionally, the sample eye self-supervised loss includes a sample left eye self-supervised loss and a sample right eye self-supervised loss.

[0053] The self-supervised loss calculation module includes:

[0054] The left and right eye self-supervised loss calculation sub-module is configured to calculate the self-supervised loss according to the sample left eye feature and the sample right eye feature respectively by using a preset self-supervised loss algorithm, to obtain the sample left eye self-supervised loss and the sample right eye self-supervised loss.

[0055] The face self-supervised loss calculation sub-module is configured to calculate the self-supervised loss according to the sample face feature by using a preset self-supervised loss algorithm, to obtain the sample face self-supervised loss.

[0056] The model output module includes:

[0057] The eye feature extraction network adjustment sub-module is configured to adjust the parameters of the eye feature extraction network to be trained according to the sample eye self-supervised loss.

[0058] The face feature extraction network adjustment sub-module is configured to adjust the parameters of the face feature extraction network to be trained according to the sample face self-supervised loss.

[0059] Optionally, the feature extraction module includes:

[0060] The data enhancement sub-module is configured to perform data enhancement on the eye region and the face region in the sample face image respectively, to generate a self-supervised sample pair, wherein the self-supervised sample pair includes an eye sample pair and a face sample pair, the eye sample pair includes the eye region before data enhancement and the eye region after data enhancement, and the face sample pair includes the face region before data enhancement and the face region after data enhancement.

[0061] The eye feature extraction sub-module is configured to input the eye sample pair into the eye feature extraction network to be trained to perform feature extraction, to obtain the eye features corresponding to the eye region before data enhancement and the eye region after data enhancement respectively.

[0062] a face feature extraction submodule configured to perform feature extraction on the face sample pair input into the face feature extraction network to be trained to obtain face features corresponding to the face region before data augmentation and the face region after data augmentation respectively;

[0063] The self-supervised loss calculation module comprises:

[0064] A sample setting submodule is configured to take the eye features corresponding to the eye region before data augmentation as positive samples, take the eye features corresponding to the eye region after data augmentation as negative samples, and calculate the sample eye self-supervised loss through a preset self-supervised loss algorithm.

[0065] A loss calculation submodule is configured to take the face features corresponding to the face region before data augmentation as positive samples, take the face features corresponding to the face region after data augmentation as negative samples, and calculate the sample face self-supervised loss through a preset self-supervised loss algorithm.

[0066] Optionally, the feature extraction module comprises:

[0067] An image reconstruction submodule is configured to perform image reconstruction on the multiple sample face images corresponding to the sample person in the sample face image of any sample person to obtain multiple reconstructed images, wherein the sample face image comprises multiple images corresponding to the same person.

[0068] A reconstructed feature extraction submodule is configured to perform feature extraction on the eye region and the face region in the reconstructed image to obtain sample eye features and sample face features.

[0069] The self-supervised loss calculation module comprises:

[0070] A feature construction submodule is configured to perform optical flow recognition according to the sample eye features and the sample face features, and construct images according to the recognized optical flow to obtain constructed eye features and constructed face features.

[0071] A constructed loss calculation submodule is configured to calculate the sample eye self-supervised loss and the sample face self-supervised loss according to the constructed eye features and the constructed face features.

[0072] Optionally, the device further comprises:

[0073] A user image acquisition module is configured to acquire a target user face image.

[0074] A user feature extraction module is configured to input the target user face image into the trained feature extraction network, and perform feature extraction on the eye region and the face region in the target user face image respectively to obtain user eye features and user face features.

[0075] a user loss calculation module configured to calculate self-supervised losses of the user eye features and the user face features respectively by using the preset self-supervised loss algorithm, and obtain user eye self-supervised losses and user face self-supervised losses;

[0076] a user network training module configured to adjust parameters of the trained feature extraction network according to the user eye self-supervised losses and the user face self-supervised losses, and return the step of inputting the target user face image into the trained feature extraction network to extract features of an eye region and a face region in the target user face image respectively, and obtaining the user eye features and the user face features, to continue to be executed until a second preset iteration stopping condition is met, and obtain a user feature extraction network.

[0077] In a third aspect, the embodiment of the present application provides a gaze recognition system, the system comprising an image sensor and a processor;

[0078] The image sensor is configured to collect a face image and input the collected face image into the processor.

[0079] The processor is configured to receive the collected face image, and use the trained feature extraction network and the trained gaze direction estimation network obtained by any of the gaze estimation model training methods to estimate a gaze.

[0080] Optionally, the system further comprises an AR / VR glasses.

[0081] The AR / VR glasses are configured to display a specified image in a gaze direction corresponding to the gaze estimation result according to the gaze estimation result.

[0082] Optionally, the system is applied to a vehicle.

[0083] The image sensor is configured to collect a face image of a driver and input the collected face image of the driver into the processor.

[0084] The processor is configured to receive the collected face image of the driver, and use the trained feature extraction network and the trained gaze direction estimation network obtained by any of the gaze estimation model training methods to estimate a gaze of the driver.

[0085] In another aspect, the embodiment of the present application provides an electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus.

[0086] The memory is configured to store a computer program.

[0087] A processor is configured to implement any of the above line-of-sight estimation model training and correction methods when executing a program stored in a memory.

[0088] In another aspect of the embodiments of the present application, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program is configured to implement any of the above line-of-sight estimation model training and correction methods when executed by a processor.

[0089] In another aspect of the embodiments of the present application, a computer program product is provided, and the computer program product contains instructions. When the computer program product is run on a computer, the computer is caused to implement any of the above line-of-sight estimation model training and correction methods.

[0090] The embodiments of the present application have the following beneficial effects:

[0091] The gaze estimation model training method, device and system provided in the embodiments of the present application can obtain a sample face image, wherein the sample face image is a face image pre-labeled with a gaze direction; the sample face image is input into a feature extraction network to be trained to perform feature extraction on an eye region and a face region in the sample face image respectively, and sample eye features and sample face features are obtained; a preset self-supervised loss algorithm is used to calculate a sample eye self-supervised loss and a sample face self-supervised loss according to the sample eye features and the sample face features respectively; the sample eye features and the sample face features are input into a gaze direction estimation network to be trained to perform gaze direction calculation, and an estimated gaze direction is obtained; a supervision loss is calculated according to the pre-labeled gaze direction of the sample face image and the estimated gaze direction; the parameters of the feature extraction network to be trained are adjusted through the sample eye self-supervised loss and the sample face self-supervised loss, the parameters of the gaze direction estimation network to be trained are adjusted through the supervision loss, and the step of inputting the sample face image into the feature extraction network to be trained to perform feature extraction on the eye region and the face region in the sample face image respectively to obtain the sample eye features and the sample face features is continued to be executed until a first preset iteration stop condition is met, and a trained feature extraction network and a trained gaze direction estimation network are obtained. Through the method of the embodiments of the present application, after obtaining the face image pre-labeled with the gaze direction, not only can the supervision loss be calculated according to the pre-labeled gaze direction of the sample face image and the estimated gaze direction, but also the sample eye self-supervised loss and the sample face self-supervised loss can be calculated through the preset self-supervised loss algorithm, so that the parameters of the feature extraction network to be trained are adjusted through the sample eye self-supervised loss and the sample face self-supervised loss, the parameters of the gaze direction estimation network to be trained are adjusted through the supervision loss, the training efficiency of the model is improved, and the problem of low training efficiency caused by the adjustment of the parameters of the feature extraction network and the gaze direction estimation network at the same time through the calculation loss during the training of the current gaze estimation model is solved.

[0092] Of course, implementing any of the products or methods of the present application does not necessarily require all the advantages described above to be achieved at the same time. BRIEF DESCRIPTION OF DRAWINGS

[0093] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other embodiments can also be obtained by those skilled in the art based on these drawings.

[0094] Figure 1 A flowchart of a gaze estimation model training method provided in the embodiments of the present application;

[0095] Figure 2 A flowchart of a process for obtaining eye features and face features is provided for the embodiments of the present application.

[0096] Figure 3 An example diagram of gaze direction prediction is provided for the embodiments of the present application.

[0097] Figure 4 A flowchart of a process for adjusting parameters of a model is provided for the embodiments of the present application.

[0098] Figure 5 A flowchart of a process for calculating a self-supervised loss is provided for the embodiments of the present application.

[0099] Figure 6 A flowchart of a process for fine-tuning a pre-trained gaze direction prediction model is provided for the embodiments of the present application.

[0100] Figure 7 An example diagram of fine-tuning a pre-trained gaze direction prediction model is provided for the embodiments of the present application.

[0101] Figure 8 An example diagram of gaze estimation is provided for the embodiments of the present application.

[0102] Figure 9 A structural diagram of a gaze estimation model training apparatus is provided for the embodiments of the present application.

[0103] Figure 10 A structural diagram of a gaze estimation system is provided for the embodiments of the present application.

[0104] Figure 11 A structural diagram of an electronic device is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0105] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art based on the present application belong to the scope of protection of the present application.

[0106] First, the professional terms that may be used in the embodiments of the present application are explained:

[0107] Gaze estimation: estimating the three-dimensional visual fixation direction of a person according to the imaging of a face and eyes.

[0108] Personalized correction: fine-tuning a general model using a small number of user samples to overcome individual differences of users and improve the performance of the gaze model for the individual user.

[0109] The first aspect of the embodiments of the present application first provides a line-of-sight estimation model training method, which is described below with reference to Figure 1 , Figure 1 A flowchart of the line-of-sight estimation model training method provided by the embodiments of the present application is shown in FIG. 1. The method includes the following steps.

[0110] In step S11, a sample face image is obtained.

[0111] The sample face image is a face image pre-labeled with a line-of-sight direction. The face image in the embodiments of the present application can include a face region and an eye region, and specifically can include a left eye region and a right eye region. In actual use, the sample face image can include face images of multiple sample tasks, and specifically, one or more images can be obtained for each sample person.

[0112] The method of the embodiments of the present application is applied to a smart terminal, and can be implemented by the smart terminal. Specifically, the smart terminal can be a computer, a mobile phone, a server, or the like. In actual use, the smart terminal and the device for line-of-sight prediction can be the same device or two different devices.

[0113] In step S12, the sample face image is input into a feature extraction network to be trained, and feature extraction is performed on the eye region and the face region in the sample face image to obtain sample eye features and sample face features.

[0114] The line-of-sight estimation model in the embodiments of the present application can include a feature extraction network and a line-of-sight estimation network. The eye region and the face region in the sample face image are extracted, and the eye region and the face region are extracted to obtain sample eye features and sample face features. Specifically, feature extraction can be performed by SIFT (Scale-Invariant Feature Transform), HOG (Histogram of Oriented Gradients), or the like.

[0115] In step S13, a preset self-supervised loss algorithm is used to calculate a self-supervised loss based on the sample eye features and the sample face features to obtain sample eye self-supervised loss and sample face self-supervised loss.

[0116] The self-supervised loss calculation in the embodiments of the present application can be performed without obtaining the label of the sample image. Specifically, the current loss can be calculated by a contrastive learning method, based on optical flow reconstructed images, and by feature decoupling reconstructed images.

[0117] In step S14, the sample eye features and the sample face features are input into a line-of-sight direction estimation network to be trained to calculate a line-of-sight direction to obtain an estimated line-of-sight direction.

[0118] In the embodiment of the present application, the line-of-sight direction estimation network to be trained can predict the line-of-sight direction of the face in the corresponding sample image according to the sample eye feature and the sample face feature. Specifically, the angle or position of the predicted line-of-sight can be calculated.

[0119] In step S15, the supervision loss is calculated according to the pre-labeled line-of-sight direction of the sample face image and the estimated line-of-sight direction.

[0120] In the embodiment of the present application, the supervision loss is calculated according to the pre-labeled line-of-sight direction of the sample face image and the estimated line-of-sight direction. For example, when the pre-labeled line-of-sight direction is g and the estimated line-of-sight direction is The supervision loss can be calculated by a preset formula

[0121] In step S16, the parameters of the feature extraction network to be trained are adjusted by the sample eye self-supervised loss and the sample face self-supervised loss, the parameters of the line-of-sight direction estimation network to be trained are adjusted by the supervision loss, and the step of inputting the sample face image into the feature extraction network to be trained to extract features of the eye region and the face region in the sample face image to obtain the sample eye feature and the sample face feature is continued to be executed until the first preset iteration stop condition is met, and the trained feature extraction network and the trained line-of-sight direction estimation network are obtained.

[0122] In the embodiment of the present application, the parameters of the feature extraction network to be trained are adjusted by the sample eye self-supervised loss and the sample face self-supervised loss, the parameters of the line-of-sight direction estimation network to be trained are adjusted by the supervision loss, and the parameters of the feature extraction network to be trained and the parameters of the line-of-sight direction estimation network to be trained can be modified by a back propagation optimization algorithm, and then the step S12 is continued to be executed until a preset iteration number is reached, and the trained feature extraction network and the trained line-of-sight direction estimation network are obtained.

[0123] In the embodiment of the present application, the entire model can be regarded as a whole, and the joint training of supervision and self-supervision is performed, so the adjustment of the parameters of the feature extraction network to be trained by the sample eye self-supervised loss and the sample face self-supervised loss and the adjustment of the parameters of the line-of-sight direction estimation network to be trained by the supervision loss can be performed simultaneously.

[0124] By the method of the embodiment of the present application, after obtaining the pre-labeled face image with the gaze direction, not only the supervision loss can be calculated according to the pre-labeled gaze direction and the estimated gaze direction of the sample face image, but also the sample eye self-supervised loss and the sample face self-supervised loss can be calculated through the preset self-supervised loss algorithm, so as to adjust the parameters of the feature extraction network to be trained through the sample eye self-supervised loss and the sample face self-supervised loss, and adjust the parameters of the gaze direction estimation network to be trained through the supervision loss, thereby improving the training efficiency of the model.

[0125] Optionally, the sample eye features include sample left eye features and sample right eye features, referring to Figure 2 , step S12 inputs the sample face image into the feature extraction network to be trained, and performs feature extraction on the eye region and the face region in the sample face image respectively to obtain sample eye features and sample face features, including:

[0126] Step S121, normalizing the sample face image to determine the corresponding left eye region, right eye region and face region;

[0127] Step S122, performing feature extraction on the left eye region and the right eye region through the eye feature extraction network to be trained to obtain sample left eye features and sample right eye features;

[0128] Step S123, performing feature extraction on the face region through the face feature extraction network to be trained to obtain sample face features.

[0129] In the embodiment of the present application, the order of step S122 and step S123 is not limited.

[0130] In the embodiment of the present application, the sample face image is normalized to determine the corresponding left eye region, right eye region and face region. After the sample face image is normalized, the normalized image can be identified for the left eye region, right eye region and face region, so as to determine the left eye region, right eye region and face region in the sample face image. The sample face image can be normalized to perform a series of standard processing and transformation, and be transformed into a fixed standard form. Specifically, it can include coordinate centering, x-shearing normalization, scaling normalization and rotation normalization, etc.

[0131] In this embodiment, feature extraction for the left and right eye regions can be performed using the same feature extraction network, such as an eye feature extraction network to be trained, to obtain sample left-eye and sample right-eye features. Furthermore, feature extraction for the face region is performed using a face feature extraction network to be trained, resulting in sample face features. The aforementioned eye feature extraction network and face feature extraction network to be trained can be different feature extraction networks.

[0132] Among them, see Figure 3 The self-supervised part in this embodiment consists of three parts: left and right eye self-supervision and face self-supervision. The total loss function is L = L face +L right-eye +L left-eye +L gaze , where L face For facial loss, L right-eye For right eye loss, L left-eye For the loss of the left eye, L gaze To monitor losses.

[0133] As can be seen, the method of this application embodiment can extract eye features and face features separately through different feature extraction networks to obtain sample left eye features, sample right eye features and sample face features. This not only improves the efficiency of feature extraction, but also improves the efficiency of gaze prediction by extracting features for the left and right eyes separately when predicting gaze using a trained model.

[0134] Optionally, the self-supervised loss for the sample eyes includes the self-supervised loss for the left eye and the self-supervised loss for the right eye; see [link to relevant documentation]. Figure 4 By using a pre-defined self-supervised loss algorithm, self-supervised losses are calculated based on the sample eye features and sample face features, respectively, to obtain the sample eye self-supervised loss and sample face self-supervised loss, including:

[0135] Step S41: Using a preset self-supervised loss algorithm, calculate the self-supervised loss based on the left eye features and right eye features of the sample respectively to obtain the self-supervised loss of the left eye and the self-supervised loss of the right eye of the sample; Using a preset self-supervised loss algorithm, calculate the self-supervised loss based on the facial features of the sample respectively to obtain the self-supervised loss of the face of the sample.

[0136] The parameters of the feature extraction network to be trained are adjusted using sample eye self-supervised loss and sample face self-supervised loss, including:

[0137] Step S42, the parameters of the eye feature extraction network to be trained are adjusted through the sample eye self-supervision loss; and the parameters of the face feature extraction network to be trained are adjusted through the sample face self-supervision loss.

[0138] The self-supervision algorithm in the embodiments of the present application can be various algorithms, for example, can be contrast learning and image reconstruction, etc. Among them, the contrast learning can use data enhancement to construct a sample pair (the same image before and after enhancement belongs to the same class, and different images belong to different classes), a feature extractor extracts sample features (there will be differences between the features after the samples pass through the feature extractor), a classification loss is calculated according to the sample pair relationship (belongs to which class), and the parameters of the feature extractor are updated and optimized by gradient. Among them, the image reconstruction can construct a sample pair A and B at random, a feature extractor extracts sample features (there will be differences between the features after the samples pass through the feature extractor), the feature difference of the sample pair is decoded into an optical flow, the sample B can be reconstructed using the optical flow, the reconstruction loss of B and B' is calculated, and the parameters of the feature extractor are updated and optimized by gradient.

[0139] Specifically, referring to Figure 5 , Single-source architecture, in the figure, Source frame s and Target frame t are a sample pair, through a feature extractor (or an encoder) Encoder, and an embedding operation embeding v t , embeding v s , the features Vs and Vt are obtained, then the features of the sample pair are sent into a decoder Decoder to be decoded into an optical flow (the optical flow can express how to transform from frame s to frame t), then frame s is added with the optical flow, and frame s' can be reconstructed through Generated frame s', the difference between frame s and s' is calculated, which is the reconstruction loss, and the parameters of the feature extractor and the decoder are updated and optimized by gradient. In the embodiments of the present application, the eye reconstruction is similar to the face, and can be referred to each other.

[0140] In the embodiments of the present application, the preset self-supervised algorithm can use various types of self-supervised learning methods, such as various contrast learning methods, reconstructing images according to optical flow, and reconstructing images through feature decoupling. Through the self-supervised method, the feature extraction capability of the network can be improved without line-of-sight labels. For example, self-supervised samples are obtained in various ways of data augmentation expansion according to the provided samples, and then the positive and negative relationships of these samples are defined. Then, the loss amount can be calculated using these positive and negative samples, and then the feature extraction model is optimized. For another example, the same individual's eye or face can be used to form a random sample pair. According to the feature difference, the optical flow is generated, and then the sample pair can be used to reconstruct the other image using the optical flow. Finally, the image difference is calculated to form a reconstruction loss, and then the part of the network branch is optimized.

[0141] Optionally, the sample face image is input into the feature extraction network to be trained, and the eye region and the face region in the sample face image are respectively subjected to feature extraction to obtain sample eye features and sample face features, including: respectively performing data augmentation on the eye region and the face region in the sample face image to generate a self-supervised sample pair, wherein the self-supervised sample pair includes an eye sample pair and a face sample pair, the eye sample pair includes the eye region before data augmentation and the eye region after data augmentation, and the face sample pair includes the face region before data augmentation and the face region after data augmentation; the eye sample pair is input into the eye feature extraction network to be trained for feature extraction to obtain the eye features corresponding to the eye region before data augmentation and the eye region after data augmentation respectively; and the face sample pair is input into the face feature extraction network to be trained for feature extraction to obtain the face features corresponding to the face region before data augmentation and the face region after data augmentation respectively.

[0142] Through the preset self-supervised loss algorithm, the sample eye self-supervised loss and the sample face self-supervised loss are calculated according to the sample eye features and the sample face features respectively, including: taking the eye features corresponding to the eye region before data augmentation as positive samples, and taking the eye features corresponding to the eye region after data augmentation as negative samples, and calculating the sample eye self-supervised loss through the preset self-supervised loss algorithm; and taking the face features corresponding to the face region before data augmentation as positive samples, and taking the face features corresponding to the face region after data augmentation as negative samples, and calculating the sample face self-supervised loss through the preset self-supervised loss algorithm.

[0143] Optionally, the sample face image is input into the feature extraction network to be trained, and feature extraction is performed on the eye region and the face region in the sample face image respectively to obtain sample eye features and sample face features, including: for each sample person, performing image reconstruction on the plurality of sample face images corresponding to the sample person in the sample face image to obtain a plurality of reconstructed images, wherein the sample face image includes a plurality of images corresponding to the same person; performing feature extraction on the eye region and the face region in the reconstructed image to obtain sample eye features and sample face features.

[0144] Through a preset self-supervised loss algorithm, self-supervised loss calculation is performed according to the sample eye features and the sample face features respectively to obtain sample eye self-supervised loss and sample face self-supervised loss, including: performing optical flow recognition according to the sample eye features and the sample face features, and constructing an image according to the recognized optical flow to obtain constructed eye features and constructed face features; and calculating the sample eye self-supervised loss and the sample face self-supervised loss according to the constructed eye features and the constructed face features.

[0145] Optionally, referring to Figure 6 , the parameters of the feature extraction network to be trained are adjusted through the sample eye self-supervised loss and the sample face self-supervised loss, the parameters of the gaze direction estimation network to be trained are adjusted through the supervised loss, and the step of inputting the sample face image into the feature extraction network to be trained to perform feature extraction on the eye region and the face region in the sample face image respectively to obtain sample eye features and sample face features is continued to be executed until a first preset iteration stop condition is met, and after the trained feature extraction network and the trained gaze direction estimation network are obtained, the method further includes:

[0146] Step S61, obtaining a target user face image;

[0147] Step S62, inputting the target user face image into the trained feature extraction network to perform feature extraction on the eye region and the face region in the target user face image respectively to obtain user eye features and user face features;

[0148] Step S63, performing self-supervised loss calculation according to the user eye features and the user face features respectively through a preset self-supervised loss algorithm to obtain user eye self-supervised loss and user face self-supervised loss;

[0149] Step S64, the parameters of the trained feature extraction network are adjusted according to the user eye self-supervised loss and the user face self-supervised loss, and the target user face image is input into the trained feature extraction network, and the feature extraction of the eye region and the face region in the target user face image is continued to be performed to obtain the user eye feature and the user face feature until the second preset iteration stop condition is met, and the user feature extraction network is obtained.

[0150] Wherein, the target user can refer to a user to be estimated, for example, when the method of the embodiment of the application is applied to a vehicle, the face image of the driver can be obtained, and then the line of sight of the driver is estimated.

[0151] Wherein, referring to Figure 7 The loss calculated in the embodiment of the application can be composed of 3 parts of self-supervised loss, as shown in the following formula:

[0152] L = L face + L right-eye + L left-eye By the method of the embodiment of the application, not only the eye feature of the user can be extracted when the line of sight is predicted, but also the face feature of the user can be extracted. Since the feature of the face often changes when the line of sight of the user changes, for example, the angle, etc., therefore, after the eye feature and the face feature of the target user are obtained, the line of sight direction is predicted by the eye feature and the face feature of the target user, which can improve the efficiency and accuracy of the prediction.

[0153] In actual use, the method of the embodiment of the application can include three processes of model training, correction and use, which can be performed in sequence. The training process can obtain a trained feature extraction network and a trained line of sight direction estimation network by the method of the embodiment of the application. The correction process can adjust the parameters of the pre-trained feature extraction network by the target user image, and obtain a user feature extraction network, as described in the above embodiment. The use process can obtain the face image of the user in real time, and use the obtained user feature extraction network and the trained line of sight direction estimation network to predict the line of sight of the user.

[0154] In order to illustrate the method of the embodiment of the application, the following will be described in combination with specific embodiments, referring to Figure 8 :

[0155] The whole scheme flow is divided into three steps:

[0156] In a first step, a general gaze estimation model is obtained by joint training of supervised and self-supervised manners using labeled gaze data. Specifically, referring to the first embodiment described above, the sample face image is input into the feature extraction network to be trained, and the eye region and face region in the sample face image are respectively subjected to feature extraction to obtain sample eye features and sample face features. The parameters of the feature extraction network to be trained are adjusted through the sample eye self-supervised loss and the sample face self-supervised loss, the parameters of the gaze direction estimation network to be trained are adjusted through the supervised loss, and the process is iterated to obtain the trained feature extraction network and the trained gaze direction estimation network.

[0157] In a second step, a plurality of frames of unlabeled data of the user are obtained without the need for the user to cooperate and without the need for professional acquisition equipment. Then, the model is fine-tuned through self-supervised learning using the unlabeled data to obtain a personalized gaze estimation model. Specifically, referring to the previous embodiment, the face image of the target user is obtained, the face image of the target user is input into the trained feature extraction network, and the eye region and face region in the face image of the target user are respectively subjected to feature extraction to obtain user eye features and user face features. Then, the user eye self-supervised loss and the user face self-supervised loss are calculated according to the user eye features and the user face features through a preset self-supervised loss algorithm. Finally, the parameters of the trained feature extraction network are adjusted according to the user eye self-supervised loss and the user face self-supervised loss, and the process is iterated to obtain the user feature extraction network.

[0158] In a third step, the user uses the fine-tuned personalized model to predict the gaze of the user. Specifically, the user feature extraction network and the trained gaze direction estimation network obtained in the previous step are used to predict the gaze of the user. Since the feature extraction network is fine-tuned through the face image of the user in the previous step, the features of the user can be better captured, thereby improving the prediction accuracy of the gaze of the user.

[0159] In a second aspect, a gaze estimation model training device is provided, referring to Figure 9 , comprising:

[0160] The sample acquisition module 901 is configured to acquire a sample face image, wherein the sample face image is a face image pre-labeled with a gaze direction.

[0161] The feature extraction module 902 is configured to input the sample face image into a feature extraction network to be trained, and respectively extract features of an eye region and a face region in the sample face image to obtain sample eye features and sample face features.

[0162] The self-supervised loss calculation module 903 is configured to calculate sample eye self-supervised loss and sample face self-supervised loss according to the sample eye feature and the sample face feature respectively by using a preset self-supervised loss algorithm.

[0163] The gaze estimation module 904 is configured to input the sample eye feature and the sample face feature into a gaze direction estimation network to be trained to calculate the gaze direction, and obtain an estimated gaze direction.

[0164] The supervised loss calculation module 905 is configured to calculate the supervised loss according to the pre-labeled gaze direction of the sample face image and the estimated gaze direction.

[0165] The model output module 906 is configured to adjust the parameters of the feature extraction network to be trained by using the sample eye self-supervised loss and the sample face self-supervised loss, adjust the parameters of the gaze direction estimation network to be trained by using the supervised loss, and return to the step of inputting the sample face image into the feature extraction network to be trained to extract features of the eye region and the face region in the sample face image, and obtain the sample eye feature and the sample face feature, until a first preset iteration stop condition is met, and obtain the trained feature extraction network and the trained gaze direction estimation network.

[0166] Optionally, the feature extraction module comprises:

[0167] The normalization submodule is configured to normalize the sample face image, and determine the left eye region, the right eye region, and the face region.

[0168] The left and right eye feature extraction submodule is configured to extract features of the left eye region and the right eye region by using the eye feature extraction network to be trained, and obtain the sample left eye feature and the sample right eye feature.

[0169] The face feature extraction submodule is configured to extract features of the face region by using the face feature extraction network to be trained, and obtain the sample face feature.

[0170] Optionally, the sample eye self-supervised loss comprises a sample left eye self-supervised loss and a sample right eye self-supervised loss.

[0171] The self-supervised loss calculation module comprises:

[0172] The left and right eye self-supervised loss calculation submodule is configured to calculate the sample left eye self-supervised loss and the sample right eye self-supervised loss according to the sample left eye feature and the sample right eye feature respectively by using a preset self-supervised loss algorithm.

[0173] The face self-supervision loss calculation submodule is configured to calculate sample face self-supervision losses according to the sample face features respectively by using a preset self-supervision loss algorithm.

[0174] The model output module comprises:

[0175] The eye feature extraction network adjustment submodule is configured to adjust parameters of the eye feature extraction network to be trained according to the sample eye self-supervision loss.

[0176] The face feature extraction network adjustment submodule is configured to adjust parameters of the face feature extraction network to be trained according to the sample face self-supervision loss.

[0177] Optionally, the feature extraction module comprises:

[0178] The data enhancement submodule is configured to perform data enhancement on the eye region and the face region in the sample face image respectively to generate a self-supervision sample pair, wherein the self-supervision sample pair comprises an eye sample pair and a face sample pair, the eye sample pair comprises the eye region before data enhancement and the eye region after data enhancement, and the face sample pair comprises the face region before data enhancement and the face region after data enhancement.

[0179] The eye feature extraction submodule is configured to input the eye sample pair into the eye feature extraction network to be trained to perform feature extraction to obtain eye features corresponding to the eye region before data enhancement and the eye region after data enhancement respectively.

[0180] The face feature extraction submodule is configured to input the face sample pair into the face feature extraction network to be trained to perform feature extraction to obtain face features corresponding to the face region before data enhancement and the face region after data enhancement respectively.

[0181] The self-supervision loss calculation module comprises:

[0182] The sample setting submodule is configured to take the eye features corresponding to the eye region before data enhancement as positive samples, take the eye features corresponding to the eye regions after data enhancement as negative samples, and calculate the sample eye self-supervision loss by using a preset self-supervision loss algorithm.

[0183] The loss calculation submodule is configured to take the face features corresponding to the face region before data enhancement as positive samples, take the face features corresponding to the face regions after data enhancement as negative samples, and calculate the sample face self-supervision loss by using a preset self-supervision loss algorithm.

[0184] Optionally, the feature extraction module comprises:

[0185] an image reconstruction submodule, configured to perform image reconstruction on a plurality of sample face images corresponding to the sample person in the sample face image of any sample person, to obtain a plurality of reconstructed images, wherein the sample face image comprises a plurality of images corresponding to the same person;

[0186] a reconstructed feature extraction submodule, configured to perform feature extraction on the eye region and the face region in the reconstructed image, to obtain sample eye features and sample face features;

[0187] a self-supervised loss calculation module, comprising:

[0188] a feature construction submodule, configured to perform optical flow recognition according to the sample eye features and the sample face features, and construct an image according to the recognized optical flow, to obtain constructed eye features and constructed face features;

[0189] a constructed loss calculation submodule, configured to calculate sample eye self-supervised loss and sample face self-supervised loss according to the constructed eye features and the constructed face features.

[0190] Optionally, the apparatus further comprises:

[0191] a user image acquisition module, configured to acquire a target user face image;

[0192] a user feature extraction module, configured to input the target user face image into the trained feature extraction network, and perform feature extraction on an eye region and a face region in the target user face image, to obtain user eye features and user face features;

[0193] a user loss calculation module, configured to calculate self-supervised loss according to the user eye features and the user face features by using a preset self-supervised loss algorithm, to obtain user eye self-supervised loss and user face self-supervised loss;

[0194] a user network training module, configured to adjust parameters of the trained feature extraction network according to the user eye self-supervised loss and the user face self-supervised loss, and continue to perform the steps of inputting the target user face image into the trained feature extraction network, and performing feature extraction on the eye region and the face region in the target user face image, to obtain the user eye features and the user face features, until a second preset iteration stopping condition is met, to obtain a user feature extraction network.

[0195] Through the device of the embodiment of the present application, after a face image pre-labeled with a line-of-sight direction is acquired, not only can a supervision loss be calculated according to the pre-labeled line-of-sight direction and the estimated line-of-sight direction of the sample face image, but also sample eye self-supervision loss and sample face self-supervision loss can be calculated through a preset self-supervision loss algorithm, so as to adjust the parameters of the feature extraction network to be trained through the sample eye self-supervision loss and the sample face self-supervision loss, and adjust the parameters of the line-of-sight direction estimation network to be trained through the supervision loss, thereby improving the training efficiency of the model.

[0196] In a third aspect, the embodiment of the present application provides a line-of-sight recognition system, referring to Figure 10 The system comprises an image sensor 1001 and a processor 1002.

[0197] The image sensor 1001 is configured to collect a face image and input the collected face image to the processor.

[0198] The processor 1002 is configured to receive the collected face image and perform line-of-sight estimation by using a trained feature extraction network and a trained line-of-sight direction estimation network trained by any of the line-of-sight estimation model training methods.

[0199] Optionally, the system further comprises AR / VR (augmented reality / virtual reality) glasses.

[0200] The AR / VR glasses are configured to display a specified image in a line-of-sight direction corresponding to the line-of-sight estimation result according to the line-of-sight estimation result.

[0201] Optionally, the system is applied to a vehicle.

[0202] The image sensor is configured to collect a face image of a driver and input the collected face image of the driver to the processor.

[0203] The processor is configured to receive the collected face image of the driver and perform line-of-sight estimation of the driver by using a trained feature extraction network and a trained line-of-sight direction estimation network trained by any of the line-of-sight estimation model training methods.

[0204] When the system of this application embodiment is applied to a vehicle, it can be applied to a smart cockpit. The system collects the driver's facial image and inputs the collected driver's facial image into a processor. The processor receives the collected driver's facial image and uses a trained feature extraction network and a trained gaze direction estimation network trained by any of the above-mentioned gaze estimation model training methods to estimate the driver's gaze. Based on the gaze estimation result, a driving-specific image is displayed, such as displaying vehicle driving information or warning information in the driver's gaze direction, or issuing a warning when the driver's gaze is detected to deviate from a preset direction for a long time, thereby improving vehicle safety.

[0205] This application also provides an electronic device, such as... Figure 11 As shown, it includes a processor 1101, a communication interface 1102, a memory 1103, and a communication bus 1104. The processor 1101, communication interface 1102, and memory 1103 communicate with each other via the communication bus 1104.

[0206] Memory 1103 is used to store computer programs;

[0207] When processor 1101 executes the program stored in memory 1103, it performs the following steps:

[0208] Obtain the target user's image;

[0209] The target user image is input into a pre-trained feature extraction network to obtain the target user's target features. The pre-trained gaze direction prediction model is a network model trained on sample face images of one or more people other than the target user.

[0210] Input the target features into the pre-trained gaze direction prediction model to obtain the current predicted gaze direction;

[0211] The current loss is calculated using a pre-defined self-supervised algorithm. Based on the current loss, the parameters of the pre-trained feature extraction network are adjusted. The process of inputting the target user image into the pre-trained feature extraction network to obtain the target user's target features continues until the adjusted network model is obtained.

[0212] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0213] The communication interface is used for communication between the above electronic device and other devices.

[0214] The memory can include a Random Access Memory (RAM) and can also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0215] The processor mentioned above can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0216] In another embodiment provided in the present application, a computer readable storage medium is also provided, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement any of the above line-of-sight estimation model training and correction methods.

[0217] In another embodiment provided in the present application, a computer program product containing instructions is also provided, and when the computer program product is run on a computer, the computer is caused to execute any of the line-of-sight estimation model training and correction methods in the above embodiments.

[0218] In the embodiments described above, all or some of the steps can be implemented by software, hardware or firmware, or any combination thereof. When implemented by software, all or some of the steps can be implemented in the form of one or more computer programs. The computer program can be stored in any computer readable medium, and when loaded into a computer system, can cause the computer system to perform one or more of the steps of the computer program. The computer readable medium can be a computer program product, a memory, a diskette, an optical disc, a hard disk, or the like. The computer readable medium can be distributed over network coupled (or network enabled) computer systems, so that the computer program segments can be stored across the network coupled computer systems and loaded into the computer system over the network when needed. For example, the computer readable medium can include a hard disk, a floppy disk, magnetic tape, optical data storage device, or the like. As another example, the computer readable medium excludes carrier waves.

[0219] It should be noted that, in the description, relational terms such as first and second, and the like, can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. In addition, terms such as "first" and "second" are used herein only to describe one entity's relationship to another entity (e.g., a first signal can be the first element in a list of elements), and do not necessarily imply that relationship (e.g., in a time sequence).

[0220] Each of the embodiments described in the specification is described in a related manner, and the same or similar parts of each of the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the device, system, electronic device, computer readable storage medium, and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0221] The above merely provides the preferred embodiment of the present application, and not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for training a gaze estimation model, characterized in that, The method comprises the following steps: obtaining a sample face image, wherein the sample face image is a face image pre-labeled with a line-of-sight direction; inputting the sample face image into a feature extraction network to be trained to respectively extract features of an eye region and a face region in the sample face image, and obtain sample eye features and sample face features; calculating a sample eye self-supervised loss and a sample face self-supervised loss according to the sample eye features and the sample face features respectively by using a preset self-supervised loss algorithm; calculating an estimated line-of-sight direction by inputting the sample eye features and the sample face features into a line-of-sight direction estimation network to be trained; calculating a supervision loss according to the pre-labeled line-of-sight direction of the sample face image and the estimated line-of-sight direction; adjusting parameters of the feature extraction network to be trained according to the sample eye self-supervised loss and the sample face self-supervised loss, adjusting parameters of the line-of-sight direction estimation network to be trained according to the supervision loss, and returning to the step of inputting the sample face image into the feature extraction network to be trained to respectively extract features of the eye region and the face region in the sample face image, and obtain sample eye features and sample face features, until a first preset iteration stop condition is met, and a trained feature extraction network and a trained line-of-sight direction estimation network are obtained.

2. The method of claim 1, wherein, The sample eye features comprise sample left eye features and sample right eye features, and the step of inputting the sample face image into the feature extraction network to be trained to respectively extract features of the eye region and the face region in the sample face image, and obtain sample eye features and sample face features comprises the following steps: normalizing the sample face image to determine a left eye region, a right eye region and a face region; extracting features of the left eye region and the right eye region by using an eye feature extraction network to be trained, and obtaining the sample left eye features and the sample right eye features; extracting features of the face region by using a face feature extraction network to be trained, and obtaining the sample face features.

3. The method of claim 2, wherein, The sample eye self-supervised loss comprises a sample left eye self-supervised loss and a sample right eye self-supervised loss. The step of calculating a sample eye self-supervised loss and a sample face self-supervised loss according to the sample eye features and the sample face features respectively by using a preset self-supervised loss algorithm comprises the following steps: calculating the sample left eye self-supervised loss and the sample right eye self-supervised loss according to the sample left eye features and the sample right eye features respectively by using a preset self-supervised loss algorithm; calculating the sample face self-supervised loss according to the sample face features by using a preset self-supervised loss algorithm; The step of adjusting parameters of the feature extraction network to be trained according to the sample eye self-supervised loss and the sample face self-supervised loss comprises the following steps: adjusting parameters of the eye feature extraction network to be trained according to the sample eye self-supervised loss; The parameters of the face feature extraction network to be trained are adjusted through the sample face self-supervision loss.

4. The method of claim 1, wherein, The sample face image is input into the feature extraction network to be trained, and features of the eye region and the face region in the sample face image are extracted to obtain sample eye features and sample face features. The eye region and the face region in the sample face image are respectively subjected to data enhancement to generate a self-supervision sample pair, wherein the self-supervision sample pair includes an eye sample pair and a face sample pair, the eye sample pair includes the eye region before data enhancement and the eye region after data enhancement, and the face sample pair includes the face region before data enhancement and the face region after data enhancement. The eye sample pair is input into the eye feature extraction network to be trained to extract features of the eye region before data enhancement and the eye region after data enhancement. The face sample pair is input into the face feature extraction network to be trained to extract features of the face region before data enhancement and the face region after data enhancement. The sample eye features and the sample face features are used to calculate self-supervision losses through a preset self-supervision loss algorithm to obtain a sample eye self-supervision loss and a sample face self-supervision loss, including: The eye features corresponding to the eye region before data enhancement are used as positive samples, and the eye features corresponding to the eye region after data enhancement are used as negative samples to calculate the sample eye self-supervision loss through the preset self-supervision loss algorithm. The face features corresponding to the face region before data enhancement are used as positive samples, and the face features corresponding to the face region after data enhancement are used as negative samples to calculate the sample face self-supervision loss through the preset self-supervision loss algorithm.

5. The method of claim 1, wherein, The sample face image is input into the feature extraction network to be trained, and features of the eye region and the face region in the sample face image are extracted to obtain sample eye features and sample face features. For each sample person, multiple sample face images corresponding to the sample person in the sample face image are reconstructed to obtain multiple reconstructed images. Features of the eye region and the face region in the reconstructed images are extracted to obtain sample eye features and sample face features. The sample eye features and the sample face features are used to calculate self-supervision losses through a preset self-supervision loss algorithm to obtain a sample eye self-supervision loss and a sample face self-supervision loss, including: Optical flow is identified according to the sample eye features and the sample face features, and an image is constructed according to the identified optical flow to obtain constructed eye features and constructed face features. The sample eye self-supervision loss and the sample face self-supervision loss are calculated according to the constructed eye features and the constructed face features.

6. The method of claim 1, wherein, The parameters of the to-be-trained feature extraction network are adjusted through the sample eye self-supervision loss and the sample face self-supervision loss, the parameters of the to-be-trained gaze direction estimation network are adjusted through the supervision loss, and the step of inputting the sample face image into the to-be-trained feature extraction network to extract features of the eye region and the face region in the sample face image respectively to obtain the sample eye feature and the sample face feature is continued to be executed until a first preset iteration stop condition is met. After obtaining the trained feature extraction network and the trained gaze direction estimation network, the method further comprises: obtaining a target user face image; inputting the target user face image into the trained feature extraction network to extract features of the eye region and the face region in the target user face image respectively to obtain a user eye feature and a user face feature; calculating the self-supervision loss according to the user eye feature and the user face feature respectively through the preset self-supervision loss algorithm to obtain a user eye self-supervision loss and a user face self-supervision loss; adjusting the parameters of the trained feature extraction network according to the user eye self-supervision loss and the user face self-supervision loss, and continuing to execute the step of inputting the target user face image into the trained feature extraction network to extract features of the eye region and the face region in the target user face image respectively to obtain the user eye feature and the user face feature until a second preset iteration stop condition is met, and obtaining a user feature extraction network.

7. A line-of-sight estimation model training apparatus, comprising: comprise: a sample acquisition module configured to acquire a sample face image, wherein the sample face image is a face image pre-labeled with a gaze direction; a feature extraction module configured to input the sample face image into a to-be-trained feature extraction network to extract features of an eye region and a face region in the sample face image respectively to obtain a sample eye feature and a sample face feature; a self-supervision loss calculation module configured to calculate a self-supervision loss according to the sample eye feature and the sample face feature respectively through a preset self-supervision loss algorithm to obtain a sample eye self-supervision loss and a sample face self-supervision loss; a gaze estimation module configured to input the sample eye feature and the sample face feature into a to-be-trained gaze direction estimation network to calculate a gaze direction to obtain an estimated gaze direction; a supervision loss calculation module configured to calculate a supervision loss according to a pre-labeled gaze direction of the sample face image and the estimated gaze direction; The model output module is configured to adjust parameters of the to-be-trained feature extraction network based on the sample eye self-supervised loss and the sample face self-supervised loss, adjust parameters of the to-be-trained gaze direction estimation network based on the supervised loss, and return to the step of inputting the sample face image into the to-be-trained feature extraction network to extract features of an eye region and a face region in the sample face image to obtain a sample eye feature and a sample face feature, until a first preset iteration stop condition is met, and finally obtain a trained feature extraction network and a trained gaze direction estimation network.

8. A line-of-sight recognition system, characterized by The system comprises an image sensor and a processor; The image sensor is configured to collect a face image and input the collected face image into the processor; The processor is configured to receive the collected face image and perform gaze estimation by using the trained feature extraction network and the trained gaze direction estimation network obtained by the gaze estimation model training method according to any one of claims 1-6.

9. The system of claim 8, wherein, The system further comprises AR / VR glasses. The AR / VR glasses are configured to display a specified image in a gaze direction corresponding to the gaze estimation result according to the gaze estimation result.

10. The system of claim 8, wherein, The system is applied to a vehicle. The image sensor is configured to collect a face image of a driver and input the collected face image of the driver into the processor; The processor is configured to receive the collected face image of the driver and perform gaze estimation of the driver by using the trained feature extraction network and the trained gaze direction estimation network obtained by the gaze estimation model training method according to any one of claims 1-6.

Images