Line of sight estimation method, device and storage medium

Abstract

The present application provides a line of sight estimation method, device and storage medium. The method includes: obtaining source domain data with labeled information and target domain data without labeled information; training the training set data in the source domain data to obtain a line of sight estimation model; adding adversarial noise to the source domain data and the target domain data using adversarial attacks; performing comparative learning on the source domain data before and after adding the adversarial noise, and performing comparative learning on the target domain data at the same time, updating the line of sight estimation model; optimizing the line of sight estimation model using adversarial training methods. The method provided in the present application adds adversarial noise related to the line of sight estimation task to the source domain data and the target domain data through adversarial attacks, thereby improving the correlation between the features extracted by the line of sight estimation model during the adversarial training process and the line of sight estimation task, and at the same time eliminating the influence of the adversarial noise on the line of sight estimation model through comparative learning, thereby improving the accuracy of the line of sight estimation model in line of sight estimation of the target domain data.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a line of sight estimation method, device and storage medium. Background Art

[0002] Gaze is an important indicator of human attention, indicating the direction a person is looking at. Gaze information has been widely used in various human-computer interaction applications, such as virtual / augmented reality games, smart cockpit systems, medical analysis, gaze / head redirection, etc. In recent years, due to the release of multiple gaze estimation datasets, the gaze estimation model trained based on a certain dataset, i.e., source domain data, has low accuracy in predicting other datasets, i.e., target domain data, and has cross-domain problems.

[0003] The existing method to solve the cross-domain problem is based on adversarial training. The core of the adversarial training method is to introduce a domain discriminator to distinguish image data from different domains, so that the line of sight estimation model can extract features that are sufficient to confuse the discriminator, thereby making the features extracted by the line of sight estimation model more universal.

[0004] When using adversarial training to solve cross-domain problems, the existing technology uses a neural network model to estimate the line of sight of the target domain data, and the obtained line of sight estimation results have low accuracy. Summary of the invention

[0005] The present application provides a line of sight estimation method, device and storage medium, which are used to solve the problem that the line of sight estimation result obtained by a neural network model for line of sight estimation on target domain data is of low accuracy due to cross-domain problems.

[0006] In a first aspect, the present application provides a line of sight estimation method, comprising:

[0007] Obtain source domain data with labeled information and target domain data without labeled information;

[0008] Train the training set data in the source domain data to obtain a line of sight estimation model;

[0009] Using adversarial attacks to add adversarial noise to source domain data and target domain data;

[0010] Perform comparative learning on the source domain data before and after adding adversarial noise, and also perform comparative learning on the target domain data to update the line of sight estimation model;

[0011] Adversarial training method is used to optimize the gaze estimation model.

[0012] In a second aspect, the present application provides a sight line estimation device, comprising:

[0013] An acquisition module is used to acquire source domain data with labeled information and target domain data without labeled information;

[0014] A training module is used to train the training set data in the source domain data to obtain a sight estimation model;

[0015] Add a module for adding adversarial noise to source domain data and target domain data using adversarial attacks;

[0016] The contrastive learning module performs contrastive learning on the source domain data before and after adding adversarial noise, and also performs contrastive learning on the target domain data to update the line of sight estimation model.

[0017] Optimization module: Use adversarial training method to optimize the gaze estimation model.

[0018] In a third aspect, the present application provides a line of sight estimation device, comprising: a processor and a memory, wherein the memory stores code, and the processor runs the code stored in the memory to execute a line of sight estimation method as described in any one of the first aspects.

[0019] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the line of sight estimation method as described in any one of the first aspects.

[0020] The present application provides a line of sight estimation method, device and storage medium. The method includes: obtaining source domain data and target domain data with annotated information, and training the training set data in the source domain data to obtain a line of sight estimation model. Using adversarial attack to add adversarial noise to the source domain data and the target domain data, the source domain data and the target domain data after adding the adversarial noise are obtained. Comparative learning is performed on the source domain data before and after the adversarial noise is added, and at the same time, comparative learning is performed on the target domain data before and after the adversarial noise is added, the line of sight estimation model is updated, and the influence of the adversarial noise on the line of sight estimation model is eliminated. The line of sight estimation model is optimized using an adversarial training method. The method provided in the present application adds adversarial noise related to the line of sight estimation task to the source domain data and the target domain data through adversarial attack, thereby improving the correlation between the features extracted by the line of sight estimation model during the adversarial training process and the line of sight estimation task, and at the same time, the influence of the adversarial noise on the line of sight estimation model is eliminated through comparative learning, thereby improving the accuracy of the line of sight estimation model for the line of sight estimation of the target domain data. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0022] Figure 1 A schematic diagram of a sight line estimation scenario provided in an embodiment of the present application;

[0023] Figure 2 A flow chart of a sight line estimation method provided in an embodiment of the present application;

[0024] Figure 3 A flowchart of an image counterattack method provided in an embodiment of the present application;

[0025] Figure 4 A flow chart of an image comparison learning method provided in an embodiment of the present application;

[0026] Figure 5 A flow chart of an image adversarial training method provided in an embodiment of the present application;

[0027] Figure 6 A flow chart of a method for correcting sight line estimation results provided in an embodiment of the present application;

[0028] Figure 7 A schematic diagram of a sight line estimation device provided in an embodiment of the present application Figure 1 ;

[0029] Figure 8 A schematic diagram of a sight line estimation device provided in an embodiment of the present application Figure 2 .

[0030] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0031] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0032] The task of gaze estimation is to estimate the user's gaze direction based on the user's eye or full face image taken by the device. In recent years, the common gaze estimation method is based on deep learning, which takes the user's facial image or eye image as input and directly estimates the user's gaze direction through a neural network model.

[0033] Although the line of sight estimation method based on deep learning can achieve relatively accurate results within the training set, namely the source domain data, in actual applications, due to changes in factors such as the subject, background, and lighting, when the neural network model trained based on the source domain data is applied to a new data set, namely the target domain data, the line of sight estimation result output by the neural network model for the target domain data has low accuracy and there will be cross-domain problems.

[0034] The existing method to solve the cross-domain problem is based on adversarial training, which uses a domain discriminator to distinguish image data from different domains, so that the line of sight estimation model extracts features that are sufficient to confuse the discriminator, making the features extracted by the line of sight estimation model more universal. This method uses a neural network model to estimate the line of sight of the target domain data, and the line of sight estimation result obtained is of low accuracy.

[0035] The present application provides a line of sight estimation method, which obtains source domain data with labeled information and target domain data without labeled information, trains the training set data in the source domain data, and obtains a line of sight estimation model. Since the line of sight estimation model is obtained by training the source domain data, when the line of sight estimation model performs line of sight estimation on the source domain data, the accuracy of the line of sight estimation result obtained is relatively high. When the line of sight estimation model performs line of sight estimation on the target domain data, the accuracy of the line of sight estimation result obtained is relatively low. Adversarial noise is added to the source domain data and the target domain data using an adversarial attack to obtain the source domain data and the target domain data after adding the adversarial noise, and the added adversarial noise is related to the line of sight estimation task. The source domain data before and after the adversarial noise is added are compared and learned, and the target domain data is compared and learned at the same time, the line of sight estimation model is updated, and the influence of the adversarial noise on the line of sight estimation model is eliminated. The line of sight estimation model is optimized by adversarial training, so that the line of sight estimation model completes cross-domain adaptation from the source domain data to the target domain data. The method provided in the present application adds adversarial noise related to the line of sight estimation task to the source domain data and the target domain data through an adversarial attack, thereby improving the relevance of the features extracted by the line of sight estimation model during the adversarial training process and the line of sight estimation task. At the same time, through contrastive learning, the influence of the adversarial noise on the line of sight estimation model is eliminated, thereby improving the accuracy of the line of sight estimation model for the target domain data.

[0036] Figure 1 A schematic diagram of a sight estimation scenario provided in an embodiment of the present application is shown in FIG. Figure 1As shown in the figure, the sight estimation model is obtained by training the source domain data with annotated information, and adversarial noise is added to the source domain data and the target domain data through adversarial attack. The sight estimation model is updated through comparative learning to eliminate the influence of adversarial noise on the sight estimation model. The sight estimation model is optimized by adversarial training method, so that the sight estimation model completes cross-domain adaptation from source domain data to target domain data. In order to further improve the accuracy of sight estimation of the target domain data by the sight estimation model, the sight estimation result is corrected.

[0037] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0038] Figure 2 This is a flow chart of a line of sight estimation method provided in an embodiment of the present application. The method of this embodiment can be executed by a line of sight estimation device and can be implemented by hardware, software, or a combination of hardware and software. Figure 2 As shown, the method provided by the present application may include:

[0039] S201: Acquire source domain data with labeled information and target domain data without labeled information.

[0040] The source domain data and the target domain data are images containing human eyes or human face images. The annotation information is the actual sight direction of the source domain data. Since the target domain data does not carry annotation information, it is necessary to perform sight estimation on the target domain data to obtain the sight estimation result of the target domain data.

[0041] A dataset can be selected from a public dataset as source domain data or target domain data. When selecting source domain data and target domain data, the source domain data must carry annotation information, and usually, the range of the source domain data is wider than that of the target domain data. In one implementation scenario, the ETH-XGaze dataset or the Gaze360 dataset can be selected as the source domain data, and the MPIIGaze dataset or the EyeDiap dataset can be selected as the target domain data. Among them, the ETH-XGaze dataset or the Gaze360 dataset carries annotation information, and the data range covers the range of the MPIIGaze dataset or the EyeDiap dataset.

[0042] It should be noted that since the target domain data does not carry annotation information, while the MPIIGaze dataset or the EyeDiap dataset does, when the MPIIGaze dataset or the EyeDiap dataset is selected as the target domain data, the annotation information carried by the MPIIGaze dataset or the EyeDiap dataset is not used. In addition to selecting a dataset from a public dataset as the target domain data, the target domain data can also be data from a real scene.

[0043] After obtaining the source domain data and the target domain data, data preprocessing operations can be performed, for example, normalizing the pixel values ​​of the images in the source domain data and the target domain data. To ensure that the source domain data and the target domain data have the same size, the resolution of the images in the source domain data and the target domain data can also be reset, that is, the resolution of the images in the source domain data and the target domain data is the same. For example, the pixel values ​​of the images in the source domain data and the target domain data can be normalized from [0,255] to [0,1], and the resolution is reset to 448*448.

[0044] S202: Training the training set data in the source domain data to obtain a line of sight estimation model.

[0045] Before training the source domain data, the source domain data can be divided into training set data and validation set data. The training set data is used for training to obtain a line of sight estimation model. The validation set data can be used to correct the line of sight estimation result output by the line of sight estimation model.

[0046] When dividing the source domain data into training set data and validation set data, the division can be performed according to a certain ratio, such as a ratio of 7:3 between the training set data and the validation set data. The division can also be performed according to the amount of data. For example, 10,000 images in the source domain data are used as validation set data, and the remaining images are used as training set data.

[0047] When training the training set data, the initial learning rate can be 0.00001 and the number of batch training samples can be 24.

[0048] It should be noted that since the sight line estimation model is obtained by training the source domain data, when the sight line estimation model performs sight line estimation on the source domain data, the sight line estimation result obtained has a higher accuracy. When the sight line estimation model performs sight line estimation on the target domain data, the sight line estimation result obtained has a lower accuracy.

[0049] In one implementation scenario, the sight line estimation result includes a sight line direction in the form of a vector.

[0050] S203: Add adversarial noise to the source domain data and the target domain data using adversarial attack.

[0051] In one implementation scenario, based on the gaze estimation model and the domain discriminator, adversarial noise is added to the source domain data according to the gaze estimation results of the source domain data and the domain discrimination results of the source domain data; adversarial noise is added to the target domain data according to the gaze estimation results of the target domain data and the domain discrimination results of the target domain data. Adding adversarial noise to the source domain data and the target domain data using adversarial attacks improves the relevance of the added adversarial noise to the gaze estimation task.

[0052] S204: performing comparative learning on the source domain data before and after adding the adversarial noise, and performing comparative learning on the target domain data, and updating the line of sight estimation model.

[0053] In one implementation scenario, with the intrinsic consistency of the source domain data before and after adding adversarial noise or the target domain data before and after adding adversarial noise as a constraint, positive samples and negative samples are constructed based on the source domain data and target domain data before and after adding adversarial noise, and the line of sight estimation model is updated according to the loss of the positive samples to eliminate the influence of the adversarial noise on the line of sight estimation model.

[0054] Among them, internal consistency means that the source domain data before and after adding adversarial noise is input into the gaze estimation model, and the same gaze estimation features are output; the target domain data before and after adding adversarial noise is input into the gaze estimation model, and the same gaze estimation features are obtained.

[0055] S205: Optimize the gaze estimation model using an adversarial training method.

[0056] When the line of sight estimation model estimates the line of sight of the source domain data, the line of sight estimation result has a high accuracy, while when it estimates the line of sight of the target domain data, the line of sight estimation result has a low accuracy. In order to improve the accuracy of the line of sight estimation model for the target domain data, the adversarial training method is used to optimize the line of sight estimation model so that the line of sight estimation model can complete the cross-domain adaptation from the source domain data to the target domain data.

[0057] Optionally, during the adversarial training process, the optimization goal of the gaze estimation model is opposite to the optimization goal of the domain discriminator, and the gaze estimation model is optimized so that the gaze estimation model extracts features that can confuse the domain discriminator, and the gaze estimation model can extract more universal features, thereby improving the accuracy of gaze estimation for target domain data.

[0058] To further improve the accuracy of the line of sight estimation results, the line of sight estimation results output by the line of sight estimation model can be corrected based on the validation set data in the source domain data.

[0059] The embodiment of the present application provides a line of sight estimation method, obtains source domain data with labeled information and target domain data without labeled information, trains the training set data in the source domain data, obtains a line of sight estimation model, and uses adversarial noise to add adversarial noise to the source domain data and the target domain data. The source domain data before and after the adversarial noise is added are compared and learned, and the target domain data is compared and learned at the same time, and the line of sight estimation model is updated to eliminate the influence of the adversarial noise on the line of sight estimation model. The line of sight estimation model is optimized using an adversarial training method, so that the line of sight estimation model completes cross-domain adaptation from source domain data to target domain data. The method provided in the present application adds adversarial noise related to the line of sight estimation task to the source domain data and the target domain data through adversarial attacks, thereby improving the correlation between the features extracted by the line of sight estimation model during the adversarial training process and the line of sight estimation task, and at the same time eliminates the influence of the adversarial noise on the line of sight estimation model through comparative learning, thereby improving the accuracy of the line of sight estimation model in line of sight estimation of the target domain data.

[0060] Based on the above embodiments, an embodiment is provided below to describe in detail the process of adding adversarial noise to source domain data and target domain data using adversarial attacks.

[0061] Figure 3 A flowchart of an image counterattack method provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the method provided by this application is as follows:

[0062] S301: Input source domain data into a sight line estimation model to obtain features of the source domain data and sight line estimation results of the source domain data; input target domain data into a sight line estimation model to obtain features of the target domain data and sight line estimation results of the target domain data.

[0063] The line of sight estimation model can be divided into an input layer, an intermediate layer, and an output layer. When the source domain data is input into the line of sight estimation model, the intermediate layer of the line of sight estimation model can extract the features of the source domain data, and the output layer of the line of sight estimation model outputs the line of sight estimation result corresponding to the source domain data.

[0064] In one implementation scenario, the features and sight estimation results exist in the form of vectors.

[0065] Optionally, when the source domain data and the target domain data are input into the line of sight estimation model, the amount of the source domain data and the amount of the target domain data are consistent.

[0066] S302: Input the features of the source domain data into the domain discriminator to obtain the domain discrimination result of the source domain data; input the features of the target domain data into the domain discriminator to obtain the domain discrimination result of the target domain data.

[0067] The domain discriminator is a deep neural network model that determines whether the data comes from the source domain or the target domain. For example, it can be a multiple layer perceptron (MLP).

[0068] S303: adding adversarial noise to the source domain data according to the sight line estimation result of the source domain data and the domain discrimination result of the source domain data; adding adversarial noise to the target domain data according to the sight line estimation result of the target domain data and the domain discrimination result of the target domain data.

[0069] In one implementation scenario, adversarial noise is added to source domain data and target domain data using adversarial attack as follows:

[0070] x′=x+ε·sign(▽ x [L gaze (x,y)+L dis (x)])

[0071] Where x represents the input source domain data and target domain data, x′ represents the source domain data and target domain data after adding adversarial noise, and y represents the label of the line of sight direction. The label of the target domain data adopts the line of sight estimation result output by the line of sight estimation model, and the label of the source domain data adopts the carried annotation information. ε is the seed of the adversarial noise, which can be 0.01. L gaze is the loss function, which can be the L1 loss function, L dis Can be binary cross entropy.

[0072] Adversarial noise is added to the target domain data based on the gaze estimation results and domain discrimination results of the target domain data, thereby ensuring that the added adversarial noise is relevant to the gaze estimation task.

[0073] The embodiment of the present application provides an image adversarial attack method, which inputs source domain data into a line of sight estimation model to obtain the features of the source domain data and the line of sight estimation results of the source domain data, and at the same time inputs target domain data into the line of sight estimation model to obtain the features of the target domain data and the line of sight estimation results of the target domain data. The features of the source domain data are input into a domain discriminator to obtain the domain discrimination results of the source domain data, and the features of the target domain data are input into a domain discriminator to obtain the domain discrimination results of the target domain data. According to the line of sight estimation results of the source domain data and the domain discrimination results of the source domain data, adversarial noise is added to the source domain data to generate source domain data after adding adversarial noise; according to the line of sight estimation results of the target domain data and the domain discrimination results of the target domain data, adversarial noise is added to the target domain data to generate target domain data after adding adversarial noise. The method provided in the present application is based on adding adversarial noise to the target domain data according to the line of sight estimation results and domain discrimination results of the target domain data, thereby improving the relevance of the added adversarial noise to the line of sight estimation task, which is conducive to improving the accuracy of the line of sight estimation model.

[0074] Based on the above embodiments, a specific embodiment is provided below to perform comparative learning on the source domain data before and after adding adversarial noise, and to perform comparative learning on the target domain data, and to describe in detail the process of updating the line of sight estimation model.

[0075] Figure 4 A flow chart of an image comparison learning method provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the method provided by this application is as follows:

[0076] S401: Input the source domain data before and after adding the adversarial noise into the line of sight estimation model to obtain the source domain data features before and after adding the adversarial noise; input the target domain data before and after adding the adversarial noise into the line of sight estimation model to obtain the target domain data features before and after adding the adversarial noise.

[0077] In one implementation scenario, the source domain data and the target domain data before adding the adversarial noise are both B images, and correspondingly, the source domain data and the target domain data after adding the adversarial noise are also both B images, so the source domain data and the target domain data before and after adding the adversarial noise are 4B images.

[0078] When inputting to the line of sight estimation model, As an input batch, where s represents the source domain data and t represents the target domain data, The i-th image before adding adversarial noise to the source domain data, The i-th image before adding adversarial noise to the target domain data, The i-th image after adding adversarial noise to the source domain data, The i-th image after adding adversarial noise to the target domain data.

[0079] S402: construct positive samples and negative samples based on the source domain data and the target domain data before and after adding adversarial noise.

[0080] When constructing positive samples and negative samples, based on the internal consistency of the source domain data or target domain data before and after adding the adversarial noise, any source domain data or target domain data before and after adding the adversarial noise can be used as a pair of positive samples, and the source domain data and target domain data before and after adding the adversarial noise other than the positive samples can be used as negative samples. For example, the source domain data and target domain data before and after adding the adversarial noise are 4B images, and any source domain data or target domain data before and after adding the adversarial noise is selected as a pair of positive samples, and the remaining 4B-2 images are regarded as negative samples.

[0081] It should be noted that, among the 4B images of the source domain data and the target domain data before and after adding the adversarial noise, any two images before and after adding the adversarial noise are used as positive samples once, so the number of positive samples is multiple.

[0082] The intrinsic consistency of the source domain data or target domain data before and after adding adversarial noise means that the source domain data before and after adding adversarial noise is input into the gaze estimation model, and the same gaze estimation features are output; the target domain data before and after adding adversarial noise is input into the gaze estimation model, and the same gaze estimation features are obtained. Therefore, the positive and negative samples are constructed with the intrinsic consistency of the source domain data or target domain data before and after adding adversarial noise as a constraint, which reduces the impact of adversarial noise on the gaze estimation model.

[0083] S403: Calculate the loss of the positive sample by using a first loss function according to the source domain data features before and after adding the adversarial noise, the target domain data features before and after adding the adversarial noise, the positive sample and the negative sample.

[0084] In one implementation scenario, the first loss function is as follows:

[0085]

[0086] Among them, l con represents the loss of contrastive learning, τ is the adjustment factor, which can be 0.5, pos represents the characteristics of the positive sample, where pos1 is the characteristics of the source domain data or the target domain data before adding adversarial noise to the positive sample, pos2 is the characteristics of the source domain data or the target domain data after adding adversarial noise to the positive sample, neg represents the characteristics of the negative sample, and sim represents the similarity, which can be the inner product of the vector.

[0087] Optionally, the similarity can be calculated by using the source domain data features before and after adding the adversarial noise and the target domain data features before and after adding the adversarial noise.

[0088] S404: Update the sight line estimation model according to the loss of the positive sample.

[0089] In one implementation scenario, the line of sight estimation model is updated using the gradient descent method according to the loss of the positive samples.

[0090] It should be noted that since there are multiple positive samples, the sight estimation model can be updated according to the sum of the losses of all positive samples.

[0091] The embodiment of the present application provides an image contrast learning method, which inputs the source domain data before and after adding adversarial noise into a sight estimation model to obtain the source domain data features before and after adding adversarial noise; the target domain data before and after adding adversarial noise is input into the sight estimation model to obtain the target domain data features before and after adding adversarial noise. Positive samples and negative samples are constructed based on the source domain data and target domain data before and after adding adversarial noise. According to the source domain data features before and after adding adversarial noise, the target domain data features before and after adding adversarial noise, the positive samples and the negative samples, the loss of the positive samples is calculated by a first loss function, and the sight estimation model is updated according to the loss of the positive samples. Since the contrast learning method provided by the present application is based on the intrinsic consistency of the source domain data or the target domain data before and after adding adversarial noise as a constraint, positive samples and negative samples are constructed, and the sight estimation model is updated according to the loss of the positive samples, thereby reducing the impact of adversarial noise on the sight estimation model and improving the accuracy of the sight estimation model.

[0092] Based on the above embodiments, a specific embodiment is provided below to describe in detail the process of optimizing the line of sight estimation model using the adversarial training method.

[0093] Figure 5 A flowchart of an image adversarial training method provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the method provided by this application is as follows:

[0094] S501: Input the source domain data features before and after adding the adversarial noise into the domain discriminator to obtain the domain discrimination result of the source domain data; input the target domain data features before and after adding the adversarial noise into the domain discriminator to obtain the domain discrimination result of the target domain data.

[0095] The features of the source domain data before and after adding the adversarial noise are extracted after the source domain data before and after adding the adversarial noise is input into the line of sight estimation model, and the features of the target domain data before and after adding the adversarial noise are obtained after the target domain data before and after adding the adversarial noise is input into the line of sight estimation model.

[0096] The domain discriminator is used to determine whether the input data is from the source domain or the target domain. For example, the source domain data features before and after adding adversarial noise to the domain discriminator input, the domain discrimination result output by the domain discriminator may be the source domain or the target domain. It can be understood that the target domain data features before and after adding adversarial noise to the domain discriminator input, the domain discrimination result output by the domain discriminator may be the target domain or the source domain.

[0097] In one implementation scenario, about 100 target domain images without labeled information can be used for adversarial training. The requirement on the number of target domain images is low and relatively certain, and there is no need to manually label the target domain images.

[0098] S502: Calculate a first loss through a second loss function according to the domain discrimination results of the source domain data before and after adding the adversarial noise and the domain discrimination results of the target domain data, and update the domain discriminator according to the first loss, wherein the first loss is the gap between the domain discrimination results of the source domain data before and after adding the adversarial noise and the source domain data in the source domain, and the gap between the domain discrimination results of the target domain data and the target domain data in the target domain.

[0099] Loss can refer to the difference between the value predicted by the model and the true value. Since the domain discrimination result output by the domain discriminator may be inconsistent with the domain of the input data, a certain loss will be generated. For example, when the input data of the domain discriminator is the features of the source domain data, the output domain discrimination result is the target domain, so there will be a certain gap. Similarly, when the input data of the domain discriminator is the features of the target domain data, the output domain discrimination result is the source domain, and there will also be a certain gap.

[0100] In one implementation scenario, the second loss function may be as follows:

[0101] L = -ln(1-D(f s ))-lnD(f t )-ln(1-D(f′ s ))-lnD(f′ t )

[0102] In the formula, L represents the loss, s represents the source domain data, t represents the target domain data, and f s represents the features of the source domain data before adding adversarial noise, f t represents the features of the target domain data before adding adversarial noise, f′ s represents the features of the source domain data after adding adversarial noise, f′ t represents the characteristics of the target domain data after adding adversarial noise, D(f s ) represents the domain discrimination result of the source domain data before adding adversarial noise, D(f t ) represents the domain discrimination result of the target domain data before adding adversarial noise, D(f′ s ) represents the domain discrimination result of the source domain data after adding adversarial noise, D(f′ t ) represents the domain discrimination result of the target domain data after adding adversarial noise.

[0103] The first loss calculated according to the second loss function is used to update the domain discriminator so that the domain discriminator can distinguish between source domain data and target domain data.

[0104] S503: Calculate the second loss through the third loss function according to the domain discrimination result of the target domain data before and after adding the adversarial noise, and update the line of sight estimation model according to the second loss, where the second loss is the gap between the domain discrimination result of the target domain data before and after adding the adversarial noise and the target domain data in the source domain.

[0105] In one implementation scenario, the third loss function may be as follows:

[0106] L = -ln(1-D(f t ))-ln(1-D(f′ t ))

[0107] In the formula, L represents the loss, f t represents the features of the target domain data before adding adversarial noise, f′ t represents the characteristics of the target domain data after adding adversarial noise, D(f t ) represents the domain discrimination result of the target domain data before adding adversarial noise, D(f′ t ) represents the domain discrimination result of the target domain data after adding adversarial noise.

[0108] The second loss calculated by the third loss function is used to update the line of sight estimation model, so that the line of sight estimation model can extract features that can confuse the domain discriminator, thereby enabling the line of sight estimation model to extract more universal features, bringing the feature similarity between the source domain data and the target domain data closer, thereby improving the accuracy of line of sight estimation for the target domain data.

[0109] The embodiment of the present application provides an image adversarial training method, which inputs the features of the source domain data before and after adding adversarial noise into a domain discriminator to obtain a domain discrimination result of the source domain data; and inputs the features of the target domain data before and after adding adversarial noise into a domain discriminator to obtain a domain discrimination result of the target domain data. According to the domain discrimination results of the source domain data before and after adding adversarial noise and the domain discrimination results of the target domain data, a first loss is calculated by a second loss function, and the domain discriminator is updated according to the first loss. According to the domain discrimination results of the target domain data before and after adding adversarial noise, a second loss is calculated by a third loss function, and a line of sight estimation model is updated according to the second loss function. The method provided in the present application uses the second loss to continuously update the line of sight estimation model, so that the line of sight estimation model extracts features that can confuse the domain discriminator, and then enables the line of sight estimation model to extract more universal features, which brings the feature similarity of the source domain data and the target domain data closer, and improves the accuracy of the line of sight estimation model for the target domain data.

[0110] Based on the above embodiments, a specific embodiment is provided below to describe in detail the process of correcting the sight line estimation result output by the sight line estimation model.

[0111] Figure 6 A flow chart of a method for correcting sight line estimation results provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the method provided by this application is as follows:

[0112] S601: Input the validation set data in the source domain data into the line of sight estimation model to obtain the line of sight estimation result corresponding to the validation set data.

[0113] Since the validation set data in the source domain data carries annotation information, the validation set data is selected to correct the line of sight estimation result. Optionally, about 10,000 source domain data can be selected as the validation set data.

[0114] S602: Calculate the line of sight estimation error, the mean value and the variance of the line of sight estimation error according to the line of sight estimation result corresponding to the validation set data and the annotation information carried by the validation set data.

[0115] The annotation information carried by the validation set data is the actual line of sight direction of the validation set data. When the validation set data is input into the line of sight estimation model, there will be a certain error between the line of sight estimation result and the actual line of sight direction.

[0116] S603: Correcting the line of sight estimation result corresponding to the validation set data according to the mean and variance of the line of sight estimation error.

[0117] In one implementation scenario, a method for reducing the bias and obtaining the final line of sight estimation result is defined as follows:

[0118]

[0119] Among them, ω is the adjustment factor, which can be taken as 0.5, μ is the mean of the line of sight estimation error, σ is the standard deviation of the line of sight estimation error, G(x) represents the line of sight estimation result output before correction, Represents the corrected line of sight estimation result.

[0120] In one implementation scenario, when correcting the line of sight estimation result, the sign of the mean value of the line of sight estimation error may be used as the correction direction, and the magnitude of the correction decreases as the variance of the line of sight estimation error distribution increases.

[0121] The embodiment of the present application provides a method for correcting a line of sight estimation result, inputting the validation set data in the source domain data into the line of sight estimation model, and obtaining the line of sight estimation result output by the line of sight estimation model. According to the line of sight estimation result and the annotation information carried by the validation set data, the line of sight estimation error, the mean and variance of the line of sight estimation error are calculated. According to the mean and variance of the line of sight estimation error, the line of sight estimation result corresponding to the validation set data is corrected. The method provided in the present application corrects the line of sight estimation result by the mean and variance of the line of sight estimation error, thereby improving the accuracy of the line of sight estimation result.

[0122] Figure 7 A schematic diagram of a sight line estimation device provided in an embodiment of the present application Figure 1 .like Figure 7 As shown, an embodiment of the present application provides a sight line estimation device 700, which may include an acquisition module 701, a training module 702, an adding module 703, a comparative learning module 704 and an optimization module 705.

[0123] An acquisition module 701 is used to acquire source domain data with labeled information and target domain data without labeled information;

[0124] A training module 702 is used to train the training set data in the source domain data to obtain a sight line estimation model;

[0125] An adding module 703 is used to add adversarial noise to the source domain data and the target domain data using an adversarial attack;

[0126] A contrastive learning module 704 performs contrastive learning on the source domain data before and after adding the adversarial noise, and also performs contrastive learning on the target domain data, and updates the sight line estimation model;

[0127] The optimization module 705 optimizes the sight line estimation model using an adversarial training method.

[0128] The device of this embodiment can be used to perform the following steps: Figure 2 The implementation principle and technical effect of the method embodiment shown are similar and will not be repeated here.

[0129] Figure 8 A schematic diagram of a sight line estimation device provided in an embodiment of the present application Figure 2 .like Figure 8 As shown, an embodiment of the present application provides a sight line estimation device 800 including a processor 801 and a memory 802 , wherein the processor 801 and the memory 802 are connected via a bus 803 .

[0130] In the specific implementation process, the memory 802 stores codes, and the processor 701 runs the codes stored in the memory 802 to execute the line of sight estimation method of the above method embodiment.

[0131] The specific implementation process of the processor 801 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0132] In the above Figure 8 In the illustrated embodiment, it should be understood that the processor 801 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0133] The memory 802 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk storage.

[0134] The bus 803 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 803 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 803 in the drawings of the present application is not limited to only one bus or one type of bus.

[0135] An embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the line of sight estimation method of the above method embodiment.

[0136] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0137] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0138] An embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, the line of sight estimation method provided by any embodiment of the present application is implemented.

[0139] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0140] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A line of sight estimation method, characterized in that: include: Obtain source domain data with labeled information and target domain data without labeled information; Training the training set data in the source domain data to obtain a sight line estimation model; Adding adversarial noise to the source domain data and the target domain data using an adversarial attack; Performing comparative learning on the source domain data before and after adding the adversarial noise, and performing comparative learning on the target domain data, and updating the sight line estimation model; Optimizing the line of sight estimation model using an adversarial training method; The adding adversarial noise to the source domain data and the target domain data by using adversarial attack includes: Inputting the source domain data into a sight line estimation model to obtain features of the source domain data and a sight line estimation result of the source domain data; inputting the target domain data into the sight line estimation model to obtain features of the target domain data and a sight line estimation result of the target domain data; Inputting the features of the source domain data into a domain discriminator to obtain a domain discrimination result of the source domain data; inputting the features of the target domain data into the domain discriminator to obtain a domain discrimination result of the target domain data; Adding adversarial noise to the source domain data according to a sight line estimation result of the source domain data and a domain discrimination result of the source domain data; adding adversarial noise to the target domain data according to a sight line estimation result of the target domain data and a domain discrimination result of the target domain data.

2. The method according to claim 1, characterized in that The performing comparative learning on the source domain data before and after adding the adversarial noise, and performing comparative learning on the target domain data, and updating the sight line estimation model, comprises: Inputting the source domain data before and after adding the adversarial noise into the sight line estimation model to obtain the source domain data features before and after adding the adversarial noise; inputting the target domain data before and after adding the adversarial noise into the sight line estimation model to obtain the target domain data features before and after adding the adversarial noise; Constructing positive samples and negative samples based on the source domain data and the target domain data before and after adding adversarial noise; Calculating the loss of the positive sample by a first loss function according to the source domain data features before and after adding the adversarial noise, the target domain data features before and after adding the adversarial noise, the positive sample and the negative sample; The line of sight estimation model is updated according to the loss of the positive sample.

3. The method according to claim 2, characterized in that The constructing positive samples and negative samples based on the source domain data and the target domain data before and after adding the adversarial noise includes: Based on the intrinsic consistency of the source domain data or the target domain data before and after adding the adversarial noise, any of the source domain data or the target domain data before and after adding the adversarial noise is taken as a pair of positive samples, and the source domain data and the target domain data before and after adding the adversarial noise except the positive samples are taken as negative samples.

4. The method according to claim 2, characterized in that: The optimizing the sight line estimation model by using the adversarial training method includes: Inputting the source domain data features before and after adding the adversarial noise into the domain discriminator to obtain the domain discrimination result of the source domain data; inputting the target domain data features before and after adding the adversarial noise into the domain discriminator to obtain the domain discrimination result of the target domain data; According to the domain discrimination results of the source domain data before and after adding the adversarial noise and the domain discrimination results of the target domain data, a first loss is calculated by a second loss function, and the domain discriminator is updated according to the first loss, wherein the first loss is the gap between the domain discrimination results of the source domain data before and after adding the adversarial noise and the source domain data in the source domain, and the gap between the domain discrimination results of the target domain data and the target domain data in the target domain; According to the domain discrimination results of the target domain data before and after adding the adversarial noise, the second loss is calculated by a third loss function, and the line of sight estimation model is updated according to the second loss, wherein the second loss is the gap between the domain discrimination results of the target domain data before and after adding the adversarial noise and the target domain data in the source domain.

5. The method according to claim 1, characterized in that The method further comprises: Based on the validation set data in the source domain data, the line of sight estimation result output by the line of sight estimation model is corrected.

6. The method according to claim 5, characterized in that The correcting the sight line estimation result output by the sight line estimation model based on the validation set data in the source domain data includes: Inputting the validation set data in the source domain data into the sight line estimation model to obtain a sight line estimation result corresponding to the validation set data; Calculate the line of sight estimation error, the mean and the variance of the line of sight estimation error according to the line of sight estimation result corresponding to the validation set data and the annotation information carried by the validation set data; The line of sight estimation result corresponding to the validation set data is corrected according to the mean and variance of the line of sight estimation error.

7. A sight line estimation device, characterized in that: include: An acquisition module is used to acquire source domain data with labeled information and target domain data without labeled information; A training module, used for training the training set data in the source domain data to obtain a sight line estimation model; An adding module, configured to add adversarial noise to the source domain data and the target domain data using an adversarial attack; A contrastive learning module, performing contrastive learning on the source domain data before and after adding the adversarial noise, and performing contrastive learning on the target domain data, and updating the sight line estimation model; Optimization module: optimizing the sight estimation model using an adversarial training method; The adding module is specifically used to input the source domain data into the sight line estimation model to obtain the characteristics of the source domain data and the sight line estimation result of the source domain data; input the target domain data into the sight line estimation model to obtain the characteristics of the target domain data and the sight line estimation result of the target domain data; Inputting the features of the source domain data into a domain discriminator to obtain a domain discrimination result of the source domain data; Inputting the features of the target domain data into the domain discriminator to obtain a domain discrimination result of the target domain data; Adding adversarial noise to the source domain data according to a sight line estimation result of the source domain data and a domain discrimination result of the source domain data; Adversarial noise is added to the target domain data according to a sight line estimation result of the target domain data and a domain discrimination result of the target domain data.

8. A sight line estimation device, comprising: A processor and a memory, wherein the memory stores codes, and the processor runs the codes stored in the memory to execute the line of sight estimation method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the line of sight estimation method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the line of sight estimation method according to any one of claims 1 to 6 is implemented.

Images