A method, apparatus, electronic device, and storage medium for cross-domain line-of-sight prediction based on minimizing the rotation variance of a multi-branch network.

Abstract

This invention discloses a cross-domain gaze prediction method based on minimizing the rotation variance of a multi-branch network, comprising the following steps: performing illumination transformation on a source domain image to obtain a source domain input image set; inputting the source domain input image set into an initial gaze prediction network, updating the parameters of the initial gaze prediction network to obtain a pre-trained gaze prediction network; inputting the illuminated target domain image and the corresponding original image into the pre-trained gaze prediction network, training the pre-trained gaze prediction network to obtain a target gaze prediction network; inputting the target domain image into the target gaze prediction network, and determining the predicted target gaze based on the output prediction results. This invention can extract more robust gaze features, reduce model uncertainty, and better adapt to a small amount of unlabeled target domain data, thus improving the accuracy of cross-domain gaze estimation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of line-of-sight prediction, and particularly relates to a cross-domain line-of-sight prediction method and device based on minimizing multi-branch network rotation variance, an electronic device and a storage medium. BACKGROUND

[0002] Humans mainly understand the world through the acquisition of visual information, and the human eye can perceive the surrounding environment, express human attention, convey personal emotions, and has important significance in natural human-external interaction. Therefore, the line-of-sight direction of the human eye can also reflect the potential cognitive process of the brain, and can be widely applied to various human-computer interaction fields, and has good commercial value. In recent years, with the rapid development of deep learning technology and the accelerated iteration of high-performance hardware devices, the line-of-sight estimation method based on deep learning has made many progresses. However, it is worth noting that in practical applications, the line-of-sight estimation network is usually trained on data collected under controlled conditions, and then the line-of-sight estimation network will be applied to a domain different and uncontrolled environment, so the accuracy of the cross-domain line-of-sight estimation method will be very low, so the breakthrough of the cross-domain line-of-sight estimation has important significance.

[0003] The existing face line-of-sight estimation method can be divided into three steps: first, the image is preprocessed, including sequentially performing face detection, alignment, enhancement and normalization on the collected single face image or video frame; then using a convolutional network or a Transformer to effectively extract high-level line-of-sight features from high-dimensional images; finally, a multi-layer perceptron is used to learn a nonlinear mapping from line-of-sight features to line-of-sight.

[0004] Dr. Xucong Zhang et al. proposed an attention mechanism-based full-face gaze estimation method in the literature "X. Zhang, Y. Sugano, and M. Fritz. It's written all over your face: Full-face appearance-based gaze estimation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017, 51-60", which learns the weights of different regions of the face through a special branch, and then enhances the weights of the eye region while suppressing the weights of other regions unrelated to the gaze direction. Bao et al. found the rotation consistency property in gaze estimation in the literature "Y. Bao, Y. Liu, and H. Wang. Generalizing gaze estimation with rotation consistency. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022, 4207-4216" and introduced it as a pseudo-label into unsupervised domain adaptation, by training a rotation enhancement model in the source domain, and then adapting the model to the target domain using synthetic images with rotation-consistent gaze directions. Cai et al. proposed an uncertainty reduction gaze estimation framework in the literature "X. Cai, J. Zeng, S. Shan, and X. Chen. Source-Free Adaptive Gaze Estimation by Uncertainty Reduction. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2023, 22035-22045", which reduces the uncertainty of cross-domain gaze estimation by applying variance minimization and pseudo-label mechanisms in the target domain.

[0005] The above methods need to train an additional rotation enhancement model in the source domain to synthesize rotation consistent images during face gaze estimation, which increases an additional difficult task for gaze estimation and increases the network burden. At the same time, uncertainty reduction is only applied in the target domain, ignoring the source domain. However, in fact, the performance of the network relies on a large number of labeled images in the source domain, rather than only a small number of unlabeled images in the target domain. SUMMARY

[0006] In order to solve the above problems existing in the prior art, the application provides a cross-domain line-of-sight prediction method based on minimizing the rotation variance of a multi-branch network, a device, an electronic equipment and a storage medium. The technical problem to be solved by the application is realized through the following technical scheme:

[0007] The application provides a cross-domain line-of-sight prediction method based on minimizing the rotation variance of a multi-branch network, comprising the following steps:

[0008] The source domain image is subjected to illumination transformation to obtain a source domain input image set;

[0009] The source domain input image set is input into an initial line-of-sight prediction network to obtain an output result, and the root mean square error between the output result and a line-of-sight real label is minimized to update the parameters of the initial line-of-sight prediction network, thereby obtaining a pre-trained line-of-sight prediction network; wherein the line-of-sight prediction network comprises a backbone network, a primary line-of-sight angle prediction branch network and a plurality of rotation prediction branch networks arranged side by side at the output end of the backbone network;

[0010] An illumination transformation image obtained by performing illumination transformation on the target domain image is input into the pre-trained line-of-sight prediction network to output a first pre-training result, and a target domain pseudo label is determined according to the first pre-training result, the rotation angle of the primary line-of-sight angle prediction branch network and the plurality of rotation prediction branch networks;

[0011] A corresponding original image of the target domain image subjected to illumination transformation is input into the pre-trained line-of-sight prediction network, and the pre-trained line-of-sight prediction network is trained in combination with the target domain pseudo label, thereby obtaining a prediction target line-of-sight prediction network;

[0012] The target domain image is input into the prediction target line-of-sight prediction network, and a prediction target line-of-sight is determined according to the output prediction result.

[0013] In an embodiment of the application, the step of inputting the source domain input image set into the initial line-of-sight prediction network to obtain an output result and minimizing the root mean square error between the output result and a line-of-sight real label to update the parameters of the initial line-of-sight prediction network to obtain a pre-trained line-of-sight prediction network comprises:

[0014] The source domain input image set is input into the backbone network of the initial line-of-sight prediction network to output a line-of-sight feature;

[0015] The line-of-sight feature is input into the primary line-of-sight angle prediction branch network of the initial line-of-sight prediction network to obtain a first output result;

[0016] The root mean square error between the first output result and a line-of-sight real label is minimized to update the parameters of the backbone network and the primary line-of-sight angle prediction branch network of the initial line-of-sight prediction network.

[0017] inputting the line-of-sight feature into a plurality of rotation prediction branch networks of an initial line-of-sight prediction network to obtain a second output result;

[0018] minimizing a root mean square error between the second output result and a line-of-sight real label to update parameters of the plurality of rotation prediction branch networks of the initial line-of-sight prediction network;

[0019] obtaining a pre-trained line-of-sight prediction network after the updating.

[0020] In an embodiment of the present application, the illumination transformed image of the target domain image is input into the pre-trained line-of-sight prediction network to output a first pre-training result, and a target domain pseudo label is determined according to the first pre-training result, an original line-of-sight angle prediction branch network and rotation angles of a plurality of rotation prediction branch networks, including:

[0021] transforming the target domain image to obtain a target domain input image set;

[0022] inputting the illumination transformed image in the target domain input image set into the pre-trained line-of-sight prediction network to output a first pre-training result;

[0023] determining a target domain pseudo label according to the first pre-training result, an original line-of-sight angle prediction branch network and rotation angles of a plurality of rotation prediction branch networks.

[0024] In an embodiment of the present application, the original image corresponding to the illumination transformed target domain image is input into the pre-trained line-of-sight prediction network, and the pre-trained line-of-sight prediction network is trained in combination with the target domain pseudo label to obtain a prediction target line-of-sight prediction network, including:

[0025] inputting the original image corresponding to the illumination transformed target domain image into the pre-trained line-of-sight prediction network to output a second pre-training result;

[0026] determining a variance minimization loss according to the second pre-training result;

[0027] determining a pseudo label supervision minimization loss according to the second pre-training result and the target domain pseudo label;

[0028] determining a target loss according to the variance minimization loss and the pseudo label supervision minimization loss;

[0029] updating parameters of the pre-trained line-of-sight prediction network according to the target loss to obtain a prediction target line-of-sight prediction network.

[0030] In an embodiment of the present application, the formula for minimizing the root mean square error between the first output result and the line-of-sight real label is:

[0031]

[0032] wherein, when θ = 0, θ represents the rotation angle of the original line-of-sight angle prediction branch network, G({I s}, θ) s represents the first output result, {I s} represents the source domain input image set, ‖·‖2 represents the calculation of the root mean square error, g s represents the line-of-sight real label;

[0033] The formula for minimizing the root mean square error between the second output result and the line-of-sight real label is:

[0034]

[0035] wherein, when θ ≠ 0, θ represents the rotation angle of the multiple rotation prediction branch networks, G({I s}, θ) s represents the second output result, represents the rotated line-of-sight label,

[0036] In an embodiment of the present application, the expression of the target domain pseudo label is:

[0037]

[0038] wherein, K represents the number of the original line-of-sight angle prediction branch network and the multiple rotation prediction branch networks; represents the first pre-training result, represents the light change image in the target domain input image set.

[0039] In an embodiment of the present application, the expression of the target loss is:

[0040]

[0041] wherein, I t represents the original image in the target domain input image set, α represents the weight hyperparameter,

[0042]

[0043] represents the second pre-training result.

[0044] The second aspect of the embodiment of the present application provides a cross-domain line-of-sight prediction device based on minimizing the rotation variance of the multiple branch networks, comprising:

[0045] a transformation module configured to perform illumination transformation on the source domain image to obtain a source domain input image set;

[0046] a pre-training module configured to input the source domain input image set into an initial line-of-sight prediction network to obtain an output result, and minimize a root mean square error between the output result and a line-of-sight real label to update parameters of the initial line-of-sight prediction network to obtain a pre-trained line-of-sight prediction network, wherein the line-of-sight prediction network comprises a backbone network, a primary line-of-sight angle prediction branch network and a plurality of rotation prediction branch networks arranged side by side at an output end of the backbone network;

[0047] a target domain training module configured to input an illumination transformation image obtained by performing illumination transformation on a target domain image into the pre-trained line-of-sight prediction network to output a first pre-training result, and determine a target domain pseudo label according to the first pre-training result, a primary line-of-sight angle prediction branch network and a plurality of rotation angles of rotation prediction branch networks;

[0048] the target domain training module is further configured to input a corresponding original image of the target domain image into the pre-trained line-of-sight prediction network, and train the pre-trained line-of-sight prediction network in combination with the target domain pseudo label to obtain a prediction target line-of-sight prediction network;

[0049] a prediction module configured to input the target domain image into the prediction target line-of-sight prediction network, and determine a prediction target line-of-sight according to an output prediction result.

[0050] A third aspect of the embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method provided in the first aspect of the embodiment of the present application when executing the program.

[0051] A fourth aspect of the embodiment of the present application provides a computer readable storage medium having a computer program stored thereon, and the computer program is executable on a processor to implement the method provided in the first aspect of the embodiment of the present application.

[0052] The present application has the following beneficial effects:

[0053] The application makes the model ignore the domain difference in the data and further extracts more robust line-of-sight related features by inhibiting the obvious distribution difference of the source domain and the target domain data caused by light and appearance and the like, and performing a data enhancement method of random light change on the input image; the prediction uncertainty of the model is reduced, and the prediction line-of-sight error is reduced by minimizing the variance of the rotation prediction branch on the same input, and performing an average operation on the prediction results of the multiple branches; when adaptively training in the target domain, pseudo labels are obtained through the light changed images, the line-of-sight estimation ability of the model is ensured, and the features are more robust to a certain extent.

[0054] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the application. The objects and other advantages of the application will be realized and attained by means of the instrumentalities particularly pointed out in the written description and claims hereof as well as the appended drawings.

[0055] The technical solutions of the present application will be further described in detail below with the help of the drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0056] The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application, and do not limit the application. In the drawings:

[0057] Figure 1 A flowchart of a cross-domain line-of-sight prediction method based on minimizing the rotation variance of a multi-branch network is provided for an embodiment of the present application.

[0058] Figure 2 An algorithm framework diagram of a cross-domain line-of-sight prediction method based on minimizing the rotation variance of a multi-branch network is provided for an embodiment of the present application.

[0059] Figure 3 A cross-domain line-of-sight prediction device based on minimizing the rotation variance of a multi-branch network is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0060] The present application will be further described in detail below in combination with specific embodiments, but the implementation manner of the present application is not limited thereto.

[0061] As Figure 1 shown, the first aspect of the embodiment of the present application provides a cross-domain line-of-sight prediction method based on minimizing the rotation variance of a multi-branch network, comprising the following steps:

[0062] Step 11, performing light transformation on the source domain image to obtain a source domain input image set.

[0063] Step 12, input the source domain input image set into the initial gaze prediction network to obtain an output result, and minimize the root mean square error between the output result and the gaze true label to update the parameters of the initial gaze prediction network to obtain a pre-trained gaze prediction network.

[0064] The gaze prediction network comprises a backbone network, a plurality of rotation prediction branch networks and a parallel original gaze angle prediction branch network located at an output end of the backbone network.

[0065] Step 13, input the illumination transformed image of the target domain image into the pre-trained gaze prediction network to output a first pre-training result, and determine a target domain pseudo label according to the first pre-training result, the original gaze angle prediction branch network and the rotation angles of the plurality of rotation prediction branch networks.

[0066] Step 14, input the original image corresponding to the target domain image subjected to illumination transformation into the pre-trained gaze prediction network, and train the pre-trained gaze prediction network in combination with the target domain pseudo label to obtain a prediction target gaze prediction network.

[0067] Step 15, input the target domain image into the prediction target gaze prediction network to determine a prediction target gaze according to the output prediction result.

[0068] In order to solve the cross-domain gaze estimation problem, in the embodiment, the sample uncertainty and model uncertainty of the unlabeled target domain data are reduced, so that the gaze estimation model trained in the source domain is better adapted to the target domain. The sample uncertainty mainly refers to the inherent noise in the input image, such as sensor noise and motion blur. The model uncertainty is caused by the implementation of the deep network itself. A simple model may not achieve the expected fitting effect, and a too complex model may cause overfitting. In the method of the embodiment, the data enhancement method reduces the sample uncertainty, can extract more robust gaze features, and the variance minimization method reduces the model uncertainty, can better adapt to a small amount of unlabeled target domain data, and improves the accuracy of cross-domain gaze estimation.

[0069] The method related in the embodiment mainly completes training in a face image dataset with labels, and predicts the gaze direction of the eyes of a person in an unlabeled face image, and can be used in the medical and health field, such as helping a patient with gradual freezing to complete daily activities with the help of an eye tracker, the auxiliary driving field, such as providing a human-computer interaction function to help free the hands of a driver, and the virtual reality (VR) field, such as realizing immersive real scene interaction of human-computer synchronization.

[0070] As shown in Figure 2 The second aspect of the embodiment of the application provides a cross-domain gaze prediction method based on minimization of multi-branch rotation variance, comprising the following steps:

[0071] Step 21, performing illumination transformation on the source domain image to obtain a source domain input image set.

[0072] In this step, if I represents the input image and g represents the line of sight, the source domain labeled data can be represented as , wherein I s represents the source domain image, g s represents the real label of the line of sight corresponding to the image. In order to reduce the uncertainty of the input sample, that is, to suppress the domain difference and noise, random illumination transformation is performed on the input image. The method of illumination transformation is gamma transformation, which is a process of nonlinear brightness transformation of the image. The implementation method is as follows:

[0073] o = c·r γ ,r∈[0,1]

[0074] Wherein, c is a constant, usually taking 1, r is the pixel value of the input image I after uniformization, o is the uniformized pixel value of the output illumination transformed image, and γ is the Gamma value of the gamma transformation, and different degrees of illumination transformation are realized by adjusting γ. Here, three degrees of gamma transformation are realized, which are γ<1, γ=1, and γ>1, so the transformed image is 3, one original image, one illumination randomly enhanced image and one illumination randomly weakened image. The transformed source domain input image can be represented as , wherein {I s} represents the set of the original image I s , the illumination randomly enhanced image and the illumination randomly weakened image , that is, the source domain input image set.

[0075] Step 22, inputting the source domain input image set into the initial line of sight prediction network to obtain an output result, and minimizing the root mean square error between the output result and the line of sight real label to update the parameters of the initial line of sight prediction network to obtain a pre-trained line of sight prediction network.

[0076] Wherein, the line of sight prediction network comprises: a backbone network, a parallel original line of sight angle prediction branch network and a plurality of rotation prediction branch networks located at the output end of the backbone network.

[0077] In this step, the method disclosed in the literature "K. He, X. Zhang, S. Ren, and J. Sun, Deep Residual Learning for Image Recognition. in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2016, 770-778" is used to construct the backbone network as ResNet18, which is used to extract the gaze feature from the input image, and then the extracted gaze feature is globally average-pooled and sent to the fully connected layer to predict the gaze. At the same time, in order to simplify the rotation enhancement model, a variance minimization method is combined to construct a raw gaze angle prediction branch network to predict the raw gaze, and multiple rotation prediction branch networks to directly predict the rotated gaze. The raw gaze angle prediction branch network and the rotation prediction branch network are parallel. Let θ represent the rotation angle, the initial parameters of the backbone network and a prediction branch network with a rotation angle θ can be represented as G(*, θ | θ = 0,..., K)0, the rotation angle of the raw gaze angle prediction branch network is θ = 0, and the rotation angle of the rotation prediction branch network is θ ≠ 0.

[0078] The specific steps of step 22 include steps 221-226:

[0079] Step 221, input the source domain input image set into the backbone network of the initial gaze prediction network, and output the gaze feature. For any input sample, the 3 kinds of input images {I s} of each sample are input into the backbone network to obtain the corresponding 3 kinds of gaze features, and the 3 kinds of gaze features should be equivalent.

[0080] Step 222, input the gaze feature into the raw gaze angle prediction branch network of the initial gaze prediction network to obtain the first output result.

[0081] Step 223, minimize the root mean square error between the first output result and the gaze true label to update the parameters of the backbone network and the raw gaze angle prediction branch network of the initial gaze prediction network.

[0082] Input the 3 kinds of gaze features into θ = 0, i.e. the raw gaze angle prediction branch network predicts the raw gaze without rotation, and then minimize the error between the predicted gaze (the first output result) and the true label, which is calculated as follows:

[0083]

[0084] Where, G({I s}, θ) s represents the network parameters obtained by training the source domain, and ‖·‖2 represents the root mean square error between the two. gs The gaze real label is represented, and the labels of the three input images are the same. The parameters of the original gaze angle prediction branch network and the backbone network are updated using the original gaze angle prediction error.

[0085] Here, the three gaze features correspond to three output results and three minimized root mean square errors, and the corresponding network parameters are updated after summing the three minimized root mean square errors.

[0086] Step 224, input the gaze feature into the multiple rotation prediction branch networks of the initial gaze prediction network to obtain a second output result.

[0087] Step 225, minimize the root mean square error between the second output result and the gaze real label to update the parameters of the multiple rotation prediction branch networks of the initial gaze prediction network.

[0088] At the same time, all the three gaze features are input into multiple parallel rotation prediction branch networks to directly predict the rotated gaze. In the multiple rotation prediction branch networks, θ represents the rotation angle, and the rotated gaze label is Similarly, the error between the predicted gaze and the real label is minimized, and the calculation method is as follows:

[0089]

[0090] Wherein, the error calculation method remains unchanged, which is still the root mean square error. However, when updating the network parameters, since the features of the backbone network are consistent, the linear rotation angle is directly predicted by the full connection layer here, and in order to prevent the ability of the backbone network to extract features from being destroyed, only the backbone network is updated when predicting the original gaze angle, and the parameters of the backbone network are frozen and not updated when predicting the rotated gaze angle.

[0091] Here, the three gaze features correspond to three output results and three minimized root mean square errors, and the corresponding network parameters are updated after summing the three minimized root mean square errors.

[0092] Wherein, step 222 and step 224 are executed in parallel.

[0093] Step 226, after updating, a pre-trained gaze prediction network is obtained. G(*, θ) s Indicates the network parameters of the pre-trained gaze prediction network.

[0094] The training in the target domain adaptation stage is based on the source domain training, and the model trained in the source domain is used as a pre-trained base model for target domain training, including the following steps 23-Step 24:

[0095] Step 23, input the illumination transformed image of the target domain image into the pre-trained gaze prediction network, output the first pre-training result, and determine the target domain pseudo label according to the first pre-training result, the original gaze angle prediction branch network and the rotation angles of the plurality of rotation prediction branch networks.

[0096] The specific steps of step 23 include steps 231-233:

[0097] Step 231, perform illumination transformation on the target domain image to obtain a target domain input image set.

[0098] Since the target domain data lacks labels, the pseudo label can be calculated by using the enhanced image. Let the target domain data be represented as {I t , which represents the set of original images I t , illumination randomly enhanced images and illumination randomly weakened images , that is, the target domain input image set.

[0099] Step 232, input the illumination transformed image in the target domain input image set into the pre-trained gaze prediction network, and output the first pre-training result.

[0100] Step 233, determine the target domain pseudo label according to the first pre-training result, the original gaze angle prediction branch network and the rotation angles of the plurality of rotation prediction branch networks.

[0101] Two randomly illuminated images are input into the network to predict a plurality of gazes, and then the original gaze prediction angle and the plurality of rotation prediction gaze angles are averaged to obtain the pseudo label p t , as shown below:

[0102]

[0103] Wherein, K represents the number of the original gaze angle prediction branch network and the plurality of rotation prediction branch networks; represents the first pre-training result, represents the illumination transformed image in the target domain input image set, that is, the set of illumination randomly enhanced images and illumination randomly weakened images . The purpose of the obtained pseudo label is to ensure that the gaze estimation ability of the model does not fluctuate too much during subsequent training in the target domain.

[0104] Step 24, input the original image corresponding to the illumination transformed image of the target domain image into the pre-trained gaze prediction network, and train the pre-trained gaze prediction network in combination with the target domain pseudo label to obtain a prediction target gaze prediction network.

[0105] The specific steps of step 24 include steps 241-245:

[0106] Step 241, input the corresponding original image of the target domain image subjected to the illumination transformation into the pre-trained gaze prediction network, and output a second pre-training result.

[0107] Step 242, determine a variance minimization loss according to the second pre-training result.

[0108] The purpose of multi-branch rotation prediction is to reduce the uncertainty of the model, which can be understood as minimizing the variance between predictions on the same input under multi-branch rotation prediction. t The input target domain original image is represented, and the variance minimization loss is as follows:

[0109]

[0110] wherein, The second pre-training result is the target domain predicted gaze of the input original image. According to the different rotation angles, there are K predicted gazes, and the variance of the K predicted gazes is minimized.

[0111] Step 243, determine a pseudo-label supervision minimization loss according to the second pre-training result and the target domain pseudo-label.

[0112] At the same time, a pseudo-label supervision minimization loss is calculated for the predicted gaze using the obtained pseudo-label, as follows:

[0113]

[0114] wherein, the same root mean square error is used to calculate the loss between the target domain predicted gaze and the pseudo-label, and the root mean square errors of multiple predicted angles are directly averaged.

[0115] Step 244, determine a target loss according to the variance minimization loss and the pseudo-label supervision minimization loss.

[0116] In order to balance the above two losses, a weight parameter is added, and therefore the final target loss of the target domain adaptive training stage is as follows:

[0117]

[0118] wherein, α is a weight hyperparameter.

[0119] Step 245, update the parameters of the pre-trained gaze prediction network according to the target loss, and obtain a predicted target gaze prediction network.

[0120] Here, step 23 and step 24 are the process of target domain adaptability training, and the initialization parameters of the network during the training process are G(*, θ)s G(*, theta) t denote the network parameters trained by the target domain.

[0121] Step 25, inputting the target domain image into the prediction target visual line prediction network, and determining the predicted target visual line according to the output prediction result.

[0122] In the inference stage, inputting the target domain image I t Using the results of the multi-branch rotation prediction, each rotation visual line is subtracted by the corresponding rotation angle, and the final predicted visual line g is obtained by averaging.

[0123]

[0124] Wherein, the angle error L angular is commonly used to evaluate the accuracy of three-dimensional visual line estimation methods. Wherein the estimated gaze direction is g element R 3 , and the actual gaze direction is

[0125] The effect of the application can be further illustrated by the following simulation experiment.

[0126] 1. Simulation conditions

[0127] The present application is performed on a central processing unit of Intel(R) Core(TM) i7-7820X CPU @ 3.60GHz, NVIDIA GeForce RTX 2080Ti, Ubuntu 22.04 operating system. The source domain dataset uses the Gaze360 dataset disclosed in the literature "P. Kellnhofer, A. Recasens, and S. Stent. Gaze360: Physically unconstrained gaze estimation in the wild. In Proceedings of IEEE International Conference on Computer Vision, 2019, 6912-6921", abbreviated as DG. The target domain dataset uses the MPIIFaceGaze dataset disclosed in the literature "X. Zhang, Y. Sugano, and M. Fritz. It's written all over your face: Full-face appearance-based gaze estimation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2017, 51-60", abbreviated as DM.

[0128] The methods compared in the experiment are as follows:

[0129] One is an outlier-guided collaborative adaptive method, denoted as PnP-GA, and the reference is "Y. Liu, R. Liu, H. Wang, and F. Lu. Generalizing Gaze Estimation with outlier-guided Collaborative Adaptation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2021, 3835-3844".

[0130] One is a method based on rotation consistency, denoted as RUDA, and the reference is "Y. Bao, Y. Liu, and H. Wang. Generalizing gaze estimation with rotation consistency. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022, 4207-4216".

[0131] One is a method based on contrastive learning, denoted as CRGA, and the reference is "Y. Wang, Y. Jiang, and J. Li. Contrastive regression for domain adaptation on gaze estimation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022, 19376-19385".

[0132] 2. Simulation content

[0133] According to the specific embodiment of the application, the angle error of cross-domain gaze estimation is calculated, the source domain is DG, the target domain is DM, and the angle error is compared with that of the RUDA method and the CRGA method, and the results are shown in Table 1.

[0134] Table 1 Angle error of cross-domain gaze estimation

[0135] Method PnP-GA RUDA CRGA The invention Error 6.18 6.20 5.89 5.78

[0136] As can be seen from Table 1, due to the use of the multi-branch rotation minimization variance method in the application, the sample uncertainty and model uncertainty of the unlabeled target domain data can be reduced, and thus a lower prediction angle error can be achieved in cross-domain gaze estimation, verifying the effectiveness of the application.

[0137] As Figure 3 shown, the third aspect of the embodiment of the application provides a cross-domain gaze prediction device based on a multi-branch network rotation variance minimization, comprising:

[0138] The transformation module 31 is configured to perform illumination transformation on the source domain image to obtain a source domain input image set;

[0139] The pre-training module 32 is configured to input a source domain input image set into an initial gaze prediction network to obtain an output result, minimize a root mean square error between the output result and a gaze real label to update parameters of the initial gaze prediction network, and obtain a pre-trained gaze prediction network; wherein the gaze prediction network comprises a backbone network, a raw gaze angle prediction branch network arranged at an output end of the backbone network, and a plurality of rotation prediction branch networks.

[0140] The target domain training module 33 is configured to input an illumination transformed image obtained by performing illumination transformation on a target domain image into the pre-trained gaze prediction network to output a first pre-training result, and determine a target domain pseudo label according to the first pre-training result, the raw gaze angle prediction branch network and rotation angles of the plurality of rotation prediction branch networks.

[0141] The target domain training module 33 is further configured to input a corresponding original image of the target domain image into the pre-trained gaze prediction network, and train the pre-trained gaze prediction network in combination with the target domain pseudo label to obtain a prediction target gaze prediction network.

[0142] The prediction module 34 is configured to input the target domain image into the prediction target gaze prediction network, and determine a prediction target gaze according to an output prediction result.

[0143] In an embodiment of the present application, the source domain input image set is input into the initial gaze prediction network to obtain an output result, a root mean square error between the output result and a gaze real label is minimized to update parameters of the initial gaze prediction network, and a pre-trained gaze prediction network is obtained, comprising:

[0144] The source domain input image set is input into the backbone network of the initial gaze prediction network to output a gaze feature;

[0145] The gaze feature is input into the raw gaze angle prediction branch network of the initial gaze prediction network to obtain a first output result;

[0146] A root mean square error between the first output result and the gaze real label is minimized to update parameters of the backbone network and the raw gaze angle prediction branch network of the initial gaze prediction network;

[0147] The gaze feature is input into the plurality of rotation prediction branch networks of the initial gaze prediction network to obtain a second output result;

[0148] A root mean square error between the second output result and the gaze real label is minimized to update parameters of the plurality of rotation prediction branch networks of the initial gaze prediction network;

[0149] After the updating is completed, the pre-trained gaze prediction network is obtained.

[0150] In one embodiment of the present invention, an illumination-transformed image of the target domain image is input into a pre-trained gaze prediction network, a first pre-training result is output, and a target domain pseudo-label is determined based on the first pre-training result, the rotation angles of the original gaze angle prediction branch network and multiple rotation prediction branch networks, including:

[0151] The target domain image is subjected to illumination transformation to obtain the target domain input image set;

[0152] The illumination transformation image in the target domain input image set is input into a pre-trained gaze prediction network, which outputs the first pre-training result.

[0153] Based on the first pre-training results, the rotation angles of the original line-of-sight prediction branch network and multiple rotation prediction branch networks, the pseudo-labels of the target domain are determined.

[0154] In one embodiment of the present invention, the original image corresponding to the illumination transformation of the target domain image is input into a pre-trained gaze prediction network, and the pre-trained gaze prediction network is trained by combining the target domain pseudo-labels to obtain a target gaze prediction network, including:

[0155] The original image corresponding to the illumination transformation of the target domain image is input into the pre-trained gaze prediction network, and the second pre-training result is output.

[0156] The variance minimization loss is determined based on the second pre-training results;

[0157] Based on the second pre-training results and the target domain pseudo-labels, determine the pseudo-label supervised minimum loss;

[0158] The target loss is determined based on the variance minimization loss and the pseudo-label supervised minimization loss;

[0159] The parameters of the pre-trained gaze prediction network are updated based on the target loss to obtain the target gaze prediction network.

[0160] In one embodiment of the present invention, the formula for calculating the root mean square error between the first output result and the true label of the line of sight is as follows:

[0161]

[0162] Where, when θ=0, θ represents the rotation angle of the original line-of-sight prediction branch network, G({I s},θ) s Indicates the first output result, {I s} represents the source domain input image set, ‖·‖2 represents the calculation of the root mean square error, g s Indicates the true gaze of the subject;

[0163] The formula for minimizing the root mean square error between the second output and the true line-of-sight label is as follows:

[0164]

[0165] Where, when θ≠0, θ represents the rotation angle of the multiple rotation prediction branch network, G({I s},θ) s This indicates the second output result. This indicates the view label after rotation.

[0166] In one embodiment of the present invention, the expression for the target domain pseudo-label is:

[0167]

[0168] Where K represents the number of the original line-of-sight prediction branch network and the number of multiple rotation prediction branch networks; This represents the first pre-training result. This represents the illumination transformation image in the target domain input image set.

[0169] In one embodiment of the present invention, the expression for the target loss is:

[0170]

[0171] Among them, I t Let α represent the original image in the input image set of the target domain, and let α represent the weight hyperparameter.

[0172]

[0173] This indicates the second pre-training result.

[0174] A fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the cross-domain line-of-sight prediction method based on minimizing the rotation variance of a multi-branch network provided by the present invention.

[0175] The fifth aspect of this invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the cross-domain line-of-sight prediction method based on minimizing the rotation variance of a multi-branch network provided in the above-described embodiments of this invention.

[0176] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0177] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware devices.

[0178] The method provided in this invention can be applied to electronic devices. Specifically, the electronic device can be a desktop computer, a portable computer, a smart mobile terminal, a server, etc. No limitation is made herein; any electronic device that can implement this invention falls within the protection scope of this invention.

[0179] For the device / electronic device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and relevant parts can be referred to in the description of the method embodiments.

[0180] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0181] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a processFigure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0182] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0183] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A cross-domain line-of-sight prediction method based on minimizing the rotational variance of a multi-leg network, characterized in that, The method comprises the following steps: The source domain image is subjected to illumination transformation to obtain a source domain input image set; The source domain input image set is input into an initial line-of-sight prediction network to obtain an output result, and the root mean square error between the output result and a line-of-sight real label is minimized to update the parameters of the initial line-of-sight prediction network to obtain a pre-trained line-of-sight prediction network, comprising: the source domain input image set is input into a backbone network of the initial line-of-sight prediction network to output a line-of-sight feature; the line-of-sight feature is input into an original line-of-sight angle prediction branch network of the initial line-of-sight prediction network to obtain a first output result; the root mean square error between the first output result and the line-of-sight real label is minimized to update the parameters of the backbone network and the original line-of-sight angle prediction branch network of the initial line-of-sight prediction network; wherein the backbone network is updated when the original line-of-sight angle is predicted, and the parameters of the backbone network are frozen and not updated when the rotated line-of-sight angle is predicted; the line-of-sight feature is input into a plurality of rotated prediction branch networks of the initial line-of-sight prediction network to obtain a second output result; the root mean square error between the second output result and the line-of-sight real label is minimized to update the parameters of the plurality of rotated prediction branch networks of the initial line-of-sight prediction network; and the pre-trained line-of-sight prediction network is obtained after the update; wherein the line-of-sight prediction network comprises: a backbone network, an original line-of-sight angle prediction branch network and a plurality of rotated prediction branch networks arranged side by side at the output end of the backbone network; The light transformation image of the target domain image is input into the pre-trained gaze prediction network, a first pre-training result is output, and a target domain pseudo label is determined according to the first pre-training result, an original gaze angle prediction branch network and rotation angles of a plurality of rotation prediction branch networks, including: performing light transformation on the target domain image to obtain a target domain input image set; inputting the light transformation image in the target domain input image set into the pre-trained gaze prediction network to output a first pre-training result; and determining a target domain pseudo label according to the first pre-training result, an original gaze angle prediction branch network and rotation angles of a plurality of rotation prediction branch networks, including: inputting two kinds of random light transformation images into the network to predict a plurality of gazes, and then averaging an original gaze prediction angle and a plurality of rotation prediction gaze angles to obtain a pseudo label p t As shown below: K represents the number of the original view angle prediction branch network and the plurality of rotation prediction branch networks; represents the first pre-training result, represents a set of light change images in the target domain input image set, that is, light random enhancement images and light random weakening images ; θ represents the rotation angle of the original view angle prediction branch network. The original image corresponding to the target domain image subjected to illumination transformation is input into the pre-trained line-of-sight prediction network, and the pre-trained line-of-sight prediction network is trained in combination with the target domain pseudo label to obtain a prediction target line-of-sight prediction network, comprising: the original image corresponding to the target domain image subjected to illumination transformation is input into the pre-trained line-of-sight prediction network to output a second pre-training result; a variance minimization loss is determined according to the second pre-training result; a pseudo label supervision minimization loss is determined according to the second pre-training result and the target domain pseudo label; a target loss is determined according to the variance minimization loss and the pseudo label supervision minimization loss; and the parameters of the pre-trained line-of-sight prediction network are updated according to the target loss to obtain the prediction target line-of-sight prediction network; wherein the expression of the target loss is: where I t represents the original image in the target domain input image set D t , and a represents the weight hyperparameter, representing a second pre-training result; The target domain image is input into the prediction target line-of-sight prediction network, and a prediction target line-of-sight is determined according to the output prediction result.

2. The method of claim 1, wherein, The calculation formula of the minimization of the root mean square error between the first output result and the line-of-sight real label is: wherein, when θ = 0, θ represents a rotation angle of the original line-of-sight angle prediction branch network, G({I s}, θ) s represents a first output result, {I s} represents a source domain input image set, ‖·‖2 represents a calculation of a root mean square error, g s represents a line-of-sight real label; The calculation formula of the minimization of the root mean square error between the second output result and the line-of-sight real label is: wherein θ represents a rotation angle of the plurality of rotated prediction branch networks when θ≠0, G({I s}, θ) s represents a second output result, represents a rotated line-of-sight label, 3. The method of claim 1, wherein, The expression of the target domain pseudo label is: Wherein, K represents the number of the original line-of-sight angle prediction branch network and the plurality of rotation prediction branch networks; denotes the first pre-training result, denotes the light change image in the target domain input image set.

4. A cross-domain line-of-sight prediction apparatus based on minimizing the rotational variance of a multi-leg network, characterized by, comprises: A transformation module configured to subject a source domain image to illumination transformation to obtain a source domain input image set; The pre-training module is configured to input a source domain input image set into an initial gaze prediction network to obtain an output result, minimize a root mean square error between the output result and a gaze real label to update parameters of the initial gaze prediction network, and obtain a pre-trained gaze prediction network. The pre-training module includes: inputting the source domain input image set into a backbone network of the initial gaze prediction network to output a gaze feature; inputting the gaze feature into an original gaze angle prediction branch network of the initial gaze prediction network to obtain a first output result; minimizing a root mean square error between the first output result and the gaze real label to update parameters of the backbone network and the original gaze angle prediction branch network of the initial gaze prediction network; wherein the backbone network is updated when an original gaze angle is predicted, and the parameters of the backbone network are frozen and not updated when a rotated gaze angle is predicted; inputting the gaze feature into a plurality of rotated prediction branch networks of the initial gaze prediction network to obtain a second output result; minimizing a root mean square error between the second output result and the gaze real label to update parameters of the plurality of rotated prediction branch networks of the initial gaze prediction network; and obtaining the pre-trained gaze prediction network after the updating is completed. The gaze prediction network includes the backbone network, the original gaze angle prediction branch network, and the plurality of rotated prediction branch networks. The target domain training module is configured to input the lightness transformed image obtained by performing lightness transformation on the target domain image into the pre-trained line-of-sight prediction network, output a first pre-training result, and determine a target domain pseudo label according to the first pre-training result, the original line-of-sight angle prediction branch network, and rotation angles of the plurality of rotation prediction branch networks. The target domain training module includes: performing lightness transformation on the target domain image to obtain a target domain input image set; inputting the lightness transformed image in the target domain input image set into the pre-trained line-of-sight prediction network to output the first pre-training result; and determining the target domain pseudo label according to the first pre-training result, the original line-of-sight angle prediction branch network, and the rotation angles of the plurality of rotation prediction branch networks. The target domain pseudo label is determined by: inputting two images transformed by random lightness into the network to predict a plurality of lines of sight, and then averaging the original line-of-sight prediction angle and the plurality of rotation prediction line-of-sight angles to obtain the pseudo label p t As shown below: K represents the number of the original view angle prediction branch network and the plurality of rotation prediction branch networks; represents the first pre-training result, represents the set of the light change images in the target domain input image set, that is, the light random enhancement images and the light random weakening images ; θ represents the rotation angle of the original view angle prediction branch network. The target domain training module is further configured to input an original image corresponding to the target domain image subjected to illumination transformation into the pre-trained gaze prediction network, and train the pre-trained gaze prediction network in combination with the target domain pseudo label to obtain a prediction target gaze prediction network. The target domain training module includes: inputting the original image corresponding to the target domain image subjected to illumination transformation into the pre-trained gaze prediction network to output a second pre-training result; determining a variance minimization loss according to the second pre-training result; determining a pseudo label supervision minimization loss according to the second pre-training result and the target domain pseudo label; determining a target loss according to the variance minimization loss and the pseudo label supervision minimization loss; and updating parameters of the pre-trained gaze prediction network according to the target loss to obtain the prediction target gaze prediction network. An expression of the target loss is: where I t represents the original image in the target domain input image set D t , and a represents the weight hyperparameter, representing a second pre-training result; The prediction module is configured to input the target domain image into the prediction target gaze prediction network, and determine a prediction target gaze according to an output prediction result.

5. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the cross-domain gaze prediction method based on minimization of rotated variance of a multi-branch network according to any one of claims 1 to 3.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the cross-domain gaze prediction method based on minimization of rotated variance of a multi-branch network according to any one of claims 1 to 3.

Images