Construction method of sight line prediction model, sight line prediction method and device, electronic equipment and storage medium

Abstract

The invention provides a sight line prediction model construction method, a sight line prediction method and device, electronic equipment and a storage medium, and the method comprises the steps: inputting a first eye sample image and a second eye sample image into an auxiliary prediction model, and obtaining sight line difference description information; inputting the first eye sample image and the second eye sample image into a to-be-constructed sight line prediction model to obtain predicted sight line information; if the loss of the to-be-constructed sight line prediction model determined on the basis of the sight line difference description information and the predicted sight line information does not meet the convergence condition, updating model parameters of the to-be-constructed sight line prediction model on the basis of the sight line difference description information and the predicted sight line information; and otherwise, determining that the to-be-constructed sight prediction model is a sight prediction model. The method provided by the invention not only can eliminate individual differences and avoid overfitting caused by sample data discretization, but also can enable the sight line prediction model to be directly used for predicting the sight line direction without firstly collecting a calibration image for calibration in a reasoning stage.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method for constructing a sight line prediction model, a sight line prediction method, a device, an electronic device, and a storage medium. Background Art

[0002] With the rapid development of computer vision, artificial intelligence and other fields, the research on line of sight estimation technology has attracted widespread attention. Line of sight estimation plays an important role in many research fields, such as human-computer interaction, virtual reality, social interaction analysis, and medical treatment. Summary of the invention

[0003] According to one aspect of the present disclosure, a method for constructing a sight line prediction model is provided, comprising:

[0004] Inputting the first eye sample image and the second eye sample image into the auxiliary prediction model to obtain sight difference description information, where the sight difference description information is used to characterize the sight difference between the first eye sample image and the second eye sample image;

[0005] Inputting the first eye sample image and the second eye sample image into the sight line prediction model to be constructed to obtain predicted sight line information, where the predicted sight line information includes a first predicted sight line parameter of the first eye sample image and a second predicted sight line parameter of the second eye sample image;

[0006] Determine the loss of the sight prediction model to be constructed based on the sight difference description information and the predicted sight information;

[0007] If the loss of the sight prediction model to be constructed does not meet the convergence condition, the model parameters of the sight prediction model to be constructed are updated based on the sight difference description information and the predicted sight information; otherwise, the sight prediction model to be constructed is determined to be the sight prediction model.

[0008] According to another aspect of the present disclosure, a sight line prediction method is provided, comprising:

[0009] Acquire a target eye image;

[0010] The target eye image is input into a sight line prediction model to obtain sight line parameters corresponding to the target eye image. The sight line prediction model is trained by the method described in the exemplary embodiment of the present disclosure.

[0011] According to another aspect of the present disclosure, a device for constructing a sight line prediction model is provided, comprising:

[0012] A training module is used to input the first eye sample image and the second eye sample image into the auxiliary prediction model to obtain line of sight difference description information, where the line of sight difference description information is used to characterize the line of sight difference between the first eye sample image and the second eye sample image; input the first eye sample image and the second eye sample image into the line of sight prediction model to be constructed to obtain predicted line of sight information, where the predicted line of sight information includes a first predicted line of sight parameter of the first eye sample image and a second predicted line of sight parameter of the second eye sample image; determine the loss of the line of sight prediction model to be constructed based on the line of sight difference description information and the predicted line of sight information; if the loss of the line of sight prediction model to be constructed does not meet the convergence condition, update the model parameters of the line of sight prediction model to be constructed based on the line of sight difference description information and the predicted line of sight information;

[0013] The determination module is used to determine that the sight line prediction model to be constructed is a sight line prediction model if the loss of the sight line prediction model to be constructed meets the convergence condition.

[0014] According to another aspect of the present disclosure, a sight line prediction device is provided, comprising:

[0015] An acquisition module, used for acquiring a target eye image;

[0016] A prediction module is used to input the target eye image into a sight line prediction model to obtain sight line parameters corresponding to the target eye image, and the sight line prediction model is trained by the device described in the exemplary embodiment of the present disclosure.

[0017] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0018] processor; and,

[0019] A memory for storing programs;

[0020] The program includes instructions, and when the instructions are executed by a processor, the processor executes the method according to the exemplary embodiment of the present disclosure.

[0021] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method according to the exemplary embodiments of the present disclosure.

[0022] One or more technical solutions provided in the exemplary embodiments of the present disclosure can input the first eye sample image and the second eye sample image into the auxiliary prediction model to obtain the sight difference description information, and input the first eye sample image and the second eye sample image into the sight prediction model to be constructed to obtain the predicted sight information, and the predicted sight information can include the first predicted sight parameter of the first eye sample image and the second predicted sight parameter of the second eye sample image. Since the sight difference description information can be used to characterize the sight difference between the first eye sample image and the second eye sample image, when the exemplary embodiments of the present disclosure use the sight difference description information output by the auxiliary prediction model as the supervision information of the eye sample image, it can not only reduce the sample annotation cost, but also eliminate the sight deviation caused by individual differences such as the angle between the eye visual axis and the optical axis, and avoid overfitting caused by "discrete sample data" during the eye sample image acquisition process. At the same time, the sight prediction model finally obtained can be directly used to predict the sight direction corresponding to the eye image to be tested in the inference stage, without the need to first collect a calibration image for calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0024] Figure 1 A flowchart showing a method for constructing a sight line prediction model according to an exemplary embodiment of the present disclosure is shown;

[0025] Figure 2 A schematic diagram of a model architecture of an exemplary embodiment of the present disclosure is shown;

[0026] Figure 3 A flowchart showing a sight line prediction method according to an exemplary embodiment of the present disclosure is shown;

[0027] Figure 4 A schematic block diagram of modules of a device for constructing a sight line prediction model according to an exemplary embodiment of the present disclosure is shown;

[0028] Figure 5 A schematic block diagram of modules of a sight line prediction device according to an exemplary embodiment of the present disclosure is shown;

[0029] Figure 6 A schematic block diagram of a chip according to an exemplary embodiment of the present disclosure is shown;

[0030] Figure 7 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0031] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0032] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0033] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0034] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0035] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0036] The exemplary embodiments of the present disclosure provide a method for constructing a sight prediction model, a sight prediction method, a device, an electronic device, and a storage medium. The sight difference description information output by the auxiliary prediction model is used as the supervision information of the eye sample image pair, which can not only reduce the sample annotation cost, but also eliminate the sight deviation caused by individual differences such as the angle between the eye visual axis and the optical axis, and avoid overfitting caused by "discrete sample data" during the collection of eye sample images. At the same time, the sight prediction model finally obtained can be directly used to predict the sight direction corresponding to the eye image to be tested in the inference stage, without the need to collect calibration images for calibration first.

[0037] The method of the exemplary embodiments of the present disclosure may be executed by an electronic device, or may be executed by a chip applied to the electronic device.

[0038] Exemplarily, the electronic device of the exemplary embodiment of the present disclosure may be an electronic device with a display function, and the electronic device may be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, or other terminals.

[0039] Figure 1 FIG. 1 is a flow chart showing a method for constructing a sight line prediction model according to an exemplary embodiment of the present disclosure. Figure 1 As shown, the method for constructing a sight line prediction model of an exemplary embodiment of the present disclosure may include:

[0040] Step 101: Input the first eye sample image and the second eye sample image into the auxiliary prediction model to obtain line of sight difference description information, where the line of sight difference description information is used to characterize the line of sight difference between the first eye sample image and the second eye sample image. The above-mentioned first eye sample image and the second eye sample image may be two eye images belonging to the same object. For a single eye image, it may be a monocular image or a binocular image, where the binocular image may be a spliced ​​image formed by left and right eye images from the same object collected at the same time, or may be a binocular image of the same object collected at the same time. For multiple eye images, they may be collected at the same time or at different times, which is determined according to the actual scene and is not limited here.

[0041] When the first eye sample image and the second eye sample image are input into the auxiliary prediction model, the auxiliary prediction model can process the first eye sample image and the second eye sample image in a shared weight manner, that is, the auxiliary prediction model can use the same weight for processing the first eye sample image and the second eye sample image.

[0042] Assuming that the eye sight direction in the first eye sample image is defined as the first sight direction, and the eye sight direction in the second eye sample image is defined as the second sight direction, an exemplary embodiment of the present disclosure can use an auxiliary prediction model to output sight difference description information between the first eye sample image and the second eye sample image, and the sight difference description information can be used to characterize the sight difference between the first sight direction and the second sight direction.

[0043] The above-mentioned auxiliary prediction model can be an untrained neural network model, an incompletely trained neural network model, or a pre-trained neural network model. Exemplarily, when the above-mentioned auxiliary prediction model is a pre-trained neural network model, the exemplary embodiment of the present disclosure can use two accurately labeled eye images and the corresponding two labeled sight lines as training samples, and determine the loss based on the predicted value of the sight difference of the two eye images output by the neural network model during the training process and the true value of the sight difference determined by the two labeled sight lines. When it is determined based on the loss that the currently trained neural network model has converged, the currently trained neural network model is determined to be an auxiliary prediction model. After the training of the auxiliary prediction model is completed, the auxiliary prediction model can be used to predict the sight difference description information corresponding to the first eye sample image and the second eye sample image.

[0044] Step 102: Input the first eye sample image and the second eye sample image into the gaze prediction model to be constructed to obtain predicted gaze information, where the predicted gaze information includes a first predicted gaze parameter of the first eye sample image and a second predicted gaze parameter of the second eye sample image.

[0045] When the first eye sample image and the second eye sample image are input into the line of sight prediction model to be constructed, the processing weights of the first eye sample image and the second eye sample image by the prediction model to be constructed are consistent, so that the prediction model to be constructed can process the first eye sample image and the second eye sample image in a shared weight manner, thereby obtaining the first predicted line of sight parameter and the second predicted line of sight parameter.

[0046] Exemplarily, the exemplary embodiments of the present disclosure may input the first eye sample image and the second eye sample image into the line of sight prediction model to be constructed at the same time, or may input them into the line of sight prediction model to be constructed separately. The line of sight prediction model to be constructed may output a first predicted line of sight parameter based on the first eye sample image, and output a second predicted line of sight parameter based on the second eye sample image. Here, the first predicted line of sight parameter may be used to characterize the prediction result of the eye line of sight direction in the first eye sample image, and the second predicted line of sight parameter may be used to characterize the prediction result of the eye line of sight direction in the second eye sample image.

[0047] Step 103: Determine the loss of the sight line prediction model to be constructed based on the sight line difference description information and the predicted sight line information.

[0048] In the related art, eye images with marked sight directions are often used to train sight estimation models, and then the eye images to be tested are directly input into the trained sight estimation model to determine the sight direction of the eye images to be tested; and when training the sight estimation model, the loss is usually determined based on the predicted value of the sight estimation model and the marked sight direction of the eye images. However, in practice, there are relatively few eye images with precise annotations, and because the angles between the visual axis and the optical axis of different individuals are different, the sight estimation model obtained by training with limited eye images with precise annotations, in its inference stage, when the object individual to which the eye image to be tested obtained by the sight estimation model belongs is different from the object individual to which the sample image processed in the training stage belongs, the sight direction determined by the sight estimation model often has a sight estimation deviation caused by individual differences. In addition, since the training sample set of eye images with precise annotations usually has the problem of "data discreteness", if the loss is determined only based on the predicted value of the sight estimation model and the marked sight direction of the eye image, it is easy to cause the sight estimation model to overfit during the training process.

[0049] The method of the exemplary embodiment of the present disclosure uses the sight difference description information output by the auxiliary prediction model as the supervision information of the eye sample image. On the one hand, it reduces the sample annotation cost. On the other hand, the sight difference prediction model can learn the sight difference information between the first eye sample image and the second eye sample image during the training process. Due to the data composition characteristics of the sight difference information itself, it can eliminate the prediction bias introduced by the aforementioned individual differences, so that the sight prediction model constructed thereby can avoid the problem of inaccurate sight direction prediction caused by individual differences when predicting the sight. In addition, by supervising the training process of the sight prediction model through the sight difference description information output by the auxiliary prediction model, the model overfitting problem caused by the "data discreteness" of the eye sample image training data set can also be avoided.

[0050] Exemplarily, the loss of the sight prediction model to be constructed in the exemplary embodiment of the present disclosure may include at least a first loss parameter, and the first loss parameter may be determined by the first predicted sight parameter, the second predicted sight parameter, and the supervision information of the eye sample image. When the sight difference description parameter of the exemplary embodiment of the present disclosure may include at least a reference sight difference parameter, the supervision information of the above-mentioned eye sample image may include the reference sight difference parameter, and the above-mentioned first loss parameter may include a first sub-loss parameter, and the first sub-loss parameter may be determined based on the reference sight difference parameter and the sight difference parameter, and the sight difference parameter may be determined by the difference between the first predicted sight parameter and the second predicted sight parameter.

[0051] The reference sight line difference parameter may actually be the difference between the first sight line direction determined by the first eye sample image and the second sight line direction determined by the second eye sample image. Of course, it may also be a related parameter used to represent the sight line direction difference. For example, the sight line direction may be defined by the sight line angle.

[0052] For example, assuming that when the eye is looking straight ahead, the eye sight is the reference sight, and then based on the reference sight, the angles between other eye sight directions and the reference sight are defined as other eye sight angles. Based on this, the exemplary embodiment of the present disclosure can define a first sight direction determined by a first eye sample image by a first sight angle, define a second sight direction determined by a second eye sample image by a second sight angle, and then determine a sight angle difference by the first sight angle and the second sight angle, and define the sight angle difference as a related parameter of the sight direction difference.

[0053] When the supervision information of the eye sample image may include a reference line of sight difference parameter, the exemplary embodiment of the present disclosure may determine the line of sight difference parameter of the first eye sample image and the second eye sample image by solving the difference between the first predicted line of sight parameter and the second predicted line of sight parameter. The essence of the above-mentioned line of sight difference parameter may be the difference between the first predicted line of sight parameter of the first eye sample image and the second predicted line of sight parameter of the second eye sample image. Of course, it may also be a related parameter used to represent the line of sight difference. For example: the first predicted line of sight parameter may be defined as the first predicted line of sight angle, and the second predicted line of sight parameter may be defined as the second predicted line of sight angle by the line of sight angle. At this time, the line of sight difference parameter may be the difference between the first predicted line of sight angle and the second predicted line of sight angle.

[0054] Based on this, the method of the exemplary embodiment of the present invention can determine the first sub-loss parameter based on the reference line of sight difference parameter and the line of sight difference parameter, and use the first sub-loss parameter to determine the line of sight direction prediction loss of the line of sight prediction model to be constructed, so as to continuously reduce the line of sight direction prediction loss of the line of sight prediction model to be constructed during the training iteration process, thereby eliminating the prediction bias caused by individual differences and avoiding the model overfitting problem caused by the "data discreteness" of the eye sample image training data set.

[0055] Step 104: Determine whether the loss of the sight line prediction model to be constructed meets the convergence condition. If the convergence condition is not met, it means that the sight line prediction accuracy of the currently trained sight line prediction model to be constructed is relatively poor and cannot be accurately used to predict the sight line information of the eye image. Further training is required to meet the sight line prediction accuracy. At this time, step 105 can be executed; if the convergence condition is met, it means that the sight line prediction accuracy of the currently trained sight line prediction model to be constructed is relatively high and can be accurately used to predict the sight line information of the eye image. At this time, step 106 can be executed.

[0056] In practical applications, the above convergence condition may include that the loss of the sight line prediction model to be constructed is less than or equal to a preset threshold, or it may refer to that the loss of the sight line prediction model to be constructed is stable, which is determined according to the actual application scenario and is not limited here.

[0057] Step 105: Update the model parameters of the sight line prediction model to be constructed based on the sight line difference description information and the predicted sight line information. The model parameters here may include weights and / or offset values.

[0058] Step 106: Determine that the sight line prediction model to be constructed is a sight line prediction model. At this point, the construction process of the sight line prediction model has been completed.

[0059] In the related art, the disparity estimation model uses eye image pairs as input and outputs sight difference information during the training process and the reasoning process. Therefore, in the actual reasoning process, it is necessary to first guide the user to look at the known direction (hereinafter referred to as the calibration direction) marker, and collect the corresponding eye image as the subsequent calibration image. When the electronic device needs to estimate the user's sight direction in real time, it is necessary to collect the user's eye image to be tested in real time, and estimate the sight difference between the eye image to be tested and the aforementioned calibration image through the disparity estimation model. Since the sight direction of the calibration image is known, the sight direction represented by the eye image to be tested can be determined based on the sight difference estimated by the disparity estimation model and the calibration direction. However, when obtaining the calibration sight, the collected calibration image may not match the calibration direction due to some reasons, such as the user not looking at the marker in the calibration direction, or not looking at the marker at all. In this case, there is a large deviation between the sight difference estimated by the disparity estimation model and the sight direction represented by the eye image to be tested determined by the calibration direction.

[0060] However, in the training phase of the sight prediction model, the sight difference description information output by the auxiliary prediction model is used as the supervision information of the eye sample image, which can not only reduce the sample annotation cost, but also eliminate the sight deviation caused by individual differences such as the angle between the eye visual axis and the optical axis, and avoid overfitting caused by "discrete sample data" during the eye sample image acquisition process. On this basis, the sight prediction model constructed by the method of the exemplary embodiment of the present disclosure can directly output the corresponding sight direction based on the input image to be tested, without the need to collect calibration images in the inference phase, thereby improving the user experience.

[0061] In practical applications, the sight difference description information of the exemplary embodiment of the present disclosure may also include the first eye image feature, and the predicted sight information may also include the second eye image feature. At this time, the loss of the sight prediction model to be constructed in the exemplary embodiment of the present disclosure may include not only the first loss parameter, but also the second loss parameter.

[0062] The first loss parameter can be determined by the first predicted sight line parameter, the second predicted sight line parameter and the supervision information of the eye sample image, which is actually the prediction loss. The specific content can be found in the previous text and will not be repeated here. The second loss parameter is determined by the first eye image feature and the second eye image feature, which is actually the feature loss.

[0063] The exemplary embodiment of the present disclosure may define eye sample images with marked sight directions as labeled eye sample images, and define eye sample images without marked sight directions as unlabeled eye sample images. From the perspective of whether the eye sample images have marked sight directions, the eye sample images of the exemplary embodiment of the present disclosure may include unlabeled eye sample images, may include labeled eye sample images, or may include both unlabeled eye sample images and labeled eye sample images. At this time, when determining the first loss parameter, the supervision information of the eye sample images needs to be determined according to the actual scenario.

[0064] It can be understood that, regardless of whether the eye sample image is an unlabeled eye sample image or a labeled eye sample image, the supervision information of the eye sample image can include a reference sight difference parameter, and the first loss parameter can include a first sub-loss parameter, which can be determined by the reference sight difference parameter and the sight difference parameter. Here, the relevant content of the first sub-loss parameter can be specifically referred to in the previous text, and will not be repeated here.

[0065] Therefore, when the eye sample image includes an unlabeled eye sample image, the exemplary embodiment of the present disclosure does not need to accurately label the unlabeled eye sample image, nor does it need to abandon the unlabeled eye sample image. Instead, the reference line of sight difference parameter output by the auxiliary prediction model can be used as the supervision information of the eye sample image, and then the prediction loss of the line of sight prediction model to be constructed is determined (i.e., the first sub-loss parameter is determined). Therefore, in the training stage of the line of sight prediction model, the unlabeled eye sample images that have not been accurately labeled can be fully learned, thereby reducing the sample labeling cost and improving the utilization efficiency of the unlabeled eye sample images.

[0066] On this basis, when the eye sample image includes a labeled eye sample image, the supervision information of the eye sample image may also include the annotated sight lines of all eye sample images. For example: in the method of the exemplary embodiment of the present disclosure, the annotated sight lines of all eye sample images include the first annotated sight line direction of the first eye sample image and the second annotated sight line direction of the second eye sample image. Here, the first annotated sight line direction can be used to annotate the eye sight line direction in the first eye sample image, and the second annotated sight line direction can be used to annotate the eye sight line direction in the second eye sample image. Based on this, the first loss parameter of the exemplary embodiment of the present disclosure may include a second sub-loss parameter. The second sub-loss parameter may be determined by the first predicted loss parameter and the second predicted loss parameter, wherein the first predicted loss parameter may be determined by the first annotated sight line direction and the first predicted sight line parameter, and the second predicted loss parameter is determined by the second annotated sight line direction and the second predicted sight line parameter.

[0067] For example, an exemplary embodiment of the present disclosure may determine a first predicted loss parameter based on a first annotated sight line direction and a first predicted sight line parameter, determine a second predicted loss parameter based on a second predicted sight line parameter and a second annotated sight line direction, and then determine a second sub-loss parameter based on a weighted result of the first predicted loss parameter and the second predicted loss parameter.

[0068] It can be seen that when the eye sample image includes a labeled eye sample image, the exemplary embodiment of the present disclosure can directly use the annotated gaze direction of the eye sample image as the supervisory information of the eye sample image, and use the annotated gaze direction of the accurately annotated eye sample image to determine the prediction loss of the gaze prediction model to be constructed (i.e., determine the second sub-loss parameter). At this time, the supervisory information of the labeled eye sample image can be fully utilized, thereby improving the utilization rate of the supervisory information of the labeled eye sample image.

[0069] On this basis, the loss of the sight prediction model to be constructed in the exemplary embodiment of the present disclosure can be determined by the weighted result of the first loss parameter and the second loss parameter. In terms of feature scale, both eye image features and predicted sight information are represented in the form of matrices, and the scale difference between the two is large.

[0070] For example, for eye image features, their scale is relatively small, while the scale of predicted sight line information is large, so the second loss parameter is relatively small, while the first loss parameter is relatively large. At this time, the weight of the second loss parameter can be set to be greater than the weight of the first loss parameter, and the second loss parameter can be amplified by the weight of the second loss parameter, so that the loss of the sight line prediction model to be constructed can fully consider the loss of eye image features, thereby improving the sight line prediction accuracy of the constructed sight line prediction model.

[0071] Therefore, the exemplary embodiment of the present disclosure can comprehensively determine the loss of the line of sight prediction model to be constructed from two aspects: prediction loss and feature loss, so that the model parameters of the obtained line of sight prediction model are optimal. The line of sight parameters predicted by the line of sight prediction model in the inference stage are closer to the real line of sight parameters of the eye image, thereby improving the line of sight prediction accuracy of the line of sight prediction model.

[0072] In practical applications, the exemplary embodiments of the present disclosure may add some branches to the line of sight prediction model to be constructed, through which the image features extracted from the first eye sample image and the second eye sample image are respectively fused, and the predicted line of sight difference parameters of the first eye sample image and the second eye sample image are output based on the fused features. Based on this, the predicted line of sight information of the exemplary embodiments of the present disclosure may also include predicted line of sight difference parameters. Here, the predicted line of sight difference parameters are different from the line of sight difference parameters of the first eye sample image and the second eye sample image determined based on the first predicted line of sight parameters and the second predicted line of sight parameters in the previous text, and they can be directly output through the branches added in the line of sight prediction model to be constructed.

[0073] In this case, the loss of the sight line prediction model to be constructed may also include a third loss parameter, the essence of which is the prediction loss, and the third loss parameter may be determined by a reference sight line difference parameter and a prediction sight line difference parameter.

[0074] When the loss of the line of sight prediction model to be constructed may include a first loss parameter, a second loss parameter and a third loss parameter, the loss of the line of sight prediction model to be constructed may be determined by a weighted result of the first loss parameter, the second loss parameter and the third loss parameter, and the weight of the first loss parameter and the weight of the third loss parameter are both smaller than the weight of the second loss parameter.

[0075] For example, the loss calculation formula of the sight prediction model to be constructed is: L 总 =X 1 *L 1 +X 2 *L 2 +X 3 *L 3 , where L 总 represents the loss of the line of sight prediction model to be constructed, L 1 , L 2 and L 3 Represent the first loss parameter, the second loss parameter and the third loss parameter respectively, X 1 , X 2 and X 3 They represent the weight of the first loss parameter, the weight of the second loss parameter, and the weight of the third loss parameter respectively.

[0076] Here, the third loss parameter, together with the first loss parameter, belongs to the prediction loss, and the scale of the annotated sight line direction of the corresponding eye sample image is also large. Therefore, when determining the loss of the sight line prediction model to be constructed, the weight of the first loss parameter and the weight of the third loss parameter are set to be smaller than the weight of the second loss parameter. At this time, the weight of the second loss parameter can be used to amplify the second loss parameter so that the loss of the sight line prediction model to be constructed can fully consider the loss of eye image features and improve the sight line prediction accuracy of the constructed sight line prediction model. For example, the values ​​of X1 and X2 can be set to 1, and the value of X3 can be set to 100.

[0077] On this basis, the exemplary embodiment of the present disclosure can determine whether the loss of the line of sight prediction model to be constructed meets the convergence condition during the training stage. If so, the currently trained line of sight prediction model to be constructed is determined to be the line of sight prediction model; otherwise, the model parameters of the line of sight prediction model to be constructed are updated based on the line of sight difference description information and the predicted line of sight information.

[0078] In practical applications, in each iterative training process, when the supervision information of the eye sample image includes a reference sight difference parameter, a first annotated sight direction, and a second annotated sight direction, the loss of the sight prediction model to be constructed is determined based on the weighted result of the first sub-loss parameter, the second loss parameter, and the third loss parameter, and then it is determined whether the loss of the sight prediction model to be constructed meets the convergence condition of the sight prediction model to be constructed. If the loss of the sight prediction model to be constructed does not meet the convergence condition of the sight prediction model to be constructed, the model parameters of the sight prediction model to be constructed are updated based on the reference sight difference parameter and the sight difference parameter, the first eye image feature and the second eye image feature, and the reference sight difference parameter and the predicted sight difference parameter. Here, the convergence condition of the sight prediction model to be constructed may include that the loss of the sight prediction model to be constructed is less than or equal to the first preset threshold, or it may refer to that the loss of the sight prediction model to be constructed is stable, which is determined according to the actual application scenario and is not limited here.

[0079] When the supervision information of the eye sample image includes a reference line of sight difference parameter, the loss of the line of sight prediction model to be constructed is determined based on the weighted result of the second sub-loss parameter, the second loss parameter and the third loss parameter, and then it is determined whether the loss of the line of sight prediction model to be constructed meets the convergence condition of the line of sight prediction model to be constructed. If the loss of the line of sight prediction model to be constructed does not meet the convergence condition of the line of sight prediction model to be constructed, the model parameters of the line of sight prediction model to be constructed are updated based on the first annotated line of sight direction and the first predicted line of sight parameter, the second annotated line of sight direction and the second predicted line of sight parameter, the first eye image feature and the second eye image feature, and the reference line of sight difference parameter and the predicted line of sight difference parameter. Here, the convergence condition of the line of sight prediction model to be constructed may include that the loss of the line of sight prediction model to be constructed is less than or equal to the second preset threshold, or it may refer to that the loss of the line of sight prediction model to be constructed is stable, which is determined according to the actual application scenario and is not limited here.

[0080] In an optional manner, when the supervision information of the eye sample image includes a reference sight difference parameter, a first annotated sight direction, and a second annotated sight direction, the auxiliary prediction model may be an untrained or pre-trained network model. At this time, the auxiliary prediction model may be trained while the sight prediction model to be constructed is being trained. Based on this, the method of the exemplary embodiment of the present disclosure may further include: updating the model parameters of the auxiliary prediction model based on the reference sight difference parameter and the annotated sight difference parameter, and the annotated sight difference parameter may be determined by the first annotated sight direction and the second annotated sight direction. The model parameters here may include weights and / or offset values.

[0081] Since the first annotated sight line direction and the second annotated sight line direction are the "real labels" of the first eye sample image and the second eye sample image, respectively, the annotated sight line difference parameters determined based on the first annotated sight line direction and the second annotated sight line direction are also the "real labels" of the eye sample images. At this time, the loss of the currently trained auxiliary prediction model can be determined based on the reference sight line difference parameters and the annotated sight line difference parameters output by the auxiliary prediction model. If the loss of the currently trained auxiliary prediction model does not meet the convergence conditions of the auxiliary prediction model, it means that the convergence degree of the currently trained auxiliary prediction model is not high. At this time, the model parameters of the auxiliary prediction model can be updated based on the reference sight line difference parameters and the annotated sight line difference parameters; otherwise, there is no need to update the model parameters of the auxiliary prediction model.

[0082] Considering that the eye sample images used in each iterative training process may or may not have annotated sight lines, the method of the exemplary embodiment of the present disclosure may update the model parameters of the sight line prediction model to be constructed separately, or may simultaneously update the model parameters of the sight line prediction model to be constructed and the model parameters of the auxiliary prediction model. It can be understood that the update of the model parameters of the sight line prediction model to be constructed exists in each iterative training, while the update of the model parameters of the auxiliary prediction model may exist in the iterative training in which the first eye sample image has a first annotated sight line direction and the second eye sample image has a second annotated sight line direction. In other words, when the first eye sample image has a first annotated sight line direction and the second eye sample image has a second annotated sight line direction, and the loss of the currently trained auxiliary prediction model does not meet the convergence conditions of the auxiliary prediction model, the model parameters of the auxiliary prediction model will be updated based on the reference sight line difference parameters and the annotated sight line difference parameters.

[0083] Exemplarily, in each iterative training process, if the first eye sample image has a first annotated sight line direction, and the second eye sample image has a second annotated sight line direction, the exemplary embodiment of the present disclosure can determine the loss of the auxiliary prediction model based on the reference sight line difference parameter and the annotated sight line difference parameter determined by the first annotated sight line direction and the second annotated sight line direction, and then determine whether the loss of the auxiliary prediction model currently being trained meets the convergence condition of the auxiliary prediction model. If the loss of the auxiliary prediction model currently being trained does not meet the convergence condition of the auxiliary prediction model, it means that the convergence degree of the auxiliary prediction model currently being trained is not high, and there is a certain gap between the reference sight line difference parameter output by the auxiliary prediction model currently being trained and the annotated sight line difference parameter. At this time, the model parameters of the auxiliary prediction model can be updated based on the reference sight line difference parameter and the annotated sight line difference parameter, so that the reference sight line difference parameter output by the auxiliary prediction model in the next iterative training process is closer to the annotated sight line difference parameter, and the sight line difference prediction accuracy of the auxiliary prediction model is higher, thereby making the prediction sight line difference parameter output by the added branch in the sight line prediction model to be constructed in the next iterative training process more accurate. Here, the convergence condition of the auxiliary prediction model may include that the loss of the auxiliary prediction model is less than or equal to a second preset threshold, or may refer to that the loss of the auxiliary prediction model is stable, which is determined according to the actual application scenario and is not limited here.

[0084] Based on this, if the loss of the auxiliary prediction model does not meet the convergence condition of the auxiliary prediction model, the method of the exemplary embodiment of the present disclosure may further include: determining a marked sight difference parameter based on the first marked sight direction and the second marked sight direction; and calibrating the reference sight difference parameter based on the marked sight difference parameter. Here, the determination of the marked sight difference parameter can refer to the above text and will not be repeated here.

[0085] If the loss of the auxiliary prediction model does not meet the loss convergence condition, it means that the convergence degree of the auxiliary prediction model currently being trained is not high, and there is a certain gap between the reference sight difference parameters output by the auxiliary prediction model currently being trained and the annotated sight difference parameters. The exemplary embodiment of the present disclosure can update the model parameters of the auxiliary prediction model based on the reference sight difference parameters and the annotated sight difference parameters, and can also calibrate the reference sight difference parameters based on the annotated sight difference parameters, so that the calibrated reference sight difference parameters are closer to the annotated sight difference parameters. Therefore, the third loss parameter determined based on the calibrated reference sight difference parameters and the predicted sight difference parameters output by the additional branch in the sight prediction model to be constructed is smaller. At this time, the prediction accuracy of the predicted sight difference parameters output by the additional branch in the sight prediction model to be constructed is higher. At the same time, the method of the exemplary embodiment of the present disclosure can also update the model parameters of the sight prediction model to be constructed based on the calibrated reference sight difference parameters and the predicted sight difference parameters output by the additional branch in the sight prediction model to be constructed during the current iteration, and can also improve the sight prediction accuracy of the sight prediction model to be constructed in the next iteration.

[0086] It can be seen that in the method of the exemplary embodiment of the present invention, since the updating of the model parameters of the line of sight prediction model to be constructed occurs in each training iteration process, in the iterative training process of updating the model parameters of the auxiliary prediction model, when the model parameters of the line of sight prediction model to be constructed are updated, the line of sight prediction model to be constructed will iteratively update the model parameters based on the reference line of sight difference parameters output by the auxiliary line of sight model and the predicted line of sight difference parameters output by the branch added in the line of sight prediction model to be constructed. Therefore, the line of sight prediction model to be constructed can learn the reference line of sight difference parameters output by the auxiliary prediction model. After multiple iterative training, the line of sight difference prediction accuracy of the auxiliary prediction model will gradually improve, and the improvement process of the line of sight difference prediction accuracy of the auxiliary line of sight model will also be learned by the line of sight prediction model to be constructed, so that the line of sight prediction model to be constructed can learn the generalization ability of the auxiliary line of sight model.

[0087] In an alternative approach, Figure 2 FIG. 1 shows a schematic diagram of a model architecture of an exemplary embodiment of the present disclosure. Figure 2 As shown, the model architecture of the exemplary embodiment of the present disclosure may include an auxiliary line of sight model 210 and a line of sight prediction model 220. The auxiliary prediction model 210 may include a first backbone network 211, a first feature splicing network 212 and a first differential prediction network 213. The line of sight prediction model 220 to be constructed may include a second backbone network 221 and a direction prediction network 222.

[0088] The first backbone network 211 contains a first basic module, which can be used to output a first eye image feature; the first feature stitching network 212 can be used to determine a first line of sight stitching feature based on the first eye image feature; the first difference prediction network 213 can be used to determine a reference line of sight difference parameter based on the first line of sight stitching feature.

[0089] Exemplarily, the first backbone network 211 can obtain the first eye image feature based on the first eye sample image and the second eye sample image, and then the first feature stitching network 212 can obtain the first sight feature of the first eye sample image and the second sight feature of the second eye sample image based on the first eye image feature, and stitch the first sight feature of the first eye sample image and the second sight feature of the second eye sample image into the first sight stitching feature. The first differential prediction network 213 fuses the first sight stitching feature to obtain the reference sight difference parameter. Based on this, since the first sight feature of the first eye sample image and the second sight feature of the second eye sample image have been spliced, individual differences such as the angle between the visual axis and the optical axis are eliminated. Therefore, when the reference sight difference parameter is used as the supervision information of the eye sample image, the sight deviation caused by individual differences can be eliminated, and the overfitting problem caused by "data discreteness" can be avoided.

[0090] The second backbone network 221 includes a second basic module, which can be used to output a second eye image feature; the direction prediction network 222 can be used to determine a first predicted sight line parameter and a second predicted sight line parameter based on the second eye image feature.

[0091] Exemplarily, the second backbone network 221 can obtain the second eye image feature based on the first eye sample image and the second eye sample image, the direction prediction network 222 can obtain the third line of sight feature of the first eye sample image based on the first eye sample image, and obtain the fourth line of sight feature of the second eye sample image based on the second eye sample image. At this time, the direction prediction network 222 can also obtain the first predicted line of sight parameter of the first eye sample image based on the third line of sight feature of the first eye sample image, and obtain the second predicted line of sight parameter of the second eye sample image based on the fourth line of sight feature of the second eye sample image.

[0092] Taking into account that the number of channels of the first eye image features output by the first basic module of each layer and the number of channels of the second eye image features output by the second basic module of the corresponding layer may be different, at this time, it is necessary to adjust the number of channels of the first eye image features output by the first basic module and the second eye image features output by the second basic module of the same layer to the same number of channels. Therefore, the eye feature difference parameters of each layer can be determined by the first eye image features and the second eye image features with the same number of channels.

[0093] For example, the feature map size of the first eye image feature is H 1 *W 1 *C 1 , the feature map size of the second eye image feature is H 2 *W 2 *C 2 After adjusting the number of channels, the feature map size of the first eye image feature is H 1 *W 1 , the feature map size of the second eye image feature is H 2 *W 2 , and then determining a second loss parameter based on the first eye image feature and the second eye image feature.

[0094] When the first backbone network contains multiple layers of first basic modules and the second backbone network contains multiple layers of second basic modules, the first basic modules and the second basic modules have the same number of layers, and each layer of the first basic modules corresponds to the second basic modules.

[0095] For example, when the number of layers of the first basic module and the second basic module are both 4, the first backbone network 211 may include 4 layers of first basic modules connected in series, each layer of first basic modules is used to extract first eye image features, the first layer of first basic modules can extract first layer first eye image features, the first layer of first eye image features are used as input of the second layer of first basic modules, the second layer of first basic modules extracts second layer first eye image features based on the first layer first eye image features, and so on, until the fourth layer of first basic modules extracts fourth layer first eye image features, and the fourth layer first eye image features are determined as the first eye image features. The second backbone network 221 may include four layers of second basic modules connected in series, each layer of second basic modules is used to extract second eye image features, the first layer of second basic modules can extract first layer of second eye image features, the first layer of second eye image features are used as input to the second layer of second basic modules, the second layer of second basic modules extract second layer of second eye image features based on the first layer of second eye image features, and so on, until the fourth layer of second basic modules extracts fourth layer of second eye image features, and the fourth layer of second eye image features are determined as second eye image features.

[0096] Based on this, the second loss parameter of the exemplary embodiment of the present disclosure is determined by multiple layers of eye feature difference parameters, and the eye feature difference parameters of each layer are determined by the first eye image features output by the first basic module of the corresponding layer and the second eye image features output by the second basic module of the corresponding layer.

[0097] Exemplarily, the exemplary embodiments of the present disclosure may first determine the eye feature difference parameter of each layer based on the first eye image feature output by the first basic module of the corresponding layer and the second eye image feature output by the second basic module of the corresponding layer; and then determine the second loss function based on the eye feature difference parameters of the multiple layers. For example, the exemplary embodiments of the present disclosure may determine the second loss function based on the weighted results of the eye feature difference parameters of the multiple layers, and the weights of the eye feature difference parameters of each layer may be the same or different, depending on the actual scenario, and are not specifically limited here.

[0098] It can be seen that the method of the exemplary embodiment of the present disclosure can determine the second loss parameter of the second eye image feature relative to the first eye image feature layer by layer, so that the line of sight prediction model can deeply learn the ability of the auxiliary prediction model to extract eye image features during the training stage, thereby improving the prediction accuracy of the line of sight prediction model in the reasoning stage.

[0099] In an optional manner, when the predicted sight line information can also include predicted sight line difference parameters, the sight line prediction model 220 to be constructed can also include a second feature stitching network 223 and a second differential prediction network 224. The second feature stitching network 223 can be used to determine the second sight line stitching feature based on the second eye image feature, and the second differential prediction network 224 can be used to determine the predicted sight line difference parameters based on the second sight line stitching feature. Here, the second feature stitching network 223 and the second differential prediction network 224 can be understood as branches added to the sight line prediction model 220 to be constructed mentioned above.

[0100] Exemplarily, the second backbone network 221 can obtain the second eye image feature based on the first eye sample image and the first eye sample image, and then the second feature stitching network 223 can obtain the fifth sight feature of the first eye sample image and the sixth sight feature of the second eye sample image based on the second eye image feature, and stitch the fifth sight feature of the first eye sample image and the sixth sight feature of the second eye sample image into the second sight stitching feature. The second difference prediction network 224 fuses the second sight stitching feature to obtain the predicted sight difference parameter.

[0101] It should be noted that, in the training stage of the line of sight prediction model, the line of sight prediction model 220 to be constructed may include a second backbone network 221 and a direction prediction network 222, as well as a second feature splicing network 223 and a second differential prediction network 224. In each iteration process, the second feature splicing network 223 and the second differential prediction network 224 may be used as additional branches in the line of sight prediction model 220 to be constructed mentioned above. When the second feature splicing network 223 and the second differential prediction network 224 output predicted line of sight difference parameters in the training stage, they may learn the process in which the first feature splicing network 212 and the first differential prediction network 213 in the auxiliary prediction model 210 output reference line of sight difference parameters, thereby improving the line of sight difference prediction accuracy of the auxiliary line of sight model through multiple iterative trainings. The line of sight prediction model to be constructed may also be learned, thereby enabling the line of sight prediction model to be constructed to learn the generalization ability of the auxiliary line of sight model. When the line of sight prediction model training is completed, the model architecture of the line of sight prediction model in the reasoning stage may include a second backbone network 221 and a direction prediction network 222. At this time, compared with the model architecture of the line of sight prediction model 220 to be constructed in the training stage, it is smaller and simpler, and more conducive to deployment on edge devices such as wearable devices.

[0102] The exemplary embodiments of the present disclosure also provide a line of sight prediction method, which can improve the line of sight prediction accuracy in the inference stage based on the line of sight prediction model trained by the exemplary embodiments of the present disclosure. It should be understood that the method of the exemplary embodiments of the present disclosure can be executed by an electronic device or by a chip applied to an electronic device. For details, please refer to the above text and will not be repeated here.

[0103] Figure 3 FIG. 4 is a flowchart of a sight line prediction method according to an exemplary embodiment of the present disclosure. Figure 3 As shown, the sight line prediction method of the exemplary embodiment of the present disclosure may include:

[0104] Step 301: Obtain a target eye image. The specific content of the target eye image is described in the above description of the eye image, which will not be described again here.

[0105] Step 302: Input the target eye image into a sight line prediction model to obtain sight line parameters corresponding to the target eye image, wherein the sight line prediction model is trained by the method of the exemplary embodiment of the present disclosure.

[0106] If the target eye image is an unlabeled eye sample image, the sight line prediction model can be used to predict the sight line parameters corresponding to the target eye image. If the target eye image is a labeled eye sample image, the sight line parameters corresponding to the target eye image predicted by the sight line prediction model can be used to calibrate the supervision information of the labeled eye sample image.

[0107] It can be seen that the method of the exemplary embodiment of the present disclosure can use the sight prediction model trained by the exemplary embodiment of the present disclosure to predict the sight parameters corresponding to the target eye image. The sight prediction model uses the sight difference description information output by the auxiliary prediction model as the supervision information of the eye sample image during the training stage, which can not only reduce the sample annotation cost, but also eliminate the sight deviation caused by individual differences such as the angle between the eye visual axis and the optical axis, and can also avoid overfitting caused by "sample data discretization" during the eye sample image acquisition process. At the same time, the sight prediction model finally obtained can be directly used to predict the sight direction corresponding to the eye image to be tested in the reasoning stage without first collecting calibration images for calibration. At the same time, after the sight prediction model training is completed, the model architecture of the sight prediction model in the reasoning stage is smaller and simpler than the model architecture of the sight prediction model to be constructed in the training stage, which is more conducive to deployment in edge devices such as wearable devices.

[0108] One or more technical solutions provided in the exemplary embodiments of the present disclosure can input the first eye sample image and the second eye sample image into the auxiliary prediction model to obtain the sight difference description information, and input the first eye sample image and the second eye sample image into the sight prediction model to be constructed to obtain the predicted sight information, and the predicted sight information can include the first predicted sight parameter of the first eye sample image and the second predicted sight parameter of the second eye sample image. Since the sight difference description information can be used to characterize the sight difference between the first eye sample image and the second eye sample image, when the exemplary embodiments of the present disclosure use the sight difference description information output by the auxiliary prediction model as the supervision information of the eye sample image, it can not only reduce the sample annotation cost, but also eliminate the sight deviation caused by individual differences such as the angle between the eye visual axis and the optical axis, and avoid overfitting caused by "discrete sample data" during the eye sample image acquisition process. At the same time, the sight prediction model finally obtained can be directly used to predict the sight direction corresponding to the eye image to be tested in the inference stage, without the need to first collect a calibration image for calibration.

[0109] The above mainly introduces the scheme provided by the embodiment of the present disclosure. It is understandable that in order to realize the above functions, the electronic device includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiment disclosed in this article, the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present disclosure.

[0110] The disclosed embodiment can divide the electronic device into functional units according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the disclosed embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0111] In the case of dividing each functional module according to each function, an exemplary embodiment of the present disclosure provides a device for constructing a sight line prediction model, and the device for constructing a sight line prediction model may be an electronic device or a chip applied to an electronic device. Figure 4 FIG. 1 is a schematic block diagram of a module of a device for constructing a sight line prediction model according to an exemplary embodiment of the present disclosure. Figure 4 As shown, the device 400 includes:

[0112] The training module 401 is used to input the first eye sample image and the second eye sample image into the auxiliary prediction model to obtain line of sight difference description information, where the line of sight difference description information is used to characterize the line of sight difference between the first eye sample image and the second eye sample image; input the first eye sample image and the second eye sample image into the line of sight prediction model to be constructed to obtain predicted line of sight information, where the predicted line of sight information includes a first predicted line of sight parameter of the first eye sample image and a second predicted line of sight parameter of the second eye sample image; and determine the loss of the line of sight prediction model to be constructed based on the line of sight difference description information and the predicted line of sight information;

[0113] The determination module 402 is used to determine that the sight line prediction model to be constructed is a sight line prediction model if the loss of the sight line prediction model to be constructed meets a convergence condition.

[0114] As a possible implementation manner, the sight difference description information includes a first eye image feature and a reference sight difference parameter, the predicted sight information also includes a second eye image feature, and the loss of the sight prediction model to be constructed includes a first loss parameter and a second loss parameter;

[0115] Among them, the first loss parameter is determined by the first predicted line of sight parameter, the second predicted line of sight parameter and the supervision information of the eye sample image, the supervision information of the eye sample image includes a reference line of sight difference parameter, and the second loss parameter is determined by the first eye image feature and the second eye image feature.

[0116] As a possible implementation manner, the loss of the sight line prediction model to be constructed is determined by a weighted result of a first loss parameter and a second loss parameter, and the weight of the second loss parameter is greater than the weight of the first loss parameter.

[0117] As a possible implementation manner, the first loss parameter includes a first sub-loss parameter and a second sub-loss parameter;

[0118] The first sub-loss parameter is determined by a reference sight difference parameter and a sight difference parameter, and the sight difference parameter is determined by a difference between a first predicted sight parameter and a second predicted sight parameter;

[0119] The supervision information of the eye sample image also includes a first annotated sight direction of the first eye sample image and a second annotated sight direction of the second eye sample image. The second sub-loss parameter is determined by the first predicted loss parameter and the second predicted loss parameter. The first predicted loss parameter is determined by the first annotated sight direction and the first predicted sight parameter. The second predicted loss parameter is determined by the second annotated sight direction and the second predicted sight parameter.

[0120] As a possible implementation method, the predicted line of sight information also includes a predicted line of sight difference parameter, and the loss of the line of sight prediction model to be constructed also includes a third loss parameter, and the third loss parameter is determined by the predicted line of sight difference parameter and the reference line of sight difference parameter. The training module 401 is also used to update the model parameters of the auxiliary prediction model based on the reference line of sight difference parameter and the labeled line of sight difference parameter, and the labeled line of sight difference parameter is determined by the first labeled line of sight direction and the second labeled line of sight direction.

[0121] As a possible implementation method, the loss of the sight prediction model to be constructed is determined by the weighted result of the first loss parameter, the second loss parameter and the third loss parameter, and the weight of the first loss parameter and the weight of the third loss parameter are both smaller than the weight of the second loss parameter.

[0122] As a possible implementation, the determination module 402 is further configured to determine a marked sight line difference parameter based on the first marked sight line direction and the second marked sight line direction;

[0123] The apparatus 400 further includes a calibration module 403, configured to calibrate the reference sight difference parameter based on the marked sight difference parameter.

[0124] As a possible implementation, the auxiliary prediction model includes a first backbone network, a first feature concatenation network, and a first differential prediction network, and the sight line prediction model to be constructed includes a second backbone network and a direction prediction network;

[0125] The first backbone network includes a first basic module for outputting a first eye image feature, a first feature stitching network for determining a first sight stitching feature based on the first eye image feature, and a first difference prediction network for determining a reference sight difference parameter based on the first sight stitching feature;

[0126] The second backbone network includes a second basic module for outputting a second eye image feature, and the direction prediction network is used to determine a first predicted sight line parameter and a second predicted sight line parameter based on the second eye image feature.

[0127] As a possible implementation method, the first backbone network contains multiple layers of first basic modules, and the second backbone network contains multiple layers of second basic modules. Each layer of the first basic modules corresponds to the second basic modules. The second loss parameter is determined by the multiple layers of eye feature difference parameters, and each layer of the eye feature difference parameters is determined by the first eye image features output by the first basic module of the corresponding layer and the second eye image features output by the second basic module of the corresponding layer.

[0128] As a possible implementation method, when the predicted line of sight information also includes predicted line of sight difference parameters, the line of sight prediction model to be constructed also includes a second feature stitching network and a second differential prediction network, the second feature stitching network is used to determine the second line of sight stitching feature based on the second eye image feature, and the second differential prediction network is used to determine the predicted line of sight difference parameters based on the second line of sight stitching feature.

[0129] An exemplary embodiment of the present disclosure further provides a sight line prediction device, which may be an electronic device or a chip applied to an electronic device. Figure 5 A schematic block diagram of modules of a sight line prediction device according to an exemplary embodiment of the present disclosure is shown.

[0130] like Figure 5 As shown, the device 500 includes:

[0131] An acquisition module 501 is used to acquire a target eye image;

[0132] The prediction module 502 is used to input the target eye image into the sight line prediction model to obtain the sight line parameters corresponding to the target eye image. The sight line prediction model is trained by the device described in the exemplary embodiment of the present disclosure.

[0133] Figure 6 Schematic block diagram of a chip of an exemplary embodiment of the present disclosure is shown. Figure 6 As shown, the chip 600 includes one or more (including two) processors 601 and a communication interface 602. The communication interface 602 can support the server to perform the data sending and receiving steps in the above method, and the processor 601 can support the server to perform the data processing steps in the above method.

[0134] Optional, such as Figure 6As shown, the chip 600 also includes a memory 603, which may include a read-only memory and a random access memory, and provides operation instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory (NVRAM).

[0135] In some embodiments, Figure 6 As shown, the processor 601 performs corresponding operations by calling the operation instructions stored in the memory (the operation instructions may be stored in the operating system). The processor 601 controls the processing operations of any one of the terminal devices, and the processor may also be called a central processing unit (CPU). The memory 603 may include a read-only memory and a random access memory, and provides instructions and data to the processor 601. A portion of the memory 603 may also include NVRAM. For example, in an application, the memory, the communication interface, and the memory are coupled together through a bus system, wherein the bus system may include a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, in Figure 6 Various buses are labeled as bus system 604 .

[0136] The method disclosed in the above-mentioned embodiment of the present disclosure can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above-mentioned processor may be a general-purpose processor, a digital signal processor (digital signal processing, DSP), an ASIC, a field-programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiment of the present disclosure can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware.

[0137] The exemplary embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to cause the electronic device to perform the method according to the embodiment of the present disclosure when executed by the at least one processor.

[0138] The exemplary embodiments of the present disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform the method according to the embodiments of the present disclosure.

[0139] The exemplary embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform the method according to the embodiments of the present disclosure.

[0140] refer to Figure 7 , a block diagram of an electronic device 700 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0141] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0142] A plurality of components in the electronic device 700 are connected to the I / O interface 705, including: an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709. The input unit 707 may be any type of device capable of inputting information to the electronic device 700, and the input unit 706 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 707 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 708 may include, but is not limited to, a disk, an optical disk. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0143] like Figure 7 As shown, the computing unit 701 can be various general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above. For example, in some embodiments, the method of the exemplary embodiments of the present disclosure may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 700 via ROM 702 and / or communication unit 709. In some embodiments, the computing unit 701 can be configured to execute the method in any other appropriate manner (e.g., by means of firmware).

[0144] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0145] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0146] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0147] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0148] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0149] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0150] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present disclosure is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a terminal, a user device or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium may be a magnetic medium, for example, a floppy disk, a hard disk, a tape; it may also be an optical medium, for example, a digital video disc (DVD); it may also be a semiconductor medium, for example, a solid state drive (SSD).

[0151] Although the present disclosure has been described in conjunction with specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present disclosure. Accordingly, this specification and the drawings are merely exemplary illustrations of the present disclosure as defined by the appended claims, and are deemed to have covered any and all modifications, variations, combinations or equivalents within the scope of the present disclosure. Obviously, those skilled in the art may make various modifications and variations to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A method for constructing a sight line prediction model, It is characterized in that The method comprises: Inputting the first eye sample image and the second eye sample image into the auxiliary prediction model to obtain sight difference description information, where the sight difference description information is used to characterize the sight difference between the first eye sample image and the second eye sample image; Inputting the first eye sample image and the second eye sample image into a sight line prediction model to be constructed to obtain predicted sight line information, where the predicted sight line information includes a first predicted sight line parameter of the first eye sample image and a second predicted sight line parameter of the second eye sample image; Determining the loss of the sight line prediction model to be constructed based on the sight line difference description information and the predicted sight line information; If the loss of the sight line prediction model to be constructed does not meet the convergence condition, the model parameters of the sight line prediction model to be constructed are updated based on the sight line difference description information and the predicted sight line information; otherwise, the sight line prediction model to be constructed is determined to be a sight line prediction model.

2. The method according to claim 1, It is characterized in that The sight line difference description information includes a first eye image feature and a reference sight line difference parameter, the predicted sight line information also includes a second eye image feature, and the loss of the sight line prediction model to be constructed includes a first loss parameter and a second loss parameter; The first loss parameter is determined by the first predicted line of sight parameter, the second predicted line of sight parameter and the supervision information of the eye sample image, the supervision information of the eye sample image includes the reference line of sight difference parameter, and the second loss parameter is determined by the first eye image feature and the second eye image feature.

3. The method according to claim 2, It is characterized in that The loss of the sight line prediction model to be constructed is determined by a weighted result of the first loss parameter and the second loss parameter, and the weight of the second loss parameter is greater than the weight of the first loss parameter.

4. The method according to claim 2, It is characterized in that The first loss parameter includes a first sub-loss parameter and a second sub-loss parameter; The first sub-loss parameter is determined by the reference sight difference parameter and a sight difference parameter, and the sight difference parameter is determined by the difference between the first predicted sight parameter and the second predicted sight parameter; The supervision information of the eye sample image also includes a first annotated sight line direction of the first eye sample image and a second annotated sight line direction of the second eye sample image, the second sub-loss parameter is determined by the first predicted loss parameter and the second predicted loss parameter, the first predicted loss parameter is determined by the first annotated sight line direction and the first predicted sight line parameter, and the second predicted loss parameter is determined by the second annotated sight line direction and the second predicted sight line parameter.

5. The method according to claim 4, It is characterized in that The predicted sight line information further includes a predicted sight line difference parameter, the loss of the sight line prediction model to be constructed further includes a third loss parameter, and the third loss parameter is determined by the predicted sight line difference parameter and the reference sight line difference parameter, and the method further includes: Based on the reference sight difference parameter and the marked sight difference parameter, the model parameters of the auxiliary prediction model are updated, and the marked sight difference parameter is determined by the first marked sight direction and the second marked sight direction.

6. The method according to claim 5, It is characterized in that The loss of the sight line prediction model to be constructed is determined by a weighted result of the first loss parameter, the second loss parameter and the third loss parameter, and the weight of the first loss parameter and the weight of the third loss parameter are both smaller than the weight of the second loss parameter.

7. The method according to claim 4, It is characterized in that The method further comprises: Determining a marked sight line difference parameter based on the first marked sight line direction and the second marked sight line direction; The reference sight difference parameter is calibrated based on the marked sight difference parameter.

8. The method according to any one of claims 2 to 7, It is characterized in that The auxiliary prediction model includes a first backbone network, a first feature splicing network and a first differential prediction network, and the sight line prediction model to be constructed includes a second backbone network and a direction prediction network; The first backbone network includes a first basic module, which is used to output the first eye image feature, the first feature stitching network is used to determine a first sight stitching feature based on the first eye image feature, and the first difference prediction network is used to determine the reference sight difference parameter based on the first sight stitching feature; The second backbone network contains a second basic module for outputting the second eye image feature, and the direction prediction network is used to determine the first predicted sight line parameter and the second predicted sight line parameter based on the second eye image feature.

9. The method according to claim 8, It is characterized in that The first backbone network contains multiple layers of first basic modules, and the second backbone network contains multiple layers of second basic modules. The first basic modules in each layer correspond to the second basic modules. The second loss parameter is determined by multiple layers of eye feature difference parameters. The eye feature difference parameters in each layer are determined by the first eye image features output by the first basic module in the corresponding layer and the second eye image features output by the second basic module in the corresponding layer.

10. The method according to claim 8, It is characterized in that When the predicted line of sight information also includes predicted line of sight difference parameters, the line of sight prediction model to be constructed also includes a second feature stitching network and a second differential prediction network, the second feature stitching network is used to determine a second line of sight stitching feature based on the second eye image feature, and the second differential prediction network is used to determine the predicted line of sight difference parameters based on the second line of sight stitching feature.

11. A line of sight prediction method, It is characterized in that The method comprises: Acquire a target eye image; The target eye image is input into a sight line prediction model to obtain sight line parameters corresponding to the target eye image, wherein the sight line prediction model is trained by the method described in any one of claims 1 to 10.

12. A device for constructing a sight line prediction model, It is characterized in that The device comprises: A training module, used for inputting a first eye sample image and a second eye sample image into an auxiliary prediction model to obtain line of sight description information, wherein the line of sight description information is used to characterize the line of sight difference between the first eye sample image and the second eye sample image; inputting the first eye sample image and the second eye sample image into a line of sight prediction model to be constructed to obtain predicted line of sight information, wherein the predicted line of sight information includes a first predicted line of sight parameter of the first eye sample image and a second predicted line of sight parameter of the second eye sample image; determining a loss of the line of sight prediction model to be constructed based on the line of sight description information and the predicted line of sight information; if the loss of the line of sight prediction model to be constructed does not meet a convergence condition, updating the model parameters of the line of sight prediction model to be constructed based on the line of sight description information and the predicted line of sight information; A determination module is used to determine that the sight line prediction model to be constructed is a sight line prediction model if the loss of the sight line prediction model to be constructed meets a convergence condition.

13. A sight line prediction device, It is characterized in that The device comprises: An acquisition module, used for acquiring a target eye image; A prediction module is used to input the target eye image into a sight line prediction model to obtain the sight line corresponding to the target eye image, and the sight line prediction model is trained by the device described in claim 12.

14. An electronic device, It is characterized in that include: processor; as well as, A memory for storing programs; The program includes instructions, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 10.

15. A non-transitory computer-readable storage medium storing computer instructions, It is characterized in that The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 10.

Images