Network model construction method, sight line estimation method, device, equipment and medium

Abstract

The embodiment of the invention relates to a construction method of a network model, a sight line estimation method and device, equipment and a medium. The construction method of the network model comprises the following steps: acquiring a plurality of training image groups; obtaining a sight line difference theoretical value corresponding to the training image group according to the angle difference between the theoretical sight line directions marked by the two eye sample images in the training image group, and determining a first weight corresponding to the training image group according to the sight line difference theoretical value; acquiring sight line difference estimated values output by a preset initial network model for the plurality of training image groups; according to the first weights corresponding to the plurality of training image groups, performing weighting processing on loss between the sight line difference theoretical values and the sight line difference estimated values corresponding to the plurality of training image groups to obtain weighted loss; and training the initial network model based on the weighted loss so as to construct a target network model based on the trained initial network model. According to the embodiment of the invention, the accuracy of sight direction estimation can be effectively guaranteed.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a network model construction method, a line of sight estimation method, a device, a equipment and a medium. Background Art

[0002] In many scenarios such as gaming, medical, and intelligent control, it is necessary to identify the direction of human gaze so that corresponding strategies can be adopted based on the estimated results of the human gaze direction. Usually, it is necessary to estimate the gaze direction with the help of a network model. However, the existing network model training method is not good, resulting in poor accuracy of gaze estimation using the output results of the network model. Summary of the invention

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a network model construction method, a line of sight estimation method, an apparatus, a device and a medium.

[0004] In a first aspect, an embodiment of the present disclosure provides a method for constructing a network model, comprising: obtaining a plurality of training image groups; wherein the training image group includes two eye sample images, and the eye sample images are annotated with theoretical sight lines; obtaining a sight line difference theoretical value corresponding to the training sample group according to the angle difference between the theoretical sight line directions annotated by the two eye sample images in the training image group, and determining a first weight corresponding to the training image group according to the sight line difference theoretical value; obtaining sight line difference estimation values ​​output by a preset initial network model for the plurality of training image groups respectively; wherein the sight line difference estimation value is obtained by estimating the angle difference between the sight line directions of the two eye sample images in the training image group by the initial network model; performing weighted processing on the loss between the sight line difference theoretical value and the sight line difference estimation value corresponding to the plurality of training image groups according to the first weights corresponding to the plurality of training image groups respectively, to obtain a weighted loss; training the initial network model based on the weighted loss to construct a target network model based on the trained initial network model; wherein the target network model is used to generate a sight line difference estimation value between two target eye images.

[0005] In a second aspect, an embodiment of the present disclosure provides a line of sight estimation method, comprising: obtaining a calibration eye image marked with a theoretical line of sight direction and a target eye image whose line of sight direction is to be estimated; obtaining an estimated value of the line of sight difference between the calibration eye image and the target eye image through a pre-constructed target network model; wherein the target network model is obtained based on the network model construction method provided in the first aspect; and estimating the line of sight direction corresponding to the target eye image according to the theoretical line of sight direction of the calibration eye image and the estimated value of the line of sight difference between the calibration eye image and the target eye image.

[0006] In a third aspect, an embodiment of the present disclosure provides a device for constructing a network model, comprising: an image group acquisition module, used to acquire multiple training image groups; wherein the training image group includes two eye sample images, and the eye sample images are annotated with theoretical sight lines; a first weight determination module, used to obtain a theoretical sight line difference value corresponding to the training sample group according to the angle difference between the theoretical sight line directions annotated by the two eye sample images in the training image group, and determine a first weight corresponding to the training image group according to the theoretical sight line difference value; a network model output module, used to obtain sight line difference estimates output by a preset initial network model for the multiple training image groups respectively. value; wherein the line of sight difference estimation value is estimated by the initial network model for the angular difference between the line of sight directions of the two eye sample images in the training image group; a weighted loss acquisition module, used to weight the loss between the line of sight difference theoretical value and the line of sight difference estimation value corresponding to each of the multiple training image groups according to the first weights corresponding to each of the multiple training image groups, so as to obtain a weighted loss; a network model training module, used to train the initial network model based on the weighted loss, so as to construct a target network model based on the trained initial network model; wherein the target network model is used to generate the line of sight difference estimation value between the two target eye images.

[0007] In a fourth aspect, an embodiment of the present disclosure provides a line of sight estimation device, comprising: an image acquisition module, used to acquire a calibration eye image marked with a theoretical line of sight direction and a target eye image whose line of sight direction is to be estimated; a line of sight difference estimation module, used to acquire an estimated value of the line of sight difference between the calibration eye image and the target eye image through a pre-constructed target network model; wherein the target network model is obtained based on the network model construction method provided in the first aspect; and a line of sight direction estimation module, used to estimate the line of sight direction corresponding to the target eye image according to the theoretical line of sight direction of the calibration eye image and the estimated value of the line of sight difference between the calibration eye image and the target eye image.

[0008] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement the network model construction method of the first aspect, or to implement the line of sight estimation method of the second aspect.

[0009] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the method for constructing a network model of the first aspect, or to implement the line of sight estimation method of the second aspect.

[0010] The above-mentioned technical solution provided by the embodiment of the present disclosure can further determine the first weight corresponding to the training image group according to the theoretical value of the line of sight difference corresponding to the training image group, based on obtaining the estimated values ​​of the line of sight difference output by the preset initial network model for multiple training image groups respectively, and the theoretical value of the line of sight difference corresponding to the training image group, so as to perform weighted processing on the loss between the theoretical value of the line of sight difference and the estimated value of the line of sight difference corresponding to each of the multiple training image groups according to the first weights corresponding to each of the multiple training image groups, thereby performing model training based on the weighted loss. This method can fully consider the influence of the line of sight difference between the two eye images on the output result of the model. During the model training process, the corresponding first weight is determined for the theoretical value of the line of sight difference corresponding to the training image group. The influence of the line of sight difference on the model training is introduced by setting the first weight, which helps to make the line of sight difference estimation result output by the target network model obtained by the final training more accurate. On this basis, the accuracy of the line of sight direction estimation result based on the line of sight difference estimated value between the calibration eye image and the target eye image output by the target network model and the theoretical line of sight direction annotated on the calibration eye image is further guaranteed.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0013] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0014] Figure 1 A schematic diagram of an angle panel provided in an embodiment of the present disclosure;

[0015] Figure 2 A schematic diagram of an angle panel provided in an embodiment of the present disclosure;

[0016] Figure 3 A schematic diagram of a flow chart of a method for constructing a network model provided in an embodiment of the present disclosure;

[0017] Figure 4 A flowchart of a line of sight estimation method provided by an embodiment of the present disclosure;

[0018] Figure 5 A schematic diagram of a line of sight estimation process provided by an embodiment of the present disclosure;

[0019] Figure 6 A schematic diagram of the structure of a network model construction device provided in an embodiment of the present disclosure;

[0020] Figure 7 A schematic diagram of the structure of a sight line estimation device provided by an embodiment of the present disclosure;

[0021] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0023] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0024] In the field of line of sight estimation, it is usually necessary to obtain multiple eye sample images and annotate the theoretical line of sight directions for the eye sample images, so as to form multiple training image groups with multiple eye sample images and train the network model used to estimate the line of sight difference between two eye images. For the convenience of research, the theoretical line of sight direction can be represented by the corresponding angle, and see Figure 1 A schematic diagram of an angle panel is shown, showing multiple concentric circles, and different circles correspond to different angles. In some embodiments, Figure 1 The angle panel shown in FIG. 1 analyzes the acquired eye sample images, for example, based on the angles corresponding to the acquired eye sample images, Figure 1 In order to analyze the angle distribution of the eye sample image, annotate it in Figure 1 Based on the above, multiple calibration points and one test point are set, and the model prediction is performed by combining the test point with each calibration point, and the accuracy of the model prediction result is measured. For ease of understanding, the following is a detailed explanation:

[0025] In practical applications, multiple calibration points can be set, and the user can be asked to look at the specified calibration points to collect the user's eye image as the calibration eye image. In addition, a test point is determined, and the eye image of the user when looking at the test point is obtained as the test eye image. For each calibration eye image, the network model is used to estimate the line of sight difference between the test eye image and the calibration eye image, and an estimated line of sight difference is obtained. Combined with the theoretical line of sight direction of the calibration eye image, the line of sight estimation result of the test eye image can be obtained. After that, the line of sight estimation results corresponding to the multiple calibration eye images are weighted to obtain the final line of sight estimation result of the model for the test eye image. For ease of understanding, please refer to Figure 2 A schematic diagram of an angle panel is shown, and the black circles are calibration points. For each calibration point, the calibration point corresponds to Figure 2 The angle in is the angle corresponding to the theoretical sight direction of the calibration eye image collected when the user looks at the calibration point; the black square is the actual test point, which corresponds to Figure 2 The angle in is the angle corresponding to the theoretical sight direction of the test eye image collected when the user looks at the test point; the gray circle is the predicted result point obtained by each calibration point relative to the test point, and the predicted result point corresponds to Figure 2 The angle in is the angle predicted by the network model based on each calibration point and the test point (that is, the model estimated angle corresponding to the test eye image); the black triangle is the comprehensive prediction result point obtained by combining multiple calibration points relative to the test point, and the comprehensive prediction result point corresponds to Figure 2 The angle in is the angle obtained by weighted averaging and other fusion processing of multiple model estimated angles corresponding to the test eye image. For the convenience of intuitive analysis, the position of each point can be directly observed below. Figure 2 It can be intuitively known that the accuracy of the angle prediction result of the calibration point close to the test point relative to the test point is relatively high (that is, the closer the corresponding gray circle is to the black square), and the accuracy of the comprehensive prediction result point obtained based on multiple calibration points relative to the test point is likely to depend on the calibration points close to the test point, that is, the degree of proximity between the black triangle and the black square mainly depends on the gray circle closer to the black square.

[0026] However, the inventors have discovered through research that there are significant differences in the application methods of the above-mentioned model testing stage and the model training stage. The main reason is that, as analyzed above, the model testing stage will be affected by the size of the line of sight difference between the two eye images (the closer the calibration point is to the test point, the smaller the line of sight difference is, and the greater the impact on the line of sight prediction result finally obtained by the model). In the model training stage, the two eye sample images in the training image group input to the model are random, and the impact of the line of sight difference between the eye sample images on the model training is not considered. The degree of influence of training image groups with different line of sight difference sizes on model training is consistent, which leads to low accuracy in estimating the line of sight direction of the eye image using the trained model in the testing stage or the subsequent reasoning stage.

[0027] To at least improve the above problems, the following embodiments of the present disclosure may be referred to: Figure 3 The flowchart of a method for constructing a network model is shown in FIG. 1 , which can be executed by a device for constructing a network model, wherein the device can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 3 As shown, the method mainly includes the following steps S302 to S310:

[0028] Step S302, obtaining a plurality of training image groups; wherein the training image groups include two eye sample images, and the eye sample images are annotated with theoretical sight lines directions.

[0029] The eye sample image may be an eye image taken when the target eye is looking at a preset fixation point. The target eye may be the eye of a target object (such as a human being). In some embodiments, multiple fixation points may be pre-set, and eye images may be taken when the target eye is looking at each fixation point, so as to obtain an eye sample image corresponding to each fixation point. In practical applications, the user may wear an electronic device such as VR glasses, smart glasses, an eye tracker, or other electronic devices that need to estimate the direction of the user's eye line. The electronic device may sense the state of the user's eyeballs, thereby determining the direction of the eye line when the eye is staring at the fixation point, and use the line of sight direction obtained by the electronic device as the theoretical line of sight direction of the collected eye sample image, thereby serving as a label for the eye sample image. In practical applications, before collecting the user's eye image, a collection prompt message may be initiated to the user, and the user may be further informed of the purpose of image collection in the prompt message. In other embodiments, the eye image may be obtained through a network or other means, and then the theoretical line of sight direction may be annotated for the eye image using an existing model. In practical applications, the theoretical line of sight direction may be represented by a spatial angle, which may be understood as the angle between the line of sight direction when the eye is staring at the fixation point and the reference direction. In some specific embodiments, the theoretical line of sight direction can be directly represented by the yaw angle (yaw) and pitch angle (pitch) of the eyeball; in other embodiments, the theoretical line of sight direction can also be represented by a target angle calculated based on the yaw angle (yaw) and the pitch angle (pitch), and the target angle can be used to represent the angle between the line of sight direction when the eyes are staring at the gaze point and the line of sight direction when the eyes are staring at the gaze point horizontally. The above are only two exemplary descriptions of the line of sight direction. In practical applications, the characterization method of the line of sight direction can be flexibly set and is not limited here. In order to ensure the reliability of model training, multiple training image groups can be obtained. The embodiment of the present disclosure does not limit the number of training image groups. Generally speaking, the more training image groups there are, the more robust the trained model is.

[0030] Step S304, according to the angle difference between the theoretical sight lines marked by the two eye sample images in the training image group, obtain the sight line difference theoretical value corresponding to the training image group, and determine the first weight corresponding to the training image group according to the sight line difference theoretical value. Each of the multiple training image groups obtained corresponds to a first weight, and the first weights corresponding to different training image groups may be the same or different, mainly depending on the sight line difference theoretical value of the training image group.

[0031] In practical applications, the angle difference between theoretical sight lines can be directly represented by the yaw angle difference and the pitch angle difference, or can be represented by the target angle difference determined based on the yaw angle difference and the pitch angle difference, which is not limited here.

[0032] In the embodiments of the present disclosure, the influence of the sight difference of the training image group on the model training is fully considered. Compared with the related art, the embodiments of the present disclosure will additionally determine the first weight corresponding to the training image group according to the sight difference theoretical value corresponding to the training image group, so as to control the influence of different training image groups on the model training based on the first weight. In some implementation examples, for any two training image groups, the first weight corresponding to the training image group with a small sight difference theoretical value is not lower than the first weight corresponding to the training image group with a large sight difference theoretical value. That is, the training image group with a small sight difference theoretical value has a greater influence on the model training, so that the trained model can focus more on the sight difference estimation results of two eye images with a small sight difference, thereby matching the model application method in the subsequent testing stage or reasoning stage.

[0033] Step S306, obtaining the sight difference estimation values ​​output by the preset initial network model for multiple training image groups respectively; wherein the sight difference estimation value is obtained by estimating the angular difference between the sight directions of two eye sample images in the training image group by the initial network model.

[0034] The initial network model may be a neural network model, and the embodiments of the present disclosure do not limit the structure of the initial network model. For example, the initial network model may be a differential model, which may specifically include a feature extraction network and a sight difference prediction network, wherein the feature extraction network is used to extract a feature vector of each eye sample image in the training image group, and the sight difference prediction network may analyze and process the feature vectors of two eye sample images in the training image group, thereby generating a sight difference estimation value (also referred to as a gaze difference value).

[0035] Step S308 , weighting the losses between the theoretical sight difference values ​​and the estimated sight difference values ​​corresponding to the plurality of training image groups according to the first weights corresponding to the plurality of training image groups, to obtain weighted losses.

[0036] In practical applications, each training image group corresponds to a first weight, and the first weight can be directly used to perform weighted averaging on the losses between the theoretical line of sight difference values ​​and the estimated line of sight difference values ​​corresponding to each of the multiple training image groups to obtain a weighted loss. Other weights can also be additionally introduced to obtain a comprehensive weight based on the first weight and other weights, and then the comprehensive weight is used to perform weighted averaging on the losses between the theoretical line of sight difference values ​​and the estimated line of sight difference values ​​corresponding to each of the multiple training image groups to obtain a weighted loss. The specific weight can be flexibly set and is not limited here. For each training image group, the loss between its corresponding theoretical line of sight difference value and the estimated line of sight difference value can be obtained by referring to the following method: Based on the difference between the theoretical line of sight difference value and the estimated line of sight difference value corresponding to the training image group, a preset loss function is used to determine the loss function value, and the loss function value is the loss between the theoretical line of sight difference value and the estimated line of sight difference value corresponding to the training image group. The embodiment of the present disclosure does not limit the loss function.

[0037] Step S310, training the initial network model based on the weighted loss to construct a target network model based on the trained initial network model; wherein the target network model is used to generate an estimated value of the line of sight difference between the two target eye images.

[0038] The initial network model is trained based on the weighted loss, and the parameters of the network model can be adjusted with the goal of reducing the weighted loss to minimize the difference, until the training is stopped when the preset training end condition is reached (such as the total loss converges to a preset threshold range). Since the weighted loss fully integrates the first weights determined based on the theoretical value of the line of sight difference for different training image groups, the target network model obtained by model training based on the weighted loss can estimate the line of sight difference more accurately, and can be better matched with the application method in the model testing or reasoning stage.

[0039] In summary, the above method can fully consider the influence of the sight difference between the two eye images on the model output results. During the model training process, the corresponding first weight is determined for the theoretical value of the sight difference corresponding to the training image group. By setting the first weight, the influence of the sight difference on the model is introduced, which helps to make the sight difference estimation result output by the final target network model more accurate.

[0040] In order to ensure the reliability of the first weight, the embodiment of the present disclosure provides an implementation method for determining the first weight corresponding to the training image group according to the sight difference theoretical value, which can be performed with reference to the following steps 1 and 2:

[0041] Step 1: Obtain a preset sight difference threshold; wherein the number of the sight difference thresholds is one or more, which can be flexibly set.

[0042] In some embodiments, the sight difference threshold includes a first sight difference threshold and a second sight difference threshold, wherein the second sight difference threshold is greater than the first sight difference threshold, and exemplarily, the first sight difference threshold may be determined based on the maximum angle difference between multiple calibration points set in the test phase, such as the first sight difference threshold may be set to range = 20. The second sight difference threshold may be determined based on the effective angle FOV seen by the human eye, such as FOV = 40.

[0043] Step 2: compare the sight difference theoretical value of the training image group with the sight difference threshold value to determine the first weight corresponding to the training image group based on the comparison result. That is, through the comparison result between the sight difference theoretical value and the sight difference threshold value, the size of the sight difference theoretical value can be objectively measured, and different first weight determination methods can be set for different comparison results, so as to reasonably determine the first weight.

[0044] In some implementation examples, the step of determining the first weight corresponding to the training image group based on the comparison result may be performed with reference to the following cases 1 to 3:

[0045] In case 1, when the theoretical value of the sight difference of the training image group is not higher than the first sight difference threshold, the first weight corresponding to the training image group is determined to be a preset first value. For example, the first value may be 1. That is, when the theoretical value of the sight difference of the training image group is not higher than the first sight difference threshold, it means that the theoretical value of the sight difference of the training image group is small and has a greater impact on the model, so the second weight weight=1 can be directly set. It should be noted that 1 is only an example, and other values ​​can be set specifically, which is not limited here.

[0046] Case 2: When the theoretical value of the sight difference of the training image group is between the first sight difference threshold and the second sight difference threshold, the second weight corresponding to the training image group is determined according to a preset algorithm; wherein the second weight is between the preset second value and the first value, the second value is smaller than the first value, and the second weight is negatively correlated with the theoretical value of the sight difference.

[0047] Exemplarily, the preset algorithm includes a linear algorithm. When the line of sight difference theoretical value is between the first line of sight difference threshold and the second line of sight difference threshold, the smaller the line of sight difference theoretical value is, the larger the second weight is. The relationship between the two can be represented by a linear function. The embodiment of the present disclosure does not limit the linear function. For example, based on the first line of sight difference threshold range and the second line of sight difference threshold FOV, the second weight weight=(1.2FOV-Δangel) / (1.2FOV-range) can be set. Among them, Δangel is used to represent the line of sight difference theoretical value.

[0048] Case 3: When the theoretical value of the sight difference of the training image group is not lower than the second sight difference threshold, the first weight corresponding to the training image group is determined to be the second value. For example, the second value may be 0.2. That is, when the theoretical value of the sight difference of the training image group is not lower than the second sight difference threshold, it means that the theoretical value of the sight difference of the training image group is large and has little impact on the model, so the second weight weight=0.2 can be directly set. It should be noted that 0.2 is only an example, and other values ​​can be set, which is not limited here.

[0049] Through the above method, the corresponding first weight can be reasonably determined based on the magnitude of the sight difference theoretical value.

[0050] On the basis of determining the first weights corresponding to the plurality of training image groups, the embodiment of the present disclosure further provides a specific implementation method for weighting the losses between the theoretical sight difference values ​​and the estimated sight difference values ​​corresponding to the plurality of training image groups according to the first weights corresponding to the plurality of training image groups, which can be performed with reference to the following steps A to C:

[0051] Step A: determining a second weight corresponding to the training image group according to the target angle interval corresponding to the theoretical sight line directions of the two eye sample images in the training image group.

[0052] In the disclosed embodiment, not only the influence of the angle difference between two eye sample images on model training is considered, but also the influence of the angle interval to which each eye sample image belongs on model training is considered. Specifically, the inventor has found through research that the currently collected eye sample images still have the problem of angle imbalance, and there are certain differences in the amount of data of eye sample images at different angles. For example, among the multiple collected eye sample images, the number of eye sample images with small angle intervals is usually large, while the number of eye sample images with large angle intervals is usually small. In practical applications, it can be used Figure 1 The angle panel diagram shown is used to mark the angles corresponding to each eye sample image. Through the marking results, it can be learned that although the angle distribution of the collected eye sample images covers a wide range, such as mainly concentrated within 0 to 35 degrees, the number of eye sample images corresponding to the small angle interval is very large, and the number of eye sample images corresponding to the large angle interval is relatively small. Therefore, the imbalance in the number of eye sample images at different angles will affect the model accuracy to a certain extent, and the estimation accuracy of the trained model for eye sample images at large angles is relatively poor. For this reason, the embodiment of the present disclosure will additionally count the angle interval (i.e., the target angle interval) to which each eye sample image belongs, so as to further determine the second weight corresponding to the training image group.

[0053] For ease of understanding, the present disclosure provides a method for determining a target angle interval corresponding to an eye sample image:

[0054] For example, assuming that the theoretical sight direction of the eye sample image is annotated with the yaw angle (yaw) and pitch angle (pitch) of the eyeball, the target angle angle corresponding to the eye sample image can be calculated based on this, which can be obtained by referring to the following formula:

[0055] angle=int(arctan(np.sqrt(tan(yaw)*tan(yaw)+tan(pitch)*tan(pitch))*180 / np.pi+0.5)where np.sqrt represents square root operation, and np.pi represents pi π. The above formula is only an example. Specifically, it is based on the Pythagorean theorem in three-dimensional space. The target angle angle is calculated based on the known yaw angle yaw and pitch angle pitch of the eyeball. The target angle can be used to represent the angle between the line of sight when the eye is staring at the fixation point and the line of sight when the eye is staring at the fixation point horizontally. The above is only an example. The calculated angle is rounded. In practical applications, the calculated angle may not be rounded, and it may be directly represented in decimal form, which is not limited here.

[0056] In addition, the embodiments of the present disclosure perform statistics based on the angle distribution of a large number of acquired eye sample images, and multiple angle intervals can be set, such as setting the angle interval to 0-1 degree, 1-2 degrees, 3-4 degrees, ..., 30-31 degrees, etc., or it can also be set to 0-0.5 degrees, 0.5-1.5 degrees, 1.5-2.5 degrees, ..., 30.5-31.5 degrees, etc., or it can be set to 0-5 degrees, 5-10 degrees, ..., 30-35 degrees, etc., and can be flexibly set according to needs. Different angle intervals can be equally spaced or unequally spaced, and there is no limitation here. The angle interval in which the target angle corresponding to each eye sample image obtained in the above manner is the target angle interval corresponding to the eye sample image.

[0057] In some specific implementations, the following steps may be performed with reference to steps A1 to A2:

[0058] Step A1, determining the image weights corresponding to the two eye sample images in the training image group according to the target angle intervals corresponding to the theoretical sight lines of the two eye sample images in the training image group.

[0059] As mentioned above, the embodiments of the present disclosure fully take into account the current problem of an imbalance in the number of eye sample images at different angles, which will also affect the accuracy of the model to a certain extent. To this end, the embodiments of the present disclosure can further obtain the target angle interval corresponding to the eye sample image, and then set the corresponding image weight based on the target angle interval, so as to improve the problem of an imbalance in the number of eye sample images at different angles during the model training process based on the way of setting image weights. Exemplarily, the embodiments of the present disclosure further provide a method for determining image weights, which can be performed with reference to the following steps A1.1 to A1.3:

[0060] Step A1.1, obtaining the number of eye sample images corresponding to each of the preset multiple angle intervals. It can be understood that each eye sample image is marked with a theoretical sight line direction, and the corresponding angle interval can be determined according to the above method. At this time, the number of eye sample images corresponding to each angle interval can be counted, and the number of eye sample images corresponding to the angle interval i can be Xi.

[0061] Step A1.2, determining the weights corresponding to the target angle intervals corresponding to the theoretical sight lines of the two eye sample images in the training image group according to the number of eye sample images corresponding to the preset multiple angle intervals.

[0062] According to the number of eye sample images corresponding to each of the multiple angle intervals obtained by statistics, it is possible to better know whether the number of eye sample images corresponding to the target angle interval to which the eye sample images in the training image group belong is higher or lower, thereby reasonably determining the corresponding image weights.

[0063] In some specific implementation examples, step A1.2 can be performed with reference to the following steps A1.2.1 to A1.2.2:

[0064] Step A1.2.1, determine the maximum number from the number of eye sample images corresponding to each of the preset multiple angle intervals, and determine the number of eye sample images corresponding to the target angle interval corresponding to the theoretical sight line direction of each of the two eye sample images in the training image group. The above maximum number is the maximum value of the number of eye sample images corresponding to each of the multiple angle intervals. By obtaining the maximum number and the number of eye sample images corresponding to the target angle interval corresponding to the eye sample image, the number of images in the target angle interval can be reasonably and objectively measured based on the maximum number.

[0065] Step A1.2.2, determine the weight corresponding to the target angle interval according to the maximum number, the number of eye sample images corresponding to the target angle interval, and the preset maximum image weight threshold. Specifically, the above step A1.2.2 can be performed with reference to the following steps 1) to 3):

[0066] Step 1) Obtain the ratio between the maximum number and the number of eye sample images corresponding to the target angle interval. Assuming that the maximum number is XB, and the number of eye sample images corresponding to the i-th angle interval is Xi, then the ratio = XB / Xi. The smaller the ratio, the larger the number of eye sample images corresponding to the angle interval. The larger the ratio, the smaller the number of eye sample images corresponding to the angle interval.

[0067] Step 2), when the ratio is not lower than the maximum image weight threshold, the weight corresponding to the target angle interval is determined as the maximum image weight threshold.

[0068] The disclosed embodiment limits the maximum image weight threshold to prevent the eye sample images in a certain angle interval from having too large a weight, which will have too great an impact on model training, and will weaken the impact of the eye sample images in other angle intervals on model training, resulting in low accuracy of the model obtained by final training. When the ratio is not lower than the maximum image weight threshold, it means that the ratio is high and the number of eye sample images corresponding to the angle interval is small. At this time, the weight corresponding to the target angle interval can be directly set to the maximum image weight threshold. By setting the maximum image weight threshold, the influence of the eye sample images corresponding to the angle interval with a small number of images on model training can be effectively improved.

[0069] Step 3), when the ratio is lower than the maximum image weight threshold, the weight corresponding to the target angle interval is determined to be the ratio. When the ratio is lower than the maximum image weight threshold, it means that the ratio is low, and the number of eye sample images corresponding to the angle interval is large. At this time, the ratio can be directly used as the weight corresponding to the target angle interval, and the number of images corresponding to the angle interval is negatively correlated with the corresponding weight. The larger the number of images, the lower the corresponding weight. By increasing the weight of the angle interval with a small number of images, the impact of the imbalance of the number of images on the model training can be effectively improved, which helps to further improve the prediction accuracy of the model for a small number of large-angle eye images.

[0070] Step A1.3: taking the weights corresponding to the target angle intervals corresponding to the theoretical sight lines of the two eye sample images in the training image group as the image weights corresponding to the two eye sample images in the training image group.

[0071] In summary, through the above step A1, the image weights corresponding to the eye sample images in the training image group can be determined reasonably and objectively.

[0072] Step A2: obtaining a second weight corresponding to the training image group according to the image weights corresponding to the two eye sample images in the training image group.

[0073] For each training image group, based on the known image weights of the two eye sample images in the training image group, the image weights corresponding to the two eye sample images in the training image group can be averaged, and the average processing result can be used as the second weight corresponding to the training image group.

[0074] Step B: obtaining comprehensive weights corresponding to the plurality of training image groups based on the first weights and the second weights corresponding to the plurality of training image groups.

[0075] It can be understood that the first weight is set based on the level of the sight difference between the two eye sample images in the training image group, and is used to alleviate the problem of mismatch between the processing methods in the model training stage and the model testing or reasoning stage. In the model training stage, it can be fully considered that the eye sample images with smaller sight difference have a higher degree of influence on the accuracy of the model prediction results. On this basis, the influence of the sight difference on the model training is adjusted by setting the first weight. The second weight is set based on the angle level of the eye sample images in the training image group, and is used to alleviate the imbalance problem of the number of eye sample images at different angles. On this basis, the influence of the number of eye sample images at different angles on the model training is adjusted by setting the second weight.

[0076] The disclosed embodiments do not limit the method for determining the comprehensive weight based on the first weight and the second weight. It may be a weighted method or a multiplication method. In some specific implementation examples, the comprehensive weight corresponding to each of the multiple training image groups may be obtained based on the product of the first weight and the second weight corresponding to each of the multiple training image groups.

[0077] Step C: performing weighted processing on the loss between the line of sight difference theoretical value and the line of sight difference estimated value corresponding to each of the plurality of training image groups according to the comprehensive weights corresponding to each of the plurality of training image groups.

[0078] By taking into account the comprehensive weight obtained by taking into account the first weight and the second weight, the model training can be performed more objectively and effectively, and the estimation accuracy of the line of sight difference estimation value between the eye images with a small line of sight difference and the accuracy of the final prediction result for the eye images with a large angle can be effectively improved.

[0079] The present disclosure also provides a line of sight estimation method. Figure 4 The flowchart of a line of sight estimation method provided by an embodiment of the present disclosure is shown in FIG. 1 , which can be executed by a line of sight estimation device, wherein the device can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 4 As shown, the method mainly includes the following steps S402 to S406:

[0080] Step S402, obtain a calibration eye image with a theoretical sight line direction and a target eye image with a sight line direction to be estimated. In practical applications, before collecting the eye image, a prompt message to collect the eye image can be sent to the user, and the eye image is collected after obtaining the user's authorization.

[0081] Step S404, obtaining an estimated value of the line of sight difference between the calibration eye image and the target eye image through a pre-constructed target network model; wherein the target network model is obtained based on the construction method of the aforementioned network model, and the details can be referred to the aforementioned related content, which will not be repeated here.

[0082] Step S406, estimating the sight direction corresponding to the target eye image according to the theoretical sight direction of the calibration eye image and the estimated sight difference between the calibration eye image and the target eye image.

[0083] Since the target network model trained by the embodiment of the present disclosure can output a reliable line of sight difference estimation value, on this basis, the accuracy of line of sight direction estimation for the target image is further guaranteed.

[0084] For example, reference may be made to Figure 5 The schematic diagram of a line of sight estimation process shown in the figure illustrates that a calibration eye image and a target eye image marked with a theoretical line of sight direction are simultaneously input into a target network model, and the target network model can output an estimated line of sight difference between the two. Based on the sum of the estimated line of sight difference and the line of sight direction marked on the calibration eye image, the line of sight direction corresponding to the target eye image can be obtained. Furthermore, the target network model can be a differential model, Figure 5 It is also shown that the target network model includes a feature extraction network and a sight difference prediction network. The feature extraction network is used to extract the feature vectors of the calibration eye image and the target eye image respectively. The sight difference prediction network can analyze and process the feature vectors of the calibration eye image and the target eye image respectively, thereby generating a sight difference estimation value. In specific implementation, the feature vectors of the calibration eye image and the target eye image can be spliced ​​to obtain spliced ​​features, and then the sight difference prediction network is used to analyze and process the spliced ​​features to obtain a sight difference estimation value. The above is only an implementation example of the target network model and should not be regarded as a limitation.

[0085] Through the above method provided by the embodiment of the present disclosure, a more accurate and reliable line of sight direction estimation result can be obtained.

[0086] Corresponding to the construction method of the aforementioned network model, Figure 6 A schematic diagram of a network model construction device provided in an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated in an electronic device, such as Figure 6As shown, the construction device of the network model includes:

[0087] The image group acquisition module 602 is used to acquire multiple training image groups; wherein the training image group includes two eye sample images, and the eye sample images are annotated with theoretical sight lines;

[0088] A first weight determination module 604 is used to obtain a theoretical value of sight difference corresponding to the training sample group according to the angle difference between the theoretical sight directions annotated by two eye sample images in the training image group, and determine a first weight corresponding to the training image group according to the theoretical sight difference value;

[0089] The network model output module 606 is used to obtain the sight difference estimation values ​​respectively outputted by the preset initial network model for the plurality of training image groups; wherein the sight difference estimation value is obtained by estimating the angle difference between the sight lines of two eye sample images in the training image group by the initial network model;

[0090] A weighted loss acquisition module 608 is used to perform weighted processing on the loss between the line of sight difference theoretical value and the line of sight difference estimated value corresponding to each of the plurality of training image groups according to the first weights corresponding to each of the plurality of training image groups, so as to obtain a weighted loss;

[0091] The network model training module 610 is used to train the initial network model based on the weighted loss to construct a target network model based on the trained initial network model; wherein the target network model is used to generate an estimated value of the line of sight difference between two target eye images.

[0092] The above-mentioned device can fully consider the influence of the sight difference between the two eye images on the model output result. During the model training process, the corresponding first weight is determined for the theoretical value of the sight difference corresponding to the training image group. By setting the first weight, the influence of the sight difference on the model is introduced, which helps to make the sight difference estimation result output by the final target network model more accurate.

[0093] In some implementations, for any two of the training image groups, the first weight corresponding to the training image group with a smaller theoretical value of sight difference is not lower than the first weight corresponding to the training image group with a larger theoretical value of sight difference.

[0094] In some embodiments, the first weight determination module 604 is specifically used to: obtain a preset line of sight difference threshold; wherein the number of the line of sight difference thresholds is one or more; compare the line of sight difference theoretical value of the training image group with the line of sight difference threshold to determine the first weight corresponding to the training image group based on the comparison result.

[0095] In some embodiments, the first weight determination module 604 is specifically used to: determine the first weight corresponding to the training image group as a preset first value when the theoretical value of the line of sight difference of the training image group is not higher than the first line of sight difference threshold; determine the second weight corresponding to the training image group according to a preset algorithm when the theoretical value of the line of sight difference of the training image group is between the first line of sight difference threshold and the second line of sight difference threshold; wherein the second weight is between the preset second value and the first value, the second value is less than the first value, and the second weight is negatively correlated with the theoretical value of the line of sight difference; and determine the first weight corresponding to the training image group as the second value when the theoretical value of the line of sight difference of the training image group is not lower than the second line of sight difference threshold.

[0096] In some implementations, the preset algorithm includes a linear algorithm.

[0097] In some embodiments, the weighted loss acquisition module 608 is specifically used to: determine the second weight corresponding to the training image group based on the target angle interval corresponding to the theoretical line of sight directions of the two eye sample images in the training image group; obtain the comprehensive weight corresponding to the multiple training image groups based on the first weights and the second weights corresponding to the multiple training image groups; and perform weighted processing on the loss between the theoretical line of sight difference value and the estimated line of sight difference value corresponding to the multiple training image groups according to the comprehensive weights corresponding to the multiple training image groups.

[0098] In some embodiments, the weighted loss acquisition module 608 is specifically used to: determine the image weight corresponding to each of the two eye sample images in the training image group according to the target angle interval corresponding to the theoretical line of sight direction of each of the two eye sample images in the training image group; and obtain the second weight corresponding to the training image group according to the image weight corresponding to each of the two eye sample images in the training image group.

[0099] In some embodiments, the weighted loss acquisition module 608 is specifically used to: obtain the number of eye sample images corresponding to each of the preset multiple angle intervals; determine the weight corresponding to the target angle interval corresponding to the theoretical line of sight direction of each of the two eye sample images in the training image group according to the number of eye sample images corresponding to each of the preset multiple angle intervals; and use the weight corresponding to the target angle interval corresponding to the theoretical line of sight direction of each of the two eye sample images in the training image group as the image weight corresponding to each of the two eye sample images in the training image group.

[0100] In some embodiments, the weighted loss acquisition module 608 is specifically used to: determine the maximum number from the numbers of eye sample images corresponding to each of the preset multiple angle intervals, and determine the number of eye sample images corresponding to the target angle interval corresponding to the theoretical line of sight direction of each of the two eye sample images in the training image group; determine the weight corresponding to the target angle interval based on the maximum number, the number of eye sample images corresponding to the target angle interval and the preset maximum image weight threshold.

[0101] In some embodiments, the weighted loss acquisition module 608 is specifically used to: obtain the ratio between the maximum number and the number of eye sample images corresponding to the target angle interval; when the ratio is not lower than the maximum image weight threshold, determine the weight corresponding to the target angle interval as the maximum image weight threshold; when the ratio is lower than the maximum image weight threshold, determine the weight corresponding to the target angle interval as the ratio.

[0102] In some implementations, the weighted loss acquisition module 608 is specifically used to: average the image weights corresponding to the two eye sample images in the training image group, and use the average processing result as the second weight corresponding to the training image group.

[0103] In some implementations, the weighted loss acquisition module 608 is specifically used to obtain the comprehensive weight corresponding to each of the multiple training image groups based on the product of the first weight and the second weight corresponding to each of the multiple training image groups.

[0104] The network model construction device provided in the embodiments of the present disclosure can execute the network model construction method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0105] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device embodiment can refer to the corresponding process in the method embodiment, and will not be repeated here.

[0106] Corresponding to the aforementioned line of sight estimation method, Figure 7 FIG. 1 is a schematic diagram of a structure of a sight line estimation device provided by an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated in an electronic device, such as Figure 7 As shown, the sight line estimation device includes:

[0107] An image acquisition module 702 is used to acquire a calibration eye image marked with a theoretical sight line direction and a target eye image whose sight line direction is to be estimated;

[0108] A sight difference estimation module 704 is used to obtain a sight difference estimation value between the calibration eye image and the target eye image through a pre-constructed target network model; wherein the target network model is obtained based on the aforementioned network model construction method;

[0109] The sight line direction estimation module 706 is used to estimate the sight line direction corresponding to the target eye image according to the theoretical sight line direction of the calibration eye image and the estimated sight line difference between the calibration eye image and the target eye image.

[0110] Since the target network model trained by the embodiment of the present disclosure can output a more accurate line of sight difference estimation value, on this basis, the accuracy of line of sight direction estimation for the target image is further guaranteed.

[0111] The gaze direction estimation device provided in the embodiments of the present disclosure can execute the gaze direction estimation method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0112] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device embodiment can refer to the corresponding process in the method embodiment, and will not be repeated here.

[0113] An embodiment of the present disclosure also provides an electronic device, which includes: a processor; a memory for storing instructions executable by the processor; and a processor for reading executable instructions from the memory and executing the instructions to implement the above-mentioned network model construction method or the above-mentioned line of sight estimation method.

[0114] Figure 8 The structure diagram of an electronic device provided by the embodiment of the present disclosure is shown in FIG. Figure 8 As shown, the electronic device 800 includes one or more processors 801 and a memory 802 .

[0115] The processor 801 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 800 to perform desired functions.

[0116] The memory 802 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 801 may run the program instructions to implement the above-mentioned network model construction method or the above-mentioned line of sight estimation method and / or other desired functions provided by the embodiments of the present disclosure described above. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0117] In one example, the electronic device 800 may further include: an input device 803 and an output device 804, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0118] In addition, the input device 803 may also include, for example, a keyboard, a mouse, etc.

[0119] The output device 804 can output various information to the outside, including the determined distance information, direction information, etc. The output device 804 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0120] Of course, to simplify, Figure 8 Only some of the components related to the present disclosure in the electronic device 800 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 800 may also include any other appropriate components.

[0121] In addition to the above-mentioned methods and devices, the embodiments of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the above-mentioned network model construction method or the above-mentioned line of sight estimation method provided by the embodiments of the present disclosure.

[0122] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0123] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the above-mentioned network model construction method or the above-mentioned line of sight estimation method provided in the embodiment of the present disclosure.

[0124] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0125] The embodiments of the present disclosure also provide a computer program product, including a computer program / instruction, which implements the XYZ method in the embodiments of the present disclosure when executed by a processor.

[0126] It is understandable that before using the technical solutions disclosed in the various embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0127] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. For example, when collecting an eye image of the user, a prompt message for collecting the eye image is sent to the user, and the user can be further informed of the purpose of the image collection in the prompt message. Thus, the user can independently choose whether to provide personal information to the electronic device, application, server or storage medium or other software or hardware that performs the operation of the technical solution of the present disclosure according to the prompt message, and perform the operation of the technical solution of the present disclosure after obtaining the user's authorization.

[0128] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0129] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0130] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0131] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for constructing a network model, characterized in that: include: Acquire multiple training image groups; wherein the training image groups include two eye sample images, and the eye sample images are annotated with theoretical sight lines; Obtaining a theoretical value of sight line difference corresponding to the training image group according to an angle difference between theoretical sight line directions annotated by two eye sample images in the training image group, and determining a first weight corresponding to the training image group according to the theoretical sight line difference value; Obtaining sight difference estimation values ​​respectively output by a preset initial network model for the plurality of training image groups; wherein the sight difference estimation value is obtained by estimating the angle difference between the sight lines of two eye sample images in the training image group by the initial network model; According to first weights corresponding to the plurality of training image groups, weighted processing is performed on the losses between the theoretical values ​​of sight difference and the estimated values ​​of sight difference corresponding to the plurality of training image groups, so as to obtain weighted losses; The initial network model is trained based on the weighted loss to construct a target network model based on the trained initial network model; wherein the target network model is used to generate an estimated value of the line of sight difference between two target eye images.

2. The method according to claim 1, characterized in that For any two of the training image groups, the first weight corresponding to the training image group with a smaller theoretical value of sight difference is not lower than the first weight corresponding to the training image group with a larger theoretical value of sight difference.

3. The method according to claim 1, characterized in that The determining the first weight corresponding to the training image group according to the sight difference theoretical value includes: Obtaining a preset sight difference threshold; wherein the number of the sight difference threshold is one or more; The sight difference theoretical value of the training image group is compared with the sight difference threshold value to determine a first weight corresponding to the training image group based on the comparison result.

4. The method according to claim 3, characterized in that Determining the first weight corresponding to the training image group based on the comparison result includes: When the sight difference theoretical value of the training image group is not higher than the first sight difference threshold, determining a first weight corresponding to the training image group to be a preset first value; When the sight difference theoretical value of the training image group is between the first sight difference threshold and the second sight difference threshold, determining a second weight corresponding to the training image group according to a preset algorithm; wherein the second weight is between a preset second value and the first value, the second value is smaller than the first value, and the second weight is negatively correlated with the sight difference theoretical value; When the theoretical value of the sight difference of the training image group is not lower than the second sight difference threshold, the first weight corresponding to the training image group is determined to be the second value.

5. The method according to claim 4, characterized in that The preset algorithm includes a linear algorithm.

6. The method according to any one of claims 1 to 5, characterized in that: The weighted processing of the loss between the line of sight difference theoretical value and the line of sight difference estimated value corresponding to each of the plurality of training image groups according to the first weights corresponding to each of the plurality of training image groups comprises: Determining a second weight corresponding to the training image group according to a target angle interval corresponding to each theoretical sight line direction of two eye sample images in the training image group; Based on the first weights and the second weights corresponding to the plurality of training image groups, respectively, obtaining the comprehensive weights corresponding to the plurality of training image groups; According to the comprehensive weights corresponding to the plurality of training image groups, weighted processing is performed on the loss between the theoretical value of the sight difference and the estimated value of the sight difference corresponding to the plurality of training image groups.

7. The method according to claim 6, characterized in that The determining, according to the target angle interval corresponding to the theoretical sight lines of the two eye sample images in the training image group, the second weight corresponding to the training image group includes: Determining image weights corresponding to the two eye sample images in the training image group according to target angle intervals corresponding to the theoretical sight lines of the two eye sample images in the training image group; According to the image weights corresponding to the two eye sample images in the training image group, a second weight corresponding to the training image group is obtained.

8. The method according to claim 7, characterized in that The determining, according to the target angle interval corresponding to the theoretical sight lines of the two eye sample images in the training image group, the image weights corresponding to the two eye sample images in the training image group respectively include: Obtaining the number of eye sample images corresponding to each of the preset multiple angle intervals; Determining the weights corresponding to the target angle intervals corresponding to the theoretical sight lines of the two eye sample images in the training image group according to the number of eye sample images corresponding to the preset multiple angle intervals; The weights corresponding to the target angle intervals corresponding to the theoretical sight lines of the two eye sample images in the training image group are used as the image weights corresponding to the two eye sample images in the training image group.

9. The method according to claim 8, characterized in that The step of determining the weights of the target angle intervals corresponding to the theoretical sight lines of the two eye sample images in the training image group according to the number of eye sample images corresponding to the preset multiple angle intervals comprises: Determine the maximum number from the numbers of eye sample images corresponding to each of the preset multiple angle intervals, and determine the number of eye sample images corresponding to the target angle interval corresponding to the theoretical sight line direction of each of the two eye sample images in the training image group; The weight corresponding to the target angle interval is determined according to the maximum number, the number of eye sample images corresponding to the target angle interval, and a preset maximum image weight threshold.

10. The method according to claim 9, characterized in that Determining the weight corresponding to the target angle interval according to the maximum number, the number of eye sample images corresponding to the target angle interval, and a preset maximum image weight threshold includes: Obtaining a ratio between the maximum number and the number of eye sample images corresponding to the target angle interval; When the ratio is not lower than the maximum image weight threshold, determining the weight corresponding to the target angle interval as the maximum image weight threshold; When the ratio is lower than the maximum image weight threshold, the weight corresponding to the target angle interval is determined to be the ratio.

11. The method according to claim 7, characterized in that The obtaining, according to the image weights corresponding to the two eye sample images in the training image group, a second weight corresponding to the training image group includes: The image weights corresponding to the two eye sample images in the training image group are averaged, and the averaged result is used as the second weight corresponding to the training image group.

12. The method according to claim 6, characterized in that The obtaining, based on the first weight and the second weight corresponding to each of the plurality of training image groups, the comprehensive weight corresponding to each of the plurality of training image groups comprises: Based on the product results of the first weights and the second weights respectively corresponding to the plurality of training image groups, the comprehensive weights respectively corresponding to the plurality of training image groups are obtained.

13. A line of sight estimation method, characterized in that: include: Acquire a calibration eye image annotated with a theoretical sight direction and a target eye image whose sight direction is to be estimated; Obtaining an estimated value of the sight difference between the calibration eye image and the target eye image through a pre-constructed target network model; wherein the target network model is obtained based on the network model construction method according to any one of claims 1 to 12; The sight direction corresponding to the target eye image is estimated according to the theoretical sight direction of the calibration eye image and the estimated sight difference between the calibration eye image and the target eye image.

14. A device for constructing a network model, characterized in that: include: An image group acquisition module, used to acquire a plurality of training image groups; wherein the training image groups include two eye sample images, and the eye sample images are annotated with theoretical sight lines; a first weight determination module, configured to obtain a theoretical value of sight difference corresponding to the training sample group according to an angle difference between theoretical sight directions annotated by two eye sample images in the training image group, and determine a first weight corresponding to the training image group according to the theoretical sight difference value; A network model output module, used to obtain the sight difference estimation values ​​respectively outputted by the preset initial network model for the plurality of training image groups; wherein the sight difference estimation value is obtained by estimating the angle difference between the sight lines of two eye sample images in the training image group by the initial network model; A weighted loss acquisition module, configured to perform weighted processing on the loss between the line of sight difference theoretical value and the line of sight difference estimated value corresponding to each of the plurality of training image groups according to the first weights corresponding to each of the plurality of training image groups, so as to obtain a weighted loss; A network model training module is used to train the initial network model based on the weighted loss to construct a target network model based on the trained initial network model; wherein the target network model is used to generate an estimated value of the line of sight difference between two target eye images.

15. A sight line estimation device, characterized in that: include: An image acquisition module, used to acquire a calibration eye image marked with a theoretical sight line direction and a target eye image whose sight line direction is to be estimated; A sight difference estimation module, used for obtaining a sight difference estimation value between the calibration eye image and the target eye image through a pre-constructed target network model; wherein the target network model is obtained based on the network model construction method according to any one of claims 1 to 12; A sight line direction estimation module is used to estimate the sight line direction corresponding to the target eye image according to the theoretical sight line direction of the calibration eye image and the estimated sight line difference between the calibration eye image and the target eye image.

16. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the network model construction method described in any one of claims 1 to 12 or the line of sight estimation method described in claim 13.

17. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the network model construction method described in any one of claims 1 to 12 or the line of sight estimation method described in claim 13.

Images