Network model construction method, gaze estimation method, apparatus, device, and medium

Abstract

Embodiments of the present disclosure relate to a network model construction method, a gaze estimation method, an apparatus, a device, and a medium. The network model construction method comprises: acquiring a plurality of training image groups; on the basis of an angular difference between theoretical gaze directions marked on two eye sample images among the training image groups, obtaining gaze difference theoretical values corresponding to the training image groups, and on the basis of the gaze difference theoretical values, determining first weights corresponding to the training image groups; acquiring gaze difference estimated values respectively output by a preset initial network model for the plurality of training image groups; on the basis of the first weights respectively corresponding to the plurality of training image groups, performing weighted processing on losses between the gaze difference theoretical values and the gaze difference estimated values respectively corresponding to the plurality of training image groups to obtain weighted losses; and training the initial network model on the basis of the weighted losses, so as to construct a target network model on the basis of the trained initial network model.

Description

Network model construction method, sight line estimation method, device, equipment and medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application No. 202310899892.2, filed on July 20, 2023, entitled “Network model construction method, line of sight estimation method, device, equipment and medium”. The entire contents of that application are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of computer technology, and in particular to a network model construction method, a line of sight estimation method, an apparatus, a device, and a medium. Background Art

[0004] In many scenarios such as gaming, medical care, and intelligent control, it is necessary to identify the direction of the human eye's gaze so that corresponding strategies can be adopted based on the estimated results of the human eye's gaze direction.

[0005] Summary of the Invention

[0006] The present disclosure provides a network model construction method, a line of sight estimation method, an apparatus, a device, and a medium.

[0007] In a first aspect, an embodiment of the present disclosure provides a method for constructing a network model, comprising: obtaining multiple training image groups; wherein the training image group includes two eye sample images, and the eye sample images are annotated with theoretical sight lines; obtaining a sight line difference theoretical value corresponding to the training sample group based on the angular difference between the theoretical sight lines annotated on the two eye sample images in the training image group, and determining a first weight corresponding to the training image group based on the sight line difference theoretical value; obtaining sight line difference estimation values ​​output by a preset initial network model for the multiple training image groups respectively; wherein the sight line difference estimation value is obtained by estimating the angular difference between the sight line directions of the two eye sample images in the training image group by the initial network model; performing weighted processing on the loss between the sight line difference theoretical value and the sight line difference estimation value corresponding to each of the multiple training image groups based on the first weight corresponding to each of the multiple training image groups to obtain a weighted loss; training the initial network model based on the weighted loss to construct a target network model based on the trained initial network model; wherein the target network model is used to generate a sight line difference estimation value between two target eye images.

[0008] In a second aspect, an embodiment of the present disclosure provides a line of sight estimation method, comprising: obtaining a calibration eye image marked with a theoretical line of sight direction and a target eye image whose line of sight direction is to be estimated; obtaining an estimated value of the line of sight difference between the calibration eye image and the target eye image through a pre-constructed target network model; wherein the target network model is obtained based on the network model construction method provided in the first aspect; and estimating the line of sight direction corresponding to the target eye image according to the theoretical line of sight direction of the calibration eye image and the estimated value of the line of sight difference between the calibration eye image and the target eye image.

[0009] In a third aspect, an embodiment of the present disclosure provides a device for constructing a network model, comprising: an image group acquisition module for acquiring a plurality of training image groups; wherein the training image group comprises two eye sample images, and the eye sample images are annotated with theoretical sight lines; a first weight determination module for obtaining a sight line difference theoretical value corresponding to the training sample group according to the angular difference between the theoretical sight line directions annotated by the two eye sample images in the training image group, and determining a first weight corresponding to the training image group according to the sight line difference theoretical value; a network model output module for obtaining sight line difference estimates outputted by a preset initial network model for the plurality of training image groups respectively. value; wherein, the line of sight difference estimation value is obtained by the initial network model estimating the angular difference between the line of sight directions of the two eye sample images in the training image group; a weighted loss acquisition module, used to weight the loss between the line of sight difference theoretical value and the line of sight difference estimation value corresponding to each of the multiple training image groups according to the first weights corresponding to each of the multiple training image groups, to obtain a weighted loss; a network model training module, used to train the initial network model based on the weighted loss, so as to construct a target network model based on the trained initial network model; wherein, the target network model is used to generate the line of sight difference estimation value between the two target eye images.

[0010] In a fourth aspect, an embodiment of the present disclosure provides a line of sight estimation device, comprising: an image acquisition module for acquiring a calibration eye image marked with a theoretical line of sight direction and a target eye image whose line of sight direction is to be estimated; a line of sight difference estimation module for acquiring an estimated value of the line of sight difference between the calibration eye image and the target eye image through a pre-constructed target network model; wherein the target network model is obtained based on the network model construction method provided in the first aspect; and a line of sight direction estimation module for estimating the line of sight direction corresponding to the target eye image based on the theoretical line of sight direction of the calibration eye image and the estimated value of the line of sight difference between the calibration eye image and the target eye image.

[0011] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor for reading the executable instructions from the memory and executing the instructions to implement the network model construction method of the first aspect, or to implement the line of sight estimation method of the second aspect.

[0012] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the network model construction method of the first aspect, or to implement the line of sight estimation method of the second aspect.

[0013] The above-mentioned technical solution provided by the embodiment of the present disclosure can further determine the first weight corresponding to the training image group based on the line of sight difference theoretical value corresponding to the training image group, based on obtaining the line of sight difference estimation values ​​output by the preset initial network model for multiple training image groups respectively, and the line of sight difference theoretical value corresponding to the training image group, so as to perform weighted processing on the loss between the line of sight difference theoretical value and the line of sight difference estimation value corresponding to each of the multiple training image groups according to the first weight corresponding to each of the multiple training image groups.

[0014] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0016] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0017] FIG1 is a schematic diagram of an angle panel provided by an embodiment of the present disclosure;

[0018] FIG2 is a schematic diagram of an angle panel provided by an embodiment of the present disclosure;

[0019] FIG3 is a flow chart of a method for constructing a network model provided by an embodiment of the present disclosure;

[0020] FIG4 is a flow chart of a line of sight estimation method provided by an embodiment of the present disclosure;

[0021] FIG5 is a schematic diagram of a line of sight estimation process provided by an embodiment of the present disclosure;

[0022] FIG6 is a schematic diagram of the structure of a device for constructing a network model provided by an embodiment of the present disclosure;

[0023] FIG7 is a schematic structural diagram of a sight line estimation device provided by an embodiment of the present disclosure;

[0024] FIG8 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features therein can be combined with each other in the absence of conflict.

[0026] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0027] In the field of line of sight estimation, it is usually necessary to obtain multiple eye sample images and annotate the theoretical line of sight directions for the eye sample images, so as to form multiple training image groups through multiple eye sample images and train the network model for estimating the line of sight difference between two eye images. For the convenience of research, the theoretical line of sight direction can be represented by the corresponding angle, and refer to a schematic diagram of an angle panel as shown in Figure 1, which illustrates multiple concentric circles, and different circles correspond to different angles. In some embodiments, the acquired eye sample images can be analyzed by the angle panel shown in Figure 1, such as, the angles corresponding to each collected eye sample image can be annotated in Figure 1 to analyze the angle distribution of the eye sample images; for example, in the model testing stage, multiple calibration points and one test point can be set based on Figure 1, and the model prediction can be performed by combining the test point with each calibration point, and the accuracy of the model prediction results can be measured. For ease of understanding, the following is a detailed explanation:

[0028] In practical applications, multiple calibration points can be set, and the user can be instructed to gaze at a designated calibration point to capture an image of the user's eye as the calibration image. A test point is then determined, and an image of the user's eye while gazing at the test point is captured as the test image. For each calibration image, the network model estimates the line of sight difference between the test eye image and the calibration image, obtaining an estimated line of sight difference. This difference, combined with the theoretical line of sight direction of the calibration image, yields the line of sight estimation result for the test eye image. The line of sight estimation results corresponding to the multiple calibration images are then weighted to produce the model's final line of sight estimation result for the test eye image. For ease of understanding, please refer to the angle panel diagram shown in Figure 2. The black circles are calibration points. For each calibration point, the angle corresponding to the calibration point in Figure 2 is the angle corresponding to the theoretical line of sight of the calibration eye image collected when the user is gazing at the calibration point. The black squares are actual test points. The angle corresponding to the actual test point in Figure 2 is the angle corresponding to the theoretical line of sight of the test eye image collected when the user is gazing at the test point. The gray circles are the predicted result points obtained by each calibration point relative to the test point. The angle corresponding to the predicted result point in Figure 2 is the angle predicted by the network model based on each calibration point and the test point individually (that is, the model-estimated angle corresponding to the test eye image). The black triangles are the comprehensive predicted result points obtained by combining multiple calibration points relative to the test point. The angle corresponding to the comprehensive predicted result point in Figure 2 is the angle obtained by fusion processing such as weighted averaging of multiple model-estimated angles corresponding to the test eye image. For intuitive analysis, the position of each point can be directly observed below. It can be seen intuitively from Figure 2 that the accuracy of the angle prediction results of the calibration points close to the test point relative to the test point is relatively higher (that is, the closer the corresponding gray circle is to the black square). The accuracy of the comprehensive prediction result obtained based on multiple calibration points relative to the test point is likely to depend on the calibration points close to the test point. That is, the degree of proximity between the black triangle and the black square mainly depends on the gray circle closer to the black square.

[0029] However, the inventors have discovered through research that there are significant differences in the application methods of the above-mentioned model testing stage and the model training stage. The main reason is that, as analyzed above, the model testing stage will be affected by the size of the line of sight difference between the two eye images (the closer the calibration point and the test point, the smaller the line of sight difference, and the greater the impact on the line of sight prediction result finally obtained by the model). In the model training stage, the two eye sample images in the training image group input to the model are random, and the impact of the line of sight difference between the eye sample images on the model training is not considered. The degree of influence of training image groups with different line of sight difference sizes on model training is consistent, which leads to low accuracy in estimating the line of sight direction of the eye image using the trained model in the testing stage or the subsequent inference stage.

[0030] To at least improve the above problems, the present disclosure provides a flowchart of a method for constructing a network model, as shown in FIG3 . This method can be performed by a network model construction device, which can be implemented using software and / or hardware and can generally be integrated into an electronic device. As shown in FIG3 , the method mainly includes the following steps S302 to S310:

[0031] Step S302 , obtaining a plurality of training image groups; wherein the training image groups include two eye sample images, and the eye sample images are annotated with theoretical sight lines.

[0032] The eye sample image can be an eye image captured when the target eye is gazing at a preset gaze point. The target eye can be the eye of a target object (such as a human). In some embodiments, multiple gaze points can be pre-set, and an eye image can be captured when the target eye is gazing at each gaze point, thereby obtaining an eye sample image corresponding to each gaze point. In practical applications, the user can wear an electronic device such as VR glasses, smart glasses, an eye tracker, or other electronic device that requires estimating the user's eye gaze direction. The electronic device can sense the user's eye state and thereby determine the gaze direction when the eye is gazing at the gaze point. The gaze direction captured by the electronic device is used as the theoretical gaze direction of the collected eye sample image, thereby serving as a label for the eye sample image. In practical applications, before collecting the user's eye image, a collection prompt message can be sent to the user, and the prompt message can further inform the user of the purpose of the image collection. In other embodiments, the eye image can be obtained through a network or other means, and then the theoretical gaze direction can be annotated for the eye image using an existing model. In practical applications, the theoretical gaze direction can be represented by a spatial angle, which can be understood as the angle between the gaze direction when the eye is gazing at the gaze point and the reference direction. In some specific embodiments, the theoretical line of sight direction can be directly represented by the yaw angle (yaw) and pitch angle (pitch) of the eyeball; in other embodiments, the theoretical line of sight direction can also be represented by a target angle calculated based on the yaw angle (yaw) and the pitch angle (pitch), and the target angle can be used to represent the angle between the line of sight direction when the eyes are staring at the gaze point and the line of sight direction when the eyes are staring horizontally at the gaze point. The above are only two exemplary descriptions of the line of sight direction. In practical applications, the characterization method of the line of sight direction can be flexibly set and is not limited here. In order to ensure the reliability of model training, multiple training image groups can be obtained. The embodiment of the present disclosure does not limit the number of training image groups. Generally speaking, the more training image groups there are, the more robust the trained model is.

[0033] Step S304: Based on the angular difference between the theoretical sight lines annotated in the two eye sample images in the training image group, a theoretical sight line difference value corresponding to the training image group is obtained, and a first weight corresponding to the training image group is determined based on the theoretical sight line difference value. Each of the multiple training image groups obtained corresponds to a first weight. The first weights corresponding to different training image groups may be the same or different, depending on the theoretical sight line difference value of the training image group.

[0034] In practical applications, the angle difference between theoretical sight lines can be directly represented by the yaw angle difference and the pitch angle difference, or by the target angle difference determined based on the yaw angle difference and the pitch angle difference, which is not limited here.

[0035] The disclosed embodiments fully consider the impact of the sight difference between training image groups on model training. Compared to related techniques, the disclosed embodiments additionally determine first weights corresponding to the training image groups based on their corresponding sight difference theoretical values, thereby controlling the degree of influence of different training image groups on model training based on the first weights. In some implementation examples, for any two training image groups, the first weight corresponding to the training image group with the smaller sight difference theoretical value is no less than the first weight corresponding to the training image group with the larger sight difference theoretical value. In other words, the training image group with the smaller sight difference theoretical value has a greater impact on model training, allowing the trained model to focus more on the sight difference estimation results for two eye images with smaller sight differences, thereby matching the model application method in the subsequent testing or inference phase.

[0036] Step S306, obtaining the sight difference estimation values ​​output by the preset initial network model for the multiple training image groups respectively; wherein the sight difference estimation value is obtained by the initial network model estimating the angular difference between the sight lines of two eye sample images in the training image group.

[0037] The initial network model may be a neural network model. The embodiments of the present disclosure do not limit the structure of the initial network model. For example, the initial network model may be a differential model, specifically including a feature extraction network and a gaze difference prediction network. The feature extraction network is used to extract feature vectors for each eye sample image in the training image set. The gaze difference prediction network can analyze and process the feature vectors of two eye sample images in the training image set to generate a gaze difference estimate (also known as a gaze difference).

[0038] Step S308 : performing weighted processing on the losses between the theoretical sight difference values ​​and the estimated sight difference values ​​corresponding to the plurality of training image groups according to the first weights corresponding to the plurality of training image groups, to obtain weighted losses.

[0039] In practical applications, each training image group corresponds to a first weight, and the first weight can be directly used to perform weighted averaging on the losses between the theoretical line of sight difference values ​​and the estimated line of sight difference values ​​corresponding to each of the multiple training image groups to obtain a weighted loss. Other weights can also be introduced to obtain a comprehensive weight based on the first weight and other weights, and then the comprehensive weight is used to perform weighted averaging on the losses between the theoretical line of sight difference values ​​and the estimated line of sight difference values ​​corresponding to each of the multiple training image groups to obtain a weighted loss. The specific weights can be flexibly set and are not limited here. For each training image group, the loss between its corresponding theoretical line of sight difference value and the estimated line of sight difference value can be obtained by referring to the following method: based on the difference between the theoretical line of sight difference value and the estimated line of sight difference value corresponding to the training image group, a preset loss function is used to determine the loss function value, and the loss function value is the loss between the theoretical line of sight difference value and the estimated line of sight difference value corresponding to the training image group. The embodiment of the present disclosure does not limit the loss function.

[0040] Step S310: training the initial network model based on the weighted loss to construct a target network model based on the trained initial network model; wherein the target network model is used to generate an estimated value of the line of sight difference between the two target eye images.

[0041] The initial network model is trained based on a weighted loss. The network model parameters are adjusted to minimize the difference, aiming to reduce the weighted loss. Training is terminated when a preset training end condition is reached (such as when the total loss converges to a preset threshold). Because the weighted loss fully incorporates the first weights determined based on the theoretical value of line of sight difference for each training image group, the target network model obtained through model training based on the weighted loss can more accurately estimate line of sight difference and better match the application methods in the model testing or inference phase.

[0042] In summary, the above method can fully consider the influence of the sight line difference between the two eye images on the model output results. During the model training process, the corresponding first weight is determined for the theoretical value of the sight line difference corresponding to the training image group. By setting the first weight, the influence of the sight line difference on the model is introduced, which helps to make the sight line difference estimation result output by the final target network model more accurate.

[0043] To ensure the reliability of the first weight, the embodiment of the present disclosure provides an implementation method for determining the first weight corresponding to the training image group based on the sight difference theoretical value, which can be performed with reference to the following steps 1 and 2:

[0044] Step 1: Obtain a preset sight difference threshold value; wherein, the number of the sight difference threshold values ​​is one or more, which can be flexibly set.

[0045] In some embodiments, the sight difference threshold includes a first sight difference threshold and a second sight difference threshold, wherein the second sight difference threshold is greater than the first sight difference threshold. For example, the first sight difference threshold can be determined based on the maximum angular difference between multiple calibration points set during the testing phase, such as the first sight difference threshold can be set to range = 20. The second sight difference threshold can be determined based on the effective angular field of view (FOV) of the human eye, such as FOV = 40.

[0046] Step 2 compares the theoretical sight-line difference value of the training image group with the sight-line difference threshold, and determines a first weight corresponding to the training image group based on the comparison result. Specifically, the comparison between the theoretical sight-line difference value and the sight-line difference threshold allows for an objective measure of the theoretical sight-line difference value. Different first weight determination methods can be used based on different comparison results, thereby rationally determining the first weight.

[0047] In some implementation examples, the step of determining the first weight corresponding to the training image group based on the comparison result may be performed with reference to the following cases 1 to 3:

[0048] In the first scenario, if the theoretical value of the sight line difference for the training image group is not higher than the first sight line difference threshold, the first weight corresponding to the training image group is determined to be a preset first value. For example, the first value can be 1. In other words, if the theoretical value of the sight line difference for the training image group is not higher than the first sight line difference threshold, it indicates that the theoretical value of the sight line difference for the training image group is small and has a greater impact on the model, so the second weight weight = 1 can be directly set. It should be noted that 1 is only an example; other values ​​can be set and are not limited here.

[0049] Case 2: When the theoretical value of the line of sight difference of the training image group is between the first line of sight difference threshold and the second line of sight difference threshold, the second weight corresponding to the training image group is determined according to a preset algorithm; wherein the second weight is between the preset second value and the first value, the second value is smaller than the first value, and the second weight is negatively correlated with the theoretical value of the line of sight difference.

[0050] Exemplarily, the preset algorithm includes a linear algorithm. When the theoretical value of the line of sight difference is between the first line of sight difference threshold and the second line of sight difference threshold, the smaller the theoretical value of the line of sight difference, the larger the second weight. The relationship between the two can be represented by a linear function. The embodiments of the present disclosure do not limit the linear function. For example, based on the setting of the first line of sight difference threshold range and the second line of sight difference threshold FOV, the second weight weight = (1.2FOV-Δangel) / (1.2FOV-range) can be set. Wherein, Δangel is used to represent the theoretical value of the line of sight difference.

[0051] In case three, if the theoretical value of the sight line difference of the training image group is not less than the second sight line difference threshold, the first weight corresponding to the training image group is determined to be the second value. For example, the second value can be 0.2. In other words, if the theoretical value of the sight line difference of the training image group is not less than the second sight line difference threshold, it indicates that the theoretical value of the sight line difference of the training image group is large and has little impact on the model, so the second weight weight = 0.2 can be directly set. It should be noted that 0.2 is only an example; other values ​​can be set and are not limited here.

[0052] Through the above method, the corresponding first weight can be reasonably determined based on the magnitude of the sight difference theoretical value.

[0053] Based on determining the first weights corresponding to the multiple training image groups, the embodiment of the present disclosure further provides a specific implementation method for weighting the losses between the theoretical sight difference values ​​and the estimated sight difference values ​​corresponding to the multiple training image groups according to the first weights corresponding to the multiple training image groups. The implementation can refer to the following steps A to C:

[0054] Step A: determining a second weight corresponding to the training image group according to the target angle interval corresponding to the theoretical sight line directions of the two eye sample images in the training image group.

[0055] In the disclosed embodiments, not only is the impact of the angular difference between two eye sample images on model training considered, but also the impact of the angular range to which each eye sample image belongs. Specifically, the inventors discovered that the currently collected eye sample images still have an angular imbalance. The amount of data for eye sample images at different angles varies. For example, among the multiple collected eye sample images, the number of eye sample images with small angular ranges is generally higher, while the number of eye sample images with large angular ranges is generally lower. In practical applications, the angle panel diagram shown in FIG1 can be used to mark the angles corresponding to each eye sample image. The marking results show that although the angular distribution of the collected eye sample images covers a wide range, such as being mainly concentrated within 0 to 35 degrees, the number of eye sample images corresponding to the small angular range is very large, while the number of eye sample images corresponding to the large angular range is relatively small. Therefore, the imbalance in the number of eye sample images at different angles will affect the model accuracy to a certain extent, and the trained model's estimation accuracy for eye sample images at large angles is also relatively poor. To this end, the disclosed embodiments additionally count the angular range (i.e., the target angular range) to which each eye sample image belongs to further determine the second weight corresponding to the training image group.

[0056] For ease of understanding, the present disclosure provides a method for determining the target angle interval corresponding to the eye sample image:

[0057] For example, assuming that the theoretical gaze direction of the eye sample image is represented by the yaw and pitch angles of the eyeball, the target angle angle corresponding to the eye sample image can be calculated based on this formula: angle = int(arctan(np.sqrt(tan(yaw)*tan(yaw)+tan(pitch)*tan(pitch))*180 / np.pi+0.5)

[0058] Here, np.sqrt represents the square root operation, and np.pi represents pi. The above formula is an example only. It is based on the Pythagorean theorem in three-dimensional space. Given the known yaw and pitch angles of the eyeball, the target angle angle is calculated. This target angle represents the angle between the eye's gaze direction when it is directed toward the fixation point and the eye's gaze direction when it is horizontally directed toward the fixation point. The above is an example only, and the calculated angle is rounded. In practical applications, the calculated angle can be expressed directly as a decimal, without any limitation.

[0059] In addition, the embodiment of the present disclosure is based on the statistics of the angle distribution of a large number of eye sample images obtained, and multiple angle intervals can be set, such as setting the angle interval to 0-1 degree, 1-2 degrees, 3-4 degrees, ..., 30-31 degrees, etc., or it can also be set to 0-0.5 degrees, 0.5-1.5 degrees, 1.5-2.5 degrees, ..., 30.5-31.5 degrees, etc., or it can be set to 0-5 degrees, 5-10 degrees, ..., 30-35 degrees, etc. It can be flexibly set according to needs, and different angle intervals can be equally spaced or unequally spaced, which is not limited here. The angle interval in which the target angle corresponding to each eye sample image obtained according to the above method is the target angle interval corresponding to the eye sample image.

[0060] In some specific implementations, the following steps may be performed with reference to steps A1 and A2:

[0061] Step A1: determining the image weights corresponding to the two eye sample images in the training image group according to the target angle intervals corresponding to the theoretical sight lines of the two eye sample images in the training image group.

[0062] As mentioned above, the embodiment of the present disclosure fully takes into account the current imbalance in the number of eye sample images at different angles, which will also affect the model accuracy to a certain extent. To this end, the embodiment of the present disclosure can further obtain the target angle interval corresponding to the eye sample image, and then set the corresponding image weight based on the target angle interval, so as to improve the imbalance in the number of eye sample images at different angles during the model training process based on the setting of image weights. Exemplarily, the embodiment of the present disclosure further provides a method for determining image weights, which can be performed with reference to the following steps A1.1 to A1.3:

[0063] Step A1.1: Obtain the number of eye sample images corresponding to each of the multiple preset angle intervals. It will be appreciated that each eye sample image is annotated with a theoretical gaze direction, and the corresponding angle interval can be determined using the aforementioned method. The number of eye sample images corresponding to each angle interval can then be counted. The number of eye sample images corresponding to angle interval i can be denoted by Xi.

[0064] Step A1.2: Determine the weights corresponding to the target angle intervals corresponding to the theoretical sight lines of the two eye sample images in the training image group according to the number of eye sample images corresponding to the preset multiple angle intervals.

[0065] Based on the statistically obtained number of eye sample images corresponding to multiple angle intervals, it is possible to better understand whether the number of eye sample images corresponding to the target angle interval to which the eye sample images in the training image group belong is higher or lower, thereby reasonably determining the corresponding image weights.

[0066] In some specific implementation examples, step A1.2 can be performed with reference to the following steps A1.2.1 to A1.2.2:

[0067] Step A1.2.1 determines the maximum number of eye sample images corresponding to each of the multiple preset angle intervals, and determines the number of eye sample images corresponding to the target angle interval corresponding to the theoretical gaze direction of each of the two eye sample images in the training image set. This maximum number is the maximum value among the numbers of eye sample images corresponding to each of the multiple angle intervals. By determining this maximum number and the number of eye sample images corresponding to the target angle interval corresponding to the eye sample images, using the maximum number as a benchmark, a reasonable and objective measurement of the number of images in the target angle interval can be achieved.

[0068] Step A1.2.2: Determine the weight corresponding to the target angle interval based on the maximum number, the number of eye sample images corresponding to the target angle interval, and the preset maximum image weight threshold. Specifically, step A1.2.2 can be performed with reference to steps 1) to 3) below:

[0069] Step 1) Obtain the ratio between the maximum number and the number of eye sample images corresponding to the target angle interval. Assuming the maximum number is XB and the number of eye sample images corresponding to the i-th angle interval is Xi, then the ratio = XB / Xi. A smaller ratio indicates a larger number of eye sample images corresponding to the angle interval. A larger ratio indicates a smaller number of eye sample images corresponding to the angle interval.

[0070] Step 2): When the ratio is not lower than the maximum image weight threshold, the weight corresponding to the target angle interval is determined to be the maximum image weight threshold.

[0071] The disclosed embodiments limit the maximum image weight threshold to prevent the eye sample images in a certain angle interval from having too large a weight, which would have an excessive impact on model training. This would weaken the influence of the eye sample images in other angle intervals on model training, resulting in low accuracy of the final trained model. When the ratio is not lower than the maximum image weight threshold, it means that the ratio is high and the number of eye sample images corresponding to the angle interval is small. In this case, the weight corresponding to the target angle interval can be directly set to the maximum image weight threshold. By setting the maximum image weight threshold, the influence of the eye sample images corresponding to the angle interval with a small number of images on model training can be effectively increased.

[0072] In step 3), when the ratio is lower than the maximum image weight threshold, the weight corresponding to the target angle interval is determined to be the ratio. When the ratio is lower than the maximum image weight threshold, it indicates that the ratio is low and the number of eye sample images corresponding to the angle interval is large. In this case, the ratio can be directly used as the weight corresponding to the target angle interval. In addition, the number of images corresponding to the angle interval is negatively correlated with the corresponding weight. The larger the number of images, the lower the corresponding weight. By increasing the weight of the angle interval with fewer images, the impact of the imbalance in the number of images on model training can be effectively improved, which helps to further improve the model's prediction accuracy for a small number of large-angle eye images.

[0073] Step A1.3: Using the weights corresponding to the target angle intervals corresponding to the theoretical sight lines of the two eye sample images in the training image group as the image weights corresponding to the two eye sample images in the training image group.

[0074] In summary, through the above step A1, the image weights corresponding to the eye sample images in the training image group can be determined reasonably and objectively.

[0075] Step A2: Obtain a second weight corresponding to the training image group according to the image weights corresponding to the two eye sample images in the training image group.

[0076] For each training image group, based on the known image weights of the two eye sample images in the training image group, the image weights corresponding to the two eye sample images in the training image group can be averaged, and the average processing result can be used as the second weight corresponding to the training image group.

[0077] Step B: obtaining comprehensive weights corresponding to the plurality of training image groups based on the first weights and the second weights corresponding to the plurality of training image groups.

[0078] It is understood that the first weight is set based on the level of the sight line difference between the two eye sample images in the training image group, and is used to alleviate the problem of mismatch between the processing methods in the model training stage and the model testing or inference stage. During the model training stage, it can be fully considered that the eye sample images with smaller sight line differences have a greater impact on the accuracy of the model prediction results. On this basis, the impact of the sight line difference on model training is adjusted by setting the first weight. The second weight is set based on the angle level of the eye sample images in the training image group, and is used to alleviate the imbalance in the number of eye sample images at different angles. On this basis, the impact of the number of eye sample images at different angles on model training is adjusted by setting the second weight.

[0079] The embodiments of the present disclosure do not limit the method of determining the comprehensive weight based on the first weight and the second weight. It can be a weighted method or a multiplication method. In some specific implementation examples, the comprehensive weight corresponding to each of the multiple training image groups can be obtained based on the product result of the first weight and the second weight corresponding to each of the multiple training image groups.

[0080] Step C: performing weighted processing on the losses between the theoretical sight difference values ​​and the estimated sight difference values ​​corresponding to the multiple training image groups according to the comprehensive weights corresponding to the multiple training image groups.

[0081] By combining the first and second weights to create a comprehensive weight, model training can be performed more objectively and effectively. This can effectively improve the accuracy of the model's estimated sight distance between eye images with small sight distances, as well as the accuracy of the final prediction results for eye images with larger angles.

[0082] The present disclosure also provides a line of sight estimation method. FIG4 is a flow chart of a line of sight estimation method provided by the present disclosure. The method can be performed by a line of sight estimation device, wherein the device can be implemented using software and / or hardware and can generally be integrated into an electronic device. As shown in FIG4, the method mainly includes the following steps S402 to S406:

[0083] Step S402: Acquire a calibration eye image annotated with the theoretical gaze direction and a target eye image whose gaze direction is to be estimated. In practical applications, before acquiring the eye image, a prompt message may be sent to the user to acquire the eye image, and the eye image is acquired after obtaining the user's authorization.

[0084] In step S404, an estimated value of the line of sight difference between the calibration eye image and the target eye image is obtained through a pre-constructed target network model. The target network model is obtained based on the aforementioned network model construction method. For details, please refer to the aforementioned related content and will not be repeated here.

[0085] Step S406 , estimating the gaze direction corresponding to the target eye image according to the theoretical gaze direction of the calibration eye image and the estimated gaze difference between the calibration eye image and the target eye image.

[0086] Since the target network model trained by the embodiment of the present disclosure can output reliable line of sight difference estimation values, on this basis, the accuracy of line of sight direction estimation for the target image is further guaranteed.

[0087] For example, a gaze estimation process diagram can be shown in FIG5 , which illustrates the simultaneous input of a calibration eye image and a target eye image, both labeled with theoretical gaze directions, into a target network model. The target network model can output an estimated gaze difference between the two. The gaze difference estimated value is then summed with the gaze direction labeled in the calibration eye image to obtain the gaze direction corresponding to the target eye image. Furthermore, the target network model can be a differential model. FIG5 also illustrates that the target network model includes a feature extraction network and a gaze difference prediction network. The feature extraction network is configured to extract feature vectors from the calibration eye image and the target eye image, respectively. The gaze difference prediction network can analyze and process the feature vectors of the calibration eye image and the target eye image to generate a gaze difference estimate. In a specific implementation, the feature vectors of the calibration eye image and the target eye image can be concatenated to obtain concatenated features, which can then be analyzed and processed by the gaze difference prediction network to obtain a gaze difference estimate. The above is merely an example implementation of the target network model and should not be considered limiting.

[0088] By using the above method provided by the embodiment of the present disclosure, a more accurate and reliable line of sight direction estimation result can be obtained.

[0089] Corresponding to the aforementioned method for constructing a network model, FIG6 is a schematic diagram of the structure of a network model construction device provided in an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated into an electronic device. As shown in FIG6 , the network model construction device includes:

[0090] The image group acquisition module 602 is used to acquire multiple training image groups; wherein the training image groups include two eye sample images, and the eye sample images are annotated with theoretical sight directions;

[0091] A first weight determination module 604 is configured to obtain a theoretical sight line difference value corresponding to the training sample group based on an angular difference between theoretical sight line directions annotated on two eye sample images in the training image group, and determine a first weight corresponding to the training image group based on the theoretical sight line difference value;

[0092] The network model output module 606 is configured to obtain sight difference estimation values ​​outputted by a preset initial network model for each of the plurality of training image groups; wherein the sight difference estimation value is an angular difference between the sight lines of two eye sample images in the training image group estimated by the initial network model;

[0093] A weighted loss acquisition module 608 is configured to perform weighted processing on the losses between the theoretical sight difference values ​​and the estimated sight difference values ​​corresponding to each of the plurality of training image groups according to first weights corresponding to each of the plurality of training image groups to obtain a weighted loss;

[0094] The network model training module 610 is used to train the initial network model based on the weighted loss to construct a target network model based on the trained initial network model; wherein the target network model is used to generate an estimated value of the line of sight difference between the two target eye images.

[0095] The above-mentioned device can fully consider the influence of the sight line difference between the two eye images on the model output results. During the model training process, the corresponding first weight is determined for the theoretical value of the sight line difference corresponding to the training image group. By setting the first weight, the influence of the sight line difference on the model is introduced, which helps to make the sight line difference estimation result output by the final target network model more accurate.

[0096] In some embodiments, for any two training image groups, the first weight corresponding to the training image group with a smaller theoretical value of sight line difference is not lower than the first weight corresponding to the training image group with a larger theoretical value of sight line difference.

[0097] In some embodiments, the first weight determination module 604 is specifically used to: obtain a preset line of sight difference threshold; wherein the number of the line of sight difference thresholds is one or more; compare the line of sight difference theoretical value of the training image group with the line of sight difference threshold to determine the first weight corresponding to the training image group based on the comparison result.

[0098] In some embodiments, the first weight determination module 604 is specifically used to: determine the first weight corresponding to the training image group as a preset first value when the theoretical value of the line of sight difference of the training image group is not higher than the first line of sight difference threshold; determine the second weight corresponding to the training image group according to a preset algorithm when the theoretical value of the line of sight difference of the training image group is between the first line of sight difference threshold and the second line of sight difference threshold; wherein the second weight is between the preset second value and the first value, the second value is less than the first value, and the second weight is negatively correlated with the theoretical value of the line of sight difference; and determine the first weight corresponding to the training image group as the second value when the theoretical value of the line of sight difference of the training image group is not lower than the second line of sight difference threshold.

[0099] In some embodiments, the preset algorithm includes a linear algorithm.

[0100] In some embodiments, the weighted loss acquisition module 608 is specifically used to: determine the second weight corresponding to the training image group based on the target angle range corresponding to the theoretical line of sight directions of the two eye sample images in the training image group; obtain the comprehensive weight corresponding to the multiple training image groups based on the first weights and second weights corresponding to the multiple training image groups; and perform weighted processing on the loss between the theoretical line of sight difference value and the estimated line of sight difference value corresponding to the multiple training image groups according to the comprehensive weights corresponding to the multiple training image groups.

[0101] In some embodiments, the weighted loss acquisition module 608 is specifically used to: determine the image weight corresponding to each of the two eye sample images in the training image group based on the target angle interval corresponding to the theoretical line of sight direction of each of the two eye sample images in the training image group; and obtain the second weight corresponding to the training image group based on the image weight corresponding to each of the two eye sample images in the training image group.

[0102] In some embodiments, the weighted loss acquisition module 608 is specifically used to: obtain the number of eye sample images corresponding to each of the preset multiple angle intervals; determine the weight corresponding to the target angle interval corresponding to the theoretical line of sight direction of each of the two eye sample images in the training image group based on the number of eye sample images corresponding to each of the preset multiple angle intervals; and use the weight corresponding to the target angle interval corresponding to the theoretical line of sight direction of each of the two eye sample images in the training image group as the image weight corresponding to each of the two eye sample images in the training image group.

[0103] In some embodiments, the weighted loss acquisition module 608 is specifically used to: determine the maximum number from the number of eye sample images corresponding to each of the preset multiple angle intervals, and determine the number of eye sample images corresponding to the target angle interval corresponding to the theoretical line of sight direction of each of the two eye sample images in the training image group; determine the weight corresponding to the target angle interval based on the maximum number, the number of eye sample images corresponding to the target angle interval and the preset maximum image weight threshold.

[0104] In some embodiments, the weighted loss acquisition module 608 is specifically used to: obtain the ratio between the maximum number and the number of eye sample images corresponding to the target angle interval; when the ratio is not lower than the maximum image weight threshold, determine the weight corresponding to the target angle interval as the maximum image weight threshold; when the ratio is lower than the maximum image weight threshold, determine the weight corresponding to the target angle interval as the ratio.

[0105] In some embodiments, the weighted loss acquisition module 608 is specifically used to: average the image weights corresponding to the two eye sample images in the training image group, and use the average processing result as the second weight corresponding to the training image group.

[0106] In some embodiments, the weighted loss acquisition module 608 is specifically configured to obtain a comprehensive weight corresponding to each of the plurality of training image groups based on a product of a first weight and a second weight corresponding to each of the plurality of training image groups.

[0107] The network model construction device provided in the embodiments of the present disclosure can execute the network model construction method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0108] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device embodiment can refer to the corresponding process in the method embodiment, and will not be repeated here.

[0109] Corresponding to the aforementioned line of sight estimation method, FIG7 is a schematic structural diagram of a line of sight estimation device provided in an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated into an electronic device. As shown in FIG7 , the line of sight estimation device includes:

[0110] An image acquisition module 702 is configured to acquire a calibration eye image annotated with a theoretical gaze direction and a target eye image whose gaze direction is to be estimated;

[0111] a sight difference estimation module 704 for obtaining an estimated sight difference between the calibration eye image and the target eye image using a pre-built target network model; wherein the target network model is obtained based on the aforementioned network model construction method;

[0112] The gaze direction estimation module 706 is configured to estimate the gaze direction corresponding to the target eye image based on the theoretical gaze direction of the calibration eye image and the estimated gaze difference between the calibration eye image and the target eye image.

[0113] Since the target network model trained by the embodiment of the present disclosure can output a more accurate line of sight difference estimation value, on this basis, the accuracy of line of sight direction estimation for the target image is further guaranteed.

[0114] The gaze direction estimation device provided in the embodiments of the present disclosure can execute the gaze direction estimation method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0115] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device embodiment can refer to the corresponding process in the method embodiment, and will not be repeated here.

[0116] An embodiment of the present disclosure also provides an electronic device, which includes: a processor; a memory for storing processor-executable instructions; and a processor for reading executable instructions from the memory and executing the instructions to implement the above-mentioned network model construction method or the above-mentioned line of sight estimation method.

[0117] FIG8 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. As shown in FIG8 , the electronic device 800 includes one or more processors 801 and a memory 802 .

[0118] The processor 801 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 800 to perform desired functions.

[0119] The memory 802 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may, for example, include read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 801 may execute the program instructions to implement the above-mentioned network model construction method or the above-mentioned line of sight estimation method provided by the embodiments of the present disclosure as described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0120] In one example, the electronic device 800 may further include an input device 803 and an output device 804 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0121] In addition, the input device 803 may also include, for example, a keyboard, a mouse, and the like.

[0122] The output device 804 can output various information to the outside, including determined distance information, direction information, etc. The output device 804 can include, for example, a display, a speaker, a printer, a communication network and its connected remote output device, etc.

[0123] Of course, for simplicity, FIG8 shows only some of the components related to the present disclosure in the electronic device 800, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 800 may further include any other appropriate components according to specific application scenarios.

[0124] In addition to the above-mentioned methods and devices, the embodiments of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the above-mentioned network model construction method or the above-mentioned line of sight estimation method provided by the embodiments of the present disclosure.

[0125] The computer program product may be written in any combination of one or more programming languages ​​to implement the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0126] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the above-mentioned network model construction method or the above-mentioned line of sight estimation method provided in an embodiment of the present disclosure.

[0127] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0128] The embodiments of the present disclosure further provide a computer program product, including a computer program / instruction, which implements the XYZ method in the embodiments of the present disclosure when executed by a processor.

[0129] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0130] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information. For example, when capturing an eye image of the user, a prompt message is sent to the user, and the user can be further informed of the purpose of the image capture in the prompt message. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message, and only perform the operations of the disclosed technical solution after obtaining user authorization.

[0131] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0132] It is understandable that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0133] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0134] The foregoing description is intended only to provide specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments described herein, but rather to be construed in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. A method for constructing a network model, comprising: Acquire multiple training image groups; wherein the training image groups include two eye sample images, and the eye sample images are annotated with theoretical sight lines; Obtaining a theoretical value of sight line difference corresponding to the training image group according to an angle difference between theoretical sight line directions annotated by two eye sample images in the training image group, and determining a first weight corresponding to the training image group according to the theoretical sight line difference; Obtaining sight difference estimation values ​​outputted by a preset initial network model for the plurality of training image groups, respectively; wherein the sight difference estimation value is obtained by estimating the angle difference between the sight lines of two eye sample images in the training image group by the initial network model; According to first weights corresponding to the plurality of training image groups, weighted processing is performed on the losses between the theoretical values ​​of sight difference and the estimated values ​​of sight difference corresponding to the plurality of training image groups, so as to obtain weighted losses; The initial network model is trained based on the weighted loss to construct a target network model based on the trained initial network model; wherein the target network model is used to generate an estimated value of the line of sight difference between two target eye images.

2. The method according to claim 1, wherein for any two of the training image groups, the first weight corresponding to the training image group with a smaller theoretical value of sight difference is not lower than the first weight corresponding to the training image group with a larger theoretical value of sight difference.

3. The method according to claim 1, wherein determining the first weight corresponding to the training image group according to the sight difference theoretical value comprises: Obtaining a preset sight difference threshold; wherein the number of the sight difference threshold is one or more; The sight difference theoretical value of the training image group is compared with the sight difference threshold value to determine a first weight corresponding to the training image group based on the comparison result.

4. The method according to claim 3, wherein determining the first weight corresponding to the training image group based on the comparison result comprises: When the sight difference theoretical value of the training image group is not higher than the first sight difference threshold, determining a first weight corresponding to the training image group to be a preset first value; When the sight difference theoretical value of the training image group is between the first sight difference threshold and the second sight difference threshold, determining a second weight corresponding to the training image group according to a preset algorithm; wherein the second weight is between a preset second value and the first value, the second value is smaller than the first value, and the second weight is negatively correlated with the sight difference theoretical value; When the theoretical value of the sight difference of the training image group is not lower than the second sight difference threshold, the first weight corresponding to the training image group is determined to be the second value. The method according to claim 4 , wherein the preset algorithm comprises a linear algorithm.

6. The method according to any one of claims 1 to 5, wherein the weighted processing of the loss between the line of sight difference theoretical value and the line of sight difference estimated value corresponding to each of the plurality of training image groups according to the first weights corresponding to each of the plurality of training image groups comprises: Determining a second weight corresponding to the training image group according to a target angle interval corresponding to each theoretical sight line direction of two eye sample images in the training image group; Based on the first weights and the second weights corresponding to the plurality of training image groups, respectively, obtaining the comprehensive weights corresponding to the plurality of training image groups; According to the comprehensive weights corresponding to the plurality of training image groups, weighted processing is performed on the loss between the theoretical value of the sight difference and the estimated value of the sight difference corresponding to the plurality of training image groups.

7. The method according to claim 6, wherein determining the second weight corresponding to the training image group according to the target angle interval corresponding to the theoretical sight lines of the two eye sample images in the training image group comprises: Determining image weights corresponding to the two eye sample images in the training image group according to target angle intervals corresponding to the theoretical sight lines of the two eye sample images in the training image group; According to the image weights corresponding to the two eye sample images in the training image group, a second weight corresponding to the training image group is obtained.

8. The method according to claim 7, wherein determining the image weights corresponding to the two eye sample images in the training image group according to the target angle intervals corresponding to the theoretical sight lines of the two eye sample images in the training image group comprises: Obtaining the number of eye sample images corresponding to each of the preset multiple angle intervals; Determining the weights corresponding to the target angle intervals corresponding to the theoretical sight lines of the two eye sample images in the training image group according to the number of eye sample images corresponding to the preset multiple angle intervals; The weights corresponding to the target angle intervals corresponding to the theoretical sight lines of the two eye sample images in the training image group are used as the image weights corresponding to the two eye sample images in the training image group.

9. The method according to claim 8, wherein determining the weights corresponding to the target angle intervals corresponding to the theoretical sight lines of the two eye sample images in the training image group according to the number of eye sample images corresponding to the preset multiple angle intervals comprises: Determine the maximum number from the numbers of eye sample images corresponding to each of the preset multiple angle intervals, and determine the number of eye sample images corresponding to the target angle interval corresponding to the theoretical sight line direction of each of the two eye sample images in the training image group; The weight corresponding to the target angle interval is determined according to the maximum number, the number of eye sample images corresponding to the target angle interval, and a preset maximum image weight threshold.

10. The method according to claim 9, wherein determining the weight corresponding to the target angle interval according to the maximum number, the number of eye sample images corresponding to the target angle interval, and a preset maximum image weight threshold comprises: Obtaining a ratio between the maximum number and the number of eye sample images corresponding to the target angle interval; When the ratio is not lower than the maximum image weight threshold, determining the weight corresponding to the target angle interval as the maximum image weight threshold; When the ratio is lower than the maximum image weight threshold, the weight corresponding to the target angle interval is determined to be the ratio.

11. The method according to claim 7, wherein obtaining the second weight corresponding to the training image group according to the image weights corresponding to the two eye sample images in the training image group comprises: The image weights corresponding to the two eye sample images in the training image group are averaged, and the averaged result is used as the second weight corresponding to the training image group.

12. The method according to claim 6, wherein the obtaining the comprehensive weights corresponding to the plurality of training image groups respectively based on the first weights and the second weights corresponding to the plurality of training image groups respectively comprises: Based on the product result of the first weight and the second weight corresponding to each of the plurality of training image groups, the comprehensive weight corresponding to each of the plurality of training image groups is obtained.

13. A line of sight estimation method, comprising: Acquire a calibration eye image annotated with a theoretical sight direction and a target eye image whose sight direction is to be estimated; Obtaining an estimated value of the sight difference between the calibration eye image and the target eye image through a pre-constructed target network model; wherein the target network model is obtained based on the network model construction method according to any one of claims 1 to 12; The sight direction corresponding to the target eye image is estimated according to the theoretical sight direction of the calibration eye image and the estimated sight difference between the calibration eye image and the target eye image.

14. A device for constructing a network model, comprising: An image group acquisition module, used to acquire a plurality of training image groups; wherein the training image groups include two eye sample images, and the eye sample images are annotated with theoretical sight lines; a first weight determination module, configured to obtain a theoretical value of sight difference corresponding to the training sample group according to an angle difference between theoretical sight directions annotated by two eye sample images in the training image group, and determine a first weight corresponding to the training image group according to the theoretical sight difference value; A network model output module, used to obtain the sight difference estimation values ​​respectively outputted by the preset initial network model for the plurality of training image groups; wherein the sight difference estimation value is obtained by estimating the angle difference between the sight lines of two eye sample images in the training image group by the initial network model; A weighted loss acquisition module, configured to perform weighted processing on the loss between the line of sight difference theoretical value and the line of sight difference estimated value corresponding to each of the plurality of training image groups according to the first weights corresponding to each of the plurality of training image groups, so as to obtain a weighted loss; A network model training module is used to train the initial network model based on the weighted loss to construct a target network model based on the trained initial network model; wherein the target network model is used to generate an estimated value of the line of sight difference between two target eye images.

15. A sight line estimation device, comprising: An image acquisition module, used to acquire a calibration eye image marked with a theoretical sight line direction and a target eye image whose sight line direction is to be estimated; A sight difference estimation module, used for obtaining a sight difference estimation value between the calibration eye image and the target eye image through a pre-constructed target network model; wherein the target network model is obtained based on the network model construction method according to any one of claims 1 to 12; A sight line direction estimation module is used to estimate the sight line direction corresponding to the target eye image according to the theoretical sight line direction of the calibration eye image and the estimated sight line difference between the calibration eye image and the target eye image.

16. An electronic device, wherein the electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the network model construction method described in any one of claims 1 to 12 or the line of sight estimation method described in claim 13.

17. A computer-readable storage medium, wherein the storage medium stores a computer program, wherein the computer program is used to execute the network model construction method described in any one of claims 1 to 12 or the line of sight estimation method described in claim 13.

Images