Network model construction method and apparatus, gaze estimation method and apparatus, and device and medium

Abstract

The embodiments of the present disclosure relate to a network model construction method and apparatus, a gaze estimation method and apparatus, and a device and a medium. The network model construction method comprises: acquiring a plurality of training images, wherein the quantity of the plurality of training images is not less than three; combining the plurality of training images into a plurality of image groups, wherein each training image belongs to two image groups among the plurality of image groups, and each image group includes two different training images; acquiring gaze difference estimation values respectively output by a preset initial network model in respect of the plurality of image groups; and training the initial network model on the basis of the gaze difference estimation values corresponding to the plurality of image groups, so as to construct a target network model on the basis of the trained initial network model.

Description

Network model construction method, sight line estimation method, device, equipment and medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Chinese Patent Application No. 202310937514.9, filed on July 27, 2023. The entire teachings of the above application are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of computer technology, and in particular to a network model construction method, a line of sight estimation method, an apparatus, a device, and a medium. Background Art

[0004] Identifying the direction of a person's gaze is crucial in many scenarios, including gaming, healthcare, and intelligent control. This allows for the implementation of appropriate strategies based on the estimated gaze direction. Typically, this is accomplished through the use of network models. However, existing network model training methods are poor, resulting in poor reliability in gaze estimation using the model's output.

[0005] Summary of the Invention

[0006] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a network model construction method, a line of sight estimation method, an apparatus, a device and a medium.

[0007] In a first aspect, an embodiment of the present disclosure provides a method for constructing a network model, comprising: obtaining a plurality of training images; wherein the number of the plurality of training images is no less than three, and the training images are eye images taken when the target eye is gazing at a preset gaze point, and different training images correspond to different gaze points; combining the plurality of training images into a plurality of image groups; wherein each of the training images belongs to two image groups of the plurality of image groups, and each of the image groups contains two different training images; obtaining sight difference estimation values ​​output by a preset initial network model for the plurality of image groups; wherein the sight difference estimation value is an estimation value of the angular difference between the respective sight directions of the two training images in the image group; according to the sight difference estimation values ​​corresponding to the plurality of image groups, training the initial network model to construct a target network model based on the trained initial network model; wherein the target network model is used to generate a sight difference estimation value between two eye images.

[0008] In a second aspect, an embodiment of the present disclosure provides a line of sight estimation method, comprising: obtaining a calibration image carrying line of sight direction information and a target image whose line of sight direction is to be estimated; wherein the calibration image and the target image are both eye images; obtaining an estimated value of the line of sight difference between the calibration image and the target image through a pre-constructed target network model; wherein the target network model is obtained based on the network model construction method provided in the first aspect; estimating the line of sight direction corresponding to the target image based on the line of sight direction information of the calibration image and the line of sight difference estimated value.

[0009] In a third aspect, an embodiment of the present disclosure provides a network model construction device, comprising: a training image acquisition module for acquiring multiple training images; wherein the number of the multiple training images is no less than three, and the training images are eye images taken when the target eye is looking at a preset gaze point, and different training images correspond to different gaze points; a training image combination module for combining the multiple training images into multiple image groups; wherein each of the training images belongs to two image groups of the multiple image groups, and each of the image groups contains two different training images; a model output module for obtaining line of sight difference estimation values ​​output by a preset initial network model for the multiple image groups; wherein the line of sight difference estimation value is an estimation value of the angular difference between the line of sight directions of each of the two training images in the image group; a model training module for training the initial network model according to the line of sight difference estimation values ​​corresponding to the multiple image groups, so as to construct a target network model based on the trained initial network model; wherein the target network model is used to generate a line of sight difference estimation value between two eye images.

[0010] In a fourth aspect, an embodiment of the present disclosure provides a line of sight estimation device, comprising: an image acquisition module, for acquiring a calibration image carrying line of sight direction information and a target image whose line of sight direction is to be estimated; wherein the calibration image and the target image are both eye images; a line of sight difference estimation module, for acquiring a line of sight difference estimation value between the calibration image and the target image through a pre-constructed target network model; wherein the target network model is obtained based on the network model construction method provided in the first aspect; a line of sight direction estimation module, for estimating the line of sight direction corresponding to the target image based on the line of sight direction information of the calibration image and the line of sight difference estimation value.

[0011] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor for reading the executable instructions from the memory and executing the instructions to implement the network model construction method of the first aspect, or to implement the line of sight estimation method of the second aspect.

[0012] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the network model construction method of the first aspect, or to implement the line of sight estimation method of the second aspect.

[0013] In a seventh aspect, an embodiment of the present disclosure provides a computer program product, comprising a computer program, which, when executed by a processor, implements the network model construction method of the first aspect, or implements the line of sight estimation method of the second aspect.

[0014] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0016] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0017] FIG1 is a flow chart of a method for constructing a network model provided by an embodiment of the present disclosure;

[0018] FIG2 is a schematic diagram of a connection relationship of gaze points provided by an embodiment of the present disclosure;

[0019] FIG3 is a flow chart of a line of sight estimation method provided by an embodiment of the present disclosure;

[0020] FIG4 is a schematic diagram of a line of sight estimation process provided by an embodiment of the present disclosure;

[0021] FIG5 is a schematic diagram of the structure of a device for constructing a network model provided by an embodiment of the present disclosure;

[0022] FIG6 is a schematic structural diagram of a sight line estimation device provided by an embodiment of the present disclosure;

[0023] FIG7 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features therein can be combined with each other in the absence of conflict.

[0025] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0026] Figure 1 is a flow chart of a method for constructing a network model provided by an embodiment of the present disclosure. The method can be executed by a network model construction device, wherein the device can be implemented using software and / or hardware and can generally be integrated into an electronic device. As shown in Figure 1, the method mainly includes the following steps S102 to S108:

[0027] Step S102, obtaining multiple training images; wherein the number of the multiple training images is no less than three, and the training images are eye images captured when the target eye is gazing at a preset gaze point, and different training images correspond to different gaze points. The target eye can be the eye of the target object to be estimated (such as a user). In the embodiment of the present disclosure, at least three gaze points can be pre-set, and eye images are captured when the target eye is gazing at each gaze point, thereby obtaining a training image corresponding to each gaze point. In actual applications, the target user can wear electronic devices such as VR glasses, smart glasses, eye trackers, or other electronic devices that need to estimate the direction of the user's eye line of sight. In order to accurately achieve eye tracking based on the activity state of the user's eyes, the user can be provided with a gaze point in advance, and the user can be asked to stare at the gaze point, and the corresponding eye image can be collected for subsequent analysis and processing.

[0028] Step S104 , combining the multiple training images into multiple image groups; wherein each training image belongs to two image groups among the multiple image groups, and each image group includes two different training images.

[0029] It should be noted that in the embodiment of the present disclosure, it is not based on random combination or arbitrary combination of two training images, but for each training image in the multiple training images, it will be combined with two different training images respectively to obtain multiple image groups. In other words, each training image belongs to two image groups in the multiple image groups. It should be noted that there are many ways to combine multiple training images. The embodiment of the present disclosure does not limit the combination method, as long as the above conditions are met. For example, assuming that there are 4 training images A, B, C, and D, the multiple image groups that can be combined are (AB)(BC)(CD)(DA), the multiple image groups that can be combined are (AC)(CD)(DB)(BA), and the multiple image groups that can be combined are (AD)(DB)(BC)(CA). They will not be listed one by one here.

[0030] Step S106, obtaining the sight difference estimation values ​​output by the preset initial network model for multiple image groups respectively; wherein the sight difference estimation value is an estimation value of the angular difference between the sight directions of each of the two training images in the image group. The sight difference estimation value can be understood as an estimation value of the spatial angle between the two sight directions. In practical applications, an angle scalar of a specified directional dimension can be used to represent the spatial angle between the two sight directions, which is not restricted here.

[0031] The initial network model may be a neural network model. The disclosed embodiments do not restrict the structure of the initial network model. For example, the initial network model may be a differential model, specifically including a feature extraction network and a gaze difference prediction network. The feature extraction network is used to extract feature vectors for each training image in the image group. The gaze difference prediction network may analyze and process feature vectors of two training images in the image group to generate gaze difference estimates (also known as gaze difference values).

[0032] Step S108: training the initial network model according to the sight difference estimation values ​​corresponding to the multiple image groups, so as to construct a target network model based on the trained initial network model; wherein the target network model is used to generate the sight difference estimation value between the two eye images.

[0033] In an embodiment of the present disclosure, each of the multiple training images is combined with two different training images to obtain multiple image groups. That is, each training image belongs to two of the multiple image groups. The combination of the image groups described above can create a certain correlation between the image groups. For example, taking the three image groups (AB), (BC), and (CA) consisting of three training images ABC as an example, (AB) and (BC) are actually related to (CA). Theoretically, the cumulative line of sight difference corresponding to image group (AB) and the line of sight difference corresponding to image group (BC) should be consistent with the line of sight difference corresponding to image group (CA). Therefore, the combination of the multiple image groups described above facilitates analysis of the line of sight difference estimation values ​​corresponding to each image group output by the initial network model, such as analyzing the correlation between different line of sight difference estimation values. For example, model training with the goal of maintaining a stable relative relationship between different line of sight difference estimation values ​​helps the network model establish correlations between different gaze directions and ensures the stability of the network model's output results, effectively ensuring that the network model can stably estimate the line of sight difference between different training images.

[0034] The above-mentioned technical solution provided by the embodiment of the present disclosure can use no less than three training images to combine into multiple image groups, and the combination method of the above-mentioned image groups can enable a certain correlation between the image groups. Therefore, model training is performed based on the line of sight difference estimation values ​​output by the initial network model for multiple image groups. This helps to enable the final target network model to establish correlations between different line of sight directions based on different line of sight difference estimation values, thereby ensuring the stability and accuracy of the line of sight difference estimation results, and the obtained line of sight difference estimation results are more reliable.

[0035] In some embodiments, the plurality of image groups includes at least two first image groups and one second image group; wherein a preset association relationship exists between the at least two first image groups and the second image group, and the preset association relationship includes, but is not limited to: the sum of the sight lines corresponding to the at least two first image groups is consistent with the sight line difference corresponding to the second image group. For ease of understanding, the following description is made in conjunction with FIG2:

[0036] In Figure 2, five gaze points A to E are pre-set, and each gaze point corresponds to a training image. It is assumed that there are at least two first image groups (AB), (BC), (CD), and (DE), and one second image group (AE). The lines between AB, BC, CD, and DE in Figure 2 form a closed loop with the line connecting AE. Accordingly, it can be understood that the sum of the sight lines corresponding to the first image groups (AB), (BC), (CD), and (DE) should theoretically be consistent with the sight line difference corresponding to the second image group (AE).

[0037] In some embodiments, the above step S104, i.e., combining the multiple training images into multiple image groups according to the positions of the gaze points corresponding to the multiple training images, can be performed with reference to the following steps 1 and 2:

[0038] Step 1: Sort the multiple training images to obtain a sorting result. In some implementation examples, the multiple training images can be randomly sorted to flexibly determine the sorting result for each training image. It should be noted that for multiple training images, different sorting methods can be used to obtain multiple sorting results. For example, for three training images, there can be six sorting results: ABC, ACB, BAC, BCA, CAB, and CBA. Only one of the sorting results obtained in step 1 is required.

[0039] Step 2: Combine adjacent training images in the sorting results to obtain at least two first image groups, and combine the first training image and the last training image in the sorting results to obtain a second image group.

[0040] For example, taking three training images, if the sorting result is ACB, then the two first image groups are (AC) and (CB), and the second image group is (AB). If the sorting result is ABC, then the two first image groups are (AB) and (BC), and the second image group is (AC). It is understood that different sorting results will result in different combinations of the first and second image groups.

[0041] Through the above method, multiple image groups that meet the requirements can be quickly determined, so that the line of sight difference estimation values ​​of each image group output by the network model can be analyzed and processed subsequently, thereby achieving the purpose of model training.

[0042] On the basis of the above, the above step S108, i.e., training the initial network model according to the estimated sight difference values ​​corresponding to the multiple image groups, can be performed with reference to the following steps (1) and (2):

[0043] Step (1) is to fuse the sight difference estimation values ​​corresponding to at least two first image groups to obtain a sight difference target value.

[0044] In some specific implementation examples, the estimated sight difference values ​​corresponding to at least two first image groups can be accumulated to obtain a sight difference accumulation value, and the sight difference target value can be further obtained based on the sight difference accumulation value. In specific implementations, the sight difference accumulation value can be directly used as the sight difference target value.

[0045] In step (2), the initial network model is trained based on the sight difference target value and the sight difference estimated value corresponding to the second image group. By using the above method, the sight difference estimated value of the first image group can be synthesized. A certain correlation exists between the synthesized result (sight difference target value) and the sight difference estimated value of the second image group. For example, in theory, the sight difference target value should be equal to the sight difference estimated value of the second image group. Based on this correlation, the initial network model is trained to ensure the reliability of the initial network model.

[0046] In some implementation examples, the initial network model may be trained based on the difference between the sight difference target value and the sight difference estimated value corresponding to the second image group. As mentioned above, in theory, the sight difference target value should be equal to the sight difference estimated value corresponding to the second image group. For ease of understanding, an explanation is given based on the example given in Figure 2: the estimated value of the line of sight difference between the training image A corresponding to the gaze point A and the training image B corresponding to the gaze point B is diff_ab=F(A, B), where F(A, B) represents the processing of the training image A corresponding to the gaze point A and the training image B corresponding to the gaze point B by the network model; accordingly, the estimated value of the line of sight difference between the training image B corresponding to the gaze point B and the training image C corresponding to the gaze point C is diff_bc=F(B, C), the estimated value of the line of sight difference between the training image C corresponding to the gaze point C and the training image D corresponding to the gaze point D is diff_cd=F(C, D), the estimated value of the line of sight difference between the training image D corresponding to the gaze point D and the training image E corresponding to the gaze point E is diff_de=F(D, E), and the target value of the line of sight difference is diff_ab+diff_bc+diff_cd+diff_de. At the same time, the estimated value of the line of sight difference between the training image A corresponding to the gaze point A and the training image E corresponding to the gaze point E, diff_ae=F(A, E), can be obtained. Since A~E form a closed loop, theoretically, diff_ab+diff_bc+diff_cd+diff_de is consistent with diff_ae. However, if the stability of the network model is poor, it will cause differences between diff_ab+diff_bc+diff_cd+diff_de and diff_ae. For this reason, the embodiment of the present disclosure can adjust the parameters of the network model towards the goal of reducing the difference to minimize the difference, thereby ensuring the stability of the trained network model.

[0047] In practical applications, the training images are annotated with the gaze direction information of the target eye, which indicates the actual gaze direction of the target eye. On this basis, when the initial network model is trained based on the difference between the sight difference target value and the sight difference estimated value corresponding to the second image group, the initial network model can be trained based on the difference between the sight difference target value and the sight difference estimated value corresponding to the second image group, and the difference between the sight difference target value and the sight difference true value corresponding to the second image group; wherein the sight difference true value corresponding to the second image group is determined based on the sight direction information annotated on each of the two training images in the second image group. In the above manner, the sight difference target value is constrained by utilizing the difference between the sight difference true values ​​corresponding to the second image group, which can further ensure the accuracy of the trained network model while ensuring the stability of the model. The disclosed embodiment can adjust the parameters of the network model towards the goal of reducing the above two differences to reduce the differences as much as possible, thereby ensuring the stability and accuracy of the trained network model. It should be noted that in practical applications, other differences can be introduced for network model training, such as rotating the same training image and performing other changes, and training the network model by comparing the differences between the network output results corresponding to the training images before and after the changes. This is not limited here.

[0048] In some specific implementation examples, the initial network model may be trained with reference to the following steps a to c:

[0049] Step a: Determine a first loss based on the difference between the target sight difference value and the estimated sight difference value corresponding to the second image group. For example, a first loss function value (i.e., first loss) is determined based on the difference between the target sight difference value and the estimated sight difference value corresponding to the second image group using a preset first loss function. The presently disclosed embodiments do not limit the first loss function; for example, the first loss function may be a Wing loss.

[0050] Exemplarily, the first loss soft_loss=Loss(diff_ab+diff_bc+diff_cd+diff_de, ​​diff_ae).

[0051] Step b: Determine a second loss based on the difference between the target sight difference value and the true sight difference value corresponding to the second image group. For example, a second loss function value (i.e., the second loss) is determined based on the difference between the target sight difference value and the true sight difference value corresponding to the second image group using a preset second loss function. The disclosed embodiments do not limit the second loss function; for example, the second loss function may also be a Wing loss.

[0052] Exemplarily, the second loss hard_loss=Loss(diff_ab+diff_bc+diff_cd+diff_de, ​​ga-ge), where ga is the sight direction annotated by the training image A, and ge is the sight direction annotated by the training image E.

[0053] Step c: training the initial network model based on the weighted result of the first loss and the second loss.

[0054] In practical applications, the weights of the first loss and the second loss can be pre-set, and the first loss and the second loss can be weighted to obtain a weighted result (which can be regarded as the total loss). The initial network model is then trained based on the weighted result of the first loss and the second loss. In practical applications, the weights of the first loss and the second loss can be flexibly set according to needs, and the sum of the weights of the two can be 1.

[0055] For example, assuming that the weight of the first loss is beta and the weight of the second loss is 1-beta, the weighted result of the first loss and the second loss (ie, the total loss) can be expressed as:

[0056] Loss=beta*soft_loss+(1-beta)*hard_loss.

[0057] In some specific implementation examples, the initial network model can be trained according to a plurality of preset stages, and the weights of the first loss and the second loss corresponding to different stages are different. That is, the weights of the first loss and the second loss are adjusted as the training stage of the initial network model changes, that is, the above-mentioned beta value can be changed during training, and the beta value can be flexibly adjusted according to actual needs. For example, assuming that the training process of the initial network model is divided into two stages, the beta value can be set to be smaller in the first stage (such as the early stage of model training), and the beta value can be set to be larger in the second stage (such as the late stage of model training). The above method can prioritize improving the accuracy of the model in the early stage and prioritize improving the stability of the model in the later stage. The above is only an example. In actual applications, three or more stages can be set, and the weights of the first loss and the second loss can be gradually adjusted in multiple stages. This is not limited here.

[0058] On this basis, the parameters of the network model can be adjusted towards the goal of reducing the total loss to minimize the difference as much as possible, until the training is stopped when the preset training end condition is reached (such as the total loss converges to a preset threshold range). Through the above approach, the stability and accuracy of the trained network model can be fully ensured, effectively guaranteeing the reliability of the network model.

[0059] In summary, the embodiments of the present disclosure construct multiple image groups that meet the requirements, and perform model training based on the line of sight estimation values ​​output by the initial network model for multiple image groups. This helps the final target network model to establish associations between different line of sight directions, ensure the stability and accuracy of the line of sight estimation results, and make the obtained line of sight estimation results more reliable.

[0060] Based on the above, the present disclosure also provides a line of sight estimation method. Figure 3 is a flow chart of a line of sight estimation method provided by the present disclosure. The method can be performed by a line of sight estimation device, wherein the device can be implemented using software and / or hardware and can generally be integrated into an electronic device. As shown in Figure 3, the method mainly includes the following steps S302 to S306:

[0061] Step S302: Acquire a calibration image containing gaze direction information and a target image whose gaze direction is to be estimated. Both the calibration image and the target image are eye images. In other words, the calibration image is an eye image with a known gaze direction, while the target image is an eye image with an unknown gaze direction. In practical applications, one or more calibration images may be used. Furthermore, before acquiring an eye image, a prompt may be sent to the user to request that the eye image be acquired. Eye image acquisition is performed only after the user authorizes the acquisition.

[0062] In step S304, an estimated value of the line of sight difference between the calibration image and the target image is obtained through a pre-constructed target network model; wherein, the target network model is obtained based on the construction method of the aforementioned network model, and the details can be referred to the aforementioned related content, which will not be repeated here.

[0063] When there are multiple calibration images, it is necessary to use the target network model to obtain the estimated value of the line of sight difference between each calibration image and the target image.

[0064] Step S306 , estimating the sight direction corresponding to the target image based on the sight direction information of the calibration image and the sight difference estimation value.

[0065] Specifically, the gaze direction indicated by the gaze direction information of the calibration image can be summed with the gaze difference estimate to obtain the gaze direction corresponding to the target image. If there are multiple calibration images, multiple gaze direction estimates corresponding to the target image can be obtained based on the gaze direction information of the multiple calibration images and the gaze difference estimates between the multiple calibration images and the target image. These multiple gaze direction estimates are then weighted, and the gaze direction corresponding to the target image is estimated based on the weighted sum. This approach effectively ensures the accuracy of the gaze direction corresponding to the target image.

[0066] Since the target network model trained by the embodiment of the present disclosure can output reliable line of sight difference estimation values, on this basis, the reliability of line of sight direction estimation for the target image is further guaranteed.

[0067] For example, a line of sight estimation process diagram as shown in FIG4 can be referred to, which illustrates that a calibration image and a target image marked with a line of sight direction are simultaneously input into a target network model. The target network model can output a line of sight difference estimation value. Based on the line of sight difference estimation value and the line of sight direction marked on the calibration image, the line of sight direction corresponding to the target image can be obtained by summing the line of sight difference estimation value and the line of sight direction marked on the calibration image. Furthermore, the target network model can be a differential model. FIG4 also illustrates that the target network model includes a feature extraction network and a line of sight difference prediction network. The feature extraction network is used to extract the feature vectors of the calibration image and the target image respectively. The line of sight difference prediction network can analyze and process the feature vectors of the calibration image and the target image respectively to generate a line of sight difference estimation value. In specific implementation, the feature vectors of the calibration image and the target image can be spliced ​​to obtain spliced ​​features, and then the line of sight difference prediction network can analyze and process the spliced ​​features to obtain a line of sight difference estimation value. The above is only an implementation example of the target network model and should not be regarded as limiting.

[0068] In practical applications, post-processing such as smoothing can be performed based on the output results of the target network model to further optimize the estimation results of the line of sight direction, which is not limited here.

[0069] By using the above method provided by the embodiment of the present disclosure, a more accurate and reliable line of sight direction estimation result can be obtained.

[0070] Corresponding to the aforementioned method for constructing a network model, FIG5 is a schematic diagram of the structure of a network model construction device provided in an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated into an electronic device. As shown in FIG5 , the network model construction device includes:

[0071] A training image acquisition module 502 is configured to acquire a plurality of training images; wherein the number of the plurality of training images is no less than three, and the training images are eye images captured when the target eye is gazing at a preset gaze point, and different training images correspond to different gaze points;

[0072] A training image combination module 504 is configured to combine a plurality of training images into a plurality of image groups; wherein each of the training images belongs to two image groups among the plurality of image groups, and each of the image groups contains two different training images;

[0073] A model output module 506 is configured to obtain line-of-sight difference estimation values ​​outputted by a preset initial network model for each of the plurality of image groups; wherein the line-of-sight difference estimation value is an estimation of the angular difference between the line-of-sight directions of two training images in the image group;

[0074] The model training module 508 is used to train the initial network model according to the line of sight difference estimation values ​​corresponding to the multiple image groups, so as to construct a target network model based on the trained initial network model; wherein the target network model is used to generate the line of sight difference estimation value between the two eye images.

[0075] The above-mentioned technical solution provided by the embodiment of the present disclosure can use no less than three training images to combine into multiple image groups, wherein each training image belongs to two image groups of the multiple image groups, and each image group contains two different training images; the combination method of the above-mentioned image groups can make there be a certain correlation between the image groups. Therefore, model training is performed based on the line of sight estimation values ​​output by the initial network model for the multiple image groups, which helps to make the final target network model establish correlations between different line of sight directions based on different line of sight estimation values, thereby ensuring the stability and accuracy of the line of sight estimation results, and the obtained line of sight estimation results are more reliable.

[0076] In some embodiments, the plurality of image groups include at least two first image groups and one second image group; wherein a preset association relationship exists between the at least two first image groups and the second image group.

[0077] In some embodiments, the training image combination module 504 is specifically used to: sort multiple gaze points to obtain a sorting result; combine the front and back adjacent training images in the sorting result to obtain at least two first image groups, and combine the first training image and the last training image in the sorting result to obtain a second image group.

[0078] In some embodiments, the model training module 508 is specifically used to: fuse the sight difference estimation values ​​corresponding to the at least two first image groups to obtain a sight difference target value; and train the initial network model based on the target value and the sight difference estimation value corresponding to the second image group.

[0079] In some embodiments, the model training module 508 is specifically configured to: accumulate the sight difference estimation values ​​corresponding to the at least two first image groups to obtain a sight difference accumulation value; and obtain a sight difference target value based on the sight difference accumulation value.

[0080] In some embodiments, the model training module 508 is specifically configured to train the initial network model according to a difference between the sight difference target value and the sight difference estimation value corresponding to the second image group.

[0081] In some embodiments, the training image is annotated with the gaze direction information of the target eye; the model training module 508 is specifically used to: train the initial network model based on the difference between the sight difference target value and the sight difference estimated value corresponding to the second image group, and the difference between the sight difference target value and the sight difference true value corresponding to the second image group; wherein the sight difference true value corresponding to the second image group is determined based on the sight direction information annotated on each of the two training images in the second image group.

[0082] In some embodiments, the model training module 508 is specifically used to: determine a first loss based on the difference between the sight difference target value and the sight difference estimated value corresponding to the second image group; determine a second loss based on the difference between the sight difference target value and the sight difference true value corresponding to the second image group; and train the initial network model according to a weighted result of the first loss and the second loss.

[0083] The network model construction device provided in the embodiments of the present disclosure can execute the network model construction method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0084] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device embodiment can refer to the corresponding process in the method embodiment, and will not be repeated here.

[0085] Corresponding to the aforementioned line of sight estimation method, FIG6 is a schematic structural diagram of a line of sight estimation device provided in an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated into an electronic device. As shown in FIG6 , the line of sight estimation device includes:

[0086] An image acquisition module 602 is configured to acquire a calibration image carrying gaze direction information and a target image whose gaze direction is to be estimated; wherein both the calibration image and the target image are eye images;

[0087] a sight line difference estimation module 604 for obtaining an estimated sight line difference between the calibration image and the eye image using a pre-built target network model; wherein the target network model is obtained based on the aforementioned network model construction method;

[0088] The sight line direction estimation module 606 is configured to estimate the sight line direction corresponding to the target image based on the sight line direction information of the calibration image and the sight line difference estimation value.

[0089] Since the target network model trained by the embodiment of the present disclosure can output reliable line of sight difference estimation values, on this basis, the reliability of line of sight direction estimation for the target image is further guaranteed.

[0090] In some embodiments, the gaze direction estimation module 606 is specifically used to: obtain multiple gaze direction estimation values ​​corresponding to the target image based on the gaze direction information of multiple calibration images and the estimated gaze difference values ​​between the multiple calibration images and the target image; perform weighted processing on the multiple gaze direction estimation values, and estimate the gaze direction corresponding to the target image based on the weighted processing result.

[0091] The gaze direction estimation device provided in the embodiments of the present disclosure can execute the gaze direction estimation method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0092] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device embodiment can refer to the corresponding process in the method embodiment, and will not be repeated here.

[0093] An embodiment of the present disclosure also provides an electronic device, which includes: a processor; a memory for storing processor-executable instructions; and a processor for reading executable instructions from the memory and executing the instructions to implement the above-mentioned network model construction method or the above-mentioned line of sight estimation method.

[0094] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device embodiment can refer to the corresponding process in the method embodiment, and will not be repeated here.

[0095] FIG7 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. As shown in FIG7 , the electronic device 700 includes one or more processors 701 and a memory 702 .

[0096] The processor 701 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 700 to perform desired functions.

[0097] The memory 702 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may, for example, include read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 701 may execute the program instructions to implement the above-mentioned network model construction method or the above-mentioned line of sight estimation method provided by the embodiments of the present disclosure as described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0098] In one example, the electronic device 700 may further include an input device 703 and an output device 704 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0099] In addition, the input device 703 may also include, for example, a keyboard, a mouse, and the like.

[0100] The output device 704 can output various information to the outside, including determined distance information, direction information, etc. The output device 704 can include, for example, a display, a speaker, a printer, a communication network and its connected remote output device, etc.

[0101] Of course, for simplicity, FIG7 only shows some of the components related to the present disclosure in the electronic device 700, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 700 may further include any other appropriate components according to specific application scenarios.

[0102] In addition to the above-mentioned methods and devices, the embodiments of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the above-mentioned network model construction method or the above-mentioned line of sight estimation method provided by the embodiments of the present disclosure.

[0103] The computer program product may be written in any combination of one or more programming languages ​​to implement the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0104] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the above-mentioned network model construction method or the above-mentioned line of sight estimation method provided in an embodiment of the present disclosure.

[0105] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0106] The embodiments of the present disclosure also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-mentioned network model construction method or the above-mentioned line of sight estimation direction in the embodiments of the present disclosure.

[0107] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0108] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information. For example, when capturing an eye image of the user, a prompt message is sent to the user, and the user can be further informed of the purpose of the image capture in the prompt message. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message, and only perform the operations of the disclosed technical solution after obtaining user authorization.

[0109] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0110] It is understandable that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0111] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0112] The foregoing description is intended only to provide specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments described herein, but rather to be construed in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. A method for constructing a network model, comprising: Acquire multiple training images; wherein the number of the multiple training images is no less than three, and the training images are eye images taken when the target eye is gazing at a preset gaze point, and different training images correspond to different gaze points; Combining the plurality of training images into a plurality of image groups; wherein each of the training images belongs to two image groups among the plurality of image groups, and each of the image groups contains two different training images; Obtaining the sight difference estimation values ​​respectively output by the preset initial network model for the plurality of image groups; wherein the sight difference estimation value is an estimation value of the angle difference between the sight lines of the two training images in the image group; The initial network model is trained according to the sight difference estimation values ​​corresponding to the multiple image groups to construct a target network model based on the trained initial network model; wherein the target network model is used to generate the sight difference estimation value between the two eye images.

2. The method according to claim 1, wherein: The multiple image groups include at least two first image groups and one second image group; wherein a preset association relationship exists between the at least two first image groups and the second image group.

3. The method according to claim 1 or 2, wherein: The combining the plurality of training images into a plurality of image groups comprises: Sorting the multiple training images to obtain a sorting result; The training images adjacent to each other in the sorting result are respectively combined to obtain at least two first image groups, and the first training image and the last training image in the sorting result are combined to obtain a second image group.

4. The method according to claim 2, wherein: The training of the initial network model according to the sight difference estimation values ​​corresponding to the multiple image groups includes: Fusing the sight difference estimation values ​​corresponding to the at least two first image groups to obtain a sight difference target value; The initial network model is trained according to the sight difference target value and the sight difference estimation value corresponding to the second image group.

5. The method according to claim 4, wherein: The fusing the sight difference estimation values ​​corresponding to the at least two first image groups to obtain a sight difference target value includes: Accumulating the sight difference estimation values ​​corresponding to the at least two first image groups to obtain a sight difference accumulation value; A sight difference target value is obtained based on the sight difference accumulated value.

6. The method according to claim 4, wherein: The training of the initial network model according to the sight difference target value and the sight difference estimation value corresponding to the second image group includes: The initial network model is trained according to the difference between the sight difference target value and the sight difference estimation value corresponding to the second image group.

7. The method according to claim 6, wherein: The training image is annotated with the sight direction information of the target eye; The training of the initial network model according to the difference between the sight difference target value and the sight difference estimation value corresponding to the second image group includes: The initial network model is trained based on the difference between the line of sight difference target value and the line of sight difference estimation value corresponding to the second image group, and the difference between the line of sight difference target value and the line of sight difference true value corresponding to the second image group; wherein the line of sight difference true value corresponding to the second image group is determined based on the line of sight direction information annotated on each of the two training images in the second image group.

8. The method according to claim 7, wherein: The training of the initial network model based on the difference between the sight difference target value and the sight difference estimated value corresponding to the second image group, and the difference between the sight difference target value and the sight difference true value corresponding to the second image group, comprises: determining a first loss based on a difference between the sight difference target value and the sight difference estimation value corresponding to the second image group; determining a second loss based on a difference between the sight difference target value and a true sight difference value corresponding to the second image group; The initial network model is trained according to a weighted result of the first loss and the second loss.

9. A line of sight estimation method, comprising: Acquire a calibration image carrying gaze direction information and a target image whose gaze direction is to be estimated; wherein both the calibration image and the target image are eye images; Obtaining an estimated value of the sight difference between the calibration image and the target image through a pre-constructed target network model; wherein the target network model is obtained based on the method for constructing a network model according to any one of claims 1 to 8; The sight direction corresponding to the target image is estimated according to the sight direction information of the calibration image and the sight difference estimation value.

10. The method according to claim 9, wherein: The number of the calibration images is multiple, and estimating the sight direction corresponding to the target image according to the sight direction information of the calibration images and the sight difference estimation value includes: Obtaining a plurality of estimated values ​​of the sight direction corresponding to the target image according to the sight direction information of the plurality of calibration images and the estimated values ​​of sight differences between the plurality of calibration images and the target image; The multiple sight line direction estimation values ​​are weighted, and the sight line direction corresponding to the target image is estimated based on the weighted processing result.

11. A network model construction device, comprising: A training image acquisition module, used to acquire a plurality of training images; wherein the number of the plurality of training images is not less than three, and the training images are eye images taken when the target eye is gazing at a preset gaze point, and different training images correspond to different gaze points; A training image combination module, used for combining the plurality of training images into a plurality of image groups; wherein each of the training images belongs to two image groups among the plurality of image groups, and each of the image groups contains two different training images; A model output module, used to obtain the sight difference estimation values ​​outputted by the preset initial network model for the plurality of image groups, wherein the sight difference estimation value is an estimation value of the angle difference between the sight lines of the two training images in the image group; A model training module is used to train the initial network model according to the line of sight difference estimation values ​​corresponding to the multiple image groups, so as to construct a target network model based on the trained initial network model; wherein the target network model is used to generate the line of sight difference estimation value between the two eye images.

12. A sight line estimation device, comprising: An image acquisition module, used to acquire a calibration image carrying sight direction information and a target image whose sight direction is to be estimated; wherein both the calibration image and the target image are eye images; A sight difference estimation module, used for obtaining a sight difference estimation value between the calibration image and the target image through a pre-constructed target network model; wherein the target network model is obtained based on the network model construction method according to any one of claims 1 to 8; The sight direction estimation module is used to estimate the sight direction corresponding to the target image according to the sight direction information of the calibration image and the sight difference estimation value.

13. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the network model construction method described in any one of claims 1-8 or the line of sight estimation method described in any one of claims 9-10.

14. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the network model construction method described in any one of claims 1 to 8 or the line of sight estimation method described in any one of claims 9 to 10.

15. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the network model construction method described in any one of claims 1 to 8 or the line of sight estimation method described in any one of claims 9 to 10 is implemented.

Images