Methods and apparatus for aligning gaze direction labels, training models, and determining gaze direction.

Abstract

The embodiment of the application provides a line-of-sight direction label alignment, model training, line-of-sight direction determination method and device, and relates to the technical field of data processing.The line-of-sight direction label alignment method comprises the following steps: obtaining a line-of-sight direction prediction label of an object in a first sample image in a to-be-aligned sample image set based on a pre-trained label prediction model; determining alignment parameters for line-of-sight direction label alignment from the to-be-aligned sample image set to a reference sample image set based on a line-of-sight direction annotation label of the object in the first sample image and the obtained line-of-sight direction prediction label; and performing alignment adjustment on a line-of-sight direction annotation label of the object in a second sample image in the to-be-aligned sample image set based on the alignment parameters. The scheme provided by the embodiment of the application can align the line-of-sight direction labels of the objects in the sample images of different sample image sets.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and apparatus for aligning gaze direction labels, training models, and determining gaze direction. Background Technology

[0002] Gaze estimation technology can determine the gaze direction of objects in an image, and this determined gaze direction has wide applications in various scenarios. For example, vehicle driver assistance systems can determine whether the driver is fatigued based on the driver's gaze direction in the acquired image, and thus issue a warning when the driver is suspected of being fatigued; similarly, smart home appliances can determine the user's focus based on the user's gaze direction in the acquired image, and thus proactively ask questions about the target.

[0003] In related technologies, a large number of sample images are generally used to train a gaze direction determination model, and the gaze direction of objects in an image is determined based on the trained model. The sample images required to train the model usually come from multiple sample image sets. The labeling method for the gaze direction of objects in sample images within the same sample image set is the same, but the labeling method for different sample image sets is often different.

[0004] The labeling of the gaze direction of objects in sample images often differs from the actual gaze direction. This error is introduced by the calibration method, and different calibration methods generally introduce different errors. Therefore, training the model on multiple sample image sets with different errors will cause the inference within the model for each sample image set to interfere with each other, resulting in a lower accuracy of the trained model in determining the gaze direction of objects in the image.

[0005] Given the above, it is necessary to align the gaze direction labels of objects in sample images from different sample image sets. Summary of the Invention

[0006] The purpose of this application is to provide a method and apparatus for aligning gaze direction labels, training models, and determining gaze direction, so as to align gaze direction labels of objects in sample images from different sample image sets. The specific technical solution is as follows:

[0007] In a first aspect, embodiments of this application provide a method for aligning line-of-sight labels, the method comprising:

[0008] Based on a pre-trained label prediction model, the gaze direction prediction label of the object in the first sample image within the set of sample images to be aligned is obtained. The label prediction model is a model used to predict the gaze direction label of the object in the image, which is obtained by training a preset model based on a set of reference sample images.

[0009] Based on the gaze direction label of the object in the first sample image and the obtained gaze direction prediction label, the alignment parameters for aligning the gaze direction labels from the set of sample images to be aligned to the set of reference sample images are determined.

[0010] Based on the alignment parameters, the gaze direction labels of objects in the second sample image within the set of sample images to be aligned are aligned and adjusted.

[0011] Secondly, embodiments of this application provide a model training method, the method comprising:

[0012] Obtain the baseline sample image set and the sample image set to be aligned;

[0013] Based on the gaze direction label alignment method, gaze direction labels are marked on the objects in the sample images within the sample image set to be aligned, and gaze direction labels are aligned from the sample image set to be aligned to the reference sample image set to obtain the aligned sample image set. The gaze direction label alignment method is the method described in the first aspect.

[0014] The preset model is trained using the reference sample image set and the aligned sample image set to obtain a gaze direction determination model for determining the gaze direction of objects in an image.

[0015] Thirdly, embodiments of this application provide a method for determining the direction of a line of sight, the method comprising:

[0016] Obtain the target image;

[0017] The target image is input into a pre-trained gaze direction determination model to obtain the gaze direction of the object in the target image output by the gaze direction determination model. The gaze direction determination model is a model trained according to the second aspect for determining the gaze direction of the object in the image.

[0018] Fourthly, embodiments of this application provide a line-of-sight label alignment device, the device comprising:

[0019] The prediction label acquisition module is used to obtain the gaze direction prediction label of the object in the first sample image in the sample image set to be aligned based on a pre-trained label prediction model. The label prediction model is a model for predicting the gaze direction label of the object in the image, which is obtained by training a preset model based on the benchmark sample image set.

[0020] The alignment parameter determination module is used to determine the alignment parameters for aligning the gaze direction labels from the set of sample images to be aligned to the set of reference sample images, based on the gaze direction label of the object in the first sample image and the obtained gaze direction prediction label;

[0021] The first label alignment module is used to align and adjust the gaze direction labels of objects in the second sample image within the sample image set to be aligned, based on the alignment parameters.

[0022] Fifthly, embodiments of this application provide a model training apparatus, the apparatus comprising:

[0023] The sample image set acquisition module is used to obtain the baseline sample image set and the sample image set to be aligned;

[0024] The second label alignment module is used to label the gaze direction of objects in the sample images within the sample image set to be aligned based on the gaze direction label alignment method, and to perform gaze direction label alignment from the sample image set to be aligned to the reference sample image set to obtain the aligned sample image set, wherein the gaze direction label alignment method is the method described in the first aspect;

[0025] The model training module is used to train the preset model using the reference sample image set and the aligned sample image set to obtain a gaze direction determination model for determining the gaze direction of objects in the image.

[0026] Sixthly, embodiments of this application provide a line-of-sight direction determination device, the device comprising:

[0027] The target image acquisition module is used to acquire the target image;

[0028] The gaze direction determination module is used to input the target image into a pre-trained gaze direction determination model to obtain the gaze direction of the object in the target image output by the gaze direction determination model, wherein the gaze direction determination model is a model trained according to the second aspect for determining the gaze direction of the object in the image.

[0029] In a seventh aspect, embodiments of this application provide an electronic device, including:

[0030] Memory, used to store computer programs;

[0031] A processor, when executing a program stored in memory, implements the method described in any of the first, second, or third aspects.

[0032] Eighthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first, second, or third aspect.

[0033] In a ninth aspect, embodiments of this application also provide a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the methods described in the first, second, or third aspect.

[0034] As can be seen from the above, when applying the scheme provided in the embodiments of this application for eye direction label alignment, a pre-trained label prediction model is first used to obtain the eye direction prediction label of the object in the first sample image in the image set to be aligned. Then, based on the eye direction label of the object in the first sample image and the obtained eye direction prediction label, the alignment parameters for eye direction label alignment from the image set to be aligned to the reference image set are determined. Then, based on the alignment parameters, the eye direction label of the object in the second sample image in the image set to be aligned can be aligned and adjusted.

[0035] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0037] Figure 1 A flowchart illustrating the first line-of-sight label alignment method provided in this application embodiment;

[0038] Figure 2 A schematic diagram illustrating the first line-of-sight label alignment process provided in this application embodiment;

[0039] Figure 3 A schematic diagram illustrating the second line-of-sight label alignment process provided in this application embodiment;

[0040] Figure 4 A flowchart illustrating the second line-of-sight label alignment method provided in this application embodiment;

[0041] Figure 5 A schematic diagram illustrating an alignment parameter determination process provided in an embodiment of this application;

[0042] Figure 6A schematic flowchart illustrating a model training method provided in an embodiment of this application;

[0043] Figure 7 A flowchart illustrating a method for determining the line of sight direction provided in an embodiment of this application;

[0044] Figure 8 This is a schematic diagram of the structure of a line-of-sight label alignment device provided in an embodiment of this application;

[0045] Figure 9 This is a schematic diagram of the structure of a model training device provided in an embodiment of this application;

[0046] Figure 10 This is a schematic diagram of the structure of a line-of-sight direction determination device provided in an embodiment of this application;

[0047] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0048] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0049] First, the implementing entity of the solution provided in the embodiments of this application will be described.

[0050] The implementation subject of the solution provided in this application embodiment is any electronic device with data processing, storage and other functions.

[0051] The line-of-sight label alignment method provided in the embodiments of this application will now be described in detail.

[0052] See Figure 1 This is a flowchart illustrating the first line-of-sight label alignment method provided in this application embodiment. The method includes the following steps S101-S103.

[0053] Step S101: Based on the pre-trained label prediction model, obtain the predicted label of the gaze direction of the object in the first sample image within the set of sample images to be aligned.

[0054] The label prediction model is a model trained on a preset model based on a set of benchmark sample images, used to predict the gaze direction labels of objects in an image. The first sample image can be a portion of the image set to be aligned, or it can be all the images in the image set to be aligned.

[0055] First, the above-mentioned set of sample images to be aligned and the set of reference sample images will be explained.

[0056] The aforementioned benchmark image set can be a set of sample images pre-determined from the entire set of sample images used to train the model for determining the gaze direction. The aforementioned set of sample images to be aligned can be a set of sample images from the entire set of sample images excluding the benchmark image set. The specific method for determining the benchmark image set will be detailed later and will not be elaborated here.

[0057] Both the baseline sample image set and the sample image set to be aligned include sample images and labels indicating the gaze direction of objects within those images. The objects can be people or animals, depending on the specific application scenario.

[0058] The following section introduces the labeling of the gaze direction of objects in the sample images.

[0059] First, let's explain the direction of the line of sight.

[0060] The gaze direction of an object in an image refers to the direction of its gaze, that is, the direction from the starting point of the gaze to the ending point, which can be represented by a direction vector. The starting point of the gaze can be the center point of the object's eyes, the center point of the object's facial region, etc., and the ending point of the gaze can be the target that the object is looking at.

[0061] For example, in an image X containing the face of person M, the center point of person M's eyes can be used as the starting point of the line of sight, and the target that person M is looking at is the point where the line of sight ends. Then, the direction from the center point of the eyes to the target is the line of sight direction of the object in image X.

[0062] It should be noted that the target being viewed by the object may or may not be included in the image.

[0063] Based on this, the aforementioned gaze direction label is: the label value of the gaze direction of the object in the sample image. That is, the gaze start point and gaze endpoint of the object in the sample image are calibrated using a certain calibration method, and the gaze direction of the object in the sample image is determined based on the calibrated gaze start point and gaze endpoint.

[0064] Within the same sample image set, the labeling method for the gaze direction annotation of objects in the sample images is the same, but the labeling method can be different for different sample image sets. That is, each sample image set can correspond to different labeling methods, and thus, the gaze direction annotation of objects in the sample images within each sample image set can be labels under different labeling methods.

[0065] In addition, the gaze direction labels of objects in the sample images within the sample image set often have an error compared to the actual gaze direction. This error is introduced by the calibration method and can be called calibration error. Different calibration methods generally introduce different calibration errors. Therefore, the calibration errors corresponding to different sample image sets are often different.

[0066] The following section will explain the predicted gaze direction labels for objects in the first sample image within the image set to be aligned.

[0067] First, the label prediction model is trained on a benchmark image set. During training, the model learns the relationship between the gaze direction labels of objects in the sample images and the target calibration methods corresponding to the benchmark image set. Therefore, the trained label prediction model can predict the gaze direction labels of objects in the images under the target calibration methods.

[0068] Therefore, by inputting the first sample image in the set of sample images to be aligned into the label prediction model, we can obtain the gaze direction label of the object in the first sample image under the target calibration method. Since the label is predicted, it can be called the gaze direction prediction label of the object in the first sample image.

[0069] Based on the above analysis, it can be seen that the aforementioned gaze direction prediction label is essentially: the gaze direction label of the object in the first sample image under the target calibration method corresponding to the reference sample image set.

[0070] Step S102: Based on the gaze direction label of the object in the first sample image and the obtained gaze direction prediction label, determine the alignment parameters for aligning the gaze direction labels from the set of sample images to be aligned to the set of reference sample images.

[0071] The aforementioned gaze direction labels refer to the gaze direction annotation labels of objects in the sample images within the sample image set. Therefore, gaze direction label alignment means adjusting the original gaze direction annotation labels of objects in the sample images within the sample image set to be aligned to the gaze direction annotation labels under the target calibration method corresponding to the reference sample image set.

[0072] In step S101, the predicted gaze direction label is obtained, and the obtained predicted gaze direction label is: the gaze direction annotation label of the object in the first sample image under the target calibration method corresponding to the reference sample image set. Therefore, in this step, the alignment parameters for aligning the gaze direction labels from the sample image set to be aligned to the reference sample image set can be determined based on the original gaze direction annotation label of the first sample image and its corresponding predicted gaze direction label.

[0073] Specifically, the alignment parameters mentioned above can be determined in the following ways.

[0074] In one implementation, gaze direction alignment labels represented by alignment parameters can be obtained first, and then the alignment parameters that minimize the total difference between the gaze direction alignment labels corresponding to the first sample image and the gaze direction prediction labels can be determined. Detailed implementation methods will be described later. Figure 4 Steps S402 and S403 in the illustrated embodiment will not be described in detail here.

[0075] In another implementation, for each first sample image, the difference between the gaze direction label and the gaze direction prediction label corresponding to the first sample image can be obtained, and the average value of the obtained difference can be used as the alignment parameter.

[0076] For example, the gaze direction label and gaze direction prediction label corresponding to the first sample image can both be represented by direction vectors. In this way, the difference vector between the gaze direction label and the gaze direction prediction label corresponding to each first sample image can be calculated, and the average vector of the obtained difference vectors can be used as the alignment parameter mentioned above.

[0077] Step S103: Based on the alignment parameters, adjust the alignment of the gaze direction labels of objects in the second sample image within the image set to be aligned.

[0078] The second sample image mentioned above can be a portion of the sample image set to be aligned, or it can be all the images in the sample image set to be aligned.

[0079] Specifically, the distribution of the aforementioned first sample image and the second sample image in this step within the set of sample images to be aligned can be as follows:

[0080] In the first case, the first sample image is a subset of the sample images to be aligned, and the second sample image is a subset of the sample images to be aligned.

[0081] The first sample image and the second sample image can be the same part of the sample image set to be aligned, or they can be two different parts of the sample image set to be aligned, or they can be two parts of the image set with overlapping images.

[0082] In the second case, the first sample image is a subset of the images in the set of sample images to be aligned, and the second sample image is all the images in the set of sample images to be aligned.

[0083] In the third case, the first sample image is a subset of the images in the set of sample images to be aligned, and the second sample image is all the images in the set of sample images to be aligned.

[0084] As can be seen, there are multiple ways to determine the first sample image used to determine the alignment parameters and the second sample image to be aligned with the visual orientation labels within the image set to be aligned. This allows for more flexible determination of the alignment parameters and the alignment of visual orientation labels based on the determined alignment parameters, thereby improving the flexibility and applicability of the solution.

[0085] As can be seen from the description in step S102 above, the alignment parameter is essentially used to adjust the gaze direction label of the object in the sample image in the sample image set to be aligned to the gaze direction label under the target calibration method corresponding to the reference sample image set.

[0086] Therefore, after aligning and adjusting the gaze direction labels of objects in the second sample image within the image set to be aligned based on the alignment parameters, the gaze direction labels of objects in the second sample image within the image set to be aligned can be adjusted to the gaze direction labels under the target calibration method corresponding to the reference image set. In other words, gaze direction labels are aligned from the image set to be aligned to the reference image set.

[0087] The following explains the specific method for aligning and adjusting the gaze direction labels of objects in the second sample image within the image set to be aligned, based on the alignment parameters.

[0088] Specifically, depending on the alignment parameters, the method for aligning and adjusting the gaze direction labels of objects in the second sample image within the image set to be aligned also varies, and will be explained below.

[0089] In one embodiment, the alignment parameters may include rotation and translation parameters. In this case, the gaze direction annotation labels of objects in the second sample image can be projected and transformed based on the aforementioned rotation and translation parameters. Detailed implementation methods will be described later. Figure 4 Step S404 in the illustrated embodiment will not be described in detail here.

[0090] In another implementation, the alignment parameter can be the average vector of the difference vector between the gaze direction label and the gaze direction prediction label corresponding to the first sample image. In this case, the sum vector of the gaze direction label of the object in the second sample image and the above average vector can be calculated. The above sum vector is the new gaze direction label after the gaze direction label of the object in the second sample image is aligned and adjusted.

[0091] As can be seen from the above, when applying the scheme provided in the embodiments of this application to perform gaze direction label alignment, a pre-trained label prediction model is first used to obtain the gaze direction prediction label of the object in the first sample image in the image set to be aligned. Then, based on the gaze direction label of the object in the first sample image and the obtained gaze direction prediction label, the alignment parameters for aligning the gaze direction labels from the image set to be aligned to the reference image set are determined. Then, based on the alignment parameters, the gaze direction label of the object in the second sample image in the image set to be aligned can be aligned and adjusted.

[0092] The aforementioned label prediction model is trained on a pre-defined model using a benchmark sample image set. During training, the model learns the relationship between the gaze direction labels of the sample images and the objects within those images under the target calibration method corresponding to the benchmark sample image set. In other words, the trained label prediction model can predict the gaze direction labels of objects in the images under the target calibration method. Therefore, the obtained gaze direction prediction labels are actually the predicted gaze direction labels of the objects in the first sample image under the target calibration method. Thus, based on the original gaze direction labels of the objects in the first sample image and the obtained gaze direction prediction labels, alignment parameters can be obtained to adjust the gaze direction labels of the objects in the first sample image to match the gaze direction labels under the aforementioned target calibration method.

[0093] Furthermore, based on the alignment parameters, the gaze direction labels of objects in the second sample image under the aforementioned target calibration method can be obtained. This ensures that the gaze direction labels of objects in the second sample image within the image set to be aligned and the gaze direction labels of objects in the sample images within the reference image set are both labels under the target calibration method. This aligns the gaze direction labels of objects in the second sample image within the image set to be aligned with the gaze direction labels of the reference image set, thereby unifying the calibration errors corresponding to the image set to be aligned and the reference image set. This reduces the differences in calibration errors between sample image sets and improves the self-consistency of each sample image set. Consequently, when training the model based on the image set to be aligned and the reference image set, the degree of mutual influence between inferences for each sample image set within the model is reduced, which is beneficial to improving model performance and thus improving the accuracy of the trained model in determining the gaze direction of objects in the image.

[0094] The method for determining the aforementioned benchmark sample image set will be explained below.

[0095] In one approach, a baseline sample image set can be determined from the entire sample image set based on the calibration method of the gaze direction label of the object in the sample image within the sample image set.

[0096] For example, the sample image set corresponding to the calibration method with the smallest calibration error can be determined as the sample image set, or the benchmark sample image set can be determined from the sample image set corresponding to the calibration method with a calibration error less than a preset threshold.

[0097] The calibration error introduced by the calibration method can be measured based on the characteristics of the calibration method itself, and this application does not limit this.

[0098] For example, the calibration method corresponding to sample image set A is based on a stereo camera, while the calibration method corresponding to sample image set B is based on a monocular camera. Since the calibration method based on a stereo camera is generally more accurate than the calibration method based on a monocular camera, it can be assumed that the calibration error corresponding to sample image set A is smaller than the calibration error corresponding to sample image set B.

[0099] This way, after aligning the gaze direction labels from the set of sample images to be aligned to the set of reference sample images, the calibration error of each set of sample images can be reduced, thereby further improving the accuracy of the gaze direction determination model trained on the set of sample images in determining the gaze direction of objects in the image.

[0100] In another implementation, the expected application scenario of the model can be determined based on the image scene and gaze direction in the sample images, and a benchmark sample image set can be determined from the above-mentioned set of all sample images.

[0101] Specifically, the set of sample images in the included sample images that match the expected application scenario of the model can be determined as the benchmark sample image set.

[0102] For example, if the model is intended to be used in a driver fatigue warning scenario, then the set of sample images in the included sample images that depict a driver driving inside a vehicle can be determined as the baseline sample image set.

[0103] In this way, after aligning the gaze direction labels from the sample image set to the reference sample image set, the calibration error of each sample image set can be the calibration error of the sample image set corresponding to the calibration method that matches the expected application scenario of the model. This allows the model to learn sample image features that are more compatible with the expected application scenario of the model based on each sample image set, thereby further improving the accuracy of the gaze direction determination model trained on the sample image set in determining the gaze direction of objects in the image.

[0104] The following is combined Figure 2 and Figure 3 Regarding the above Figure 1 The line-of-sight label alignment process described in the illustrated embodiment will be explained in its entirety.

[0105] First see Figure 2 This is a schematic diagram of the first line-of-sight label alignment process provided in the embodiments of this application.

[0106] Figure 2 The process shown is: the process of visually aligning the sample image set B to the reference sample image set A.

[0107] Figure 2 In the image set I, the predicted label corresponds to the aforementioned gaze direction predicted label. A Let A represent the set of sample images contained in the sample image set, and L represent the label set. A This represents the set of gaze direction labels contained in the sample image set A; similarly, image set I... B and label set L B These represent the set of sample images contained in sample image set B and the set of line-of-sight direction labels, respectively.

[0108] It should be noted that image set I B and label set L B It can be either the set of all sample images contained in the sample image set B and the set of all line-of-sight labels, or it can be the set of some sample images contained in the sample image set B and the set of some line-of-sight labels.

[0109] Depend on Figure 2 As can be seen, firstly, based on image set I in sample image set A A and tag set L A After pre-training, a label prediction model M can be obtained. A Secondly, the image set I B The sample image input model M A This allows us to obtain the predicted label L′ corresponding to the sample image. B Thus, based on the predicted label L′ corresponding to the sample image B and tag set L B The original labels corresponding to the sample images can be used to determine alignment parameters. These alignment parameters are used to visually align the labels from sample image set B to sample image set A. Therefore, label alignment can be performed based on these alignment parameters, i.e., for label set L... B Perform alignment adjustments to obtain the adjusted tag set.

[0110] See also Figure 3 This is a schematic diagram of the second line-of-sight label alignment process provided in the embodiments of this application.

[0111] Figure 3Label A, label B, and label C correspond to the aforementioned line-of-sight direction labels, and prediction labels A, prediction labels B, and prediction labels C correspond to the aforementioned line-of-sight direction prediction labels. Figure 3 The process shown is as follows: the process of aligning the line-of-sight labels from the sample image set to be aligned, including the sample image set AC, to the reference sample image set.

[0112] As can be seen from the figure, Figure 3 The process shown can be divided into three sub-processes: label prediction model training, alignment parameter determination, and label alignment. These will be explained below.

[0113] Label prediction model training:

[0114] The preset model is trained based on a set of benchmark sample images to obtain a label prediction model for predicting the gaze direction labels of objects in the image.

[0115] Alignment parameters determined:

[0116] As can be seen, for each set of sample images to be aligned, such as sample image set AC in the figure, the predicted labels of the objects in the sample images within the sample image set are obtained based on the label prediction model. Thus, based on the predicted labels corresponding to each sample image set and its original annotation labels, the alignment parameters can be determined.

[0117] For example, for a sample image set A, the alignment parameter A corresponding to sample image set A is determined based on the labeled A and predicted A of the objects in the sample images. Similarly, the alignment parameters B and C corresponding to sample image sets B and C can be determined.

[0118] Label alignment:

[0119] Based on the alignment parameters corresponding to each sample image set, the annotation labels of objects in the sample images within each sample image set are aligned and adjusted to obtain an aligned image set.

[0120] For example, given the alignment parameter A corresponding to sample image set A, the labels of objects in the sample images within sample image set A are aligned and adjusted to determine the aligned image set A. Similarly, aligned image sets B and C can be obtained.

[0121] exist Figure 1 Based on the illustrated embodiment, when determining the alignment parameter in step S102 above, the gaze direction alignment label represented by the alignment parameter can be obtained first, and then the alignment parameter that minimizes the total difference between the gaze direction alignment label corresponding to the first sample image and the gaze direction prediction label can be determined. In view of the above, this application embodiment provides a second gaze direction label alignment method.

[0122] See Figure 4This is a flowchart illustrating the second line-of-sight label alignment method provided in this application embodiment. The method includes the following steps S401-S404.

[0123] Step S401: Based on the pre-trained label prediction model, obtain the predicted label of the gaze direction of the object in the first sample image within the sample image set to be aligned.

[0124] The above step S401 is the same as the aforementioned Figure 1 Step S101 is the same in the illustrated embodiment, and will not be repeated here.

[0125] Step S402: Obtain the gaze direction alignment label represented by the alignment parameter.

[0126] Among them, the gaze direction alignment label refers to the gaze direction label of the object in the first sample image after the gaze direction label is aligned from the sample image set to be aligned to the reference sample image set.

[0127] It should be noted that the gaze direction alignment label obtained here is only a label represented by the alignment parameter. Since the alignment parameter has not yet been determined, the gaze direction alignment label is also unknown.

[0128] Specifically, the line-of-sight alignment label, represented by the alignment parameter, can be obtained in the following way.

[0129] In one implementation, the gaze direction alignment label represented by the alignment parameters can be obtained directly based on the relationship between the gaze direction alignment label, the gaze direction label of the object in the first sample image, and the alignment parameters.

[0130] For example, let the alignment parameter be align, and let the gaze direction label of the object in the first sample image be L. ori Then the line of sight alignment L is represented by the alignment parameter. align For: F1(align, L) ori ), where F1 represents the operation relationship used to obtain the line-of-sight alignment label.

[0131] In one case, if the alignment parameter align includes the rotation parameter R and the translation parameter T, then F1(align, L) ori ) can be R·L ori +T, thus aligning the line of sight with label L. align It can be represented by the following expression:

[0132] L align =R·L ori +T

[0133] In another implementation, the alignment point corresponding to the first gaze direction landing point after aligning the gaze direction labels from the set of sample images to be aligned to the set of reference sample images, represented by alignment parameters, can be obtained. Based on the gaze starting point used to determine the gaze direction label of the object in the first sample image and the obtained alignment point, the gaze direction alignment label of the object in the first sample image can be determined.

[0134] The first gaze point is the gaze point used to determine the gaze direction label of the object in the first sample image.

[0135] Line-of-sight (LOS) direction labels are generally determined based on the starting and ending points of the line of sight. Specifically, the starting and ending points of the line of sight can be obtained from the object in the sample image, and the difference between the ending point and the starting point is used as the line-of-sight direction label.

[0136] Generally speaking, a fixed position or a random position on a known plane or screen can be predetermined as the gaze point. Then, an image acquisition device can be used to acquire sample images of people or other objects looking at the gaze point. The gaze direction of the object in the sample image can be determined by the gaze starting point and the gaze point.

[0137] The starting point and ending point of the line of sight are described below.

[0138] Regarding the starting point of the line of sight:

[0139] For sample images in different sample image sets, the calibration method for the starting point of the line of sight of the object in the sample image is often consistent. Therefore, it can be considered that the error introduced by the calibration method used to determine the starting point of the line of sight is the same for different sample image sets, that is, there is no difference between the introduced errors.

[0140] For example, taking a face image as the sample image, the face region in the sample image can be determined based on a face detection algorithm, then the center point of the face region can be determined, and the starting point of the person's gaze in the sample image can be determined based on the determined center point; or, face feature points can be extracted from the sample image, the midpoint of the extracted binocular feature points can be determined, and the starting point of the person's gaze in the sample image can be determined based on the determined midpoint, etc.

[0141] Furthermore, even if the calibration methods for the gaze origin of objects in sample images from different sample image sets are inconsistent, since all calibration methods can use facial features as a clear reference to accurately determine the gaze origin, the error between the gaze origin of objects in sample images from different sample image sets and the actual gaze origin can be ignored. Therefore, there is naturally no difference in the error introduced by the gaze origin calibration methods corresponding to different sample image sets.

[0142] In summary, regardless of whether the calibration method of the gaze starting point of the object in the sample images of different sample image sets is consistent, it can be assumed that there is no difference in the error introduced by the gaze starting point calibration method corresponding to different sample image sets. Therefore, it can be assumed that the gaze starting point used to determine the gaze direction label in each sample image set is the gaze starting point under the same calibration method. In other words, it can be assumed that the gaze starting points used to determine the gaze direction label in each sample image set are naturally aligned.

[0143] Regarding the point of view:

[0144] For sample images in different sample image sets, the calibration method for the gaze direction label of the object in the sample image is often different. Therefore, it can be assumed that there are differences in the error introduced by the gaze direction labeling method corresponding to different sample image sets.

[0145] Taking the point of view as being located on a screen or plane as an example, the following is an example of how to calibrate the point of view.

[0146] In one embodiment, a target can be displayed on a screen or pasted on a plane, and an image containing the target can be captured using a camera. This allows the pose relationship between the camera and the screen or plane to be calibrated based on the camera's intrinsic parameters. Furthermore, the spatial position of the actual line of sight on the screen or plane in the camera coordinate system can be determined based on the pose relationship, thus completing the line of sight calibration.

[0147] In another implementation, a laser rangefinder can be used to measure the position of the line of sight in the rangefinder coordinate system, and then a camera can be used to calibrate the pose of the laser rangefinder in the camera coordinate system. This obtains the coordinate transformation relationship between the laser rangefinder coordinate system and the camera coordinate system, and then the position of the line of sight in the rangefinder coordinate system can be converted into the spatial position in the camera coordinate system, thus completing the line of sight calibration.

[0148] Unlike determining the starting point of a gaze, which can be based on clear facial features, different gaze placement methods have significantly different procedures for determining the gaze placement point. These methods include both high-precision and low-precision methods. Therefore, the errors introduced by different gaze placement methods vary. Consequently, it can be assumed that the gaze placement points used to determine the gaze direction labels in each sample image set are not gaze placement points under the same calibration method, or that the gaze placement points used to determine the gaze direction labels in each sample image set are not aligned.

[0149] In summary, the starting points of the gaze direction labels used to determine the gaze direction in each sample image set are naturally aligned, while the ending points of the gaze direction labels used to determine the gaze direction in each sample image set are not aligned. Therefore, when aligning gaze direction labels from the aligned sample image set to the reference sample image set, it is only necessary to align the ending points of the gaze from the aligned sample image set to the reference sample image set.

[0150] Therefore, the alignment point corresponding to the first gaze direction landing point after aligning the gaze direction label from the sample image set to the reference sample image set can be obtained. The difference between the obtained alignment landing point and the gaze starting point of the object in the first sample image can be calculated to obtain the gaze direction alignment label of the object in the first sample image.

[0151] The following explains the specific method for obtaining the alignment point represented by the alignment parameter.

[0152] Similarly, let the alignment parameter be 'align', and let P be the point where the gaze falls on the first sample image. end Then the alignment point P′ is represented by the alignment parameter. end It can be: F2(align, P) end ), where F2 represents the operation relationship used to obtain the alignment point.

[0153] In one case, if the alignment parameter align includes the rotation parameter R and the translation parameter T, then F2(align, P) end ) can be R·P end +T, thus aligning the landing point P′ end It can be represented by the following expression:

[0154] P′ end =R·P end +T

[0155] As can be seen, when the alignment parameters include rotation and translation parameters, the alignment point can be accurately and quickly represented according to the above expression.

[0156] After obtaining the alignment point, calculate the alignment point P′. end The gaze starting point P corresponding to the first sample image start The difference is enough to obtain the line-of-sight alignment label. At the alignment point P′ end When expressed using the above expression, the line of sight is aligned with label L. align It can be represented by the following expression:

[0157] L align =P end -P start =R·P end +TP start

[0158] As can be seen from the above, since the starting points of the gaze of objects in each sample image set are naturally aligned, while the ending points of the gaze of objects in different sample image sets are often not aligned, we can first obtain the alignment point corresponding to the first gaze ending point after aligning the gaze direction labels from the sample image set to be aligned to the reference sample image set, expressed by alignment parameters. Based on the starting point of the gaze direction label used to determine the gaze direction label of the object in the first sample image and the obtained alignment ending point, we can more reasonably and accurately determine the gaze direction alignment label of the object in the first sample image after aligning the gaze direction labels from the sample image set to be aligned to the reference sample image set.

[0159] Step S403: Determine the alignment parameter that minimizes the total difference between the gaze direction alignment label and the gaze direction prediction label corresponding to the first sample image.

[0160] In this step, a globally optimal approach can be used to obtain alignment parameters that minimize the total difference between the gaze direction alignment label and the gaze direction prediction label corresponding to the first sample image.

[0161] In one implementation, for each first sample image, the difference representation value between the gaze direction alignment label and the gaze direction prediction label corresponding to the first sample image can be obtained. Based on the obtained difference representation value, the total difference representation value is determined, and then the alignment parameter that minimizes the total difference representation value is determined.

[0162] First, let's explain the above-mentioned difference representation values.

[0163] In one case, the aforementioned difference representation value can be the absolute error between the gaze direction alignment label and the gaze direction prediction label corresponding to the first sample image.

[0164] The absolute error Loss1 mentioned above can be expressed by the following expression:

[0165] Loss1=|L pre -L align |

[0166] Among them, L pre L represents the predicted label for the gaze direction corresponding to the first sample image. align This indicates the gaze direction alignment label corresponding to the first sample image.

[0167] In another case, the aforementioned difference characterization value can be the angular error between the gaze direction alignment label and the gaze direction prediction label corresponding to the first sample image.

[0168] The aforementioned angle error Loss2 can be expressed by the following expression:

[0169]

[0170] Among them, L pre L represents the predicted label for the gaze direction corresponding to the first sample image. align This indicates the gaze direction alignment label corresponding to the first sample image.

[0171] The aforementioned absolute error or angular error can accurately characterize the difference between the gaze direction alignment label and the gaze direction prediction label corresponding to the first sample image, thereby obtaining a more accurate total difference characterization value based on the obtained difference characterization value.

[0172] The total difference representation value mentioned above can be the sum of the obtained difference representation values, or the average value of the obtained difference representation values, etc.

[0173] After obtaining the total difference characterization value, a global optimal algorithm can be used to determine the alignment parameter that minimizes the total difference characterization value. The algorithm can be the least squares method, the quasi-Newton method, etc., and this application embodiment does not limit it.

[0174] As can be seen, for each first sample image, the difference representation value between the gaze direction alignment label and the gaze direction prediction label corresponding to the first sample image is obtained. Thus, the total difference representation value determined based on the obtained difference representation value can more accurately reflect the difference between the gaze direction alignment label and the gaze direction prediction label corresponding to the first sample image. In turn, a more accurate alignment parameter can be obtained based on the above total difference representation value.

[0175] Step S404: Based on the alignment parameters, adjust the alignment of the gaze direction labels of objects in the second sample image within the set of sample images to be aligned.

[0176] Specifically, if the alignment parameters include rotation and translation parameters, the gaze direction labels of objects in the second sample image within the set of sample images to be aligned can be aligned and adjusted according to the following expression:

[0177] L′=R con ·P end +T con -P start

[0178] Where L′ represents the aligned and adjusted line-of-sight label, R con and T con Let P represent the rotation and translation parameters that minimize the total difference between the gaze direction alignment label and the gaze direction prediction label corresponding to the first sample image. start and P endThese represent the starting point and ending point of the gaze direction label used to determine the gaze direction of objects in the second sample image, respectively.

[0179] It can be seen that the above expression is similar to the expression for the alignment label of the gaze direction of the object in the first sample image, which is represented by the alignment parameter in step S402. The difference is that in step S402, the alignment label of the gaze direction of the object in the first sample image is represented by the undetermined alignment parameter, while here the alignment label of the gaze direction of the object in the second sample image is adjusted according to the determined alignment parameter obtained by the final solution.

[0180] As can be seen from the above, after obtaining the gaze direction alignment label represented by the alignment parameter, the alignment parameter that minimizes the total difference between the gaze direction alignment label and the gaze direction prediction label corresponding to the first sample image can be determined. Since the gaze direction prediction label is the label of the object in the first sample image under the target calibration method corresponding to the reference sample image set, after adjusting the alignment of the gaze direction label of the object in the second sample image within the sample image set to be aligned based on the above alignment parameter, the adjusted gaze direction label can be made as close as possible to the predicted gaze direction label of the object in the second sample image under the target calibration method, thus improving the alignment effect of the gaze direction label.

[0181] The following is combined Figure 5 Regarding the above Figure 4 The alignment parameter determination process involved in the illustrated embodiment is explained in its entirety.

[0182] See Figure 5 This is a schematic diagram of an alignment parameter determination process provided in an embodiment of this application.

[0183] Figure 5 The process shown is as follows: determining the alignment parameters for visual orientation label alignment from the sample image set B to the reference sample image set A.

[0184] Depend on Figure 5 It can be seen that for the label set L of the sample image set B B For cases involving line-of-sight direction labels, the starting point of the line of sight used to determine the line-of-sight direction label is first obtained. and the point of view Then obtain the point of view. Using rotation parameter R BtoA Translation parameter T BtoA The indicated alignment point In this way, it can be based on the starting point of the line of sight. and alignment landing point Obtain the alignment label of the line-of-sight direction annotation using the alignment parameter. Based on the above alignment tags The line-of-sight direction prediction label L′ corresponding to the line-of-sight direction label B By calculating the loss value between the two values, we can obtain the alignment parameter that minimizes the loss.

[0185] The aforementioned loss value (Loss) corresponds to the difference between the gaze direction alignment label and the gaze direction prediction label of the sample image described above.

[0186] Corresponding to the above-mentioned gaze direction label alignment method, this application embodiment also provides a model training method.

[0187] See Figure 6 This is a flowchart illustrating a model training method provided in an embodiment of this application. The method includes the following steps S601-S603.

[0188] Step S601: Obtain the baseline sample image set and the sample image set to be aligned.

[0189] The concepts of the aforementioned reference sample image set and the sample image set to be aligned have been explained in the embodiments corresponding to the aforementioned line-of-sight label alignment method, and will not be repeated here.

[0190] Step S602: Based on the gaze direction label alignment method, label the gaze direction of objects in the sample images within the sample image set to be aligned, and align the gaze direction labels from the sample image set to be aligned to the reference sample image set to obtain the aligned sample image set.

[0191] The method for aligning gaze direction labels is as described above.

[0192] After this step, the gaze direction labels of the sample image set to be aligned are aligned with those of the reference sample image set. In other words, the gaze direction labels of objects in the sample images of the sample image set to be aligned are all adjusted to the labels under the target calibration method corresponding to the reference sample image set. As a result, the calibration error corresponding to the sample image set to be aligned and the calibration error corresponding to the reference sample image set become unified.

[0193] Step S603: Use the benchmark sample image set and the aligned sample image set to train the preset model to obtain a gaze direction determination model for determining the gaze direction of objects in the image.

[0194] Specifically, the model can be trained using the baseline sample image set and the sample images in the aligned sample image set as model inputs, and the gaze direction annotation labels corresponding to the sample images as training annotations. The model training ends when the preset training termination condition is met, thus obtaining the gaze direction determination model.

[0195] As can be seen from the above, when using the scheme provided in this application for model training, the gaze direction labels of objects in the sample images within the sample image set to be aligned are first aligned from the sample image set to the reference sample image set based on the gaze direction label alignment method, resulting in an aligned sample image set. This adjusts the gaze direction labels of objects in the sample images within the sample image set to be aligned to the labels under the target calibration method corresponding to the reference sample image set. As a result, the calibration errors corresponding to the sample image set to be aligned and the reference sample image set become unified, reducing the differences between the calibration errors of the sample image sets and improving the self-consistency of each sample image set. Consequently, when training the model based on the sample image set to be aligned and the reference sample image set, the degree of mutual influence between the inferences of each sample image set within the model is reduced, which is beneficial to improving model performance and thus improving the accuracy of the trained model in determining the gaze direction of objects in the image.

[0196] Corresponding to the above-mentioned method for aligning line-of-sight labels, this application also provides a method for determining line-of-sight direction.

[0197] See Figure 7 This is a flowchart illustrating a method for determining the direction of sight provided in an embodiment of this application. The method includes the following steps S701-S702.

[0198] Step S701: Obtain the target image.

[0199] The target image mentioned above can be an image containing objects such as people and animals.

[0200] Step S702: Input the target image into the pre-trained gaze direction determination model to obtain the gaze direction of the object in the target image as output by the gaze direction determination model.

[0201] The gaze direction determination model is a model trained according to the model training method described above, used to determine the gaze direction of objects in an image.

[0202] As can be seen from the above, when using the scheme provided in the embodiments of this application to determine the gaze direction, after obtaining the target image, the target image can be input into the pre-trained gaze direction determination model to obtain the gaze direction of the object in the target image output by the gaze direction determination model.

[0203] As can be seen from the aforementioned model training method embodiment, before training the above-mentioned gaze direction model, the gaze direction labels of objects in the sample images within the sample image set to be aligned are aligned from the sample image set to the reference sample image set, resulting in an aligned sample image set. This adjusts the gaze direction labels of objects in the sample images within the sample image set to be aligned to the labels under the target calibration method corresponding to the reference sample image set. As a result, the calibration errors corresponding to the sample image set to be aligned and the reference sample image set become unified, reducing the differences between the calibration errors of the sample image sets and improving the self-consistency of each sample image set. Consequently, when training the model based on the sample image set to be aligned and the reference sample image set, the degree of mutual influence between the inferences of each sample image set within the model is reduced, which is beneficial to improving model performance. This results in a gaze direction determination model with higher accuracy, and based on the above model, the accuracy of recognizing the gaze direction of objects in images can be improved.

[0204] Corresponding to the above-described method for aligning line-of-sight labels, this application also provides a device for aligning line-of-sight labels.

[0205] See Figure 8 This is a schematic diagram of a line-of-sight label alignment device provided in an embodiment of this application. The device includes the following modules:

[0206] The prediction label acquisition module 801 is used to obtain the gaze direction prediction label of the object in the first sample image in the sample image set to be aligned based on a pre-trained label prediction model. The label prediction model is a model for predicting the gaze direction label of the object in the image, which is obtained by training a preset model based on the benchmark sample image set.

[0207] The alignment parameter determination module 802 is used to determine the alignment parameters for aligning the gaze direction labels from the set of sample images to be aligned to the set of reference sample images based on the gaze direction label of the object in the first sample image and the obtained gaze direction prediction label.

[0208] The first label alignment module 803 is used to align and adjust the gaze direction labels of objects in the second sample image within the sample image set to be aligned based on the alignment parameters.

[0209] As can be seen from the above, when applying the scheme provided in the embodiments of this application for eye direction label alignment, a pre-trained label prediction model is first used to obtain the eye direction prediction label of the object in the first sample image in the image set to be aligned. Then, based on the eye direction label of the object in the first sample image and the obtained eye direction prediction label, the alignment parameters for eye direction label alignment from the image set to be aligned to the reference image set are determined. Then, based on the alignment parameters, the eye direction label of the object in the second sample image in the image set to be aligned can be aligned and adjusted.

[0210] The aforementioned label prediction model is trained on a pre-defined model using a benchmark sample image set. During training, the model learns the relationship between the gaze direction labels of the sample images and the objects within those images under the target calibration method corresponding to the benchmark sample image set. In other words, the trained label prediction model can predict the gaze direction labels of objects in the images under the target calibration method. Therefore, the obtained gaze direction prediction labels are actually the predicted gaze direction labels of the objects in the first sample image under the target calibration method. Thus, based on the original gaze direction labels of the objects in the first sample image and the obtained gaze direction prediction labels, alignment parameters can be obtained to adjust the gaze direction labels of the objects in the first sample image to match the gaze direction labels under the aforementioned target calibration method.

[0211] Furthermore, based on the alignment parameters, the gaze direction labels of objects in the second sample image under the aforementioned target calibration method can be obtained. This ensures that the gaze direction labels of objects in the second sample image within the image set to be aligned and the gaze direction labels of objects in the sample images within the reference image set are both labels under the target calibration method. This aligns the gaze direction labels of objects in the second sample image within the image set to be aligned with the gaze direction labels of the reference image set, thereby unifying the calibration errors corresponding to the image set to be aligned and the reference image set. This reduces the differences in calibration errors between sample image sets and improves the self-consistency of each sample image set. Consequently, when training the model based on the image set to be aligned and the reference image set, the degree of mutual influence between inferences for each sample image set within the model is reduced, which is beneficial to improving model performance and thus improving the accuracy of the trained model in determining the gaze direction of objects in the image.

[0212] In one embodiment of this application, the alignment parameter determination module 802 includes:

[0213] The alignment label acquisition submodule is used to obtain the gaze direction alignment label represented by the alignment parameter, wherein the gaze direction alignment label represents the gaze direction label of the object in the first sample image after the gaze direction label is aligned from the sample image set to be aligned to the reference sample image set;

[0214] The alignment parameter determination submodule is used to determine the alignment parameter that minimizes the total difference between the gaze direction alignment label and the gaze direction prediction label corresponding to the first sample image.

[0215] As can be seen from the above, after obtaining the gaze direction alignment label represented by the alignment parameter, the alignment parameter that minimizes the total difference between the gaze direction alignment label corresponding to the first sample image and the gaze direction prediction label can be determined. Since the gaze direction prediction label is the label of the object in the predicted first sample image under the target calibration method corresponding to the reference sample image set, after adjusting the alignment of the gaze direction label of the object in the second sample image within the sample image set to be aligned based on the above alignment parameter, the adjusted gaze direction label can be made as close as possible to the gaze direction label of the predicted object in the second sample image under the target calibration method, thus improving the alignment effect of the gaze direction label.

[0216] In one embodiment of this application, the alignment parameter determination submodule is specifically used to obtain, for each first sample image, the difference representation value between the gaze direction alignment label and the gaze direction prediction label corresponding to the first sample image; determine the total difference representation value based on the obtained difference representation value; and determine the alignment parameter that minimizes the total difference representation value.

[0217] As can be seen, for each first sample image, the difference representation value between the gaze direction alignment label and the gaze direction prediction label corresponding to the first sample image is obtained. Thus, the total difference representation value determined based on the obtained difference representation value can more accurately reflect the difference between the gaze direction alignment label and the gaze direction prediction label corresponding to the first sample image. In turn, a more accurate alignment parameter can be obtained based on the above total difference representation value.

[0218] In one embodiment of this application, the difference characterization value is: the absolute error between the gaze direction alignment label and the gaze direction prediction label corresponding to the first sample image; or the difference characterization value is: the angular error between the gaze direction alignment label and the gaze direction prediction label corresponding to the first sample image.

[0219] The aforementioned absolute error or angular error can accurately characterize the difference between the gaze direction alignment label and the gaze direction prediction label corresponding to the first sample image, thereby obtaining a more accurate total difference characterization value based on the obtained difference characterization value.

[0220] In one embodiment of this application, the alignment label obtaining submodule includes:

[0221] The alignment landing point acquisition unit is used to obtain the alignment landing point corresponding to the first line of sight landing point after the line of sight direction label alignment is performed from the sample image set to be aligned to the reference sample image set, represented by the alignment parameter. The first line of sight landing point is: the line of sight landing point used to determine the line of sight direction label of the object in the first sample image.

[0222] The alignment label obtaining unit is used to determine the gaze direction alignment label of the object in the first sample image based on the gaze starting point used to determine the gaze direction label of the object in the first sample image and the obtained alignment landing point.

[0223] As can be seen from the above, since the starting points of the gaze of objects in each sample image set are naturally aligned, while the ending points of the gaze of objects in different sample image sets are often not aligned, we can first obtain the alignment point corresponding to the first gaze ending point after aligning the gaze direction labels from the sample image set to be aligned to the reference sample image set, expressed by alignment parameters. Based on the starting point of the gaze direction label used to determine the gaze direction label of the object in the first sample image and the obtained alignment ending point, we can more reasonably and accurately determine the gaze direction alignment label of the object in the first sample image after aligning the gaze direction labels from the sample image set to be aligned to the reference sample image set.

[0224] In one embodiment of this application, the alignment parameters include: rotation parameters and translation parameters;

[0225] The alignment point acquisition unit is specifically used to obtain, according to the following expression, the alignment point P′ corresponding to the first line-of-sight point after line-of-sight label alignment from the sample image set to be aligned to the reference sample image set, expressed by the rotation parameters and translation parameters. end :

[0226] P end =R·P end +T

[0227] Where R represents the rotation parameter, T represents the translation parameter, and P end This indicates the first point of view.

[0228] As can be seen, when the alignment parameters include rotation and translation parameters, the alignment point can be accurately and quickly represented according to the above expression.

[0229] In one embodiment of this application, the first sample image is a portion of the sample image set to be aligned, and the second sample image is a portion of the sample image set to be aligned; or the first sample image is a portion of the sample image set to be aligned, and the second sample image is all the images in the sample image set to be aligned; or the first sample image is all the images in the sample image set to be aligned, and the second sample image is all the images in the sample image set to be aligned.

[0230] As can be seen, there are multiple ways to determine the first sample image used to determine the alignment parameters and the second sample image to be aligned with the visual orientation labels within the image set to be aligned. This allows for more flexible determination of the alignment parameters and the alignment of visual orientation labels based on the determined alignment parameters, thereby improving the flexibility and applicability of the solution.

[0231] Corresponding to the above-described model training method, this application also provides a model training apparatus.

[0232] See Figure 9 The above is a schematic diagram of a model training device provided in an embodiment of this application. The device includes the following modules:

[0233] The sample image set acquisition module 901 is used to acquire a baseline sample image set and a sample image set to be aligned;

[0234] The second label alignment module 902 is used to label the gaze direction of objects in the sample images within the sample image set to be aligned based on the gaze direction label alignment method, and to perform gaze direction label alignment from the sample image set to be aligned to the reference sample image set to obtain the aligned sample image set. The gaze direction label alignment method is any of the aforementioned gaze direction label alignment methods.

[0235] The model training module 903 is used to train the preset model using the reference sample image set and the aligned sample image set to obtain a gaze direction determination model for determining the gaze direction of objects in the image.

[0236] As can be seen from the above, when using the scheme provided in this application for model training, the gaze direction labels of objects in the sample images within the sample image set to be aligned are first aligned from the sample image set to the reference sample image set based on the gaze direction label alignment method, resulting in an aligned sample image set. This adjusts the gaze direction labels of objects in the sample images within the sample image set to be aligned to the labels under the target calibration method corresponding to the reference sample image set. As a result, the calibration errors corresponding to the sample image set to be aligned and the reference sample image set become unified, reducing the differences between the calibration errors of the sample image sets and improving the self-consistency of each sample image set. Consequently, when training the model based on the sample image set to be aligned and the reference sample image set, the degree of mutual influence between the inferences of each sample image set within the model is reduced, which is beneficial to improving model performance and thus improving the accuracy of the trained model in determining the gaze direction of objects in the image.

[0237] Corresponding to the above-described method for determining the direction of sight, this application also provides a device for determining the direction of sight.

[0238] See Figure 10 This is a schematic diagram of a line-of-sight direction determination device provided in an embodiment of this application. The device includes the following modules:

[0239] The target image acquisition module 1001 is used to acquire the target image;

[0240] The gaze direction determination module 1002 is used to input the target image into a pre-trained gaze direction determination model to obtain the gaze direction of the object in the target image output by the gaze direction determination model. The gaze direction determination model is a model trained according to the aforementioned model training method for determining the gaze direction of the object in the image.

[0241] As can be seen from the above, when using the scheme provided in the embodiments of this application to determine the gaze direction, after obtaining the target image, the target image can be input into the pre-trained gaze direction determination model to obtain the gaze direction of the object in the target image output by the gaze direction determination model.

[0242] As can be seen from the aforementioned model training method embodiment, before training the above-mentioned gaze direction model, the gaze direction labels of objects in the sample images within the sample image set to be aligned are aligned from the sample image set to the reference sample image set, resulting in an aligned sample image set. This adjusts the gaze direction labels of objects in the sample images within the sample image set to be aligned to the labels under the target calibration method corresponding to the reference sample image set. As a result, the calibration errors corresponding to the sample image set to be aligned and the reference sample image set become unified, reducing the differences between the calibration errors of the sample image sets and improving the self-consistency of each sample image set. Consequently, when training the model based on the sample image set to be aligned and the reference sample image set, the degree of mutual influence between the inferences of each sample image set within the model is reduced, which is beneficial to improving model performance. This results in a gaze direction determination model with higher accuracy, and based on the above model, the accuracy of recognizing the gaze direction of objects in images can be improved.

[0243] In the solutions provided in this application, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information are all carried out with the user's knowledge and authorization, and comply with relevant laws and regulations, and do not violate public order and good morals.

[0244] It should be noted that the two-dimensional face images in this embodiment are from a publicly available dataset.

[0245] This application also provides an electronic device, such as... Figure 11 As shown, it includes:

[0246] Memory 1101 is used to store computer programs;

[0247] The processor 1102, when executing the program stored in the memory 1101, implements the aforementioned gaze direction label alignment method, model training method, or gaze direction determination method.

[0248] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 1102, the communication interface, and the memory 1101 communicating with each other via the communication bus.

[0249] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0250] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0251] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0252] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0253] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements any of the above-described gaze direction label alignment methods, model training methods, or gaze direction determination methods.

[0254] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the gaze direction label alignment methods, model training methods, or gaze direction determination methods in the above embodiments.

[0255] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or other storage media (e.g., solid state disk (SSD)).

[0256] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0257] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, and storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0258] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A line-of-sight label alignment method, characterized by, The method comprises: obtaining a gaze direction prediction label of an object in a first sample image in a set of sample images to be aligned based on a pre-trained label prediction model, wherein the label prediction model is a model for predicting a gaze direction label of an object in an image obtained by training a preset model based on a set of reference sample images, the set of reference sample images is a set of sample images corresponding to a calibration mode with a calibration error less than a preset threshold in a set of all sample images for training the gaze direction determination model, and the set of sample images to be aligned is a set of sample images other than the set of reference sample images in the set of all sample images; determining alignment parameters for aligning the gaze direction labels from the set of sample images to be aligned to the set of reference sample images based on the gaze direction annotation label of the object in the first sample image and the obtained gaze direction prediction label; adjusting the gaze direction annotation label of the object in a second sample image in the set of sample images to be aligned based on the alignment parameters.

2. The method of claim 1, wherein, The method comprises: obtaining a gaze direction alignment label represented by the alignment parameters, wherein the gaze direction alignment label represents the gaze direction label of the object in the first sample image after aligning the gaze direction labels from the set of sample images to be aligned to the set of reference sample images; determining the alignment parameters that minimize the total difference between the gaze direction alignment label and the gaze direction prediction label corresponding to the first sample image.

3. The method of claim 2, wherein, The method comprises: for each first sample image, obtaining a difference representation value between the gaze direction alignment label and the gaze direction prediction label corresponding to the first sample image; determining a total difference representation value based on the obtained difference representation values; determining the alignment parameters that minimize the total difference representation value.

4. The method of claim 3, wherein: the difference representation value is an absolute error between the gaze direction alignment label and the gaze direction prediction label corresponding to the first sample image; or the difference representation value is an angle error between the gaze direction alignment label and the gaze direction prediction label corresponding to the first sample image.

5. The method of claim 4, wherein: the method comprises: said absolute error is obtained according to the following expression: and / or the angle error is obtained according to the following expression: wherein, represents a gaze direction prediction label corresponding to the first sample image, represents a gaze direction alignment label corresponding to the first sample image.

6. The method of claim 2, wherein, obtaining an alignment landing point corresponding to a first gaze landing point after aligning the gaze direction labels from the set of sample images to be aligned to the set of reference sample images, wherein the first gaze landing point is a gaze landing point used to determine the gaze direction annotation label of the object in the first sample image; determining the gaze direction alignment label of the object in the first sample image based on a gaze starting point used to determine the gaze direction annotation label of the object in the first sample image and the obtained alignment landing point. ​ 7. The method of claim 6, wherein, The alignment parameters comprise a rotation parameter and a translation parameter. The obtaining of the alignment landing point corresponding to the first line-of-sight landing point of the set of reference sample images after the line-of-sight direction label alignment from the set of sample images to be aligned to the set of reference sample images by using the alignment parameters comprises: An aligned landing point corresponding to a first line-of-sight landing point after line-of-sight direction label alignment from the set of to-be-aligned sample images to the set of reference sample images is obtained using the rotation parameter and the translation parameter according to the following expression : wherein, denotes the rotation parameters, T denotes the translation parameters, denotes the first line of sight impact point.

8. The method of any one of claims 1-7, characterized in that, the first sample image is a part of the set of sample images to be aligned, and the second sample image is a part of the set of sample images to be aligned; or the first sample image is a part of the set of sample images to be aligned, and the second sample image is all of the set of sample images to be aligned; or the first sample image is all of the set of sample images to be aligned, and the second sample image is all of the set of sample images to be aligned.

9. A model training method, comprising: The method comprises: obtaining a set of reference sample images and a set of sample images to be aligned, wherein the set of reference sample images is a set of sample images corresponding to a calibration mode with a calibration error less than a preset threshold in a set of all sample images used for training a line-of-sight direction determination model, and the set of sample images to be aligned is a set of sample images other than the set of reference sample images in the set of all sample images; aligning a line-of-sight direction label of an object in a sample image in the set of sample images to be aligned to the set of reference sample images based on a line-of-sight direction label alignment method to obtain an aligned set of sample images, wherein the line-of-sight direction label alignment method is the method of any one of claims 1-7; training a preset model using the set of reference sample images and the aligned set of sample images to obtain a line-of-sight direction determination model for determining a line-of-sight direction of an object in an image.

10. A line-of-sight direction determination method characterized by, The method comprises: obtaining a target image; inputting the target image into a pre-trained line-of-sight direction determination model to obtain a line-of-sight direction of an object in the target image output by the line-of-sight direction determination model, wherein the line-of-sight direction determination model is a model for determining a line-of-sight direction of an object in an image trained according to claim 9.

11. A line-of-sight label alignment device, characterized by, The device comprises: a predicted label obtaining module configured to obtain a line-of-sight direction predicted label of an object in a first sample image in a set of sample images to be aligned based on a pre-trained label prediction model, wherein the label prediction model is a model for predicting a line-of-sight direction label of an object in an image trained by training a preset model based on a set of reference sample images, the set of reference sample images is a set of sample images corresponding to a calibration mode with a calibration error less than a preset threshold in a set of all sample images used for training a line-of-sight direction determination model, and the set of sample images to be aligned is a set of sample images other than the set of reference sample images in the set of all sample images; an alignment parameter determining module configured to determine alignment parameters for line-of-sight direction label alignment from the set of sample images to be aligned to the set of reference sample images based on a line-of-sight direction labeled label of an object in the first sample image and the obtained line-of-sight direction predicted label. The first label alignment module is configured to align and adjust the line-of-sight direction annotation label of the object in the second sample image in the set of sample images to be aligned based on the alignment parameter.

12. The apparatus of claim 11, wherein, The alignment parameter determination module comprises: an alignment label obtaining sub-module configured to obtain a line-of-sight direction alignment label represented by an alignment parameter, wherein the line-of-sight direction alignment label represents a line-of-sight direction label of the object in the first sample image after the line-of-sight direction label alignment from the set of sample images to be aligned to the set of reference sample images; and an alignment parameter determination sub-module configured to determine the alignment parameter that minimizes the total difference between the line-of-sight direction alignment label corresponding to the first sample image and the line-of-sight direction prediction label. Or The alignment parameter determination sub-module is specifically configured to, for each first sample image, obtain a difference representation value between the line-of-sight direction alignment label corresponding to the first sample image and the line-of-sight direction prediction label; determine a total difference representation value based on the obtained difference representation value; and determine the alignment parameter that minimizes the total difference representation value. Or The difference representation value is an absolute error between the line-of-sight direction alignment label corresponding to the first sample image and the line-of-sight direction prediction label; or the difference representation value is an angle error between the line-of-sight direction alignment label corresponding to the first sample image and the line-of-sight direction prediction label. Or the absolute error is obtained according to the following expression: ; and / or the angular error is obtained according to the following expression: wherein, denotes a gaze direction prediction label corresponding to the first sample image, denotes a gaze direction alignment label corresponding to the first sample image; The alignment label obtaining sub-module comprises: an alignment landing point obtaining unit configured to obtain an alignment landing point corresponding to a first line-of-sight landing point after the line-of-sight direction label alignment from the set of sample images to be aligned to the set of reference sample images, the line-of-sight landing point being used to determine the line-of-sight direction annotation label of the object in the first sample image; and an alignment label obtaining unit configured to determine the line-of-sight direction alignment label of the object in the first sample image based on a line-of-sight starting point used to determine the line-of-sight direction annotation label of the object in the first sample image and the obtained alignment landing point. Or The first sample image is a part of the set of sample images to be aligned, and the second sample image is a part of the set of sample images to be aligned; or the first sample image is a part of the set of sample images to be aligned, and the second sample image is all of the set of sample images to be aligned; or the first sample image is all of the set of sample images to be aligned, and the second sample image is all of the set of sample images to be aligned. The alignment parameters comprise a rotation parameter and a translation parameter; the alignment landing point obtaining unit is specifically configured to obtain an alignment landing point corresponding to a first line-of-sight landing point after performing line-of-sight direction label alignment from the set of to-be-aligned sample images to the set of reference sample images according to the following expression : , wherein, represents the rotation parameter, and T represents the translation parameter, represents the first line-of-sight landing point; The apparatus comprises: A sample image set obtaining module configured to obtain a set of reference sample images and a set of sample images to be aligned, the set of reference sample images being a set of sample images corresponding to a calibration mode with a calibration error less than a preset threshold in a set of all sample images used for training a line-of-sight direction determination model, and the set of sample images to be aligned being a set of sample images other than the set of reference sample images in the set of all sample images.

13. A model training apparatus, comprising: ​ ​ a second label alignment module configured to perform gaze direction label alignment on the reference sample image set based on a gaze direction label alignment method, to obtain an aligned sample image set, wherein the gaze direction label alignment method is any one of the methods in claims 1-8; a model training module configured to perform model training on a preset model by using the reference sample image set and the aligned sample image set, to obtain a gaze direction determination model for determining a gaze direction of an object in an image.

14. A line-of-sight direction determination apparatus characterized by comprising: The apparatus comprises: a target image obtaining module configured to obtain a target image; a gaze direction determination module configured to input the target image into a pre-trained gaze direction determination model, to obtain a gaze direction of an object in the target image output by the gaze direction determination model, wherein the gaze direction determination model is a model for determining a gaze direction of an object in an image, which is trained according to claim 9.

15. An electronic device, comprising: comprise: a memory configured to store a computer program; a processor configured to implement the method in any one of claims 1-8 or 9 or 10 when executing the program stored in the memory.

16. A computer-readable storage medium, characterized in that, The computer program stored in the computer readable storage medium is configured to implement the method in any one of claims 1-8 or 9 or 10 when executed by the processor.

Images