Gaze point detection model training method, gaze point detection method and device

Abstract

The gaze point detection model training method, the gaze point detection method and the device provided by the application, the gaze point detection model training method comprises: in the training process, with the increase of the training round, the dependence of the updated target model on the auxiliary feature is smaller and smaller, at the end of the training, the dependence on the auxiliary feature is removed, and the latest target model is determined as the gaze point detection model, while ensuring rich training data of the gaze point detection model to improve the precision of the gaze point detection model, the input of the gaze point detection model can be simplified, since the dependence on the auxiliary feature is removed, the size of the gaze point detection model can be reduced, and the deployability of the gaze point detection model can be enhanced; when the gaze point detection model generated by the method is applied to the gaze point detection method, only the data corresponding to the target feature except the auxiliary feature needs to be input, and the coordinate value of the relatively accurate line of sight direction can be obtained, thereby improving the accuracy of the gaze point detection method.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, in particular to a gaze point detection model training method, a gaze point detection method and device. BACKGROUND

[0002] Gaze point detection refers to a process of detecting the direction of a person's eyesight in an image or video by using computer vision technology and artificial intelligence algorithms. Gaze point detection has a wide range of applications, such as in user experience design, where it can help understand what users pay most attention to when viewing a page, and improve page layout and content presentation.

[0003] Currently, gaze point detection methods can be generally divided into geometry-based methods and appearance-based methods. Geometry-based methods have more accurate detection results, but they require high image quality and equipment, and are not easy to deploy. Although appearance-based methods have lower requirements for image quality, their detection results are less accurate than those of geometry-based methods, making it difficult to reduce the size of gaze point detection models while ensuring accuracy and enhancing the deployability of gaze point detection models. SUMMARY

[0004] The present application aims to at least solve one of the above technical defects, particularly the technical defect that it is difficult to reduce the size of gaze point detection models while ensuring accuracy and enhancing the deployability of gaze point detection models in the prior art.

[0005] In a first aspect, the present application provides a gaze point detection model training method, comprising:

[0006] obtaining target data corresponding to a plurality of preset dimensions;

[0007] In one training round, the target data corresponding to each preset dimension is divided into a plurality of data groups, and a target data group is determined from each data group corresponding to each preset dimension; wherein the target data group is any data group that does not perform feature fusion in the current training round;

[0008] Based on the target data group corresponding to each preset dimension, the target feature corresponding to each preset dimension is determined;

[0009] determining the weight of the auxiliary feature and the target feature other than the auxiliary feature in the target feature corresponding to each preset dimension; wherein the auxiliary feature is the target feature corresponding to a target preset dimension pre-determined in the plurality of preset dimensions, and the weight of the auxiliary feature is inversely proportional to the current training round;

[0010] According to the weight of the target feature corresponding to each preset dimension, the target features corresponding to each preset dimension are weighted and fused to obtain a fused feature;

[0011] According to the fused feature, a target model is iteratively trained to obtain a new target model, the target data groups corresponding to each preset dimension and the target features are re-determined, the target features corresponding to each preset dimension are weighted and fused according to the weight of the target features corresponding to each preset dimension, a new fused feature is obtained, and the new target model is iteratively trained until the re-determined target data groups corresponding to each preset dimension are all empty and the current training round is not less than a preset threshold, and the latest target model is determined as the gaze point detection model;

[0012] If the re-determined target data groups corresponding to each preset dimension are all empty and the current training round is less than the preset threshold, a next training round is entered.

[0013] In one of the embodiments, the method further comprises:

[0014] According to the target data groups corresponding to each preset dimension, the target features corresponding to each preset dimension are determined respectively, including:

[0015] In one of the embodiments, after the step of iteratively training the target model according to the fused feature to obtain a new target model is performed, the method further comprises:

[0016] According to the new target model, the preset models corresponding to each preset dimension are updated.

[0017] In one of the embodiments, the method further comprises:

[0018] The weight of the auxiliary feature and the weight of the target features other than the auxiliary feature in the target features corresponding to each preset dimension are determined, including:

[0019] The current training round is substituted into a preset feature fusion formula to obtain the weight of the auxiliary feature, and according to the feature fusion formula, the weight of the target features other than the auxiliary feature is set to 1, so as to complete the determination of the weight of the target features corresponding to each preset dimension.

[0020] In a second aspect, the application provides a gaze point detection method, which comprises:

[0021] The data to be detected is input into the gaze point detection model to obtain the coordinate values ​​of the gaze direction corresponding to the data to be detected; wherein, the gaze point detection model is generated using the gaze point detection model training method described in any of the above embodiments;

[0022] Based on a monocular camera, the distance to the object being gazed upon is obtained; wherein, the distance to the object being gazed upon is the distance between the gazer and the object being gazed upon.

[0023] The coordinates of the distance to the object being gazed upon and the direction of the gaze are input into a preset gaze point estimation module to obtain the coordinates of the gaze point corresponding to the data to be detected.

[0024] In one embodiment, the step of inputting the coordinate values ​​of the gaze distance and the gaze direction to a preset gaze point estimation module to obtain the coordinate values ​​of the gaze point corresponding to the data to be detected includes:

[0025] Obtain the camera intrinsic parameter matrix of the monocular camera;

[0026] Based on the camera intrinsic parameter matrix and the coordinate values ​​of the line of sight, calculate the vector from the camera optical center of the monocular camera to the center of the eyeball and the vector from the center of the eyeball to the object being gazed upon.

[0027] The vector from the camera optical center of the monocular camera to the center of the eyeball and the vector from the center of the eyeball to the object being gazed are transformed into a coordinate system to obtain the vector from the camera optical center to the object being gazed.

[0028] The spatial coordinates of the gaze point corresponding to the object being gazed upon are calculated using triangulation.

[0029] Thirdly, this application provides a gaze point detection model training device, the device comprising:

[0030] The target data acquisition module is used to acquire target data corresponding to multiple preset dimensions.

[0031] The target data group determination module is used to divide the target data corresponding to each preset dimension into multiple data groups in a training round, and determine the target data group from each data group corresponding to each preset dimension; wherein, the target data group is any data group that has not undergone feature fusion in the current training round.

[0032] The target feature determination module is used to determine the target features corresponding to each preset dimension based on the target data group corresponding to each preset dimension.

[0033] The weight determination module is used to determine the weights of auxiliary features and target features other than the auxiliary features in the target features corresponding to each preset dimension; wherein, the auxiliary features are target features corresponding to the target preset dimensions that are predetermined in each preset dimension, and the weight of the auxiliary features is inversely proportional to the current training round.

[0034] The feature fusion module is used to perform weighted fusion of the target features corresponding to each preset dimension according to the weights of the target features corresponding to each preset dimension, so as to obtain the fused features;

[0035] The first iterative training module is used to iteratively train the target model according to the fusion features to obtain a new target model, redetermine the target data groups and target features corresponding to each preset dimension, and perform weighted fusion of the target features corresponding to each preset dimension according to the weights of the target features corresponding to each preset dimension to obtain new fusion features, so as to iteratively train the new target model until the target data groups corresponding to each preset dimension are all empty and the current training round is not less than a preset threshold, and the latest target model is determined as the fixation detection model.

[0036] The second iterative training module is used to enter the next training round if the target data groups corresponding to each preset dimension that have been redefined are all empty and the current training round is less than the preset threshold.

[0037] Fourthly, this application provides a gaze point detection device, the device comprising:

[0038] The data acquisition module is used to acquire the data to be detected; wherein, the data to be detected is the data corresponding to the target features excluding auxiliary features;

[0039] The first coordinate value determination module is used to input the data to be detected into the gaze point detection model to obtain the coordinate values ​​of the gaze direction corresponding to the data to be detected; wherein, the gaze point detection model is generated using the gaze point detection model training method described in any of the above embodiments;

[0040] A distance determination module is used to obtain the distance to the object being gazed upon based on a monocular camera; wherein, the distance to the object being gazed upon is the distance between the gazer and the object being gazed upon;

[0041] The second coordinate value determination module is used to input the coordinate values ​​of the distance to the object being gazed upon and the direction of the gaze to a preset gaze point estimation module to obtain the coordinate values ​​of the gaze point corresponding to the data to be detected.

[0042] Fifthly, this application provides a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the gaze point detection model training method as described in any of the foregoing embodiments, and / or to perform the steps of the gaze point detection method as described in any of the foregoing embodiments.

[0043] Sixthly, this application provides a computer device, including: one or more processors, and a memory;

[0044] The memory stores computer-readable instructions, which, when executed by the one or more processors, perform the steps of the gaze point detection model training method as described in any of the above embodiments, and / or perform the steps of the gaze point detection method as described in any of the above embodiments.

[0045] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0046] The gaze point detection model training method, gaze point detection method, and apparatus provided in this application include: During training, a target model is trained using multi-dimensional data. Since auxiliary features are inversely proportional to the current training epoch, the updated target model's dependence on auxiliary features decreases with each training epoch. At the end of training, the dependence on auxiliary features is removed, and the latest target model is determined as the gaze point detection model. This approach ensures rich training data for the gaze point detection model to improve its accuracy while simplifying the input. Furthermore, removing the dependence on auxiliary features reduces the size of the gaze point detection model and enhances its deployability. When the gaze point detection model generated by this method is applied to the gaze point detection method, only data corresponding to the target features other than auxiliary features needs to be input to obtain relatively accurate coordinate values ​​of the gaze direction, thereby improving the accuracy of the gaze point detection method. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 A flowchart illustrating a gaze point detection model training method provided in an embodiment of this application;

[0049] Figure 2 A flowchart illustrating the process of determining the weights of target features corresponding to each preset dimension, provided in an embodiment of this application.

[0050] Figure 3 A schematic flowchart illustrating a gaze point detection method provided in an embodiment of this application;

[0051] Figure 4 A flowchart illustrating the process of determining the coordinate values ​​of the gaze point corresponding to the data to be detected, as provided in an embodiment of this application.

[0052] Figure 5 This is a schematic diagram of the structure of a gaze point detection model training device provided in an embodiment of this application;

[0053] Figure 6 This is a schematic diagram of the structure of a gaze point detection device provided in an embodiment of this application;

[0054] Figure 7 This is an internal structural diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0055] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] like Figure 1 As shown, in one embodiment, this application provides a method for training a gaze detection model. The following embodiments illustrate the application of this method to a server. It is understood that the gaze detection model training method can be performed on a single server or a server cluster consisting of multiple servers; this application does not impose specific limitations on this. The method includes:

[0057] Step S101: Obtain target data corresponding to multiple preset dimensions.

[0058] The target data corresponding to multiple preset dimensions can be understood as target data corresponding to different aspects. Among them, the preset dimensions include, but are not limited to, face image aspects, binocular image aspects, facial key points aspects, etc.

[0059] In this step, when the server receives a model training instruction, it can obtain the training data corresponding to the model training instruction. The training data includes target data corresponding to multiple preset dimensions.

[0060] Furthermore, acquiring target data corresponding to multiple preset dimensions can enrich the training data of the gaze detection model, thereby improving the accuracy of the gaze detection model.

[0061] Step S102: In a training round, the target data corresponding to each preset dimension is divided into multiple data groups, and the target data group is determined from each data group corresponding to each preset dimension.

[0062] The target data group is any data group that has not undergone feature fusion in the current training round. Feature fusion refers to generating new fused features using multiple existing feature sets. It can be understood that the advantage of feature fusion is that it can obtain the most differentiated information from multiple existing feature sets, thereby eliminating redundant information caused by the correlation between different feature sets.

[0063] In this step, when the server obtains target data corresponding to multiple preset dimensions, it will divide the target data corresponding to each preset dimension into multiple data groups. The strategy for dividing the target data corresponding to each preset dimension can be based on the storage order or a pre-defined division strategy. This application does not impose any specific restrictions on this.

[0064] Furthermore, the purpose of dividing the target data corresponding to each preset dimension into multiple data groups is to reduce the amount of data for a single feature extraction, so as to avoid the model used for feature extraction having to extract features from a large amount of data at once, which could lead to model crashes or runtime errors.

[0065] For example, suppose there are preset dimensions A, B, and C, corresponding to target data a, target data b, and target data c, respectively. Target data a is divided into data group a1 and data group a2, target data b is divided into data group b1 and data group b2, and target data c is divided into data group c1 and data group c2. In this case, the target data group corresponding to preset dimension A is data group a1, the target data group corresponding to preset dimension B is data group b2, and the target data group corresponding to preset dimension C is data group c1. Then, feature extraction is performed on data group a1, data group b2, and data group c1.

[0066] Step S103: Determine the target features corresponding to each preset dimension based on the target data group corresponding to each preset dimension.

[0067] In this step, when the server completes the partitioning of the target data corresponding to each preset dimension, for any target data group corresponding to a preset dimension, feature extraction is performed on the target data group to determine the target features corresponding to the preset dimension. Feature extraction can extract useful features from the target data group, that is, transform the target data group into representative features, so that the subsequent processing model can understand and process the target data group.

[0068] Furthermore, methods for feature extraction from target datasets include, but are not limited to, statistical methods, dimensionality reduction methods, model methods, feature selection methods, and so on.

[0069] Understandably, if feature extraction of a target data set corresponding to a certain preset dimension is difficult, a dilated convolution module can be added to expand the receptive field when extracting the target data set corresponding to that preset dimension, thereby improving the feature extraction capability of the target data set corresponding to that preset dimension. For example, if the target data set corresponding to a certain preset dimension is the target data set corresponding to the face, since the human eye accounts for a small proportion of the face, feature extraction is difficult. Therefore, a dilated convolution module can be added to extract features from the target data set corresponding to the face.

[0070] Step S104: Determine the auxiliary features and the weights of the target features other than the auxiliary features in each preset dimension.

[0071] Among them, the auxiliary features are the target features corresponding to the target preset dimensions that are predetermined in each preset dimension, and the weight of the auxiliary features is inversely proportional to the current training round.

[0072] It is understandable that the weight of auxiliary features is inversely proportional to the current training epoch. That is, as the training epoch increases, the weight of auxiliary features decreases. For the target preset dimension, the auxiliary features corresponding to the target preset dimension can be selected from multiple preset dimensions based on the experience value of the R&D personnel or the needs of the trained model. The number of selected auxiliary features is generally less than the number of preset dimensions, but not less than 1.

[0073] In this step, in order to perform subsequent feature fusion, it is necessary to determine the weight of the target features corresponding to each preset dimension and assign corresponding weights to the target features corresponding to each preset dimension, that is, feature weighting, so that the target features of each preset dimension can be weighted and fused during feature fusion.

[0074] Step S105: Based on the weights of the target features corresponding to each preset dimension, perform weighted fusion on the target features corresponding to each preset dimension to obtain fused features.

[0075] In this step, once the server determines the weights of the target features corresponding to each preset dimension, it can perform weighted fusion of the target features corresponding to each preset dimension to obtain the corresponding fused features.

[0076] It is understandable that weighted fusion of target features corresponding to each preset dimension can improve the accuracy and performance of the final generated gaze detection model.

[0077] Step S106: Iteratively train the target model based on the fusion features to obtain a new target model. Redetermine the target data groups and target features corresponding to each preset dimension. Based on the weights of the target features corresponding to each preset dimension, perform weighted fusion on the target features corresponding to each preset dimension to obtain new fusion features. Iteratively train the new target model until the target data groups corresponding to each preset dimension are all empty.

[0078] In this step, if the target data sets corresponding to the newly determined preset dimensions are not all empty, the target model will be iteratively trained based on the newly obtained data until the target data sets corresponding to the newly determined preset dimensions are empty.

[0079] It is understandable that when the target data groups corresponding to each preset dimension are all empty, it means that the multiple data groups corresponding to each preset dimension have been fused. At this time, it is possible to determine whether to enter the next training round by judging whether the current training round is less than the preset threshold.

[0080] Step S107: Determine whether the current training round is less than the preset threshold.

[0081] The preset threshold is an empirical value.

[0082] Step S108: When the current training round is less than the preset threshold, proceed to the next training round.

[0083] In this step, when the target data groups corresponding to each preset dimension are all empty and the current training round is less than the preset threshold, the next training round can be entered. After entering the next round, the target data corresponding to each preset dimension will be re-divided, the new target data groups and new target features corresponding to each preset dimension will be determined, and the features will be weighted and fused according to the weights of the target features corresponding to each preset dimension to obtain new fused features for iterative training of the target model.

[0084] Step S109: When the current training round is not less than the preset threshold, the latest target model is determined as the fixation detection model.

[0085] In the steps, if the target data groups corresponding to each preset dimension are all empty and the current training round is not less than the preset threshold, it means that the target model has completed all training rounds. At this time, the latest target model is determined as the fove detection model.

[0086] The gaze point detection model training method, gaze point detection method, and apparatus provided in this application include: During training, a target model is trained using multi-dimensional data. Since auxiliary features are inversely proportional to the current training epoch, the updated target model's dependence on auxiliary features decreases with each training epoch. At the end of training, the dependence on auxiliary features is removed, and the latest target model is determined as the gaze point detection model. This approach ensures rich training data for the gaze point detection model to improve its accuracy while simplifying the input. Furthermore, removing the dependence on auxiliary features reduces the size of the gaze point detection model and enhances its deployability. When the gaze point detection model generated by this method is applied to the gaze point detection method, only data corresponding to the target features other than auxiliary features needs to be input to obtain relatively accurate coordinate values ​​of the gaze direction, thereby improving the accuracy of the gaze point detection method.

[0087] In one embodiment, the target features corresponding to each preset dimension are determined based on the target data group corresponding to each preset dimension, including:

[0088] Based on the preset models corresponding to each preset dimension, feature extraction is performed on any one of the target data groups corresponding to each preset dimension to obtain the target features corresponding to each preset dimension.

[0089] The preset model is a pre-trained feature extraction model corresponding to a preset dimension, which can be used to extract features from the target data group corresponding to the preset dimension.

[0090] In another embodiment, the data groups for feature extraction in the target data groups of each preset dimension can be determined according to preset data group determination rules.

[0091] It is understandable that feature extraction from target data sets of various preset dimensions can yield representative features, which is beneficial to improving the accuracy of the fixation detection model.

[0092] In one embodiment, after performing iterative training of the target model based on the fused features to obtain a new target model, the method further includes:

[0093] Based on the new target model, update the preset models corresponding to each preset dimension.

[0094] In this embodiment, the step of updating the preset model corresponding to each preset dimension based on the new target model may include:

[0095] Based on the new target model, the corresponding loss function is calculated, and the backpropagation gradient is determined based on the loss function. An optimization algorithm is then used to update the parameters of the preset model corresponding to each preset dimension, thereby achieving the update of the preset model corresponding to each preset dimension.

[0096] Furthermore, optimization algorithms include, but are not limited to, stochastic gradient descent, gradient descent, momentum method, adaptive learning rate method, conjugate gradient method, and so on.

[0097] It is understandable that after updating each preset model, there may be some differences between the preset models used for feature extraction each time. Therefore, the features extracted for the same data may be the same or different. Using a loss function to update each preset model can improve the feature extraction capability of each preset model and ultimately improve the accuracy of the fixation detection model.

[0098] like Figure 2 As shown, in one embodiment, determining the auxiliary features and the weights of the target features other than the auxiliary features in each preset dimension includes:

[0099] Step S201: Substitute the current training round into the preset feature fusion formula to obtain the weights of the auxiliary features.

[0100] Step S202: According to the feature fusion formula, set the weight of the target features other than the auxiliary features to 1 to complete the determination of the weight of the target features corresponding to each preset dimension.

[0101] In this embodiment, the expression for the feature fusion formula can be represented as follows:

[0102] feature out =feature1 + ... + feature n +(1-λ)×feature n+1 +…

[0103] +(1-λ)×feature m

[0104] λ = epoch / epochs

[0105] In the formula, m≥n+1, n≥1, feature out The fused features are output after feature fusion, from feature1 to feature2. nFor target features other than auxiliary features, feature n+1 To feature m is the auxiliary feature, (1-λ) is the weight of the auxiliary feature, epoch is the current training epoch, and epochs is the total number of training epochs, i.e. the preset threshold.

[0106] As can be seen from the above expression, based on the feature fusion formula and the current training round, the weights of the auxiliary features and the target features other than the auxiliary features can be obtained, i.e., 1. In other words, based on the feature fusion formula and the current training round, the weights of the target features corresponding to each preset dimension can be determined, and then the target features corresponding to each preset dimension can be weighted and fused according to the weights of the target features corresponding to each preset dimension.

[0107] Understandably, as the number of training rounds increases, the updated target model becomes less dependent on auxiliary features. At the end of training, the dependence on auxiliary features is removed, and the latest target model is determined as the fove detection model. This ensures that the training data for the fove detection model is rich enough to improve its accuracy, while simplifying the input of the fove detection model.

[0108] The data used in the training of the gaze detection model provided in this application were all obtained through legal means.

[0109] like Figure 3 As shown, in one embodiment, this application provides a gaze point detection method. The following embodiments illustrate the application of this method to a server. It is understood that the gaze point detection method can be performed by a single server or a server cluster consisting of multiple servers; this application does not impose specific limitations on this. The method includes:

[0110] Step S301: Obtain the data to be detected.

[0111] The data to be detected is the data corresponding to the target features excluding auxiliary features.

[0112] When executing the gaze detection method, the server will receive a detection instruction and obtain the data to be detected based on the received instruction.

[0113] Understandably, when the gaze detection model training ends, based on the weights of the auxiliary features, the gaze detection model's dependence on the auxiliary features will be removed. Therefore, the data to be detected only needs to include data corresponding to the target features other than the auxiliary features. In this way, while ensuring the accuracy of the gaze detection method, the input of the gaze detection model can be simplified, and a smaller gaze detection model can be used to detect gaze points, saving server resources.

[0114] Step S302: Input the data to be detected into the gaze point detection model to obtain the coordinate values ​​of the gaze direction corresponding to the data to be detected.

[0115] The gaze detection model is generated using the gaze detection model training method described in any of the above embodiments.

[0116] In this step, the data to be detected is used as input to the gaze detection model. The gaze detection model obtains the coordinates of the gaze direction corresponding to the data to be detected. At this time, the coordinates of the gaze direction can only roughly determine the location of the object being gazed at, rather than the precise coordinates.

[0117] Step S303: Based on the monocular camera, obtain the distance to the object being gazed upon.

[0118] The object-gazing distance is the distance between the gazer and the object being gazed upon.

[0119] Using a monocular camera, the distance to the object being gazed upon can be obtained. Because monocular cameras can be mounted in fixed positions to achieve accurate 3D detection, gaze point detection methods can be applied in open retail settings. Due to the complexity of open retail settings and the high requirements for deployment, using a monocular camera can improve the deployability of gaze point detection methods. When applied to open retail settings, gaze point detection methods can identify the products that customers are focusing on, thereby adjusting sales strategies based on customer gaze behavior. In addition, gaze point detection methods can also be applied in examination environments to identify abnormal behavior of examinees. This application does not impose specific limitations on the application scenarios of gaze point detection methods.

[0120] In another embodiment, the method of obtaining the gaze distance can be adjusted according to the application scenario. For example, when applied in a retail open scenario, the gaze distance can be obtained by gridding the planar map of the retail open scenario.

[0121] Step S304: Input the coordinate values ​​of the distance to the object being gazed at and the direction of the gaze into the preset gaze point estimation module to obtain the coordinate values ​​of the gaze point corresponding to the data to be detected.

[0122] Based on the distance to the object being gazed upon, methods such as linear regression can be used to decouple the coordinate values ​​of the gaze direction in order to obtain the coordinate values ​​of the gaze point corresponding to the data to be detected.

[0123] The gaze detection model training method, gaze detection method, and apparatus provided in this application include: when the gaze detection model generated by the gaze detection model training method is applied to the gaze detection method, only data corresponding to the target features other than auxiliary features need to be input to obtain relatively accurate coordinate values ​​of the gaze direction, thereby improving the accuracy of the gaze detection method.

[0124] like Figure 4 As shown, in one embodiment, the coordinate values ​​of the object being gazed at and the direction of the gaze are input into a preset gaze point estimation module to obtain the coordinate values ​​of the gaze point corresponding to the data to be detected, including:

[0125] Step S401: Obtain the camera intrinsic parameter matrix of the monocular camera.

[0126] The camera intrinsic matrix, also known as the camera projection matrix or camera calibration matrix, is one of the internal parameter descriptions of a camera. It can be used to describe the internal geometric features of the camera, including focal length, camera optical center position, and image plane scaling, etc.

[0127] Step S402: Based on the camera intrinsic parameter matrix and the coordinate values ​​of the viewing direction, calculate the vector from the camera optical center of the monocular camera to the center of the eyeball and the vector from the center of the eyeball to the object being gazed at.

[0128] Based on the inverse of the camera intrinsic matrix, the coordinate values ​​of the viewing direction can be inversely projected, thereby calculating the vector from the camera optical center of the monocular camera to the center of the eyeball and the vector from the center of the eyeball to the object being viewed.

[0129] Step S403: Perform coordinate system transformation on the vector from the camera optical center of the monocular camera to the eye center and the vector from the eye center to the object being gazed upon, to obtain the vector from the camera optical center to the object being gazed upon.

[0130] By converting the vector from the camera optical center of the monocular camera pointing to the center of the eyeball and the vector from the center of the eyeball pointing to the object being viewed into the world coordinate system, the vector from the camera optical center pointing to the object being viewed can be calculated.

[0131] Step S404: Calculate the spatial coordinates of the gaze point corresponding to the object being gazed upon using triangulation.

[0132] Among them, the spatial coordinates of the gaze point corresponding to the object being gazed upon are the coordinate values ​​of the gaze point.

[0133] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0134] The gaze detection method provided in this application can be applied to terminals that include a display interface and acquisition components, such as mobile phones and computers. With user authorization, the gaze detection method provided in this application can be used to collect the biometric features of authorized users. For example, with user authorization, an examination monitoring system can use the gaze detection method of this application to collect the biometric features of authorized users to identify abnormal behavior in the examination room.

[0135] The gaze point detection model training device provided in the embodiments of this application is described below. The gaze point detection model training device described below can be referred to in correspondence with the gaze point detection model training method described above.

[0136] like Figure 5 As shown, this application provides a gaze point detection model training device 500, the device comprising:

[0137] The target data acquisition module 501 is used to acquire target data corresponding to multiple preset dimensions;

[0138] The target data group determination module 502 is used to divide the target data corresponding to each preset dimension into multiple data groups in a training round, and determine the target data group from each data group corresponding to each preset dimension; wherein, the target data group is any data group that has not undergone feature fusion in the current training round.

[0139] The target feature determination module 503 is used to determine whether the target data groups corresponding to each preset dimension are all empty. If not, it determines the target features corresponding to each preset dimension based on the target data groups corresponding to each preset dimension.

[0140] The weight determination module 504 is used to determine the weights of auxiliary features and target features other than auxiliary features in the target features corresponding to each preset dimension; wherein, the auxiliary features are the target features corresponding to the preset dimensions of the target in advance, and the weight of the auxiliary features is inversely proportional to the current training round.

[0141] The feature fusion module 505 is used to perform weighted fusion of the target features corresponding to each preset dimension according to the weight of the target features corresponding to each preset dimension, so as to obtain the fused features.

[0142] The first iteration training module 506 is used to iteratively train the target model based on the fusion features to obtain a new target model, redetermine the target data groups and target features corresponding to each preset dimension, and perform weighted fusion of the target features corresponding to each preset dimension according to the weights of the target features corresponding to each preset dimension to obtain new fusion features, so as to iteratively train the new target model until the target data groups corresponding to each preset dimension are all empty and the current training round is not less than a preset threshold, and the latest target model is determined as the fixation detection model.

[0143] The second iteration training module 507 is used to enter the next training round if the current training round is less than a preset threshold.

[0144] In one embodiment, the target feature determination module includes:

[0145] The target feature extraction submodule is used to extract features from any one of the target data groups corresponding to each preset dimension based on the preset model corresponding to each preset dimension, so as to obtain the target features corresponding to each preset dimension.

[0146] In one embodiment, the weight determination module includes:

[0147] The weight setting submodule is used to substitute the current training round into the preset feature fusion formula to obtain the weights of the auxiliary features. According to the feature fusion formula, the weights of the target features other than the auxiliary features are set to 1, so as to complete the determination of the weights of the target features corresponding to each preset dimension.

[0148] The gaze point detection device provided in the embodiments of this application is described below. The gaze point detection device described below can be referred to in correspondence with the gaze point detection method described above.

[0149] like Figure 6 As shown, this application provides a gaze point detection device 600, the device comprising:

[0150] The data acquisition module 601 is used to acquire the data to be detected; wherein, the data to be detected is the data corresponding to the target features excluding auxiliary features;

[0151] The first coordinate value determination module 602 is used to input the data to be detected into the gaze point detection model to obtain the coordinate values ​​of the gaze direction corresponding to the data to be detected; wherein, the gaze point detection model is generated using the gaze point detection model training method as described in any of the above embodiments;

[0152] The distance determination module 603 is used to obtain the distance to the object being gazed upon based on a monocular camera; wherein, the distance to the object being gazed upon is the distance between the gazer and the object being gazed upon;

[0153] The second coordinate value determination module 604 is used to input the coordinate values ​​of the distance to the object being gazed at and the direction of the gaze to a preset gaze point estimation module to obtain the coordinate values ​​of the gaze point corresponding to the data to be detected.

[0154] In one embodiment, the second coordinate value determination module includes:

[0155] The matrix acquisition submodule is used to obtain the camera intrinsic parameter matrix of a monocular camera;

[0156] The first vector determination submodule is used to calculate the vector from the camera optical center of the monocular camera to the center of the eyeball and the vector from the center of the eyeball to the object being gazed upon, based on the camera intrinsic parameter matrix and the coordinate values ​​of the gaze direction.

[0157] The second vector determination submodule is used to perform coordinate system transformation on the vector of the camera optical center pointing to the eye center and the vector of the eye center pointing to the object being gazed at, so as to obtain the vector of the camera optical center pointing to the object being gazed at.

[0158] The coordinate calculation submodule is used to calculate the spatial coordinates of the gaze point corresponding to the object being gazed upon, based on triangulation methods.

[0159] The division of modules in the aforementioned gaze point detection model training device and / or gaze point detection device is merely illustrative. In other embodiments, the gaze point detection model training device and / or gaze point detection device may be divided into different modules as needed to complete all or part of the functions of the aforementioned gaze point detection model training device and / or gaze point detection device. Each module in the aforementioned gaze point detection model training device and / or gaze point detection device may be implemented entirely or partially through software, hardware, or a combination thereof. Each module may be embedded in hardware or independent of the processor in a computer device, or stored in software in the memory of a computer device, so that the processor can call and execute the operations corresponding to each module.

[0160] In one embodiment, this application also provides a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the gaze point detection model training method as described in any of the foregoing embodiments, and / or to perform the steps of the gaze point detection method as described in any of the foregoing embodiments.

[0161] In one embodiment, this application also provides a computer device storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the gaze point detection model training method as described in any of the foregoing embodiments, and / or to perform the steps of the gaze point detection method as described in any of the foregoing embodiments.

[0162] Indicatively, such as Figure 7 As shown, Figure 7 This is a schematic diagram of the internal structure of a computer device 700 provided in an embodiment of this application. The computer device 700 can be provided as a server. (Refer to...) Figure 7 The computer device 700 includes a processing component 702, which further includes one or more processors, and memory resources represented by memory 701 for storing instructions executable by the processing component 702, such as application programs. The application programs stored in memory 701 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 702 is configured to execute instructions to perform the gaze detection model training method of any of the above embodiments, and / or to perform the gaze detection method as described in any of the above embodiments.

[0163] The computer device 700 may also include a power supply component 703 configured to perform power management of the computer device 700, a wired or wireless network interface 704 configured to connect the computer device 700 to a network, and an input / output (I / O) interface 705. The computer device 700 can operate on an operating system stored in memory 701, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.

[0164] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0165] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising a…" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. In this document, the singular forms "a," "an," and "the" may also include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising / including” or “having” specify the presence of the stated features, wholes, steps, operations, components, parts or combinations thereof, but do not exclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts or combinations thereof. Meanwhile, the term “and / or” as used in this specification includes any and all combinations of the associated listed items.

[0166] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0167] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for training a gaze point detection model, characterized in that, The method includes: Acquire target data corresponding to multiple preset dimensions; the preset dimensions include human eye image dimension, binocular image dimension, and facial key point dimension; In a training round, the target data corresponding to each preset dimension is divided into multiple data groups, and the target data group is determined from each data group corresponding to each preset dimension; wherein, the target data group is any data group that has not undergone feature fusion in the current training round. Based on the target data group corresponding to each preset dimension, determine the target features corresponding to each preset dimension; Determine the auxiliary features and the weights of the target features other than the auxiliary features in each preset dimension; wherein the auxiliary features are the target features corresponding to the preset target dimensions that are predetermined in each preset dimension, and the weights of the auxiliary features are inversely proportional to the current training round. Based on the weights of the target features corresponding to each preset dimension, the target features corresponding to each preset dimension are weighted and fused to obtain the fused features; The target model is iteratively trained based on the fusion features to obtain a new target model. The target data groups and target features corresponding to each preset dimension are redefined. The target features corresponding to each preset dimension are weighted and fused according to their weights to obtain new fusion features. The new target model is iteratively trained until the target data groups corresponding to each preset dimension are all empty and the current training round is not less than a preset threshold. The latest target model is then determined as the fixation detection model. If the target data groups corresponding to each of the newly determined preset dimensions are all empty, and the current training round is less than the preset threshold, then proceed to the next training round.

2. The gaze point detection model training method according to claim 1, characterized in that, The step of determining the target features corresponding to each preset dimension based on the target data group corresponding to each preset dimension includes: Based on the preset models corresponding to each preset dimension, feature extraction is performed on any one of the target data groups corresponding to each preset dimension to obtain the target features corresponding to each preset dimension.

3. The gaze point detection model training method according to claim 2, characterized in that, After performing iterative training on the target model based on the fused features to obtain a new target model, the method further includes: Based on the new target model, update the preset models corresponding to each preset dimension.

4. The gaze point detection model training method according to claim 1, characterized in that, The step of determining the auxiliary features and the weights of the target features other than the auxiliary features in the target features corresponding to each preset dimension includes: Substitute the current training round into the preset feature fusion formula to obtain the weight of the auxiliary feature. According to the feature fusion formula, set the weight of the target feature other than the auxiliary feature to 1 to complete the determination of the weight of the target feature corresponding to each preset dimension.

5. A method for detecting gaze points, characterized in that, The method includes: Acquire the data to be detected; wherein, the data to be detected is the data corresponding to the target features excluding auxiliary features; The data to be detected is input into the gaze point detection model to obtain the coordinate values ​​of the gaze direction corresponding to the data to be detected; wherein, the gaze point detection model is generated using the gaze point detection model training method as described in any one of claims 1 to 4; Based on a monocular camera, the distance to the object being gazed upon is obtained; wherein, the distance to the object being gazed upon is the distance between the gazer and the object being gazed upon. The coordinate values ​​of the distance to the object being gazed at and the direction of the gaze are input into a preset gaze point estimation module to obtain the coordinate values ​​of the gaze point corresponding to the data to be detected.

6. The fixation point detection method according to claim 5, characterized in that, The step of inputting the coordinate values ​​of the gaze distance and the gaze direction into a preset gaze point estimation module to obtain the coordinate values ​​of the gaze point corresponding to the data to be detected includes: Obtain the camera intrinsic parameter matrix of the monocular camera; Based on the camera intrinsic parameter matrix and the coordinate values ​​of the line of sight, calculate the vector from the camera optical center of the monocular camera to the center of the eyeball and the vector from the center of the eyeball to the object being gazed upon. The vector from the camera optical center of the monocular camera to the center of the eyeball and the vector from the center of the eyeball to the object being gazed are transformed into a coordinate system to obtain the vector from the camera optical center to the object being gazed. The spatial coordinates of the gaze point corresponding to the object being gazed upon are calculated using triangulation.

7. A gaze point detection model training device, characterized in that, The device includes: The target data acquisition module is used to acquire target data corresponding to multiple preset dimensions; the preset dimensions include human eye image dimension, binocular image dimension, and facial key point dimension. The target data group determination module is used to divide the target data corresponding to each preset dimension into multiple data groups in a training round, and determine the target data group from each data group corresponding to each preset dimension; wherein, the target data group is any data group that has not undergone feature fusion in the current training round. The target feature determination module is used to determine the target features corresponding to each preset dimension based on the target data group corresponding to each preset dimension. The weight determination module is used to determine the weights of auxiliary features and target features other than the auxiliary features in the target features corresponding to each preset dimension; wherein, the auxiliary features are target features corresponding to the target preset dimensions that are predetermined in each preset dimension, and the weight of the auxiliary features is inversely proportional to the current training round. The feature fusion module is used to perform weighted fusion of the target features corresponding to each preset dimension according to the weights of the target features corresponding to each preset dimension, so as to obtain the fused features; The first iterative training module is used to iteratively train the target model according to the fusion features to obtain a new target model, redetermine the target data groups and target features corresponding to each preset dimension, and perform weighted fusion of the target features corresponding to each preset dimension according to the weights of the target features corresponding to each preset dimension to obtain new fusion features, so as to iteratively train the new target model until the target data groups corresponding to each preset dimension are all empty and the current training round is not less than a preset threshold, and the latest target model is determined as the fixation detection model. The second iterative training module is used to enter the next training round if the target data groups corresponding to each preset dimension that have been redefined are all empty and the current training round is less than the preset threshold.

8. A gaze point detection device, characterized in that, The device includes: The data acquisition module is used to acquire the data to be detected; wherein, the data to be detected is the data corresponding to the target features excluding auxiliary features; The first coordinate value determination module is used to input the data to be detected into the gaze point detection model to obtain the coordinate values ​​of the gaze direction corresponding to the data to be detected; wherein, the gaze point detection model is generated using the gaze point detection model training method as described in any one of claims 1 to 4; A distance determination module is used to obtain the distance to the object being gazed upon based on a monocular camera; wherein, the distance to the object being gazed upon is the distance between the gazer and the object being gazed upon; The second coordinate value determination module is used to input the coordinate values ​​of the distance to the object being gazed upon and the direction of the gaze to a preset gaze point estimation module to obtain the coordinate values ​​of the gaze point corresponding to the data to be detected.

9. A storage medium, characterized in that: The storage medium stores computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the gaze point detection model training method as described in any one of claims 1 to 4, and / or to perform the steps of the gaze point detection method as described in any one of claims 5 to 6.

10. A computer device, characterized in that, include: One or more processors, and a memory; the memory storing computer-readable instructions, which, when executed by the one or more processors, perform the steps of the gaze point detection model training method as claimed in any one of claims 1 to 4, and / or perform the steps of the gaze point detection method as claimed in any one of claims 5 to 6.

Images