Training method and device of line-of-sight estimation model, line-of-sight estimation method and device

Abstract

The application provides a gaze estimation model training method and device and a gaze estimation method and device, and relates to the technical field of computer vision. The training method comprises the following steps: determining a sample image pair of a target object based on a binocular region image set of the target object, wherein the sample image pair comprises two sample images; determining a first gaze feature corresponding to each of the two sample images; determining a loss function of a to-be-trained neural network model based on the first gaze feature corresponding to each of the two sample images; and training the to-be-trained neural network model based on the loss function to obtain a gaze estimation model. Compared with the existing gaze estimation model training method, the representation ability of the gaze feature is significantly improved, the effective working range of the gaze estimation model is expanded, and the accuracy of the gaze direction prediction is improved. In addition, the gaze estimation model trained and generated by the embodiment of the application can provide comprehensive and accurate support for analyzing the driving state of a driver, thereby improving the safety of vehicle driving.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, specifically to a training method and apparatus for a gaze estimation model, and a gaze estimation method and apparatus. Background Technology

[0002] Gaze estimation has wide applications in driver status monitoring, human-computer interaction, and psychological research. Current gaze estimation methods are mainly appearance-based. Appearance-based gaze estimation methods generally use monocular (or binocular) images with head pose information, binocular images with facial images, or facial images as input to the neural network model to be trained. The input image and head pose information are directly mapped to gaze direction through an end-to-end convolutional neural network (CNN) structure.

[0003] However, appearance-based methods lack constraints on gaze features at the feature level during training, making it impossible to achieve feature decoupling or feature purification. This results in a lack of feature representation capabilities and interpretability. Summary of the Invention

[0004] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide a training method and apparatus for a gaze estimation model, as well as a gaze estimation method and apparatus.

[0005] In a first aspect, one embodiment of this application provides a training method for a gaze estimation model. The training method includes: determining a pair of sample images of the target object based on a set of binocular region images of the target object, wherein the pair of sample images includes two sample images; determining a first gaze feature corresponding to each of the two sample images; determining a loss function of the neural network model to be trained based on the first gaze feature corresponding to each of the two sample images; and training the neural network model to be trained based on the loss function to obtain a gaze estimation model.

[0006] In conjunction with the first aspect, in some implementations of the first aspect, the loss function of the neural network model to be trained is determined based on the first gaze features corresponding to each of the two sample images, including: determining the feature consistency loss function of the neural network model to be trained based on the first gaze features corresponding to each of the two sample images; determining the training loss function of the neural network model to be trained; and determining the loss function of the neural network model to be trained based on the feature consistency loss function and the training loss function.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, the feature consistency loss function of the neural network model to be trained is determined based on the first gaze direction features corresponding to each of the two sample images, including: determining the gaze direction transformation relationship between the two sample images; for each of the two sample images, performing feature transformation on the first gaze direction features corresponding to the sample image using the gaze direction transformation relationship to obtain the second gaze direction features corresponding to the sample image, wherein the second gaze direction features corresponding to the sample image and the first gaze direction features represent the same gaze direction; and determining the feature consistency loss function based on the first gaze direction features and the second gaze direction features corresponding to each of the two sample images.

[0008] In conjunction with the first aspect, in certain implementations of the first aspect, determining the gaze direction transformation relationship between two sample images includes: determining the gaze direction labels corresponding to each of the two sample images; determining the rotation matrices corresponding to each of the two sample images based on the gaze direction labels and the reference direction unit vector; determining the transformation matrix between the two sample images based on the rotation matrices; and determining the gaze direction transformation relationship based on the gaze direction labels and the transformation matrix.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, determining the training loss function of the neural network model to be trained includes: determining the gaze direction labels corresponding to each of the two sample images; determining the predicted gaze direction corresponding to each of the two sample images; for each of the two sample images, determining the gaze angle corresponding to the sample image based on the gaze direction label and the predicted gaze direction, wherein the gaze angle refers to the angle between the gaze direction label and the predicted gaze direction; and determining the training loss function based on the gaze angle corresponding to each of the two sample images.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the neural network model to be trained includes a gaze prediction multilayer perceptron, which determines the predicted gaze direction corresponding to each of the two sample images, including: for each of the two sample images, performing feature transformation of a first gaze feature and a second gaze feature corresponding to the sample image with a preset dimension; and using the gaze prediction multilayer perceptron to process the first gaze feature and the second gaze feature after feature transformation to obtain the predicted gaze direction corresponding to the sample image.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, determining the first gaze feature corresponding to each of the two sample images includes: determining the head pose information of the target object; determining the head pose feature of the target object based on the head pose information; extracting features from the sample image pair to obtain the appearance features corresponding to each of the two sample images; and for each of the two sample images, stitching together the appearance feature and head pose feature corresponding to the sample image to obtain the first gaze feature corresponding to the sample image.

[0012] Secondly, one embodiment of this application provides a gaze estimation method, which includes acquiring a frame of binocular region image of a target object; inputting the frame of binocular region image into a gaze estimation model, extracting features to obtain gaze features of the target object, and predicting gaze estimation information corresponding to the target object based on the gaze features of the target object; wherein the gaze estimation model is generated based on the training method of the gaze estimation model mentioned in the first aspect.

[0013] Thirdly, one embodiment of this application provides a training apparatus for a gaze estimation model, comprising: a first determining module, configured to determine a pair of sample images of a target object based on a set of binocular region images of the target object, wherein the pair of sample images includes two sample images; a second determining module, configured to determine a first gaze feature corresponding to each of the two sample images; a third determining module, configured to determine a loss function of a neural network model to be trained based on the first gaze feature corresponding to each of the two sample images; and a training module, configured to train the neural network model to be trained based on the loss function to obtain a gaze estimation model.

[0014] Fourthly, one embodiment of this application provides a gaze estimation device, which includes: an acquisition module for acquiring a frame of binocular region image of a target object; and a prediction module for inputting the frame of binocular region image into a gaze estimation model, extracting features to obtain gaze features of the target object, and predicting gaze estimation information corresponding to the target object based on the gaze features of the target object; wherein the gaze estimation model is generated based on the training method of the gaze estimation model mentioned in the first aspect.

[0015] Fifthly, one embodiment of this application provides a computer-readable storage medium storing computer program instructions for performing the methods described in the first aspect and / or the second aspect.

[0016] In a sixth aspect, one embodiment of this application provides an electronic device comprising: a processor; a memory for storing processor-executable instructions; the processor being configured to perform the methods described in the first aspect and / or the second aspect.

[0017] In this embodiment, based on a set of binocular region images of the target object, sample image pairs of the target object are determined, each pair including two sample images. First gaze features corresponding to each of the two sample images are determined. Based on the first gaze features corresponding to each of the two sample images, a loss function for the neural network model to be trained is determined. The neural network model to be trained is then trained based on the loss function to obtain a gaze estimation model. Compared with existing gaze estimation model training methods, this significantly improves the representation capability of gaze features, expands the effective working range of the gaze estimation model, and improves the accuracy of gaze direction estimation. Furthermore, the gaze estimation model generated in this embodiment can provide comprehensive and accurate support and basis for analyzing the driver's driving state, thereby further improving vehicle safety during driving and possessing the advantage of wide applicability. Attached Figure Description

[0018] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0019] Figure 1 The diagram shown is a scenario applicable to an embodiment of this application.

[0020] Figure 2 The diagram shown illustrates another scenario applicable to the embodiments of this application.

[0021] Figure 3 The diagram shown is a flowchart illustrating a training method for a gaze estimation model provided in an exemplary embodiment of this application.

[0022] Figure 4 The diagram shown is a flowchart illustrating how a loss function for a neural network model to be trained is determined based on the first gaze features corresponding to two sample images, according to an exemplary embodiment of this application.

[0023] Figure 5 The diagram shown is a schematic flowchart of an exemplary embodiment of this application, which describes the process of determining the feature consistency loss function of a neural network model to be trained based on the first gaze features corresponding to two sample images.

[0024] Figure 6 The diagram shown is a schematic flowchart of a method for determining the relationship between the line-of-sight direction transformations of two sample images, provided in an exemplary embodiment of this application.

[0025] Figure 7The diagram shown is a flowchart illustrating the process of determining the training loss function of a neural network model to be trained, provided in an exemplary embodiment of this application.

[0026] Figure 8 The diagram shown is a flowchart illustrating the process of determining the training loss function of a neural network model to be trained, provided in another exemplary embodiment of this application.

[0027] Figure 9 The diagram shown is a flowchart illustrating the process of determining the first gaze features corresponding to two sample images, as provided in an exemplary embodiment of this application.

[0028] Figure 10 The diagram shown is a flowchart illustrating a line-of-sight estimation method provided in an exemplary embodiment of this application.

[0029] Figure 11 The diagram shown is a schematic representation of the structure of a training device for a gaze estimation model provided in an exemplary embodiment of this application.

[0030] Figure 12 The diagram shown is a schematic diagram of the line-of-sight estimation device provided in an exemplary embodiment of this application.

[0031] Figure 13 The diagram shown is a structural schematic of an electronic device provided in an exemplary embodiment of this application. Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] Exemplary System

[0035] Figure 1 The diagram shown illustrates a scenario applicable to an embodiment of this application. Figure 1 As shown, the embodiments of this application are applicable to the scenario of a driver status monitoring platform, which includes a server 1 and an image acquisition device 2, wherein there is a communication connection between the server 1 and the image acquisition device 2.

[0036] Specifically, the image acquisition device 2 can be installed inside the vehicle to acquire multiple frames of user images of the driver. The image acquisition device 2 is a camera system including multiple cameras. The cameras can be ordinary visible light cameras or near-infrared cameras. The multiple cameras can be of the same or different models, and the number of multiple cameras can be two or more, as long as the image acquisition device 2 can acquire multiple frames of user images of the driver. This application does not impose specific limitations on the structure of the image acquisition device 2.

[0037] Server 1 can be a single server, a server group consisting of multiple servers, a virtualization platform, or a cloud computing service center. This application does not specifically limit the type of server 1. Server 1 acquires a set of binocular region images of the driver acquired by image acquisition device 2; based on the set of binocular region images of the target object, it determines sample image pairs of the target object, each sample image pair including two sample images; it determines the first gaze feature corresponding to each of the two sample images; based on the first gaze feature corresponding to each of the two sample images, it determines the loss function of the neural network model to be trained; and it trains the neural network model to be trained based on the loss function to obtain a gaze estimation model. That is, this scenario implements a training method for a gaze estimation model.

[0038] because Figure 1 The scenario shown above utilizes server 1 to implement the training method for the gaze estimation model. Therefore, this scenario not only improves the adaptability of the scenario but also effectively reduces the computational load of the image acquisition device 2.

[0039] It should be noted that this application also applies to another scenario. Figure 2 The diagram illustrates another scenario applicable to the embodiments of this application. Specifically, the embodiments of this application are applicable to a vehicle driving scenario, which includes an image processing device 3 installed on the A-pillar inside the vehicle. The image processing device 3 includes an image acquisition module 30 and a calculation module 31, and there is a communication connection between the image acquisition module 30 and the calculation module 31.

[0040] Specifically, the image acquisition module 30 in the image processing device 3 is used to acquire a set of images of the driver's binocular regions. The calculation module 31 in the image processing device 3, based on the set of binocular region images of the target object, determines a pair of sample images of the target object, each pair containing two sample images; determines the first gaze feature corresponding to each of the two sample images; determines the loss function of the neural network model to be trained based on the first gaze feature corresponding to each of the two sample images; and trains the neural network model to be trained based on the loss function to obtain a gaze estimation model. That is, this scenario implements a method for training a gaze estimation model.

[0041] because Figure 2 The scenario shown above utilizes image processing device 3 to implement the training method for the gaze estimation model, eliminating the need for data transmission with servers or other related devices. Therefore, Figure 2 The scenario shown above ensures the real-time performance of the training method for the gaze estimation model.

[0042] It should be understood that the gaze estimation model trained by the training method provided in this application can be applied to various scenarios, such as intelligent cockpits determining whether a driver is fatigued or using gaze for human-vehicle interaction. It can also be applied in the medical field, for example, to detect and diagnose mental or psychological illnesses; or in virtual reality, to perform refined scene rendering only on areas of human eye focus; and it can also be applied to offline retail or human-computer interaction. This application does not specifically limit the application scenarios of the gaze estimation model generated by the training method.

[0043] Exemplary methods

[0044] Figure 3 The diagram shown is a flowchart illustrating a training method for a gaze estimation model provided in an exemplary embodiment of this application. Figure 3 As shown in the embodiments of this application, the training method for the gaze estimation model includes the following steps.

[0045] Step S300: Based on the set of binocular region images of the target object, determine the sample image pair of the target object.

[0046] For example, a sample image pair includes two sample images. The set of images of both eyes' regions can be used as a training set, and pairs of samples can be randomly sampled from the training set to serve as sample image pairs for the target object.

[0047] For example, the binocular region image set can be obtained based on a camera system. The camera system includes multiple cameras. That is, the camera system can be a camera matrix constructed with different installation locations and different numbers of cameras for different application scenarios. The installation locations of the multiple cameras can be determined based on the gaze location of the target object, and the number of cameras can be determined based on the range of line-of-sight coverage. The cameras can be ordinary visible light or near-infrared cameras, and can be the same model or different models; there can be two or more cameras.

[0048] For example, the binocular region image can be a near-infrared image captured by a near-infrared camera, a visible light image captured by a visible light camera, or an infrared image captured by an infrared camera. This application embodiment does not limit the specific form of the user image; it can be the original captured image, an image of the original captured image after preprocessing operations such as noise reduction, or a portion of the original captured image. Furthermore, the object corresponding to the binocular region image can be the driver or other target objects.

[0049] In practical applications, a camera system is used to acquire a set of binocular region images of the target object in real time. For a camera system consisting of C cameras, the set of binocular region images X0, X1, ..., X can be acquired simultaneously at each moment. C-1 A pair of samples is randomly selected as the sample image pair X0 and X1 of the target object.

[0050] Step S301: Determine the first gaze features corresponding to each of the two sample images.

[0051] For example, the first line of sight feature refers to a feature that can reflect changes in line of sight.

[0052] Step S302: Determine the loss function of the neural network model to be trained based on the first gaze features corresponding to each of the two sample images.

[0053] For example, the loss function is used to improve the semantic consistency between gaze features and gaze direction, and to optimize the accuracy of the prediction results.

[0054] Step S303: Train the neural network model to be trained based on the loss function to obtain the gaze estimation model.

[0055] In the actual training process, pairs of samples are randomly sampled from the training set as inputs to the neural network model to be trained. The neural network model is trained based on the loss function until it converges, thereby determining the gaze estimation model.

[0056] For example, a gaze estimation model is used to estimate the direction of human eye gaze to obtain gaze estimation information. The gaze estimation information includes gaze region information and / or gaze point information.

[0057] For example, the gaze estimation model can be a neural network model such as MobilenetV2, ResNet18, Fast-RCNN, Faster-RCNN, R-FCN, YOLO (You Only Look Once), and SSD (Single Shot Multibox Detector). It should be understood that the gaze estimation model can also be other neural network models that can output gaze estimation information corresponding to sample image pairs.

[0058] The gaze estimation model training method provided in this application embodiment is based on a set of binocular region images of the target object, determining sample image pairs of the target object, each sample image pair including two sample images; determining the first gaze feature corresponding to each of the two sample images; determining the loss function of the neural network model to be trained based on the first gaze feature corresponding to each of the two sample images; and training the neural network model to be trained based on the loss function to obtain the gaze estimation model. Compared with existing gaze estimation model training methods, this method significantly improves the representation ability of gaze features, expands the effective working range of the gaze estimation model, and improves the accuracy of gaze direction estimation. Furthermore, the gaze estimation model generated in this application embodiment can provide comprehensive and accurate support and basis for analyzing the driver's driving state, thereby further improving the safety of vehicle driving and having the advantage of wide applicability.

[0059] Figure 4 The diagram illustrates a flowchart of an exemplary embodiment of this application, illustrating the process of determining the loss function of a neural network model to be trained based on the first gaze features corresponding to two sample images. Figure 3 This application extends from the embodiments shown. Figure 4 The illustrated embodiment will be described in detail below. Figure 4 The illustrated embodiments and Figure 3 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0060] like Figure 4 As shown, in the training method of the gaze estimation model provided in this application embodiment, the loss function of the neural network model to be trained is determined based on the first gaze features corresponding to the two sample images (step S302), including the following steps.

[0061] Step S401: Based on the first gaze features corresponding to each of the two sample images, determine the feature consistency loss function of the neural network model to be trained.

[0062] For example, the feature consistency loss function is used to improve the semantic consistency between gaze features and gaze direction during training.

[0063] Step S402: Determine the training loss function of the neural network model to be trained.

[0064] For example, the training loss function is used to optimize the accuracy of the prediction results.

[0065] Step S403: Determine the loss function of the neural network model to be trained based on the feature consistency loss function and the training loss function.

[0066] For example, the loss function of the neural network model to be trained is used as the overall loss function during the training process.

[0067] As an example, the feature consistency loss function is used This indicates that the training loss function is used The loss function is expressed as follows:

[0068] The training method for the gaze estimation model provided in this application determines the loss function of the neural network model to be trained based on the feature consistency loss function and the training loss function. This significantly improves the correlation between gaze features and gaze direction, has better interpretability, and enhances the representation ability of gaze features and the accuracy of gaze direction estimation.

[0069] Figure 5 The diagram illustrates a flowchart of an exemplary embodiment of this application, showing how to determine the feature consistency loss function of a neural network model to be trained based on the first gaze features corresponding to two sample images. Figure 4 This application extends from the embodiments shown. Figure 5 The illustrated embodiment will be described in detail below. Figure 5 The illustrated embodiments and Figure 4 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0070] like Figure 5 As shown, in the training method of the gaze estimation model provided in this application embodiment, the feature consistency loss function of the neural network model to be trained is determined based on the first gaze features corresponding to the two sample images (step S401), including the following steps.

[0071] Step S501: Determine the relationship between the line-of-sight direction transformations of the two sample images.

[0072] For example, the sample image pair is X0 and X1, and the gaze direction labels of sample image X0 and sample image X1 are g0 and g1, respectively. The gaze direction transformation relationship between the two sample images is determined based on the gaze direction labels corresponding to each of the two sample images. The gaze direction transformation relationship is used to rotate the gaze features to the same direction as the paired samples.

[0073] Perform the following steps for each of the two sample images.

[0074] Step S502: The first gaze feature corresponding to the sample image is transformed using the gaze direction transformation relationship to obtain the second gaze feature corresponding to the sample image. The second gaze feature corresponding to the sample image represents the same gaze direction as the first gaze feature.

[0075] Specifically, the first gaze feature corresponding to the sample image is the original gaze feature. The second gaze feature corresponding to the sample image is the transformed gaze feature.

[0076] As an example, the first gaze feature corresponding to sample image X0 is z0, and the second gaze feature corresponding to sample image X0 is z0'. The first gaze feature corresponding to sample image X1 is z1, and the second gaze feature corresponding to sample image X1 is z1'. z0 and z0', and z1 and z1' in the feature space represent the same gaze direction, and the features should be identical or approximately the same. z0 and z1 represent different gaze directions.

[0077] Step S503: Determine the feature consistency loss function based on the first gaze feature and the second gaze feature corresponding to the two sample images respectively.

[0078] Specifically, the first gaze feature z1 corresponding to sample image X1 and the second gaze feature z1' corresponding to sample image X1 obtained by transforming the first gaze feature z0 corresponding to sample image X0 are semantically consistent in the feature space, and their feature values ​​should be made close during training. Therefore, a feature consistency loss function is added during training.

[0079]

[0080] in,

[0081] The training method for the gaze estimation model provided in this application adds a feature consistency loss function at the feature level. Compared with existing gaze estimation model training methods, it significantly improves the correlation between gaze features and gaze direction, has better interpretability, and improves the accuracy of gaze direction estimation.

[0082] Figure 6 The diagram shown is a schematic flowchart illustrating the process of determining the gaze direction transformation relationship between two sample images according to an exemplary embodiment of this application. Figure 5 This application extends from the embodiments shown. Figure 6 The illustrated embodiment will be described in detail below. Figure 6 The illustrated embodiments and Figure 5 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0083] like Figure 6 As shown, in the training method of the gaze estimation model provided in this application embodiment, determining the gaze direction transformation relationship between two sample images (step S501) includes the following steps.

[0084] Step S601: Determine the gaze direction labels corresponding to each of the two sample images.

[0085] For example, the sample image pair is X0 and X1, and the gaze direction labels of sample image X0 and sample image X1 are g0 and g1, respectively.

[0086] Step S602: Based on the gaze direction labels and reference direction unit vectors corresponding to the two sample images, determine the rotation matrix corresponding to each of the two sample images.

[0087] For example, in the camera coordinate system, the line-of-sight direction is typically represented by two angles: Pitch (vertical pitch angle) and Yaw (horizontal yaw angle), and can be converted into a unit vector g in three-dimensional space. When the target object is looking straight ahead, the reference direction of the line of sight is Pitch = 0 and Yaw = 0. For any line-of-sight direction Pitch = θ and Yaw = φ, it can be obtained by rotating the reference direction unit vector using the following rotation matrix.

[0088]

[0089] Based on this, according to the gaze direction labels g0 and g1 of sample images X0 and X1, the rotation matrices R0(φ0,θ0) and R1(φ1,θ1) transformed by the reference direction unit vector can be calculated.

[0090] Step S603: Determine the transformation matrix between the two sample images based on the rotation matrices corresponding to each of the two sample images.

[0091] For example, the transformation matrix obtained between the viewing directions g0 and g1 is R. 0→1 =R1(R0) -1 The transformation matrix obtained for the line-of-sight directions g1 to g0 is R. 1→0 =R0(R1) -1 .

[0092] Step S604: Determine the gaze direction transformation relationship based on the gaze direction labels and transformation matrices corresponding to the two sample images.

[0093] For example, based on the gaze direction labels and transformation matrices corresponding to the two sample images, the gaze direction transformation relationship between the two sample images is determined as shown in the following formula.

[0094] g′1=R 0→1 g0 = R1(R0) -1 g0=g1 (3)

[0095] g′0=R 1→0 g1 = R0(R1) -1 g1 = g0 (4)

[0096] The training method for the gaze estimation model provided in this application determines the gaze direction transformation relationship based on the gaze direction labels and transformation matrices corresponding to two sample images. Since gaze features and gaze directions are consistent, the same transformation can be performed in the feature space using the gaze direction transformation relationship, thereby determining the same representation of paired samples and achieving the purpose of rotating gaze features to the same direction of paired samples using the gaze direction relationship between sample pairs.

[0097] Figure 7 The diagram shown is a schematic flowchart illustrating the process of determining the training loss function of a neural network model to be trained, provided in an exemplary embodiment of this application. Figure 4 This application extends from the embodiments shown. Figure 7 The illustrated embodiment will be described in detail below. Figure 7 The illustrated embodiments and Figure 4 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0098] like Figure 7 As shown, in the training method of the gaze estimation model provided in this application embodiment, the training loss function of the neural network model to be trained is determined (step S402), and the following steps are performed.

[0099] Step S701: Determine the gaze direction labels corresponding to each of the two sample images.

[0100] As an example, the sample image pair is X0 and X1, and the gaze direction labels of sample image X0 and sample image X1 are g0 and g1, respectively.

[0101] Step S702: Determine the predicted gaze direction corresponding to each of the two sample images.

[0102] Specifically, the neural network model to be trained includes a feature extraction module, a feature transformation module, and a gaze prediction module. Sample images X0 and X1 are processed by the feature extraction and feature transformation modules to obtain corresponding gaze features z0 and z0′, z1 and z1′, respectively. These features are then input into the gaze prediction module to obtain the predicted gaze directions for each of the two sample images.

[0103] Perform the following steps for each of the two sample images.

[0104] Step S703: Based on the gaze direction label and the predicted gaze direction corresponding to the sample image, determine the gaze angle corresponding to the sample image. The gaze angle refers to the angle between the gaze direction label and the predicted gaze direction.

[0105] Step S704: Determine the training loss function based on the eye angles corresponding to the two sample images.

[0106] For example, the training loss function is shown in the following formula (5).

[0107]

[0108] Where α and β are hyperparameters, The angle between the lines of sight.

[0109] In summary, the overall loss function during training is shown in the following formula (6).

[0110]

[0111] in, This can improve the semantic consistency between gaze representation and gaze direction. Optimize the accuracy of prediction results.

[0112] The training method for the gaze estimation model provided in this application uses the angle between the gaze direction label and the predicted gaze direction as the training loss function, which significantly improves the correlation between gaze representation and gaze direction, has better interpretability, and improves the accuracy of gaze direction estimation.

[0113] Figure 8 The diagram shown is a flowchart illustrating the process of determining the training loss function of a neural network model to be trained, provided in another exemplary embodiment of this application. Figure 7 This application extends from the embodiments shown. Figure 8 The illustrated embodiment will be described in detail below. Figure 8 The illustrated embodiments and Figure 7 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0114] like Figure 8 As shown, in the training method of the gaze estimation model provided in this application embodiment, the neural network model to be trained includes a gaze prediction multilayer perceptron, which determines the predicted gaze direction corresponding to each of the two sample images (step S702), and performs the following steps for each of the two sample images.

[0115] Step S801: Perform feature transformation of preset dimensions on the first gaze feature and the second gaze feature corresponding to the sample image respectively.

[0116] Step S802: Use the gaze prediction multilayer perceptron to process the first gaze feature and the second gaze feature after feature transformation to obtain the predicted gaze direction corresponding to the sample image.

[0117] For example, the first gaze feature and the second gaze feature corresponding to the sample image are respectively transformed into a d=48 dimension vector and input into the gaze prediction multilayer perceptron (MLP) to obtain the predicted gaze direction corresponding to the sample image as shown in the following formulas (7) and (8).

[0118]

[0119]

[0120] The training method for the gaze estimation model provided in this application utilizes a gaze prediction multilayer perceptron to process the first gaze feature and the second gaze feature after feature transformation, which can achieve the purpose of obtaining the predicted gaze direction corresponding to the sample image, providing a prerequisite for further using gaze direction labels and determining the loss function based on the predicted gaze direction.

[0121] Figure 9 The diagram shown is a schematic flowchart illustrating the process of determining the first gaze features corresponding to two sample images, as provided in an exemplary embodiment of this application. Figure 3 This application extends from the embodiments shown. Figure 9 The illustrated embodiment will be described in detail below. Figure 9 The illustrated embodiments and Figure 3 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0122] like Figure 9 As shown, in the training method of the gaze estimation model provided in this application embodiment, determining the first gaze features corresponding to each of the two sample images (step S301) includes the following steps.

[0123] Step S901: Determine the head pose information of the target object.

[0124] Step S902: Determine the head pose features of the target object based on the head pose information.

[0125] Step S903: Extract features from the sample image pairs to obtain the appearance features corresponding to each of the two sample images.

[0126] Perform the following steps for each of the two sample images.

[0127] Step S904: Segment the appearance features and head pose features corresponding to the sample image to obtain the first gaze feature corresponding to the sample image.

[0128] In one embodiment, the neural network model to be trained includes a feature extraction module consisting of a series of convolutional and pooling layers, and a gaze prediction module consisting of fully connected (FC) layers. The feature extraction module uses a commonly used convolutional neural network structure (e.g., MobileNetV2, ResNet18, etc.). For multi-input schemes such as eye images and head pose information, or eye images and face images, a multi-branch structure can be used, and feature concatenation can be used to fuse them at the gaze feature layer. Specifically, the gaze feature module uses binocular region images and head pose information (X = [I eyes ,h head Taking this as an example, let's first look at the binocular region image. Apparent features are obtained after CNN and pooling layers. Head posture information Head pose features are obtained through mapping using the FC layer. Then, the appearance features and head pose features are concatenated and fused, and the gaze features are obtained by mapping using the FC layer. In order to perform a three-dimensional transformation of the gaze features in the feature space, d is generally a multiple of 3 (e.g., d = 48), and is further transformed into Therefore, for sample image X0 and sample image X1, the gaze characteristics can be obtained. and visual characteristics

[0129] The training method for the gaze estimation model provided in this application involves stitching together the appearance features and head pose features corresponding to each of two sample images to obtain the first gaze feature corresponding to the sample image. This provides a prerequisite for training the neural network model to be trained, thereby achieving the goal of obtaining the gaze estimation model.

[0130] Figure 10 The diagram shown is a schematic flowchart of a line-of-sight estimation method provided in an exemplary embodiment of this application. Figure 10 As shown, the line-of-sight estimation method provided in this application embodiment includes the following steps.

[0131] Step S1000: Obtain a single frame of binocular region image of the target object.

[0132] The embodiments of this application do not limit the specific form of the binocular region images. They can be the original captured images, images of the original captured images after preprocessing operations such as noise reduction, or a portion of the original captured images. Furthermore, the object corresponding to the binocular region images can be a driver or other target objects.

[0133] Step S1001: Input a frame of binocular region image into the gaze estimation model, extract features to obtain the gaze features of the target object, and predict the gaze estimation information corresponding to the target object based on the gaze features of the target object.

[0134] Specifically, during the inference process, only a single sample is needed as input to the gaze estimation model. After feature extraction, gaze features are obtained. These gaze features are then used to predict the final gaze direction, which is the gaze estimation information of the target object. The gaze estimation model is generated based on the training method of the gaze estimation model mentioned in the first aspect above.

[0135] The gaze estimation method provided in this application acquires a single frame of binocular region image of a target object; it then processes the binocular region image using a gaze estimation model to obtain gaze estimation information of the target object. Compared with existing gaze estimation methods, this method effectively improves the representation capability of gaze features, expands the effective working range of the gaze estimation system, and enhances the accuracy of gaze direction estimation. It can provide comprehensive and accurate support and basis for analyzing the driver's driving state, thereby further improving vehicle safety during driving and possessing the advantage of wide applicability.

[0136] Figure 11 The diagram shown is a schematic representation of the structure of a training apparatus for a gaze estimation model provided in an exemplary embodiment of this application. Figure 11 As shown, the training device for the gaze estimation model provided in this application embodiment includes a first determining module 1100, a second determining module 1101, a third determining module 1102, and a training module 1103. The first determining module 1100 is used to determine sample image pairs of the target object based on a set of binocular region images of the target object, wherein each sample image pair includes two sample images. The second determining module 1101 is used to determine the first gaze feature corresponding to each of the two sample images. The third determining module 1102 is used to determine the loss function of the neural network model to be trained based on the first gaze feature corresponding to each of the two sample images. The training module 1103 is used to train the neural network model to be trained based on the loss function to obtain the gaze estimation model.

[0137] In one embodiment of this application, the third determining module 1102 is further configured to: determine the feature consistency loss function of the neural network model to be trained based on the first gaze features corresponding to the two sample images; determine the training loss function of the neural network model to be trained; and determine the loss function of the neural network model to be trained based on the feature consistency loss function and the training loss function.

[0138] In one embodiment of this application, the third determining module 1102 is further configured to: determine the gaze direction transformation relationship between two sample images; for each of the two sample images, perform feature transformation on the first gaze feature corresponding to the sample image using the gaze direction transformation relationship to obtain the second gaze feature corresponding to the sample image, wherein the second gaze feature corresponding to the sample image and the first gaze feature represent the same gaze direction; and determine the feature consistency loss function based on the first gaze feature and the second gaze feature corresponding to each of the two sample images.

[0139] In one embodiment of this application, the third determining module 1102 is further configured to: determine the gaze direction labels corresponding to each of the two sample images; determine the rotation matrix corresponding to each of the two sample images based on the gaze direction labels and the reference direction unit vector; determine the transformation matrix between the two sample images based on the rotation matrix corresponding to each of the two sample images; and determine the gaze direction transformation relationship based on the gaze direction labels and the transformation matrix corresponding to each of the two sample images.

[0140] In one embodiment of this application, the third determining module 1102 is further configured to: determine the gaze direction label corresponding to each of the two sample images; determine the predicted gaze direction corresponding to each of the two sample images; for each of the two sample images, determine the gaze angle corresponding to the sample image based on the gaze direction label and the predicted gaze direction, wherein the gaze angle refers to the angle between the gaze direction label and the predicted gaze direction; and determine the training loss function based on the gaze angle corresponding to each of the two sample images.

[0141] In one embodiment of this application, the neural network model to be trained includes a gaze prediction multilayer perceptron. The third determining module 1102 is further configured to perform feature transformation of a preset dimension on the first gaze feature and the second gaze feature corresponding to each of the two sample images; and use the gaze prediction multilayer perceptron to process the first gaze feature and the second gaze feature after feature transformation to obtain the predicted gaze direction corresponding to the sample image.

[0142] In one embodiment of this application, the second determining module 1101 is further configured to: determine the head pose information of the target object; determine the head pose features of the target object based on the head pose information; extract features from the sample image pair to obtain the appearance features corresponding to each of the two sample images; and for each of the two sample images, stitch together the appearance features and head pose features corresponding to the sample image to obtain the first gaze feature corresponding to the sample image.

[0143] Figure 12 The diagram shown is a schematic representation of the line-of-sight estimation device provided in an exemplary embodiment of this application. Figure 12As shown, the gaze estimation device provided in this embodiment includes an acquisition module 1200 and a prediction module 1201. The acquisition module 1200 is used to acquire a frame of binocular region image of a target object. The prediction module 1201 is used to input the frame of binocular region image into a gaze estimation model, perform feature extraction to obtain the gaze features of the target object, and predict the gaze estimation information corresponding to the target object based on the gaze features of the target object; wherein, the gaze estimation model is generated based on the training method of the gaze estimation model mentioned in the first aspect above.

[0144] Exemplary electronic devices

[0145] Below, for reference Figure 13 This describes an electronic device according to embodiments of the present application. Figure 13 The diagram shown is a structural schematic of an electronic device provided in an exemplary embodiment of this application.

[0146] like Figure 13 As shown, the electronic device 130 includes one or more processors 1301 and memory 1302.

[0147] The processor 1301 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 130 to perform desired functions.

[0148] The memory 1302 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1301 may execute the program instructions to implement the training methods of the gaze estimation models of the various embodiments of this application described above, and / or other desired functions. Various contents, such as a set of binocular region images of a target object, may also be stored in the computer-readable storage medium.

[0149] In one example, the electronic device 130 may also include an input device 1303 and an output device 1304, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0150] The input device 1303 may include, for example, a keyboard, a mouse, etc.

[0151] The output device 1304 can output various information to the outside, including a defined line-of-sight estimation model. The output device 1304 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0152] Of course, for the sake of simplicity, Figure 13 Only some of the components of the electronic device 130 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 130 may include any other suitable components depending on the specific application.

[0153] In addition to the methods and apparatus described above, embodiments of this application may also be computer program products, comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the training method and / or gaze estimation method of the gaze estimation model according to various embodiments of this application described above.

[0154] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0155] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the training method and / or gaze estimation method of the gaze estimation model according to various embodiments of this application described above.

[0156] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0157] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0158] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0159] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0160] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0161] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A training method for a gaze estimation model, characterized in that, include: Based on the binocular region image set of the target object, a sample image pair of the target object is determined, wherein the sample image pair includes two sample images; wherein, the binocular region image set of the target object includes: binocular region images of the target object acquired simultaneously by at least two cameras; Determine the first gaze feature corresponding to each of the two sample images; Determine the relationship between the line-of-sight transformations of the two sample images; The first gaze feature corresponding to the first sample image is transformed using the gaze direction transformation relationship to obtain the second gaze feature corresponding to the second sample image; the first gaze feature corresponding to the second sample image is transformed using the gaze direction transformation relationship to obtain the second gaze feature corresponding to the first sample image; wherein, the second gaze feature corresponding to the same sample image and the first gaze feature represent the same gaze direction; Based on the feature consistency loss of the first gaze feature and the second gaze feature corresponding to the first sample image and the feature consistency loss of the first gaze feature and the second gaze feature corresponding to the second sample image, a feature consistency loss function is determined. Determine the training loss function for the neural network model to be trained; Based on the feature consistency loss function and the training loss function, the loss function of the neural network model to be trained is determined; The neural network model to be trained is trained based on the loss function to obtain the gaze estimation model.

2. The training method for the gaze estimation model according to claim 1, characterized in that, Determining the gaze direction transformation relationship between the two sample images includes: Determine the gaze direction label corresponding to each of the two sample images; Based on the gaze direction labels and reference direction unit vectors corresponding to the two sample images, the rotation matrix corresponding to each of the two sample images is determined. Based on the rotation matrices corresponding to the two sample images, the transformation matrix between the two sample images is determined; Based on the gaze direction labels corresponding to the two sample images and the transformation matrix, the gaze direction transformation relationship is determined.

3. The training method for the gaze estimation model according to claim 1, characterized in that, Determining the training loss function of the neural network model to be trained includes: Determine the gaze direction label corresponding to each of the two sample images; Determine the predicted gaze direction corresponding to each of the two sample images; For each of the two sample images, based on the gaze direction label and the predicted gaze direction corresponding to the sample image, the gaze angle corresponding to the sample image is determined. The gaze angle refers to the angle between the gaze direction label and the predicted gaze direction. The training loss function is determined based on the angle of view corresponding to each of the two sample images.

4. The training method for the gaze estimation model according to claim 3, characterized in that, The neural network model to be trained includes a gaze prediction multilayer perceptron, and determining the predicted gaze direction corresponding to each of the two sample images includes: For each of the two sample images, a feature transformation of a preset dimension is performed on the first gaze feature and the second gaze feature corresponding to the sample image; The predicted gaze direction corresponding to the sample image is obtained by using the first gaze feature and the second gaze feature after feature transformation processed by the gaze prediction multilayer perceptron.

5. The training method for the gaze estimation model according to any one of claims 1 to 4, characterized in that, Determining the first gaze feature corresponding to each of the two sample images includes: Determine the head pose information of the target object; Based on the head pose information, the head pose features of the target object are determined; Feature extraction is performed on the sample image pairs to obtain the appearance features corresponding to each of the two sample images; For each of the two sample images, the appearance features corresponding to the sample image and the head pose features are stitched together to obtain the first gaze feature corresponding to the sample image.

6. A line-of-sight estimation method, characterized in that, include: Obtain a single frame of binocular region image of the target object; The gaze estimation model is input into a frame of binocular region image, and the gaze features of the target object are obtained after feature extraction. Based on the gaze features of the target object, the gaze estimation information corresponding to the target object is predicted. The gaze estimation model is generated based on the training method of the gaze estimation model according to any one of claims 1 to 5.

7. A training device for a gaze estimation model, characterized in that, include: The first determining module is used to determine a pair of sample images of the target object based on a set of binocular region images of the target object, wherein the pair of sample images includes two sample images; wherein the set of binocular region images of the target object includes: binocular region images of the target object acquired simultaneously by at least two cameras; The second determining module is used to determine the first gaze feature corresponding to each of the two sample images; The third determining module is used to determine the relationship between the line-of-sight direction transformation between the two sample images; The first gaze feature corresponding to the first sample image is transformed using the gaze direction transformation relationship to obtain the second gaze feature corresponding to the second sample image; the first gaze feature corresponding to the second sample image is transformed using the gaze direction transformation relationship to obtain the second gaze feature corresponding to the first sample image; wherein, the second gaze feature corresponding to the same sample image and the first gaze feature represent the same gaze direction; Based on the feature consistency loss of the first gaze feature and the second gaze feature corresponding to the first sample image and the feature consistency loss of the first gaze feature and the second gaze feature corresponding to the second sample image, a feature consistency loss function is determined. Determine the training loss function for the neural network model to be trained; Based on the feature consistency loss function and the training loss function, the loss function of the neural network model to be trained is determined; The training module is used to train the neural network model to be trained based on the loss function to obtain the gaze estimation model.

8. A line-of-sight estimation device, characterized in that, include: The acquisition module is used to acquire a single frame of binocular region image of the target object; The prediction module is used to input the one-frame binocular region image into the gaze estimation model, extract features to obtain the gaze features of the target object, and predict the gaze estimation information corresponding to the target object based on the gaze features of the target object; wherein, the gaze estimation model is generated based on the training method of the gaze estimation model according to any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for performing the method described in any one of claims 1 to 6.

10. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to perform the method described in any one of claims 1 to 6.

Images