Training, Estimation Method, Device, System and Medium of Line-of-Sight Prediction Model

Abstract

The disclosed embodiments provide a training and estimation method, device, system, and medium for a gaze prediction model. The training method includes obtaining each original facial image containing each pre-identified facial key point; each pre-identified facial key point includes the center of the pupil of each eye; based on a cropping window of a preset size, each original facial image is cropped with each pupil center as the image center to obtain at least one pair of feature images corresponding to each original facial image; head posture information, eye key point information, and gaze angle information are annotated based on each feature image, and training data is formed based on the feature image; each training data is input into a gaze prediction model; and using a loss function, the loss is calculated based on the output of the training data based on the gaze prediction model and the model parameters of the gaze prediction model are updated based on the loss. By constructing concise but feature-rich training data, a gaze prediction model with high prediction accuracy is trained.

Description

Technical Field

[0001] The present disclosure relates to the technical field of line of sight estimation, and in particular to the training, estimation method, device, system and medium of a line of sight prediction model. Background Art

[0002] Gaze state estimation technology helps determine where a person is looking at an object, providing feedback on blink rate and pupil response to various stimuli. Therefore, it helps determine the direction of a person's gaze, and thus, whether their eyes are drowsy, and where their eyes are focused.

[0003] Fatigue driving carries a high risk of causing accidents. Therefore, in the smart car sector, gaze tracking and head posture tracking technologies are used to monitor the driver's gaze area in real time, such as the driver's windshield, passenger windshield, and left rearview mirror, to identify blind spots and provide timely warnings of potential hazards. These warnings are also provided if the driver's eyelids droop (due to drowsiness) or blink less frequently (due to distraction).

[0004] However, the line of sight state estimation method needs to balance efficiency and accuracy, but the current solution still has room for improvement in balancing accuracy and efficiency. Summary of the Invention

[0005] In view of the above-mentioned shortcomings of the prior art, the purpose of the present disclosure is to provide a training and estimation method, device, system and medium for a line of sight prediction model to solve the problems in the related art.

[0006] The first aspect of the present disclosure provides a training method for a gaze prediction model, comprising: obtaining each original face image including each pre-recognized face key point; the pre-recognized face key point includes the pupil center of each eye; based on a cropping window of a preset size, each original face image is cropped with a preset size with each pupil center as the image center to obtain at least one pair of feature images corresponding to each original face image; the at least one pair of feature images includes: a left-eye feature image and a right-eye feature image; based on each of the feature images, head posture information, eye key point information, and gaze angle information are annotated to obtain each pair of feature images carrying the annotations, and training data is formed based on each pair of feature images; each training data is input into a gaze prediction model; wherein the gaze prediction model is configured to output the predicted head posture information, eye key point information, and gaze angle information based on the face input image; using a loss function, the loss is calculated based on the output result of the training data based on the gaze prediction model and the model parameters of the gaze prediction model are updated according to the loss.

[0007] In an embodiment of the first aspect, each original facial image containing pre-recognized facial key points is obtained; the pre-recognized facial key points include the pupil center of each eye, including: determining the center point of the geometric figure surrounded by the pre-recognized facial key points marked on the eye contour as the pupil center.

[0008] In an embodiment of the first aspect, the cropping window based on a preset size is used to crop each original facial image of a preset size with each pupil center as the image center to obtain at least one pair of feature images corresponding to each original facial image, including: based on the relative position of the non-overlapping area of the image cropping result and the original facial image relative to the image center in the four directions of up, down, left and right, determining the required number of pixels to be filled from the boundary of the overlapping area along the four directions, and performing filling with preset pixel values to obtain the feature image; and converting the feature image to the input size of the gaze prediction model to obtain the training data.

[0009] In an embodiment of the first aspect, each of the feature images contains at least 3 / 4 of the image area of the original facial image.

[0010] In an embodiment of the first aspect, the sight line prediction model includes: a first image feature encoding module, a second image feature encoding module, an image feature fusion module, and an image decoding module;

[0011] In an embodiment of the first aspect, the method is configured to perform feature extraction based on the left eye feature image to obtain first feature data;

[0012] In an embodiment of the first aspect, the method is configured to perform feature extraction based on the right eye feature image to obtain second feature data;

[0013] In an embodiment of the first aspect, the first feature data and the second feature data are used to fuse to obtain a fused feature result; and the image decoding module is used to decode the fused feature result to obtain the output result.

[0014] In an embodiment of the first aspect, the image feature encoding module includes at least two convolutional layers, at least one activation layer, and at least one batch normalization layer; wherein each of the activation layers is located between two convolutional layers, and each of the batch normalization layers is located between a convolutional layer and the activation layer; and / or the image decoding module includes: at least two convolutional layers, at least two atrous convolutional layers, and a fully connected layer connected in sequence.

[0015] In an embodiment of the first aspect, the loss calculated by the loss function includes: a weighted sum of a first loss part of the predicted head posture information relative to the labeled head posture information, a second loss part of the predicted eye key point information relative to the labeled eye key point information, and a third loss part of the predicted sight angle information relative to the labeled sight angle information; wherein the weight of the third loss part is higher than the weight of the second loss part, and the weight of the second loss part is higher than the weight of the first loss part.

[0016] A second aspect of the present disclosure provides a line of sight estimation method, comprising: obtaining a face image to be predicted, and obtaining at least one pair of feature images to be predicted based on the face image to be predicted; the at least one pair of feature images includes a left-eye feature image and a right-eye feature image; inputting the at least one pair of feature images to be predicted into a trained line of sight prediction model to obtain corresponding predicted head posture information, eye key point information, and line of sight angle information; wherein the line of sight prediction model is trained by a training method as described in any one of the first aspects; and performing line of sight state estimation based on the predicted head posture information, eye key point information, and line of sight angle information.

[0017] In an embodiment of the second aspect, the line of sight state estimation based on the predicted head posture information, eye key point information, and line of sight angle information includes: judging whether the head posture is abnormal based on the head posture information; if the head posture is abnormal, judging the line of sight state is abnormal; if the head posture is normal, judging whether it is abnormal based on the eye key point information and judging whether it is abnormal based on the line of sight angle information; if it is judged to be abnormal based on the eye key point information and the line of sight angle information, judging the line of sight state is abnormal; otherwise, judging the line of sight state is normal.

[0018] In an embodiment of the second aspect, at least one of the following is included: 1) the head posture information includes: head heading angle, head pitch angle, and head roll angle; the judgment of whether the head posture is abnormal based on the head posture information includes: judging whether any one of the head heading angle, head pitch angle, and head roll angle exceeds a preset angle threshold range; if exceeded, the head posture is judged to be abnormal; 2) the sight line angle information includes eye heading angle and eye pitch angle; the judgment of whether it is abnormal based on the sight line angle information includes: judging whether the eye heading angle and eye pitch angle exceed the preset angle threshold range respectively, if both do not exceed the preset angle threshold range, the sight line state is judged to be normal; otherwise, the sight line state is judged to be abnormal; 3) the eye key point information includes pupil center position information; the judgment of whether it is abnormal based on the eye key point information includes: judging whether the predicted pupil center position information is different from the left eye and the right eye respectively. Whether the deviation distance between the position information of the corresponding pupil centers is greater than a preset distance threshold; if the deviation distances of the left eye and the right eye are both greater than the preset distance threshold, the line of sight state is judged to be abnormal; otherwise, the line of sight state is judged to be normal; 4) the eye key point information includes: at least the eye key points that draw the eye outline; the judgment based on the eye key point information whether it is abnormal includes: corresponding to the left eye and the right eye respectively, calculating the aspect ratio of the eye based on each eye key point and comparing it with the preset aspect ratio threshold; if the aspect ratio of the left eye and the right eye are both greater than the preset aspect ratio threshold, the line of sight state is judged to be abnormal; otherwise, the line of sight state is judged to be normal; 5) in the case that 3) and 4) both judge that the line of sight state is abnormal, determine that the judgment based on the eye key point information is abnormal; 6) the line of sight state estimation method is executed periodically; the number of abnormal line of sight states judged within the preset time range accounts for more than a preset proportion of the total number, and a warning message is generated.

[0019] According to a third aspect of the present disclosure, there is provided a training device for a gaze prediction model, comprising: an image acquisition module for acquiring each original face image including each pre-identified face key point; the pre-identified face key point includes the pupil center of each eye; an image cropping module for cropping each original face image of a preset size with each pupil center as the image center based on a cropping window of a preset size, so as to obtain at least one pair of feature images corresponding to each original face image; the at least one pair of feature images includes a left-eye feature image and a right-eye feature image; a training data acquisition module for annotating head posture information, eye key point information, and gaze angle information based on each feature image, so as to obtain each pair of feature images carrying the annotations, and forming training data based on each pair of feature images; a training module for inputting each training data into a gaze prediction model; wherein the gaze prediction model is configured to output the predicted head posture information, eye key point information, and gaze angle information based on the face input image; and, using a loss function, calculating a loss based on the output result of the training data based on the gaze prediction model and updating the model parameters of the gaze prediction model according to the loss.

[0020] The fourth aspect of the present disclosure provides a line of sight estimation device, comprising: an image acquisition module, used to acquire a face image to be predicted, and obtain at least one pair of feature images to be predicted based on the face image to be predicted; the at least one pair of feature images includes a left-eye feature image and a right-eye feature image; a prediction module, used to input the at least one pair of feature images to be predicted into a trained line of sight prediction model, and obtain corresponding predicted head posture information, eye key point information, and line of sight angle information; wherein the line of sight prediction model is trained by a training method as described in any one of the first aspects; a line of sight state estimation module, used to perform line of sight state estimation based on the predicted head posture information, eye key point information, and line of sight angle information.

[0021] A fifth aspect of the present disclosure provides an electronic device, comprising: a processor and a memory; the memory stores program instructions; the processor is used to run the program instructions to execute the training method of the line of sight prediction model as described in any one of the first aspects, or to execute the line of sight estimation method as described in any one of the second aspects.

[0022] The sixth aspect of the present disclosure provides a line of sight monitoring system, which is installed in a vehicle and includes: a camera device, which is provided in the vehicle and is configured to capture an original image of the driver's face; a control device, which includes: a communicator, a processor and a memory; the memory stores program instructions; the communicator is communicatively connected to the camera device; the processor is connected to the memory and the communicator, and is used to run the program instructions to execute the line of sight estimation method as described in any one of the second aspects based on the original image of the face read from the camera device, so as to obtain an estimation result of the driver's line of sight state.

[0023] A seventh aspect of the present disclosure provides a computer-readable storage medium, characterized in that program instructions are stored therein, and the program instructions are run to execute the training method of the line of sight prediction model as described in any one of the first aspects, or to execute the line of sight estimation method as described in any one of the second aspects.

[0024] As described above, the present disclosure provides a training, estimation method, device, system, and medium for a gaze prediction model. The training method includes obtaining each original facial image containing each pre-identified facial key point; each pre-identified facial key point includes the pupil center of each eye; based on a cropping window of a preset size, each original facial image is cropped with each pupil center as the image center to obtain at least one pair of feature images corresponding to each original facial image; head posture information, eye key point information, and gaze angle information are annotated based on each feature image, and training data is formed based on each pair of feature images; each training data is input into a gaze prediction model; using a loss function, the loss is calculated based on the output result of the training data based on the gaze prediction model and the model parameters of the gaze prediction model are updated based on the loss. By constructing concise but feature-rich training data, a gaze prediction model with high prediction accuracy is trained. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A flowchart illustrating a method for training a gaze prediction model in accordance with an embodiment of the present disclosure is shown.

[0026] Figure 2 A schematic diagram showing feature image capture in one embodiment of the present disclosure.

[0027] Figure 3 A schematic diagram showing the structure of a sight line prediction model in one embodiment of the present disclosure.

[0028] Figure 4 A schematic diagram showing predicted eye key point information in one embodiment of the present disclosure.

[0029] Figure 5 A schematic diagram showing an example of executing a method for training a gaze prediction model in one embodiment of the present disclosure.

[0030] Figure 6 A flowchart illustrating a method for estimating a line of sight state in an embodiment of the present disclosure is shown.

[0031] Figure 7 A schematic diagram showing the flow of line of sight state judgment in one embodiment of the present disclosure.

[0032] Figure 8 A schematic diagram showing a module of a training device for a gaze prediction model in one embodiment of the present disclosure.

[0033] Figure 9 A module diagram showing a sight line state estimation device according to an embodiment of the present disclosure is shown.

[0034] Figure 10 A schematic diagram showing the circuit structure of an electronic device in one embodiment of the present disclosure is shown.

[0035] Figure 11 A schematic structural diagram of a sight line monitoring system in one embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0036] The following describes the embodiments of the present disclosure through specific examples. Those skilled in the art can easily understand the other advantages and effects of the present disclosure from the information disclosed in this disclosure. The present disclosure can also be implemented or applied through different specific embodiments. The details of the present disclosure can also be modified or changed according to different viewpoints and application modules without departing from the spirit of the present disclosure. It should be noted that the embodiments and features in the embodiments of the present disclosure can be combined with each other unless there is a conflict.

[0037] The following is a detailed description of the embodiments of the present disclosure with reference to the accompanying drawings so that those skilled in the art can easily implement the present disclosure. The present disclosure can be embodied in many different forms and is not limited to the embodiments described herein.

[0038] Throughout the present disclosure, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present disclosure. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or a group of embodiments or examples. Furthermore, those skilled in the art may combine and integrate different embodiments or examples, and features of different embodiments or examples, as described in the present disclosure, without conflicting requirements.

[0039] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the context of this disclosure, "a group" means two or more, unless otherwise specifically defined.

[0040] In order to clearly describe the present disclosure, components not related to the description are omitted, and the same or similar components throughout the specification are denoted by the same reference numerals.

[0041] Throughout this specification, when a device is said to be "connected" to another device, this includes not only "direct connection" but also "indirect connection" with other elements interposed therebetween. Furthermore, when a device is said to "include" a certain component, unless otherwise stated, this does not exclude the inclusion of other components but rather implies that the device may include other components.

[0042] Although the terms first, second, etc. are used in this document to represent various elements in some examples, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, the first interface and the second interface, etc. are represented. Furthermore, as used in this document, the singular forms "one," "an," and "the" are intended to also include the plural forms, unless there is a contrary indication in the context. It should be further understood that the terms "comprise" and "include" indicate the presence of the described features, steps, operations, elements, modules, projects, types, and / or groups, but do not exclude the presence, occurrence, or addition of one or a group of other features, steps, operations, elements, modules, projects, types, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Therefore, "A, B, or C" or "A, B, and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B, and C." Exceptions to this definition only occur when the combination of elements, functions, steps, or operations is inherently mutually exclusive in some way.

[0043] The technical terms used herein are intended only to refer to specific embodiments and are not intended to limit the present disclosure. The singular form used herein also includes the plural form, unless the statement explicitly indicates otherwise. The term "comprising" as used in this specification specifies specific features, regions, integers, steps, operations, elements, and / or components, and does not exclude the presence or addition of other features, regions, integers, steps, operations, elements, and / or components.

[0044] Although not defined differently, all terms used herein, including technical and scientific terms, have the same meanings as those generally understood by those skilled in the art to which this disclosure belongs. Terms defined in commonly used dictionaries are additionally interpreted as having meanings consistent with relevant technical literature and the current message. Unless otherwise defined, they should not be overly interpreted as ideal or highly formalized meanings.

[0045] Estimating the human eye's gaze has always been a crucial technology. For example, in the field of intelligent driving, drivers' gaze estimation can be used to determine whether they are drowsy, asleep, or distracted, which could lead to traffic accidents, and thus provide warnings. Another example is in education, where cheating can be detected by estimating a candidate's gaze during an exam. Alternatively, in offline or online classrooms, students' gaze estimation can be used to determine whether they are distracted and provide warnings, which can help improve learning efficiency.

[0046] Although current technology has developed to the point where neural network models can be used to predict a user's line of sight, since neural network models require training data, the selection of training data must take into account both the accuracy of model predictions and the efficiency of model training. Current solutions still have room for improvement in balancing accuracy and efficiency.

[0047] In view of this, some embodiments of the present disclosure may provide a method for training a line of sight prediction model to solve the problems of model training and prediction accuracy.

[0048] When people are tired, they will have more typical facial expressions or movement characteristics, such as longer blinking duration, slower eyelid movement, nodding, yawning, etc. These characteristics are closely related to eye characteristics. Therefore, this technical solution proposes a gaze state estimation method based on head posture constraints to determine whether the gaze state is abnormal.

[0049] like Figure 1 , which is a flow chart showing a method for training a gaze prediction model in one embodiment of the present disclosure.

[0050] exist Figure 1 , the training method includes:

[0051] Step S101: Acquire each original face image containing each pre-recognized face key point; each pre-recognized face key point includes the pupil center of each eye.

[0052] For example, the original face image can be a picture. Further, the original face image can be a color picture or a black and white picture taken by an infrared camera, which is not limited here.

[0053] For example, see Figure 2 As shown, 68 key points pre-annotated in the original face image are displayed, including key points corresponding to facial features, eyebrows, and facial contours. Each of the pre-annotated key points can be automatically annotated by software, such as by using an algorithm based on facial key point recognition to identify and annotate key points in the original face image, or can also be obtained by manual annotation.

[0054] In some embodiments, the center point of the geometric figure surrounded by the pre-identified facial key points marked on the eye contour can be determined as the pupil center. Figure 2 The center point of the key points 44, 45, 47, and 48 of the right eye is used as the pupil center of the eye, as shown by the circular point center1 in the figure.

[0055] In some embodiments, each of the original facial images may be normalized to a uniform size, such as height and width.

[0056] Step S102: Based on a cropping window of a preset size, cropping each original face image with a preset size with the center of each pupil as the image center, so as to obtain at least one pair of feature images corresponding to each original face image.

[0057] The at least one pair of characteristic images includes: a left-eye characteristic image and a right-eye characteristic image. The following describes the reason for intercepting the left and right pair of characteristic images.

[0058] Since the line of sight of the two eyes is consistent, this correlation will be ignored if the images of the two eyes are processed separately. Therefore, when making a training data set, as shown in steps S101 and S102, the center of the pupil of the left and right eyes is used as the center to intercept the feature image with a height Hc and a width Wc to form the training data. In addition, most current line of sight estimation schemes do not use head information as reference information for line of sight estimation. However, in reality, head posture has a strong correlation with line of sight estimation. When the head changes, the line of sight will also change. Therefore, when intercepting the feature image, its size should be considered to be able to bring in enough head posture feature data. For example, the size of each feature image after interception can be at least 3 / 4 of the original face image.

[0059] like Figure 2 The figure shows an example of capturing the right eye feature image based on the pupil center of the right eye. Specifically, for the original face image A, we can use the pupil center center1 as the center, and capture an area with a height of Hc / 2 above and below, and a width of Wc / 2 on the left and right, i.e., window B.

[0060] exist Figure 2In the example, the overlapping area of A and B (size is wc2*hc2) has the data of the original face image, while the non-overlapping area of A and B (that is, the area outside A) has no data. Generally speaking, the input width and height of the neural network model are generally consistent, which is determined by the characteristics of the neural network. In other words, after obtaining the feature image, the captured image needs to be resized to scale it to the input size of the neural network model. However, for this non-overlapping area of B outside of A, if the overlapping part is directly scaled in width and height, this violent operation will cause the image to deform, and the eyes and face will be distorted, which is very unfavorable for model convergence and reduces the effect. Therefore, the non-overlapping area can be filled with pixels.

[0061] In some embodiments, the filled pixel value may be a preset value, such as a gray pixel with the pixel point (R, G, B) being (114, 114, 114), or a black pixel with the pixel point (0, 0, 0).

[0062] In some embodiments, based on the relative position of the non-overlapping area of the image capture result and the original facial image relative to the center of the image in the four directions of up, down, left and right, the required number of pixels to be filled from the boundary of the overlapping area along the four directions can be determined, and filling with preset pixel values can be performed to obtain the feature image.

[0063] As an example, let's assume the bounding box of the original facial image is gt, the height of the original facial image is Height, and the width is Weight. A portion of the facial image is captured with the left and right pupil centers as the center points, i.e., the feature images have a height of Hc and a width of Wc. The pupil center coordinates are set to (X, Y). The following calculation can be used to determine the number of pixels to be padded in the left, right, top, and bottom directions. The number of padded pixels to the left is the left parameter, the right parameter is the right parameter, the top parameter is the top parameter, and the bottom parameter is the bottom parameter:

[0064] left=0ifX>Wc / 2elseWc / 2-X (If X>Wc / 2, the left amount is 0, otherwise the left amount is Wc / 2-X, and the same principle applies to the following)

[0065] right=0ifWeight-X>Wc / 2elseWc / 2-Weight+X

[0066] Top=0ifY>Hc / 2elseHc / 2-Y

[0067] Bottom=0ifHeight-Y>Hc / 2elseHc / 2-Height+Y

[0068] Based on these four parameters, the number of pixels to be padded in four directions based on the size of the original face image can be specified. The padded pixel values can be (114, 114, 114).

[0069] Specifically, the original image is expanded to a width of Weight + left + right and a height of Height + left + right. Then, based on the pupil center coordinates (X, Y), the boundary coordinates of the feature image are determined upward, downward, left, and right, i.e., (x1, y1, x2, y2).

[0070] x1=X-Wc / 2;

[0071] y1=Y-Hc / 2;

[0072] x2=X+Wc / 2;

[0073] y2=Y+Hc / 2.

[0074] In this way, the left eye feature image and the right eye feature image after pixel filling can be obtained, and then converted to the input size of the gaze prediction model.

[0075] In some embodiments, one or more pairs of left-eye feature images and right-eye feature images may be generated based on each original facial image, for example, based on scale changes of the feature images, and is not limited to a pair of feature images.

[0076] Step S103 labels head posture information, eye key point information, and sight angle information based on each of the feature images to obtain each pair of feature images carrying the labels, and forms training data based on each pair of the feature images.

[0077] In some embodiments, the feature image can be annotated using a facial key point annotation software tool or manually. For example, the eye key point information is about the key points of the eyelid, iris, pupil, pupil center, etc. Figure 4 As shown, there are 16 key points for each eyelid, 8 key points for the iris, 8 key points for the pupil, and 1 key point for the pupil center, for a total of 33 key points. Exemplarily, the sight angle information can include 2 angles, the heading angle (yaw) and pitch angle (pitch) of the sight direction. The sight angle and the deflection angle of the eye are uniquely related, so the heading angle (yaw) and pitch angle (pitch) of the sight direction can be defined as the heading angle (yaw) and pitch angle (pitch) of the eye. Exemplarily, the head posture information can include 3 angles, namely the heading angle, pitch angle, and roll angle.

[0078] A pair of feature images obtained from each original face image is a piece of training data, and the training data set of multiple original face images forms a training data set.

[0079] Step S104: Input each training data into the sight line prediction model.

[0080] The gaze prediction model is configured to output predicted head pose information, eye key point information, and gaze angle information based on the face input image. The head pose information, eye key point information, and gaze angle information annotated in step S104 can be used as the ground truth when calculating the loss.

[0081] like Figure 3 , which shows a functional module diagram of the sight line prediction model 300 described in one embodiment of the present disclosure.

[0082] exist Figure 3 In the embodiment, the sight line prediction model 300 includes: a first image feature encoding module 301, a second image feature encoding module 302, an image feature fusion module 303, and an image decoding module 304.

[0083] The first image feature encoding module 301 is configured to extract features based on the left eye feature image to obtain first feature data. In some embodiments, the image feature encoding module includes at least two convolutional layers, at least one activation layer, and at least one batch normalization (BN) layer. Each activation layer is located between two convolutional layers, and each batch normalization layer is located between a convolutional layer and an activation layer. Convolutional layers are used to perform convolution calculations to extract features from previous outputs. The feature image of the training data is input into the gaze prediction model 300, where each convolutional layer performs increasingly deeper feature extraction, including both low-level semantic features and high-level semantic features. Low-level semantic features have higher resolution and contain more location and detail information, corresponding to the location and detail information features of the eye image. High-level semantic features have stronger semantic information but lower resolution and poorer detail perception. From low-level semantic features to high-level semantic features, the process becomes increasingly abstract but also increasingly global. For example, the activation layer includes an activation function, which can be ReLU, Leaky, or Maxout, all of which are nonlinear functions. The batch normalization layer, located after the convolutional layer, limits the value range of the convolutional layer's output to between 0 and 1. This prevents the amplitude from being too large or too small when used as input to subsequent layers, thus preventing issues such as gradient vanishing that can occur with increasing layers.

[0084] In a similar principle, the second image feature encoding module 302 can be used to extract features based on the right eye feature image to obtain second feature data. In some embodiments, the first image feature encoding module 301 and the second image feature encoding module 302 can be two independent branch networks, which are respectively dedicated to processing the left eye feature image and the right eye feature image to improve the accuracy of feature extraction. In some embodiments, with respect to a pair of feature images obtained based on each original face image, one of the feature images can be first input into a corresponding image feature encoding module to obtain a feature data, and then the other feature image can be input into another corresponding image feature encoding module to obtain another feature data, and then the two feature data can be input into the image feature fusion module 303 together. For example, the left eye feature image is first input into the first image feature encoding module 301 to obtain the first feature data, and then the right eye feature image is input into the second image feature encoding module 302 to obtain the second feature data, and then the first feature data and the second feature data are input into the image feature fusion module 303. Alternatively, in another embodiment, the left-eye feature image and the right-eye feature image may be input in parallel to the first image feature encoding module 301 and the second image feature encoding module 302 respectively, and it is only necessary to ensure that the association index between the feature data of the left-eye feature image and the right-eye feature image belonging to the same pair is not lost.

[0085] As an example, the first image feature encoding module 301 and the second image feature encoding module 302 can be implemented based on a feature extraction network, such as MobilenetV1, MobilenetV2, etc., or can be implemented using, for example, a deep multi-layer residual network ResNEt.

[0086] The image feature fusion module 303 is used to fuse the first feature data and the second feature data to obtain a fused feature result. As mentioned above, each pair of left-eye feature images and right-eye feature images contains rich head posture information. Therefore, after the two feature images are separately feature extracted, feature fusion is performed to fuse the features of the left and right eyes with the head features. The semantic information is richer, and the convergence of the model in line of sight estimation is increased. At the same time, the eyes also have a certain constraint on the head posture estimation. The two gain each other. This feature not only aggregates the features between different layers, but also ensures the integrity and diversity of the features.

[0087] In some embodiments, the first feature data and the second feature data may include a feature map of size H×W×N obtained based on a feature image of the input training data, where H is the height, W is the width, and N is the number of channels, which may be determined by the size and number of convolution kernels of the convolution layer and the structure of each layer of the feature extraction network. As an example, the image feature fusion module 303 may perform fusion by simple concatenation, i.e., concatenating the left and right feature maps of the same size.

[0088] The image decoding module 304 is configured to decode the fused feature result to obtain the output result. In some embodiments, the image decoding module 304 includes: at least two sequentially connected convolutional layers, at least two dilated convolutional layers, and a fully connected layer. Dilated convolution, also known as expanded convolution or dilated convolution, is simply the process of enlarging the convolution kernel by adding spaces (zeros) between the kernel elements. Dilated convolution can control the receptive field while maintaining the size of the feature map, thereby extracting multi-scale information. Compared to downsampling operations such as pooling, which lose information in the feature map, dilated convolution can increase the receptive field and reduce the amount of computation (adding zeros to the convolution kernel effectively leaves the number of parameters unchanged). Specifically, the feature map is first obtained through each convolutional layer, then through the dilated convolution layer, and finally through the fully connected layer to obtain the prediction result. The prediction result is in the form of a multidimensional vector, for example, a 76-dimensional vector, including 33 key points and 66 data points; 2 data points for the line of sight angle; and 3 head posture angles.

[0089] Based on the above, the line of sight prediction model 300 extracts features of the left and right eyes and their respective parts of the head posture through the first image feature encoding module 301 and the second image feature encoding module 302, fuses the left and right eye features and the head posture features in the image feature fusion module 303 to obtain fused feature data, and predicts the eye key point information, line of sight angle, head posture angle and other results based on the fused feature data in the image decoding module 304.

[0090] Step S105: using a loss function, calculating a loss based on the output result of the sight line prediction model according to the training data and updating the model parameters of the sight line prediction model according to the loss.

[0091] In some embodiments, the loss calculated by the loss function includes: a weighted sum of a first loss part of the predicted head posture information relative to the labeled head posture information, a second loss part of the predicted eye key point information relative to the labeled eye key point information, and a third loss part of the predicted sight angle information relative to the labeled sight angle information.

[0092] Exemplarily, during training, the output variables all belong to regression problems, so the loss function may adopt, for example, the mean square error loss function (MSE).

[0093] In this task, gaze angle is the most important, eye keypoints are second, and head pose state loss is third. Therefore, the weight of the third loss component is higher than the weight of the second loss component, and the weight of the second loss component is higher than the weight of the first loss component. For example, the weight of the third loss component of gaze angle is 20, the weight of the second loss component is 10, and the weight of the first loss component is 5.

[0094] It can be understood that the line of sight prediction model is repeatedly trained using the training data in the training dataset until the loss converges to complete the training.

[0095] It should be noted that, as an example, the training data of the gaze prediction model uses 33 eye key points for training and prediction. Figure 4 As shown in the figure, the 33 eye key points contain enough eye parameter information, which can make the model converge better and more robust, that is, the predicted results are more accurate, and the accuracy is also better when the predicted results are used to estimate the line of sight state.

[0096] To explain the principle of the above method in a simple and intuitive way, you can also refer to Figure 5 , which is a schematic diagram showing an example of executing a training method for a gaze prediction model in one embodiment of the present disclosure.

[0097] As an example, in Figure 5 In the paper, the front-end face detection model is used to detect the face position information in the original face image, and then the face position information is input into the face key point model to infer the face key points (including the pupil center point), and 68 key points are obtained. Figure 2 As shown. The original face image with pre-labeled facial key points is as follows Figure 2 In the example, a pair of left-eye feature images and right-eye feature images are captured with the center of the left pupil and the center of the right pupil as the center, and the non-overlapping area of the captured window and the original face image is padded with pixels such as (114, 114, 114). Using an annotation tool (which can be automatic or manual), the left-eye feature image and the right-eye feature image are annotated with eye key point information (33 points for the left eye + 33 points for the right eye = 66 key points), gaze angle information (2 angles), and head posture information (3 angles). Each pair of annotated left-eye feature images and right-eye feature images are adjusted to the input size of the gaze prediction model and used as training data to form a training dataset. The gaze prediction model is trained based on the training dataset, and the parameters are updated by backpropagation according to the loss function until training is completed. The gaze prediction model predicts eye key point information, gaze angle information, and head posture information based on the input of the left and right eye feature images.

[0098] Therefore, the trained gaze prediction model can be used to estimate the gaze state of people in actual scenes.

[0099] like Figure 6 , which is a flow chart showing a line of sight estimation method in one embodiment of the present disclosure.

[0100] exist Figure 6 , the sight line estimation method includes:

[0101] Step S601: Acquire a face image to be predicted, and obtain at least one pair of feature images to be predicted based on the face image to be predicted; the at least one pair of feature images includes a left-eye feature image and a right-eye feature image.

[0102] For example, in a driving scenario, a camera inside the car captures the driver's face. Alternatively, in an educational scenario, a camera in an offline classroom or on an online electronic terminal (such as a computer, laptop, tablet, or mobile phone) captures the student's face.

[0103] Further, please refer to Figure 5 , step S601 also includes obtaining 68 key points through pre-labeling of the face image to be predicted, and Figure 2 The principle shown is based on intercepting and filling the centers of the left and right pupils in the face image to be predicted to obtain a pair of feature images to be predicted.

[0104] Exemplarily, the gaze estimation method may be executed periodically at a preset frequency, and the facial image to be predicted may be acquired periodically.

[0105] Step S602: inputting the at least one pair of feature images to be predicted into the trained gaze prediction model to obtain the corresponding predicted head posture information, eye key point information, and gaze angle information.

[0106] The sight line prediction model is trained by the sight line prediction model training method in the previous embodiment (such as Figure 1 ) is obtained through training.

[0107] For example, the gaze prediction model outputs 33 key points of the left eye, 33 key points of the right eye, the heading and pitch angles of the eye gaze, and the heading, pitch, and roll angles of the head posture based on the facial image to be predicted. Based on this information, it can determine whether the user has an abnormal gaze state.

[0108] Step S603: Estimating the gaze state based on the predicted head posture information, eye key point information, and gaze angle information.

[0109] In some embodiments, when head posture is abnormal, it's often impossible to fully capture eye parameters. In this case, head posture is used directly to determine gaze status. Only when head posture is normal are gaze and eye key points used for judgment. This configuration maximizes accuracy and robustness, avoiding misjudgments. Therefore, head posture information can be prioritized for determining gaze status anomalies. If head posture information is normal, judgments can be made based on eye key points and gaze angle information, respectively.

[0110] Therefore, if Figure 7 As shown, step S603 may exemplarily include:

[0111] Step S701: determining whether the head posture is abnormal based on the head posture information;

[0112] If the head posture is abnormal, proceed to step S702: determine whether the line of sight state is abnormal; if the head posture is normal, proceed to step S703.

[0113] Step S702: Determine whether there is an abnormality based on the eye key point information and determine whether there is an abnormality based on the sight angle information.

[0114] If the eye key point information determines that the state is abnormal, and the line of sight angle information determines that the state is abnormal, then go to step S703: determine that the line of sight state is abnormal; otherwise, go to step S704: determine that the line of sight state is normal.

[0115] As an example, the principles of judging whether the line of sight state is abnormal based on head posture information, eye key point information, and line of sight angle information are respectively explained.

[0116] In some examples, the head posture information includes: a head heading angle, a head pitch angle, and a head roll angle. Determining whether the head posture is abnormal based on the head posture information includes: determining whether any one of the head heading angle, the head pitch angle, and the head roll angle exceeds a preset angle threshold range; if so, determining that the head posture is abnormal.

[0117] As a specific example, assuming that the three predicted head posture angles are yaw_head, pitch_head, and roll_head, and the corresponding preset angle threshold ranges are yaw_head_thre1~yaw_head_thre2, pitch_head_thre1~pitch_head_thre2, and roll_head_thre1~roll_head_thre, respectively. If it is determined in step S701 that yaw_head, pitch_head, and roll_head all fall within their respective preset threshold ranges, indicating that the head posture is normal, then step S701 is entered for judgment; otherwise, it can be directly determined that the line of sight state is abnormal.

[0118] In some examples, the sight angle information includes an eye heading angle and an eye pitch angle. Determining whether the sight angle information is abnormal based on the sight angle information includes: determining whether the eye heading angle and the eye pitch angle exceed a preset angle threshold range, respectively; if both do not exceed the preset angle threshold range, determining that the sight state is normal; otherwise, determining that the sight state is abnormal.

[0119] As a specific example, assume the predicted eye yaw angle is y_eye and the predicted eye pitch angle is p_eye. The preset yaw and pitch angle thresholds are yaw_eye_thre1 to yaw_eye_thre2 and pitch_eye_thre1 to pitch_eye_thre2, respectively. These thresholds may differ from the previously set yaw_head_thre1 to yaw_head_thre2 and pitch_head_thre1 to pitch_head_thre2. If both y_eye and p_eye are within their respective preset thresholds, the gaze state is considered normal; otherwise, it is considered abnormal.

[0120] In some examples, the eye key point information includes pupil center position information. Determining whether an abnormality exists based on the eye key point information includes: determining, for each of the left and right eyes, whether a deviation distance between the predicted pupil center position information and the corresponding pupil center position information is greater than a preset distance threshold; if the deviation distances for both the left and right eyes are greater than the preset distance threshold, determining that the sight state is abnormal; otherwise, determining that the sight state is normal.

[0121] As a specific example, suppose the pupil center point centerh coordinates (xh, yh) predicted by the model, and the pupil center point obtained by the front-end face detection model based on the geometry of the eye key points is center1 (see Figure 2), calculate the distance between centerh and center1 and compare it to the preset threshold distance, distance_thre. If the distance is greater than distance_thre, it indicates that the iris is deviated too much. If the distance between the pupil centers of both eyes is greater than the preset distance, the gaze state is considered abnormal; otherwise, the gaze state is considered normal.

[0122] In some examples, the eye key point information includes: at least the key points of the eyes that draw the eye outline, such as Figure 4 The 33 key points of a single eye in the image are used. Determining whether the gaze state is abnormal based on the eye key point information includes: calculating the eye aspect ratio based on each of the eye key points for the left and right eyes, respectively, and comparing the eye aspect ratio with a preset aspect ratio threshold; if the aspect ratios of both the left and right eyes are greater than the preset aspect ratio threshold, determining that the gaze state is abnormal; otherwise, determining that the gaze state is normal.

[0123] As a specific example, the aspect ratio of the eyes is calculated based on the eye key points in the facial key points. Figure 4 As shown, taking the inferred key points of the right eye as an example, first determine the width of the eye based on point 0 and point 8, determine the height of the eye based on point 18 and point 22, and then calculate the aspect ratio of each eye and compare it with the preset aspect ratio threshold. The aspect ratio of the eyes is used to determine the degree of openness and closure of the eyes. Compared with using the absolute height alone, the influence of the zoom scale of the eyes due to distance can be ignored. The closer to the closed state, the smaller the eye height and the larger the aspect ratio. Therefore, if the aspect ratio is greater than the preset aspect ratio threshold, it means that the eyes may be in a closed state. When both eyes are in a closed state, it is judged that the line of sight state is abnormal. If a single eye is closed, it cannot be determined that the line of sight state is abnormal.

[0124] In some examples, the line of sight state estimation method is executed periodically. If the number of abnormal line of sight states determined within a preset time range exceeds a preset proportion of the total number, a warning message is generated. For example, within time T, if the number of abnormal line of sight states accounts for more than 80% (modifiable) of the total number of line of sight states, a warning is issued. The statistical results of multiple abnormal line of sight states in the time dimension are used to obtain a more accurate judgment of the line of sight state, that is, to more accurately determine whether the line of sight is abnormal. Avoid misjudgment of abnormality due to events such as a high frequency of blinking by the user within a certain period of time.

[0125] Combined with the above examples, give a corresponding Figure 7A specific example of the process. Suppose the above head posture judgment is A, the judgment of the sight angle information is B, the judgment of the offset of the pupil center is C, the judgment of the eye aspect ratio is D, and the judgment of the proportion of abnormal sight state within time T is E. First, the head posture (A) is judged. If it is abnormal, the sight state is judged to be abnormal; if the head posture (A) is judged to be normal, the sight angle (B) and the eye key point information (C, D) are judged separately. The eye key point information judgment includes two groups of judgments of key point 1 (C, pupil center) and key point 2 (D, aspect ratio of other key points). When the judgment results of C and D are both abnormal, the output sight state of the key point judgment is considered to be abnormal, otherwise it is normal. Among them, E is also executed. Within time T, if the abnormal sight state accounts for more than 80% of the total state (replaceable), a warning is issued.

[0126] like Figure 8 As shown, a module diagram of a training device for a sight line prediction model in one embodiment of the present disclosure is shown. It should be noted that the principle and technical implementation of the training device for the sight line prediction model can refer to the training method embodiment of the sight line prediction model in the previous embodiment (for example Figure 1 ), so it will not be repeated in this embodiment.

[0127] exist Figure 8 In the embodiment, the training device 800 includes:

[0128] The image acquisition module 801 is used to acquire each original face image containing each pre-recognized face key point; the pre-recognized face key point includes the pupil center of each eye.

[0129] An image capture module 802 is configured to capture each original facial image of a preset size using a capture window of a preset size, with the center of each pupil as the image center, to obtain at least one pair of feature images corresponding to each original facial image; the at least one pair of feature images comprising a left eye feature image and a right eye feature image;

[0130] The training data acquisition module 803 is used to annotate head posture information, eye key point information, and sight angle information based on each feature image to obtain each pair of feature images carrying annotations, and form training data based on each pair of feature images.

[0131] The training module 804 is used to input each training data into the gaze prediction model; wherein the gaze prediction model is configured to output the predicted head posture information, eye key point information, and gaze angle information based on the face input image; and, using a loss function, calculate the loss based on the output results of the training data based on the gaze prediction model and update the model parameters of the gaze prediction model according to the loss.

[0132] like Figure 9 As shown, a schematic diagram of a module of a sight line estimation device in one embodiment of the present disclosure is shown. It should be noted that the principle and technical implementation of the sight line estimation device can refer to the sight line estimation method embodiment in the previous embodiment (for example Figure 6 ), so it will not be repeated in this embodiment.

[0133] exist Figure 9 In the embodiment, the sight line estimation device 900 includes:

[0134] The image acquisition module 901 is used to acquire a face image to be predicted and obtain at least one pair of feature images to be predicted based on the face image to be predicted; the at least one pair of feature images includes a left-eye feature image and a right-eye feature image.

[0135] The prediction module 902 is used to input at least one pair of feature images to be predicted into a trained gaze prediction model to obtain the corresponding predicted head posture information, eye key point information, and gaze angle information; wherein, the gaze prediction model is trained by a training method as in any of the previous embodiments.

[0136] The sight line state estimation module 903 is used to estimate the sight line state based on the predicted head posture information, eye key point information, and sight line angle information.

[0137] It should be noted that in Figure 8 and Figure 9 The various functional modules in the embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, they can be implemented in whole or in part in the form of a program instruction product. The program instruction product includes one or a group of program instructions. When the program instruction instructions are loaded and executed on a computer, the process or function according to the present disclosure is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The program instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium.

[0138] and, Figure 8 and Figure 9The devices disclosed in the embodiments can be implemented using other module division methods. The device embodiments shown above are merely illustrative. For example, the module division is merely a logical functional division. In actual implementation, other division methods may be used, such as a group of modules or modules that can be combined or dynamically integrated into another system, or some features that can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between devices or modules shown or discussed can be through some interface, and the indirect coupling or communication connection between devices or modules can be electrical or other forms.

[0139] in addition, Figure 8 and Figure 9 Each functional module and submodule in the embodiments may be dynamically integrated into a single processing component, each module may exist physically independently, or two or more modules may be dynamically integrated into a single component. The aforementioned dynamic components may be implemented in hardware or as software functional modules. If the aforementioned dynamic components are implemented as software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium. The storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc.

[0140] It should be noted that the processes or methods represented by the flowcharts of the above embodiments of the present disclosure can be understood as modules, segments, or portions of code that include one or more sets of executable instructions configured to implement specific logical functions or steps of a process. Furthermore, the scope of the preferred embodiments of the present disclosure includes alternative implementations in which functions may be performed in a different order than that shown or discussed, including performing functions substantially simultaneously or in reverse order depending on the functions involved.

[0141] For example, Figure 1 、 Figure 6 The order of the steps in the method embodiments may be changed in specific scenarios and is not limited to the above.

[0142] like Figure 10 FIG. 1 is a schematic diagram showing the structure of an electronic device in an embodiment of the present disclosure.

[0143] The electronic device can execute computer program instructions to perform the following Figure 1 or Figure 6 For example, the electronic device may be a distributed computing node system, a server group / server, a desktop computer, a laptop computer, etc., so as to run the method as follows: Figure 1Alternatively, the electronic device may be a vehicle-mounted controller, or a cloud-based server / server group, a distributed computing node system, etc. that remotely communicates with a local terminal (such as a terminal used by students for education, or a vehicle-mounted controller, etc.), and executes Figure 6 The gaze state estimation method in the embodiment is to estimate the gaze state based on the captured face image of the student or driver and obtain the result of whether the gaze state is abnormal.

[0144] The electronic device 1000 includes a bus 1001, a processor 1002, and a memory 1003. The processor 1002 and the memory 1003 can communicate with each other via the bus 1001. The memory 1003 can store program instructions. The processor 1002 executes the program instructions in the memory 1003 to implement the method steps in the previous embodiment, such as Figure 1 or Figure 6 The method in .

[0145] Bus 1001 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of illustration, although only one thick line is used in the figure, this does not mean that there is only one bus or only one type of bus.

[0146] In some embodiments, the processor 1002 may be implemented as a central processing unit (CPU), a microprocessor unit (MCU), a system on a chip (SoC), or a field programmable gate array (FPGA). The memory 1003 may include volatile memory, such as random access memory (RAM), for temporarily storing data while running a program.

[0147] The memory 1003 may also include a non-volatile memory (non-volatile memory) for data storage, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid-state disk (SSD).

[0148] In some embodiments, the electronic device 1000 may further include a communicator 1004. The communicator 1004 is used for external communication. In a specific embodiment, the communicator 1004 may include one or a group of wired and / or wireless communication circuit modules. For example, the communicator 1004 may include one or more of a wired network card, a USB module, a serial interface module, and the like. The wireless communication protocols followed by the wireless communication module include, for example, one or more of: near field communication (NFC) technology, infrared (IR) technology, Global System for Mobile communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), Bluetooth (BT), Global Navigation Satellite System (GNSS), etc.

[0149] like Figure 11 As shown, for vehicle driving scenarios, an eye-line monitoring system can also be provided in the embodiment of the present disclosure. The eye-line monitoring system can be installed in a vehicle.

[0150] The sight line monitoring system 1100 includes:

[0151] The camera device 1101 is installed in the vehicle and configured to capture the original image of the driver's face. The camera device 1101 may include one or multiple cameras at different locations.

[0152] The control device 1102 includes a communicator, a processor, and a memory. The memory stores program instructions; the communicator is connected to the camera device; the processor is connected to the memory and the communicator, and is used to run the program instructions to perform, for example, Figure 6 The sight line estimation method is used to obtain the driver's sight line state estimation result. In some embodiments, the communicator, processor and memory can refer to Figure 10As described in the embodiment, the communicator may include a multimedia data interface capable of communicating with the camera device to transmit shooting data, such as USB2.0, GigE, USB3.0, CameraLink, CoaXPress, etc.

[0153] In an embodiment of the present disclosure, a computer-readable storage medium may be provided, wherein program instructions are stored, and the program instructions are executed by running, for example Figure 1 The training method of the sight line prediction model in the embodiment, or executing e.g. Figure 6 The line of sight estimation method in the embodiment.

[0154] That is, the method steps in the above embodiments are implemented as software or computer code that can be stored in a recording medium (such as a CDROM, RAM, floppy disk, hard disk or magneto-optical disk), or are implemented as computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded via a network and to be stored in a local recording medium, so that the method represented herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an ASIC or FPGA).

[0155] In summary, the embodiments of the present disclosure provide a training, estimation method, device, system, and medium for a gaze prediction model. The training method includes obtaining each original facial image containing each pre-identified facial key point; each pre-identified facial key point includes the pupil center of each eye; based on a cropping window of a preset size, each original facial image is cropped with each pupil center as the image center to obtain at least one pair of feature images corresponding to each original facial image; head posture information, eye key point information, and gaze angle information are annotated based on each feature image, and training data is formed based on each pair of feature images; each training data is input into a gaze prediction model; using a loss function, the loss is calculated based on the output result of the training data based on the gaze prediction model and the model parameters of the gaze prediction model are updated according to the loss. By constructing concise but feature-rich training data, a gaze prediction model with high prediction accuracy is trained.

[0156] The above embodiments are merely illustrative of the principles and effects of this disclosure and are not intended to limit this disclosure. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of this disclosure. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical concepts disclosed herein shall be covered by the claims of this disclosure.

Claims

1. A training method for a gaze prediction model, characterized in that: include: Obtain each original face image containing each pre-identified face key point; Each of the pre-identified facial key points includes the pupil center of each eye; Based on a cropping window of a preset size, cropping each original facial image with a preset size with the center of each pupil as the image center, so as to obtain at least one pair of feature images corresponding to each original facial image; The at least one pair of feature images includes: a left-eye feature image and a right-eye feature image; Annotating head posture information, eye key point information, and sight angle information based on each of the feature images to obtain each pair of feature images carrying the annotations, and forming training data based on each pair of the feature images; Inputting each training data into a gaze prediction model; wherein the gaze prediction model is configured to output predicted head posture information, eye key point information, and gaze angle information based on the face input image; Using a loss function, the loss is calculated based on the output results of the training data according to the line of sight prediction model and the model parameters of the line of sight prediction model are updated according to the loss; the loss calculated by the loss function includes: a first loss part of the predicted head posture information relative to the labeled head posture information, a second loss part of the predicted eye key point information relative to the labeled eye key point information, and a weighted sum of a third loss part of the predicted line of sight angle information relative to the labeled line of sight angle information; wherein the weight of the third loss part is higher than the weight of the second loss part, and the weight of the second loss part is higher than the weight of the first loss part.

2. The training method according to claim 1, characterized in that The method includes: performing image cropping of each original face image with a preset size based on a cropping window of a preset size and taking each pupil center as the image center to obtain at least one pair of feature images corresponding to each original face image. Based on the relative positions of the non-overlapping areas of the image capture result and the original facial image in the four directions of up, down, left, and right relative to the center of the image, determining the number of pixels required to be filled in along the four directions from the boundary of the overlapping area, and performing filling with preset pixel values to obtain the feature image; The feature image is converted to an input size of a sight line prediction model to obtain the training data.

3. The training method according to claim 1, characterized in that Each of the feature images contains at least 3 / 4 of the image area of the original face image.

4. The training method according to claim 1, characterized in that The sight line prediction model includes: a first image feature encoding module, a second image feature encoding module, an image feature fusion module, and an image decoding module; The first image feature encoding module is configured to extract features based on the left eye feature image to obtain first feature data; The second image feature encoding module is used to extract features based on the right eye feature image to obtain second feature data; The image feature fusion module is used to fuse the first feature data and the second feature data to obtain a fusion feature result; The image decoding module is used to decode the fusion feature result to obtain the output result.

5. A line of sight estimation method, characterized in that: include: Acquire a face image to be predicted, and obtain at least one pair of feature images to be predicted based on the face image to be predicted; The at least one pair of feature images includes a left-eye feature image and a right-eye feature image; Inputting the at least one pair of feature images to be predicted into a trained gaze prediction model to obtain corresponding predicted head posture information, eye key point information, and gaze angle information; wherein the gaze prediction model is trained by the training method according to any one of claims 1 to 4; The gaze state is estimated based on the predicted head posture information, eye key point information, and gaze angle information.

6. The sight line estimation method according to claim 5, wherein: The estimation of the sight state based on the predicted head posture information, eye key point information, and sight angle information includes: Determining whether the head posture is abnormal based on the head posture information; If the head posture is abnormal, the sight state is judged to be abnormal; If the head posture is normal, determine whether it is abnormal based on the key point information of the eyes and the line of sight angle information; If an abnormality is determined based on the eye key point information and the sight line angle information, the sight line state is determined to be abnormal; otherwise, the sight line state is determined to be normal.

7. The sight line estimation method according to claim 6, wherein: Include at least one of the following: 1) The head posture information includes: a head heading angle, a head pitch angle, and a head roll angle; and determining whether the head posture is abnormal based on the head posture information includes: determining whether any one of the head heading angle, the head pitch angle, and the head roll angle exceeds a preset angle threshold range; if so, determining that the head posture is abnormal; 2) the sight line angle information includes an eye heading angle and an eye pitch angle; the determining whether the sight line angle is abnormal based on the sight line angle information includes: determining whether the eye heading angle and the eye pitch angle exceed a preset angle threshold range respectively; if both do not exceed the preset angle threshold range, determining that the sight line state is normal; otherwise, determining that the sight line state is abnormal; 3) The eye key point information includes pupil center position information; the determining whether there is an abnormality based on the eye key point information includes: determining, for each of the left and right eyes, whether a deviation distance between the predicted pupil center position information and the corresponding pupil center position information is greater than a preset distance threshold; if the deviation distances for both the left and right eyes are greater than the preset distance threshold, determining that the sight state is abnormal; otherwise, determining that the sight state is normal; 4) The eye key point information includes: at least each eye key point that depicts the eye outline; the determining whether the eye is abnormal based on the eye key point information includes: corresponding to the left eye and the right eye, calculating the eye aspect ratio based on each eye key point, and comparing it with a preset aspect ratio threshold; if the aspect ratio of the left eye and the right eye is greater than the preset aspect ratio threshold, determining that the sight state is abnormal; otherwise, determining that the sight state is normal; 5) If both 3) and 4) determine that the sight line state is abnormal, determine the abnormality based on the key point information of the eye; 6) The line of sight state estimation method is executed periodically; if the number of abnormal line of sight states determined within a preset time range exceeds a preset proportion of the total number, a warning message is generated.

8. A training device for a sight line prediction model, characterized in that: include: An image acquisition module is used to acquire each original face image containing each pre-identified face key point; Each of the pre-identified facial key points includes the pupil center of each eye; An image capture module is configured to capture each original facial image of a preset size based on a capture window of a preset size, with the center of each pupil as the image center, to obtain at least one pair of feature images corresponding to each original facial image; The at least one pair of feature images includes: a left-eye feature image and a right-eye feature image; a training data acquisition module, configured to annotate head posture information, eye key point information, and sight angle information based on each of the feature images to obtain each pair of feature images carrying the annotations, and to form training data based on each pair of the feature images; A training module is used to input each training data into a gaze prediction model; wherein the gaze prediction model is configured to output predicted head posture information, eye key point information, and gaze angle information based on a face input image; and, using a loss function, calculate the loss based on the output result of the training data according to the gaze prediction model and update the model parameters of the gaze prediction model according to the loss; the loss calculated by the loss function includes: a weighted sum of a first loss part of the predicted head posture information relative to the labeled head posture information, a second loss part of the predicted eye key point information relative to the labeled eye key point information, and a third loss part of the predicted gaze angle information relative to the labeled gaze angle information; wherein the weight of the third loss part is higher than the weight of the second loss part, and the weight of the second loss part is higher than the weight of the first loss part.

9. A sight line estimation device, characterized in that: include: An image acquisition module, configured to acquire a face image to be predicted and obtain at least one pair of feature images to be predicted based on the face image to be predicted; The at least one pair of feature images includes a left-eye feature image and a right-eye feature image; A prediction module, configured to input the at least one pair of feature images to be predicted into a trained gaze prediction model to obtain corresponding predicted head posture information, eye key point information, and gaze angle information; wherein the gaze prediction model is trained by the training method according to any one of claims 1 to 4; The sight line state estimation module is used to estimate the sight line state based on the predicted head posture information, eye key point information, and sight line angle information.

10. An electronic device, characterized in that: include: processor and memory; The memory stores program instructions; The processor is configured to run the program instructions to execute the line of sight prediction model training method according to any one of claims 1 to 4, or execute the line of sight estimation method according to any one of claims 5 to 7.

11. A sight line monitoring system, characterized in that: Installed in the vehicle, including: a camera device, disposed in the vehicle and configured to capture an original image of the driver's face; A control device comprising: a communicator, a processor and a memory; the memory storing program instructions; the communicator being communicatively connected to the camera device; the processor being connected to the memory and the communicator and configured to run the program instructions to execute the line of sight estimation method as described in any one of claims 5 to 7 based on an original face image read from the camera device, so as to obtain an estimation result of the driver's line of sight state.

12. A computer-readable storage medium, characterized in that Program instructions are stored, and the program instructions are executed to perform the training method of the sight line prediction model according to any one of claims 1 to 4, or to perform the sight line estimation method according to any one of claims 5 to 7.

Images