Method, device, storage medium and electronic device for generating sight area prediction model
By combining end-to-end training of eye posture and line of sight area labels in the line of sight area prediction model, the prediction deviation problem caused by multi-model fit in the prior art is solved, and the accuracy and efficiency of line of sight area prediction are improved.
Patent Information
- Application Number
- CN202011244436.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-11-10
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2040-11-10
AI Technical Summary
The existing line-of-sight area prediction method involves fitting multiple models, and the optimization goals are inconsistent, resulting in deviations in prediction results and affecting performance.
A line of sight area prediction model is adopted. By first training the eye posture prediction module and the line of sight area prediction module, combining the eye posture label and line of sight area label, end-to-end model training is achieved to improve prediction accuracy.
Through end-to-end model training, the accuracy and efficiency of line-of-sight area prediction are improved, and the complexity of model training and prediction is reduced.
Smart Images

Figure CN112308006B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a method and device for generating a sight area prediction model, a line area prediction method and device, a computer-readable storage medium, and an electronic device. Background Art
[0002] With the development of artificial intelligence technology, sight area prediction technology has been applied to many fields, such as fatigue driving detection, sight tracking, smart cockpit, intelligent human-computer interaction, etc. The current sight area prediction method usually uses multiple models to perform predictions of different functions, and then further determines the sight area based on these prediction results. For example, the sight angle prediction model, key point detection model, human eye depth prediction model, sight three-dimensional area calculation model, etc. are trained, and each model is usually trained separately. Summary of the invention
[0003] Embodiments of the present disclosure provide a method, device, computer-readable storage medium, and electronic device for generating a sight area prediction model.
[0004] An embodiment of the present disclosure provides a method for generating a line of sight region prediction model, wherein the line of sight region prediction model includes an eye posture prediction module and a line of sight region prediction module, and the method includes: using a first facial image sample set with an eye posture label as an input of an initial model to train the eye posture prediction module; using a second facial image sample set with a line of sight region label as an input of the initial model to obtain an eye posture prediction result; training the line of sight region prediction module based on the second facial image sample set; and determining the initial model after the training as the line of sight region prediction model.
[0005] According to another aspect of an embodiment of the present disclosure, a line of sight area prediction method is provided, the method comprising: obtaining a facial image taken by a target camera; inputting the facial image into a pre-trained line of sight area prediction model, and obtaining an eye posture prediction result based on an eye posture prediction module included in the line of sight area prediction model, wherein the line of sight area prediction model is pre-trained based on the generation method of the above-mentioned line of sight area prediction model; inputting the eye posture prediction result into the line of sight area prediction module included in the line of sight area prediction model to obtain line of sight area information corresponding to the facial image.
[0006] According to another aspect of an embodiment of the present disclosure, a device for generating a line of sight region prediction model is provided, wherein the line of sight region prediction model includes an eye posture prediction module and a line of sight region prediction module, and the device includes: a first training module, used to take a first facial image sample set with an eye posture label as an input of an initial model to train the eye posture prediction module; a first determination module, used to take a second facial image sample set with a line of sight region label as an input of the initial model to obtain an eye posture prediction result; a second training module, used to train the line of sight region prediction module based on the second facial image sample set; and a second determination module, used to determine the initial model after the training as the line of sight region prediction model.
[0007] According to another aspect of an embodiment of the present disclosure, a sight area prediction device is provided, the device comprising: an acquisition module, for acquiring a face image taken by a target camera; a first prediction module, for inputting the face image into a pre-trained sight area prediction model, and obtaining an eye posture prediction result based on an eye posture prediction module included in the sight area prediction model, wherein the sight area prediction model is pre-trained based on a generation method of the sight area prediction model; a second prediction module, for inputting the eye posture prediction result into the sight area prediction module included in the sight area prediction model, and obtaining sight area information corresponding to the face image.
[0008] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and the computer program is used to execute the above-mentioned line-of-sight area prediction model generation method or line-of-sight area prediction method.
[0009] According to another aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; and a processor for reading executable instructions from the memory and executing the instructions to implement the above-mentioned line of sight area prediction model generation method or line of sight area prediction method.
[0010] Based on the generation method, device, computer-readable storage medium and electronic device of the line of sight area prediction model provided by the above-mentioned embodiments of the present disclosure, the eye posture prediction module included in the initial model is first trained with the first face image sample set, and then the eye posture prediction result predicted by the eye posture prediction module and the line of sight area prediction module included in the initial model are trained with the second face image sample set, and finally the trained initial model is used as the line of sight area prediction model. Since the eye posture prediction module and the line of sight area prediction module are included in the line of sight area prediction model, the parameters of each component included in the model will be updated at the same time during training, thereby realizing end-to-end model training and improving the prediction accuracy of the line of sight area prediction model. When using the line of sight area prediction model, the face image can be directly input to obtain the line of sight area information, thereby improving the efficiency of line of sight area prediction.
[0011] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.
[0013] Figure 1 is a system diagram to which the present disclosure is applicable.
[0014] Figure 2 It is a flowchart of a method for generating a sight area prediction model provided by an exemplary embodiment of the present disclosure.
[0015] Figure 3 It is a flowchart of a method for generating a sight area prediction model provided by another exemplary embodiment of the present disclosure.
[0016] Figure 4 It is a flowchart of a method for generating a sight area prediction model provided by another exemplary embodiment of the present disclosure.
[0017] Figure 5 It is an exemplary schematic diagram of the model training process according to the method for generating the line of sight area prediction model disclosed in the present invention.
[0018] Figure 6 It is a flowchart of a sight area prediction method provided by an exemplary embodiment of the present disclosure.
[0019] Figure 7It is a structural diagram of a device for generating a sight area prediction model provided by an exemplary embodiment of the present disclosure.
[0020] Figure 8 It is a structural schematic diagram of a device for generating a sight area prediction model provided by another exemplary embodiment of the present disclosure.
[0021] Fig. 9 It is a structural diagram of a sight area prediction device provided by an exemplary embodiment of the present disclosure.
[0022] Fig.10 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0023] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.
[0024] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.
[0025] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.
[0026] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.
[0027] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.
[0028] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.
[0029] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.
[0030] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.
[0031] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.
[0032] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.
[0033] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.
[0034] The disclosed embodiments can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.
[0035] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.
[0036] Application Overview
[0037] Existing solutions involve multiple model fitting steps when calculating the line of sight area, and the optimization goal of the model is not completely consistent with the prediction result of the final line of sight area, which leads to deviations in the optimization results and affects the performance of the final line of sight area. For example, the current line of sight area prediction scheme generally involves the fitting of four key models (including line of sight angle prediction model, key point detection model, human eye depth prediction model, and line of sight three-dimensional area calculation model), and each model needs to be fitted through annotated original data. Each model generally needs to be trained separately offline to meet the corresponding accuracy requirements of each model. This method ignores the guidance of the line of sight area prediction error on the learning objectives of each model, resulting in reduced prediction accuracy of the model.
[0038] Exemplary Systems
[0039] Figure 1 An exemplary system architecture 100 of a method for generating a sight line area prediction model or an apparatus for generating a sight line area prediction model, and a sight line area prediction method or an apparatus for generating a sight line area prediction model according to an embodiment of the present disclosure is shown.
[0040] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0041] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as search applications, web browser applications, shopping applications, instant messaging tools, etc.
[0042] The terminal device 101 can be various electronic devices, including but not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc.
[0043] The server 103 may be a server that provides various services, such as a background server that performs model training using a face image sample set uploaded by the terminal device 101, or performs sight area prediction using a face image uploaded by the terminal device 101. The background server may perform model training using a face image sample set to obtain a sight area prediction model, or perform sight area prediction on a received face image using a sight area prediction model to obtain sight area information.
[0044] It should be noted that the line of sight area prediction model generation method or line of sight area prediction method provided in the embodiments of the present disclosure can be executed by the server 103 or by the terminal device 101. Accordingly, the line of sight area prediction model generation device or line of sight area prediction device can be set in the server 103 or in the terminal device 101.
[0045] It should be understood that Figure 1 The number of terminal devices, networks and servers in the system is only for illustration. Any number of terminal devices, networks and servers may be provided as required. When the face image sample set or the face image for line of sight area prediction does not need to be acquired remotely, the above system architecture may not include a network but only a server or a terminal device.
[0046] Exemplary Methods
[0047] Figure 2 FIG. 1 is a flow chart of a method for generating a sight area prediction model provided by an exemplary embodiment of the present disclosure. Figure 1 On the terminal device 101 or server 103 shown in FIG. Figure 2 As shown, the method comprises the following steps:
[0048] Step 201: Using a first face image sample set with eye posture labels as input of an initial model to train an eye posture prediction module.
[0049] In this embodiment, the electronic device can use the first facial image sample set with eye posture labels as the input of the initial model to train the eye posture prediction module. The facial images in the first facial image sample set include eye areas. The sight line area prediction model in this embodiment includes an eye posture prediction module and a sight line area prediction module. The eye posture prediction module is used to determine the eye posture of the person indicated by the input facial image, such as the sight line angle, pupil depth, etc. The sight line area prediction module is used to determine the sight line area of the person indicated by the input facial image, that is, the area where the human eye is looking.
[0050] The eye posture label may be information for pre-labeling the face images in the first face image sample set. The eye posture label is obtained in advance through an existing eye posture collection method. For example, the distance between the pupil and the camera is determined by determining the position of the pupil in three-dimensional space, and the pupil depth label is obtained; the angle of the straight line between the two points in three-dimensional space is determined by determining the position of the pupil in three-dimensional space and the three-dimensional space position of the gaze point.
[0051] The eye posture prediction module can generally be composed of an artificial neural network, for example, a convolutional neural network including a convolutional layer, a pooling layer, a fully connected layer, etc. The electronic device can use a machine learning method to take the face image in the first face image sample set as the input of the initial model. The initial model can first extract the features of the eye area in the face image, take the features of the eye area as the input of the eye posture prediction module, take the eye posture label corresponding to the input face image as the expected output (i.e., the target of optimizing the model parameters), train the initial model, and obtain the actual output for each training input face image. The actual output is the data actually output by the initial model, which is used to characterize the eye posture. Then, the electronic device can use the gradient descent method and the back propagation method to adjust the parameters of the eye posture prediction module included in the initial model based on the actual output and the expected output, and use the model obtained after each parameter adjustment as the initial model for the next training, and end the training when the preset training end condition for the eye posture prediction module is met, thereby obtaining the trained eye posture prediction module. The above training end condition may include but is not limited to at least one of the following: the number of training times reaches a preset number of times, and the loss value of the loss function corresponding to the eye posture prediction module reaches a preset loss value.
[0052] Step 202: Using the second face image sample set with sight area labels as input to the initial model to obtain eye posture prediction results.
[0053] In this embodiment, the electronic device can use the second face image sample set with the sight area label as the input of the initial model to obtain the eye posture prediction result. Specifically, the trained eye posture prediction module can be used to perform eye posture prediction on the input face image to obtain the eye posture prediction result. The eye posture result may include but is not limited to information such as sight angle and pupil depth.
[0054] The sight area label may be obtained by pre-marking the face image. The sight area label usually includes a plurality of coordinate information, which is used to determine an area in a three-dimensional space.
[0055] Step 203: training a sight area prediction module based on the second face image sample set.
[0056] In this embodiment, the electronic device can train a sight area prediction module based on the second face image sample set, wherein the sight area prediction module is used to obtain sight area information representing the gaze area of the human eye according to the eye posture prediction result.
[0057] Specifically, the sight area prediction module can generally be composed of an artificial neural network, for example, a convolutional neural network including a convolutional layer, a pooling layer, a fully connected layer, etc. This module can classify the three-dimensional space and determine which areas are the areas where the human eye is looking.
[0058] The electronic device can use a machine learning method to take the face image in the second face image sample set as the input of the initial model, obtain the eye posture prediction result after processing by the eye posture prediction module, take the eye posture prediction result as the input of the sight area prediction module, take the sight area label corresponding to the input face image as the expected output (i.e., the target of optimizing the model parameters), train the initial model, and obtain the actual output for each training input face image. The actual output is the data actually output by the initial model, which is used to characterize the sight area. Then, the electronic device can use the gradient descent method and the back propagation method to adjust the parameters of the sight area prediction module included in the initial model based on the actual output and the expected output, and use the model obtained after each parameter adjustment as the initial model for the next training, and end the training when the preset training end condition for the sight area prediction module is met, thereby obtaining the trained sight area prediction module. The above-mentioned training end condition may include but is not limited to at least one of the following: the number of training times reaches the preset number of times, and the loss value of the loss function corresponding to the sight area prediction module reaches the preset loss value.
[0059] Step 204: determine the initial model after training as the sight area prediction model.
[0060] In this embodiment, the electronic device may determine the initial model after training as the sight area prediction model. The sight area prediction model may receive a face image including an eye area and output sight area information.
[0061] The method provided by the above-mentioned embodiments of the present disclosure first uses the first facial image sample set to train the eye posture prediction module included in the initial model, then uses the eye posture prediction results predicted by the eye posture prediction module and the second facial image sample set to train the line of sight area prediction module included in the initial model, and finally uses the trained initial model as the line of sight area prediction model. Since the eye posture prediction module and the line of sight area prediction module are included in the line of sight area prediction model, the parameters of each component included in the model will be updated simultaneously during training, thereby realizing end-to-end model training and improving the prediction accuracy of the line of sight area prediction model.
[0062] In some optional implementations, the number of face images included in the first face image sample set is less than the number of face images included in the second face image sample set. Since the annotation of the sight area label is relatively simple, the method usually used to collect the eye posture label is relatively complex, requiring professional tools and personnel. For example, the eye posture label can include a sight angle label and / or a pupil depth label. The sight angle label can be obtained by determining the position of the pupil in three-dimensional space and the three-dimensional space position of the gazed viewpoint in advance, and determining the angle of the straight line between the two points in three-dimensional space. The pupil depth label can be obtained by using a depth collector, such as a binocular camera, a TOF camera, a structured light camera, etc. Therefore, the number of face images included in the first face image sample set with eye posture labels collected by this implementation is relatively small. When training the eye posture prediction module, fewer samples can be used to perform preliminary training on the eye posture prediction module, and more samples can be used to train the sight area prediction module, so as to obtain better training results. Thereby, the difficulty of collecting labels can be reduced and the efficiency of model training can be improved.
[0063] In some optional implementations, the above step 201 may be performed as follows:
[0064] In response to determining that the eye posture label corresponding to the input face image includes a sight angle label, a sight angle prediction submodule included in the eye posture prediction module is trained based on the sight angle label.
[0065] In response to determining that the eye posture label corresponding to the input face image includes a pupil depth label, a pupil depth prediction submodule included in the eye posture prediction module is trained based on the pupil depth label.
[0066] Among them, the eye posture prediction module provided by this implementation may include a sight angle prediction submodule and / or a pupil depth prediction submodule. The sight angle submodule is used to determine the sight angle of the person indicated by the input face image. The pupil depth prediction submodule is used to determine the pupil depth of the person indicated by the input face image. The pupil depth represents the distance from the pupil to the camera. The sight angle prediction submodule and the pupil depth prediction submodule can generally be composed of an artificial neural network (e.g., a convolutional neural network).
[0067] It should be understood that the method of training the sight angle prediction submodule and the pupil depth prediction submodule is basically the same as step 201, and will not be repeated here.
[0068] This implementation method can more comprehensively determine the eye posture and improve the accuracy of the line of sight area prediction by setting the line of sight angle prediction submodule and the pupil depth prediction submodule.
[0069] In some optional implementations, step 201 may be performed as follows:
[0070] First, the face images included in the first face image sample set are input into a feature extraction module included in the initial model to obtain eye feature data and face feature data.
[0071] Among them, the feature extraction module is used to extract eye feature data and face feature data. The eye feature data can characterize the features of the eye, such as the size, proportion, pupil position, etc. of the eye; the face feature data is used to characterize the features of the face, such as the direction and size of the face. The feature extraction module can adopt the existing multi-layer convolutional neural network structure.
[0072] Then, based on the eye posture label, the eye feature data and face feature data are used to train the eye posture prediction module.
[0073] Specifically, during the training process, the module parameters can be continuously adjusted based on the loss values of the loss functions corresponding to the eye posture prediction module and the line of sight area prediction module. It should be noted that if the eye posture prediction module includes the line of sight angle prediction submodule and / or pupil depth prediction submodule shown in the above optional implementation, the eye feature data and the face feature data can be used as the input of the line of sight angle prediction submodule and / or the pupil depth prediction submodule, and the line of sight angle label and / or pupil depth label can be used for training.
[0074] This implementation method can extract eye features and facial features of an input face image by setting a feature extraction module in the line of sight area prediction model, thereby making full use of the eye features and facial features for eye posture prediction and improving the accuracy of eye posture prediction and line of sight area prediction.
[0075] Further references Figure 3 , shows a flow chart of another embodiment of a method for generating a sight area prediction model. Figure 3 As shown in the above Figure 2 Based on the illustrated embodiment, step 201 may include the following steps:
[0076] Step 2011, in response to determining that the input sample face image also has a corresponding line of sight area label, the weight of the eye posture loss function corresponding to the eye posture prediction module is set to a first weight, and the weight of the line of sight area loss function corresponding to the line of sight area prediction module is set to a second weight.
[0077] In this embodiment, the first weight is greater than the second weight.
[0078] Specifically, the facial images in the first facial image sample set in the present embodiment have (some facial images may have, or all facial images may have) corresponding eye posture labels and sight area labels. During training, a larger first weight (for example, 0.9) may be set for the eye posture loss function, and a smaller second weight (for example, 0.1) may be set for the sight area loss function. During the training process, since the weight of the sight area loss function is smaller, the amplitude of the parameter update of the sight area prediction module is smaller, that is, the update of the model parameters can be concentrated on the eye posture prediction module. Among them, the eye posture loss function may be a regression loss function, such as an L1 norm loss function, an L2 norm loss function, etc. The sight area loss function may be a loss function used for classification, such as a cross entropy loss function.
[0079] Step 2012, based on the eye posture label and the sight area label corresponding to the input sample face image, determine the first loss value of the eye posture loss function and the second loss value of the sight area loss function.
[0080] In this embodiment, based on the eye posture loss function, a first loss value representing the difference between the predicted eye posture and the actually annotated eye posture can be determined during training. Based on the sight area loss function, a second loss value representing the difference between the predicted sight area and the actually annotated sight area can be determined during training.
[0081] Step 2013: training an eye posture prediction module based on the first weight, the second weight, the first loss value, and the second loss value.
[0082] As an example, assuming that the eye posture loss function is l1, the sight area loss function is l2, the first weight is a, and the second weight is b, then the total loss function is l = a × l1 + b × l2. During the training process, the gradient descent method and back propagation method are used to adjust the parameters of the initial model so that the total loss value continues to decrease until convergence.
[0083] In some optional implementations, step 2011 may be performed as follows:
[0084] The sum of the weights of the sight angle loss function and the pupil depth loss function included in the eye posture loss function is set as the first weight, wherein the sight angle loss function and the pupil depth loss function correspond to the sight angle prediction submodule and the pupil depth prediction submodule included in the eye posture prediction module, respectively.
[0085] As an example, assuming that the sight angle loss function is l11, the pupil depth loss function is l12, the sight area loss function is l2, the weight of l11 is a1, the weight of l12 is a2, the first weight is a=a1+a2, and the second weight is b, then the total loss function is l=a1×l11+a2×l12+b×l2. Figure 2 The sight angle prediction submodule and pupil depth prediction submodule described in the corresponding embodiments are basically the same and will not be described in detail here.
[0086] This implementation method sets weights for the loss functions of the gaze angle prediction submodule and the pupil depth prediction submodule included in the eye posture prediction module, thereby more flexibly adjusting the contribution of the gaze angle prediction submodule and the pupil depth prediction submodule to the gaze area prediction and improving the targetedness of the model training.
[0087] Above Figure 3 The method provided in the corresponding embodiment, by setting the weight of the eye posture loss function to a larger one and the line of sight area loss function to a smaller one, can make it possible to take into account the training of the line of sight area prediction module when training the eye posture prediction module, which helps to improve the prediction accuracy of the line of sight area prediction model after training.
[0088] Further references Figure 4 , shows a flow chart of another embodiment of a method for generating a sight area prediction model. Figure 4 As shown in the above Figure 2 Based on the illustrated embodiment, step 203 may include the following steps:
[0089] Step 2031, in response to determining that the input sample face image also has a corresponding eye posture label, the weight of the eye posture loss function corresponding to the eye posture prediction module is set to a third weight, and the weight of the sight area loss function corresponding to the sight area prediction module is set to a fourth weight.
[0090] Among them, the third weight is smaller than the fourth weight.
[0091] Specifically, the facial images in the second facial image sample set in this embodiment have (some facial images may have, or all facial images may have) corresponding eye posture labels and sight area labels. During training, a smaller third weight (e.g., 0.1) may be set for the eye posture loss function, and a larger fourth weight (e.g., 0.9) may be set for the sight area loss function. During the training process, since the weight of the eye posture loss function is smaller, the amplitude of the parameter update of the eye posture prediction module is smaller, that is, the update of the model parameters can be concentrated in the sight area prediction module.
[0092] Step 2032, based on the eye posture label and the sight area label corresponding to the input sample face image, determine the first loss value of the eye posture loss function and the second loss value of the sight area loss function.
[0093] In this embodiment, based on the eye posture loss function, a first loss value representing the difference between the predicted eye posture and the actually annotated eye posture can be determined during training. Based on the sight area loss function, a second loss value representing the difference between the predicted sight area and the actually annotated sight area can be determined during training.
[0094] Step 2033: training a sight area prediction module based on the third weight, the fourth weight, the first loss value, and the second loss value.
[0095] As an example, assuming that the eye posture loss function is l1, the sight area loss function is l2, the third weight is c, and the fourth weight is d, the total loss function is l = c × l1 + d × l2. During the training process, the gradient descent method and back propagation method are used to adjust the parameters of the initial model so that the total loss value is continuously reduced until convergence.
[0096] It should be understood that the eye posture module of this embodiment may also include a sight angle prediction submodule and a pupil depth prediction submodule. Accordingly, the eye posture loss function may include a sight angle loss function and a pupil depth loss function, and the sum of the weights of the sight angle loss function and the pupil depth loss function may be set to a third weight.
[0097] Above Figure 4 In the method provided in the corresponding embodiment, by setting the weight of the eye posture loss function to be smaller and the sight area loss function to be larger, the training of the sight area prediction module can be taken into account while training the sight area prediction module, which helps to improve the prediction accuracy of the trained sight area prediction model. Figure 3 and Figure 4 The corresponding embodiments are combined, that is, by adjusting the weight of the loss function and training the various modules included in the initial model at the same time, so that the line of sight area prediction error guides the optimization target for each module during learning, improves the efficiency of model training, and makes the prediction accuracy of the trained line of sight area prediction model higher.
[0098] See also Figure 5 , which shows an exemplary schematic diagram of the model training process of the method for generating the sight area prediction model according to the present disclosure. Figure 5In the training process, the face image 501 including the eye area is first input into the feature extraction module 5021 of the initial model 502 to obtain the eye feature data F1 and the face feature data F2. Then, F1 and F2 are input into the sight angle prediction submodule 5022 and the pupil depth prediction submodule 5023 as two branches to obtain the sight angle information O1 and the pupil depth information O2. Next, O1 and O2 are input into the sight area prediction module 5024 to obtain the sight area information O3. During the training process, the sight angle loss function L1 is used to determine the difference between the sight angle information O1 and the sight angle label 503, the pupil depth loss function L2 is used to determine the difference between the pupil depth information O2 and the pupil depth label 504, and the sight area loss function L3 is used to determine the difference between the sight area information O3 and the sight area label 505. Based on the above differences, the parameters of each module included in the initial model are adjusted to reduce the difference to convergence, and the initial model after training is finally used as the sight area prediction model.
[0099] Continue to refer Figure 6 , Figure 6 FIG. 1 is a flow chart of a method for predicting a sight area provided by an exemplary embodiment of the present disclosure. Figure 1 On the terminal device 101 or server 103 shown in FIG. Figure 6 As shown, the method comprises the following steps:
[0100] Step 601, obtaining a face image captured by a target camera.
[0101] In this embodiment, the electronic device can obtain a facial image captured by a target camera locally or remotely. The target camera can be a camera set at a specific position in a specific space. For example, the target camera can be set inside a vehicle to capture the face of a person in the vehicle and obtain a facial image.
[0102] Step 602: input the face image into a pre-trained sight line area prediction model, and obtain an eye posture prediction result based on an eye posture prediction module included in the sight line area prediction model.
[0103] In this embodiment, the electronic device can input the face image into a pre-trained sight area prediction model, and obtain the eye posture prediction result based on the eye posture prediction module included in the sight area prediction model. Figure 2 The method described in the corresponding embodiment is trained.
[0104] Step 603: input the eye posture prediction result into the sight area prediction module included in the sight area prediction model to obtain the sight area information corresponding to the face image.
[0105] In this embodiment, the electronic device can input the eye posture prediction result into the sight area prediction module included in the sight area prediction model to obtain the sight area information corresponding to the face image. The sight area information is used to characterize the area where the person indicated by the face image is looking at, and the sight area information can include multiple coordinates, which can characterize the area in the three-dimensional space. Usually, after obtaining the sight area information, the sight area information can be output in various ways. For example, it can be displayed on a display, stored in a preset storage area, or sent to other electronic devices.
[0106] The above embodiment of the present disclosure provides a method for predicting the sight area by inputting the captured face image into a Figure 2 The sight area prediction model trained in the corresponding embodiment can obtain the sight area information, thereby effectively utilizing the end-to-end trained model to perform sight area prediction, thereby improving the accuracy and efficiency of sight area prediction.
[0107] Exemplary Devices
[0108] Figure 7 is a schematic diagram of a structure of a device for generating a sight area prediction model provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices. The sight area prediction model includes an eye posture prediction module and a sight area prediction module. Figure 7 As shown, the generating device of the sight area prediction model includes: a first training module 701, used to take the first face image sample set with eye posture labels as the input of the initial model to train the eye posture prediction module; a first determination module 702, used to take the second face image sample set with the sight area label as the input of the initial model to obtain the eye posture prediction result; a second training module 703, used to train the sight area prediction module based on the second face image sample set; a second determination module 704, used to determine the initial model after the training as the sight area prediction model.
[0109] In this embodiment, the first training module 701 can use the first facial image sample set with eye posture labels as the input of the initial model to train the eye posture prediction module. The facial images in the first facial image sample set include eye areas. The sight area prediction model in this embodiment includes an eye posture prediction module and a sight area prediction module. The eye posture prediction module is used to determine the eye posture of the person indicated by the input facial image, such as the sight angle, pupil depth, etc. The sight area prediction module is used to determine the sight area of the person indicated by the input facial image, that is, the area where the human eye is looking.
[0110] The eye posture label may be information for pre-labeling the face images in the first face image sample set. The eye posture label is obtained in advance through an existing eye posture collection method. For example, the distance between the pupil and the camera is determined by determining the position of the pupil in three-dimensional space, and the pupil depth label is obtained; the angle of the straight line between the two points in three-dimensional space is determined by determining the position of the pupil in three-dimensional space and the three-dimensional space position of the gaze point.
[0111] The eye posture prediction module can usually be composed of an artificial neural network, for example, a convolutional neural network including a convolutional layer, a pooling layer, a fully connected layer, etc. The first training module 701 can use a machine learning method to take the face image in the first face image sample set as the input of the initial model. The initial model can first extract the features of the eye area in the face image, take the features of the eye area as the input of the eye posture prediction module, take the eye posture label corresponding to the input face image as the expected output (i.e., the target of optimizing the model parameters), train the initial model, and obtain the actual output for each training input face image. Wherein, the actual output is the data actually output by the initial model, which is used to characterize the eye posture. Then, the first training module 701 can use the gradient descent method and the back propagation method to adjust the parameters of the eye posture prediction module included in the initial model based on the actual output and the expected output, and use the model obtained after each parameter adjustment as the initial model for the next training, and end the training when the preset training end condition for the eye posture prediction module is met, thereby obtaining the trained eye posture prediction module. The above training end condition may include but is not limited to at least one of the following: the number of training times reaches a preset number of times, and the loss value of the loss function corresponding to the eye posture prediction module reaches a preset loss value.
[0112] In this embodiment, the first determination module 702 can use the second facial image sample set with the sight area label as the input of the initial model to obtain the eye posture prediction result. Specifically, the trained eye posture prediction module can be used to perform eye posture prediction on the input facial image to obtain the eye posture prediction result. The eye posture result may include but is not limited to information such as sight angle and pupil depth.
[0113] The sight area label may be obtained by pre-marking the face image. The sight area label usually includes a plurality of coordinate information, which is used to determine an area in a three-dimensional space.
[0114] In this embodiment, the second training module 703 can train a sight area prediction module based on the second face image sample set, wherein the sight area prediction module is used to obtain sight area information representing the gaze area of the human eye according to the eye posture prediction result.
[0115] Specifically, the sight area prediction module can generally be composed of an artificial neural network, for example, a convolutional neural network including a convolutional layer, a pooling layer, a fully connected layer, etc. This module can classify the three-dimensional space and determine which areas are the areas where the human eye is looking.
[0116] The second training module 703 can use a machine learning method to take the face image in the second face image sample set as the input of the initial model, obtain the eye posture prediction result after processing by the eye posture prediction module, take the eye posture prediction result as the input of the sight area prediction module, take the sight area label corresponding to the input face image as the expected output (i.e., the target of optimizing the model parameters), train the initial model, and obtain the actual output for each training input face image. The actual output is the data actually output by the initial model, which is used to characterize the sight area. Then, the second training module 703 can use the gradient descent method and the back propagation method to adjust the parameters of the sight area prediction module included in the initial model based on the actual output and the expected output, and use the model obtained after each parameter adjustment as the initial model for the next training, and terminate the training when the preset training end condition for the sight area prediction module is met, thereby obtaining the trained sight area prediction module. The above-mentioned training end condition may include but is not limited to at least one of the following: the number of training times reaches the preset number of times, and the loss value of the loss function corresponding to the sight area prediction module reaches the preset loss value.
[0117] In this embodiment, the second determination module 704 may determine the initial model after training as the sight line area prediction model. The sight line area prediction model may receive a face image including an eye area and output sight line area information.
[0118] Reference Figure 8 , Figure 8 It is a structural schematic diagram of a device for generating a sight area prediction model provided by another exemplary embodiment of the present disclosure.
[0119] In some optional implementations, the first training module 701 may include: a first training unit 7011, for, in response to determining that the eye posture label corresponding to the input facial image includes a line of sight angle label, based on the line of sight angle label, training the line of sight angle prediction submodule included in the eye posture prediction module; a second training unit 7012, for, in response to determining that the eye posture label corresponding to the input facial image includes a pupil depth label, based on the pupil depth label, training the pupil depth prediction submodule included in the eye posture prediction module.
[0120] In some optional implementations, the first training module 701 may include: an extraction unit 7013, used to input the facial images included in the first facial image sample set into a feature extraction module included in the initial model to obtain eye feature data and facial feature data; a third training unit 7014, used to train an eye posture prediction module based on eye posture labels using the eye feature data and facial feature data.
[0121] In some optional implementations, the first training module includes 701, which can: a first setting unit 7015, which is used to set the weight of the eye posture loss function corresponding to the eye posture prediction module to a first weight in response to determining that the input sample face image also has a corresponding line of sight area label, and set the weight of the line of sight area loss function corresponding to the line of sight area prediction module to a second weight, wherein the first weight is greater than the second weight; a first determination unit 7016, which is used to determine a first loss value of the eye posture loss function and a second loss value of the line of sight area loss function based on the eye posture label and the line of sight area label corresponding to the input sample face image; a fourth training unit 7017, which is used to train the eye posture prediction module based on the first weight, the second weight, and the first loss value and the second loss value.
[0122] In some optional implementations, the first setting unit 7015 can be further used to: set the sum of the weights of the sight angle loss function and the pupil depth loss function included in the eye posture loss function as the first weight, wherein the sight angle loss function and the pupil depth loss function respectively correspond to the sight angle prediction submodule and the pupil depth prediction submodule included in the eye posture prediction module.
[0123] In some optional implementations, the second training module 703 may include: a second setting unit 7031, used to set the weight of the eye posture loss function corresponding to the eye posture prediction module to a third weight in response to determining that the input sample face image also has a corresponding eye posture label, and set the weight of the line of sight area loss function corresponding to the line of sight area prediction module to a fourth weight, wherein the third weight is less than the fourth weight; a second determination unit 7032, used to determine a first loss value of the eye posture loss function and a second loss value of the line of sight area loss function based on the eye posture label and the line of sight area label corresponding to the input sample face image; a fifth training unit 7033, used to train the line of sight area prediction module based on the third weight, the fourth weight, and the first loss value and the second loss value.
[0124] In some optional implementations, the number of facial images included in the first facial image sample set is smaller than the number of facial images included in the second facial image sample set.
[0125] The above-mentioned embodiment of the present disclosure provides a device for generating a line of sight area prediction model. First, the eye posture prediction module included in the initial model is trained with the first facial image sample set. Then, the eye posture prediction result predicted by the eye posture prediction module and the line of sight area prediction module included in the initial model are trained with the second facial image sample set. Finally, the trained initial model is used as the line of sight area prediction model. Since the eye posture prediction module and the line of sight area prediction module are included in the line of sight area prediction model, the parameters of each component included in the model will be updated simultaneously during training, thereby realizing end-to-end model training and improving the prediction accuracy of the line of sight area prediction model.
[0126] Fig. 9 FIG. 1 is a schematic diagram of a structure of a sight area prediction device provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as Fig. 9 As shown, the sight area prediction device includes: an acquisition module 901, used to acquire a face image taken by a target camera; a first prediction module 902, used to input the face image into a pre-trained sight area prediction model, and obtain an eye posture prediction result based on an eye posture prediction module included in the sight area prediction model, wherein the sight area prediction model is pre-trained based on a generation method of the sight area prediction model; a second prediction module 903, used to input the eye posture prediction result into the sight area prediction module included in the sight area prediction model, and obtain the sight area information corresponding to the face image.
[0127] In this embodiment, the acquisition module 901 can acquire the face image captured by the target camera locally or remotely. The target camera can be a camera set at a specific position in a specific space. For example, the target camera can be set inside a vehicle to capture the face of the person in the vehicle and obtain the face image.
[0128] In this embodiment, the first prediction module 902 can input the face image into a pre-trained sight area prediction model, and obtain the eye posture prediction result based on the eye posture prediction module included in the sight area prediction model. Figure 2 The method described in the corresponding embodiment is trained.
[0129] In this embodiment, the second prediction module 903 can input the eye posture prediction result into the sight area prediction module included in the sight area prediction model to obtain the sight area information corresponding to the face image. Among them, the sight area information is used to characterize the area of the sight of the person indicated by the face image, and the sight area information may include multiple coordinates, which can characterize the area in the three-dimensional space. Usually, after obtaining the sight area information, the sight area information can be output in various ways. For example, it can be displayed on a display, stored in a preset storage area, or sent to other electronic devices.
[0130] The generation device of the sight area prediction model provided by the above embodiment of the present disclosure is to input the captured face image into the Figure 2 The sight area prediction model trained in the corresponding embodiment can obtain the sight area information, thereby effectively utilizing the end-to-end trained model to perform sight area prediction, thereby improving the accuracy and efficiency of sight area prediction.
[0131] Exemplary Electronic Devices
[0132] Below, reference Fig.10 To describe an electronic device according to an embodiment of the present disclosure. The electronic device may be as follows Figure 1 Any one or both of the terminal device 101 and the server 103 shown, or a stand-alone device independent of them, the stand-alone device can communicate with the terminal device 101 and the server 103 to receive the collected input signals from them.
[0133] Fig.10 A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated.
[0134] like Fig.10 As shown, the electronic device 1000 includes one or more processors 1001 and a memory 1002 .
[0135] The processor 1001 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1000 to perform desired functions.
[0136] The memory 1002 may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on a computer-readable storage medium, and the processor 1001 may run the program instructions to implement the generation method of the sight area prediction model or the sight area prediction method of the various embodiments of the present disclosure above and / or other desired functions. Various contents such as a facial image sight area prediction model may also be stored in the computer-readable storage medium.
[0137] In one example, the electronic device 1000 may further include: an input device 1003 and an output device 1004 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).
[0138] For example, when the electronic device is a terminal device 101 or a server 103, the input device 1003 may be a camera, a mouse, a keyboard, etc., for inputting a facial image. When the electronic device is a stand-alone device, the input device 1003 may be a communication network connector for receiving an input facial image from the terminal device 101 and the server 103.
[0139] The output device 1004 can output various information to the outside, including the predicted sight area information. The output device 1004 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.
[0140] Of course, to simplify, Fig.10 Only some of the components related to the present disclosure in the electronic device 1000 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 1000 may also include any other appropriate components.
[0141] Exemplary computer program products and computer-readable storage media
[0142] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps in the method for generating a line of sight area prediction model or the line of sight area prediction method according to various embodiments of the present disclosure described in the above "Exemplary Method" section of this specification.
[0143] The computer program product may be written in any combination of one or more programming languages to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0144] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the processor executes the steps in the method for generating a line of sight area prediction model or the line of sight area prediction method according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.
[0145] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0146] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.
[0147] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0148] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.
[0149] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.
[0150] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.
[0151] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
[0152] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.
Claims
1. A method for generating a sight area prediction model, the sight area prediction model comprising an eye posture prediction module and a sight area prediction module, the method comprising: Using a first face image sample set with eye pose labels as input to an initial model to train the eye pose prediction module; Using a second facial image sample set with gaze region labels as input to the initial model to obtain eye posture prediction results, wherein at least some of the facial images in the second facial image sample set also have eye posture labels; Based on the second facial image sample set, training the eye posture prediction module and the sight area prediction module; The initial model after training is determined as the sight area prediction model.
2. The method according to claim 1, wherein The method of using the first face image sample set with eye posture labels as input of the initial model to train the eye posture prediction module includes: In response to determining that the eye posture label corresponding to the input facial image includes a gaze angle label, training the gaze angle prediction submodule included in the eye posture prediction module based on the gaze angle label; In response to determining that the eye posture label corresponding to the input facial image includes a pupil depth label, the pupil depth prediction submodule included in the eye posture prediction module is trained based on the pupil depth label.
3. The method according to claim 1, wherein The method of using the first face image sample set with eye posture labels as input of the initial model to train the eye posture prediction module includes: Inputting the facial images included in the first facial image sample set into the feature extraction module included in the initial model to obtain eye feature data and facial feature data; Based on the eye posture label, the eye posture prediction module is trained using the eye feature data and the facial feature data.
4. The method according to claim 1, wherein The training of the eye posture prediction module comprises: In response to determining that the input sample face image also has a corresponding gaze region label, setting the weight of the eye pose loss function corresponding to the eye pose prediction module to a first weight, and setting the weight of the gaze region loss function corresponding to the gaze region prediction module to a second weight, wherein the first weight is greater than the second weight; Determining a first loss value of the eye posture loss function and a second loss value of the sight line area loss function based on the eye posture label and the sight line area label corresponding to the input sample face image; The eye posture prediction module is trained based on the first weight, the second weight, the first loss value, and the second loss value.
5. The method according to claim 1, wherein The training of the eye posture prediction module and the sight area prediction module includes: In response to determining that the input sample face image also has a corresponding eye pose label, setting the weight of the eye pose loss function corresponding to the eye pose prediction module to a third weight, and setting the weight of the gaze area loss function corresponding to the gaze area prediction module to a fourth weight, wherein the third weight is less than the fourth weight; Determining a first loss value of the eye posture loss function and a second loss value of the sight line area loss function based on the eye posture label and the sight line area label corresponding to the input sample face image; The eye posture prediction module and the sight area prediction module are trained based on the third weight, the fourth weight, the first loss value, and the second loss value.
6. A method for predicting a sight area, comprising: Get the face image taken by the target camera; Inputting the facial image into a pre-trained gaze area prediction model, and obtaining an eye posture prediction result based on an eye posture prediction module included in the gaze area prediction model, wherein the gaze area prediction model is pre-trained based on the method of any one of claims 1-5; The eye posture prediction result is input into the sight area prediction module included in the sight area prediction model to obtain the sight area information corresponding to the face image.
7. A device for generating a sight area prediction model, the sight area prediction model comprising an eye posture prediction module and a sight area prediction module, the device comprising: a first training module, configured to use a first facial image sample set with eye posture labels as input of an initial model to train the eye posture prediction module; a first determining module, configured to use a second facial image sample set with a gaze region label as input to an initial model to obtain an eye posture prediction result, wherein at least some of the facial images in the second facial image sample set also have an eye posture label; A second training module, configured to train the eye posture prediction module and the sight area prediction module based on the second facial image sample set; The second determination module is used to determine the initial model after training as the sight area prediction model.
8. A sight area prediction device, comprising: An acquisition module is used to acquire a face image captured by a target camera; a first prediction module, configured to input the facial image into a pre-trained gaze area prediction model, and obtain an eye posture prediction result based on an eye posture prediction module included in the gaze area prediction model, wherein the gaze area prediction model is pre-trained based on the method according to any one of claims 1 to 5; The second prediction module is used to input the eye posture prediction result into the sight area prediction module included in the sight area prediction model to obtain the sight area information corresponding to the face image.
9. A computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 6.
10. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method according to any one of claims 1 to 6.
