Data processing method, image processing method and apparatus, electronic device, and storage medium

Abstract

A data processing method, an image processing method and apparatus, an electronic device, and a storage medium, relating to the technical field of computers. The data processing method comprises: acquiring a first face image set, a first face image being correspondingly tagged with an eyeball rotation angle based on a head coordinate system; on the basis of the eyeball rotation angle correspondingly tagged for the first face image, determining an eyeball rotation angle category tag for the first face image; acquiring a second face image set, and performing key point detection on an eye area of a second face image, to obtain an eye key point; on the basis of the eye key point, determining an eyeball rotation angle category tag of the second face image; and by means of the first face image and the eyeball rotation angle category tag thereof, and the second face image and the eyeball rotation angle category tag thereof, performing training on a network model, to obtain a target network model used to predict an eyeball rotation angle based on the head coordinate system in a face image.

Description

Data processing method, image processing method, device, electronic device and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to Chinese patent application number 202410217326.3, filed on February 27, 2024, entitled “Data processing method, image processing method, device, electronic device and storage medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of computer technology, and in particular to a data processing method, an image processing method, a data processing device, an electronic device, and a computer-readable storage medium. Background Art

[0004] In the production of 3D games and animations, it is necessary to synchronize the facial changes of real people with virtual characters so that the virtual characters can have the same facial movements as real people. In this process, it is necessary to calculate the eye rotation angle to drive the virtual character to move the eyeballs accordingly.

[0005] Current work on eye rotation angle calculation primarily focuses on calculating the eye gaze angle (i.e., line of sight estimation) based on the camera coordinate system. This eye gaze angle is related not only to the eye state but also to the head posture. However, this eye gaze angle cannot be used to drive virtual characters because line of sight estimation is based on the camera coordinate system, while the virtual character's eye rotation is based on the head coordinate system. Therefore, current related technologies lack a means to determine the eye rotation angle based on the head coordinate system. Summary of the Invention

[0006] The present disclosure provides a data processing method, an image processing method, a data processing device, an electronic device, and a computer-readable storage medium to solve or at least partially solve the above-mentioned problems, as follows.

[0007] In a first aspect, the present disclosure provides a data processing method, the method comprising:

[0008] Acquire a first facial image set, wherein a first facial image in the first facial image set is correspondingly annotated with an eye rotation angle based on a head coordinate system;

[0009] determining an eye rotation angle category label for the first facial image according to the eye rotation angle based on the head coordinate system marked corresponding to the first facial image;

[0010] Acquire a second facial image set, and perform key point detection on the eye regions of the second facial images in the second facial image set to obtain eye key points of the second facial images;

[0011] determining an eye rotation angle category label of the second facial image based on eye key points of the second facial image;

[0012] The network model to be trained is trained using the first face image set, the eye rotation angle category labels of the first face images, the second face image set and the eye rotation angle category labels of the second face images to obtain a target network model for predicting the eye rotation angle based on the head coordinate system in face images.

[0013] In a second aspect, the present disclosure further provides an image processing method, comprising:

[0014] Obtaining a target face image for which eye rotation angles based on a head coordinate system are to be predicted;

[0015] The target facial image is input into a target network model so that the target network model outputs an eye rotation angle predicted for the target facial image based on a head coordinate system, and the target network model is trained according to the method described in the first aspect above.

[0016] In a third aspect, the present disclosure further provides a data processing device, comprising:

[0017] an image acquisition module configured to acquire a first facial image set, wherein a first facial image in the first facial image set is labeled with an eye rotation angle based on a head coordinate system;

[0018] a first label determination module configured to determine an eye rotation angle category label of the first facial image based on the eye rotation angle based on the head coordinate system annotated corresponding to the first facial image;

[0019] an image acquisition and key point detection module, configured to acquire a second facial image set, and perform key point detection on the eye regions of the second facial images in the second facial image set to obtain eye key points of the second facial images;

[0020] A second label determination module is configured to determine an eye rotation angle category label of the second facial image based on eye key points of the second facial image;

[0021] The model training module is configured to train the network model to be trained through the first face image set, the eye rotation angle category label of the first face image, the second face image set and the eye rotation angle category label of the second face image, and obtain a target network model for predicting the eye rotation angle based on the head coordinate system in the face image.

[0022] In a fourth aspect, the present disclosure further provides an electronic device, comprising: a processor, a memory, and computer program instructions stored in the memory and executable on the processor;

[0023] When the processor executes the computer program instructions, it implements the data processing method described in the first aspect and / or the image processing method described in the second aspect.

[0024] In a fifth aspect, the present disclosure also provides a computer-readable storage medium, which stores computer program instructions. When the computer program instructions are executed by a processor, they are used to implement the data processing method described in the first aspect above, and / or the image processing method described in the second aspect.

[0025] The exemplary embodiments of the present disclosure have the following beneficial effects:

[0026] The data processing method provided by the embodiment of the present disclosure obtains a first facial image set, wherein the first facial image in the first facial image set is labeled with an eye rotation angle based on a head coordinate system; determines the eye rotation angle category label of the first facial image based on the eye rotation angle labeled with the first facial image based on the head coordinate system; obtains a second facial image set, and performs key point detection on the eye area of ​​the second facial image in the second facial image set to obtain the eye key points of the second facial image; determines the eye rotation angle category label of the second facial image based on the eye key points of the second facial image; trains a network model to be trained using the first facial image set, the eye rotation angle category label of the first facial image, the second facial image set, and the eye rotation angle category label of the second facial image to obtain a target network model for predicting the eye rotation angle based on the head coordinate system in facial images. The present disclosure introduces facial images without eye rotation angle labels to perform weakly supervised training on the network model, so that the network model can take into account both accuracy and generalization and can be applied to a variety of scenarios. For facial images without labeled eye rotation angles, the eye rotation angle labels can be categorized based on the eye key points, and the categorized eye rotation angle labels can be used for weakly supervised training of the network model. Therefore, only a small number of facial images labeled with eye rotation angles are needed to train a network model with excellent accuracy and generalization for predicting eye rotation angles in facial images based on the head coordinate system, without the need to collect large-scale labeled datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] FIG1 is a flow chart of a data processing method provided by one embodiment of the present disclosure;

[0028] FIG2 is a schematic diagram of a head coordinate system provided by one embodiment of the present disclosure;

[0029] FIG3 is a schematic diagram of a network model training process provided by one embodiment of the present disclosure;

[0030] FIG4 is a block diagram of a data processing device provided by one embodiment of the present disclosure;

[0031] FIG5 is a schematic diagram of a logical structure of an electronic device for implementing data processing provided by one embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the disclosure for which protection is sought, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, every other embodiment obtained by those skilled in the art without making creative work falls within the scope of protection of the present disclosure.

[0033] The terms "a", "an", "the" and "said" are used in this specification to indicate the presence of one or more elements / components / etc.; the terms "including" and "having" are used to express open-ended inclusion and mean that additional elements / components / etc. may exist in addition to the listed elements / components / etc.; the terms "first" and "second" etc. are used only as labels and are not intended to limit the quantity of their objects.

[0034] It should be understood that in the embodiments of the present disclosure, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "Including A, B and / or C" means including any one, any two, or any three of A, B, and C.

[0035] It should be understood that in the embodiments of the present disclosure, "B corresponding to A," "B corresponding to A," "A corresponds to B," or "B corresponds to A" means that B is associated with A and B can be determined based on A. Determining B based on A does not mean determining B based solely on A; B can also be determined based on A and / or other information.

[0036] Before describing the embodiments of the present disclosure in detail, the related art will be further introduced.

[0037] Current work related to eye angle estimation mainly focuses on calculating the eye gaze direction in the camera coordinate system, including geometry-based methods and appearance-based methods.

[0038] The basic steps of geometry-based methods include feature detection and angle estimation. First, features are detected from the input image, such as the locations of key points like the pupil and eye corners. This method primarily includes corner detection methods based on traditional image features and those based on convolutional neural networks. Then, the eye gaze direction is calculated using rules designed based on these features. These methods place high demands on image quality and resolution. Poor image quality results in poor accuracy.

[0039] The basic steps of appearance-based methods include data collection, angle estimation model training, and model prediction. This involves directly learning a model that maps images to gaze using large amounts of data. During the data collection phase, specialized hardware (such as eye trackers) is primarily used to collect large amounts of facial image data with gaze directions. During model training, most methods leverage the fitting capabilities of convolutional neural networks and improve model accuracy by designing different network structures, such as combining features from the entire face and the eye region. Finally, the predicted image is input into the trained model to output the prediction result. However, these solutions have low generalizability across different scenarios and individuals. Prediction accuracy decreases when the actual scenario deviates significantly from the training data.

[0040] The above-mentioned solutions mainly have the following problems:

[0041] 1. The relevant solutions calculate the eye gaze angle in the camera coordinate system, and cannot obtain the eye rotation angle based on the head coordinate system. Therefore, they cannot be directly used to drive virtual characters.

[0042] 2. The datasets used in related solutions are all annotated with eye gaze angles in the camera coordinate system. Currently, there is no large-scale dataset with annotations of eye rotation angles in the head coordinate system that can directly support model training based on eye rotation angles in the head coordinate system. Recreating a usable dataset requires a lot of manpower, material resources, and time, and is also limited by the collection environment.

[0043] Based on this, the present disclosure provides a data processing method by which a network model for estimating eye rotation angles can be trained. Facial images without eye rotation angle labels are introduced for weak supervision training, so that the network model can take into account both accuracy and generalization and can be applied to a variety of scenarios. For facial images without eye rotation angle labels, a classification method for eye rotation angle labels based on eye key points is designed, and the categorized eye rotation angle labels are used for weak supervision training of the network model. Therefore, only a small number of facial images with eye rotation angle labels are needed to train a network model with excellent accuracy and generalization, without the need to collect a large-scale labeled data set.

[0044] The execution subject of the method can be a terminal device or a server, and the terminal device can be a desktop computer, a laptop computer, a tablet computer, a mobile phone, etc., or other electronic devices, which is not specifically limited in this disclosure. The server is used to provide background services for the client of the application in the terminal device. For example, the server can be the background server of the above-mentioned application. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms, which is not specifically limited in this disclosure.

[0045] FIG1 shows a data processing method provided by one embodiment of the present disclosure. As shown in FIG1 , the method includes the following steps S101 to S105 .

[0046] Step S101: Acquire a first facial image set, wherein the first facial image in the first facial image set is correspondingly annotated with an eye rotation angle based on a head coordinate system.

[0047] The first facial image set includes multiple first facial images, each of which is marked with an eye rotation angle in the image, and the eye rotation angle is the rotation angle of the eye in the first facial image based on the head coordinate system. Optionally, for the first facial image, a facial capture device can be used to obtain the eye rotation angle when shooting the first facial image, thereby obtaining the true value of the eye rotation angle corresponding to the first facial image. Exemplarily, a head-mounted camera device can be used to collect data. Since the head-mounted camera device and the head remain relatively still, the collected eye rotation angle is based on the head coordinate system.

[0048] Optionally, the head coordinate system may include a Fick coordinate system, as shown in Figure 2. In the Fick coordinate system, a plane that is orthogonal to the sagittal plane of the head and passes through the rotation centers of both eyes is called the Listing plane, also known as the frontal plane.

[0049] In the Fick coordinate system, eye movement can be regarded as a rotation around the three coordinate axes XX', YY' and ZZ', wherein the rotation along the XX' axis includes upward rotation and downward rotation (hereinafter collectively referred to as up and down rotation), the rotation along the ZZ' axis includes inward rotation and outward rotation (hereinafter collectively referred to as left and right rotation), and the rotation along the YY' axis includes internal rotation and external rotation. In the embodiment of the present disclosure, the internal rotation and external rotation of the eye are ignored, and only the up and down rotation and left and right rotation of the eye are considered, wherein up and down and left and right both refer to the up and down and left and right of the face.

[0050] Keep the relative position relationship between the YY' coordinate axis and the eyeball unchanged. The up and down rotation is the rotation of the eyeball around the XX' coordinate axis in the Fick coordinate system. The angle of the up and down rotation of the eyeball is the angle between the YY' coordinate axis after rotation and the YY' coordinate axis before rotation.

[0051] Keep the relative position relationship between the YY' coordinate axis and the eyeball unchanged. Left and right rotation means the eyeball rotates around the ZZ' coordinate axis in the Fick coordinate system. The angle of left and right rotation of the eyeball is the angle between the YY' coordinate axis after rotation and the YY' coordinate axis before rotation.

[0052] Step S102: determining an eye rotation angle category label of the first facial image according to the eye rotation angle based on the head coordinate system marked corresponding to the first facial image.

[0053] In one embodiment of the present disclosure, angle category labels for the eyeballs in the first direction and the eyeballs in the second direction in the first facial image can be generated according to the left-right and up-down rotations of the eyeballs, respectively. Accordingly, step S10:2 can be implemented through the following steps S1021 to S1022.

[0054] It should be noted in advance that in the following steps S1021 to S1022, the first direction and the second direction are perpendicular to each other, and the second direction is the direction of the facial center axis in the face image (including the first face image and the second face image described below).

[0055] Step S1021: Determine the angle category label of the eye rotation in the first direction in the first facial image based on the head coordinate system, the preset eye rotation angle range, and the equal angles obtained by dividing the preset eye rotation angle range into equal parts, according to the eye rotation angle corresponding to the first facial image.

[0056] In an optional embodiment, the angle category label Labelcls_zy2 of the eyeball rotating in the first direction in the first face image, that is, the angle category label of left and right rotation, can be determined by the following formula (4);

[0057] Among them, Labelreg_zy represents the eye rotation angle based on the head coordinate system in the first direction corresponding to the first face image. The preset eye rotation angle range is [-m, m]. Every n degrees corresponds to an eye rotation angle category, and n represents the equal angle obtained by dividing the preset eye rotation angle range [-m, m] into equal parts.

[0058] in, Is the floor symbol. If a value x is an integer, then x is rounded down to x itself, that is, If a value x is not an integer, then x is rounded down to the integer part obtained by removing the decimal part of x.

[0059] In an exemplary embodiment of the present disclosure, the eye movement angle category and the angle category label may be integers starting from 0, ie, [0, 1, 2, 3, ...].

[0060] Step S1022: Determine the angle category label of the eye rotation in the second direction in the first facial image based on the head coordinate system, the preset eye rotation angle range, and the equal angles obtained by dividing the preset eye rotation angle range into equal parts, according to the eye rotation angle corresponding to the first facial image.

[0061] In an optional embodiment, the angle category label Labelcls_ss2 of the eyeball in the first face image rotating in the second direction, i.e., the angle category label of the up-down rotation, can be determined by the following formula (5);

[0062] Among them, Labelreg_ss represents the eye rotation angle based on the head coordinate system in the second direction corresponding to the first face image. The preset eye rotation angle range is [-m, m]. Every n degrees corresponds to an eye rotation angle category, and n represents the equal angle obtained by dividing the preset eye rotation angle range [-m, m] into equal parts.

[0063] Step S103: obtaining a second facial image set, and performing key point detection on the eye regions of the second facial images in the second facial image set to obtain eye key points of the second facial images.

[0064] The second facial images in the second facial image set are not labeled with eye rotation angles. Alternatively, ordinary RGB images can be obtained as the second facial images by capturing them with any camera or crawling the web. Since no labeling is required, a large number of second facial images can be obtained at a low cost.

[0065] It should be noted that the first face image and the second face image that can be used to train the model need to be face images that can identify eye movement, and face images that include conditions such as closed eyes need to be screened out.

[0066] In one embodiment of the present disclosure, the eye key points of the second facial image may include a key point of the left corner of the image, a key point of the right corner of the image, a key point of the pupil center, and a key point of the lower eyelid center.

[0067] It should be noted that in the embodiments of the present disclosure, the determination of the eye rotation angle category label is for a single eye (left eye or right eye). Therefore, the key points of the left corner of the image and the right corner of the image described in this embodiment belong to the same eye. For a face image, the left corner of the left eye in the image (i.e., the actual person's right eye) is the outer corner of the eye, and the right corner of the eye is the inner corner of the eye; the left corner of the right eye in the image (i.e., the actual person's left eye) is the inner corner of the eye, and the right corner of the eye is the outer corner of the eye. Among them, the inner corner of the eye refers to the corner of the eye close to the bridge of the nose, and the outer corner of the eye refers to the corner of the eye close to the temple.

[0068] In an optional embodiment, before proceeding to step S104 below, data cleaning can be performed on the second facial image set with identified eye key points. Because some key point detection may be inaccurate, such facial image data can affect network model training. Therefore, it is necessary to filter out data with inaccurate key points. This filtered data can be processed by manually re-labeling key points or directly removing them. Furthermore, second facial images with closed eyes, where the eye rotation angle cannot be discerned, should also be filtered out to facilitate subsequent model training.

[0069] Step S104: determining an eye rotation angle category label of the second facial image based on the eye key points of the second facial image.

[0070] In one embodiment of the present disclosure, step S104 may be implemented through the following steps S1041 to S1042.

[0071] Step S1041: Determine the angle category label of the eye rotation in the first direction in the second facial image based on the distance between the key point of the left eye corner of the image and the key point of the pupil center in the first direction, the distance between the key point of the right eye corner of the image and the key point of the pupil center in the first direction, the corresponding first bias parameter in the first direction, the preset eye rotation angle range, and the equal angles obtained by dividing the preset eye rotation angle range into equal parts.

[0072] In an optional embodiment, the angle category label Labelcls_zy1 of the eyeball in the second face image rotating in the first direction, i.e., the angle category label of left and right rotation, can be determined by the following formula (2);

[0073] Among them, dleft represents the distance between the key point of the left eye corner of the image and the key point of the pupil center in the first direction, dright represents the distance between the key point of the right eye corner of the image and the key point of the pupil center in the first direction, offsethorrizontal represents the first offset parameter corresponding to the first direction, the preset eye rotation angle range is [-m, m], every n degrees corresponds to an eye rotation angle category, and n represents the equal angle obtained by dividing the preset eye rotation angle range [-m, m] into equal parts.

[0074] Step S1042: Determine the angle category label of the eyeball rotation in the second direction in the second facial image based on the directed distance between the lower eyelid center key point and the pupil center key point in the second direction, the distance between the right eye corner key point and the pupil center key point in the first direction, the distance between the left eye corner key point and the pupil center key point in the first direction, the first bias parameter corresponding to the first direction, the second bias parameter corresponding to the second direction, the preset eyeball rotation angle range, and the equal angle obtained by dividing the preset eyeball rotation angle range into equal parts.

[0075] In an optional embodiment, the angle category label Labelcls_ss1 of the eyeball in the second face image rotating in the second direction, i.e., the angle category label of the eyeball rotating up and down, can be determined by the following formula (3);

[0076] Among them, dvertical represents the signed distance between the lower eyelid center key point and the pupil center key point in the second direction. When the pupil center key point is above the center point of the left eye corner key point and the right eye corner key point of the image, the signed distance is positive. When the pupil center key point is below the center point of the left eye corner key point and the right eye corner key point of the image, the signed distance is negative. offsethorrizontal represents the first offset parameter corresponding to the first direction. dright represents the distance between the right eye corner key point and the pupil center key point in the first direction. dleft represents the distance between the left eye corner key point and the pupil center key point in the first direction. offsetvertical represents the second offset parameter corresponding to the second direction. The preset eye rotation angle range is [-m, m]. Every n degrees corresponds to an eye rotation angle category. n represents the equal angle obtained by dividing the preset eye rotation angle range [-m, m] into equal parts.

[0077] For the second face image without the true value annotation of the eye rotation angle, the determination of the angle category label for the up and down rotation takes into account the distance between the pupil center and the lower eyelid, thus avoiding the problem that the angle category label cannot be directly obtained by the distance between the upper and lower eyelids and the pupil center due to the different closure states of the upper and lower eyelids.

[0078] In one embodiment of the present disclosure, the eye rotation angle category label for the first facial image includes an angle category label for eye rotation in a first direction in the first facial image, and an angle category label for eye rotation in a second direction in the first facial image. Based on this, the first offset parameter "offsethorrizontal" and the second offset parameter "offsetvertical" in the above formula can be determined by the following method, including steps S201 to S204.

[0079] Step S201: determining a reference first bias parameter corresponding to the first direction according to the angle category label of the eyeball rotation in the first direction in the first face image using formula (2);

[0080] Step S202: averaging the reference first offset parameters determined for each first facial image to obtain a first offset parameter corresponding to the first direction;

[0081] Step S203: determining a reference second bias parameter corresponding to the second direction according to the angle category label of the eyeball rotation in the second direction in the first face image and the first bias parameter using formula (3);

[0082] Step S204: averaging the reference second offset parameters determined for each first facial image to obtain a corresponding second offset parameter in the second direction.

[0083] In this embodiment, for the first facial image with the true value of the eyeball rotation angle annotated, the above formula (4) and formula (5) can be used to determine the angle category label Labelcls_zy2 for the left and right eyeball rotation in the first facial image, and the angle category label Labelcls_ss2 for the up and down eyeball rotation in the first facial image. Labelcls_zy2 can then be substituted into formula (2). At this time, only the first offset parameter offsethorrizontal is unknown, so the reference first offset parameter offset'horrizontal for each first facial image can be determined. The first offset parameter offsethorrizontal can be obtained by determining the average value of the reference first offset parameter offset'horrizontal for each first facial image.

[0084] Similarly, Labelcls_ss2 and the first offset parameter offsethorrizontal determined in the above steps are substituted into formula (3). At this point, only the second offset parameter offsetvertical is unknown. This allows the reference second offset parameter offset'vertical for each first facial image to be determined. The second offset parameter offsetvertical is obtained by averaging the reference second offset parameters offset'vertical for each first facial image.

[0085] Step S105: Train the network model to be trained using the first face image set, the eye rotation angle category label of the first face image, the second face image set, and the eye rotation angle category label of the second face image to obtain a target network model for predicting the eye rotation angle based on the head coordinate system in the face image.

[0086] The network model to be trained is trained using a first facial image set, an eye rotation angle category label determined for each first facial image in the first facial image set, and a second facial image set, an eye rotation angle category label determined for each second facial image in the second facial image set. The second facial images, which are not labeled with true eye rotation angle values, and the eye rotation angle category labels for the second facial images are used for weakly supervised training of the network model.

[0087] Weakly supervised training refers to using noisy or inaccurately labeled data for model training. In the disclosed embodiment, for a second facial image that does not have the corresponding true value of the eye rotation angle labeled, equations (2) and (3) are used to generate eye rotation angle category labels based on the eye key points in the second facial image. However, this label itself is not necessarily accurate, so using the second facial image and its corresponding eye rotation angle category label for model training is called weakly supervised training.

[0088] Among them, the reasons why the eye rotation angle category labels generated in this way are inaccurate include: 1. There are errors in the determination of eye key points, errors in algorithm recognition, and errors in manual labeling; 2. The design of formula (2) and formula (3) itself is incomplete and cannot take into account various special cases. For example, the eye shapes of different people are different, which is difficult to consider separately. Therefore, the generated eye rotation angle category labels may be inaccurate.

[0089] In the disclosed embodiment, by using a set of facial images without the true values ​​of eye rotation angles to join the model training, weakly supervised model training can be introduced. Facial images without the true values ​​of eye rotation angles can contain more characters and other factors (such as different head postures and facial expressions). Therefore, when the network model learns this data information, its generalization ability in different scenarios will be improved. By introducing weakly supervised model training, only a small number of facial images with accurate true values ​​of eye rotation angles are needed to enable the network model to achieve higher accuracy and generalization.

[0090] In one embodiment of the present disclosure, referring to FIG3 , the training process of the network model includes determining a classification loss and a regression loss for a first face image, and determining a classification loss for a second face image.

[0091] In this embodiment, the classification loss and regression loss are determined for the face images with the true value of the eye rotation angle annotated. Since the true value of the eye rotation angle is relatively accurate, it is equivalent to introducing more supervision information for model training, and therefore they can be used simultaneously.

[0092] For facial images without true eye rotation angle annotations, if a specific eye rotation angle label is generated using a certain strategy, the accuracy of the generated eye rotation angle will be relatively low, and supervision through regression loss will deteriorate the learning quality of the model. This embodiment categorizes the eye rotation angle labels, thereby generating eye rotation angle category labels for facial images without true eye rotation angle annotations, rather than specific eye rotation angle labels. This makes the labels relatively accurate, and such labels can effectively supervise the training process of the network model.

[0093] For example, humans may not be able to determine the exact angle of an eye movement, but they can determine the approximate angle range, such as whether it is approximately 10 degrees, 30 degrees, or 60 degrees, and whether it is to the left or right. Therefore, by categorizing eye movement angle labels, we can effectively supervise the training process of the network model.

[0094] In an optional implementation, determining the classification loss and the regression loss for the first facial image may be achieved through the following steps S1051 to S1053.

[0095] Step S1051: For the first face image input into the network model, the network model outputs the probability values ​​of the eye rotation angles based on the head coordinate system in the first face image belonging to each eye rotation angle category, wherein the eye rotation angle category is obtained by dividing the preset eye rotation angle range into equal angles.

[0096] Assume that the preset eye rotation angle range [-m, m] can be divided into an angle range of every 5 degrees. Taking the division of 5 angle ranges as an example, 5 corresponding eye rotation angle categories are obtained, namely [0, 1, 2, 3, 4]. For the first face image input into the network model, the network model can output the eye rotation angle based on the head coordinate system in the first face image, which belongs to the probability value P(0) of eye rotation angle category 0, the probability value P(1) of eye rotation angle category 1, the probability value P(2) of eye rotation angle category 2, the probability value P(3) of eye rotation angle category 3, and the probability value P(4) of eye rotation angle category 4.

[0097] Exemplarily, the probability value may be a softmax probability value.

[0098] Among them, the eye rotation angle category corresponding to the highest probability value is the prediction result of the network model for the eye rotation angle category corresponding to the first face image.

[0099] Step S1052: Determine the classification loss function value of the first facial image based on the first target eye rotation angle category with the highest probability value output by the network model for the first facial image and the eye rotation angle category label of the first facial image.

[0100] In this step, the classification loss function value of the first face image can be determined by comparing the prediction result a of the network model for the eye rotation angle category corresponding to the first face image and the eye rotation angle category label b determined based on the true value annotation of the eye rotation angle of the first face image.

[0101] The present disclosure does not limit the functional form of the classification loss function. Exemplarily, the classification loss function can be a cross entropy loss function.

[0102] Step S1053: Determine the regression loss function value of the first face image according to the probability values ​​of the eye rotation angles based on the head coordinate system in the first face image output by the network model belonging to each eye rotation angle category, the equal angles obtained by dividing the preset eye rotation angle range into equal parts, and the preset eye rotation angle range.

[0103] In an optional embodiment, the regression loss function value Lreg of the first face image can be determined by the following formula (1): L reg =[∑P(x i )*x i ]*nm Formula (1)

[0104] Among them, xi represents the eye rotation angle category, P(xi) represents the probability value of the eye rotation angle based on the head coordinate system in the first face image output by the network model belongs to the eye rotation angle category xi, the first face image corresponds to the eye rotation angle based on the head coordinate system marked in the second direction, the preset eye rotation angle range is [-m, m], every n degrees corresponds to an eye rotation angle category, and n represents the equal angle obtained by dividing the preset eye rotation angle range [-m, m] into equal parts.

[0105] Continuing with the above example, xi in the above formula (1) can be substituted into [0,1,2,3,4].

[0106] Correspondingly, the training process of the network model may further include: adjusting parameters of the network model according to the classification loss function value and the regression loss function value of the first face image.

[0107] In one embodiment, the classification loss function value and the regression loss function value of the first facial image may be summed, and the network model parameters may be adjusted based on the summation result.

[0108] In an optional implementation, determining the classification loss for the second facial image may be achieved through the following steps S1054 to S1055.

[0109] Step S1054: For the second face image input into the network model, the network model outputs the probability values ​​of the eye rotation angles based on the head coordinate system in the second face image belonging to each eye rotation angle category, wherein the eye rotation angle category is obtained by dividing the preset eye rotation angle range into equal angles.

[0110] Step S1055: Determine the classification loss function value of the second facial image based on the second target eye rotation angle category with the highest probability value output by the network model for the second facial image and the eye rotation angle category label of the second facial image.

[0111] The implementation of steps S1054 to S1055 may refer to the implementation of steps S1051 to S1052 above, and will not be described in detail here.

[0112] Correspondingly, the training process of the network model may further include: adjusting parameters of the network model according to the classification loss function value of the second face image.

[0113] In practical applications, there is no restriction on the order in which the first face image set and the second face image set are input into the network model during training.

[0114] Exemplarily, the first face image set can be input and the classification loss and regression loss can be determined. After the network model parameters are adjusted according to the classification loss and regression loss of the first face image, the second face image set can be input and the classification loss can be determined. Then, the network model parameters can be adjusted according to the classification loss of the second face image.

[0115] Exemplarily, the second face image set can be input first and the classification loss can be determined. After adjusting the network model parameters according to the classification loss of the second face image, the first face image set can be input and the classification loss and regression loss can be determined. Then, the network model parameters can be adjusted according to the classification loss and regression loss of the first face image.

[0116] Exemplarily, the first face image set and the second face image set can also be input at the same time, and the two can be marked to distinguish them, so that the network model can determine which face images need to determine the classification loss and regression loss, and which face images only need to determine the classification loss based on the marks.

[0117] In addition, the present disclosure does not specifically limit the network model. Exemplarily, the network model may include a residual convolutional neural network (Residual Network, ResNet) model.

[0118] This data processing method can train a network model for eye rotation angle estimation. By replacing simple regression training with a classification-then-regression training method and introducing facial images without eye rotation angle labels for weak supervision, the network model achieves both accuracy and generalization, making it applicable to a variety of scenarios. For facial images without eye rotation angle labels, a method for generating eye rotation angle category labels based on eye key points is designed. These categorized eye rotation angle labels are then used for weak supervision training of the network model. As a result, only a small number of facial images with eye rotation angle labels are needed to train a network model with excellent accuracy and generalization, without the need for collecting large-scale labeled datasets.

[0119] The data processing method provided by the embodiment of the present disclosure obtains a first facial image set, wherein the first facial image in the first facial image set is labeled with an eye rotation angle based on a head coordinate system; determines the eye rotation angle category label of the first facial image based on the eye rotation angle labeled with the first facial image based on the head coordinate system; obtains a second facial image set, and performs key point detection on the eye area of ​​the second facial image in the second facial image set to obtain the eye key points of the second facial image; determines the eye rotation angle category label of the second facial image based on the eye key points of the second facial image; trains a network model to be trained using the first facial image set, the eye rotation angle category label of the first facial image, the second facial image set, and the eye rotation angle category label of the second facial image to obtain a target network model for predicting the eye rotation angle based on the head coordinate system in facial images. The present disclosure introduces facial images without eye rotation angle labels to perform weakly supervised training on the network model, so that the network model can take into account both accuracy and generalization and can be applied to a variety of scenarios. For facial images without labeled eye rotation angles, the eye rotation angle labels can be categorized based on the eye key points, and the categorized eye rotation angle labels can be used for weakly supervised training of the network model. Therefore, only a small number of facial images labeled with eye rotation angles are needed to train a network model with excellent accuracy and generalization for predicting eye rotation angles in facial images based on the head coordinate system, without the need to collect large-scale labeled datasets.

[0120] The present disclosure also provides an image processing method, comprising:

[0121] Step S301: Obtain a target face image for which eye rotation angles based on a head coordinate system are to be predicted.

[0122] Step S302: Input the target face image into the target network model so that the target network model outputs the eye rotation angle based on the head coordinate system predicted for the target face image, and the target network model is trained according to the data processing method provided by the present disclosure.

[0123] Corresponding to the data processing method provided in the embodiment of the present disclosure, the embodiment of the present disclosure also provides a data processing device. As shown in FIG4 , the data processing device 700 includes:

[0124] The image acquisition module 701 is configured to acquire a first facial image set, wherein a first facial image in the first facial image set is labeled with an eye rotation angle based on a head coordinate system;

[0125] A first label determination module 702 is configured to determine an eye rotation angle category label of the first facial image based on the eye rotation angle based on the head coordinate system annotated corresponding to the first facial image;

[0126] The image acquisition and key point detection module 703 is configured to acquire a second facial image set and perform key point detection on the eye regions of the second facial images in the second facial image set to obtain eye key points of the second facial images;

[0127] The second label determination module 704 is configured to determine the eye rotation angle category label of the second facial image based on the eye key points of the second facial image;

[0128] The model training module 705 is configured to train the network model to be trained through the first face image set, the eye rotation angle category label of the first face image, the second face image set and the eye rotation angle category label of the second face image, and obtain a target network model for predicting the eye rotation angle based on the head coordinate system in the face image.

[0129] Optionally, the model training module includes:

[0130] A first loss determination submodule is configured to determine a classification loss and a regression loss for the first face image, and a second loss determination submodule is configured to determine a classification loss for the second face image.

[0131] Optionally, the first loss determination submodule includes:

[0132] a first output unit configured to execute, for the first facial image input into the network model, outputting, through the network model, probability values ​​of eye rotation angles in the first facial image based on a head coordinate system belonging to each eye rotation angle category, wherein the eye rotation angle category is obtained by equally dividing a preset eye rotation angle range;

[0133] a first classification loss determining unit configured to determine a classification loss function value of the first facial image based on the first target eye movement angle category with the highest probability value output by the network model for the first facial image and the eye movement angle category label of the first facial image;

[0134] a regression loss determining unit configured to determine a regression loss function value of the first facial image based on a probability value of each eye rotation angle category in the first facial image based on the head coordinate system output by the network model, an equal angle obtained by equally dividing the preset eye rotation angle range, and the preset eye rotation angle range;

[0135] The model training module is specifically configured to perform: adjusting the parameters of the network model according to the classification loss function value and the regression loss function value of the first facial image.

[0136] Optionally, the regression loss determination unit is specifically configured to execute:

[0137] The regression loss function value Lreg of the first face image is determined by the following formula (1); L reg =[∑P(x i )*x i ]*nm Formula (1)

[0138] Among them, xi represents the eye rotation angle category, P(xi) represents the probability value of the eye rotation angle based on the head coordinate system in the first face image output by the network model belongs to the eye rotation angle category xi, the first face image corresponds to the eye rotation angle based on the head coordinate system marked in the second direction, the preset eye rotation angle range is [-m, m], every n degrees corresponds to an eye rotation angle category, and n represents the equal angle obtained by dividing the preset eye rotation angle range [-m, m] into equal parts.

[0139] Optionally, the second loss determination submodule includes:

[0140] a second output unit configured to execute, for the second facial image input into the network model, outputting, through the network model, probability values ​​of eye rotation angles in the second facial image based on the head coordinate system belonging to each eye rotation angle category, wherein the eye rotation angle category is obtained by equally dividing a preset eye rotation angle range;

[0141] a second classification loss determining unit configured to determine a classification loss function value of the second facial image based on the second target eye movement angle category with the highest probability value output by the network model for the second facial image and the eye movement angle category label of the second facial image;

[0142] The model training module is specifically configured to perform: adjusting parameters of the network model according to the classification loss function value of the second facial image.

[0143] Optionally, the eye key points of the second facial image include a left eye corner key point, a right eye corner key point, a pupil center key point, and a lower eyelid center key point;

[0144] The second tag determination module includes:

[0145] The first left-right rotation label determination submodule is configured to determine an angle category label of the eyeball rotation in the first direction in the second facial image based on the distance between the left eye corner key point and the pupil center key point in the first direction, the distance between the right eye corner key point and the pupil center key point in the first direction, a first offset parameter corresponding to the first direction, a preset eyeball rotation angle range, and an equal angle obtained by equally dividing the preset eyeball rotation angle range;

[0146] The first up-down rotation label determination submodule is configured to determine the angle category label of the eyeball rotation in the second direction in the second facial image based on the directed distance between the lower eyelid center key point and the pupil center key point in the second direction, the distance between the right eye corner key point and the pupil center key point in the first direction, the distance between the left eye corner key point and the pupil center key point in the first direction, a first offset parameter corresponding to the first direction, a second offset parameter corresponding to the second direction, a preset eyeball rotation angle range, and an equal angle obtained by equally dividing the preset eyeball rotation angle range;

[0147] The first direction and the second direction are perpendicular to each other, and the second direction is the direction of the facial center axis in the face image.

[0148] Optionally, the first left-right rotation label determination submodule is specifically configured to execute:

[0149] Determine the angle category label Labelcls_zy1 of the eyeball rotation in the first direction in the second face image by the following formula (2);

[0150] Wherein, dleft represents the distance between the left eye corner key point of the image and the pupil center key point in the first direction, dright represents the distance between the right eye corner key point of the image and the pupil center key point in the first direction, offsethorrizontal represents the first offset parameter corresponding to the first direction, the preset eye rotation angle range is [-m, m], each n degrees corresponds to an eye rotation angle category, and n represents the equal angle obtained by dividing the preset eye rotation angle range [-m, m] into equal parts;

[0151] The first up-and-down rotation label determination submodule is specifically configured to execute:

[0152] Determine the angle category label Labelcls_ss1 of the eyeball rotation in the second direction in the second face image by the following formula (3);

[0153] Among them, dvertical represents the directed distance between the lower eyelid center key point and the pupil center key point in the second direction. When the pupil center key point is above the center point of the left eye corner key point and the right eye corner key point of the image, the directed distance is positive. When the pupil center key point is below the center point of the left eye corner key point and the right eye corner key point of the image, the directed distance is negative. offsetvertical represents the second offset parameter corresponding to the second direction.

[0154] Optionally, the eyeball rotation angle category label of the first facial image includes an angle category label for determining eyeball rotation in a first direction in the first facial image, and an angle category label for determining eyeball rotation in a second direction in the first facial image, wherein the first direction and the second direction are perpendicular to each other, and the second direction is a direction of a facial central axis in the facial image;

[0155] The method for determining the first bias parameter and the second bias parameter includes:

[0156] Determine a reference first bias parameter corresponding to the first direction using formula (2) according to the angle category label of the eyeball rotation in the first direction in the first facial image;

[0157] averaging the reference first bias parameters determined for each first facial image to obtain a corresponding first bias parameter in the first direction;

[0158] Determine a reference second bias parameter corresponding to the second direction using formula (3) according to the angle category label of the eyeball rotation in the second direction in the first facial image and the first bias parameter;

[0159] An average of the reference second offset parameters determined for each first facial image is taken to obtain a corresponding second offset parameter in the second direction.

[0160] Optionally, the first tag determination module includes:

[0161] a second left-right rotation label determination submodule configured to determine an angle category label for eye rotation in the first direction in the first facial image based on the eye rotation angle in the first direction marked in the first facial image based on the head coordinate system, a preset eye rotation angle range, and equal angles obtained by equally dividing the preset eye rotation angle range;

[0162] a second up-down rotation label determination submodule configured to determine an angle category label for eye rotation in the second direction in the first facial image based on the eye rotation angle in the second direction marked in the first facial image based on the head coordinate system, a preset eye rotation angle range, and equal angles obtained by equally dividing the preset eye rotation angle range;

[0163] The first direction and the second direction are perpendicular to each other, and the second direction is the direction of the facial center axis in the face image.

[0164] Optionally, the second left-right rotation label determination submodule is specifically configured to execute:

[0165] Determine the angle category label Labelcls_zy2 of the eyeball rotation in the first direction in the first face image by the following formula (4);

[0166] Wherein, Labelreg_zy represents the eye rotation angle based on the head coordinate system in the first direction corresponding to the first facial image, the preset eye rotation angle range is [-m, m], each n degrees corresponds to an eye rotation angle category, and n represents the equal angle obtained by dividing the preset eye rotation angle range [-m, m] into equal parts;

[0167] The second up-and-down rotation label determination submodule is specifically configured to execute:

[0168] Determine the angle category label Labelcls_ss2 of the eyeball rotation in the second direction in the first face image by the following formula (5);

[0169] Among them, Labelreg_ss represents the eye rotation angle based on the head coordinate system in the second direction corresponding to the first facial image.

[0170] The data processing device provided by the embodiment of the present disclosure obtains a first facial image set, wherein the first facial image in the first facial image set is labeled with an eye rotation angle based on a head coordinate system, and determines the eye rotation angle category label of the first facial image based on the eye rotation angle labeled corresponding to the first facial image; obtains a second facial image set, and performs key point detection on the eye area of ​​the second facial image in the second facial image set to obtain the eye key points of the second facial image; determines the eye rotation angle category label of the second facial image based on the eye key points of the second facial image; trains a network model to be trained using the first facial image set, the eye rotation angle category label of the first facial image, the second facial image set, and the eye rotation angle category label of the second facial image to obtain a target network model for predicting the eye rotation angle based on the head coordinate system in facial images. The present disclosure introduces facial images without eye rotation angle labels to perform weakly supervised training on the network model, so that the network model can balance accuracy and generalization and can be applied to a variety of scenarios. For facial images without labeled eye rotation angles, the eye rotation angle labels can be categorized based on the eye key points, and the categorized eye rotation angle labels can be used for weakly supervised training of the network model. Therefore, only a small number of facial images labeled with eye rotation angles are needed to train a network model with excellent accuracy and generalization for predicting eye rotation angles in facial images based on the head coordinate system, without the need to collect large-scale labeled datasets.

[0171] Next, an electronic device provided by an embodiment of the present disclosure is introduced. Please refer to Figure 5, which is a structural diagram of an electronic device provided by an embodiment of the present disclosure. Among them, the data processing device described in the embodiment of the present disclosure can be deployed on the electronic device 800 to implement the functions in the embodiment of the present disclosure. Specifically, the electronic device 800 includes: a receiver 801, a transmitter 802, a processor 803 and a memory 804 (wherein the number of processors 803 in the electronic device 800 can be one or more, and Figure 5 takes one processor as an example), wherein the processor 803 may include an application processor 8031 ​​and a communication processor 8032. In some embodiments of the present disclosure, the receiver 801, the transmitter 802, the processor 803 and the memory 804 may be connected via a bus or other means.

[0172] The memory 804 may include a read-only memory and a random access memory, and provides instructions and data to the processor 803. A portion of the memory 804 may also include non-volatile random access memory (NVRAM). The memory 804 stores processor and operation instructions, executable modules, or data structures, or subsets or extended sets thereof. The operation instructions may include various operation instructions for implementing various operations.

[0173] Processor 803 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all bus systems are referred to as a bus system in the figure.

[0174] The methods disclosed in the above embodiments of the present disclosure can be applied to or implemented by the processor 803. The processor 803 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor 803 or by software instructions. The above processor 803 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and can further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 803 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present disclosure. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present disclosure can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 804, and processor 803 reads information in memory 804 and, in conjunction with its hardware, completes the steps of the above method.

[0175] Receiver 801 can be used to receive input digital or character information and generate signal input related to executing device-related settings and function control. Transmitter 802 can be used to output digital or character information through the first interface. Transmitter 802 can also be used to send instructions to the disk pack through the first interface to modify data in the disk pack. Transmitter 802 can also include a display device such as a display screen.

[0176] In the embodiments of the present disclosure, the application processor 8031 ​​in the processor 803 is configured to execute the data processing method in the embodiments of the present disclosure. It should be noted that the specific manner in which the application processor 8031 ​​executes each step is based on the same concept as the various method embodiments in the present disclosure, and the technical effects achieved are the same as those of the various method embodiments in the present disclosure. For details, please refer to the description of the method embodiments shown above in the present disclosure, and will not be repeated here.

[0177] The embodiment of the present disclosure further provides a chip for executing instructions, which is used to execute the technical solution of the data processing method in the above embodiment.

[0178] An embodiment of the present disclosure further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on a processor, the processor executes the technical solution of the data processing method in the above embodiment.

[0179] The embodiments of the present disclosure further provide a computer program product, including a computer program, which is used to execute the technical solution of the data processing method in the above embodiments when executed by a processor.

[0180] The computer-readable storage medium may be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The computer-readable storage medium may be any available medium that can be accessed by a general-purpose or dedicated server.

[0181] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

[0182] Although the present disclosure is disclosed as above in terms of preferred embodiments, it is not intended to limit the present disclosure. Any person skilled in the art may make possible changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the scope of protection of the present disclosure shall be based on the scope defined by the claims of the present disclosure.

Claims

1. A data processing method, comprising: Acquire a first facial image set, wherein a first facial image in the first facial image set is correspondingly annotated with an eye rotation angle based on a head coordinate system; determining an eye rotation angle category label for the first facial image according to the eye rotation angle based on the head coordinate system marked corresponding to the first facial image; Acquire a second facial image set, and perform key point detection on the eye regions of the second facial images in the second facial image set to obtain eye key points of the second facial images; determining an eye rotation angle category label of the second facial image based on eye key points of the second facial image; The network model to be trained is trained using the first face image set, the eye rotation angle category labels of the first face images, the second face image set and the eye rotation angle category labels of the second face images to obtain a target network model for predicting the eye rotation angle based on the head coordinate system in face images.

2. The method according to claim 1, wherein The training of the network model to be trained includes: A classification loss and a regression loss are determined for the first face image, and a classification loss is determined for the second face image.

3. The method according to claim 2, wherein: The determining of the classification loss and the regression loss for the first face image includes: For the first facial image input into the network model, the network model outputs, through the network model, probability values ​​of eye rotation angles in the first facial image based on the head coordinate system belonging to each eye rotation angle category, wherein the eye rotation angle categories are obtained by equally dividing a preset eye rotation angle range; determining a classification loss function value of the first facial image based on the first target eye rotation angle category with the highest probability value output by the network model for the first facial image and the eye rotation angle category label of the first facial image; determining a regression loss function value for the first facial image based on probability values ​​of eye rotation angles based on the head coordinate system in the first facial image output by the network model belonging to each eye rotation angle category, equal angles obtained by equally dividing the preset eye rotation angle range, and the preset eye rotation angle range; The training of the network model to be trained includes: adjusting parameters of the network model according to the classification loss function value and the regression loss function value of the first face image.

4. The method according to claim 3, wherein: The method further comprises determining a regression loss function value of the first facial image based on the probability values ​​of the eye rotation angles based on the head coordinate system in the first facial image output by the network model belonging to each eye rotation angle category, the equal angles obtained by equally dividing the preset eye rotation angle range, and the preset eye rotation angle range, including: Determine the regression loss function value Lreg of the first face image by the following formula (1); L reg =[∑P(x i )*x i ]*nm Formula (1) Among them, xi represents the eye rotation angle category, P(xi) represents the probability value of the eye rotation angle based on the head coordinate system in the first face image output by the network model belongs to the eye rotation angle category xi, the first face image corresponds to the eye rotation angle based on the head coordinate system marked in the second direction, the preset eye rotation angle range is [-m, m], every n degrees corresponds to an eye rotation angle category, and n represents the equal angle obtained by dividing the preset eye rotation angle range [-m, m] into equal parts.

5. The method according to claim 1, wherein The determining of the classification loss for the second facial image includes: For the second facial image input into the network model, outputting, through the network model, probability values ​​of eye rotation angles in the second facial image based on the head coordinate system belonging to each eye rotation angle category, wherein the eye rotation angle categories are obtained by equally dividing a preset eye rotation angle range; determining a classification loss function value for the second facial image based on the second target eye rotation angle category with the highest probability value output by the network model for the second facial image and the eye rotation angle category label of the second facial image; The training of the network model to be trained includes: adjusting parameters of the network model according to the classification loss function value of the second facial image.

6. The method according to claim 1, wherein The eye key points of the second face image include the key points of the left corner of the image, the key points of the right corner of the image, the key points of the pupil center and the key points of the lower eyelid center; The step of determining the eye rotation angle category label of the second facial image according to the eye key points of the second facial image includes: Determining an angle category label for eye rotation in the first direction in the second facial image based on a distance between a left eye corner key point and a pupil center key point in the first direction, a distance between a right eye corner key point and the pupil center key point in the first direction, a first offset parameter corresponding to the first direction, a preset eye rotation angle range, and angles obtained by equally dividing the preset eye rotation angle range; Determine, based on the directed distance between the lower eyelid center key point and the pupil center key point in the second direction, the distance between the right eye corner key point and the pupil center key point in the first direction, the distance between the left eye corner key point and the pupil center key point in the first direction, a first offset parameter corresponding to the first direction, a second offset parameter corresponding to the second direction, a preset eye rotation angle range, and equal angles obtained by equally dividing the preset eye rotation angle range, an angle category label for the eye rotation in the second direction in the second facial image; The first direction and the second direction are perpendicular to each other, and the second direction is the direction of the facial center axis in the face image.

7. The method according to claim 6, wherein: The determining, based on the distance between the left eye corner key point of the image and the pupil center key point in the first direction, the distance between the right eye corner key point of the image and the pupil center key point in the first direction, a first offset parameter corresponding to the first direction, a preset eye rotation angle range, and equal angles obtained by equally dividing the preset eye rotation angle range, the angle category label of the eye rotation in the second facial image in the first direction includes: Determine the angle category label Labelcls_zy1 of the eyeball rotation in the first direction in the second face image by the following formula (2); Wherein, dleft represents the distance between the left eye corner key point of the image and the pupil center key point in the first direction, dright represents the distance between the right eye corner key point of the image and the pupil center key point in the first direction, offsethorrizontal represents the first offset parameter corresponding to the first direction, the preset eye rotation angle range is [-m, m], each n degrees corresponds to an eye rotation angle category, and n represents the equal angle obtained by dividing the preset eye rotation angle range [-m, m] into equal parts; The method of determining the angle category label of the eyeball rotation in the second direction in the second facial image based on the directed distance between the lower eyelid center key point and the pupil center key point in the second direction, the distance between the right eye corner key point and the pupil center key point in the first direction, the distance between the left eye corner key point and the pupil center key point in the first direction, a first offset parameter corresponding to the first direction, a second offset parameter corresponding to the second direction, a preset eyeball rotation angle range, and an equal angle obtained by equally dividing the preset eyeball rotation angle range, includes: Determine the angle category label Labelcls_ss1 of the eyeball rotation in the second direction in the second face image by the following formula (3); Among them, dvertical represents the directed distance between the lower eyelid center key point and the pupil center key point in the second direction. When the pupil center key point is above the center point of the left eye corner key point and the right eye corner key point of the image, the directed distance is positive. When the pupil center key point is below the center point of the left eye corner key point and the right eye corner key point of the image, the directed distance is negative. offsetvertical represents the second offset parameter corresponding to the second direction.

8. The method according to claim 7, wherein: The eyeball rotation angle category label of the first facial image includes an angle category label for determining eyeball rotation in a first direction in the first facial image, and an angle category label for determining eyeball rotation in a second direction in the first facial image, wherein the first direction and the second direction are perpendicular to each other, and the second direction is a direction of a facial central axis in the facial image; The method for determining the first bias parameter and the second bias parameter includes: Determine a reference first bias parameter corresponding to the first direction using formula (2) according to the angle category label of the eyeball rotation in the first direction in the first facial image; averaging the reference first bias parameters determined for each first facial image to obtain a corresponding first bias parameter in the first direction; Determine a reference second bias parameter corresponding to the second direction using formula (3) according to the angle category label of the eyeball rotation in the second direction in the first facial image and the first bias parameter; An average of the reference second offset parameters determined for each first facial image is taken to obtain a corresponding second offset parameter in the second direction.

9. The method according to claim 1, wherein The step of determining the eye rotation angle category label of the first facial image according to the eye rotation angle based on the head coordinate system marked corresponding to the first facial image includes: determining an angle category label for eye rotation in the first direction in the first facial image based on the head coordinate system and an eye rotation angle in the first direction corresponding to the eye rotation angle annotated in the first facial image, a preset eye rotation angle range, and angles obtained by equally dividing the preset eye rotation angle range; determining an angle category label for eye rotation in the second direction in the first facial image based on the head coordinate system and corresponding to the eye rotation angle in the second direction annotated in the first facial image, a preset eye rotation angle range, and equal angles obtained by equally dividing the preset eye rotation angle range; The first direction and the second direction are perpendicular to each other, and the second direction is the direction of the facial center axis in the face image.

10. The method according to claim 9, wherein: The determining, based on the eye rotation angle based on the head coordinate system in the first direction corresponding to the eye rotation angle annotated in the first facial image, a preset eye rotation angle range, and equal angles obtained by equally dividing the preset eye rotation angle range, includes: Determine the angle category label Labelcls_zy2 of the eyeball rotation in the first direction in the first face image by the following formula (4); Wherein, Labelreg_zy represents the eye rotation angle based on the head coordinate system in the first direction corresponding to the first facial image, the preset eye rotation angle range is [-m, m], each n degrees corresponds to an eye rotation angle category, and n represents the equal angle obtained by dividing the preset eye rotation angle range [-m, m] into equal parts; The determining, based on the eye rotation angle in the second direction corresponding to the eye rotation angle in the head coordinate system marked in the second direction in the first facial image, a preset eye rotation angle range, and equal angles obtained by equally dividing the preset eye rotation angle range, includes: Determine the angle category label Labelcls_ss2 of the eyeball rotation in the second direction in the first face image by the following formula (5); Among them, Labelreg_ss represents the eye rotation angle based on the head coordinate system in the second direction corresponding to the first facial image.

11. An image processing method, comprising: Obtaining a target face image for which eye rotation angles based on a head coordinate system are to be predicted; The target facial image is input into a target network model so that the target network model outputs an eye rotation angle predicted for the target facial image based on a head coordinate system, and the target network model is trained according to the method according to any one of claims 1 to 10.

12. A data processing device, comprising: an image acquisition module configured to acquire a first facial image set, wherein a first facial image in the first facial image set is labeled with an eye rotation angle based on a head coordinate system; a first label determination module configured to determine an eye rotation angle category label of the first facial image based on the eye rotation angle based on the head coordinate system annotated corresponding to the first facial image; an image acquisition and key point detection module, configured to acquire a second facial image set, and perform key point detection on the eye regions of the second facial images in the second facial image set to obtain eye key points of the second facial images; A second label determination module is configured to determine an eye rotation angle category label of the second facial image based on eye key points of the second facial image; The model training module is configured to train the network model to be trained through the first face image set, the eye rotation angle category label of the first face image, the second face image set and the eye rotation angle category label of the second face image, and obtain a target network model for predicting the eye rotation angle based on the head coordinate system in the face image.

13. An electronic device comprising: a processor, a memory, and computer program instructions stored on the memory and executable on the processor; When the processor executes the computer program instructions, it implements the data processing method according to any one of claims 1 to 10 and / or the image processing method according to claim 11.

14. A computer-readable storage medium, wherein computer program instructions are stored in the computer-readable storage medium, and when the computer program instructions are executed by a processor, they are used to implement the data processing method according to any one of claims 1 to 10 and / or the image processing method according to claim 11.

Images