A method and device for measuring eye position deviation angle based on three-dimensional reconstruction

Abstract

The application belongs to the technical field of eye position measurement, and discloses a kind of measurement method and device of eye position deflection angle based on three-dimensional reconstruction.The face image sequence of the person to be tested is collected;In the first neural network, the covering condition of the face image sequence is obtained, and the key frame image is determined;The key frame image is input into the second neural network, the feature point heat map is obtained, and is converted into face feature point coordinates;The objective function between face feature points and the projection point of face three-dimensional model is constructed, and the head posture corresponding to the key frame image is obtained;The eye position is initialized, the eye rotation angle of reference eye position is fixed as a preset angle based on the head posture, the three-dimensional coordinates of the eye in the head coordinate system in the reference eye position image are solved;The three-dimensional coordinates of the eye in the head coordinate system in the image to be measured are fixed, the eye rotation angle in the image to be measured is solved, and the deflection angle of the eye is obtained.The convenience and measurement accuracy of measuring eye position deflection angle are improved.

Description

Technical Field

[0001] This application relates to the field of eye position measurement technology, and in particular to a method and apparatus for measuring eye position deviation angle based on three-dimensional reconstruction. Background Technology

[0002] Strabismus refers to the intermittent or persistent inability of a patient's two eyes to simultaneously focus on a target. The line of sight from the eye in a strabismic state cannot be directed to the same object seen by the other eye; examples include horizontal esotropia (inward strabismus) or vertical strabismus (upward or downward strabismus). Strabismus is a common eye problem, affecting approximately 3% to 8% of the population, with about 3% of children experiencing it. Early diagnosis and intervention can lead to a cure, but delayed treatment can result in difficulties with binocular depth perception, causing serious psychological developmental delays in children. In adults, strabismus also presents numerous problems, such as impaired reading ability, increased risk of falls, and social anxiety, significantly impacting physical and mental health and greatly reducing quality of life. Therefore, it is crucial to measure and determine strabismus promptly. Measuring the degree of strabismus primarily involves measuring the angle of eye deviation, which is very helpful for subsequent auxiliary strabismus surgery planning, disease type diagnosis, and disease assessment. Currently, the widely used methods for measuring the angle of eye deviation mainly include: prism and cover test, corneal reflection test, visual field arc test, etc. However, the above measurement methods all depend on the patient's high level of cooperation to varying degrees, and are affected by the subjective factors of the examiner, which can lead to large deviations. Different examiners may obtain inconsistent test results, lacking objectivity and consistency. Summary of the Invention

[0003] Therefore, embodiments of this application provide a method and apparatus for measuring eye deviation angle based on three-dimensional reconstruction, which improves the convenience and accuracy of measuring eye deviation angle.

[0004] In one aspect, this application provides a method for measuring eye deviation angle based on three-dimensional reconstruction.

[0005] This application is achieved through the following technical solution:

[0006] A method for measuring eye deviation angle based on three-dimensional reconstruction, the method comprising: acquiring video stream information of a test subject's face, and obtaining a sequence of face images of the test subject based on the video stream information;

[0007] The face image sequence is input into the first neural network to obtain the occlusion status of the face image sequence, and the keyframe images of the face image sequence are determined based on the occlusion status.

[0008] The keyframe image is input into the second neural network to obtain a feature point heatmap of the keyframe image, and the feature point heatmap is converted into facial feature point coordinates.

[0009] A 3D face model is constructed. Based on the coordinates of the face feature points and the 3D face model, an objective function is obtained between the coordinates of the face feature points and the projected coordinates of the 3D face model. Based on the objective function, the head pose corresponding to the face feature point coordinates of the keyframe image is obtained.

[0010] One reference eye position image is selected from the keyframe images for the left eye and one for the right eye to initialize the eye position. Based on the head posture, the eye rotation angle of the reference eye position is fixed as a preset angle, and the reference three-dimensional coordinates of the eye in the head coordinate system in the reference eye position image are solved.

[0011] Based on the reference three-dimensional coordinates, the three-dimensional coordinates of the eyeball in the head coordinate system in the image to be measured are fixed, and the rotation angle of the eyeball in the image to be measured is solved. Based on the rotation angle of the eyeball and the rotation degree of the eyeball in the reference eye position, the deviation angle of the eyeball in the image to be measured is obtained.

[0012] In a preferred embodiment of this application, the step of inputting the face image sequence into a first neural network to obtain the occlusion status of the face image sequence may further include:

[0013] The first neural network is trained as follows: an initial first neural network is constructed, a sequence of face images is collected, and the actual occlusion of the face image sequence is labeled; each frame of the face image sequence is sequentially input into the initial first neural network as a training image to obtain the predicted occlusion of the training image; a cross-entropy loss function is constructed based on the predicted occlusion and the corresponding actual occlusion; the initial first neural network is trained under supervision based on the cross-entropy loss function to obtain a trained neural network, which is used as the first neural network.

[0014] In a preferred embodiment of this application, the step of inputting the keyframe image into a second neural network to obtain a feature point heatmap of the keyframe image may further include:

[0015] The second neural network is trained as follows: An initial second neural network is constructed by acquiring a face image from a public dataset and annotating the face image with the coordinates of real feature points, converting these coordinates into a real feature point heatmap; the face image is input into the initial second neural network to obtain a predicted feature point heatmap; an L1 loss function is constructed based on the real and predicted feature point heatmaps; supervised training of the initial second neural network is performed based on the L1 loss function until the L1 loss function converges, resulting in a trained initial second neural network, which serves as the second neural network.

[0016] In a preferred embodiment of this application, the step of fixing the three-dimensional coordinates of the eyeball in the head coordinate system in the image to be measured based on the reference three-dimensional coordinates and solving the eyeball rotation angle in the image to be measured further includes: fixing the three-dimensional coordinates of the eyeball in the head coordinate system in the image to be measured, constructing an objective function between iris feature points and iris model projection points, and solving the eyeball rotation angle in the image to be measured through iterative optimization.

[0017] In a preferred embodiment of this application, the eyeball rotation angle of the fixed reference eye position can be further configured as a preset angle, wherein the preset angle is set to 0 degrees.

[0018] In a preferred embodiment of this application, the step of collecting video stream information of the test subject's face may be further configured to include:

[0019] A first visual target is set at a first preset distance in front of the test subject, and a second visual target is set at a second preset distance. A camera is used to capture video stream information of the test subject looking at the first visual target and the second visual target when the left eye is covered, the right eye is covered, both the left and right eyes are covered, and neither the left nor the right eye is covered.

[0020] In a preferred example of this application, the occlusion of the face image sequence can be further configured as one of the following: left eye occlusion, right eye occlusion, both left and right eyes occlusion, and neither left nor right eye occlusion.

[0021] Secondly, this application provides a device for measuring the eye deviation angle based on three-dimensional reconstruction.

[0022] This application is achieved through the following technical solution:

[0023] A device for measuring eye deviation angle based on three-dimensional reconstruction, the device comprising:

[0024] The image acquisition module is used to acquire video stream information of the test subject's face and obtain a sequence of facial images of the test subject based on the video stream information;

[0025] A keyframe extraction module is used to input the face image sequence into a first neural network to obtain the occlusion status of the face image sequence, and determine the keyframe images of the face image sequence based on the occlusion status.

[0026] A 3D face reconstruction module is used to input the keyframe image into a second neural network to obtain a feature point heatmap of the keyframe image, convert the feature point heatmap into facial feature point coordinates, construct a 3D face model, and based on the facial feature point coordinates and the 3D face model, obtain an objective function between the facial feature point coordinates and the projected coordinates of the 3D face model, and obtain the head pose corresponding to the facial feature point coordinates of the keyframe image based on the objective function.

[0027] The human eye 3D reconstruction module is used to select one frame of reference eye position image for the left eye and one frame of reference eye position image for the right eye from the key frame image to initialize the eye position. Based on the head posture, the eye rotation angle of the reference eye position is fixed as a preset angle, and the reference 3D coordinates of the eye in the head coordinate system in the reference eye position image are solved.

[0028] The skew angle measurement module is used to fix the three-dimensional coordinates of the eyeball in the head coordinate system in the image to be measured based on the reference three-dimensional coordinates, solve the eyeball rotation angle in the image to be measured, and obtain the skew angle of the eyeball in the image to be measured based on the eyeball rotation angle and the eyeball rotation degree of the reference eye position.

[0029] Thirdly, this application provides a computer device.

[0030] This application achieves its goal through the following technical solution:

[0031] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above-described methods for measuring eye deviation angle based on three-dimensional reconstruction.

[0032] Fourthly, this application provides a computer-readable storage medium.

[0033] This application is achieved through the following technical solution:

[0034] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described methods for measuring eye deviation angle based on three-dimensional reconstruction.

[0035] In summary, compared with the prior art, the beneficial effects of the technical solution provided by the embodiments of this application include at least the following: acquiring a sequence of face images of the test subject; obtaining the occlusion of the face image sequence through a first neural network to determine key frame images; obtaining a feature point heatmap through a second neural network and converting it into face feature point coordinates; constructing an objective function between the face feature points and the projection points of the face 3D model to obtain the head pose corresponding to the key frame image; initializing the eye position, fixing the eye rotation angle of the reference eye position as a preset angle based on the head pose, and solving the three-dimensional coordinates of the eye in the reference eye position image under the head coordinate system; fixing the three-dimensional coordinates of the eye in the image to be measured under the head coordinate system, solving the eye rotation angle in the image to be measured, and obtaining the skew angle of the eye. The system requires only manual operation of the image acquisition module to collect appropriate video stream information and transmit the data. Subsequent processing is handled by various modules stored in the cloud platform. Without the limitations of hospital space, users can upload data to the device at any time and place using smart devices such as mobile phones or computers and quickly obtain analysis results, greatly shortening examination time and improving convenience. The system uses a neural network model to analyze and process the acquired images, which is less affected by the subjective factors of the collector and the measured person, thus improving measurement accuracy and consistency. Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating a method for measuring eye deviation angle based on three-dimensional reconstruction, provided as an exemplary embodiment of this application. Detailed Implementation

[0037] This specific embodiment is merely an explanation of this application and is not intended to limit it. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they fall within the scope of the claims of this application.

[0038] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0039] Furthermore, the term "and / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this application, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0040] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor are there any restrictions on quantity or execution order.

[0041] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.

[0042] In one embodiment of this application, a method for measuring eye deviation angle based on three-dimensional reconstruction is provided, such as... Figure 1 As shown, the main steps are described below:

[0043] S10: Collect video stream information of the test subject's face, and obtain the test subject's face image sequence based on the video stream information.

[0044] In some embodiments, a first visual target is set at a first preset distance in front of the test subject, and a second visual target is set at a second preset distance. A camera is used to capture video stream information of the test subject looking at the first visual target and the second visual target when the left eye is covered, the right eye is covered, both the left and right eyes are covered, and both the left and right eyes are covered.

[0045] Specifically, firstly, the subject's head is fixed using a head positioning device, with the chin resting on a chin rest and the forehead close to a headband. The position is adjusted appropriately to ensure the subject's line of sight is horizontal as much as possible. A first visual target is set at a first preset distance in front of the subject's line of sight, which is 3-6 meters from the subject's head; this first visual target is for distance vision. A second visual target is set at a second preset distance in front of the subject's line of sight, which is 0.2-0.5 meters from the subject's head; this second visual target is for near vision. The first and second visual targets are aligned in a straight line, and a high-definition camera is positioned on the same axis as the first and second visual targets. High-definition cameras were used to capture video stream information of test subjects when they were focusing on a first visual target under the following conditions: one blinder covering only the left eye, one blinder covering only the right eye, two blinders covering both the left and right eyes, and neither eye was obscured. Simultaneously, video stream information was captured when the test subjects were focusing on a second visual target under the following conditions: one blinder covering only the left eye, one blinder covering only the right eye, two blinders covering both the left and right eyes, and neither eye was obscured. Facial image sequences of the test subjects were obtained from the captured video stream information.

[0046] S20: Input the face image sequence into the first neural network to obtain the occlusion status of the face image sequence, and determine the key frame images of the face image sequence based on the occlusion status.

[0047] The first neural network model is a deep residual network (ResNet). The obtained sequence of face images of the test subject is input into ResNet to obtain the occlusion status of each frame. The occlusion status is divided into four categories: left eye occlusion, right eye occlusion, complete occlusion of both left and right eyes, and no occlusion of either left or right eye. The occlusion status of the face image sequence is sorted chronologically, and frames where the occlusion status changes are designated as keyframe images. These keyframe images are further divided into reference eye position, manifest strabismus eye position, and maximum strabismus angle eye position according to different transformation modes. The reference eye position can also be represented as the monocular fixation eye position. The changes in the coverage can be divided into 12 transition scenarios: from neither left nor right eye covered to left eye covered; from neither left nor right eye covered to right eye covered; from neither left nor right eye covered to both left and right eye covered; from left eye covered to neither left nor right eye covered; from left eye covered to right eye covered; from left eye covered to both left and right eye covered; from left eye covered to both left and right eye covered; from right eye covered to neither left nor right eye covered; from right eye covered to left eye covered; from right eye covered to both left and right eye covered; from both left and right eye covered to left eye covered; from both left and right eye covered to right eye covered; and from both left and right eye covered to left eye covered.

[0048] Preferably, to improve the output accuracy of the first network model, it is necessary to pre-train the first network model. The pre-training steps are as follows: construct an initial first neural network, collect a sequence of face images and label the actual occlusion of the face image sequence; input each frame of the face image sequence as a training image into the initial first neural network to obtain the predicted occlusion of the training images; construct a cross-entropy loss function based on the predicted occlusion and the corresponding actual occlusion; perform supervised training on the initial first neural network based on the cross-entropy loss function to obtain a trained neural network, which serves as the first neural network.

[0049] Specifically, an initial first neural network is constructed. The obtained face image sequences are manually labeled with actual occlusion conditions to obtain face image sequences with classification labels showing actual occlusion. Each frame of the face image sequence is sequentially input into the untrained initial first neural network. After recognition processing, the predicted occlusion condition of each frame is obtained. A cross-entropy loss function is applied based on the predicted and actual occlusion conditions. The initial first neural network is then trained based on the cross-entropy loss function to obtain a trained neural network, which serves as the final first neural network for keyframe extraction. Training with a large amount of data in the early stages improves the output accuracy of the first neural network, ensuring that the output results achieve predictive accuracy and further improving the accuracy of subsequent eye deviation angle measurement.

[0050] S30: Input the keyframe image into the second neural network to obtain the feature point heatmap of the keyframe image, and convert the feature point heatmap into the coordinates of facial feature points.

[0051] The second neural network model is the U-net model. When a face image sequence is input into the U-net model, N predicted feature point heatmaps of the face image sequence can be obtained, where N is greater than 1.

[0052] Preferably, to improve the output accuracy of the second neural network model, the second neural network model is pre-trained. The pre-training steps are as follows: construct an initial second neural network, obtain face images from a public dataset, and annotate the coordinates of real feature points on the face images. Convert the real feature point coordinates into a real feature point heatmap. Specifically, obtain face images from a public face image database, process the face images, annotate N real feature point coordinates, and annotate the two-dimensional positions of the nose, mouth, and iris. Further, use Gaussian distribution sampling to convert the N real feature point coordinates into N feature point heatmaps. Input the face image into the initial second neural network to obtain the predicted feature point heatmap of the face image. Construct an L1 loss function based on the real feature point heatmap and the predicted feature point heatmap. Perform supervised training on the initial second neural network based on the L1 loss function until the L1 loss function converges to obtain the trained initial second neural network, which serves as the second neural network.

[0053] After training the second neural network to a preset accuracy, the collected face image sequence is input into the trained second neural network to obtain a feature point heatmap of the face image sequence. The feature point heatmap is then converted into feature point coordinates, which are used in the subsequent face 3D reconstruction process to model the face 3D model.

[0054] S40: Construct a 3D face model. Based on the face feature point coordinates and the 3D face model, obtain the objective function between the face feature point coordinates and the projected coordinates of the 3D face model. Based on the objective function, obtain the head pose corresponding to the face feature point coordinates of the keyframe image.

[0055] Specifically, firstly, a 3D face model is constructed using facial 3D modeling software. Combining the facial feature point coordinates obtained in step S30 with the 3D face model, an objective function is constructed between the facial feature points and the corresponding projection points of the 3D face model. The head pose is then solved using an iterative optimization method, which may be the Gauss-Newton method or the conjugate gradient method.

[0056] S50: Select one reference eye position image for each of the left and right eyes from the keyframe images to initialize the eye position. Based on the head posture, fix the eye rotation angle of the reference eye position to a preset angle, and solve the reference three-dimensional coordinates of the eye in the head coordinate system in the reference eye position image.

[0057] Specifically, based on the extracted keyframe images and facial feature point coordinates, one reference eye position image for each of the left and right eyes is selected to initialize the eyeball position. The initialization steps are as follows: fix the rotation degree of the eyeball in the current reference eye position image to 0 degrees, construct the objective function between the iris feature points and the iris model projection points, and solve for the reference three-dimensional coordinates of the eyeball in the head coordinate system using the Forbidden City iterative optimization method.

[0058] S60: Based on the above-mentioned reference three-dimensional coordinates, fix the three-dimensional coordinates of the eyeball in the head coordinate system in the image to be measured, solve the eyeball rotation angle in the image to be measured, and obtain the eyeball deviation angle in the image to be measured based on the eyeball rotation angle and the eyeball rotation degree of the reference eye position.

[0059] Based on the extracted keyframe images and facial feature point coordinates, the eyeball deviation angles for the dominant strabismus position and the maximum strabismus angle position can be calculated. Here, the dominant strabismus position and the maximum strabismus angle position in the keyframe images are taken as the images to be measured. It should be noted that the images to be measured can be any images in the acquired facial image sequence. The specific solution steps are as follows: Based on the reference three-dimensional coordinates of the reference eye position obtained above, the three-dimensional coordinates of the eyeball in the image to be measured under the head coordinate system are fixed. An objective function is constructed between the iris feature points and the iris model projection points. The eyeball rotation angle in the image to be measured is solved using an iterative optimization method. The difference between the eyeball rotation angle corresponding to the image to be measured and the eyeball rotation angle of the reference eye position is used to obtain the eyeball deviation angle in the image to be measured.

[0060] In some embodiments, the method for measuring eye deviation angle based on 3D reconstruction further includes: utilizing video stream data of a large batch of standardized strabismus and normal population facial images collected previously, and constructing a regression model of eyeball 3D rotation and prism degree based on their corresponding prism strabismus degrees. After inputting new facial image samples into this regression model, the corresponding prism strabismus degree can be obtained. Based on the regression model, further assistance can be provided in classifying strabismus into esotropia, exotropia, vertical strabismus, latent strabismus, and normal strabismus. Combined with clinical information, it can assist in providing diagnostic and treatment decision-making opinions, including but not limited to risk assessment of amblyopia, recommendations for wearing glasses, or recommendations for surgery.

[0061] This application also provides a device for measuring eye deviation angle based on three-dimensional reconstruction, the device comprising:

[0062] The image acquisition module is used to acquire video stream information of the test subject's face and obtain a sequence of facial images of the test subject based on the video stream information;

[0063] A keyframe extraction module is used to input the face image sequence into a first neural network to obtain the occlusion status of the face image sequence, and determine the keyframe images of the face image sequence based on the occlusion status.

[0064] The face 3D reconstruction module is used to input the keyframe image into the second neural network to obtain the feature point heatmap of the keyframe image, convert the feature point heatmap into face feature point coordinates, construct a face 3D model, construct an objective function between the face feature point coordinates and the projection coordinates of the face 3D model based on the face feature point coordinates and the face 3D model, and obtain the head pose corresponding to the face feature point coordinates of the keyframe image based on the objective function.

[0065] The human eye 3D reconstruction module is used to select one frame of reference eye position image for the left eye and one frame of reference eye position image for the right eye from the key frame image to initialize the eye position. Based on the head posture, the eye rotation angle of the reference eye position is fixed as a preset angle, and the reference 3D coordinates of the eye in the head coordinate system in the reference eye position image are solved.

[0066] The skew angle measurement module is used to fix the three-dimensional coordinates of the eyeball in the head coordinate system in the image to be measured based on the above-mentioned reference three-dimensional coordinates, solve the eyeball rotation angle in the image to be measured, and obtain the skew angle of the eyeball in the image to be measured based on the eyeball rotation angle and the eyeball rotation degree of the reference eye position.

[0067] The system only requires manual operation of the image acquisition module to collect appropriate video stream information and transmit the data. Subsequent processing is handled by various modules stored in the cloud platform. Without the limitations of hospital space, users can upload data to the device using smart devices such as mobile phones or computers at any time and place to quickly obtain analysis results, greatly shortening the examination time. At the same time, it is less affected by the subjective factors of the collector and the measured, resulting in high measurement accuracy and consistency.

[0068] In one embodiment, a computer device is provided, which may be a server.

[0069] The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media contains an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs on the non-volatile storage media. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements any of the aforementioned methods for measuring eye deviation angles based on three-dimensional reconstruction.

[0070] In one embodiment, a computer-readable storage medium is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-described methods for measuring eye deviation angle based on three-dimensional reconstruction.

[0071] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink, DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0072] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system described in this application can be divided into different functional units or modules to complete all or part of the functions described above.

Claims

1. A method for measuring eye deviation angle based on three-dimensional reconstruction, characterized in that, The method includes: The test subject's face video stream information is collected, and the test subject's face image sequence is obtained based on the video stream information; wherein, a first visual target is set at a first preset distance in front of the test subject, and a second visual target is set at a second preset distance; the test subject is captured by a camera looking at the first visual target and the second visual target respectively when the left eye is covered, the right eye is covered, both the left and right eyes are completely covered, and neither the left nor the right eye is covered. The face image sequence is input into the first neural network to obtain the occlusion status of the face image sequence, and the keyframe images of the face image sequence are determined based on the occlusion status. The keyframe image is input into the second neural network to obtain the feature point heatmap of the keyframe image, and the feature point heatmap is converted into facial feature point coordinates. A 3D face model is constructed. Based on the coordinates of the face feature points and the 3D face model, an objective function is obtained between the coordinates of the face feature points and the projected coordinates of the 3D face model. Based on the objective function, the head pose corresponding to the face feature point coordinates of the keyframe image is obtained. One reference eye position image is selected from the keyframe images for the left eye and one for the right eye to initialize the eye position. Based on the head posture, the eye rotation angle of the reference eye position is fixed as a preset angle, and the reference three-dimensional coordinates of the eye in the head coordinate system in the reference eye position image are solved. Based on the reference three-dimensional coordinates, the three-dimensional coordinates of the eyeball in the head coordinate system in the image to be measured are fixed, and the rotation angle of the eyeball in the image to be measured is solved. Based on the rotation angle of the eyeball and the rotation degree of the eyeball in the reference eye position, the deviation angle of the eyeball in the image to be measured is obtained.

2. The method for measuring eye deviation angle based on three-dimensional reconstruction according to claim 1, characterized in that, Before the step of inputting the face image sequence into the first neural network to obtain the occlusion status of the face image sequence, the method further includes: The first neural network is trained as follows: an initial first neural network is constructed, a sequence of face images is collected, and the actual occlusion of the face image sequence is labeled; each frame of the face image sequence is sequentially input into the initial first neural network as a training image to obtain the predicted occlusion of the training image; a cross-entropy loss function is constructed based on the predicted occlusion and the corresponding actual occlusion; the initial first neural network is trained under supervision based on the cross-entropy loss function to obtain a trained neural network, which is used as the first neural network.

3. The method for measuring eye deviation angle based on three-dimensional reconstruction according to claim 1, characterized in that, Before the step of inputting the keyframe image into the second neural network to obtain the feature point heatmap of the keyframe image, the method further includes: The second neural network is trained as follows: An initial second neural network is constructed by acquiring a face image from a public dataset and annotating the face image with the coordinates of real feature points, converting these coordinates into a real feature point heatmap; the face image is input into the initial second neural network to obtain a predicted feature point heatmap; an L1 loss function is constructed based on the real and predicted feature point heatmaps; supervised training of the initial second neural network is performed based on the L1 loss function until the L1 loss function converges, resulting in a trained initial second neural network, which serves as the second neural network.

4. The method for measuring eye deviation angle based on three-dimensional reconstruction according to claim 1, characterized in that, Based on the aforementioned reference three-dimensional coordinates, the step of fixing the three-dimensional coordinates of the eyeball in the head coordinate system in the image to be measured, and solving the eyeball rotation angle in the image to be measured, further includes: The three-dimensional coordinates of the eyeball in the head coordinate system in the image to be measured are fixed, and an objective function between the iris feature points and the projection points of the iris model is constructed. The eyeball rotation angle in the image to be measured is solved by iterative optimization.

5. The method for measuring eye deviation angle based on three-dimensional reconstruction according to claim 1, characterized in that, The eyeball rotation angle of the fixed reference eye position is a preset angle, wherein the preset angle is set to 0 degrees.

6. The method for measuring eye deviation angle based on three-dimensional reconstruction according to claim 1, characterized in that, The occlusion situation of the face image sequence is one of the following: left eye occlusion, right eye occlusion, both left and right eyes occlusion, and neither left nor right eye occlusion.

7. A device for measuring eye deviation angle based on three-dimensional reconstruction, characterized in that, The measuring device is used to perform the method as described in any one of claims 1 to 6, the device comprising: The image acquisition module is used to acquire video stream information of the test subject's face and obtain a sequence of facial images of the test subject based on the video stream information; wherein, a first visual target is set at a first preset distance in front of the test subject and a second visual target is set at a second preset distance, and a camera is used to acquire video stream information of the test subject looking at the first visual target and the second visual target respectively when the left eye is covered, the right eye is covered, both the left and right eyes are completely covered, and neither the left nor the right eye is covered; A keyframe extraction module is used to input the face image sequence into a first neural network to obtain the occlusion status of the face image sequence, and determine the keyframe images of the face image sequence based on the occlusion status. A 3D face reconstruction module is used to input the keyframe image into a second neural network to obtain a feature point heatmap of the keyframe image, convert the feature point heatmap into facial feature point coordinates, construct a 3D face model, and based on the facial feature point coordinates and the 3D face model, obtain an objective function between the facial feature point coordinates and the projected coordinates of the 3D face model, and obtain the head pose corresponding to the facial feature point coordinates of the keyframe image based on the objective function. The human eye 3D reconstruction module is used to select one frame of reference eye position image for the left eye and one frame of reference eye position image for the right eye from the key frame image to initialize the eye position. Based on the head posture, the eye rotation angle of the reference eye position is fixed as a preset angle, and the reference 3D coordinates of the eye in the head coordinate system in the reference eye position image are solved. The skew angle measurement module is used to fix the three-dimensional coordinates of the eyeball in the head coordinate system in the image to be measured based on the reference three-dimensional coordinates, solve the eyeball rotation angle in the image to be measured, and obtain the skew angle of the eyeball in the image to be measured based on the eyeball rotation angle and the eyeball rotation degree of the reference eye position.

8. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6.

Images