Method and apparatus for measuring eye position deviation angle based on three-dimensional reconstruction

Abstract

To provide a method and an apparatus for measuring a deviation angle of a gaze position based on 3D reconstruction for improving convenience and measurement accuracy of measurement for a deviation angle of a gaze position.SOLUTION: A method collects a face image sequence of a subject, determines an image of a key frame by acquiring a shielding state of the face image sequence in a first neural network, inputs the image of the key frame into a second neural network, acquires a feature point heat map, converts the feeature point heat map into face feature point coordinates, constructs an objective function between the face feature point and a projection point of a face 3D model, acquires a head posture corresponding to the image of the key frame, initializes a position of an eyeball, fixes an eyeball rotation angle at the reference gaze position to a preset angle on the basis of the head posture, obtains the three-dimensional coordinates of the eyeball in the head coordinate system in the reference gaze position image, fixes the three-dimensional coordinates of the eyeball in the head coordinate system in a measured image, and obtains the eyeball rotation angle in the measured image to obtain a deviation angle of the eyeball.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present application relates to the technical field of eye position measurement, and in particular to a method and apparatus for measuring eye position deviation angles based on three-dimensional reconstruction. [Background technology]

[0002] Strabismus is a condition in which the patient's eyes cannot focus on a target simultaneously, either intermittently or continuously. The gaze of one eye that is in a strabismus state cannot be directed toward the other eye, for example, esotropia in the horizontal direction or up-down strabismus in the vertical direction. Strabismus is a common ophthalmic disease that affects about 3% to 8% of the population, of which about 3% of children occur. When the strabismus phenomenon occurs, it can be cured by early diagnosis and intervention, but if treatment is delayed, it becomes difficult to perceive depth with both eyes, which may cause serious psychological development disorders in children. For adults, strabismus can cause many problems, such as affecting reading ability, increasing the risk of falling, and causing social anxiety, which can have a serious impact on physical and mental health and greatly reduce the quality of life, so when the strabismus phenomenon occurs, it is necessary to measure and determine it early. Measurement of the degree of strabismus is mainly a measurement of the angle of eye position deviation, and is very useful for supporting the design of strabismus surgery, diagnosing the disease type and estimating the condition. Currently, the methods for measuring the angle of eye position deviation that are widely used mainly include the prism occlusion test, the Hirschberg method, and the visual field test. However, all of the above measurement methods rely to a certain extent on the high level of cooperation of the patient and are influenced by the subjective factors of the examiner, which may result in large deviations. There may also be cases where the test results obtained by different examiners are inconsistent, and therefore lack objective consistency. Summary of the Invention [Problem to be solved by the invention]

[0003] Meanwhile, the embodiments of the present application provide a method and apparatus for measuring an eye position deviation angle based on three-dimensional reconstruction, which improves the convenience and measurement accuracy of the eye position deviation angle measurement.

[0004] According to a first aspect, the present application provides a method for measuring an eye misalignment angle based on three-dimensional reconstruction. [Means for solving the problem]

[0005] The present application is realized by the following technical solutions:

[0006] A method for measuring an eye position deviation angle based on three-dimensional reconstruction, comprising: collecting video streaming information of a face of the subject, and obtaining a facial image sequence of the subject based on the video streaming information; inputting the facial image sequence into a first neural network to obtain an occlusion state of the facial image sequence, and determining an image of a key frame of the facial image sequence based on the occlusion state; inputting the image of the key frame into a second neural network to obtain a feature point heat map of the image of the key frame, and converting the feature point heat map into facial feature point coordinates; constructing a three-dimensional face model, obtaining an objective function between the facial feature point coordinates and the projection coordinates of the three-dimensional face model based on the facial feature point coordinates and the three-dimensional face model, and obtaining a head pose corresponding to the facial feature point coordinates of the image of the key frame based on the objective function; Selecting one frame of a reference eye position image for each of the left eye and the right eye from the key frame images to initialize the position of the eyeball, fixing the eyeball rotation angle of the reference eye position to a preset angle based on the head posture, and determining the reference three-dimensional coordinates of the eyeball in the reference eye position image in the head coordinate system; The method includes fixing the three-dimensional coordinates of the eyeball in the measured image in the head coordinate system based on the reference three-dimensional coordinates, determining the eyeball rotation angle in the measured image, and obtaining the eyeball deviation angle in the measured image based on the eyeball rotation angle and the eyeball rotation angle of a reference eye position.

[0007] In a preferred embodiment of the present application, the step of inputting the face image sequence into a first neural network and obtaining an occlusion state of the face image sequence includes: The method may further include training a first neural network, and a specific training process may be configured to: construct an initial first neural network, collect a facial image sequence, and label the actual occlusion state of the facial image sequence, input images of each frame in the facial image sequence into the initial first neural network in sequence as training images, obtain a predicted occlusion state of the training image, construct a cross-entropy loss function according to the predicted occlusion state and the corresponding actual occlusion state, supervisedly train the initial first neural network based on the cross-entropy loss function, and obtain a trained neural network as the first neural network.

[0008] In a preferred embodiment of the present application, the step of inputting the image of the key frame into a second neural network and obtaining a feature point heat map of the image of the key frame includes the steps of: The method may further include training a second neural network, and a specific training process may be configured to: construct an initial second neural network, obtain a face image from a public dataset, mark actual feature point coordinates on the face image, convert the actual feature point coordinates into an actual feature point heat map, and input the face image into the initial second neural network to obtain a predicted feature point heat map of the face image, construct an L1 loss function based on the actual feature point heat map and the predicted feature point heat map, and supervise-train the initial second neural network based on the L1 loss function until the L1 loss function converges, and obtain the trained initial second neural network as the second neural network.

[0009] In a preferred example of the present application, fixing the three-dimensional coordinates of the eyeball in the measured image in the head coordinate system and determining the eyeball rotation angle in the measured image based on the reference three-dimensional coordinates as described above may be configured to further include fixing the three-dimensional coordinates of the eyeball in the head coordinate system in the measured image, constructing an objective function between iris feature points and projection points of an iris model, and determining the eyeball rotation angle in the measured image by iterative optimization.

[0010] In a preferred example of the present application, with regard to fixing the eyeball rotation angle of the reference eye position at a preset angle, the preset angle may be configured to be set to 0 degrees.

[0011] In a preferred embodiment of the present application, the collecting of the face video streaming information of the subject further comprises: The method may include providing a first visual target at a first preset distance forward from the subject, providing a second visual target at a second preset distance, and using a camera to collect video streaming information when the subject gazes at the first visual target and the second visual target in each of the following states: with the left eye occluded, with the right eye occluded, with both the left and right eyes occluded, and with neither the left eye nor the right eye occluded.

[0012] In a preferred example of the present application, the occlusion state of the facial image sequence may further be configured to be one of a left eye occluded state, a right eye occluded state, a left eye and right eye both occluded state, and a left eye and right eye neither occluded state.

[0013] According to a second aspect, the present application provides an apparatus for measuring an eye position deviation angle based on three-dimensional reconstruction.

[0014] The present application is realized by the following technical solutions:

[0015] An apparatus for measuring an eye position deviation angle based on three-dimensional reconstruction, comprising: an image acquisition module for acquiring video streaming information of a face of the subject and acquiring a facial image sequence of the subject based on the video streaming information; a keyframe extraction module for inputting the facial image sequence into a first neural network to obtain an occlusion state of the facial image sequence, and determining an image of a keyframe of the facial image sequence based on the occlusion state; a 3D face reconstruction module for inputting the image of the key frame into a second neural network, obtaining a feature point heat map of the image of the key frame, converting the feature point heat map into facial feature point coordinates, and constructing a 3D face model, obtaining an objective function between the facial feature point coordinates and the projection coordinates of the 3D face model based on the facial feature point coordinates and the 3D face model, and obtaining a head pose corresponding to the facial feature point coordinates corresponding to the image of the key frame based on the objective function; a human eye three-dimensional reconstruction module for initializing an eye position by selecting a reference eye position image of one frame for each of the left eye and the right eye from the key frame image, fixing an eye rotation angle of the reference eye position to a preset angle based on the head posture, and calculating a reference three-dimensional coordinate of the eye in the reference eye position image in the head coordinate system; and a deviation angle measurement module for fixing the three-dimensional coordinates of the eyeball in the measured image in the head coordinate system based on the reference three-dimensional coordinates, determining the eyeball rotation angle in the measured image, and obtaining the deviation angle of the eyeball in the measured image based on the eyeball rotation angle and the eyeball rotation angle of a reference eye position.

[0016] According to a third aspect, the present application provides a computing device.

[0017] The present application is realized by the following technical solutions:

[0018] A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer device realizing, when the processor executes the computer program, steps of any one of the above methods for measuring an eye position deviation angle based on three-dimensional reconstruction.

[0019] According to a fourth aspect, the present application provides a computer-readable storage medium.

[0020] The present application is realized by the following technical solutions:

[0021] A computer-readable storage medium has stored thereon a computer program that, when executed by a processor, implements the steps of any one of the above methods for measuring an eye position deviation angle based on three-dimensional reconstruction. Effect of the Invention

[0022] As mentioned above, compared with the prior art, the beneficial effects of the technical solutions according to the embodiments of the present application at least include the following effects: collect a face image sequence of a person to be measured, obtain the occlusion state of the face image sequence in a first neural network to determine a key frame image, obtain a feature point heat map in a second neural network and convert it into face feature point coordinates, construct an objective function between the face feature points and the projection points of a face three-dimensional model, obtain a head posture corresponding to the key frame image, initialize the eye position, fix the eye rotation angle of the reference eye position to a preset angle based on the head posture, obtain the three-dimensional coordinates of the eye in the head coordinate system in the reference eye position image, fix the three-dimensional coordinates of the eye in the head coordinate system in the measured image, obtain the eye rotation angle in the measured image, and initialize the head posture corresponding to the head posture; By obtaining the ball deviation angle, the image collection module only needs to be manually operated to collect appropriate video streaming information and transmit the data. All subsequent operations are calculated and processed by each module stored in the cloud platform, and are not limited by the space in the hospital. Users can upload data using smart devices such as mobile phones and computers at any time and place, and then quickly obtain analysis results, which greatly shortens the examination time and increases convenience. In addition, by using a neural network model to perform analysis and processing on the collected images, the influence of subjective factors between the collector and the person being measured is reduced, and the measurement accuracy and consistency are improved. [Brief description of the drawings]

[0023] [Figure 1] 1 is a flowchart of a method for measuring an eye misalignment angle based on three-dimensional reconstruction according to an exemplary embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] The specific examples are merely for the purpose of illustrating the present application and are not intended to limit the present application. After reading this specification, a person skilled in the art may make modifications to the present examples without making any creative contributions, if necessary, and such modifications are protected by the patent law as long as they are within the scope of the claims of the present application.

[0025] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, a clear and complete description of the technical solutions in the embodiments of the present application will be given below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor belong to the scope of protection of the present application.

[0026] In addition, the term "and / or" in this specification is merely used to describe the relationship between related objects and means that three relationships may exist, for example, "A and / or B" can mean three states: A exists alone, A and B exist simultaneously, and B exists alone. In addition, unless otherwise specified, the symbol " / " in this application generally indicates that the relationship between the related objects before and after it is an "or" relationship.

[0027] It should be understood that the terms "first", "second", and the like in this application are intended to distinguish between elements that are substantially identical in action and function or between similar elements, and that "first", "second", and "nth" do not imply any logical or timing dependency between them, nor do they limit the number or order of execution.

[0028] Hereinafter, the embodiments of the present application will be described in more detail with reference to the specification and drawings.

[0029] In one embodiment of the present application, a method for measuring eye position deviation angle based on 3D reconstruction is provided, as shown in FIG. 1, the main steps are described as follows:

[0030] S10: Collect video streaming information of the subject's face, and obtain a facial image sequence of the subject based on the video streaming information.

[0031] In some embodiments, a first visual target is provided at a first preset distance forward from the subject, and a second visual target is provided at a second preset distance, and a camera collects video streaming information as the subject gazes at the first and second visual targets with the left eye occluded, the right eye occluded, both the left and right eyes occluded, and neither the left eye nor the right eye occluded.

[0032] Specifically, first, using a head positioning device, the lower chin of the person to be measured is placed on a chin rest, the head of the person to be measured is fixed so that the forehead is in close contact with a headband, and the position is appropriately adjusted to make the line of sight of both eyes of the person to be measured as horizontal as possible. A first visual target is provided at a position at a first preset distance forward from the line of sight of the person to be measured, the first preset distance being the distance between the first visual target and the head of the person to be measured, which is 3 to 6 m, and the first visual target is a visual target for looking at a distant object. A second visual target is provided at a second preset distance forward from the line of sight of the person to be measured, the second preset distance being the distance between the second visual target and the head of the person to be measured, which is 0.2 to 0.5 m, and the second visual target is a visual target for looking at a nearby object. The first visual target and the second visual target are on a straight line, and a high-definition camera is provided at a coaxial position of the first visual target and the second visual target. The video streaming information is collected by a high-definition camera when the subject gazes at a first visual target in the following states: a state in which only the left eye is covered by one occluding plate, a state in which only the right eye is covered by one occluding plate, a state in which both the left and right eyes are covered by two occluding plates, and a state in which neither the left eye nor the right eye is covered; and at the same time, the video streaming information is collected when the subject gazes at a second visual target in the following states: a state in which only the left eye is covered by one occluding plate, a state in which only the right eye is covered by one occluding plate, a state in which both the left and right eyes are covered by two occluding plates, and a state in which neither the left eye nor the right eye is covered. The facial image sequence of the subject is obtained from the collected video streaming information. The acquired facial image sequence of the subject is a sequence of facial images acquired during the period from the start of measurement to the end of measurement of the patient. For example, if the total time of this period is 40 seconds and the video frame rate is 30 frames / second, a total of 40*30 facial images are acquired as the facial image sequence.

[0033] S20: Input a face image sequence into a first neural network to obtain an occlusion state of the face image sequence, and determine an image of a key frame of the face image sequence based on the occlusion state.

[0034] The network model of the first neural network is a deep residual network Resnet, and the face image sequence of the subject obtained above can be input into the deep residual network Resnet to obtain the occlusion state of the image of each frame. Specifically, the occlusion state is divided into four types, which are one of the following states: a state in which the left eye is occluded, a state in which the right eye is occluded, a state in which both the left eye and the right eye are occluded, and a state in which neither the left eye nor the right eye is occluded. The occlusion states obtained from the face image sequence are sorted in time series, and the frame in which the occlusion state has changed is taken as the image of the key frame. According to different change situations, the image of the key frame is further divided into images of key frames such as a reference eye position, an obvious strabismus eye position, and a maximum deviation eye position. The reference eye position may be called a 0 degree eye position or a one-eye gaze position, and is defined as the eye position in three-dimensional coordinates of the gaze eye after one eye is occluded. The target to be gazed upon can be located directly in front of the face or in any other direction that induces eye movement. Here, we will explain the reference eye position, apparent strabismus position, and maximum deviation eye position using gaze straight ahead as an example.

[0035] Apparent strabismus: The angle at which the strabismus eye is misaligned when both eyes are fixed on one another (one eye is the gazing eye and the other eye is the strabismus eye).

[0036] Maximum deviation position: By alternately blocking one eye with an opaque blocking plate, alternating monocular gaze can be formed, and binocular fusion can be destroyed, at which time the maximum deviation position can be obtained. For example, during the process of alternate blocking, when the blocking plate placed on the right eye is moved to the left eye, the eye position immediately after the blocking plate of the right eye is removed is the maximum deviation position of the right eye (this is common knowledge in the industry).

[0037] In this technique, the reference eye position, apparent strabismus eye position, and maximum deviation eye position are calculated for each eye pair.

[0038] After gazing at the first target (near viewing), calculate the reference eye position, apparent strabismus eye position, and maximum deviation eye position, then gaze at the second target (far viewing), and repeat the above calculations. The order of looking at the far or near can be determined according to actual clinical needs.

[0039] The situations in which the occlusion state changes include a situation in which neither the left eye nor the right eye is occluded changes to the left eye being occluded, a situation in which neither the left eye nor the right eye is occluded changes to the right eye being occluded, a situation in which neither the left eye nor the right eye is occluded changes to the left and right eyes being both occluded, a situation in which the left eye is occluded changes to neither the left eye nor the right eye being occluded, a situation in which the left eye is occluded changes to the right eye being occluded, a situation in which the left eye is occluded changes to the left and right eye being both occluded, These can be divided into 12 types of change situations: a situation in which the right eye changes from being covered to neither the left nor the right eye being covered, a situation in which the right eye changes to being covered to the left eye, a situation in which the right eye changes to both the left and right eyes being covered, a situation in which both the left and right eyes change to being covered by the left eye, a situation in which both the left and right eyes change to being covered by the right eye, and a situation in which both the left and right eyes change to neither the left nor the right eye being covered.

[0040] Preferably, in order to improve the output accuracy of the network model of the first neural network, the network model of the first neural network needs to be pre-trained. The pre-training steps are as follows: construct an initial first neural network, collect a face image sequence, label the actual occlusion state of the face image sequence, input the image of each frame in the face image sequence into the initial first neural network in sequence as a training image, obtain the predicted occlusion state of the training image, construct a cross-entropy loss function according to the predicted occlusion state and the corresponding actual occlusion state, supervise-train the initial first neural network according to the cross-entropy loss function, and obtain the trained neural network as the first neural network. Here, after processing the image of each frame with a neural network, one predicted label is output, and the predicted occlusion state includes the predicted label, which includes a label indicating either the left eye is occluded, the right eye is occluded, both the left eye and the right eye are occluded, or neither the left eye nor the right eye is occluded, and the number of predicted labels is equal to the number of image sequences.

[0041] Specifically, an initial first neural network is constructed, the acquired face image sequence is manually labeled with actual occlusion states, a face image sequence with classification labels of actual occlusion states is acquired, the images of each frame in the face image sequence are input in sequence to the untrained initial first neural network, a discrimination process is performed to acquire a predicted occlusion state of the images of each frame, a cross-entropy loss function is created according to the predicted occlusion state and the actual occlusion state, the initial first neural network is trained according to the cross-entropy loss function, and the trained neural network is finally acquired as the first neural network used for extracting key frames. The training of a large amount of data in the previous period can improve the output accuracy of the first neural network, ensure that the output result of the first neural network reaches the predicted accuracy (the classification accuracy rate of the occlusion state reaches 90% or more, even 95% or more), and the measurement accuracy of the eye deviation angle can be further improved in the future.

[0042] S30: Input the image of the keyframe into a second neural network, obtain a feature point heat map of the image of the keyframe, and convert the feature point heat map into facial feature point coordinates.

[0043] The network model of the second neural network is a U-net model, and by inputting a face image sequence into the U-net model, N predicted feature point heat maps of the face image sequence can be obtained, where N is greater than 1.

[0044] Preferably, the second neural network model is pre-trained to improve the output accuracy of the second neural network model. The pre-training steps are as follows: construct an initial second neural network, obtain face images from a public dataset, mark actual feature point coordinates on the face images, and convert the actual feature point coordinates into actual feature point heat maps, specifically, obtain face images from a public face image database, process the face images to mark N actual feature point coordinates, mark the two-dimensional positions of the nose, mouth, and iris, and further convert the N actual feature point coordinates into N feature point heat maps through Gaussian distribution sampling, input the face images into the initial second neural network, obtain a predicted feature point heat map of the face images, construct an L1 loss function according to the actual feature point heat map and the predicted feature point heat map, and supervise-train the initial second neural network according to the L1 loss function until the L1 loss function converges, and obtain the trained initial second neural network as the second neural network.

[0045] After training the second neural network to a preset accuracy (standardized pixel error is less than or equal to 5%), the collected facial image sequence can be input into the trained second neural network to obtain a feature point heat map of the facial image sequence, and the feature point heat map can be converted into feature point coordinates and used in the subsequent 3D facial reconstruction process to construct a 3D facial model.

[0046] S40: Construct a 3D face model through 3D facial reconstruction, construct an objective function between the facial feature point coordinates and the projection coordinates of the 3D face model based on the facial feature point coordinates and the 3D facial model, and obtain the head pose corresponding to the facial feature point coordinates of the image of the key frame based on the objective function.

[0047] Specifically, first, a parameterized 3D face model is constructed based on 3D scan data of multiple people with multiple expressions, and the facial feature point coordinates obtained in step S30 are combined to construct an objective function between the facial feature points and the corresponding projection points of the 3D face model, and their two-dimensional Euclidean distances are constrained to be close to each other, and the head pose is obtained by an iterative optimization method, which can be the Gauss-Newton method or the conjugate gradient method.

[0048] S50: One frame of reference eye position images for each of the left eye and right eye is selected from the key frame images to initialize the eye position, the eye rotation angle of the reference eye position is fixed to a preset angle based on the head posture, and reference three-dimensional coordinates in the head coordinate system of the eyeball in the reference eye position image are obtained, the reference three-dimensional coordinates being the three-dimensional coordinates of the eyeball in the head coordinate system in the reference eye position image, and the head coordinate system is the coordinate system of the three-dimensional face model.

[0049] Specifically, based on the extracted key frame images and facial feature point coordinates, one frame of reference eye position images for each of the left and right eyes is selected, and the eye positions are initialized. The initialization steps are as follows: the rotation angle of the eye in the current reference eye position image is fixed to 0 degrees, an objective function between the iris feature points and the projection points of the iris model is constructed, and the reference 3D coordinates of the eye in the head coordinate system are obtained by an iterative optimization method, and the iris model projection is the projection of the iris model in the coordinate system where the facial feature points are located.

[0050] S60: Based on the above-mentioned reference three-dimensional coordinates, the three-dimensional coordinates of the eyeball in the measured image in the head coordinate system are fixed, the eyeball rotation angle in the measured image is obtained, and the eyeball deviation angle in the measured image is obtained based on the eyeball rotation angle and the eyeball rotation angle of the reference eye position.

[0051] According to the extracted key frame image and face feature point coordinates, the eye deviation angle of the obvious strabismus eye position and the maximum deviation eye position can be obtained. Here, the image of the obvious strabismus eye position and the maximum deviation eye position in the key frame image is taken as the measured image, and the measured image may be all images in the acquired face image sequence. The specific acquisition steps are as follows: According to the acquired reference three-dimensional coordinate of the reference eye position, the three-dimensional coordinate of the eye in the head coordinate system in the measured image is fixed, an objective function between the iris feature point and the projection point of the iris model is constructed, and the eye rotation angle in the measured image is obtained by an iterative optimization method, and the difference between the eye rotation angle corresponding to the measured image and the eye rotation angle of the reference eye position is obtained, so that the eye deviation angle in the measured image can be obtained.

[0052] In some embodiments, the method for measuring the eye position deviation angle based on the three-dimensional reconstruction further includes: using a large amount of standardized strabismus and normal person's face images collected in the previous period, constructing a regression model of the three-dimensional rotation of the eyeball and the prism angle according to the corresponding prism strabismus degree, and inputting a new face image sample into the regression model to obtain the corresponding prism strabismus degree. Based on the regression model, the classification judgment of esotropia, exotropia, vertical strabismus, recessive strabismus, normal, etc. can be further supported. In combination with clinical information, the clinical decision-making opinion can be supported, where the clinical decision-making opinion includes, but is not limited to, evaluation of amblyopia risk, suggestion to wear glasses or suggestion to perform surgery, etc.

[0053] The present application further provides an apparatus for measuring an eye position deviation angle based on 3D reconstruction, the apparatus includes: an image acquisition module, a key frame extraction module, a face 3D reconstruction module, a human eye 3D reconstruction module, and a deviation angle measurement module; The image acquisition module is for collecting video streaming information of the face of the person to be measured, and obtaining a facial image sequence of the person to be measured based on the video streaming information, and the image acquisition module may be, for example, a high-definition camera, a three-dimensional camera, a CAMERA or a WEBCAM; The keyframe extraction module is for inputting the face image sequence into a first neural network, obtaining an occlusion state of the face image sequence, and determining a keyframe image of the face image sequence based on the occlusion state, and the keyframe extraction module may be, for example, a graphics processing unit (GPU), a central processing unit (CPU), or a device equipped with a CPU and a GPU, such as a server, a personal computer, or a smartphone; The 3D face reconstruction module is for inputting the image of the key frame into a second neural network to obtain a feature point heat map of the image of the key frame, converting the feature point heat map into facial feature point coordinates, constructing a 3D face model, constructing an objective function between the facial feature point coordinates and the projection coordinates of the 3D face model based on the facial feature point coordinates and the 3D face model, and obtaining a head pose corresponding to the facial feature point coordinates corresponding to the image of the key frame based on the objective function. The 3D face reconstruction module may be, for example, a central processing unit (CPU), or a device equipped with a CPU such as a server, a personal computer, or a smartphone, The human eye three-dimensional reconstruction module is for initializing the position of the eyeball by selecting one frame of a reference eye position image for each of the left eye and the right eye from the key frame image, fixing the eyeball rotation angle of the reference eye position to a preset angle based on the head posture, and obtaining the reference three-dimensional coordinates of the eyeball in the reference eye position image in the head coordinate system. The human eye three-dimensional reconstruction module may be, for example, a central processing unit (CPU), or a device equipped with a CPU such as a server, a personal computer, or a smartphone. The deviation angle measurement module fixes the three-dimensional coordinates of the eyeball in the head coordinate system in the measured image based on the above-mentioned reference three-dimensional coordinates, calculates the eyeball rotation angle in the measured image, and obtains the deviation angle of the eyeball in the measured image based on the eyeball rotation angle and the eyeball rotation angle of the reference eye position.The deviation angle measurement module may be, for example, a central processing unit (CPU), or an apparatus equipped with a CPU such as a server, a personal computer, or a smartphone.

[0054] The image collection module only needs to be manually operated to collect appropriate video streaming information and transmit the data. All subsequent operations are calculated and processed by each module stored in the cloud platform, and are not limited by the space in the hospital. Users can upload data using smart devices such as mobile phones and computers at any time and place, and then quickly obtain analysis results. This greatly shortens the examination time and reduces the influence of subjective factors between the collector and the person being measured, resulting in high measurement accuracy and consistency.

[0055] In one embodiment, a computing device is provided, which may be a server.

[0056] The computer device includes a processor, a memory, a network interface, and a database, which are connected via a system bus. Here, the processor of the computer device is for providing calculation and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium has an operating system, a computer program, and a database. The internal memory provides an environment for the execution of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is for connecting to and communicating with an external terminal via a network. When the computer program is executed by the processor, any one of the above methods for measuring an eye position deviation angle based on three-dimensional reconstruction is realized.

[0057] In one embodiment, a computer-readable storage medium is provided, the computer-readable storage medium including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is executed by the processor to realize any one of the above methods for measuring an eye position deviation angle based on three-dimensional reconstruction.

[0058] Those skilled in the art can understand that all or part of the flow in the method of the above embodiment may be achieved by issuing instructions to relevant hardware by a computer program that can be stored in a non-volatile computer readable storage medium, which, when executed, can include the flow as in the above embodiment of the method. Any reference to memory, storage, database, or other medium used in each embodiment according to the present application may include non-volatile and / or volatile memory. Non-volatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. RAM can take various forms, including but not limited to, static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), Rambus Direct RAM (RDRAM), Direct Memory Bus Dynamic RAM (DRDRAM), and Memory Bus Dynamic RAM (RDRAM).

[0059] As can be clearly understood by those skilled in the art, for convenience and conciseness of explanation, only the division of each of the above-mentioned functional units and modules has been described as an example, but in actual application, the above functions may be implemented by different functional units or modules as necessary, that is, all or part of the functions described above can be realized by dividing the internal structure of the system of the present application into different functional units or modules.

Claims

1. A method for measuring an eye position deviation angle based on three-dimensional reconstruction, comprising: collecting video streaming information of a face of the subject, and obtaining a facial image sequence of the subject based on the video streaming information; inputting the facial image sequence into a first neural network to obtain an occlusion state of the facial image sequence, and determining an image of a key frame of the facial image sequence based on the occlusion state; inputting the image of the key frame into a second neural network to obtain a feature point heat map of the image of the key frame, and converting the feature point heat map into facial feature point coordinates; constructing a three-dimensional face model, obtaining an objective function between the facial feature point coordinates and the projection coordinates of the three-dimensional face model based on the facial feature point coordinates and the three-dimensional face model, and obtaining a head pose corresponding to the facial feature point coordinates of the image of the key frame based on the objective function; selecting one frame of a reference eye position image for each of the left eye and the right eye from the key frame images to initialize the position of the eyeballs, fixing an eyeball rotation angle of the reference eye position to a preset angle based on the head posture, and determining reference three-dimensional coordinates of the eyeballs in the reference eye position image in the head coordinate system; a measurement method for an eye position deviation angle based on three-dimensional reconstruction, comprising: fixing three-dimensional coordinates in a head coordinate system of the eyeball in the measured image based on the reference three-dimensional coordinates; determining an eyeball rotation angle in the measured image; and obtaining an eyeball deviation angle in the measured image based on the eyeball rotation angle and an eyeball rotation angle of a reference eye position.

2. Prior to inputting the facial image sequence into a first neural network and obtaining an occlusion state of the facial image sequence, 2. The method for measuring an eye position deviation angle based on 3D reconstruction according to claim 1, further comprising: training a first neural network, wherein the training process includes: constructing an initial first neural network; collecting a facial image sequence and labeling the facial image sequence with respect to an actual occlusion state; inputting images of each frame in the facial image sequence into the initial first neural network in sequence as training images to obtain a predicted occlusion state of the training image; constructing a cross-entropy loss function according to the predicted occlusion state and the corresponding actual occlusion state; training the initial first neural network according to the cross-entropy loss function; and obtaining a trained neural network as the first neural network.

3. Before inputting the image of the key frame into a second neural network and obtaining a feature point heat map of the image of the key frame, 2. The method of claim 1, further comprising: training a second neural network, wherein the training process includes: constructing an initial second neural network; obtaining a face image from a public dataset, marking actual feature point coordinates on the face image, converting the actual feature point coordinates into an actual feature point heat map, inputting the face image into the initial second neural network to obtain a predicted feature point heat map of the face image, constructing an L1 loss function based on the actual feature point heat map and the predicted feature point heat map, and training the initial second neural network based on the L1 loss function until the L1 loss function converges, thereby obtaining a trained initial second neural network as the second neural network.

4. Fixing the three-dimensional coordinates of the eyeball in the measured image in the head coordinate system based on the reference three-dimensional coordinates, and calculating the eyeball rotation angle in the measured image, 2. The method for measuring an eye position deviation angle based on three-dimensional reconstruction according to claim 1, further comprising: fixing a three-dimensional coordinate of the eyeball in the measured image in a head coordinate system, constructing an objective function between iris feature points and a projection point of an iris model, and obtaining an eyeball rotation angle of the measured image through iterative optimization.

5. 2. The method for measuring an eye position deviation angle based on three-dimensional reconstruction according to claim 1, characterized in that, in fixing the eyeball rotation angle of the reference eye position at a preset angle, the preset angle is set to 0 degrees.

6. The above-mentioned collecting video streaming information of the subject's face includes:

2. The method for measuring an eye position deviation angle based on three-dimensional reconstruction according to claim 1, further comprising: providing a first visual target at a position at a first preset distance forward from a subject; providing a second visual target at a position at a second preset distance forward from the subject; and collecting, with a camera, video streaming information when the subject gazes at the first visual target and the second visual target in each of the following states: a left eye is occluded, a right eye is occluded, both the left eye and the right eye are occluded, and neither the left eye nor the right eye is occluded.

7. The method for measuring an eye position deviation angle based on three-dimensional reconstruction according to claim 6, characterized in that the occlusion state of the face image sequence is one of a left eye occluded state, a right eye occluded state, a left eye and a right eye both occluded state, and a left eye and a right eye both not occluded state.

8. An apparatus for measuring an eye position deviation angle based on three-dimensional reconstruction, comprising: an image acquisition module for acquiring video streaming information of a face of the subject and acquiring a facial image sequence of the subject based on the video streaming information; a keyframe extraction module for inputting the facial image sequence into a first neural network to obtain an occlusion state of the facial image sequence, and determining an image of a keyframe of the facial image sequence based on the occlusion state; a 3D face reconstruction module for inputting the image of the key frame into a second neural network, obtaining a feature point heat map of the image of the key frame, converting the feature point heat map into facial feature point coordinates, and constructing a 3D face model, obtaining an objective function between the facial feature point coordinates and the projection coordinates of the 3D face model based on the facial feature point coordinates and the 3D face model, and obtaining a head pose corresponding to the facial feature point coordinates of the image of the key frame based on the objective function; a human eye three-dimensional reconstruction module for initializing an eye position by selecting one frame of a reference eye position image for each of the left eye and the right eye from the key frame image, fixing an eye rotation angle of the reference eye position to a preset angle based on the head posture, and calculating a reference three-dimensional coordinate of the eye in the reference eye position image in the head coordinate system; and a deviation angle measurement module for fixing three-dimensional coordinates in a head coordinate system of the eyeball in the measured image based on the reference three-dimensional coordinates, determining an eyeball rotation angle in the measured image, and obtaining an eyeball deviation angle in the measured image based on the eyeball rotation angle and an eyeball rotation angle of a reference eye position.

9. A computing device comprising a memory, a processor and a computer program stored in the memory, the processor executing the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored therein, the computer program implementing the steps of the method according to any one of claims 1 to 7 when executed by a processor.

Images