A device and method for measuring strabismus based on pupil localization

Abstract

A strabismus measuring device based on pupil positioning belongs to the technical field of strabismus measurement. The optical system includes a near vision nine-point sight mark optical path and a far vision optical path; the electromechanical control system includes an application module for controlling the lighting selection of each sight mark according to the corneal reflection method and selecting the cover method measurement, a sight mark display module for realizing the corneal reflection method and the cover method, and a human eye cover module for covering the left eye or the right eye; the image processing system is used to perform image segmentation using a fully convolutional neural network through the eye image collected by the infrared camera to realize pupil positioning, and then a three-dimensional pupil model is fitted through a series of time-series eye images to realize the detection of the pupil center position, and realize the calculation of the measurement results of the reflection method and the cover method. A strabismus measurement method based on pupil positioning is provided. The invention has a high degree of automation, convenient operation, and accurate and objective measurement results.

Description

Technical Field

[0001] The invention belongs to the technical field of strabismus measurement and relates to a strabismus measurement device and method. Background Art

[0002] Strabismus is a common eye disease characterized by one eye deviating from the normal direction of sight when gazing at an object. In China, the incidence of strabismus is about 3% to 5%, while the incidence of the disease is about 2% worldwide. It is worth noting that about 65% of strabismus patients are diagnosed with this disease in childhood.

[0003] At present, the main methods for checking the eye position of strabismus are as follows: (1) Cover-uncover method: detect manifest strabismus and distinguish manifest strabismus from latent strabismus / orthostatic eyes; (2) Alternating cover method: The purpose of the alternating cover test is to separate the binocular fusion function to detect all strabismus including manifest strabismus and latent strabismus, and to distinguish orthostatic eyes from manifest strabismus or latent strabismus; (3) Prism alternating cover test: The prism alternating cover test can measure the total strabismus of manifest strabismus and latent strabismus, and is the most accurate method for measuring strabismus; (4) Corneal light reflection method (Hirschberg method): The examiner determines the type and degree of strabismus by observing the position of the patient's corneal light reflection and its relationship with the pupil. A 1 mm deviation of the corneal light reflection from the center of the pupil is approximately equivalent to 7°, and is generally used to check the patient's strabismus for near vision. In addition, there is also a method for diagnosing strabismus: using accommodative sight marks to check the movement of the spouse muscles of both eyes when the subject is looking at nine directions (also known as: diagnostic eye position); the diagnostic strabismus examination includes quantitative examination of strabismus when the eyeball rotates from the original position, horizontally to the right, horizontally to the left, to the upper left, to the lower left, to the upper right, to the lower right, vertically upward, and vertically downward. It can analyze and determine the affected muscles of non-common strabismus and is used for diagnosis and surgical planning.

[0004] The examination methods vary according to the degree and type of strabismus and the age of the patient. At present, the commonly used methods for measuring strabismus in clinical practice include cover measurement and corneal reflection measurement. These measurement methods mainly rely on the experience of ophthalmologists, the operation process is relatively complicated and the results are somewhat subjective. The diagnostic accuracy of strabismus is affected by the professional level of ophthalmologists, especially when measuring the strabismus angle of young children with poor cooperation ability. This situation leads to the lack of sufficient accuracy in technical design, which increases the risk of secondary surgery and may cause greater harm to patients.

[0005] Chinese patent publication number CN117617889A discloses a strabismus degree detection system based on cover test, including a headrest, a calibration point, a display, a camera, a light source, a voice announcer and a control device, which defines the eye state, including an initial state, a ready state, a blinking state, an occlusion state, a transient state and a stable state, and automatically saves image information in the transient state and the stable state to calculate the strabismus degree, and uses a voice announcer to interact with the user, voice prompts the subject to perform a cover test, and can automatically detect the occlusion state of the subject's eyes, and obtain the strabismus detection result after the cover test. However, the detection system has the following disadvantages: (1) The detection system is not an integrated instrument and needs to be reinstalled and calibrated by professionals according to changes in the measurement environment, which is inconvenient to use; (2) To complete the cover method measurement, the subject needs to cover his / her eyes according to the reminder, which increases the requirements for the subject's cooperation; (3) When detecting the subject's eye state, only the eye information in the uncovered state can be captured, and the eye information in the covered state cannot be captured and analyzed, resulting in low detection accuracy.

[0006] Studies have pointed out that existing inspection methods based on photography or video (such as the commonly used corneal reflection method) generally have errors. The reason is that the measurement reference point based on the photography or video inspection method - the projection position of the "pupil center" on the corneal surface moves with the movement of the detection point (that is, the corneal reflection point), that is, the three-dimensional spatial distance is measured in a two-dimensional plane without taking into account the arc angle of the eye strabismus, so the accurate pupil center position cannot be obtained.

[0007] As a key optical device, prisms are widely used in the measurement and treatment of strabismus. Traditionally, medical prisms are usually made of glass or plastic materials with a fixed refractive index. Their deflection ability is expressed in prismatic degree, and the strabismus of strabismus patients can also be quantified by prismatic degree. In recent years, with the advancement of micro-manufacturing technology, more and more researchers have begun to explore the use of microstructured pressed Fresnel prisms to replace traditional prisms. This new type of prism is usually called a microprism. Microprisms can achieve the same refractive effect as solid prisms by arranging prism structures in an array on the surface of the lens, while significantly reducing the volume and weight of the prism lens, greatly improving the wearing comfort. Summary of the invention

[0008] In order to overcome the shortcomings of the existing technology, such as low automation, inconvenient operation and low accuracy of measurement results, the present invention provides a strabismus measurement device and method based on pupil positioning, which has a high degree of automation, convenient operation and accurate and objective measurement results. A three-dimensional pupil model is reconstructed through a two-dimensional pupil image, thereby solving the error problem that the current strabismus inspection method based on photography or videography cannot accurately locate the pupil center.

[0009] The technical solution adopted by the present invention to solve its technical problem is:

[0010] A strabismus measuring device based on pupil positioning comprises an optical system, an electromechanical control system and an image processing system, wherein the optical system comprises a near vision nine-point sight mark optical path and a far vision optical path; the electromechanical control system comprises an application module for controlling the lighting selection of each sight mark according to the corneal reflection method and selecting the measurement of the covering method, a sight mark display module for realizing the corneal reflection method and the covering method, and a human eye covering module for covering control of the left eye or the right eye; the image processing system is used to perform image segmentation using a fully convolutional neural network on an eye image collected by an infrared camera to realize pupil positioning, and then to detect the pupil center position by fitting a three-dimensional pupil model through a series of time-series eye images, and to calculate the measurement results of the reflection method and the covering method.

[0011] Further, the nine-point sight mark optical path for near vision includes a nine-point sight mark optical path for the left eye and a nine-point sight mark optical path for the right eye, wherein the nine-point sight mark optical path for the left eye includes nine sight marks in different directions connected in sequence, a first semi-transparent and semi-reflective mirror, a first liquid crystal light valve on the left side, a polarizing film on the left side, a second semi-transparent and semi-reflective mirror on the left side, and a second liquid crystal light valve on the left side, and the left infrared light source is aimed at the left eye position; the nine-point sight mark optical path for near vision of the right eye includes nine sight marks in different directions connected in sequence, a first semi-transparent and semi-reflective mirror, a first liquid crystal light valve on the right side, a polarizing film on the right side, a second semi-transparent and semi-reflective mirror on the right side, and a second liquid crystal light valve on the right side, and the right infrared light source is aimed at the right eye position;

[0012] The telescopic optical path includes a telescopic left-eye optical path and a telescopic right-eye optical path, wherein the telescopic left-eye optical path includes a telescopic sight mark, an aperture stop, a telescopic lens, a telescopic semi-transparent mirror, a first left-side liquid crystal light valve, a left-side polarizing film, a left-side second semi-transparent mirror, and a left-side second liquid crystal light valve connected in sequence; the telescopic right-eye optical path includes a telescopic sight mark, an aperture stop, a telescopic lens, a telescopic semi-transparent mirror, a telescopic reflective mirror, a first right-side liquid crystal light valve, a right-side polarizing film, a right-side second semi-transparent mirror, and a right-side second liquid crystal light valve connected in sequence;

[0013] The left infrared camera is aimed at the reflection direction of the left second semi-transparent and semi-reflective mirror to capture the left eye image; the right infrared light source is aimed at the right eye position, and the right infrared camera is aimed at the reflection direction of the right second semi-transparent and semi-reflective mirror to capture the right eye image.

[0014] Furthermore, the strabismus measuring device also includes a left microprism and a right microprism, the left microprism is located between the second semi-transparent and semi-reflective mirror on the left and the second liquid crystal light valve on the left, and the right microprism is located between the second semi-transparent and semi-reflective mirror on the right and the second liquid crystal light valve on the right, and the electromechanical control system also includes a microprism displacement module for controlling the displacement of the left microprism and the right microprism.

[0015] The nine sight marks in different directions include nine red LEDs in the middle, upper, lower, left, right, upper left, upper right, lower left and lower right directions, and the distance between two adjacent sight marks is equal.

[0016] In the application module, the selection of covering method includes alternating covering, covering and uncovering, prism alternating covering method and prism covering and uncovering method; in the sight mark display module, the corneal reflection method adjusts the brightness of the sight mark to high brightness by adjusting the duty cycle of the PWM signal, and lights up nine sight marks in turn for 2 seconds, in the order of upper left, upper, upper right, left, middle, right, lower left, lower, and lower right; the covering method adjusts the brightness of the sight mark to low brightness by adjusting the duty cycle of the PWM signal, and then selects one of the sight mark points through the application module to complete the covering, so as to realize the strabismus measurement of the eye position.

[0017] The microprism displacement module includes a longitudinal displacement motor, a lateral displacement motor and an opposing displacement motor, so as to realize the displacement control of the left microprism and the right microprism. The left microprism and the right microprism are installed in corresponding prism sheet slots. The spacing of the prism sheet slots is first controlled by the opposing displacement motor to match the pupil distance of the subject. After selecting a certain position sight mark for prism cover test, the longitudinal displacement motor and the lateral displacement motor are controlled to achieve the field of view requirement of the human eye looking at the nine-point sight mark.

[0018] Preferably, the left eye or the right eye is covered by controlling the power on and off of the first liquid crystal light valve on the left, the first liquid crystal light valve on the right, the second liquid crystal light valve on the left, and the second liquid crystal light valve on the right;

[0019] The left infrared light source and the right infrared light source respectively include four infrared LED lamp beads and a constant current module. The infrared LED lamp beads are placed around the eyes. The eight infrared LED lamp beads are divided into two groups, and every four are connected in parallel in the constant current module to power the infrared LED lamp beads.

[0020] The left infrared camera and the right infrared camera are respectively installed on the left and right sides of the instrument, and the eye images are captured and collected through the second semi-transparent and semi-reflective mirror on the left and the second semi-transparent and semi-reflective mirror on the right.

[0021] The image processing system includes a pupil recognition module and a light reflection recognition module. The pupil recognition module uses a deep learning method. The deep learning method uses a fully convolutional neural network to segment the entire pupil area. The fully convolutional neural network is trained on manually annotated pupil images to achieve pupil positioning. A three-dimensional pupil model is then fitted through a series of time-series eye images that complete pupil positioning, and finally the estimation of the human eye's line of sight is completed. The pupil image is extracted by collecting eye images taken by an infrared camera under infrared light supplementary lighting, and the pupil area is manually labeled to complete the training of the neural network. The light reflection recognition module includes digital image processing and a deep learning neural network. The digital image processing uses image binarization processing to identify and mark the light reflection points on the eyeball, and then compares the pupil position information identified by the neural network to calculate the strabismus degree based on the positional relationship between the two.

[0022] The implementation process of the image processing system is as follows:

[0023] (1) Consider the center of the infrared camera as the observation point o. Construct a cone on the image plane that passes through the observation point o and the edge of the pupil ellipse. Then find the circular intersection of the edge of the cone projected outward, which is the pupil circle. At any given distance, two circular intersections, i.e., two pupil circles, can be drawn to obtain the pupil position, gaze vector, and pupil radius: (p + ,n + ,r),(p - ,n - ,r), where p + represents the pupil position of the positive pupil circle, n + represents the gaze vector towards the pupil circle, p - Indicates the pupil position of the negative pupil circle, n - represents the gaze vector of the negative pupil circle, r represents the pupil radius of the pupil circle;

[0024] (2) For each eye image, consider the pupil circle ( , , ), where i is the index of the image, represents the pupil position of the pupil circle of the i-th image, represents the gaze vector of the pupil circle of the i-th image, represents the pupil radius of the pupil circle of the i-th image. The cone of each pupil ellipse projection can draw two pupil circles, and it is unknown whether the two circles ( + , + ,r)or( - , - , ) is correct, but the projections of the normal vectors of the two pupil circles on the plane are parallel: + ∝ - ,in, + represents the pupil position of the positive pupil circle of the i-th image, + represents the gaze vector of the positive pupil circle of the i-th image, - represents the pupil position of the negative pupil circle of the i-th image, - represents the gaze vector of the positive pupil circle of the i-th image, + represents the projection of the normal vector of the pupil circle of the i-th image onto the plane, - represents the projection of the normal vector of the negative pupil circle of the i-th image on the plane;

[0025] (3) Similarly, the pupil positions of the two pupil circles projected on the plane are + and - The straight line between ± ,in, + represents the pupil position of the projection of the positive pupil circle of the i-th image on the plane, - represents the pupil position of the projection of the negative pupil circle of the i-th image on the plane, ± represents the projection of the normal vectors of the positive pupil circle and the negative pupil circle of the i-th image on the plane, which means,

[0026] + = - +s + = - +t + ;

[0027] Among them, s represents the negative proportional coefficient and t represents the positive proportional coefficient;

[0028] (4) Find the center of the eyeball projection surface by calculating the intersection of straight lines that correspond to the projected gaze direction: each straight line passes through the projected pupil center and is parallel to the projected pupil normal. Find the set of projection lines Li The intersection of

[0029] L i ={(x, y)= +s };

[0030] Among them, x and y represent the coordinates in the projection space, represents the pupil position of the projection of the pupil circle of the i-th image on the plane, represents the normal vector of the projection of the pupil circle of the i-th image on the plane;

[0031] (5) By calculating and using the least squares method, find the point closest to each line.

[0032] = - ;

[0033] in, represents the center of the eyeball projection plane, represents the identity matrix, express The transpose of

[0034] Then reverse scale the two-dimensional coordinates to three-dimensional space, that is, through the center of the eyeball projection surface Find the center c of the 3D sphere;

[0035] (6) The center of the eyeball projection surface obtained by solving and the 3D sphere center c, find the normal vector of each pupil circle They all point in the opposite direction of the 3D sphere center c, and the normal vector of the projection of each pupil circle on the plane All point to the center of the eyeball projection surface The opposite direction of the projection on the plane is obtained Point to the center of the eyeball projection surface The pupil circle in the opposite direction of , , );

[0036] (7) Since the size of the pupil circle cannot be determined, it cannot be used directly , consider introducing candidate pupil circle centers i and the candidate pupil circle radius i , candidate pupil circle center i Located through the observation point o and the center of the pupil circle , and consider the circle ( i , , i ) is tangent to the 3D sphere with center c, passing from the center c to i The gaze line observation point o and the pupil circle center The candidate pupil circle center is found by intersecting the lines of i ;

[0037] (8) Since straight lines in 3D will never intersect, find the least squares intersection again, and then calculate the sphere radius R by finding the average distance from the center of the sphere to the center of each pupil,

[0038] R=mean({R i = i -c i});

[0039] Where R represents the radius of the sphere, R i Represents the distance from the center of the i-th pupil circle i The distance to the center of the sphere c, i The center of the pupil circle of the i-th image, c represents the center of the 3D sphere;

[0040] (9) Due to the ambiguity of distance and size, the pupil circle cannot be guaranteed. , , ) is on the sphere, so we introduce a new circle ( , , ) ,in,

[0041] -c 2 = R 2 ;

[0042] = s ;

[0043] =c+R ;

[0044] = ;

[0045] in, After angle scaling , represents the z coordinate of the i-th pupil circle in three-dimensional space, represents the z coordinate of the i-th pupil circle in three-dimensional space after scaling. , find a scaling factor s such that s Located on the surface of the sphere (c, R), it can be calculated as the intersection of the line and the sphere, and two scaling factor solutions s1 and s2 are obtained. The smallest scaling factor is taken and the calculation is obtained and ;

[0046] (10) Confidence factor for each image frame that will be considered during 3D model fitting and gaze estimation Included in; the center of the eyeball projection surface The confidence-weighted fitting formula of and sphere radius R is expressed as:

[0047] - ;

[0048] R= ;

[0049] in, represents the center of the eyeball projection plane, represents the confidence factor of the i-th image, represents the center of the pupil circle where the i-th image is tangent to the sphere;

[0050] The pupil deviation angle is then calculated through the angle between the three-dimensional eye gaze vectors, and then the strabismus and prism degree of the subject are calculated.

[0051] A method for measuring strabismus based on pupil positioning, the measuring method comprising a corneal light reflection method and a covering method, the process of the corneal light reflection method is as follows:

[0052] Step 1.1, record the subject's information, fix the subject's head, place both eyes in front of the second liquid crystal light valve, use an infrared light source to fill the eyes, select the corneal reflection method in the application module to start measurement, and the camera starts to collect the subject's eye image information;

[0053] Step 1.2, adjust the red LED sight mark to high brightness, and light up the nine red LED sight marks in sequence for 2 seconds, in the order of upper left, upper, upper right, left, middle, right, lower left, lower, and lower right;

[0054] Step 1.3, the image processing system is used to process the eye images of the subject gazing at the nine sight marks, locate the distance between the reflected light point and the pupil center, and calculate the subject's strabismus in nine directions based on the principle that a deviation of 1 mm is equivalent to a strabismus of 7 degrees, and the measurement results are saved and displayed;

[0055] The process of the covering method is as follows:

[0056] Step 2.1, record the subject's information, fix the subject's head, place both eyes in front of the second liquid crystal light valve, use infrared light source to fill the eyes, select a covering method in the application and select a certain direction sight mark to start measurement, and the camera starts to collect the subject's eye image information;

[0057] Step 2.2, executing the masking program corresponding to the masking method through the human eye masking module, breaking the visual fusion of the human eye by controlling the masking of the left and right eyes, pausing image acquisition after the masking program is completed, and calculating the pupil center position information through the image processing system;

[0058] Step 2.3, by calculating the deviation angle of the pupil center position, converting the strabismus and prism degree, the measurement results are saved and displayed.

[0059] Furthermore, in step 2.2, when the masking program selects the prism alternating masking method or the prism masking and unmasking method, the microprism displacement module is called to displace the microprism to a position that matches the pupil distance of the subject, and after matching the position with the selected sight mark, the masking program is completed through the human eye masking module.

[0060] The technical concept of the present invention is: adopting an automated control system to reduce dependence on physician experience. The device can automatically guide the subject to complete the cover test, monitor the eye status in real time, automatically identify and record the occlusion status, and ensure the integrity of the data; optical-mechanical-electrical-computing integrated design: combining optical, mechanical, electronic and computer technologies, designing an integrated device to achieve automated measurement of strabismus. By integrating sensors, cameras and processing units, the operation process is simplified, the volume of the device is reduced, and portability is improved; three-dimensional reconstruction technology: reconstructing a three-dimensional pupil model through a two-dimensional pupil image overcomes the error of measuring the three-dimensional space distance in a two-dimensional planar manner in traditional methods. This technology can more accurately capture the true position and movement angle of the eyeball, providing a more reliable data basis for the calculation of strabismus. Comprehensive information collection: designing multiple optical path systems and cameras to capture the eye information of the subject in different states, including occlusion and unocclusion. By comprehensively analyzing this information, the accuracy and reliability of strabismus measurement are improved.

[0061] The beneficial effects of the present invention are mainly manifested in: high degree of automation, convenient operation, and accurate and objective measurement results. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a schematic diagram of the optical path of the near nine-point sight mark in the optical system of the present invention;

[0063] Figure 2It is a schematic diagram of the optical path of the nine-point sight mark of the right eye for near vision and the optical path of the right eye for far vision in the optical system of the present invention;

[0064] Figure 3 It is a schematic diagram of the telescopic beam splitting optical path in the optical system of the present invention;

[0065] Figure 4 is a schematic diagram of a micro-prism displacement module in an electromechanical control system of the present invention;

[0066] Figure 5 is a schematic diagram of a micro-prism structure of the present invention;

[0067] Figure 6 It is an organizational structure block diagram of the electromechanical control system of the present invention.

[0068] The reference numerals are as follows: left-side myopia mark 1, middle longitudinal axis myopia mark 2, right-side myopia mark 3, first semi-transparent and semi-reflective mirror 4, left-side first liquid crystal light valve 5, left-side polarizing film 6, left-side second semi-transparent and semi-reflective mirror 7, left-side infrared camera 8, left microprism 9, left-side second liquid crystal light valve 10, left-side infrared light source 11, left eye of the subject 12, right-side first liquid crystal light valve 13, right-side polarizing film 14, right-side second semi-transparent and semi-reflective mirror 15, right-side second liquid crystal light valve 16, left-side second liquid crystal light valve 17, left-side second liquid crystal light valve 18, left microprism 19, left microprism 20, left microprism 21, left microprism 22, left microprism 23, left microprism 24, left microprism 25, left microprism 26, left microprism 27, left microprism 28, left microprism 29, left microprism 30, left microprism 31, left microprism 32, left microprism 33, right microprism 34, right microprism 35, left microprism 36, left microprism 37, left microprism 38, left microprism 39, left microprism 40, left microprism 41, left microprism 42, left microprism 43, left microprism 44, right microprism 45, left microprism 46, left microprism 47, left microprism 48, left microprism 49, left microprism 50, left microprism 51, left microprism 52, left microprism 53, left microprism 54, left microprism 55, left micro Crystal light valve 16, right infrared light source 17, right eye of the subject 18, right infrared camera 19, upper near vision mark 20, middle horizontal axis near vision mark 21, lower near vision mark 22, far vision mark 23, aperture diaphragm 24, far vision beam splitting path 25, far vision lens 26, far vision semi-transparent and semi-reflective mirror 27, far vision reflecting mirror 28, prism sheet slot 29, lateral displacement motor 30, opposite displacement motor 31, longitudinal displacement motor 32, right microprism 33. DETAILED DESCRIPTION

[0069] The present invention will be further described below in conjunction with the accompanying drawings.

[0070] Reference Figure 1 to Figure 6 , a strabismus measuring device based on pupil positioning, comprising an optical system, an electromechanical control system and an image processing system, wherein the optical system comprises a near vision nine-point sight mark optical path and a far vision optical path; the electromechanical control system comprises an application module for controlling the lighting selection of each sight mark according to the corneal reflection method and selecting the cover method measurement, a sight mark display module for realizing the corneal reflection method and the cover method, and a human eye covering module for covering the left eye or the right eye; the image processing system is used to perform image segmentation using a fully convolutional neural network on the eye image collected by an infrared camera to realize pupil positioning, and then detect the pupil center position by fitting a three-dimensional pupil model through a series of time-series eye images, and realize the calculation of the measurement results of the reflection method and the cover method.

[0071] Further, the near vision nine-point sight mark optical path includes a near vision left eye nine-point sight mark optical path and a near vision right eye nine-point sight mark optical path, wherein the near vision left eye nine-point sight mark optical path includes nine sight marks in different directions connected in sequence, a first semi-transparent and semi-reflective mirror 4, a first left liquid crystal light valve 5, a left polarizing film 6, a left second semi-transparent and semi-reflective mirror 7 and a left second liquid crystal light valve 10; the near vision right eye nine-point sight mark optical path includes nine sight marks in different directions connected in sequence, a first semi-transparent and semi-reflective mirror 4, a right first liquid crystal light valve 13, a right polarizing film 14, a right second semi-transparent and semi-reflective mirror 15 and a right second liquid crystal light valve 16;

[0072] The telescopic optical path includes a telescopic left-eye optical path and a telescopic right-eye optical path, wherein the telescopic left-eye optical path includes a telescopic sight mark 23, an aperture diaphragm 24, a telescopic lens 26, a telescopic semi-transparent mirror 27, a first left-side liquid crystal light valve 5, a left-side polarizing film 6, a left-side second semi-transparent mirror 7 and a left-side second liquid crystal light valve 10 connected in sequence; the telescopic right-eye optical path includes a telescopic sight mark 23, an aperture diaphragm 24, a telescopic lens 26, a telescopic semi-transparent mirror 27, a telescopic reflector 28, a first right-side liquid crystal light valve 13, a right-side polarizing film 14, a right-side second semi-transparent mirror 15 and a right-side second liquid crystal light valve 16 connected in sequence.

[0073] The left infrared light source 11 is aimed at the left eye position, and the left infrared camera 8 is aimed at the reflection direction of the left second semi-transparent and semi-reflective mirror 7 to capture the left eye image; the right infrared light source 17 is aimed at the right eye position, and the right infrared camera 19 is aimed at the reflection direction of the right second semi-transparent and semi-reflective mirror 15 to capture the right eye image.

[0074] Preferably, the nine sight marks in different directions include nine red light LEDs in nine directions of middle, top, bottom, left, right, upper left, upper right, lower left and lower right, and the distance between two adjacent sight marks is equal.

[0075] The left eye or the right eye is covered by controlling the power on and off of the first liquid crystal light valve 5 on the left, the first liquid crystal light valve 13 on the right, the second liquid crystal light valve 10 on the left, and the second liquid crystal light valve 16 on the right. In this scheme, the second liquid crystal light valve in front of the eye only retains a single-side polarization film. When the voltage on both sides is zero, the polarization direction is parallel to the polarization film and thus does not block light. In the powered-on state, the polarization direction is perpendicular to the polarization film to block light, and at the same time, it does not affect the infrared camera's acquisition of eye images; the light-transmitting and light-blocking state of the first liquid crystal light valve is synchronized with the second liquid crystal light valve to achieve enhanced light-blocking effect.

[0076] The left infrared light source 11 and the right infrared light source 17 respectively include four infrared LED lamp beads and a constant current module. The infrared LED lamp beads all have a power of 0.2W and a wavelength of 940nm. They are placed around the eyes. The eight infrared LED lamp beads are divided into two groups, and every four are connected in parallel in the constant current module to power the infrared LED lamp beads.

[0077] The left infrared camera 8 and the right infrared camera 19 are respectively installed on the left and right sides of the instrument, and the eye images are captured and collected through the left second semi-transparent half-reflective mirror 7 and the right second semi-transparent half-reflective mirror 15.

[0078] Furthermore, the strabismus measuring device also includes a left microprism 9 and a right microprism 33. The left microprism 9 is located between the second semi-transparent and semi-reflective mirror 7 on the left and the second liquid crystal light valve 10 on the left. The right microprism 33 is located between the second semi-transparent and semi-reflective mirror 15 on the right and the second liquid crystal light valve 16 on the right. The electromechanical control system also includes a microprism displacement module for controlling the displacement of the left microprism 9 and the right microprism 33.

[0079] In the application module, the selection of masking methods includes alternating masking, masking and demasking, prism alternating masking method and prism masking and demasking method.

[0080] The application module is developed on the Linux operating system and uses an SQLite database to save the measurement results. The SQLite database runs inside the application and does not require the configuration or operation of a separate server program.

[0081] In the sight mark display module, the corneal reflection method adjusts the brightness of the sight mark to high brightness by adjusting the duty cycle of the PWM signal, and lights up nine infrared light sources in sequence for 2s, in the order of upper left, upper, upper right, left, middle, right, lower left, lower, and lower right; the covering method adjusts the brightness of the sight mark to low brightness by adjusting the duty cycle of the PWM signal, and then selects one of the sight mark points through the application module to complete the covering method, so as to realize the strabismus measurement of the eye position.

[0082] The microprism displacement module includes a longitudinal displacement motor 32, a lateral displacement motor 30 and an opposite displacement motor 31, so as to realize the displacement control of the left microprism 9 and the right microprism 33. The microprism 9 and the right microprism 33 are respectively installed in the corresponding prism sheet slots. The opposite displacement motor 31 first controls the spacing of the prism sheet slots 29 to match the pupil distance of the subject. After selecting a certain position sight mark for prism cover test, the longitudinal displacement motor 32 and the lateral displacement motor 30 are controlled to achieve the field of view requirement of the human eye looking at the nine-point sight mark;

[0083] The image processing system includes a pupil recognition module and a light reflection recognition module. The pupil recognition module uses a deep learning method. The deep learning method uses a fully convolutional neural network to segment the entire pupil area. The fully convolutional neural network is trained on manually annotated pupil images to achieve pupil positioning. A three-dimensional pupil model is then fitted through a series of time-series eye images that complete pupil positioning, and finally the estimation of the human eye's line of sight is completed. The pupil image is extracted by collecting eye images taken by an infrared camera under infrared light supplementary lighting, and the pupil area is manually labeled to complete the training of the neural network. The light reflection recognition module includes digital image processing and a deep learning neural network. The digital image processing uses image binarization processing to identify and mark the light reflection points on the eyeball, and then compares the pupil position information identified by the neural network to calculate the strabismus degree based on the positional relationship between the two.

[0084] The implementation process of the image processing system is as follows:

[0085] (1) Consider the center of the infrared camera as the observation point o. Construct a cone on the image plane that passes through the observation point o and the edge of the pupil ellipse. Then find the circular intersection of the edge of the cone projected outward, which is the pupil circle. At any given distance, two circular intersections, i.e., two pupil circles, can be drawn to obtain the pupil position, gaze vector, and pupil radius: (p + ,n + ,r),(p - ,n - ,r), where p + represents the pupil position of the positive pupil circle, n + represents the gaze vector towards the pupil circle, p - Indicates the pupil position of the negative pupil circle, n - represents the gaze vector of the negative pupil circle, r represents the pupil radius of the pupil circle;

[0086] (2) For each eye image, consider the pupil circle ( , , ), where i is the index of the image, represents the pupil position of the pupil circle of the i-th image, represents the gaze vector of the pupil circle of the i-th image, represents the pupil radius of the pupil circle of the i-th image. The cone of each pupil ellipse projection can draw two pupil circles, and it is unknown whether the two circles ( + , + ,r)or( - , - , ) is correct, but the projections of the normal vectors of the two pupil circles on the plane are parallel: + ∝ - ,in, + represents the pupil position of the positive pupil circle of the i-th image, + represents the gaze vector of the positive pupil circle of the i-th image, - represents the pupil position of the negative pupil circle of the i-th image, - represents the gaze vector of the positive pupil circle of the i-th image, + represents the projection of the normal vector of the pupil circle of the i-th image onto the plane, - represents the projection of the normal vector of the negative pupil circle of the i-th image on the plane;

[0087] (3) Similarly, the pupil positions of the two pupil circles projected on the plane are + and - The straight line between ± , in, + represents the pupil position of the projection of the positive pupil circle of the i-th image on the plane, - represents the pupil position of the projection of the negative pupil circle of the i-th image on the plane, ± represents the projection of the normal vectors of the positive pupil circle and the negative pupil circle of the i-th image on the plane, which means,

[0088] + = - +s + = - +t + ;

[0089] Among them, s represents the negative proportional coefficient and t represents the positive proportional coefficient;

[0090] (4) Find the center of the eyeball projection surface by calculating the intersection of straight lines that correspond to the projected gaze direction: each straight line passes through the projected pupil center and is parallel to the projected pupil normal. Find the set of projection lines L i The intersection of

[0091] L i ={(x, y)= +s };

[0092] Among them, x and y represent the coordinates in the projection space, represents the pupil position of the projection of the pupil circle of the i-th image on the plane, represents the normal vector of the projection of the pupil circle of the i-th image on the plane;

[0093] (5) By calculating and using the least squares method, find the point closest to each line.

[0094] = - ;

[0095] in, represents the center of the eyeball projection plane, represents the identity matrix, express The transpose of

[0096] Then reverse scale the two-dimensional coordinates to three-dimensional space, that is, through the center of the eyeball projection surface Find the center c of the 3D sphere;

[0097] (6) The center of the eyeball projection surface obtained by solving and the 3D sphere center c, find the normal vector of each pupil circle They all point in the opposite direction of the 3D sphere center c, and the normal vector of the projection of each pupil circle on the plane All point to the center of the eyeball projection surface The opposite direction of the projection on the plane is obtained Point to the center of the eyeball projection surface The pupil circle in the opposite direction of , , );

[0098] (7) However, since the size of the pupil circle cannot be determined, it cannot be used directly , consider introducing candidate pupil circle centers i and the candidate pupil circle radius i, candidate pupil circle center i Located through the observation point o and the center of the pupil circle , and consider the circle ( i , , i ) is tangent to the 3D sphere with center c, passing from the center c to i The gaze line observation point o and the pupil circle center The candidate pupil circle center is found by intersecting the lines of i ;

[0099] (8) Since straight lines in 3D will never intersect, find the least squares intersection again, and then calculate the sphere radius R by finding the average distance from the center of the sphere to the center of each pupil,

[0100] R=mean({R i = i -c i});

[0101] Where R represents the radius of the sphere, R i Represents the distance from the center of the i-th pupil circle i The distance to the center of the sphere c, i The center of the pupil circle of the i-th image, c represents the center of the 3D sphere;

[0102] (9) Due to the ambiguity of distance and size, the pupil circle cannot be guaranteed. , , ) is on the sphere, so we introduce a new circle ( , , ) ,in,

[0103] -c 2 = R 2 ;

[0104] = s ;

[0105] =c+R ;

[0106] = ;

[0107] in, After angle scaling , represents the z coordinate of the i-th pupil circle in three-dimensional space, represents the z coordinate of the i-th pupil circle in three-dimensional space after scaling. , find a scaling factor s such that s Located on the surface of the sphere (c, R), it can be calculated as the intersection of the line and the sphere, and two scaling factor solutions s1 and s2 are obtained. The smallest scaling factor is taken and the calculation is obtained and ;

[0108] (10) Confidence factor for each image frame that will be considered during 3D model fitting and gaze estimation Included in; the center of the eyeball projection surface The confidence-weighted fitting formula of and sphere radius R is expressed as:

[0109] - ;

[0110] R= ;

[0111] in, represents the center of the eyeball projection plane, represents the confidence factor of the i-th image, represents the center of the pupil circle where the i-th image is tangent to the sphere;

[0112] The pupil deviation angle is then calculated through the angle between the three-dimensional eye gaze vectors, and then the strabismus and prism degree of the subject are calculated.

[0113] A method for measuring strabismus based on pupil positioning, the measuring method comprising a corneal light reflection method and a covering method, the process of the corneal light reflection method is as follows:

[0114] Step 1.1, record the subject's information, fix the subject's head, place both eyes in front of the second liquid crystal light valve, use an infrared light source to fill the eyes, select the corneal reflection method in the application module to start measurement, and the camera starts to collect the subject's eye image information;

[0115] Step 1.2, adjust the red LED sight mark to high brightness, and light up the nine red LED sight marks in sequence for 2 seconds, in the order of upper left, upper, upper right, left, middle, right, lower left, lower, and lower right;

[0116] Step 1.3, the image processing system is used to process the eye images of the subject gazing at the nine sight marks, locate the distance between the reflected light point and the pupil center, and calculate the subject's strabismus in nine directions based on the principle that a deviation of 1 mm is equivalent to a strabismus of 7 degrees, and the measurement results are saved and displayed;

[0117] The process of the covering method is as follows:

[0118] Step 2.1, record the subject's information, fix the subject's head, place both eyes in front of the second liquid crystal light valve, use infrared light source to fill the eyes, select a covering method in the application and select a certain direction sight mark to start measurement, and the camera starts to collect the subject's eye image information;

[0119] Step 2.2, executing the masking program corresponding to the masking method through the human eye masking module, breaking the visual fusion of the human eye by controlling the masking of the left and right eyes, pausing image acquisition after the masking program is completed, and calculating the pupil center position information through the image processing system;

[0120] Step 2.3, by calculating the deviation angle of the pupil center position, converting the strabismus and prism degree, the measurement results are saved and displayed.

[0121] The covering method includes an alternating covering method, a covering and de-covering method, a prism alternating covering method and a prism covering and de-covering method.

[0122] Preferably, the process of alternating cover measurement is as follows:

[0123] Step 101, record the information of the subject, fix the head of the subject, place both eyes in front of the second liquid crystal light valve, fill the eyes with infrared light, select the alternating cover method in the application and select a certain direction sight mark to start measurement, and the camera starts to collect the subject's eye image information;

[0124] Step 1022, in the initial state, the left first liquid crystal light valve 5, the right first liquid crystal light valve 13, the left second liquid crystal light valve 10, and the right second liquid crystal light valve 16 are all in the power-off transparent state, and the left first liquid crystal light valve 5 and the left second liquid crystal light valve 10 are powered on to cover the left eye. At this time, the right eye is the dominant eye, and the right eye image information is collected. The pupil center position information is calculated by the image processing system as the right eye reference eye position;

[0125] Step 103, the left eye is transparent by turning off the power of the first liquid crystal light valve 5 and the second liquid crystal light valve 10, and the right eye is covered by turning on the first liquid crystal light valve 13 and the second liquid crystal light valve 16. At this time, the left eye is the dominant eye, and the image information of the left eye is collected. The pupil center position information is calculated by the image processing system as the reference eye position of the left eye.

[0126] Step 104, the left eye is covered by energizing the first liquid crystal light valve 5 and the second liquid crystal light valve 10 on the left side, and the right eye is transparent by de-energizing the first liquid crystal light valve 13 and the second liquid crystal light valve 16 on the right side. At this time, the right eye is the dominant eye, and the image information of the left eye and the right eye is collected, and the pupil angle values ​​A1 of the left eye, the right eye and the left eye reference eye position and the right eye reference eye position are calculated respectively;

[0127] Step 105, the left eye is transparent by turning off the power of the left first liquid crystal light valve 5 and the left second liquid crystal light valve 10, and the right eye is covered by the power of the right first liquid crystal light valve 13 and the right second liquid crystal light valve 16. At this time, the left eye is the dominant eye, and the image information of the left eye and the right eye is collected. The pupil angle value A2 of the left eye, the right eye and the left eye reference eye position and the right eye reference eye position is calculated respectively, and the offset of the left eye and the right eye is calculated by the angle difference between A2 and A1;

[0128] Step 106, repeating steps 104 and 105 and calculating the angle difference, taking the average of the angle differences calculated twice to determine the offset of the left eye and the right eye, and obtaining the strabismus and prism degree of the subject.

[0129] Alternatively, the specific process of the cover-uncover method measurement is as follows:

[0130] Step 111, record the information of the subject, fix the head of the subject, place both eyes in front of the second liquid crystal light valve, fill the eyes with infrared light, select the cover and uncover method in the application, select a certain direction sight mark, choose to cover the left eye / right eye, and then start measurement, and the camera starts to collect the subject's eye image information;

[0131] Step 112, in the initial state, the first left liquid crystal light valve 5, the first right liquid crystal light valve 13, the second left liquid crystal light valve 10, and the second right liquid crystal light valve 16 are all in a power-off transparent state, and the left eye is covered by powering on the first left liquid crystal light valve 5 and the second left liquid crystal light valve 10. At this time, the right eye is the dominant eye, and the right eye image information is collected. The pupil center position information is calculated by the image processing system as the right eye reference eye position;

[0132] Step 113, the left eye is transparent by turning off the power of the left first liquid crystal light valve 5 and the left second liquid crystal light valve 10, and the right eye is covered by turning on the power of the right first liquid crystal light valve 13 and the right second liquid crystal light valve 16. At this time, the left eye is the dominant eye, and the left eye image information is collected. The pupil center position information is calculated by the image processing system as the left eye reference eye position;

[0133] Step 114, the left / right eye is covered by energizing the first left / right liquid crystal light valve 13 and the second left / right liquid crystal light valve 16, and the right / left eye is transparent by de-energizing the first right / left liquid crystal light valve 5 and the second right / left liquid crystal light valve 10. At this time, the right eye / left eye is the dominant eye, and the image information of the left eye and the right eye is collected, and the pupil angle values ​​B1 of the left eye, the right eye and the left eye reference eye position and the right eye reference eye position are calculated respectively;

[0134] Step 115, the left eye is transparent by turning off the power of the first liquid crystal light valve 5 and the second liquid crystal light valve 10 on the left side, and the right eye is transparent by turning off the power of the first liquid crystal light valve 13 and the second liquid crystal light valve 16 on the right side, and the left eye and right eye image information are collected, and the pupil angle values ​​B2 of the left eye, the right eye and the left eye reference eye position and the right eye reference eye position are calculated respectively at this time, and the displacement of the left eye and the right eye is calculated by the angle difference between B2 and B1;

[0135] Step 116, repeat step 114 and step 115 and calculate the angle difference. Take the average of the two calculated angle differences to determine the offset of the left eye and the right eye, and obtain the subject's dominant eye, strabismus, and prism degree.

[0136] Alternatively, the measurement process using the prism alternating cover method is as follows:

[0137] Step 121, record the information of the subject, fix the head of the subject, place both eyes in front of the second liquid crystal light valve, fill the eyes with infrared light, select the alternating cover method in the application and select a certain direction sight mark to start measurement, and the camera starts to collect the subject's eye image information;

[0138] Step 122, select the microprism 9 and insert it into the prism sheet slot 29, adjust the spacing of the prism sheet slot 29 through the opposite displacement motor 31 so that the microprism 9 corresponds to the pupil distance, and then call the microprism displacement module according to the orientation of the sight mark to align the subject's viewing angle with the position of the microprism 9.

[0139] Step 123, in the initial state, the first left liquid crystal light valve 5, the first right liquid crystal light valve 13, the second left liquid crystal light valve 10, and the second right liquid crystal light valve 16 are all in the power-off transparent state, and the left eye is covered by powering on the first left liquid crystal light valve 5 and the second left liquid crystal light valve 10. At this time, the right eye is the dominant eye, and the right eye image information is collected. The pupil center position information is calculated by the image processing system as the right eye reference eye position;

[0140] Step 124, the left eye is transparent by turning off the power of the left first liquid crystal light valve 5 and the left second liquid crystal light valve 10, and the right eye is covered by turning on the power of the right first liquid crystal light valve 13 and the right second liquid crystal light valve 16. At this time, the left eye is the dominant eye, and the left eye image information is collected. The pupil center position information is calculated by the image processing system as the left eye reference eye position;

[0141] Step 125, the left eye is covered by energizing the first liquid crystal light valve 5 and the second liquid crystal light valve 10 on the left side, and the right eye is transparent by de-energizing the first liquid crystal light valve 13 and the second liquid crystal light valve 16 on the right side. At this time, the right eye is the dominant eye, and the image information of the left eye and the right eye is collected, and the pupil angle values ​​A1 of the left eye, the right eye and the left eye reference eye position and the right eye reference eye position are calculated respectively;

[0142] Step 126, the left eye is transparent by turning off the power of the left first liquid crystal light valve 5 and the left second liquid crystal light valve 10, and the right eye is covered by the power of the right first liquid crystal light valve 13 and the right second liquid crystal light valve 16. At this time, the left eye is the dominant eye, and the image information of the left eye and the right eye is collected. The pupil angle value A2 of the left eye, the right eye and the left eye reference eye position and the right eye reference eye position is calculated respectively, and the offset of the left eye and the right eye is calculated by the angle difference between A2 and A1;

[0143] Step 127, repeat steps 124 and 125 and calculate the angle difference. Take the average of the two calculated angle differences to determine the offset of the left eye and the right eye, and verify whether the strabismus of the subject matches the microprism 9 used. If not, how big the difference is.

[0144] Step 128, replace the corresponding microprism 9 according to the obtained prismatic power difference, and repeat steps 122 to 127 multiple times until there is no deviation between the left eye and the right eye of the subject. At this time, the power of the microprism 9 is the prismatic power of the subject.

[0145] Alternatively, the process of measuring by prism covering and uncovering method is as follows:

[0146] Step 131, record the information of the subject, fix the head of the subject, place both eyes in front of the second liquid crystal light valve, fill the eyes with infrared light, select the cover and uncover method in the application, select a certain direction sight mark, choose to cover the left eye / right eye, and then start measurement, and the camera starts to collect the subject's eye image information;

[0147] Step 132, select the microprism 9 and insert it into the prism sheet slot 29, adjust the spacing of the prism sheet slot 29 by the opposite displacement motor 31 so that the microprism 9 corresponds to the pupil distance of the subject, and then call the displacement module of the microprism 9 according to the orientation of the sight mark to align the subject's viewing angle with the position of the microprism 9;

[0148] Step 1313, in the initial state, the first left liquid crystal light valve 5, the first right liquid crystal light valve 13, the second left liquid crystal light valve 10, and the second right liquid crystal light valve 16 are all in the power-off transparent state, and the left first liquid crystal light valve 5 and the left second liquid crystal light valve 10 are powered on to cover the left eye. At this time, the right eye is the dominant eye, and the right eye image information is collected. The pupil center position information is calculated by the image processing system as the right eye reference eye position;

[0149] Step 314, the left eye is transparent by turning off the power of the left first liquid crystal light valve 5 and the left second liquid crystal light valve 10, and the right eye is covered by turning on the power of the right first liquid crystal light valve 13 and the right second liquid crystal light valve 16. At this time, the left eye is the dominant eye, and the left eye image information is collected. The pupil center position information is calculated by the image processing system as the left eye reference eye position;

[0150] Step 135, the left / right eye is covered by energizing the first left / right liquid crystal light valve 13 and the second left / right liquid crystal light valve 16, and the right / left eye is transparent by de-energizing the first right / left liquid crystal light valve 5 and the second right / left liquid crystal light valve 10. At this time, the right eye / left eye is the dominant eye, and the image information of the left eye and the right eye is collected, and the pupil angle values ​​B1 of the left eye, the right eye and the left eye reference eye position and the right eye reference eye position are calculated respectively;

[0151] Step 136, the left eye is transparent by turning off the power of the first liquid crystal light valve 5 and the second liquid crystal light valve 10 on the left side, and the right eye is transparent by turning off the power of the first liquid crystal light valve 13 and the second liquid crystal light valve 16 on the right side, and the left eye and right eye image information are collected, and the pupil angle values ​​B2 of the left eye, the right eye and the left eye reference eye position and the right eye reference eye position are calculated respectively at this time, and the displacement of the left eye and the right eye is calculated by the angle difference between B2 and B1;

[0152] Step 137, repeat steps 134 and 135 and calculate the angle difference. Take the average of the angle differences calculated twice to determine the offset of the left eye and the right eye, and verify whether the strabismus of the subject matches the microprism 9 used. If not, how big the difference is.

[0153] Step 138, replace the corresponding microprism 9 according to the obtained prismatic power difference, and repeat steps 132 to 137 multiple times until there is no deviation between the left eye and the right eye of the subject. At this time, the power of the microprism 9 is the prismatic power of the subject.

[0154] In this embodiment, the second liquid crystal light valve only retains a single-side polarization film. When the voltage on both sides is zero, the polarization direction is parallel to the polarization film and does not block light. When powered on, the polarization direction is perpendicular to the polarization film to block light. The second semi-transparent and semi-reflective mirror allows the infrared camera to capture the eye image information reflected by it; the polarizer allows the second liquid crystal light valve to control the light blocking and light transmission. The first liquid crystal light valve enhances the light blocking effect. The first semi-transparent and semi-reflective mirror 4 separates the near nine-point sight mark optical path from the far sight optical path, and reflects the two beams of light of the far sight splitting optical path 25 to the left eye 12 and the right eye 18 of the subject respectively. The aperture diaphragm 24 controls the incident angle and luminous flux of the light of the far sight mark 23. The far sight lens 26 causes the far sight mark 23 optical path to emit parallel light. The far sight semi-transparent and semi-reflective mirror 27 allows half of the light intensity of the emitted parallel light to enter the left eye through the far sight left eye optical path. The telescopic reflector 28 allows half of the light intensity after passing through the telescopic semi-transparent and semi-reflective mirror 27 to enter the right eye through the telescopic right eye optical path.

[0155] Furthermore, the telescopic beam splitting optical path 25 includes a telescopic sight mark 23 , an aperture stop 24 , a telescopic lens 26 , a telescopic semi-transparent mirror 27 and a telescopic reflecting mirror 28 .

[0156] The near vision nine-point sight mark of this embodiment is respectively located at the first eye position, the second eye position, and the third eye position of the human eye during measurement, and the viewing distance is 33 cm. The far vision sight mark 23 is imaged at a 6-meter position after passing through the aperture diaphragm 24 and the far vision lens 26, which is used to simulate a 6-meter viewing distance. Because the near vision nine-point sight mark measurement and the far vision measurement are not performed at the same time, the first semi-transparent and semi-reflective mirror 4 separates the two measurement optical paths to achieve a viewing distance switching function.

[0157] The micro-prismatic lens of this embodiment is composed of a group of Fresnel micro-structured prisms arranged neatly, which is used to achieve refraction control of incident light. Among them, the intersection end of the refraction surface is thinner, called the tip, and the other end is thicker, called the bottom. The cross section of the prism is a triangle. If the light passes through the prism along the cross section, it will be deflected toward the bottom. The degree of deflection can be quantified by the prismatic degree.

[0158] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A strabismus measurement device based on pupil positioning, characterized in that: The strabismus measuring device comprises an optical system, an electromechanical control system and an image processing system, wherein the optical system comprises a near vision nine-point sight mark optical path and a far vision optical path; the electromechanical control system comprises an application module for controlling the lighting selection of each sight mark according to the corneal reflection method and selecting the cover method measurement, a sight mark display module for realizing the corneal reflection method and the cover method, and a human eye cover module for covering the left eye or the right eye; the image processing system is used to perform image segmentation using a fully convolutional neural network through the eye image collected by the infrared camera to realize pupil positioning, and then to estimate the pupil line of sight by fitting a three-dimensional pupil model through a series of time-series eye images, and to realize the calculation of the measurement results of the reflection method and the cover method; The near vision nine-point sight mark optical path includes a near vision nine-point sight mark optical path for the left eye and a near vision nine-point sight mark optical path for the right eye, wherein the near vision nine-point sight mark optical path for the left eye includes nine sight marks in different directions connected in sequence, a first semi-transparent and semi-reflective mirror, a first liquid crystal light valve on the left side, a polarizing film on the left side, a second semi-transparent and semi-reflective mirror on the left side, and a second liquid crystal light valve on the left side, and the left infrared light source is aimed at the left eye position; the near vision nine-point sight mark optical path for the right eye includes nine sight marks in different directions connected in sequence, a first semi-transparent and semi-reflective mirror, a first liquid crystal light valve on the right side, a polarizing film on the right side, a second semi-transparent and semi-reflective mirror on the right side, and a second liquid crystal light valve on the right side, and the right infrared light source is aimed at the right eye position; The telescopic optical path includes a telescopic left-eye optical path and a telescopic right-eye optical path, wherein the telescopic left-eye optical path includes a telescopic sight mark, an aperture stop, a telescopic lens, a telescopic semi-transparent mirror, a first left-side liquid crystal light valve, a left-side polarizing film, a left-side second semi-transparent mirror, and a left-side second liquid crystal light valve connected in sequence; the telescopic right-eye optical path includes a telescopic sight mark, an aperture stop, a telescopic lens, a telescopic semi-transparent mirror, a telescopic reflective mirror, a first right-side liquid crystal light valve, a right-side polarizing film, a right-side second semi-transparent mirror, and a right-side second liquid crystal light valve connected in sequence; The left infrared camera is aimed at the reflection direction of the left second semi-transparent and semi-reflective mirror to capture the left eye image; the right infrared light source is aimed at the right eye position, and the right infrared camera is aimed at the reflection direction of the right second semi-transparent and semi-reflective mirror to capture the right eye image.

2. A strabismus measurement device based on pupil positioning as claimed in claim 1, characterized in that: The strabismus measuring device also includes a left microprism and a right microprism. The left microprism is located between the second semi-transparent and semi-reflective mirror on the left and the second liquid crystal light valve on the left. The right microprism is located between the second semi-transparent and semi-reflective mirror on the right and the second liquid crystal light valve on the right. The electromechanical control system also includes a microprism displacement module for controlling the displacement of the left microprism and the right microprism.

3. The strabismus measurement device based on pupil positioning according to claim 1, characterized in that: The nine sight marks in different directions include nine red LEDs in the middle, upper, lower, left, right, upper left, upper right, lower left and lower right directions, and the distance between two adjacent sight marks is equal.

4. A strabismus measurement device based on pupil positioning according to any one of claims 1 to 3, characterized in that: In the application module, the selection of covering method includes alternating covering, covering and uncovering, prism alternating covering method and prism covering and uncovering method; in the sight mark display module, the corneal reflection method adjusts the brightness of the sight mark to high brightness by adjusting the duty cycle of the PWM signal, and lights up nine sight marks in turn for 2 seconds, in the order of upper left, upper, upper right, left, middle, right, lower left, lower, and lower right; the covering method adjusts the brightness of the sight mark to low brightness by adjusting the duty cycle of the PWM signal, and then selects one of the sight mark points through the application module to complete the covering, so as to realize the strabismus measurement of the eye position.

5. The strabismus measurement device based on pupil positioning according to claim 2, characterized in that: The microprism displacement module includes a longitudinal displacement motor, a lateral displacement motor and an opposing displacement motor, so as to realize the displacement control of the left microprism and the right microprism. The left microprism and the right microprism are installed in corresponding prism sheet slots. The spacing of the prism sheet slots is first controlled by the opposing displacement motor to match the pupil distance of the subject. After selecting a certain position sight mark for prism cover test, the longitudinal displacement motor and the lateral displacement motor are controlled to achieve the field of view requirement of the human eye looking at the nine-point sight mark.

6. A strabismus measurement device based on pupil positioning according to any one of claims 1 to 3, characterized in that: The image processing system includes a pupil recognition module and a light reflection recognition module. The pupil recognition module uses a deep learning method. The deep learning method uses a fully convolutional neural network to segment the entire pupil area. The fully convolutional neural network is trained on manually annotated pupil images to achieve pupil positioning. A three-dimensional pupil model is then fitted through a series of time-series eye images that complete pupil positioning, and finally the estimation of the human eye's line of sight is completed. The pupil image is extracted by collecting eye images taken by an infrared camera under infrared light supplementary lighting, and the pupil area is manually labeled to complete the training of the neural network. The light reflection recognition module includes digital image processing and a deep learning neural network. The digital image processing uses image binarization processing to identify and mark the light reflection points on the eyeball, and then compares the pupil position information identified by the neural network to calculate the strabismus degree based on the positional relationship between the two.

7. The strabismus measurement device based on pupil positioning according to claim 6, characterized in that: The implementation process of the image processing system is as follows: (1) Consider the center of the infrared camera as the observation point o. Construct a cone on the image plane that passes through the observation point o and the edge of the pupil ellipse. Then find the circular intersection of the edge of the cone projected outward, which is the pupil circle. At any given distance, two circular intersections, i.e., two pupil circles, can be drawn to obtain the pupil position, gaze vector, and pupil radius: (p + ,n + ,r),(p - ,n - ,r), where p + represents the pupil position of the positive pupil circle, n + represents the gaze vector towards the pupil circle, p - Indicates the pupil position of the negative pupil circle, n - represents the gaze vector of the negative pupil circle, r represents the pupil radius of the pupil circle; (2) For each eye image, consider the pupil circle ( , , ), where i is the index of the image, represents the pupil position of the pupil circle of the i-th image, represents the gaze vector of the pupil circle of the i-th image, represents the pupil radius of the pupil circle of the i-th image. The cone of each pupil ellipse projection can draw two pupil circles, and it is unknown whether the two circles ( + , + ,r)or( - , - , ) is correct, but the projections of the normal vectors of the two pupil circles on the plane are parallel: + ∝ - ,in, + represents the pupil position of the positive pupil circle of the i-th image, + represents the gaze vector of the positive pupil circle of the i-th image, - represents the pupil position of the negative pupil circle of the i-th image, - represents the gaze vector of the positive pupil circle of the i-th image, + represents the projection of the normal vector of the pupil circle of the i-th image onto the plane, - represents the projection of the normal vector of the negative pupil circle of the i-th image on the plane; (3) Similarly, the pupil positions of the two pupil circles projected on the plane are + and - The straight line between ± ,in, + represents the pupil position of the projection of the positive pupil circle of the i-th image on the plane, - represents the pupil position of the projection of the negative pupil circle of the i-th image on the plane, ± represents the projection of the normal vectors of the positive pupil circle and the negative pupil circle of the i-th image on the plane, which means, + = - +s + = - +t + ; Among them, s represents the negative proportional coefficient and t represents the positive proportional coefficient; (4) Find the center of the eyeball projection surface by calculating the intersection of straight lines that correspond to the projected gaze direction: each straight line passes through the projected pupil center and is parallel to the projected pupil normal. Find the set of projection lines L i The intersection of L i ={(x, y)= +s }; Among them, x and y represent the coordinates in the projection space, represents the pupil position of the projection of the pupil circle of the i-th image on the plane, represents the normal vector of the projection of the pupil circle of the i-th image on the plane; (5) By calculating and using the least squares method, find the point closest to each line. = - ； in, represents the center of the eyeball projection plane, represents the identity matrix, express The transpose of Then reverse scale the two-dimensional coordinates to three-dimensional space, that is, through the center of the eyeball projection surface Find the center c of the 3D sphere; (6) The center of the eyeball projection surface obtained by solving and the 3D sphere center c, find the normal vector of each pupil circle They all point in the opposite direction of the 3D sphere center c, and the normal vector of the projection of each pupil circle on the plane All point to the center of the eyeball projection surface The opposite direction of the projection on the plane is obtained Point to the center of the eyeball projection surface The pupil circle in the opposite direction of , , ); (7) Since the size of the pupil circle cannot be determined, it cannot be used directly , consider introducing candidate pupil circle centers i and the candidate pupil circle radius i , candidate pupil circle center i Located through the observation point o and the center of the pupil circle , and consider the circle ( i , , i ) is tangent to the 3D sphere with center c, passing from the center c to i The gaze line observation point o and the pupil circle center The candidate pupil circle center is found by intersecting the lines of i ; (8) Since straight lines in 3D will never intersect, find the least squares intersection again, and then calculate the sphere radius R by finding the average distance from the center of the sphere to the center of each pupil, R=mean({R i = i -c i}); Where R represents the radius of the sphere, R i Represents the distance from the center of the i-th pupil circle i The distance to the center of the sphere c, i The center of the pupil circle of the i-th image, c represents the center of the 3D sphere; (9) Due to the ambiguity of distance and size, the pupil circle cannot be guaranteed. , , ) is on the sphere, so we introduce a new circle ( , , ) ,in, -c 2 = R 2 ; = s ； =c+R ; = ； in, After angle scaling , represents the z coordinate of the i-th pupil circle in three-dimensional space, represents the z coordinate of the i-th pupil circle in three-dimensional space after scaling. , find a scaling factor s such that s Located on the surface of the sphere (c, R), it can be calculated as the intersection of the line and the sphere, and two scaling factor solutions s1 and s2 are obtained. The smallest scaling factor is taken and the calculation is obtained and ; (10) Confidence factor for each image frame that will be considered during 3D model fitting and gaze estimation Included in; the center of the eyeball projection surface The confidence-weighted fitting formula of and sphere radius R is expressed as: - ； R= ； in, represents the center of the eyeball projection plane, represents the confidence factor of the i-th image, represents the center of the pupil circle where the i-th image is tangent to the sphere; The pupil deviation angle is then calculated through the angle between the three-dimensional eye gaze vectors, and then the strabismus and prism degree of the subject are calculated.

8. A measurement method implemented by the strabismus measurement device based on pupil positioning as claimed in claim 1, characterized in that: The measurement method includes corneal reflection method and covering method. The process of corneal reflection method is as follows: Step 1.1, record the subject's information, fix the subject's head, place both eyes in front of the second liquid crystal light valve, use an infrared light source to fill the eyes, select the corneal reflection method in the application module to start measurement, and the camera starts to collect the subject's eye image information; Step 1.2, adjust the red LED sight mark to high brightness, and light up the nine red LED sight marks in sequence for 2 seconds, in the order of upper left, upper, upper right, left, middle, right, lower left, lower, and lower right; Step 1.3, the image processing system is used to process the eye images of the subject gazing at the nine sight marks, locate the distance between the reflected light point and the pupil center, and calculate the subject's strabismus in nine directions based on the principle that a deviation of 1 mm is equivalent to a strabismus of 7 degrees, and the measurement results are saved and displayed; The process of the covering method is as follows: Step 2.1, record the subject's information, fix the subject's head, place both eyes in front of the second liquid crystal light valve, use infrared light source to fill the eyes, select a covering method in the application and select a certain direction sight mark to start measurement, and the camera starts to collect the subject's eye image information; Step 2.2, executing the masking program corresponding to the masking method through the human eye masking module, breaking the visual fusion of the human eye by controlling the masking of the left and right eyes, pausing image acquisition after the masking program is completed, and calculating the pupil center position information through the image processing system; Step 2.3, by calculating the deviation angle of the pupil center position, converting the strabismus and prism degree, the measurement results are saved and displayed.

9. The measuring method according to claim 8, characterized in that: In step 2.2, when the masking program selects the prism alternating masking method or the prism masking and unmasking method, the microprism displacement module is called to displace the microprism to a position that matches the pupil distance of the subject, and after matching the position with the selected sight mark, the masking program is completed through the human eye masking module.

Images