An interactive binocular vision function training system and method based on line of sight estimation
Through an interactive binocular visual function training system based on line of sight estimation, the deep learning neural network and virtual occlusion model are used to solve the problem of singularity and balance of the existing amblyopia training methods, and the effective training of binocular visual functions is achieved, which improves the Training effect and fitness.
Patent Information
- Application Number
- CN202411133910.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-08-19
AI Technical Summary
The existing amblyopia training methods are large in size and single in training mode, which ignore the balance of the normal visual function of the dominant eye and the visual information of the binocular eye, resulting in poor training results, especially for younger patients.
Adopt an interactive binocular visual function training system based on line of sight estimation, combined with optical system and processing control system, balance binocular visual information input through real-time line of sight estimation technology, including optical system and processing control system, and predict gaze coordinates using deep learning neural networks. And visual function training is carried out through virtual occlusion model and three-dimensional virtual environment.
It improves the training effect and adaptability, stimulates the enthusiasm of patients, balances the development of visual functions of both eyes, avoids the suppression of dominant eyes in traditional methods, and enhances the durability and effectiveness of vision training.
Smart Images

Figure CN118649070B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of vision training, and in particular relates to an interactive binocular vision function training system and method based on line of sight estimation. Background Art
[0002] Amblyopia is a common clinical ophthalmological disease caused by abnormal interaction between brain development and visual experience, which leads to neurodevelopmental disorders of the visual system and disordered spatial visual development. Amblyopia has a significant impact on the visual input of the brain, disrupting the normal development of visual function and causing different vision defects.
[0003] Binocular vision is divided into three levels. The first level is simultaneous vision. Establishing normal simultaneous vision can not only balance the inhibition between the superior and inferior eyes, but also establish a solid prerequisite for binocular advanced vision. The second level is fusion vision, which assists in forming a normal retinal correspondence, coordinates binocular movement, and increases the fusion range. The third level is stereoscopic vision, which allows the brain to fuse the images observed by the two eyes into a complete stereoscopic impression to distinguish the distance and shape of the scene. Only when the first level of vision function is perfected can the second level of vision function be established; only when the second level of vision function is perfected can the foundation for the development of the third level of vision function be laid. The three levels of vision are linked together and work in sequence, so that we can observe the world clearly, permanently and comfortably. Normal and perfect binocular vision is indispensable in human daily life, learning and work scenes. However, due to some congenital pathological reasons, improper use of eyes in life, eye fatigue and other conditions, more and more people suffer from varying degrees of amblyopia or even strabismus. Not only do they often feel visual fatigue, eye soreness, eye swelling, difficulty in near vision, drowsiness, etc., but they also lose some visual functions, which greatly affects their normal life. The current visual function training methods mainly use reversal photography, stereoscopic atlases, occlusion therapy, synoptophore training, etc. These training methods are not only monotonous in form and content, easily causing numb memory and weakening the training effect, but also cannot fully mobilize the enthusiasm of patients, especially for young patients. Summary of the invention
[0004] In order to overcome the shortcomings of existing amblyopia training methods, such as large size, single training mode, ignoring the normal visual function of the dominant eye while training the visual function of the amblyopic eye, and ignoring the balance of binocular visual information, the present invention provides an interactive binocular vision function training system and method based on line of sight estimation, which combines real-time line of sight estimation technology to solve the technical problem of how to balance binocular vision information input as an optimized medium for visual function training, thereby improving the effect and adaptability of user training.
[0005] To achieve the above technical purpose, the present invention adopts an interactive binocular vision function training system based on line of sight estimation, including an optical system and a processing control system, wherein the optical system refers to a symmetrically arranged left eye amblyopia optical path and a right eye amblyopia optical path; the left eye amblyopia optical path includes a left infrared light source, a left red / blue filter, a polarizer and an infrared camera connected in sequence; the right eye amblyopia optical path includes a right infrared light source, a right red / blue filter, a polarizer and an infrared camera connected in sequence; the polarizer finally passed by the left and right amblyopia optical paths is the same polarizer, and the receiving infrared camera is the same infrared camera;
[0006] The processing and control system includes a computer display, a binocular vision function training module, a human eye fixation point prediction module and a dominant eye suppression module. The computer display is connected to a left infrared light source, a right infrared light source and an infrared camera. The binocular vision function training module includes a three-level visual function training game for the computer display as a terminal display device; the human eye fixation point prediction module stores the facial posture image acquired by the infrared camera, segments and analyzes the binocular and facial images to obtain the fixation point prediction coordinates, and performs multi-software interaction with the binocular vision function training module; the dominant eye suppression module includes the generation and real-time display of an occlusion model, and the generation of the occlusion model refers to generating a virtual occlusion model using a three-dimensional modeling software; the real-time display of the occlusion model refers to superimposing and displaying the virtual occlusion model on the user fixation point coordinate position output by a deep learning neural network based on line of sight estimation, and the superimposed display is only on a preset dominant eye terminal display device.
[0007] Furthermore, the eye gaze point prediction module includes a deep learning neural network, which is based on a deep learning neural network model based on line of sight estimation and is improved and trained by changing the data loading method and the coordinate generation method: the user's current gaze information is obtained by calling an infrared camera, and the gaze point coordinate data is predicted in real time; the infrared camera is located at the top of the center of the computer screen in a system with a computer screen as a terminal display device, and the real-time user gaze information obtained by it is: eye position and facial posture information, and the gaze point is predicted by extracting features from images of the left eye, right eye and facial information, and then using a fully connected layer regression to achieve prediction of the gaze point, and at the same time using facial posture information as supervision information to recalibrate the eye features. After the improvement, the camera can be directly called to achieve the effect of generating predicted point coordinates in real time;
[0008] By taking the face image , left eye image , right eye image and the position of the face relative to the image Input into the gaze point prediction model and finally get the predicted point , first and Input to Get the output feature vector and ,
[0009] ;
[0010] ;
[0011] Then connect the feature vectors of the left eye and the right eye.
[0012] ;
[0013] And finally processed by the fully connected layer,
[0014] ;
[0015] For the feature extraction of face images, the face images are Input to middle,
[0016] ;
[0017] The same is true for the position of the face relative to the image. , enter it into middle,
[0018] ;
[0019] Integration , and All the data information contained is predicted through the fully connected layer to obtain the standard coordinates of the real-time gaze point ,
[0020] ;
[0021] In the formula, , and The left eye image and right eye image , face image and the position of the face relative to the image The output obtained after passing through the respective network models.
[0022] Preferably, the gaze point coordinate data refers to the real-time coordinate data of the user's gaze point falling on the terminal display device screen, which is the coordinate data in a plane rectangular coordinate system established with the center of the screen as the origin: in view of the differences in the sizes of different computer screens, in order to ensure that the output gaze point coordinate data is always based on the current computer center origin, the following formula is used to achieve the versatility of gaze point tracking:
[0023] ;
[0024] ;
[0025] In the formula, and refers to the actual coordinates of the predicted gaze point on the user's computer screen, and Refers to the standard prediction point coordinates calculated by the deep learning neural network. and It refers to the actual length and width of the user's computer screen. and It refers to the offset in length and width of the user's computer screen specifications. The coordinate origin of the predicted gaze point coordinate data is calibrated to the center of the user's computer screen through the formula.
[0026] In the binocular vision function training module, the third-level vision function training game for the computer screen as the terminal display device is to construct a three-dimensional virtual environment scene and Gabor sight mark through a computer, and to interact with the game in accordance with the principle of controlling the screen perspective with a mouse and controlling the movement of the perspective with a keyboard. During the game, the screen is always displayed from the first perspective or the third perspective, and the game requirements are met or the conditions are met to pass the level. After the game is over, the game duration and score records are saved.
[0027] Preferably, the three-level visual function training game includes a simultaneous vision training game, a fusion vision training game and a stereoscopic vision training game based on a computer screen as a terminal display device, and PC game development is performed through Unity 3D software: the card game uses a random arrangement algorithm Gabor sight mark to stimulate the user's cerebral cortex to train the user's vision, and achieves simultaneous vision training by focusing on and distinguishing the orientation and proximity of the Gabor sight mark; the Gabor collision game uses a random arrangement algorithm and a mobile component interaction algorithm to mobilize the cooperation of hands, eyes and brain based on the principle of perceptual training, thereby training the user's binocular acceptance of objects and images at the same time, so as to achieve simultaneous vision training; the animal gathering and dispersion classification game uses a random arrangement algorithm and a mobile component interaction algorithm to perceive the monocular fusion distance range of the two eyes as the principle, and mobilizes the brain to adjust the binoculars to maintain sensory fusion and motor fusion capabilities. To train fusion vision; the Marsden dodgeball game uses a random falling algorithm and a rigid body collision detection algorithm to simulate the movement trajectory of the Marsden ball in a variety of real-life scenarios, satisfying the far and near left and right longitudinal movement space to train the user's perception of depth, in order to achieve stereoscopic vision training; the maze game uses a perspective manipulation algorithm and a rigid body collision detection algorithm to stimulate the user's cerebral cortex with grating stimulation of different spatial frequencies and near and far distance displacement, train the user's visual function, mobilize the hand, eye, and brain to collaborate through visual perception training to identify and grasp Gabor visual marks, improve the defective pathways of single and binocular vision perception, observe the moving stripe gratings of different depths in space, train the depth motion perception, and strengthen the establishment of stereoscopic perception.
[0028] The difficulty and type of the simultaneous vision training game, fusion vision training game and stereoscopic vision training game are adjusted according to the user's training plan, and are divided into Gabor sight mark parameter and obstacle parameter adjustment, including but not limited to the number of Gabor sight marks, the direction of Gabor sight marks, the spatial frequency of Gabor sight marks, the proximity of Gabor sight marks, the number of obstacles, the speed of obstacles, the size of obstacles, the contrast clarity of obstacles, and the adjustment of the horizontal and vertical distance parameters of obstacles.
[0029] In the dominant eye suppression module, the virtual occlusion model is designed and constructed specifically according to the user's specific visual function and the formulated training program, and can be adjusted according to the results of user training feedback. The adjustable parameters include the color, shape, material, size, contrast, clarity, frequency of occurrence and speed of the virtual occlusion model.
[0030] In the dominant eye suppression module, only the display is superimposed on the preset dominant eye terminal display device. When the terminal display device is a computer screen, red and blue glasses are required. Only red light can pass through the red lens, which can filter out the red content in the picture; only blue light can pass through the blue lens, which can filter out the blue content in the picture. In this way, the left and right eyes see different pictures through the red and blue lenses. By adjusting the color of the virtual occlusion model, the dominant eye position can be preset in advance to achieve the suppression effect of the dominant eye.
[0031] The processing and control system also includes a user information management module, which includes a user information entry system and a user information query. The user information entry system includes obtaining user personal information, diagnostic data and game training data, and requires the user to register or log in to an account when using the binocular vision function module training to facilitate the storage of information data; the user information query system includes a user viewing the user training effect independent of the binocular vision function module training, and requires the user to register or log in to the same account;
[0032] The personal information includes the user's age, gender, name and amblyopia; the amblyopia includes binocular naked eye vision, optometry, corrected vision, strabismus (degree), superior / abdominal eye position, stereoscopic vision and intensity parameters of the formulated training program; the game training data is timely saved in the system through the completion of the third-level visual function game in the binocular vision function training module, including training duration, training cycle and game score information.
[0033] The system also includes a chin rest, a bracket, a knob and a lifting screw, a gear is installed on the rotating shaft of the knob, the gear is meshed with the lifting screw, and the chin rest is installed on the upper end of the lifting screw. The knob is turned to drive the lifting screw to rotate by rotating the gear, so that the chin rest can move up and down.
[0034] In the present invention, an infrared light source illuminates the human eye and plays a role of fill light, so that the camera can easily obtain images of the human eyes and facial posture; the infrared camera is located on the same plane as the computer monitor as the terminal display device, and is at the uppermost end of the center of the computer monitor. During measurement, it is located directly in front of the human eye, and the viewing distance is 0.70 meters; the system is provided with a symmetrical optical system, that is, a set of the system is provided for each left eye; the left infrared light source and the right infrared light source are both infrared LED light sources, and the light emission wavelength is 940nm.
[0035] An interactive binocular vision function training method based on line of sight estimation comprises the following steps:
[0036] Step 1: The user fixes his head and puts both eyes in a natural gaze state. The left infrared light source and the right infrared light source illuminate the human eyes for fill light, and the type and intensity of the three-level visual function training game for the computer monitor as the terminal display device are set;
[0037] Step 2: When starting the game, turn on the infrared camera to capture the user's head posture and eye information in real time, predict the coordinate data of the gaze point in real time, generate a virtual occlusion model and overlay it on the computer screen of the dominant eye;
[0038] Step 3: At the beginning of the game, Gabor sight marks are randomly generated according to the game type and training intensity. The user needs to wear red and blue glasses to pass the game level according to the clearance prompts;
[0039] Step 4: After completing a game, the game type, play time and score will be automatically saved; you can also choose other game types and training intensities to conduct binocular vision function training again.
[0040] The technical concept of the present invention is that Gabor sight marks are a kind of neural vision or perceptual vision training based on visual stimulation and promotion of neural connections responsible for vision. In the contour perception task, because Gabor sight marks have no edge prompts, contrast, size, spatial frequency, orientation and spatial position can all be used and processed, and are regarded as suitable stimulus elements, which are conducive to the mastery of spatial integration ability of amblyopic eyes and play a pivotal role in perceptual training. At the same time, orientation information is an important attribute of visual stimulus signals. In perceptual learning research, orientation discrimination is usually represented by the ability to resolve the orientation difference of sinusoidal bars. Perceptual learning can significantly improve the user's orientation discrimination ability. Therefore, it is very necessary to invent new technical methods and equipment to train the visual function of amblyopic eyes of patients and balance the ability of binocular reception of information and image processing fusion. The present invention proposes a new system and method, which overcomes the problems existing in the above-mentioned existing technologies. Different from traditional monocular repair training, it directly trains binocular vision function, which can not only achieve good training effects, but also help to improve binocular vision function and maintain training effects.
[0041] Gaze estimation is the process of predicting a person's gaze direction or gaze point position through the state of the eyes and the posture of the head. It has important value in the fields of neuroscience, psychology, computer science, etc. The goal of gaze estimation is to derive the gaze direction from a face image or an eye image. Although gaze estimation technology is now used in many fields, including game interaction, virtual reality, medical treatment, assisted driving, and web page analysis, there is no application of gaze estimation in the field of visual function training. Based on the important features of gaze estimation in capturing and visualizing people's focus points, the present invention proposes for the first time an interactive binocular visual function training system and method based on gaze estimation. Through the gaze tracking algorithm, the local area of human eye focus can be rendered with high precision, thereby improving the compliance of visual function training, saving energy consumption and reducing hardware costs, and also helping doctors to better understand the patient's training situation.
[0042] The beneficial effects of the present invention are mainly manifested in:
[0043] 1) Innovation in the cross-integration of optics and ophthalmology: This invention is based on the principle of amblyopia training, integrated with multimedia technology, and cleverly and efficiently combines optics and ophthalmology according to medical training requirements and basic optical knowledge. It independently designs a new binocular training game that meets the third-level visual function, and invents a training system, equipment and method for vision training.
[0044] 2) Balance binocular visual information input: Compared with traditional visual function training methods, the present invention avoids the suppression of the dominant eye's visual function in complete occlusion therapy, cultivates the user's binocular cooperation habits, and not only completes targeted single and binocular visual function training for the inferior eye, but also protects the normal visual function of the dominant eye during the training process, creates special conditions for the superior and inferior eyes to work together, stimulates specific neurons in the visual cortex and trains them to process image information more efficiently, thereby gradually forming new neural connections and achieving lasting vision improvement.
[0045] 3) High training compliance, simple and convenient operation: The visual function training games developed by the present invention, which combine the fun of games with the rigor of amblyopia training, are of various types, with rich training content, in line with children's psychological characteristics, and improve user training compliance. It is easy to use, and you can start binocular vision function training at home, making it easier to persist in training for a long time. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 It is a light path diagram of an interactive binocular vision function training system based on line of sight estimation;
[0047] Figure 2 It is a schematic diagram of the technical route of an interactive binocular vision function training system based on line of sight estimation;
[0048] Figure 3It is a flow chart based on the computer screen as the terminal display device;
[0049] Figure 4 It is a schematic diagram of the chin rest lifting mechanism;
[0050] Figure 5 It is a schematic diagram of the structure of the terminal display device based on the computer screen;
[0051] Serial numbers and names of the accompanying drawings: 1 left infrared light source, 2 right infrared light source, 3 left red / blue filter, 4 right red / blue filter, 5 polarizer, 6 infrared camera, 7 human eye, 8 display, 9 chin rest, 10 bracket, 11 knob, 12 lifting screw. DETAILED DESCRIPTION
[0052] The present invention will be further described below in conjunction with the accompanying drawings.
[0053] Reference Figure 1~Figure 5 , an interactive binocular vision function training device based on line of sight estimation, including an optical system and a processing control system, wherein the optical system refers to a symmetrically arranged left eye amblyopia optical path and a right eye amblyopia optical path; the left eye amblyopia optical path includes a left infrared light source 1, a left red / blue filter 3, a polarizer 5, and an infrared camera 6 connected thereto; the right eye amblyopia optical path includes a right infrared light source 2, a right red / blue filter 4, a polarizer 5, and an infrared camera 6 connected thereto; wherein the infrared light source irradiates the human eye 7 to play a fill light role, so that the infrared camera can obtain eye images conveniently; the polarizer 5 finally passed by the left and right amblyopia optical paths is the same polarizer, and the receiving infrared camera 6 is the same infrared camera;
[0054] The processing and control system includes a computer display, a binocular vision function training module, a human eye fixation point prediction module and a dominant eye suppression module. The computer display is connected to a left infrared light source, a right infrared light source and an infrared camera. The binocular vision function training module includes a three-level visual function training game for the computer display as a terminal display device; the human eye fixation point prediction module stores the facial posture image acquired by the infrared camera, segments and analyzes the binocular and facial images to obtain the fixation point prediction coordinates, and performs multi-software interaction with the binocular vision function training module; the dominant eye suppression module includes the generation and real-time display of an occlusion model, and the generation of the occlusion model refers to generating a virtual occlusion model using a three-dimensional modeling software; the real-time display of the occlusion model refers to superimposing and displaying the virtual occlusion model on the user fixation point coordinate position output by a deep learning neural network based on line of sight estimation, and the superimposed display is only on a preset dominant eye terminal display device.
[0055] Furthermore, the eye gaze point prediction module includes a deep learning neural network, which is based on a deep learning neural network model based on line of sight estimation and is improved and trained by changing the data loading method and the coordinate generation method: the user's current gaze information is obtained by calling an infrared camera, and the gaze point coordinate data is predicted in real time; the infrared camera is located at the top of the center of the computer screen in a system with a computer screen as a terminal display device, and the real-time user gaze information obtained by it is: the position of both eyes and facial posture information, and the prediction of the gaze point is achieved by extracting features from images of the left eye, right eye and facial information, and then using a fully connected layer regression to achieve the prediction of the gaze point, and at the same time using the facial posture information as the supervision information to recalibrate the eye features. After the improvement, the infrared camera can be directly called to achieve the effect of generating the predicted point coordinates in real time;
[0056] By taking the face image , left eye image , right eye image and the position of the face relative to the image Input into the gaze point prediction model and finally get the predicted point , first and Input to Get the output feature vector and ,
[0057] ;
[0058] ;
[0059] Then connect the feature vectors of the left eye and the right eye.
[0060] ;
[0061] And finally processed by the fully connected layer,
[0062] ;
[0063] For the feature extraction of face images, the face images are Input to middle,
[0064] ;
[0065] The same is true for the position of the face relative to the image. , enter it into middle,
[0066] ;
[0067] Integration , and All the data information contained is predicted through the fully connected layer to obtain the standard coordinates of the real-time gaze point ,
[0068] ;
[0069] In the formula, , and The left eye image and right eye image , face image and the position of the face relative to the image The output obtained after passing through the respective network models.
[0070] Preferably, the gaze point coordinate data refers to the real-time coordinate data of the user's gaze point falling on the terminal display device screen, which is the coordinate data in a plane rectangular coordinate system established with the center of the screen as the origin: in view of the differences in the sizes of different computer screens, in order to ensure that the output gaze point coordinate data is always based on the current computer center origin, the following formula is used to achieve the versatility of gaze point tracking:
[0071] ;
[0072] ;
[0073] In the formula, and refers to the actual coordinates of the predicted gaze point on the user's computer screen, and Refers to the standard prediction point coordinates calculated by the deep learning neural network. and It refers to the actual length and width of the user's computer screen. and It refers to the offset in length and width of the user's computer screen specifications. The coordinate origin of the predicted gaze point coordinate data is calibrated to the center of the user's computer screen through the formula.
[0074] In the binocular vision function training module, the third-level vision function training game for the computer screen as the terminal display device is to construct a three-dimensional virtual environment scene and Gabor sight mark through a computer, and to interact with the game in accordance with the principle of controlling the screen perspective with a mouse and controlling the movement of the perspective with a keyboard. During the game, the screen is always displayed from the first perspective or the third perspective, and the game requirements are met or the conditions are met to pass the level. After the game is over, the game duration and score records are saved.
[0075] Preferably, the three-level visual function training game includes a simultaneous vision training game, a fusion vision training game and a stereoscopic vision training game based on a computer screen as a terminal display device, and PC game development is performed through Unity 3D software: the card game uses a random arrangement algorithm Gabor sight mark to stimulate the user's cerebral cortex to train the user's vision, and achieves simultaneous vision training by focusing on and distinguishing the orientation and proximity of the Gabor sight mark; the Gabor collision game uses a random arrangement algorithm and a mobile component interaction algorithm to mobilize the cooperation of hands, eyes and brain based on the principle of perceptual training, thereby training the user's binocular acceptance of objects and images at the same time, so as to achieve simultaneous vision training; the animal gathering and dispersion classification game uses a random arrangement algorithm and a mobile component interaction algorithm to perceive the monocular fusion distance range of the two eyes as the principle, and mobilizes the brain to adjust the binoculars to maintain sensory fusion and motor fusion capabilities. To train fusion vision; the Marsden dodgeball game uses a random falling algorithm and a rigid body collision detection algorithm to simulate the movement trajectory of the Marsden ball in a variety of real-life scenarios, satisfying the far and near left and right longitudinal movement space to train the user's perception of depth, in order to achieve stereoscopic vision training; the maze game uses a perspective manipulation algorithm and a rigid body collision detection algorithm to stimulate the user's cerebral cortex with grating stimulation of different spatial frequencies and near and far distance displacement, train the user's visual function, mobilize the hand, eye, and brain to collaborate through visual perception training to identify and grasp Gabor visual marks, improve the defective pathways of single and binocular vision perception, observe the moving stripe gratings of different depths in space, train the depth motion perception, and strengthen the establishment of stereoscopic perception.
[0076] The difficulty and type of the simultaneous vision training game, fusion vision training game and stereoscopic vision training game are adjusted according to the user's training plan, and are divided into Gabor sight mark parameter and obstacle parameter adjustment, including but not limited to the number of Gabor sight marks, the direction of Gabor sight marks, the spatial frequency of Gabor sight marks, the proximity of Gabor sight marks, the number of obstacles, the speed of obstacles, the size of obstacles, the contrast clarity of obstacles, and the adjustment of the horizontal and vertical distance parameters of obstacles.
[0077] In the dominant eye suppression module, the virtual occlusion model is designed and constructed specifically according to the user's specific visual function and the formulated training program, and can be adjusted according to the results of user training feedback. The adjustable parameters include the color, shape, material, size, contrast, clarity, frequency of occurrence and speed of the virtual occlusion model.
[0078] In the dominant eye suppression module, only the display is superimposed on the preset dominant eye terminal display device. When the terminal display device is a computer screen, red and blue glasses are required. Only red light can pass through the red lens, which can filter out the red content in the picture; only blue light can pass through the blue lens, which can filter out the blue content in the picture. In this way, the left and right eyes see different pictures through the red and blue lenses. By adjusting the color of the virtual occlusion model, the dominant eye position can be preset in advance to achieve the suppression effect of the dominant eye.
[0079] The processing and control system also includes a user information management module, the user information management module includes a user information entry system and a user information query, the user information entry system includes obtaining user personal information, diagnostic data and game training data, and requires the user to register or log in to an account when using the binocular vision function module training to facilitate the storage of information data; the user information query system includes a user viewing the user training effect independent of the binocular vision function module training, and requires the user to register or log in to the same account;
[0080] The personal information includes the user's age, gender, name and amblyopia; the amblyopia includes binocular naked eye vision, optometry, corrected vision, strabismus (degree), superior / abdominal eye position, stereoscopic vision and intensity parameters of the formulated training program; the game training data is timely saved in the system through the completion of the third-level visual function game in the binocular vision function training module, including training duration, training cycle and game score information.
[0081] The system also includes a chin rest, a bracket, a knob and a lifting screw, a gear is installed on the rotating shaft of the knob, the gear is meshed with the lifting screw, and the chin rest is installed on the upper end of the lifting screw. The knob is turned to drive the lifting screw to rotate by rotating the gear, so that the chin rest can move up and down.
[0082] In the present invention, an infrared light source illuminates the human eye and plays a role of fill light, so that the camera can easily obtain images of the human eyes and facial posture; the infrared camera is located on the same plane as the computer monitor as the terminal display device, and is at the uppermost end of the center of the computer monitor. During measurement, it is located directly in front of the human eye, and the viewing distance is 0.70 meters; the system is provided with a symmetrical optical system, that is, a set of the system is provided for each left eye; the left infrared light source and the right infrared light source are both infrared LED light sources, and the light emission wavelength is 940nm.
[0083] This embodiment uses a computer screen as the terminal display device for amblyopia training, and its flow chart is as follows: Figure 3 As shown, the following steps are included:
[0084] Step 1: The user fixes his head on the chin rest 9 and keeps his eyes in a natural gaze state. The left infrared light source 1 and the right infrared light source 2 illuminate the eyes for supplementary light. The user registers an account and logs in with a valid account and password to enter the homepage of the user information management system. On this page, you can upload user personal information and diagnostic data, select visual function training games, and view historical training records;
[0085] Among them, the uploaded user personal information includes the user's age, gender, name, training background, training duration, training cycle, game scores and other personal information, which is convenient for establishing the user's personal training file for subsequent testing of training effects;
[0086] Among them, the diagnostic data includes the naked eye vision of both eyes, optometry, corrected vision, strabismus (degree), superior / weak vision, stereoscopic vision, and the intensity parameters of the training program formulated by the doctor;
[0087] Among them, the selection of visual function training games includes simultaneous vision training games, fusion vision training games and stereoscopic vision training games, and the difficulty setting of each game can be adjusted;
[0088] For the simultaneous vision training game in the present embodiment, as an example of the present embodiment, the card game: three difficulty levels can be set, corresponding to Gabor sight marks of different numbers and orientations. The first level contains a single Gabor sight mark corresponding to a single orientation and a single adjacency, which is used to stimulate the user's cerebral cortex to simply distinguish the Gabor sight mark. The second level contains 12 Gabor sight marks corresponding to 6 groups of orientations and a single adjacency. The third level contains 36 Gabor sight marks corresponding to 18 groups of orientations and multiple adjacencies. The level requires the user to click on the card and memorize the content of the card. Each click is limited to two cards. After 3 seconds, the card information is initialized to the initial screen. When exactly the same Gabor sight mark appears on the two cards clicked in a single click, the pairing is successful. When all cards are successfully paired, the level can be passed.
[0089] For the simultaneous vision training game in the present embodiment, as an example of the present embodiment, the Gabor matching game: the level contains 63 randomly appearing Gabor sight marks in different directions and multiple adjacent Gabor sight marks are randomly arranged. The user needs to move the horizontal or vertical Gabor sight marks so that three or more completely identical Gabor sight marks are adjacent in the horizontal and vertical directions to eliminate these Gabor sight marks, and each Gabor sight mark will get 10 points. A new Gabor sight mark will be randomly generated at the top of the vertical column where the eliminated Gabor sight mark is located to move downward. The game can be passed if the required lower limit of points is reached within the specified time. The initial specified time is 20 seconds, and the initial required points are 100 points. After each successful clearance, the specified time increases by 10 seconds, and the required points increase by 80 points;
[0090] For the fusion vision training game in this embodiment, as an example of this embodiment, the animal gathering and dispersing classification game: the level contains 10-30 randomly appearing animal images and 1-10 cages of different shapes. The animal images and cages are all graphics with red and blue color difference principle after special image processing. As the number of cages increases and the volume of cages decreases, the difficulty of the game increases. The animal images corresponding to cages of different shapes in each level are different. According to the prompt information of the level, the randomly generated animal images are classified into the cages to which they belong. The game can be passed by completing the classification of all animal images within the specified time. The initial specified time is 90 seconds, and the specified time is reduced by 15 seconds after each successful clearance.
[0091] For the stereoscopic vision training game in this embodiment, as an example of this embodiment, the Marsden dodgeball game: the level includes 1-2 Marsden balls that appear randomly and fall from the top of the slope in the distance of the game. The user needs to control the little man to move left and right to avoid the falling Marsden balls. The game is considered over when the little man falls off the platform or hits the Marsden ball. The little man gets 1 point each time he successfully passes a Marsden ball. The level is divided into three levels, which are divided into slow, medium and fast according to the falling speed of the Marsden ball. You can pass the game if you reach the lower limit of the required points within the specified time. The initial level has a specified time of 30 seconds and an initial required score of 15 points. After each successful clearance, the specified time increases by 10 seconds and the required score increases by 10 points;
[0092] For the visual function training game in this embodiment, as an example of this embodiment, the maze game can be divided into three difficulty levels, corresponding to different numbers of striped gratings to form maps with increasing difficulty. The first level contains 10 striped gratings, the second level contains 18 striped gratings, and the third level contains 30 striped gratings. The user needs to use the left and right handles to control the first-person perspective to move, and cross the maze path to reach the exit to pass the game.
[0093] Among them, checking the historical training records includes finding the number of times the browsing user uses binocular training, the game selection, duration and score;
[0094] Step 2: When starting the game, turn on the infrared camera 6 to capture the user's head posture and binocular information in real time, predict the coordinate data of the gaze point in real time, generate a virtual occlusion model and overlay it on the computer screen of the dominant eye;
[0095] The coordinate data of the real-time prediction of the gaze point is the prediction result obtained by training a deep learning neural network model with the user's head posture and binocular information as input. , left eye image , right eye image and the position of the face relative to the image Input into the gaze point prediction model and finally get the predicted point , first and Input to Get the output feature vector and ,
[0096] ;
[0097] ;
[0098] Then connect the feature vectors of the left eye and the right eye.
[0099] ;
[0100] And finally processed by the fully connected layer,
[0101] ;
[0102] For the feature extraction of face images, the face images are Input to middle,
[0103] ;
[0104] The same is true for the position of the face relative to the image. , enter it into middle,
[0105] ;
[0106] Integration , and All the data information contained is predicted through the fully connected layer to obtain the standard coordinates of the real-time gaze point ,
[0107] ;
[0108] In the formula, , and The left eye image and right eye image , face image and the position of the face relative to the image The output obtained after passing through the respective network models.
[0109] Furthermore, according to the data projected onto the coordinate axis with the center of the computer screen as the origin according to the algorithm: In view of the differences in the sizes of different computer screens, in order to ensure that the output gaze point coordinate data is always based on the current computer center origin, the following formula is used to achieve the versatility of gaze point tracking:
[0110] ;
[0111] ;
[0112] In the formula, and refers to the actual coordinates of the predicted gaze point on the user's computer screen, and Refers to the standard prediction point coordinates calculated by the deep learning neural network. and It refers to the actual length and width of the user's computer screen. and It refers to the offset in length and width of the user's computer screen specifications. The coordinate origin of the predicted gaze point coordinate data is calibrated to the center of the user's computer screen through the formula.
[0113] The game screen observed by the dominant eye and the amblyopic eye is a pair of scene images that conform to the parallax of the human eye, and there is also a visual inhibition effect of the virtual occlusion model in the content observed by the dominant eye;
[0114] Step 3: At the beginning of the game, Gabor sight marks are randomly generated according to the appropriate game type and training intensity. The user needs to wear red and blue glasses to pass the game level according to the clearance prompts;
[0115] Among them, the appropriate game type and training intensity means that the user meets the doctor's training plan, appropriately weakens the visual information input by the dominant eye at the current training intensity, effectively mobilizes the visual function of the inferior eye, and achieves the effect of binocular collaboration;
[0116] Among them, randomly generating Gabor sight marks is in line with the principles and training requirements of the third-level visual function;
[0117] The user needs to wear left and right red / blue filter glasses 3 and 4 that match the position of the superior and amblyopic eyes;
[0118] Step 4. After completing a game, the game type, playing time and score will be automatically saved in the user information management system for easy review by users and doctors. Users can also choose other game types and training intensities to conduct binocular vision function training again.
[0119] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.
Claims
1. An interactive binocular vision function training system based on line of sight estimation, characterized in that: The system includes an optical system and a processing control system. The optical system refers to a symmetrically arranged left-eye amblyopia optical path and a right-eye amblyopia optical path; the left-eye amblyopia optical path includes a left-side infrared light source, a left-side red / blue filter, a polarizer, and an infrared camera connected in sequence; the right-eye amblyopia optical path includes a right-side infrared light source, a right-side red / blue filter, a polarizer, and an infrared camera connected in sequence; the polarizer that the left and right amblyopia optical paths finally pass through is the same polarizer, and the receiving infrared camera is the same infrared camera; The processing and control system includes a computer display, a binocular vision function training module, a human eye fixation point prediction module and a dominant eye suppression module. The computer display is connected to a left infrared light source, a right infrared light source and an infrared camera. The binocular vision function training module includes a three-level visual function training game for a computer display as a terminal display device; the human eye fixation point prediction module stores the facial posture image acquired by the infrared camera, and performs binocular and facial image segmentation and analysis to obtain the fixation point prediction coordinates, and performs multi-software interaction with the binocular vision function training module; the dominant eye suppression module includes the generation and real-time display of an occlusion model, and the generation of the occlusion model refers to the generation of a virtual occlusion model using a three-dimensional modeling software; the real-time display of the occlusion model refers to the superimposition and display of the virtual occlusion model on the user fixation point coordinate position output by a deep learning neural network based on line of sight estimation, and the superimposition and display is only on a preset dominant eye terminal display device; The human eye gaze point prediction module includes a deep learning neural network. The deep learning neural network is based on a deep learning neural network model based on line of sight estimation, and is improved and trained by changing the data loading method and the coordinate generation method: the user's current gaze information is obtained by calling an infrared camera, and the gaze point coordinate data is predicted in real time; the infrared camera is located at the top of the center of the computer screen in a system with a computer screen as a terminal display device, and the real-time user gaze information obtained by it is: the position of both eyes and the facial posture information, and the prediction of the gaze point is achieved by extracting features from the pictures of the left eye, the right eye and the facial information, and then using the fully connected layer regression to achieve the prediction of the gaze point, and at the same time, the facial posture information is used as the supervision information to recalibrate the eye features. After the improvement, the effect of generating the predicted point coordinates in real time can be achieved by directly calling the camera; By taking the face image , left eye image , right eye image and the position of the face relative to the image Input into the gaze point prediction model and finally get the predicted point , first and Input to Get the output feature vector and ; ; ; Then connect the feature vectors of the left eye and the right eye. ; And finally processed by the fully connected layer, ; For the feature extraction of face images, the face images are Input to middle, ; The same is true for the position of the face relative to the image. , enter it into middle, ; Integration , and All the data information contained is predicted through the fully connected layer to obtain the standard coordinates of the real-time gaze point , ; In the formula, , and The left eye image and right eye image , face image and the position of the face relative to the image The output obtained after passing through the respective network models.
2. The interactive binocular vision function training system based on line of sight estimation according to claim 1, characterized in that: The gaze point coordinate data refers to the real-time coordinate data of the user's gaze point falling on the terminal display device screen, which is the coordinate data in a plane rectangular coordinate system established with the center of the screen as the origin: In view of the differences in the sizes of different computer screens, in order to ensure that the output gaze point coordinate data is always based on the current computer center origin, the following formula is used to achieve the versatility of gaze point tracking: ; ; In the formula, and refers to the actual coordinates of the predicted gaze point on the user's computer screen, and Refers to the standard prediction point coordinates calculated by the deep learning neural network. and It refers to the actual length and width of the user's computer screen. and It refers to the offset in length and width of the user's computer screen specifications. The coordinate origin of the predicted gaze point coordinate data is calibrated to the center of the user's computer screen through the formula.
3. An interactive binocular vision function training system based on line of sight estimation as claimed in claim 1 or 2, characterized in that: In the binocular vision function training module, the third-level vision function training game for the computer screen as the terminal display device is to construct a three-dimensional virtual environment scene and Gabor sight marks through a computer, and to interact with the game in accordance with the principle of controlling the screen perspective with a mouse and controlling the movement of the perspective with a keyboard. During the game, the screen is always displayed from the first perspective or the third perspective, and the game requirements are met or the conditions are met to pass the level. After the game is over, the game duration and score records are saved.
4. The interactive binocular vision function training system based on line of sight estimation according to claim 3, characterized in that: The three-level visual function training game includes a simultaneous vision training game, a fusion vision training game and a stereoscopic vision training game based on a computer screen as a terminal display device, and PC-side game development is performed through Unity 3D software: the card game uses a random arrangement algorithm Gabor sight mark to stimulate the user's cerebral cortex to train the user's vision, and achieves simultaneous vision training by focusing on and distinguishing the orientation and proximity of the Gabor sight mark; the Gabor collision game uses a random arrangement algorithm and a mobile component interaction algorithm to mobilize the cooperation of hands, eyes and brain according to the principle of perceptual training, thereby training the user's ability to simultaneously accept objects and images with both eyes, so as to achieve simultaneous vision training; The animal gathering and dispersing classification game uses a random arrangement algorithm and a mobile component interaction algorithm to perceive the distance range of monocular fusion of the two eyes, mobilize the brain to adjust the two eyes to maintain sensory fusion ability, and train fusion vision; The Marsden dodgeball game uses a random drop algorithm and a rigid body collision detection algorithm to simulate the movement trajectory of the Marsden ball in a variety of real-life scenarios, satisfying the long and short left and right vertical movement space to train the user's depth perception and achieve stereoscopic vision training; The maze game uses the perspective manipulation algorithm and rigid body collision detection algorithm to stimulate the user's cerebral cortex with grating stimulation of different spatial frequencies and long and short distance displacement, train the user's visual function, mobilize the hand, eye, and brain to cooperate through visual perception training to identify and grasp Gabor sight marks, improve the defective pathways of single and binocular visual perception, observe the moving stripe gratings of different depths in space to train depth motion perception, and strengthen the establishment of three-dimensional perception.
5. The interactive binocular vision function training system based on line of sight estimation according to claim 4, characterized in that: The difficulty and type of the simultaneous vision training game, fusion vision training game and stereoscopic vision training game are adjusted according to the user's training plan, and are divided into Gabor sight mark parameter and obstacle parameter adjustment, including but not limited to the number of Gabor sight marks, the direction of Gabor sight marks, the spatial frequency of Gabor sight marks, the proximity of Gabor sight marks, the number of obstacles, the speed of obstacles, the size of obstacles, the contrast clarity of obstacles, and the adjustment of the horizontal and vertical distance parameters of obstacles.
6. An interactive binocular vision function training system based on line of sight estimation as claimed in claim 1 or 2, characterized in that: In the dominant eye suppression module, the virtual occlusion model is designed and constructed specifically according to the user's specific visual function and the formulated training program, and can be adjusted according to the results of user training feedback. The adjustable parameters include the color, shape, material, size, contrast, clarity, frequency of occurrence and speed of the virtual occlusion model.
7. The interactive binocular vision function training system based on line of sight estimation according to claim 6, characterized in that: In the dominant eye suppression module, only the display is superimposed on the preset dominant eye terminal display device. When the terminal display device is a computer screen, red and blue glasses are required. Only red light can pass through the red lens, which can filter out the red content in the picture; only blue light can pass through the blue lens, which can filter out the blue content in the picture. In this way, the left and right eyes see different pictures through the red and blue lenses. By adjusting the color of the virtual occlusion model, the dominant eye position can be preset in advance to achieve the suppression effect of the dominant eye.
8. An interactive binocular vision function training system based on line of sight estimation as claimed in claim 1 or 2, characterized in that: The processing and control system also includes a user information management module, which includes a user information entry system and a user information query system. The user information entry system includes obtaining user personal information, diagnostic data and game training data, and requires the user to register or log in to an account when using the binocular vision function module training to facilitate the storage of information data; the user information query system includes a user viewing the user training effect independently of the binocular vision function module training, and requires the user to register or log in to the same account; The personal information includes the user's age, gender, name and amblyopia; the amblyopia includes binocular naked eye vision, optometry, corrected vision, strabismus, superior / abdominal eye position, stereoscopic vision and intensity parameters of the formulated training program; the game training data is timely saved in the system by saving the training duration, training cycle and game score information in the binocular vision function training module after the third-level visual function game is completed.
