VR Interactive Binocular Vision Training System Based on Gaze Estimation

Through the VR interactive binocular visual function training system based on line of sight estimation, the micro infrared camera and deep learning neural network predict the gaze point, combined with VR games, the problem of imbalance in binocular visual information input in the existing training methods is solved, and the correct Effective training of binocular vision function improves training effect and compliance.

CN118649069BActive Publication Date: 2024-11-12ZHEJIANG UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411133905.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-19
Publication Date
2024-11-12
Estimated Expiration
2044-08-19

AI Technical Summary

Technical Problem

The existing amblyopia training method is large in size and a single training mode, which cannot effectively balance the input of binocular visual information, ignores the normal visual function of the dominant eye, and is not effective for young patients.

Method used

Using a VR interactive binocular vision function training system based on line of sight estimation, combined with optical system and processing control system, the user's gaze point is predicted through a micro infrared camera and a deep learning neural network, and a virtual occlusion model is generated to achieve balanced training of binocular vision function. Develop a variety of VR visual function detection and training games.

Benefits of technology

It improves training effect and adaptability, enhances the balance and collaboration of binocular visual functions, improves training compliance, avoids the suppression of dominant eyes in traditional methods, and promotes efficient processing of the visual cortex and long-lasting vision improvement.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118649069B_ABST
    Figure CN118649069B_ABST
Patent Text Reader

Abstract

A VR interactive binocular vision function training system based on line of sight estimation belongs to the field of visual training technology. The VR interactive binocular vision function training system includes an optical system and a processing control system. The optical system includes a left infrared LED light source, a right infrared LED light source, a left eye micro infrared camera and a right eye micro infrared camera. The left infrared LED light source is aimed at the left eye, and the right infrared LED light source is aimed at the right eye. The processing control system includes a left eye VR display screen, a right eye VR display screen, a binocular vision function training module, a human eye fixation point prediction module and a dominant eye suppression module. The present invention combines real-time line of sight estimation technology to improve the effect and adaptability of user training.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of vision training, and in particular relates to a VR interactive binocular vision function training system based on line of sight estimation. Background Art

[0002] Amblyopia is a common clinical ophthalmological disease caused by abnormal interaction between brain development and visual experience, which leads to neurodevelopmental disorders of the visual system and disordered spatial visual development. Amblyopia has a significant impact on the visual input of the brain, disrupting the normal development of visual function and causing different vision defects.

[0003] Binocular vision is divided into three levels. The first level is simultaneous vision. Establishing normal simultaneous vision can not only balance the inhibition between the superior and inferior eyes, but also establish a solid prerequisite for binocular advanced vision. The second level is fusion vision, which assists in forming a normal retinal correspondence, coordinates binocular movement, and increases the fusion range. The third level is stereoscopic vision, which allows the brain to fuse the images observed by the two eyes into a complete stereoscopic impression to distinguish the distance and shape of the scene. Only when the first level of vision function is perfected can the second level of vision function be established; only when the second level of vision function is perfected can the foundation for the development of the third level of vision function be laid. The three levels of vision are linked together and work in sequence, so that we can observe the world clearly, permanently and comfortably. Normal and perfect binocular vision is indispensable in human daily life, learning and work scenes. However, due to some congenital pathological reasons, improper use of eyes in life, eye fatigue and other conditions, more and more people suffer from varying degrees of amblyopia or even strabismus. Not only do they often feel visual fatigue, eye soreness, eye swelling, difficulty in near vision, drowsiness, etc., but they also lose some visual functions, which greatly affects their normal life. The current visual function training methods mainly use reversal photography, stereoscopic atlases, occlusion therapy, synoptophore training, etc. These training methods are not only monotonous in form and content, easily causing numb memory and weakening the training effect, but also cannot fully mobilize the enthusiasm of patients, especially for young patients. Summary of the invention

[0004] In order to overcome the shortcomings of existing amblyopia training methods, such as large size, single training mode, ignoring the normal visual function of the dominant eye while training the visual function of the amblyopic eye, and ignoring the balance of binocular visual information, the present invention provides a VR interactive binocular vision function training system based on line of sight estimation, which combines real-time line of sight estimation technology to solve the technical problem of how to balance binocular vision information input as an optimized medium for visual function training, thereby improving the effect and adaptability of user training.

[0005] To achieve the above technical objectives, the present invention adopts a VR interactive binocular vision function training system based on line of sight estimation, including an optical system and a processing control system, wherein the optical system includes a left infrared LED light source, a right infrared LED light source, a left eye micro infrared camera, and a right eye micro infrared camera, wherein the left infrared LED light source is aimed at the left eye, and the right infrared LED light source is aimed at the right eye;

[0006] The processing and control system includes a left-eye VR display screen, a right-eye VR display screen, a binocular vision function training module, a human eye gaze point prediction module and a dominant eye suppression module. The left-eye micro infrared camera is located below the outer side of the left-eye VR display screen, and the right-eye micro infrared camera is located below the outer side of the right-eye VR display screen. The binocular vision function training module includes a visual function training game for a VR head-mounted display screen as a terminal display device; the human eye gaze point prediction module stores the facial posture image acquired by the infrared camera, segments and analyzes the binocular and facial images to obtain the gaze point prediction coordinates, and performs multi-software interaction with the binocular vision function training module; the dominant eye suppression module includes the generation and real-time display of an occlusion model, and the generation of the occlusion model refers to generating a virtual occlusion model using a three-dimensional modeling software; the real-time display of the occlusion model refers to superimposing and displaying the virtual occlusion model on the user gaze point coordinate position output by a deep learning neural network based on line of sight estimation, and only superimposing and displaying it on a preset dominant eye terminal display device screen.

[0007] Furthermore, the eye gaze point prediction module includes a deep learning neural network, which is based on a deep learning neural network model for line of sight estimation and is trained by changing the data loading method and coordinate generation method: the user's current gaze information is obtained by calling a miniature infrared camera to predict the coordinate data of the gaze point in real time;

[0008] The two miniature infrared cameras are located below the outer side of binocular lenses in a system with a VR head-mounted display as a terminal display device, and the angle between the two-binocular display screen center horizontal line is 45°. The real-time user gaze information obtained by the miniature infrared cameras is specifically: binocular pupil information, by extracting features from left and right eye images, by directly calling the miniature infrared cameras to achieve the effect of generating prediction point coordinates in real time, and by using fully connected layer regression to achieve gaze point prediction;

[0009] By placing the left eye image , right eye image Input into the gaze point prediction model and finally get the predicted point ,

[0010] First, and Input to Get the output feature vector and ,

[0011] ;

[0012] ;

[0013] Then connect the feature vectors of the left eye and the right eye.

[0014] ;

[0015] And finally processed by the fully connected layer,

[0016] ;

[0017] Integration All the data information contained is predicted through the fully connected layer to obtain the standard coordinates of the real-time gaze point ,

[0018] ;

[0019] In the formula Is the left eye image and right eye image The output obtained after passing through the network model.

[0020] In the binocular vision function training module, the visual function detection game and visual function training game for the VR head-mounted display as the terminal display device are to construct a three-dimensional virtual environment scene, visual function detection pictures, Gabor sight marks and spatial frequency gratings through the VR device, so that the left handle controls the movement of the character and the right handle switches the perspective screen. The first-person perspective display screen is always maintained during the game process, and the game requires scores or conditions to be completed. After the game is over, the game duration and score records are saved.

[0021] Preferably, the visual function detection game includes a simultaneous vision detection game, a fused vision detection game and a stereoscopic vision detection game based on a VR head-mounted display as a terminal display device, and the VR-end detection game is developed through Unity 3D software: the simultaneous vision detection game uses the VR device to achieve a binocular vision effect, and the left and right VR display screens are individually rendered and presented to the user for observation; the fused vision detection game uses a random algorithm to call two images that conform to the parallax of the left and right eyes, and the user observes the image aggregation relationship to judge the range of sensory fusion and motor fusion capabilities; the stereoscopic vision detection game uses a random algorithm to call a random point stereoscopic vision inspection image that conforms to the principle of color difference, and the user judges the stereoscopic vision sharpness value by distinguishing the stereoscopic graphics.

[0022] Preferably, the visual function training game includes a simultaneous vision training game, a fusion vision training game and a stereoscopic vision training game based on a VR head-mounted display as a terminal display device, and VR-side game development is performed through Unity 3D software: the card game uses a random arrangement algorithm Gabor sight mark to stimulate the user's cerebral cortex to train the user's vision, and achieves simultaneous vision training by focusing on and distinguishing the orientation and proximity of the Gabor sight mark; the Gabor collision game uses a random arrangement algorithm and a mobile component interaction algorithm to mobilize the cooperation of hands, eyes and brain based on the principle of perceptual training, thereby training the user's binocular acceptance of objects and images at the same time, so as to achieve simultaneous vision training; the animal gathering and dispersion classification game uses a random arrangement algorithm and a mobile component interaction algorithm to perceive the monocular fusion distance range of the two eyes as the principle, mobilizes the brain to adjust the binoculars to maintain sensory fusion and motor fusion capabilities, so as to train fusion vision; the Marsden dodgeball game uses a random falling algorithm and a rigid body collision detection algorithm to simulate The movement trajectory of the simulated Marsden ball in a variety of real-life scenarios satisfies the long and short distances and left and right longitudinal movement space, thereby training the user's perception of depth and achieving stereoscopic vision training; the step-by-step climbing game uses the principles of physical mechanics and computer graphics to simulate the stair space with different step heights to imitate the different stair paths in real life scenes, thereby training the user's perception of depth in the horizontal and vertical directions and achieving stereoscopic vision training; the maze game uses the perspective manipulation algorithm and the rigid body collision detection algorithm to stimulate the user's cerebral cortex with gratings of different spatial frequencies to train the user's visual function, and through visual perception training, mobilize the hand, eye, and brain to collaborate to identify and grasp Gabor visual marks, improve the defective pathways of single and binocular visual perception, observe the moving stripe gratings of different depths in the space, train the depth motion perception, and strengthen the establishment of stereoscopic perception.

[0023] The image types of the simultaneous vision detection game, fused vision detection game and stereoscopic vision detection game can be adjusted according to the user's training plan, including the image size, contrast, clarity, hue and image types corresponding to the stereoscopic vision acuity value.

[0024] The difficulty and type of the simultaneous vision training game, fusion vision training game and stereoscopic vision training game can be adjusted according to the user's training plan, and are divided into Gabor sight mark parameter and obstacle parameter adjustment, including the number of Gabor sight marks, the direction of Gabor sight marks, the spatial frequency of Gabor sight marks, the proximity of Gabor sight marks, the number of obstacles, the speed of obstacles, the size of obstacles, the contrast clarity of obstacles and the horizontal and vertical distance parameters of obstacles.

[0025] In the dominant eye suppression module, the virtual occlusion model is designed and constructed specifically according to the user's specific visual function and the formulated training program, and can be adjusted according to the results of user training feedback. The adjustable parameters include the color, shape, material, size, contrast, clarity, frequency of occurrence and speed of the virtual occlusion model.

[0026] Furthermore, the method of only superimposing and displaying on the preset dominant eye terminal display device screen is used to develop a VR game for a visual function training game in a system with a VR head-mounted display as a terminal display device, and a visual function training game scene that conforms to binocular parallax is designed and established, and the left and right display screens of the VR are rendered respectively. A virtual occlusion model is generated through three-dimensional modeling and a single rendering is performed on the display screen of the preset dominant eye, thereby achieving a suppression effect on the dominant eye.

[0027] Preferably, the angle between the left-eye micro-infrared camera and the center horizontal line of the left-eye VR display screen is 45°, and the angle between the right-eye micro-infrared camera and the center horizontal line of the right-eye VR display screen is 45°.

[0028] The processing and control system also includes a user information management module, which includes a user information entry system and a user information query system. The user information entry system includes obtaining user personal information, diagnostic data and game training data, and requires the user to register or log in to an account when using the binocular vision function module training to facilitate the storage of information data; the user information query system includes a user viewing the user training effect independently of the binocular vision function module training, and requires the user to register or log in to the same account;

[0029] The personal information includes the user's age, gender, name, and amblyopia; the amblyopia includes binocular naked eye vision, optometry, corrected vision, strabismus (degree), superior / abdominal eye position, stereoscopic vision, and intensity parameters of the formulated training program; the game training data is timely saved in the system through the completion of the third-level visual function game in the binocular vision function training module, including training duration, training cycle, and game score information.

[0030] The technical concept of the present invention is that Gabor sight marks are a kind of neural vision or perceptual vision training based on visual stimulation and promotion of neural connections responsible for vision. In the contour perception task, because Gabor sight marks have no edge prompts, contrast, size, spatial frequency, orientation and spatial position can all be used and processed, and are regarded as suitable stimulus elements, which are conducive to the mastery of spatial integration ability of the amblyopic eye and play a pivotal role in perceptual training. At the same time, orientation information is an important attribute of visual stimulation signals. In perceptual learning research, orientation discrimination is usually represented by the ability to resolve the orientation difference of sinusoidal bars. Perceptual learning can significantly improve the user's orientation discrimination ability. Therefore, it is very necessary to use the system of the present invention to train the visual function of the patient's amblyopic eye and balance the binocular information reception and image processing fusion ability. The present invention proposes a new system, which overcomes the problems existing in the above-mentioned existing technologies. Different from the traditional monocular repair training, it directly trains the binocular vision function, which can not only achieve good training effects, but also help to improve the binocular vision function and maintain the training effect.

[0031] Gaze estimation is the process of predicting a person's gaze direction or gaze point position through the state of the eyes and the posture of the head. It has important value in the fields of neuroscience, psychology, computer science, etc. The goal of gaze estimation is to derive the gaze direction from a face image or an eye image. Although gaze estimation technology is now used in many fields, including game interaction, virtual reality, medical treatment, assisted driving, and web page analysis, there is no application of gaze estimation in the field of visual function training. Based on the important features of gaze estimation in capturing and visualizing people's focus points, the present invention proposes for the first time an interactive binocular visual function training system based on gaze estimation. In VR, the gaze tracking algorithm can be used to render the local area of ​​human eye focus with high precision, thereby improving the compliance of visual function training, saving energy consumption and reducing hardware costs, and also helping doctors to better understand the patient's training situation.

[0032] The beneficial effects of the present invention are mainly manifested in:

[0033] 1) Innovation in the cross-integration of optics and ophthalmology: This invention is based on the principle of amblyopia training, integrated with multimedia technology and virtual reality technology, and cleverly and efficiently combines optics and ophthalmology according to medical training requirements and basic optical knowledge. It independently designs a new binocular training game that meets the third-level visual function, and invents a training system for vision training.

[0034] 2) Balance binocular visual information input: Compared with traditional visual function training methods, the present invention avoids the suppression of the dominant eye's visual function in complete occlusion therapy, cultivates the user's binocular cooperation habits, and not only completes targeted single and binocular visual function training for the inferior eye, but also protects the normal visual function of the dominant eye during the training process, creates special conditions for the superior and inferior eyes to work together, stimulates specific neurons in the visual cortex and trains them to process image information more efficiently, thereby gradually forming new neural connections and achieving lasting vision improvement.

[0035] 3) High training compliance, simple and convenient operation: The visual function training games developed by the present invention, which combine the fun of games with the rigor of amblyopia training, are of various types, with rich training content, in line with children's psychological characteristics, and improve user training compliance. It is easy to use, and you can start binocular vision function training at home, making it easier to persist in training for a long time. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a schematic diagram of the structure of a miniature infrared camera for VR interactive binocular vision function training based on line of sight estimation;

[0037] Figure 2 This is a schematic diagram of the technical route of the VR interactive binocular vision function training system based on line of sight estimation;

[0038] Figure 3 It is a flow chart based on VR head mounted display as the terminal display device.

[0039] Serial numbers and names of the accompanying drawings: 1 left eye VR display screen, 2 right eye VR display screen, 3 left infrared LED light source, 4 right infrared LED light source, 5 left eye micro infrared camera, 6 right eye micro infrared camera. DETAILED DESCRIPTION

[0040] The present invention will be further described below in conjunction with the accompanying drawings.

[0041] Reference Figure 1~Figure 3 , a VR interactive binocular vision function training device based on line of sight estimation, comprising an optical system and a processing control system, wherein the optical system comprises a left infrared LED light source 3, a right infrared LED light source 4, a left eye micro infrared camera 5 and a right eye micro infrared camera 6, wherein the left infrared LED light source 3 is aimed at the left eye, and the right infrared LED light source 4 is aimed at the right eye;

[0042] The processing and control system comprises a left-eye VR display screen 1, a right-eye VR display screen 2, a binocular vision function training module, a human eye gaze point prediction module and a dominant eye suppression module. The left-eye micro infrared camera 5 is located below the outer side of the left-eye VR display screen 1, and the angle with the central horizontal line of the left-eye VR display screen 1 is 45°; the right-eye micro infrared camera 6 is located below the outer side of the right-eye VR display screen 2, and the angle with the central horizontal line of the right-eye VR display screen 2 is 45°. The binocular vision function training module comprises a visual function training game for a VR head-mounted display screen as a terminal display device; the human eye gaze point prediction module stores the facial posture image acquired by the infrared camera, and performs binocular and facial image segmentation and analysis to obtain the gaze point prediction coordinates, and performs multi-software interaction with the binocular vision function training module at the same time; the dominant eye suppression module comprises the generation and real-time display of an occlusion model, and the generation of the occlusion model refers to the generation of a specific virtual occlusion model by using a three-dimensional modeling software; the real-time display of the occlusion model refers to the superimposition and display of the virtual occlusion model on the user gaze point coordinate position output by the deep learning neural network based on line of sight estimation, and the superimposition and display is only on the preset dominant eye terminal display device screen.

[0043] Furthermore, the eye gaze point prediction module includes a deep learning neural network, which is based on a deep learning neural network model for line of sight estimation and is trained by changing the data loading method and coordinate generation method: the user's current gaze information is obtained by calling a miniature infrared camera to predict the coordinate data of the gaze point in real time;

[0044] The two miniature infrared cameras are located below the outer side of binocular lenses in a system with a VR head-mounted display as a terminal display device, and the angle between the two-binocular display screen center horizontal line is 45°. The real-time user gaze information obtained by the miniature infrared cameras is specifically: binocular pupil information, and feature extraction is performed on left and right eye images. After improvement, the miniature infrared cameras can be directly called to achieve the effect of generating prediction point coordinates in real time, and the prediction of the gaze point is realized by using the fully connected layer regression;

[0045] By placing the left eye image , right eye image Input into the gaze point prediction model and finally get the predicted point ,

[0046] First, and Input to Get the output feature vector and ,

[0047] ;

[0048] ;

[0049] Then connect the feature vectors of the left eye and the right eye.

[0050] ;

[0051] And finally processed by the fully connected layer,

[0052] ;

[0053] Integration All the data information contained is predicted through the fully connected layer to obtain the standard coordinates of the real-time gaze point ,

[0054] ;

[0055] In the formula Is the left eye image and right eye image The output obtained after passing through the network model.

[0056] In the binocular vision function training module, the visual function detection game and visual function training game for the VR head-mounted display as the terminal display device are constructed by VR devices to construct three-dimensional virtual environment scenes, visual function detection pictures, Gabor sight marks and spatial frequency gratings, and the left handle controls the movement of the character and the right handle switches the perspective screen. The game process always maintains the first-person perspective display screen, completes the game requirements to score or completes the clearance under conditions, and saves the game duration and score records after the game ends;

[0057] Preferably, the visual function detection game includes a simultaneous vision detection game, a fused vision detection game and a stereoscopic vision detection game based on a VR head-mounted display as a terminal display device, and the VR-end detection game is developed through Unity 3D software: the simultaneous vision detection game uses the VR device to achieve a binocular vision effect, and the left and right VR display screens are individually rendered and presented to the user for observation; the fused vision detection game uses a random algorithm to call two images that conform to the parallax of the left and right eyes, and the user observes the image aggregation relationship to judge the range of sensory fusion and motor fusion capabilities; the stereoscopic vision detection game uses a random algorithm to call a random point stereoscopic vision inspection image that conforms to the principle of color difference, and the user judges the stereoscopic vision sharpness value by distinguishing the stereoscopic graphics.

[0058] Preferably, the visual function training game includes a simultaneous vision training game, a fusion vision training game and a stereoscopic vision training game based on a VR head-mounted display as a terminal display device, and VR-side game development is performed through Unity 3D software: the card game uses a random arrangement algorithm Gabor sight mark to stimulate the user's cerebral cortex to train the user's vision, and achieves simultaneous vision training by focusing on and distinguishing the orientation and proximity of the Gabor sight mark; the Gabor collision game uses a random arrangement algorithm and a mobile component interaction algorithm to mobilize the cooperation of hands, eyes and brain based on the principle of perceptual training, thereby training the user's binocular acceptance of objects and images at the same time, so as to achieve simultaneous vision training; the animal gathering and dispersion classification game uses a random arrangement algorithm and a mobile component interaction algorithm to perceive the monocular fusion distance range of the two eyes as the principle, mobilizes the brain to adjust the binoculars to maintain sensory fusion and motor fusion capabilities, so as to train fusion vision; the Marsden dodgeball game uses a random falling algorithm and a rigid body collision detection algorithm to simulate The movement trajectory of the simulated Marsden ball in a variety of real-life scenarios satisfies the long and short distances and left and right longitudinal movement space, thereby training the user's perception of depth and achieving stereoscopic vision training; the step-by-step climbing game uses the principles of physical mechanics and computer graphics to simulate the stair space with different step heights to imitate the different stair paths in real life scenes, thereby training the user's perception of depth in the horizontal and vertical directions and achieving stereoscopic vision training; the maze game uses the perspective manipulation algorithm and the rigid body collision detection algorithm to stimulate the user's cerebral cortex with gratings of different spatial frequencies to train the user's visual function, and through visual perception training, mobilize the hand, eye, and brain to collaborate to identify and grasp Gabor visual marks, improve the defective pathways of single and binocular visual perception, observe the moving stripe gratings of different depths in the space, train the depth motion perception, and strengthen the establishment of stereoscopic perception.

[0059] The image types of the simultaneous vision detection game, fused vision detection game and stereoscopic vision detection game can be adjusted according to the user's training plan, including the image size, contrast, clarity, hue and image types corresponding to the stereoscopic vision acuity value.

[0060] The difficulty and type of the simultaneous vision training game, fusion vision training game and stereoscopic vision training game can be adjusted according to the user's training plan, which is divided into Gabor sight mark parameter and obstacle parameter adjustment, including but not limited to the number of Gabor sight marks, the direction of Gabor sight marks, the spatial frequency of Gabor sight marks, the proximity of Gabor sight marks, the number of obstacles, the speed of obstacles, the size of obstacles, the contrast clarity of obstacles, the lateral and longitudinal distances of obstacles and other parameters.

[0061] In the dominant eye suppression module, the virtual occlusion model is designed and constructed in a targeted manner according to the user's specific visual function and the formulated training program, and can be adjusted according to the results of user training feedback. The specific adjustable parameters include but are not limited to the color, shape, material, size, contrast, clarity, frequency of occurrence and speed of the virtual occlusion model;

[0062] Furthermore, the method of only superimposing and displaying on the preset dominant eye terminal display device screen is used to develop a VR game for a visual function training game in a system with a VR head-mounted display as a terminal display device, and a visual function training game scene that conforms to binocular parallax is designed and established, and the left and right display screens of the VR are rendered respectively. A specific virtual occlusion model is generated through three-dimensional modeling, and a single rendering is performed on the specific display screen of the preset dominant eye, thereby achieving a suppression effect on the dominant eye.

[0063] The processing and control system also includes a user information management module, which includes a user information entry system and a user information query system. The user information entry system includes obtaining user personal information, diagnostic data and game training data, and requires the user to register or log in to an account when using the binocular vision function module training to facilitate the storage of information data; the user information query system includes a user or doctor viewing the user training effect independently of the binocular vision function module training, and requires the user to register or log in to the same account;

[0064] The personal information includes the user's age, gender, name, and amblyopia; the amblyopia includes binocular naked eye vision, optometry, corrected vision, strabismus (degree), superior / abdominal eye position, stereoscopic vision, and intensity parameters of the formulated training program; the game training data is timely saved in the system through the completion of the third-level visual function game in the binocular vision function training module, including training duration, training cycle, and game score information.

[0065] This embodiment uses a flow chart based on a VR head mounted display as a terminal display device, as shown in FIG. Figure 3 As shown, the following steps are included:

[0066] Step 1: The user wears the VR device of the present invention, with both eyes in a natural gaze state, and the left infrared LED light source 3 and the right infrared LED light source 4 illuminate the human eyes for supplementary light. The user registers an account, logs in with a valid account and password to enter the homepage of the user information management system. On this page, user information and diagnostic data can be uploaded, visual function detection games, visual function training games can be selected, and historical training records can be checked;

[0067] Among them, the selection of visual function testing games includes simultaneous vision testing games, fusion vision testing games and stereoscopic vision testing games, and the difficulty setting of each test can be adjusted;

[0068] For the simultaneous vision detection game in this embodiment, as an example of this embodiment, the simultaneous vision detection game uses the VR device to render the left and right VR display screens alternately and separately, so that the simultaneous vision detection visual mark images are displayed in front of the user's left and right eyes in turn, to achieve binocular dichoptic effect, and the user observes the displayed image and selects the only matching correct answer from 4 options. The visual conditions of the simultaneous vision function are distinguished by changing the complexity of the simultaneous vision detection visual mark image. The initial difficulty stipulates a single Gabor visual mark and a single direction, and each subsequent difficulty is based on increasing the number of Gabor visual marks and the orientation transformation as the transformation basis;

[0069] For the fusion vision detection game in this embodiment, as an example of this embodiment, the fusion vision detection game: using the VR device to randomly call a group of images in the image set that meets the left and right eye parallax as the current detection visual mark, the user moves the image by manipulating the handle to observe the image aggregation relationship, and judges the range of sensory fusion and motor fusion capabilities;

[0070] For the stereoscopic vision detection game in this embodiment, as an example of this embodiment, the stereoscopic vision detection game uses the VR device to randomly call a random point stereoscopic vision inspection map that conforms to the principle of color difference, and renders a blue filter material for the VR left eye and a red filter material for the VR right eye to simulate a real stereoscopic vision detection scene. The user identifies the stereoscopic graphics to determine the corresponding stereoscopic vision sharpness value;

[0071] Among them, the selection of visual function training games includes simultaneous vision training games, fusion vision training games and stereoscopic vision training games, and the difficulty setting of each game can be adjusted;

[0072] For the simultaneous vision training game in this embodiment, as an example of this embodiment, the card game: three difficulty levels can be set, corresponding to Gabor sight marks of different numbers and orientations. The first level contains a single Gabor sight mark corresponding to a single orientation and a single adjacency, which is used to stimulate the user's cerebral cortex to simply distinguish the Gabor sight marks. The second level contains 12 Gabor sight marks corresponding to 6 groups of orientations and single adjacency. The third level contains 36 Gabor sight marks corresponding to 18 groups of orientations and multiple adjacencies. The level requires the user to click on the cards to memorize the content of the cards. Each click is limited to two cards. After 3 seconds, the card information is initialized to the initial screen. When exactly the same Gabor sight mark appears on the two cards clicked once, the pairing is successful. When all cards are successfully paired, the level can be passed;

[0073] For the simultaneous vision training game in the present embodiment, as an example of the present embodiment, the Gabor matching game: the level contains 63 randomly appearing Gabor sight marks in different directions and multiple adjacent Gabor sight marks are randomly arranged. The user needs to move the horizontal or vertical Gabor sight marks so that three or more completely identical Gabor sight marks are adjacent in the horizontal and vertical directions to eliminate these Gabor sight marks, and each Gabor sight mark will get 10 points. A new Gabor sight mark will be randomly generated at the top of the vertical column where the eliminated Gabor sight mark is located to move downward. The game can be passed if the required lower limit of points is reached within the specified time. The initial specified time is 20 seconds, and the initial required points are 100 points. After each successful clearance, the specified time increases by 10 seconds, and the required points increase by 80 points;

[0074] For the fusion vision training game in this embodiment, as an example of this embodiment, the animal gathering and dispersing classification game: the level contains 10-30 randomly appearing animal images and 1-10 cages of different shapes. The animal images and cages are all graphics with red and blue color difference principle after special image processing. As the number of cages increases and the volume of cages decreases, the difficulty of the game increases. The animal images corresponding to cages of different shapes in each level are different. According to the prompt information of the level, the randomly generated animal images are classified into the cages to which they belong. The game can be passed by completing the classification of all animal images within the specified time. The initial specified time is 90 seconds, and the specified time is reduced by 15 seconds after each successful clearance.

[0075] For the stereoscopic vision training game in this embodiment, as an example of this embodiment, the Marsden dodgeball game: the level includes 1-2 Marsden balls that appear randomly and fall from the top of the slope in the distance of the game. The user needs to control the little man to move left and right to avoid the falling Marsden balls. The game is considered over when the little man falls off the platform or hits the Marsden ball. The little man gets 1 point each time he successfully passes a Marsden ball. The level is divided into three levels, which are divided into slow, medium and fast according to the falling speed of the Marsden ball. You can pass the game if you reach the lower limit of the required points within the specified time. The initial level has a specified time of 30 seconds and an initial required score of 15 points. After each successful clearance, the specified time increases by 10 seconds and the required score increases by 10 points;

[0076] For the stereoscopic vision training game of this embodiment, as an example of this embodiment, the step-by-step climbing game can be divided into three difficulty levels, which correspond to simulated staircase spaces with different step heights to imitate different staircase paths in real life scenes. The user needs to manipulate the handle to select the correct step position to climb the stairs to reach the top of the building to pass the game. Passing a staircase will get 1 point. The initial level has a prescribed time of 90 seconds and a required score of 20 points. After each successful pass, the prescribed time is reduced by 10 seconds and the required score is increased by 5 points.

[0077] For the visual function training game in this embodiment, as an example of this embodiment, the maze game can be divided into three difficulty levels, corresponding to different numbers of striped gratings to form maps with increasing difficulty. The first level contains 10 striped gratings, the second level contains 18 striped gratings, and the third level contains 30 striped gratings. The user needs to use the left and right handles to control the first-person perspective to move, and cross the maze path to reach the exit to pass the game.

[0078] Step 2: When starting the game, turn on the left-eye micro infrared camera 5 and the right-eye micro infrared camera 6 on the VR device to capture the pupil information of both eyes of the user in real time, predict the coordinate data of the gaze point in real time, and generate a virtual occlusion model to be superimposed and displayed on the VR single-side display screen of the dominant eye;

[0079] The left-eye micro-infrared camera 5 and the right-eye micro-infrared camera 6 are a pair of identical micro-infrared cameras placed below the outer sides of the left-eye VR display screen 1 and the right-eye VR display screen 2;

[0080] The coordinate data of the real-time predicted gaze point is the prediction result obtained by training a deep learning neural network model using the user's binocular pupil information as input, and is projected onto a coordinate axis with the visual center of the VR head-mounted display screen as the origin according to the algorithm;

[0081] By placing the left eye image , right eye image Input into the gaze point prediction model and finally get the predicted point ,

[0082] First, and Input to Get the output feature vector and ,

[0083] ;

[0084] ;

[0085] Then connect the feature vectors of the left eye and the right eye.

[0086] ;

[0087] And finally processed by the fully connected layer,

[0088] ;

[0089] Integration All the data information contained is predicted through the fully connected layer to obtain the standard coordinates of the real-time gaze point ,

[0090] ;

[0091] In the formula Is the left eye image and right eye image The output obtained after passing through the network model;

[0092] Step 3: At the beginning of the game, Gabor sight marks are randomly generated according to the appropriate game type and training intensity. The user needs to wear a VR head-mounted display device and pass the game level according to the clearance prompts;

[0093] Step 4. After completing a game, the game type, playing time and score will be automatically saved in the user information management system for easy review by users and doctors. Users can also choose other game types and training intensities to conduct binocular vision function training again.

[0094] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A VR interactive binocular vision function training system based on line of sight estimation, characterized in that: The system comprises an optical system and a processing control system, wherein the optical system comprises a left infrared LED light source, a right infrared LED light source, a left-eye micro infrared camera and a right-eye micro infrared camera, wherein the left infrared LED light source is aimed at the left eye, and the right infrared LED light source is aimed at the right eye; The processing and control system includes a left-eye VR display screen, a right-eye VR display screen, a binocular vision function training module, a human eye gaze point prediction module, and a dominant eye suppression module. The left-eye micro infrared camera is located below the outer side of the left-eye VR display screen, and the right-eye micro infrared camera is located below the outer side of the right-eye VR display screen. The binocular vision function training module includes a visual function training game for a VR head-mounted display screen as a terminal display device; The eye gaze point prediction module stores the facial posture image acquired by the infrared camera, segments and analyzes the binocular and facial images to obtain the predicted coordinates of the gaze point, and performs multi-software interaction with the binocular vision function training module; the dominant eye suppression module includes the generation and real-time display of the occlusion model, and the generation of the occlusion model refers to the generation of a virtual occlusion model using a three-dimensional modeling software; the real-time display of the occlusion model refers to the superimposition and display of the virtual occlusion model on the user gaze point coordinate position output by the deep learning neural network based on line of sight estimation, and the superimposition and display is only on the preset dominant eye terminal display device screen; The human eye gaze point prediction module includes a deep learning neural network. The deep learning neural network is based on a deep learning neural network model based on line of sight estimation, and is trained by changing the data loading method and the coordinate generation method: the user's current gaze information is obtained by calling a micro infrared camera, and the coordinate data of the gaze point is predicted in real time; The two miniature infrared cameras are located below the outer side of binocular lenses in a system with a VR head-mounted display as a terminal display device, and the angle between the two-binocular display screen center horizontal line is 45°. The real-time user gaze information obtained by the miniature infrared cameras is specifically: binocular pupil information, by extracting features from left and right eye images, by directly calling the miniature infrared cameras to achieve the effect of generating prediction point coordinates in real time, and by using fully connected layer regression to achieve gaze point prediction; By placing the left eye image , right eye image Input into the gaze point prediction model and finally get the predicted point , First, and Input to Get the output feature vector and , ; ; Then connect the feature vectors of the left eye and the right eye. ; And finally processed by the fully connected layer, ; Integration All the data information contained is predicted through the fully connected layer to obtain the standard coordinates of the real-time gaze point , ; In the formula Is the left eye image and right eye image The output obtained after passing through the network model.

2. The VR interactive binocular vision function training system based on line of sight estimation according to claim 1, characterized in that: In the binocular vision function training module, the visual function detection game and visual function training game for the VR head-mounted display as the terminal display device are to construct a three-dimensional virtual environment scene, visual function detection pictures, Gabor sight marks and spatial frequency gratings through the VR device, so that the left handle controls the movement of the character and the right handle switches the perspective screen. The first-person perspective display screen is always maintained during the game process, and the game requires scores or conditions to be completed. After the game is over, the game duration and score records are saved.

3. The VR interactive binocular vision function training system based on line of sight estimation according to claim 2, characterized in that: The visual function detection game includes a simultaneous vision detection game, a fused vision detection game and a stereoscopic vision detection game based on a VR head-mounted display as a terminal display device, and the VR-side detection game is developed through Unity 3D software: the simultaneous vision detection game uses VR equipment to achieve binocular dichoptic effect, and renders the left and right VR display screens separately and alternately for the user to observe; the fused vision detection game uses a random algorithm to call two images that conform to the parallax of the left and right eyes, and the user observes the image aggregation relationship to judge the range of sensory fusion and motor fusion capabilities; the stereoscopic vision detection game uses a random algorithm to call a random point stereoscopic vision inspection image that conforms to the principle of color difference, and the user judges the stereoscopic vision sharpness value by identifying the stereoscopic graphics; The visual function training games include simultaneous vision training games, fused vision training games and stereoscopic vision training games based on VR head-mounted displays as terminal display devices, and VR-side game development is performed through Unity 3D software.

4. The VR interactive binocular vision function training system based on line of sight estimation according to claim 3, characterized in that: The image types of the simultaneous vision detection game, fusion vision detection game and stereoscopic vision detection game are adjusted according to the user's training plan, including the image size, contrast, clarity, hue and image type corresponding to the stereoscopic acuity value; The difficulty and type of the simultaneous vision training game, fusion vision training game and stereoscopic vision training game are adjusted according to the user's training plan, and are divided into Gabor sight mark parameter and obstacle parameter adjustment, including the number of Gabor sight marks, the direction of Gabor sight marks, the spatial frequency of Gabor sight marks, the proximity of Gabor sight marks, the number of obstacles, the speed of obstacles, the size of obstacles, the contrast clarity of obstacles and the horizontal and vertical distance parameters of obstacles.

5. The VR interactive binocular vision function training system based on line of sight estimation according to claim 1, characterized in that: In the dominant eye suppression module, the virtual occlusion model is designed and constructed specifically according to the user's specific visual function and the formulated training program, and can be adjusted according to the results of user training feedback. The adjustable parameters include the color, shape, material, size, contrast, clarity, frequency of occurrence and speed of the virtual occlusion model.

6. The VR interactive binocular vision function training system based on line of sight estimation according to claim 5, characterized in that: The method of only superimposing and displaying on the preset dominant eye terminal display device screen is used to develop a VR game for a visual function training game in a system with a VR head-mounted display as a terminal display device, design and establish a visual function training game scene that conforms to binocular parallax, render the left and right display screens of the VR respectively, generate a virtual occlusion model through three-dimensional modeling, and perform a single rendering on the display screen of the preset dominant eye, thereby achieving an inhibitory effect on the dominant eye.

7. The VR interactive binocular vision function training system based on line of sight estimation according to claim 1, characterized in that: The processing and control system also includes a user information management module, which includes a user information entry system and a user information query system. The user information entry system includes obtaining user personal information, diagnostic data and game training data, and requires the user to register or log in to an account when using the binocular vision function module training to facilitate the storage of information data; the user information query system includes a user viewing the user training effect independently of the binocular vision function module training, and requires the user to register or log in to the same account; The personal information includes the user's age, gender, name, and amblyopia; the amblyopia includes binocular naked eye vision, optometry, corrected vision, strabismus, superior / abdominal eye position, stereoscopic vision, and intensity parameters of the formulated training program; the game training data is timely saved in the system by saving the training duration, training cycle, and game score information after the third-level visual function game in the binocular vision function training module is completed.