Binocular display image optimization method and system for VR visual training

Abstract

The invention relates to the technical field of VR visual training, and provides a binocular display image optimization method and system for VR visual training, and the method comprises the steps: collecting the coordinates of fixation points of a right eye and a left eye of a VR head-mounted display, and determining a binocular fixation point difference vector; calculating a pixel compensation vector, determining a compensation eye, and performing offset adjustment on a picture seen by the compensation eye of the VR head-mounted display at the second acquisition moment; and determining a compensation gradient, calculating a pixel gradient compensation vector of a next adjacent acquisition moment of the acquisition moment by combining the pixel compensation vector, and respectively carrying out offset adjustment on images watched by the compensation eyes at the corresponding acquisition moments according to the pixel gradient compensation vectors of all the acquisition moments starting from the third acquisition moment. According to the invention, the quality and the matching degree of the optimized image can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of VR visual training, in particular to a binocular display image optimization method and system for VR visual training. BACKGROUND

[0002] VR visual training is a visual function training method based on virtual reality technology, which can improve the visual ability of patients with amblyopia, strabismus and abnormal visual function in game interaction. In order to make the user perceive the stereoscopic virtual environment, the images presented to the left eye and the right eye in the VR head-mounted display of VR visual training are slightly different, and the images seen by the two eyes are the visual information received when observing the object at the corresponding angle respectively. Therefore, the fusion quality and matching degree of the images seen by the two eyes can directly affect the accuracy and stability of stereoscopic vision, as well as the training effect and visual comfort.

[0003] Generally, the images of the two eyes in the VR head-mounted display are optimized using the offline calibration idea. Specifically, at the time of factory shipment, a camera is used to shoot a lens distortion map, and the distortion grids of the left and right eyes are generated at one time. The vertex displacement or texture resampling of each frame of original image is performed according to the distortion grid, and then the left and right images are respectively pre-distorted to the head-mounted screen according to the fixed IPD (Interpupillary Distance) pupillary distance. However, this optimization method can only compensate for the lens distortion with a fixed pupillary distance, without considering the real-time change of the user's eyeball position. When the user's eye position deviates slightly or drifts due to fatigue during the training process, if the original grid is still used for projection, it will cause unpredictable horizontal or vertical deviation of the pixel landing points of the left and right eyes, directly inducing diplopia and misplacement, and causing visual fatigue of the user. SUMMARY

[0004] The present application provides a binocular display image optimization method and system for VR visual training to solve the problem that the real-time drift of the user's eye position causes dynamic misplacement of the pixel landing points of the left and right eyes during the binocular display image optimization process of VR visual training, resulting in insufficient image quality and matching degree after optimization. The technical solution adopted is as follows: In a first aspect, an embodiment of the present application provides a binocular display image optimization method for VR visual training, which comprises the following steps: Collecting the gaze point coordinates of the right eye and the left eye of the VR head-mounted display at each collection time, and determining the binocular gaze point difference vector at each collection time according to the gaze point coordinates; According to the binocular gaze point difference vectors of adjacent collection times, the pixel compensation vectors at each collection time are calculated respectively, the compensation eye at the collection time is determined according to the gaze point coordinates of the two eyes at the collection time, and the screen seen by the compensation eye of the VR head-mounted display at the second collection time is adjusted by offsetting in combination with the pixel compensation vector at the second collection time; The pixel gradient compensation vector of the next adjacent collection time point of the collection time point is calculated by combining the pixel compensation vector of the collection time point and the compensation gradient of the collection time point. The picture seen by the compensation eye at the corresponding collection time point is adjusted by offset according to the pixel gradient compensation vectors of all collection time points starting from the third collection time point.

[0005] Further, the binocular fixation point difference vector of the collection time point is the difference between the fixation point coordinates of the right eye and the left eye at the same collection time point.

[0006] Further, the calculation formula of the pixel compensation vector of the collection time point is: wherein, represents the pixel compensation vector of the i-th collection time point; represents a clipping function; represents a preset maximum allowed compensation amount; represents the gain scaling factor of the i-th collection time point; represents the binocular fixation point difference vector of the i-th collection time point. Further, the calculation formula of the gain scaling factor of the collection time point is:

[0007] represents the binocular fixation point difference vector of the i-th collection time point; represents a natural constant with a natural constant as a base number; represents a modulus of

[0008] Further, the specific determination method of the compensation eye of the collection time point is: Any one collection time point is recorded as a target collection time point. The offset distances of the left eye and the right eye at the target collection time point are determined respectively according to the positions of the fixation point coordinates of the left eye and the right eye at the target collection time point. It is judged whether the eye corresponding to the maximum value of the offset distance of the target collection time point is the dominant eye. If yes, the dominant eye is taken as the compensation eye of the target collection time point. If not, the other eye is taken as the compensation eye of the target collection time point.

[0009] Further, the specific determination method of the offset distances of the left eye and the right eye at the target collection time point is: The Euclidean distance between the fixation point coordinates of the left eye at the target collection time point and the center point of the image seen by the left eye in the VR head-mounted display is recorded as the offset distance of the left eye at the target collection time point. ​​​​​The Euclidean distance between the gaze point coordinate of the right eye at the target acquisition time and the center point of the image seen by the left eye in the VR headset is denoted as the offset distance of the right eye at the target acquisition time.

[0010] Further, the pixel compensation vector at the second acquisition time is used to adjust the image seen by the compensated eye of the VR headset at the second acquisition time, and the specific method comprises: The pixel compensation vector at the second acquisition time is used as the pixel-level offset to adjust the image seen by the compensated eye of the VR headset at the second acquisition time.

[0011] Further, the compensation gradient at the acquisition time is a gradient of a cost function with respect to the pixel compensation vector at the acquisition time estimated by using a numerical method.

[0012] Further, the calculation formula of the pixel gradient compensation vector at the next adjacent acquisition time of the acquisition time is: wherein, represents the pixel gradient compensation vector at the first acquisition time; represents the pixel gradient compensation vector at the first acquisition time; represents the compensation gradient at the first acquisition time.

[0013] In a second aspect, the embodiments of the present application also provide a VR visual training binocular display image optimization system, which comprises a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor implements the steps of the method according to any one of the above aspects when executing the computer program.

[0014] The present application has the following beneficial effects: ​The application first determines a binocular fixation point difference vector of the collection moment according to the fixation point coordinate difference of the right eye and the left eye of the VR head-mounted display at the same collection moment, and the binocular fixation point difference vector can represent the difference between the landing points of the left and right eyes; the dominant eye is determined to avoid the active modification of the dominant eye image causing visual discomfort, and then for the second collection moment, the misalignment caused by the eye position drift or device sliding in the image in the VR head-mounted display at the next adjacent collection moment is compensated in pixel units according to the binocular fixation point difference vector of the second collection moment, the monocular image is gradually modified, the brain has enough time to re-fuse the double images, the discomfort caused by the picture shaking is avoided, and the images of the right eye and the left eye in the VR head-mounted display are re-aligned on the retina of the user; further, for all collection moments starting from the third collection moment, the compensation amount of the offset adjustment is real-time corrected using the gradient, so that the picture offset can always follow the real eye position and dynamically adaptively converge, preventing double vision caused by drift accumulation, while avoiding dizziness caused by excessive correction, thereby improving the fusion probability and visual comfort of the user in the whole training process, solving the problem that in the double-eye display image optimization process of VR visual training, the real-time drift of the user's eye position causes dynamic misalignment of the left and right eye pixel landing points, the optimized image quality and matching degree are insufficient, and improving the stability and quality of the double-eye display image optimization of VR visual training. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0016] Figure 1 The flowchart of the double-eye display image optimization method for VR visual training provided by an embodiment of the present application. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0018] Please refer to Figure 1 which shows the flowchart of the double-eye display image optimization method for VR visual training provided by an embodiment of the present application, and the method comprises the following steps: Step S001, the right eye and left eye of the VR head-mounted display at each acquisition time point is collected, and the difference vector of the binocular gaze point at each acquisition time point is determined according to the gaze point coordinates.

[0019] Whether the human eye can fuse two images into a single stereoscopic vision depends on the correspondence of the left and right retinas, which is manifested in the VR head-mounted display as whether the same pixel in the images seen by the left and right eyes falls on the same corresponding point. The user will constantly fine-tune during the training process, and may also have strabismus or fatigue eye position drift, resulting in deviation in the position judgment of the same pixel by the left and right eyes. Only when the deviation of the falling points of the lines of sight of the two eyes on the display screen remains within the Panum fusion area, can the brain fuse the two images into a single stereoscopic image.

[0020] Once the falling points deviate, pixel-level reverse translation of the next frame of image in the VR head-mounted display is needed according to the changes of the two eyes to offset the misplacement caused by eye position drift, slight strabismus or device sliding, maximize the fusion probability and prevent diplopia and visual fatigue.

[0021] According to the infrared reflection images of each frame of the right eye and the left eye respectively photographed by the eye movement tracking camera in the VR head-mounted display, the pupil center is located by the bright-dark pupil difference method, the corneal reflection vector and the pupil center offset are calculated, the unit vector of the right eye line of sight in the three-dimensional head-mounted display coordinate system is solved by using the eye tracker, the unit vector and the screen plane are intersected by perspective, and the lens distortion is reversely mapped to obtain the gaze point coordinates of the right eye and the left eye in each frame respectively.

[0022] Among them, the right eye and left eye gaze point coordinates obtained from the infrared reflection images of the right eye and left eye are known technologies and will not be described again.

[0023] The difference value of the gaze point coordinates of the right eye and the left eye of the same frame is recorded as the binocular gaze point difference vector of the acquisition time point corresponding to the same frame.

[0024] It can be understood that each frame corresponds to an acquisition time, so each acquisition time also corresponds to a binocular gaze point difference vector, and the binocular gaze point difference vector of each acquisition time can be obtained.

[0025] In the coordinate system with the upper left corner of the screen as the origin, the x-axis to the right as positive, and the y-axis downward as positive, the value of the x-axis component of the binocular gaze point difference vector is greater than 0, indicating that the right eye falling point is more to the right relative to the left eye falling point, the value of the x-axis component of the binocular gaze point difference vector is less than 0, indicating that the right eye falling point is more to the left relative to the left eye falling point, the value of the y-axis component of the binocular gaze point difference vector is greater than 0, indicating that the right eye falling point is more to the bottom relative to the left eye falling point, and the value of the y-axis component of the binocular gaze point difference vector is less than 0, indicating that the right eye falling point is more to the top relative to the left eye falling point.

[0026] The VR headset's IMU is used to read the linear acceleration, tracking confidence, and actual inter-frame acquisition time interval of all acquisition moments within one consecutive second prior to each acquisition moment.

[0027] Preferably, as an embodiment of this application, 60 acquisition times are uniformly set per second.

[0028] At this point, the binocular gaze difference vectors at all acquisition times are obtained.

[0029] Step S002: Calculate the pixel compensation vector for each acquisition time based on the binocular gaze point difference vector at adjacent acquisition times. Determine the compensation eye at the acquisition time based on the gaze point coordinates of both eyes at the acquisition time. Combine the pixel compensation vector at the second acquisition time to adjust the offset of the image seen by the compensation eye of the VR headset at the second acquisition time.

[0030] Instead of adjusting the image of the VR headset at the first acquisition moment, for the second acquisition moment, based on the binocular gaze difference vector at the second acquisition moment, the misalignment caused by eye drift or device sliding in the image of the VR headset at the next adjacent acquisition moment is compensated in pixels, so that the images of the right and left eyes in the VR headset are realigned on the user's retina.

[0031] Calculate the pixel compensation vector for each acquisition time based on the binocular gaze difference vector at adjacent acquisition times.

[0032] in, Indicates the first Pixel compensation vector at each acquisition time; This represents the cropping function. The purpose of the cropping function is to prevent excessive instantaneous translation of the image caused by extreme eye movement noise or blinking. When the x-axis value and y-axis value of the first position in the cropping function are greater than the preset maximum allowable compensation amount, the value greater than the preset maximum allowable compensation amount is directly assigned as the maximum allowable compensation amount. This indicates the preset maximum allowable compensation amount. The maximum allowable compensation amount should be greater than or equal to 5px and less than or equal to 30px. In this embodiment, the maximum allowable compensation amount is set to 15px. Indicates the first The gain scaling factor at each acquisition moment is used to control the degree of pixel compensation vector correction and avoid image jitter caused by full compensation. Indicates the first The binocular fixation difference vector at each acquisition time; Indicates the first The binocular fixation difference vector at each acquisition time; represents a natural constant with a natural constant as a base number; represents a modulus.

[0033] The human eye is very sensitive to the instantaneous displacement of the image, and if all the misalignments are compensated at once, the vestibular-visual conflict and dizziness will be triggered at a high refresh rate. Therefore, a part of the error is offset by a gain scaling factor, the image is gradually corrected, the brain has enough time to re-integrate the double images, and the discomfort caused by the picture jitter is avoided. At the same time, the direction and power of convergence to zero error are preserved. Therefore, the pixel compensation vector at the second acquisition time calculated according to the difference vector of the fixation points of the two eyes at the first and second acquisition times can avoid picture jumping caused by too large correction amplitude and quickly correct the error.

[0034] When calculating the pixel compensation vector, the value after clipping is taken negative, so that the image movement direction and the eye position error direction are opposite. When the right eye landing point is right, the right eye image needs to be moved to the left to offset the measured misalignment and achieve pixel-level alignment. Therefore, the x-axis component of the pixel compensation vector greater than 0 indicates that the image corresponding to the eye in the VR headset is translated to the right, the x-axis component of the pixel compensation vector less than 0 indicates that the image corresponding to the eye in the VR headset is translated to the left, the y-axis component of the pixel compensation vector greater than 0 indicates that the image corresponding to the eye in the VR headset is translated downward, and the y-axis component of the pixel compensation vector less than 0 indicates that the image corresponding to the eye in the VR headset is translated upward.

[0035] It should be understood that the pixel compensation amount only needs to be applied to a single eye to pull the relative positions of the two retinas back to the fusion zone. This processing can not only save an additional rendering channel and reduce the load, but also avoid the secondary alignment error introduced by the simultaneous movement of the two eyes. For example, when it is found that the right eye is right, only the right eye image needs to be moved to the left, and the left eye remains stationary, and the binocular disparity can be zeroed to complete the spatial correction.

[0036] Specifically, the compensation eye compensated by the pixel compensation amount is the only eye to which the compensation is applied, and the compensation eye compensated by the pixel compensation amount is determined according to the fixation point coordinates of the two eyes.

[0037] Preferably, as an embodiment of the present application, any one collection time is recorded as a target collection time, the Euclidean distance between the gaze point coordinates of the left eye of the target collection time and the center point of the image seen by the left eye in the VR headset is recorded as the offset distance of the left eye at the target collection time, and the Euclidean distance between the gaze point coordinates of the right eye of the target collection time and the center point of the image seen by the left eye in the VR headset is recorded as the offset distance of the right eye at the target collection time; it is judged whether the eye corresponding to the maximum offset distance of the target collection time is the dominant eye, if so, the dominant eye is taken as the compensation eye of the target collection time, if not, the other eye is taken as the compensation eye of the target collection time.

[0038] The process of judging whether it is a dominant eye is to avoid the active modification of the dominant eye image causing visual discomfort.

[0039] It should be noted that since the image of the VR headset at the first collection time is not adjusted, the compensation eye at the first collection time is not judged, and the picture seen by the compensation eye of the VR headset at the first collection time is not offset adjusted.

[0040] The pixel compensation vector at the second collection time is taken as the pixel-level offset, and the picture seen by the compensation eye of the VR headset at the second collection time is offset adjusted.

[0041] Preferably, as an embodiment of the present application, the projection matrix adjustment method and the fragment shader adjustment method can be used for offset adjustment.

[0042] Thus, the offset adjustment of the picture seen by the compensation eye of the VR headset at the second collection time is realized, and the compensation eyes at all collection times are obtained.

[0043] Step S003, determine the compensation gradient of the collection time, combine the pixel compensation vector of the collection time, calculate the pixel gradient compensation vector of the next adjacent collection time of the collection time, and respectively offset adjust the picture seen by the compensation eye of the corresponding collection time according to the pixel gradient compensation vector of all collection times starting from the third collection time.

[0044] The eye position, fatigue degree and fusion ability of a person change all the time, and a fixed compensation amount will soon appear unable to match the new error, so for all collection times starting from the third collection time, the gradient is used to real-time correct the compensation amount of offset adjustment, so that the picture offset can always follow the real eye position dynamic convergence, that is, it can prevent diplopia caused by drift accumulation, and avoid dizziness caused by excessive correction, thereby improving the fusion probability and visual comfort of the user in the whole training process.

[0045] The gradient of the cost function with respect to the pixel compensation vector of the acquisition time is estimated using a numerical method, denoted as a compensation gradient of the acquisition time.

[0046] The gradient of the cost function with respect to the pixel compensation vector of the acquisition time is estimated using a numerical method, denoted as a compensation gradient of the acquisition time.

[0047] The pixel gradient compensation vector of the next adjacent acquisition time of the acquisition time is calculated according to the pixel compensation vector of the acquisition time and the compensation gradient.

[0048] The pixel gradient compensation vector of the next adjacent acquisition time of the acquisition time is calculated according to the pixel compensation vector of the acquisition time and the compensation gradient. The pixel gradient compensation vector of the next adjacent acquisition time of the acquisition time is calculated according to the pixel compensation vector of the acquisition time and the compensation gradient. The pixel gradient compensation vector of the next adjacent acquisition time of the acquisition time is calculated according to the pixel compensation vector of the acquisition time and the compensation gradient. The pixel gradient compensation vector of the next adjacent acquisition time of the acquisition time is calculated according to the pixel compensation vector of the acquisition time and the compensation gradient. The pixel gradient compensation vector of the next adjacent acquisition time of the acquisition time is calculated according to the pixel compensation vector of the acquisition time and the compensation gradient. The pixel gradient compensation vector of the next adjacent acquisition time of the acquisition time is calculated according to the pixel compensation vector of the acquisition time and the compensation gradient. The pixel gradient compensation vector of the next adjacent acquisition time of the acquisition time is calculated according to the pixel compensation vector of the acquisition time and the compensation gradient. The pixel gradient compensation vector of the next adjacent acquisition time of the acquisition time is calculated according to the pixel compensation vector of the acquisition time and the compensation gradient. The pixel gradient compensation vector of the next adjacent acquisition time of the acquisition time is calculated according to the pixel compensation vector of the acquisition time and the compensation gradient.

[0049] The pixel gradient compensation vector of the next adjacent acquisition time of the acquisition time is calculated according to the pixel compensation vector of the acquisition time and the compensation gradient.

[0050] Preferably, as an embodiment of the present application, the projection matrix adjustment method and the fragment shader adjustment method can be used for offset adjustment.

[0051] The pixel gradient compensation vector can establish a mapping relationship between the cost function and the picture offset optimization, so that the picture offset adjustment can be adaptively converged, the image translation can be more smoothly realized, the pixel-level alignment accuracy of the binocular image can be significantly improved, and the training efficiency and hardware stability can be ensured.

[0052] Thus, the binocular display image optimization of the VR visual training is realized.

[0053] Based on the same inventive concept as the above method, the embodiments of the present application also provide a binocular display image optimization system for VR visual training, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor implements the steps of any one of the above-mentioned binocular display image optimization methods for VR visual training when executing the computer program.

[0054] The above merely describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A binocular display image optimization method for VR visual training, characterized in that, The method includes the following steps: Collect the gaze coordinates of the right and left eyes of the VR headset at each collection moment, and determine the binocular gaze difference vector at each collection moment based on the gaze coordinates. Based on the difference vector between the gaze points of the two eyes at adjacent acquisition times, the pixel compensation vector at each acquisition time is calculated. Based on the gaze point coordinates of the two eyes at the acquisition time, the compensation eye at the acquisition time is determined. Combined with the pixel compensation vector at the second acquisition time, the image seen by the compensation eye of the VR headset at the second acquisition time is offset and adjusted. Determine the compensation gradient at the acquisition time, combine it with the pixel compensation vector at the acquisition time, calculate the pixel gradient compensation vector of the next adjacent acquisition time, and adjust the offset of the image seen by the compensated eye at the corresponding acquisition time according to the pixel gradient compensation vectors of all acquisition times starting from the third acquisition time.

2. The VR visual training binocular display image optimization method of claim 1, wherein, The binocular fixation point difference vector at the acquisition time is the difference between the fixation point coordinates of the right eye and the left eye at the same acquisition time.

3. The VR visual training binocular display image optimization method of claim 1, wherein, The formula for calculating the pixel compensation vector at the acquisition time is: wherein, represents a pixel compensation vector for the th acquisition time instant; represents a clipping function; represents a preset maximum allowed compensation; represents a gain scaling factor for the th acquisition time instant; represents a binocular fixation point difference vector for the th acquisition time instant.

4. The VR visual training binocular display image optimization method of claim 3, wherein, The formula for calculating the gain scaling factor at the acquisition time is: represents a binocular fixation point difference vector at a represents a natural constant with a natural constant as a base number. represents a modulus of .​ 5. The VR visual training binocular display image optimization method of claim 1, wherein, The specific method for determining the compensating eye at the acquisition time is as follows: Record any acquisition time as the target acquisition time. Based on the position of the fixation point coordinates of the left and right eyes at the target acquisition time, determine the offset distance of the left and right eyes at the target acquisition time. Determine whether the eye corresponding to the maximum offset distance at the target acquisition time is the dominant eye. If so, use the dominant eye as the compensation eye at the target acquisition time; otherwise, use the other eye as the compensation eye at the target acquisition time.

6. The VR visual training binocular display image optimization method of claim 5, wherein, The specific method for determining the offset distance between the left and right eyes at the target acquisition time is as follows: The Euclidean distance between the gaze point coordinates of the left eye at the time of target acquisition and the center point of the image seen by the left eye in the VR headset is denoted as the offset distance of the left eye at the time of target acquisition. The Euclidean distance between the gaze point coordinates of the right eye at the time of target acquisition and the center point of the image seen by the left eye in the VR headset is denoted as the offset distance of the right eye at the time of target acquisition.

7. The VR visual training binocular display image optimization method of claim 1, wherein, The method for adjusting the offset of the image seen by the compensated eye of the VR headset at the second acquisition time by combining the pixel compensation vector at the second acquisition time includes: The pixel compensation vector at the second acquisition moment is used as a pixel-level offset to adjust the image seen by the compensated eye of the VR headset at the second acquisition moment.

8. The VR visual training binocular display image optimization method of claim 1, wherein, The compensation gradient at the acquisition time is the gradient of the cost function with respect to the pixel compensation vector at the acquisition time, estimated using numerical methods.

9. The VR visual training binocular display image optimization method of claim 3, wherein, The formula for calculating the pixel gradient compensation vector of the next adjacent acquisition time is: wherein, represents the pixel gradient compensation vector at the acquisition time point; represents the compensation gradient at the acquisition time point.

10. A binocular display image optimization system for VR visual training, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1-9.

Images