Binocular display image optimization method and system for VR visual training

Abstract

The application relates to the technical field of VR visual training, and discloses a binocular display image optimization method and system for VR visual training, which comprises the following steps: collecting the gaze point coordinates of the right eye and the left eye of a VR head-mounted display, determining a binocular gaze point difference vector; calculating a pixel compensation vector, determining a compensation eye, and performing offset adjustment on the picture seen by the compensation eye of the VR head-mounted display at a second collection time; determining a compensation gradient, combining the pixel compensation vector to calculate a pixel gradient compensation vector at a next adjacent collection time of the collection time, and respectively performing offset adjustment on the picture seen by the compensation eye at the corresponding collection time according to the pixel gradient compensation vectors at all collection times starting from the third collection time. The application can improve the image quality and matching degree after optimization.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of VR visual training, in particular to a binocular display image optimization method and system for VR visual training. BACKGROUND

[0002] VR visual training is a visual function training method based on virtual reality technology, which can improve the visual ability of patients with amblyopia, strabismus and abnormal visual function in game interaction. In order to make the user perceive a stereoscopic virtual environment, the images presented to the left eye and the right eye in the VR head-mounted display of VR visual training are slightly different, and the binoculars see the visual information received when observing the object at the corresponding angle respectively. Therefore, the fusion quality and matching degree of the images seen by the binoculars can directly affect the accuracy and stability of stereoscopic vision, as well as the training effect and visual comfort.

[0003] Generally, the images of the binoculars in the VR head-mounted display are optimized using the offline calibration idea. Specifically, at the time of factory shipment, a camera is used to shoot a lens distortion map, and a left-eye and a right-eye distortion grid are generated at one time. The vertex displacement or texture resampling of each frame of original image is performed according to the distortion grid, and then the left and right images are respectively pre-distorted to the head-mounted screen according to the fixed IPD (Interpupillary Distance) pupil distance. However, this optimization method can only compensate for the lens distortion with a fixed pupil distance, without considering the real-time position changes of the user's eyeballs. When the user's eye position deviates slightly or drifts due to fatigue during the training process, if the original grid is still used for projection, it will cause unpredictable horizontal or vertical deviation of the pixel landing points of the left and right eyes, directly inducing diplopia and misplacement, and causing visual fatigue of the user. SUMMARY

[0004] The present application provides a binocular display image optimization method and system for VR visual training to solve the problem that the real-time drift of the user's eye position during the binocular display image optimization process of VR visual training causes dynamic misplacement of the pixel landing points of the left and right eyes, resulting in insufficient image quality and matching degree after optimization. The technical solution adopted is as follows:

[0005] In a first aspect, an embodiment of the present application provides a binocular display image optimization method for VR visual training, which comprises the following steps:

[0006] The gaze point coordinates of the right eye and the left eye of the VR head-mounted display at each collection time are collected, and the binocular gaze point difference vector at each collection time is determined according to the gaze point coordinates;

[0007] The pixel compensation vector at each collection time is calculated respectively according to the binocular gaze point difference vectors of adjacent collection times, the compensation eye at the collection time is determined according to the gaze point coordinates of the binoculars at the collection time, and the screen seen by the compensation eye of the VR head-mounted display at the second collection time is adjusted by offsetting in combination with the pixel compensation vector at the second collection time.

[0008] Determine the compensation gradient at the acquisition time, combine it with the pixel compensation vector at the acquisition time, calculate the pixel gradient compensation vector of the next adjacent acquisition time, and adjust the offset of the image seen by the compensated eye at the corresponding acquisition time according to the pixel gradient compensation vectors of all acquisition times starting from the third acquisition time.

[0009] Furthermore, the binocular gaze point difference vector at the acquisition time is the difference between the gaze point coordinates of the right eye and the left eye at the same acquisition time.

[0010] Furthermore, the formula for calculating the pixel compensation vector at the acquisition time is:

[0011]

[0012] in, Indicates the first Pixel compensation vector at each acquisition time; Represents the clipping function; This indicates the preset maximum allowable compensation amount; Indicates the first Gain scaling factor at each acquisition time; Indicates the first The difference vector of gaze points between the two eyes at each acquisition time.

[0013] Furthermore, the formula for calculating the gain scaling factor at the acquisition time is:

[0014]

[0015] Indicates the first The binocular fixation difference vector at each acquisition time; Represents the natural constant with the natural constant as the base; express The model.

[0016] Furthermore, the specific method for determining the compensating eye at the acquisition time is as follows:

[0017] Record any acquisition time as the target acquisition time. Based on the position of the fixation point coordinates of the left and right eyes at the target acquisition time, determine the offset distance of the left and right eyes at the target acquisition time.

[0018] Determine whether the eye corresponding to the maximum offset distance at the target acquisition time is the dominant eye. If so, use the dominant eye as the compensation eye at the target acquisition time; otherwise, use the other eye as the compensation eye at the target acquisition time.

[0019] Further, the specific determination method of the offset distance of the left eye and the right eye at the target acquisition moment is as follows:

[0020] The Euclidean distance between the gaze point coordinate of the left eye at the target acquisition moment and the center point of the image seen by the left eye in the VR head-mounted display is recorded as the offset distance of the left eye at the target acquisition moment.

[0021] The Euclidean distance between the gaze point coordinate of the right eye at the target acquisition moment and the center point of the image seen by the left eye in the VR head-mounted display is recorded as the offset distance of the right eye at the target acquisition moment.

[0022] Further, the specific method for adjusting the image seen by the compensated eye of the VR head-mounted display at the second acquisition moment by combining the pixel compensation vector of the second acquisition moment includes the following steps.

[0023] The pixel compensation vector of the second acquisition moment is used as the pixel-level offset to adjust the image seen by the compensated eye of the VR head-mounted display at the second acquisition moment.

[0024] Further, the compensation gradient of the acquisition moment is the gradient of the cost function with respect to the pixel compensation vector of the acquisition moment estimated by using a numerical method.

[0025] Further, the calculation formula of the pixel gradient compensation vector of the next adjacent acquisition moment of the acquisition moment is as follows:

[0026]

[0027] wherein, represents the pixel gradient compensation vector of the first acquisition moment; represents the compensation gradient of the first acquisition moment. represents the pixel gradient compensation vector of the first acquisition moment; represents the compensation gradient of the first acquisition moment.

[0028] In a second aspect, the embodiments of the present application further provide a VR visual training binocular display image optimization system, which comprises a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor implements the steps of the method according to any one of the above aspects when executing the computer program.

[0029] The present application has the following beneficial effects:

[0030] The application first determines a binocular fixation point difference vector according to the fixation point coordinate difference of the right eye and the left eye of the VR head-mounted display at the same collection moment, the binocular fixation point difference vector can represent the difference between the landing points of the left and right eyes; the dominant eye is determined to avoid the active modification of the dominant eye image causing visual discomfort, then for the second collection moment, the misalignment caused by the eye position drift or device sliding in the image in the VR head-mounted display at the next adjacent collection moment is compensated in pixel units according to the binocular fixation point difference vector at the second collection moment, the monocular image is gradually modified, the brain has enough time to re-fuse the double images, the discomfort caused by the picture shaking is avoided, and the images of the right eye and the left eye in the VR head-mounted display are re-aligned on the retina of the user; further, for all collection moments starting from the third collection moment, the compensation amount of the offset adjustment is real-time corrected using the gradient, so that the picture offset can always follow the real eye position and dynamically adaptively converge, preventing double vision caused by drift accumulation, while avoiding dizziness caused by excessive correction, thereby improving the fusion probability and visual comfort of the user in the whole training process, solving the problem that in the binocular display image optimization process of VR visual training, the real-time drift of the user's eye position causes dynamic misalignment of the left and right eye pixel landing points, the optimized image quality and matching degree are insufficient, and improving the stability and quality of the binocular display image optimization of VR visual training. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only show some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0032] Figure 1 The flowchart of the binocular display image optimization method for VR visual training provided by an embodiment of the present application. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only show some embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0034] Please refer to Figure 1 which shows the flowchart of the binocular display image optimization method for VR visual training provided by an embodiment of the present application, the method comprises the following steps:

[0035] In step S001, the gaze point coordinates of the right eye and the left eye of the VR head-mounted display at each collection time are collected, and the binocular gaze point difference vector at each collection time is determined according to the gaze point coordinates.

[0036] Whether the human eye can fuse two images into a single stereoscopic vision depends on the correspondence of the left and right retinas, which is manifested in the VR head-mounted display as whether the same pixel in the images seen by the left and right eyes falls on the same corresponding point. The user will constantly fine-tune during the training process, and may also have strabismus or fatigue eye position drift, resulting in a deviation in the position judgment of the same pixel by the left and right eyes. Only when the deviation of the falling points of the lines of sight of the two eyes on the display screen remains within the Panum fusion area, can the brain fuse the two images into a single stereoscopic image.

[0037] Once the falling points deviate, pixel-level reverse translation needs to be performed on the next frame of image in the VR head-mounted display according to the changes of the two eyes to offset the misplacement caused by eye position drift, slight strabismus or device sliding, maximize the fusion probability and prevent diplopia and visual fatigue.

[0038] According to the infrared reflection images of each frame of the right eye and the left eye respectively photographed by the eye movement tracking camera in the VR head-mounted display, the pupil center is located by the bright-dark pupil difference method, the corneal reflection vector and the pupil center offset are calculated, the unit vector of the right eye line of sight in the three-dimensional head-mounted display coordinate system is solved by using the eye tracker, the unit vector and the screen plane are intersected by perspective, and the lens distortion is reversely mapped to respectively obtain the gaze point coordinates of the right eye and the left eye in each frame.

[0039] Among them, the acquisition of the gaze point coordinates of the right eye and the left eye according to the infrared reflection images of the right eye and the left eye is a known technology and will not be described again.

[0040] The difference between the gaze point coordinates of the right eye and the left eye of the same frame is recorded as the binocular gaze point difference vector at the collection time corresponding to the same frame.

[0041] It can be understood that each frame corresponds to a collection time, so each collection time also corresponds to a binocular gaze point difference vector, and the binocular gaze point difference vector at each collection time can be obtained.

[0042] In the coordinate system with the upper left corner of the screen as the origin, the x-axis to the right as positive, and the y-axis downward as positive, the value of the x-axis component of the binocular gaze point difference vector greater than 0 indicates that the right eye falling point is more to the right relative to the left eye falling point, the value of the x-axis component of the binocular gaze point difference vector less than 0 indicates that the right eye falling point is more to the left relative to the left eye falling point, the value of the y-axis component of the binocular gaze point difference vector greater than 0 indicates that the right eye falling point is more downward relative to the left eye falling point, and the value of the y-axis component of the binocular gaze point difference vector less than 0 indicates that the right eye falling point is more upward relative to the left eye falling point.

[0043] The IMU of the VR headset reads the linear acceleration, tracking confidence, and actual inter-frame acquisition time interval of all acquisition moments within one consecutive second prior to each acquisition moment.

[0044] Preferably, as an embodiment of this application, 60 acquisition times are uniformly set per second.

[0045] At this point, the binocular gaze difference vectors at all acquisition times are obtained.

[0046] Step S002: Calculate the pixel compensation vector for each acquisition time based on the binocular gaze point difference vector at adjacent acquisition times. Determine the compensation eye at the acquisition time based on the gaze point coordinates of both eyes at the acquisition time. Combine the pixel compensation vector at the second acquisition time to adjust the offset of the image seen by the compensation eye of the VR headset at the second acquisition time.

[0047] Instead of adjusting the image of the VR headset at the first acquisition moment, for the second acquisition moment, based on the binocular gaze difference vector at the second acquisition moment, the misalignment caused by eye drift or device sliding in the image of the VR headset at the next adjacent acquisition moment is compensated in pixels, so that the images of the right and left eyes in the VR headset are realigned on the user's retina.

[0048] Calculate the pixel compensation vector for each acquisition time based on the binocular gaze difference vector at adjacent acquisition times.

[0049]

[0050]

[0051] in, Indicates the first Pixel compensation vector at each acquisition time; This represents the cropping function. The purpose of the cropping function is to prevent excessive instantaneous translation of the image caused by extreme eye movement noise or blinking. When the x-axis value and y-axis value of the first position in the cropping function are greater than the preset maximum allowable compensation amount, the value greater than the preset maximum allowable compensation amount is directly assigned as the maximum allowable compensation amount. This indicates the preset maximum allowable compensation amount. The maximum allowable compensation amount should be greater than or equal to 5px and less than or equal to 30px. In this embodiment, the maximum allowable compensation amount is set to 15px. Indicates the first The gain scaling factor at each acquisition moment is used to control the degree of pixel compensation vector correction and avoid image jitter caused by full compensation. Indicates the first The binocular fixation difference vector at each acquisition time; Indicates the first The binocular fixation difference vector at each acquisition time; Represents the natural constant with the natural constant as the base; express The model.

[0052] The human eye is very sensitive to instantaneous displacement of images. If all misalignments are compensated at once, it can trigger vestibular-visual conflict and dizziness at high refresh rates. Therefore, by using a gain scaling factor to offset some of the error and gradually correcting the image, the brain has enough time to re-integrate the two images, avoiding discomfort caused by image jitter. At the same time, it retains the direction and momentum for convergence towards zero error. Therefore, the pixel compensation vector at the second acquisition time, calculated based on the difference vector between the two gaze points at the first and second acquisition times, can avoid excessive correction amplitude that could cause image jumps and quickly correct errors.

[0053] When calculating the pixel compensation vector, the cropped value is negativeed to ensure that the image movement direction and the eye position error direction are inversely related. If the right eye's position is off-center to the right, the right eye image needs to be shifted to the left to compensate for the measured misalignment and achieve pixel-level alignment. Therefore, a pixel compensation vector with an x-axis component greater than 0 indicates a rightward shift of the corresponding eye image in the VR headset, a pixel compensation vector with an x-axis component less than 0 indicates a leftward shift of the corresponding eye image in the VR headset, a pixel compensation vector with a y-axis component greater than 0 indicates a downward shift of the corresponding eye image in the VR headset, and a pixel compensation vector with a y-axis component less than 0 indicates an upward shift of the corresponding eye image in the VR headset.

[0054] It is important to understand that pixel compensation only needs to be applied to one eye to pull the relative positions of the two retinas back to the fusion area. This approach can save an extra rendering pass, reduce the load, and avoid secondary alignment errors introduced by the simultaneous movement of both eyes. For example, when the right eye is found to be deviating to the right, the right eye image only needs to be shifted to the left while the left eye remains stationary, and the binocular parallax can be reduced to zero, thus completing spatial correction.

[0055] Specifically, the compensation eye for pixel compensation is the eye that receives the only compensation, and the compensation eye for pixel compensation is determined based on the coordinates of the gaze points of both eyes.

[0056] Preferably, as an embodiment of the present application, any one collection time is recorded as a target collection time, the Euclidean distance between the gaze point coordinates of the left eye of the target collection time and the center point of the image seen by the left eye in the VR headset is recorded as the offset distance of the left eye at the target collection time, and the Euclidean distance between the gaze point coordinates of the right eye of the target collection time and the center point of the image seen by the left eye in the VR headset is recorded as the offset distance of the right eye at the target collection time; it is judged whether the eye corresponding to the maximum offset distance of the target collection time is the dominant eye, if so, the dominant eye is taken as the compensation eye of the target collection time, if not, the other eye is taken as the compensation eye of the target collection time.

[0057] The process of judging whether it is a dominant eye is to avoid the active modification of the dominant eye image causing visual discomfort.

[0058] It should be noted that since the image of the VR headset at the first collection time is not adjusted, the compensation eye at the first collection time is not judged, and the picture seen by the compensation eye of the VR headset at the first collection time is not offset adjusted.

[0059] The pixel compensation vector at the second collection time is taken as the pixel-level offset, and the picture seen by the compensation eye of the VR headset at the second collection time is offset adjusted.

[0060] Preferably, as an embodiment of the present application, the projection matrix adjustment method and the fragment shader adjustment method can be used for offset adjustment.

[0061] At this point, the offset adjustment of the picture seen by the compensation eye of the VR headset at the second collection time is realized, and the compensation eyes at all collection times are obtained.

[0062] Step S003, determine the compensation gradient of the collection time, combine the pixel compensation vector of the collection time, calculate the pixel gradient compensation vector of the next adjacent collection time of the collection time, and respectively offset adjust the picture seen by the compensation eye of the corresponding collection time according to the pixel gradient compensation vector of all collection times starting from the third collection time.

[0063] The eye position, fatigue degree and fusion ability of a person change all the time, and a fixed compensation amount will soon appear unable to match the new error, so for all collection times starting from the third collection time, the gradient is used to real-time correct the compensation amount of offset adjustment, so that the picture offset can always follow the real eye position dynamic convergence, that is, it can prevent diplopia caused by drift accumulation, and avoid dizziness caused by excessive correction, thereby improving the fusion probability and visual comfort of the user in the whole training process.

[0064] The gradient of the cost function with respect to the pixel compensation vector of the acquisition time is estimated using a numerical method, denoted as a compensation gradient of the acquisition time.

[0065] The gradient of the cost function with respect to the pixel compensation vector of the acquisition time is estimated using a numerical method, denoted as a compensation gradient of the acquisition time.

[0066] The pixel gradient compensation vector of the next adjacent acquisition time of the acquisition time is calculated according to the pixel compensation vector of the acquisition time and the compensation gradient.

[0067]

[0068] The pixel gradient compensation vector of the next adjacent acquisition time of the acquisition time is calculated according to the pixel compensation vector of the acquisition time and the compensation gradient. The pixel gradient compensation vector of the next adjacent acquisition time of the acquisition time is calculated according to the pixel compensation vector of the acquisition time and the compensation gradient. The pixel gradient compensation vector of the next adjacent acquisition time of the acquisition time is calculated according to the pixel compensation vector of the acquisition time and the compensation gradient. The pixel gradient compensation vector of the next adjacent acquisition time of the acquisition time is calculated according to the pixel compensation vector of the acquisition time and the compensation gradient. The pixel gradient compensation vector of the next adjacent acquisition time of the acquisition time is calculated according to the pixel compensation vector of the acquisition time and the compensation gradient. The pixel gradient compensation vector of the next adjacent acquisition time of the acquisition time is calculated according to the pixel compensation vector of the acquisition time and the compensation gradient. The pixel gradient compensation vector of the next adjacent acquisition time of the acquisition time is calculated according to the pixel compensation vector of the acquisition time and the compensation gradient. The pixel gradient compensation vector of the next adjacent acquisition time of the acquisition time is calculated according to the pixel compensation vector of the acquisition time and the compensation gradient. The pixel gradient compensation vector of the next adjacent acquisition time of the acquisition time is calculated according to the pixel compensation vector of the acquisition time and the compensation gradient.

[0069] The pixel gradient compensation vector of the next adjacent acquisition time of the acquisition time is calculated according to the pixel compensation vector of the acquisition time and the compensation gradient.

[0070] Preferably, as an embodiment of the present application, the projection matrix adjustment method and the fragment shader adjustment method can be used for offset adjustment.

[0071] The pixel gradient compensation vector can establish a mapping relationship between the cost function and the picture offset optimization, so that the picture offset adjustment can be adaptively converged, the image translation can be more smoothly realized, the pixel-level alignment accuracy of the binocular image can be significantly improved, and the training efficiency and hardware stability can be ensured.

[0072] Thus, the double-eye display image optimization of the VR visual training is realized.

[0073] Based on the same inventive concept as the above method, the embodiments of the present application also provide a binocular display image optimization system for VR visual training, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor implements the steps of any one of the above-mentioned binocular display image optimization methods for VR visual training when executing the computer program.

[0074] The above merely describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for optimizing binocular display images in VR vision training, characterized in that, The method includes the following steps: Collect the gaze coordinates of the right and left eyes of the VR headset at each collection moment, and determine the binocular gaze difference vector at each collection moment based on the gaze coordinates. Based on the difference vector between the gaze points of the two eyes at adjacent acquisition times, the pixel compensation vector at each acquisition time is calculated. Based on the gaze point coordinates of the two eyes at the acquisition time, the compensation eye at the acquisition time is determined. Combined with the pixel compensation vector at the second acquisition time, the image seen by the compensation eye of the VR headset at the second acquisition time is offset and adjusted. Determine the compensation gradient at the acquisition time, combine it with the pixel compensation vector at the acquisition time, calculate the pixel gradient compensation vector of the next adjacent acquisition time, and adjust the offset of the image seen by the compensated eye at the corresponding acquisition time according to the pixel gradient compensation vectors of all acquisition times starting from the third acquisition time. The formula for calculating the pixel compensation vector at the acquisition time is: in, Indicates the first Pixel compensation vector at each acquisition time; Represents the clipping function; This indicates the preset maximum allowable compensation amount; Indicates the first Gain scaling factor at each acquisition time; Indicates the first The binocular fixation difference vector at each acquisition time; The specific method for determining the compensating eye at the acquisition time is as follows: Record any acquisition time as the target acquisition time. Based on the position of the fixation point coordinates of the left and right eyes at the target acquisition time, determine the offset distance of the left and right eyes at the target acquisition time. Determine whether the eye corresponding to the maximum offset distance at the target acquisition time is the dominant eye. If so, use the dominant eye as the compensation eye at the target acquisition time; otherwise, use the other eye as the compensation eye at the target acquisition time. The method for adjusting the offset of the image seen by the compensated eye of the VR headset at the second acquisition time by combining the pixel compensation vector at the second acquisition time includes: The pixel compensation vector at the second acquisition moment is used as a pixel-level offset to adjust the image seen by the compensated eye of the VR headset at the second acquisition moment. The compensation gradient at the acquisition time is the gradient of the cost function with respect to the pixel compensation vector at the acquisition time, estimated using numerical methods. The formula for calculating the pixel gradient compensation vector of the next adjacent acquisition time is: in, Indicates the first Pixel gradient compensation vector at each acquisition time; Indicates the first The compensation gradient at each acquisition time.

2. The method for optimizing binocular display images in VR vision training according to claim 1, characterized in that, The binocular fixation point difference vector at the acquisition time is the difference between the fixation point coordinates of the right eye and the left eye at the same acquisition time.

3. The method for optimizing binocular display images in VR vision training according to claim 1, characterized in that, The formula for calculating the gain scaling factor at the acquisition time is: Indicates the first The binocular fixation difference vector at each acquisition time; Represents the natural constant with the natural constant as the base; express The model.

4. The method for optimizing binocular display images in VR vision training according to claim 1, characterized in that, The specific method for determining the offset distance between the left and right eyes at the target acquisition time is as follows: The Euclidean distance between the gaze point coordinates of the left eye at the time of target acquisition and the center point of the image seen by the left eye in the VR headset is denoted as the offset distance of the left eye at the time of target acquisition. The Euclidean distance between the gaze point coordinates of the right eye at the time of target acquisition and the center point of the image seen by the left eye in the VR headset is denoted as the offset distance of the right eye at the time of target acquisition.

5. A binocular display image optimization system for VR vision training, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1-4.

Images