Method for optimizing the fidelity of visual image display of near-eye display devices

Abstract

The present invention relates to the field of image data processing technology, and more specifically to a method for optimizing the fidelity of visual image display for a near-eye display device. The method comprises: determining the motion distortion of the target area in the binocular image using the movement information of the target area in the binocular image and the binocular disparity information of each frame during the user's head movement; determining the matching degree of the target area in the binocular image of adjacent frames, and determining the rendering reference degree of the target area in the rendered monocular image using the matching degree and the motion distortion degree; and determining the pixel adjustment value of the target pixel in the unrendered monocular image using the rendering reference degree and the pixel value of the target pixel in the binocular image of adjacent frames. The technical solution of the present invention can significantly improve the efficiency of binocular image rendering, optimize the image fusion effect while ensuring real-time rendering performance, make the displayed content appear more realistic, and enhance the virtual or augmented reality experience.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to a method for optimizing the fidelity of visual image display of a near-eye display device. Background Art

[0002] Near-eye displays (NEDs) are wearable displays that project images directly in front of the user's eyes, enabling an immersive experience of virtual or augmented reality content. Visual fidelity is a key factor influencing user immersion and quality of experience. Optimizing the fidelity of NEDs is crucial for enhancing user experience, reducing visual fatigue, and improving application effectiveness. Currently, the immersion and realism of visual images are generally enhanced through personalized rendering using real-time eye tracking and optimization methods that incorporate the human eye's visual perception mechanism.

[0003] In existing technologies, high dynamic range rendering is often used to enhance brightness, contrast, and color saturation, thereby improving the realism of visual image display. However, high dynamic range rendering requires higher computing resources and storage bandwidth, which may introduce additional delays or reduce frame rates in real-time rendering scenarios. When the user's head moves, the different viewing angles of the left and right eyes cause an imbalance in the rendering load and rendering time. The left and right eyes receive the images at different time points, and the binocular parallax information does not match. This can cause deviations in the position and depth perception of virtual objects, which may drift or jump, resulting in distorted visual images. Summary of the Invention

[0004] In order to solve the technical problem of visual image distortion caused by unbalanced rendering load and different rendering times between the left and right eyes during user head movement, the present invention aims to provide a method for optimizing the fidelity of visual image display on a near-eye display device. The technical solution adopted is as follows:

[0005] The present invention provides a method for optimizing the fidelity of visual image display of a near-eye display device, the method comprising:

[0006] When the user's head moves, the motion distortion of the target area is determined by using the movement information of the target area in the binocular image and the binocular disparity information of each frame;

[0007] Determine the matching degree of the target area in the binocular images of adjacent frames, and use the matching degree and motion distortion to determine the rendering reference degree of the target area in the rendered monocular image;

[0008] Determine the pixel adjustment value of the target pixel in the unrendered monocular image by using the rendering reference degree and the pixel value of the target pixel in the target area in the adjacent frame binocular image;

[0009] The binocular images of adjacent frames include the rendered monocular image of the previous frame and the unrendered monocular image of the subsequent frame.

[0010] Furthermore, determining the movement information of the target area in the binocular image includes:

[0011] Determining a first displacement in adjacent frames of the rendered monocular image and a second displacement in adjacent frames of the unrendered monocular image;

[0012] The relative movement amplitude of the target area in the binocular image is determined using the first displacement and the second displacement.

[0013] Furthermore, determining binocular disparity information of each frame of the target area in the binocular image includes:

[0014] The disparity of the target area is determined by using the pixel values of the respective target pixels in the target area in the binocular image;

[0015] The disparity of the target region is used to determine the disparity change rate of each frame of the target region relative to the previous frame.

[0016] Furthermore, the method of determining the motion distortion of the target area by using the movement information of the target area in the binocular image and the binocular disparity information of each frame includes:

[0017] The DTW distance between the motion information and the binocular disparity information is used to determine the motion distortion of the target area.

[0018] Furthermore, determining the matching degree of the target area in the binocular images of adjacent frames includes:

[0019] Determine the vector description of each pair of feature points of the target area in the binocular images of adjacent frames;

[0020] The matching degree of the target area in the binocular images of adjacent frames is calculated using the vector description of each pair of feature points and the number of each pair of feature points.

[0021] Furthermore, the determining of the rendering reference degree of the target area in the rendered monocular image by using the matching degree and the motion distortion degree includes:

[0022] Determine the relationship coefficient between the matching degree and the motion distortion degree;

[0023] The rendering reference degree of the target area in the rendered monocular image is determined using the relationship coefficient, matching degree, and motion distortion.

[0024] Furthermore, the determining of the rendering reference degree of the target area in the rendered monocular image by using the relationship coefficient, the matching degree, and the motion distortion degree includes:

[0025] The rendering reference degree of the target area in the rendered monocular image is calculated using the mean and standard deviation of the relationship coefficient, matching degree and motion distortion degree.

[0026] Furthermore, the pixel adjustment value of the target pixel in the unrendered monocular image is determined by using the rendering reference degree and the pixel value of the target pixel in the target area in the binocular image of the adjacent frame, including:

[0027] The compensation contribution of the target area in the rendered monocular image is determined using the rendering reference degree;

[0028] The pixel adjustment value of the target pixel in the unrendered monocular image is determined by using the compensation contribution and the pixel value of the target pixel in the target area in the adjacent frame binocular image.

[0029] Furthermore, the determining of the compensation contribution of the target area in the rendered monocular image by using the rendering reference degree includes:

[0030] Determine the gaze distance between the target area and the user's gaze point in the unrendered monocular image;

[0031] The compensation contribution of the target area in the rendered monocular image is calculated using the rendering reference degree and gaze distance.

[0032] Furthermore, after determining the pixel adjustment value of the target pixel in the unrendered monocular image, the method further includes:

[0033] The binocular fusion filtering algorithm is used to perform adaptive weight distribution and smoothing on the overlapping areas of the binocular image after pixel value adjustment to obtain the optimized binocular image.

[0034] The present invention has the following beneficial effects:

[0035] When optimizing the visual image display of near-eye display devices to improve image fidelity, the binocular image requires high rendering and display quality when the head moves. This causes disparity information to mismatch when the binocular images are received at different times, leading to visual image distortion. Therefore, the present invention significantly improves binocular image rendering efficiency by tracking the movement trajectory of the binocular gaze points in real time, analyzing the reference areas of adjacent binocular image frames based on the binocular image refresh rate during user movement, and dynamically compensating for disparity. While ensuring real-time rendering performance, it also optimizes image fusion, resulting in higher fidelity of displayed content and an enhanced virtual or augmented reality experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 A flowchart of a method for optimizing the fidelity of visual image display of a near-eye display device provided by one embodiment of the present invention;

[0038] Figure 2 A detailed flow chart of step S1 in a method for optimizing the fidelity of visual image display of a near-eye display device provided by one embodiment of the present invention;

[0039] Figure 3 A detailed flow chart of step S2 in a method for optimizing the fidelity of visual image display of a near-eye display device provided by one embodiment of the present invention;

[0040] Figure 4 A detailed flowchart of step S2 in a method for optimizing the fidelity of visual image display of a near-eye display device provided by another embodiment of the present invention;

[0041] Figure 5 A detailed flowchart of step S3 in a method for optimizing the fidelity of visual image display of a near-eye display device provided by one embodiment of the present invention;

[0042] Figure 6 This is a schematic diagram of the structure of the hardware operating environment of a device for optimizing the fidelity of visual image display of a near-eye display device involved in an embodiment of the present invention;

[0043] Figure 7 Schematic diagram of the framework structure of a system for optimizing the fidelity of visual image display of a near-eye display device according to an embodiment of the present invention;

[0044] Figure 8 This is a schematic diagram of the binocular visual field area before the user wears the mobile near-eye display device from left to right according to the embodiment of the present invention;

[0045] Figure 9 This is a schematic diagram of the change in binocular field of view when a user wears a mobile near-eye display device from left to right according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] To further illustrate the technical means and effects employed by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, describes in detail a method for optimizing the fidelity of visual image display for a near-eye display device according to the present invention, including its specific implementation, structure, features, and effects. In the following description, references to different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics of one or more embodiments may be combined in any suitable manner.

[0047] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0048] The following describes in detail a method for optimizing the fidelity of visual image display of a near-eye display device provided by the present invention with reference to the accompanying drawings.

[0049] Example 1:

[0050] For the method for optimizing the fidelity of visual image display of a near-eye display device provided by the present invention, please refer to Figure 1 , which shows a flowchart of the steps of a method for optimizing the visual image display fidelity of a near-eye display device provided by one embodiment of the present invention.

[0051] The method comprises:

[0052] Step S1, during the user's head movement, using the movement information of the target area in the binocular image and the binocular disparity information of each frame to determine the motion distortion of the target area;

[0053] In this embodiment, first, binocular images need to be captured and preprocessed while the user's head is moving:

[0054] In order to study the implementation plan for optimizing the realism of visual image display of user motion, static image materials covering more scenes can be used. The static image can be transmitted to a near-eye display device and split into two images to be transmitted to the micro-displays of the left and right eyes.

[0055] The user is instructed to move their head along a predetermined path (e.g., left or right), ensuring that the trajectory covers the dynamic range required for the study. The built-in eye tracking system simultaneously captures eye movements. As the user's head moves, the device continuously captures consecutive frames of binocular images from both eyes to capture subtle changes during movement, and simultaneously records the timestamp and gaze position of each frame.

[0056] Adaptive filtering is applied to each frame of binocular image collected to reduce ambient light interference and sensor noise. The image is geometrically corrected using the camera intrinsic parameters obtained in advance to eliminate lens distortion. Color and white balance correction is performed on the left and right eye images to ensure consistent color and brightness on both sides.

[0057] Regarding step S1, specifically, in one embodiment, determining the movement information of the target area in the binocular image includes:

[0058] Determining a first displacement in adjacent frames of the rendered monocular image and a second displacement in adjacent frames of the unrendered monocular image;

[0059] The relative movement amplitude of the target area in the binocular image is determined using the first displacement and the second displacement.

[0060] For ease of understanding, the example here uses the user's left-right head movement as an example. Therefore, the right eye image in the binocular image is rendered faster than the left eye image. Therefore, the rendered monocular image is referred to as the right eye image, and the unrendered monocular image is referred to as the left eye image.

[0061] When a user wearing a near-eye display device moves their head, new left and right eye images (binocular images) are rendered directly in the GPU (Graphics Processing Unit) and output to the display. If the full-frame rendering calculation is large and rendering delays may occur, the two displays will receive images with large parallax. The brain cannot fuse the image stimulation of both eyes and will perceive a motion trajectory that does not conform to physical reality, ultimately leading to visual confusion or difficulty focusing, making the scene look blurry or drifting.

[0062] For dynamic fusion distortion, the impact of head motion on binocular image fusion is determined by analyzing the motion trajectories of various regions in the binocular image. This allows the potential locations of delayed images on the display screen to be predicted, and then reprojection compensation is performed using the rendering results of the other display screen image. First, the impact of head motion must be analyzed to determine whether dynamic compensation is necessary and the strength of the compensation.

[0063] Analyze the motion trajectory (movement information) of each area (target area) in the binocular image:

[0064] Different regions in a binocular image contribute differently to visual fusion and overall image display. By independently matching and reprojecting each region and dynamically adjusting pixel offsets based on the motion and parallax characteristics of each region, we can reduce computational complexity while improving overall processing efficiency and visual fidelity. Therefore, binocular images are segmented.

[0065] The graph cut algorithm is used to segment the continuous frame binocular images into regions to obtain several regions of each frame image of the left and right eyes.

[0066] As the head moves, the refresh synchronization between the left and right eye images becomes increasingly severe. The larger the binocular image parallax, the more the asynchronous visual information causes visual "smearing" or "afterimages" during the visual fusion process. The impact of head movement needs to be analyzed. Each region in the binocular image shifts with head movement, resulting in different rendering speeds for the left and right eyes, which increases the impact on binocular image parallax. The effect of head movement on visual image display distortion can be determined by comparing the changes in the position of each region in consecutive binocular image frames with the changes in binocular image parallax.

[0067] Use the optical flow method to obtain the position of each region in the continuous frame binocular image, and determine the relative movement amplitude of each region based on the offset of the region center:

[0068]

[0069] Indicates any The relative movement amplitude of each region in the continuous frame binocular image; Indicates the The displacement of a region in adjacent frames of the right eye image (first displacement), that is, the distance between the center points of the region in two adjacent frames of the image; Indicates the The displacement of the region in the adjacent frame of the left eye image (second displacement); Indicates the The average displacement of the regions in the binocular image.

[0070] For details, please refer to Figure 2 In one embodiment, determining binocular disparity information of a target area in each frame of a binocular image includes:

[0071] Step S11, determining the disparity of the target area using the pixel values of the respective target pixels in the target area in the binocular image;

[0072] Step S12: using the disparity of the target area, determining the disparity change rate of each frame of the target area relative to the previous frame.

[0073] Binocular image parallax is a key mechanism for the human brain to perceive three-dimensional space and depth information, and determines the fidelity of visual image display. Analyze the parallax and changes of binocular images caused by head movement:

[0074]

[0075] Indicates the The disparity of the region in the binocular image; Indicates the The area in any right eye image The pixel value of a pixel (target pixel); Indicates the The same region in the left eye image The pixel value of a pixel (target pixel). Indicates the The total number of pixels in the area.

[0076] The rate of change of binocular disparity between adjacent frames can be used to capture the dynamic change trend of the user's perspective in real time. Based on the change of disparity in the binocular images of adjacent frames, the rate of change of disparity in each binocular image is obtained:

[0077]

[0078] Indicates the The region in The disparity change rate in the frame image, that is, the disparity change rate of each frame of the target area relative to the previous frame; Indicates the The region in The parallax of the binocular image at the frame image moment; Indicates the The region in The disparity of the binocular images at the frame moment.

[0079] Specifically, in one embodiment, step S1 includes:

[0080] The DTW distance between the motion information and the binocular disparity information is used to determine the motion distortion of the target area.

[0081] To quantify the degree of image distortion in dynamic scenes, the motion distortion of each region is obtained based on the relative movement amplitude and parallax change rate of the region in consecutive frames of binocular images:

[0082]

[0083] Indicates the The motion distortion of each region; Indicates the A sequence consisting of the relative movement amplitudes of the regions in consecutive frames of binocular images; Indicates the The disparity change rate sequence of a region in consecutive frames of binocular images. Indicates the The relative movement of the regions is similar to the DTW (Dynamic Time Warping) distance of the parallax change rate, reflecting the impact of motion on binocular image distortion.

[0084] Step S2, determining the matching degree of the target area in the binocular images of adjacent frames, and determining the rendering reference degree of the target area in the rendered monocular image using the matching degree and the motion distortion degree;

[0085] For details, please refer to Figure 3 In one embodiment, step S2 includes:

[0086] Step S21, determining vector descriptions of each pair of feature points of the target area in the binocular images of adjacent frames;

[0087] Step S22 : using the vector description of each pair of feature points and the number of each pair of feature points, calculate the matching degree of the target area in the binocular images of adjacent frames.

[0088] Due to head movement, binocular image rendering delays and different update rates for the left and right eye images on the two displays can cause significant parallax, resulting in binocular image fusion distortion. Since the left and right eye view cameras are positioned close together, the scenes seen overlap significantly. Compensation can be performed by referencing the previously rendered frame on the display on the other side as a reference. For example, when the head moves to the right, the majority of the scene rendered by the right eye in the current frame will appear within the field of view of the left eye in the next frame, resulting in minimal parallax and nearly identical pixels.

[0089] Please refer to Figure 8 and Figure 9 , Figure 8 This is a schematic diagram of the binocular visual field area before the user wears the mobile near-eye display device from left to right according to the embodiment of the present invention; Figure 9 This is a schematic diagram of the change in binocular field of view when a user wears a mobile near-eye display device from left to right according to an embodiment of the present invention.

[0090] As shown in the figure, the circles represent the left and right eyes respectively, and the triangle represents the target area. In the first part of the image, it is on the left side of the right eye's perspective. As the head moves to the right, it is located on the left side of the left eye's perspective in the second part of the image. The rendered image of the current frame of the left eye can refer to the previous frame image of the right eye.

[0091] Taking the movement of the head from left to right as an example, the right eye perspective is the first to be exposed to the latest field of view.

[0092] Analyze the reference area for rendering in the binocular image:

[0093] When the head moves to the right, the right eye renders the image relatively quickly. To prevent excessive parallax between the left and right eye images, we can refer to the matching effect of each area in the right eye's rendered image in the unfinished left eye image to perform reprojection and fast rendering, improving image fidelity. Feature matching is performed on adjacent binocular images. The vector description between each feature point pair in each region determines the matching degree of each region in the adjacent binocular images.

[0094] The SIFT (Scale-invariant feature transform) algorithm is used to perform feature matching on the left eye image of the current frame (the unrendered monocular image of the subsequent frame) and the right eye image of the adjacent previous frame (the rendered monocular image of the previous frame) to obtain a vector description of each feature point in each area.

[0095] According to the vector description of the feature point pairs in the same area of the current frame left eye image and the adjacent previous frame right eye image, the matching degree of each area in the adjacent frame binocular image is obtained:

[0096]

[0097] Indicates the The region in The matching degree in the frame image, that is, the matching degree of the target area in the adjacent frame binocular image; Indicates the number of feature point pairs in each region (the number of each pair of feature points); Indicates the The area in the right eye The first frame in the image Vector description of feature points; Indicates the The area in the left eye The same frame image A vector description of feature points.

[0098] For details, please refer to Figure 4 In one embodiment, step S2 includes:

[0099] Step S210, determining the relationship coefficient between the matching degree and the motion distortion degree;

[0100] Step S220 : Determine a rendering reference degree of the target area in the rendered monocular image by using the relationship coefficient, the matching degree, and the motion distortion degree.

[0101] More specifically, step S220 includes:

[0102] The rendering reference degree of the target area in the rendered monocular image is calculated using the mean and standard deviation of the relationship coefficient, matching degree and motion distortion degree.

[0103] When the user's head moves relatively quickly, resulting in high motion distortion, the matching degree is usually reduced. The relationship between the motion distortion and matching degree in consecutive binocular images can be used to predict the matching degree of the current frame. The relationship coefficient between the motion distortion of each region and the matching degree of the binocular images in the adjacent frames is obtained:

[0104]

[0105] Indicates the The relationship coefficient between the motion distortion of a region and the matching degree of the binocular images of adjacent frames; Indicates the The motion distortion of each region; Indicates the The matching degree of each region in the binocular images of adjacent frames; Indicates the The Pearson correlation coefficient between the motion distortion of a region and its matching degree in the binocular images of adjacent frames.

[0106] To improve the rendering efficiency of the left-eye image loading, the matching degree between the previous right-eye image and the current left-eye image is predicted for each region to determine whether the rendered result can be used as a reference. The rendering reference degree of each region is predicted based on the relationship coefficient between the motion distortion degree of each region and the matching degree of the binocular image of the adjacent frame, as well as the motion distortion degree of the current binocular image:

[0107]

[0108] Indicates the The area in the right eye The rendering reference degree in the frame image (the rendered monocular image of the previous frame); Indicates the The average matching degree of each region in the continuous frame graphics; Indicates the The relationship coefficient between the motion distortion of a region and the matching degree of the binocular images of adjacent frames; Indicates the The standard deviation of the matching degree of each region in the consecutive frames of binocular images; Indicates the The standard deviation of the motion distortion of the region in the consecutive frames of binocular images; Indicates the The region in Motion distortion in the frame image; Indicates the The average value of the motion distortion of the region in the consecutive frames of binocular images. Indicates the The linear regression prediction matching degree of each region (based on the least squares linear regression prediction formula) is larger. The larger the formula is, the higher the regional matching degree between the previous frame image of the right eye (the rendered monocular image of the previous frame) and the current frame image of the left eye (the rendered monocular image of the next frame) is, and the higher the rendering reference is.

[0109] Step S3: Determine a pixel adjustment value of the target pixel in the unrendered monocular image using the rendering reference degree and the pixel value of the target pixel in the target area in the adjacent frame binocular image; wherein the adjacent frame binocular image includes the rendered monocular image of the previous frame and the unrendered monocular image of the subsequent frame.

[0110] For details, please refer to Figure 5 In one embodiment, step S3 includes:

[0111] Step S31 : determining the compensation contribution of the target area in the rendered monocular image using the rendering reference degree.

[0112] More specifically, step S31 includes:

[0113] Determine the gaze distance between the target area and the user's gaze point in the unrendered monocular image;

[0114] The compensation contribution of the target area in the rendered monocular image is calculated using the rendering reference degree and gaze distance.

[0115] By analyzing the contribution of each area in the binocular image during the reprojection compensation process, a stable and highly consistent target area on one display screen is extracted, which is called the reference area. The local projection change is constructed in combination with the influence of head movement, and the image information of the reference area is dynamically reprojected and mapped to the predicted position on the other display screen, thereby achieving precise alignment and natural fusion of the left and right eye images, and ultimately generating an optimized image with high stereoscopic sense and realism, enhancing the immersive experience of near-eye display devices.

[0116] The greater the rendering reference degree of each area in the binocular image, the greater the weight of reprojection to the unrendered image; at the same time, when the reference area is near the user's gaze point, the faster it should be rendered, the more it needs to directly refer to the rendered image, which can improve the fidelity of the visual image display. Analyze the compensation contribution of each area to visual fusion:

[0117]

[0118] Indicates the The area in the right eye Compensation contribution in the frame image; Indicates the The area in the left eye The gaze distance between the frame image and the user's gaze point. The smaller the value, the more the area is the user's focus, and the faster the rendering should be completed; Indicates the The area in the right eye The rendering reference degree in the frame image.

[0119] Step S32 : Determine a pixel adjustment value of the target pixel in the unrendered monocular image by using the compensation contribution and the pixel value of the target pixel in the target area in the binocular image of the adjacent frame.

[0120] Head movement produces dynamic image distortion, and the rendered image in the binocular image needs to be used as a reference to quickly render the other image. The greater the compensation contribution of each area, the more correction is needed in the distortion compensation process to ensure image accuracy and visual comfort. Based on the compensation contribution of each area to visual fusion, the pixel adjustment value is obtained:

[0121]

[0122] Indicates the The area in the left eye Any frame image The pixel adjustment value of a pixel point (target pixel point); Indicates the The area in the right eye Compensation contribution in the frame image; Indicates the The area in the left eye Frame image The pixel value of the pixel point is the pixel value of the image received without rendering; Indicates the The area in the right eye Frame image The pixel value of the pixel point is the pixel value that has been rendered. Indicates that according to The compensation contribution of each area is used as the reprojection weight to reproject the rendered pixel value of the previous frame of the right eye image into the left eye image, and combined with the original unrendered pixel value of the left eye to obtain the pixel adjustment value.

[0123] In addition, after step S3, the method further includes:

[0124] The binocular fusion filtering algorithm is used to perform adaptive weight distribution and smoothing on the overlapping areas of the binocular image after pixel value adjustment to obtain the optimized binocular image.

[0125] After dynamic compensation, some pixels in the binocular image may shift due to local reprojection caused by head movement, especially in edge areas or areas rich in detail. This can cause jagged edges or unnatural jumps. A binocular fusion filtering algorithm adaptively weights and smoothes the overlapping areas of the left and right eye images, effectively fusing the differences at the edges of the two images. This restores local details, reduces image jumps and jagged edges, and creates a smoother, more natural visual experience.

[0126] Since user head movement and rendering delays may cause motion blur or smearing, asynchronous time warp (ATW) technology is used to eliminate the blur and smearing caused by rendering delays, making images in moving scenes clearer and more stable. It also enables near-eye display devices to maintain high-precision synchronization and natural fusion of binocular images when the user moves, thereby improving the realism of visual image display.

[0127] When optimizing the visual image display of near-eye display devices to improve image fidelity, the binocular image requires high rendering and display quality when the head moves. This causes disparity information to mismatch when the binocular images are received at different times, leading to visual image distortion. Therefore, the present invention significantly improves binocular image rendering efficiency by tracking the movement trajectory of the binocular gaze points in real time, analyzing the reference areas of adjacent binocular image frames based on the binocular image refresh rate during user movement, and dynamically compensating for disparity. While ensuring real-time rendering performance, it also optimizes image fusion, resulting in higher fidelity of displayed content and an enhanced virtual or augmented reality experience.

[0128] By real-time monitoring of the movement trajectory of the user's gaze point in the binocular image, head movement information can be accurately captured, thereby identifying image displacement or distortion caused by head movement; according to the reception speed of the new image, the previous and next frame images are matched to extract a stable and reliable reference area, reducing the time error caused by inconsistent frame rates, reusing historical image data to improve rendering efficiency, and providing a reliable benchmark for parallax compensation; by comparing the positional relationship of the reference area within the overlapping range of the left and right eye images of the current frame, the importance of each area in visual fusion can be evaluated, and the parts that have a greater impact on depth perception and stereo fusion can be compensated; a weighted projection algorithm is used to perform pixel-level dynamic adjustment on the slow-refresh side image to ensure that the binocular image is finely fused while being efficiently rendered, thereby significantly improving the visual image realism and viewing comfort of near-eye display devices.

[0129] Example 2:

[0130] The present invention also provides a device for optimizing the fidelity of visual image display for a near-eye display device. The device can be an augmented reality (AR), virtual reality (VR), or mixed reality (MR) near-eye display device.

[0131] like Figure 6 As shown, Figure 6 It is a structural diagram of the hardware operating environment of the visual image display fidelity optimization processing device of the near-eye display device involved in the embodiment of the present invention.

[0132] like Figure 6 As shown, the near-eye display device's visual image display fidelity optimization processing device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display and an input unit, such as a control panel. Optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a WiFi interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory, such as a disk drive. The memory 1005 may also be a storage device independent of the aforementioned processor 1001. The memory 1005, which serves as a computer storage medium, may include a visual image display fidelity optimization processing program.

[0133] Those skilled in the art will understand that Figure 6 The hardware structure shown in the figure does not constitute a limitation of the device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0134] Continue to refer to Figure 6 , Figure 6 The memory 1005 as a computer-readable storage medium may include an operating system, a user interface module, a network communication module, and a visual image display fidelity optimization processing program.

[0135] exist Figure 6 In the embodiment, the network communication module is mainly used to connect to the server and perform data communication with the server; and the processor 1001 can call the visual image display fidelity optimization processing program stored in the memory 1005 and execute the steps in the above embodiments.

[0136] The hardware structure of the visual image display fidelity optimization processing device based on the above-mentioned near-eye display device is used to implement various embodiments of the visual image display fidelity optimization processing method of the near-eye display device of the present invention.

[0137] In addition, the present invention also provides a near-eye display device visual image display fidelity optimization processing system, please refer to Figure 7 The visual image display fidelity optimization processing system of the near-eye display device includes:

[0138] The trajectory analysis module A10 is used to determine the motion distortion of the target area by using the movement information of the target area in the binocular image and the binocular disparity information of each frame during the user's head movement;

[0139] Matching reference module A20, used to determine the matching degree of the target area in the binocular images of adjacent frames, and determine the rendering reference degree of the target area in the rendered monocular image using the matching degree and motion distortion;

[0140] A pixel optimization module A30 is configured to determine a pixel adjustment value of a target pixel in an unrendered monocular image by using a rendering reference and a pixel value of a target pixel in a target area in an adjacent frame of a binocular image;

[0141] Furthermore, the trajectory analysis module A10 is further configured to:

[0142] Determining a first displacement in adjacent frames of the rendered monocular image and a second displacement in adjacent frames of the unrendered monocular image;

[0143] The relative movement amplitude of the target area in the binocular image is determined using the first displacement and the second displacement.

[0144] Furthermore, the trajectory analysis module A10 is further configured to:

[0145] The disparity of the target area is determined by using the pixel values of the respective target pixels in the target area in the binocular image;

[0146] The disparity of the target region is used to determine the disparity change rate of each frame of the target region relative to the previous frame.

[0147] Furthermore, the trajectory analysis module A10 is further configured to:

[0148] The DTW distance between the motion information and the binocular disparity information is used to determine the motion distortion of the target area.

[0149] Furthermore, the matching reference module A20 is further configured to:

[0150] Determine the vector description of each pair of feature points of the target area in the binocular images of adjacent frames;

[0151] The matching degree of the target area in the binocular images of adjacent frames is calculated using the vector description of each pair of feature points and the number of each pair of feature points.

[0152] Furthermore, the matching reference module A20 is further configured to:

[0153] Determine the relationship coefficient between the matching degree and the motion distortion degree;

[0154] The rendering reference degree of the target area in the rendered monocular image is determined using the relationship coefficient, matching degree, and motion distortion.

[0155] Furthermore, the matching reference module A20 is further configured to:

[0156] The rendering reference degree of the target area in the rendered monocular image is calculated using the mean and standard deviation of the relationship coefficient, matching degree and motion distortion degree.

[0157] Furthermore, the pixel optimization module A30 is further configured to:

[0158] The compensation contribution of the target area in the rendered monocular image is determined using the rendering reference degree;

[0159] The pixel adjustment value of the target pixel in the unrendered monocular image is determined by using the compensation contribution and the pixel value of the target pixel in the target area in the adjacent frame binocular image.

[0160] Furthermore, the pixel optimization module A30 is further configured to:

[0161] Determine the gaze distance between the target area and the user's gaze point in the unrendered monocular image;

[0162] The compensation contribution of the target area in the rendered monocular image is calculated using the rendering reference degree and gaze distance.

[0163] Furthermore, the pixel optimization module A30 is further configured to:

[0164] The binocular fusion filtering algorithm is used to perform adaptive weight distribution and smoothing on the overlapping areas of the binocular image after pixel value adjustment to obtain the optimized binocular image.

[0165] The specific implementation of the visual image display fidelity optimization processing system of the near-eye display device of the present invention is basically the same as the various embodiments of the visual image display fidelity optimization processing method of the near-eye display device mentioned above, and will not be repeated here.

[0166] The present invention also provides a computer-readable storage medium having a visual image display fidelity optimization processing program stored thereon. When executed by a processor, the visual image display fidelity optimization processing program implements the steps of the above-described method for optimizing visual image display fidelity for a near-eye display device.

[0167] Among them, the method implemented when the visual image display fidelity optimization processing program is executed can refer to the various embodiments of the visual image display fidelity optimization processing method of the near-eye display device of the present invention, and will not be repeated here.

[0168] It should be noted that the order in which the embodiments of the present invention are described above is for illustrative purposes only and does not necessarily represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0169] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

[0170] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0171] The above description is only a preferred embodiment of the present invention and does not limit the scope of protection of the present invention. All equivalent structural / method transformations made by using the contents of the present invention description and drawings under the inventive concept of the present invention, or direct / indirect application in other related technical fields are included in the scope of protection of the present invention.

Claims

1. A method for optimizing the fidelity of visual image display of a near-eye display device, characterized in that: The method comprises: When the user's head moves, the motion distortion of the target area is determined by using the movement information of the target area in the binocular image and the binocular disparity information of each frame; Determine the matching degree of the target area in the binocular images of adjacent frames, and use the matching degree and motion distortion to determine the rendering reference degree of the target area in the rendered monocular image; Determine the pixel adjustment value of the target pixel in the unrendered monocular image by using the rendering reference degree and the pixel value of the target pixel in the target area in the adjacent frame binocular image; The binocular images of adjacent frames include the rendered monocular image of the previous frame and the unrendered monocular image of the subsequent frame.

2. The method for optimizing the fidelity of visual image display of a near-eye display device according to claim 1, characterized in that: Determine the motion information of the target area in the binocular image, including: Determining a first displacement in adjacent frames of the rendered monocular image and a second displacement in adjacent frames of the unrendered monocular image; The relative movement amplitude of the target area in the binocular image is determined using the first displacement and the second displacement.

3. The method for optimizing the visual image display fidelity of a near-eye display device according to claim 1, characterized in that: Determine the binocular disparity information of each frame of the target area in the binocular image, including: The disparity of the target area is determined by using the pixel values of the respective target pixels in the target area in the binocular image; The disparity of the target region is used to determine the disparity change rate of each frame of the target region relative to the previous frame.

4. The method for optimizing the fidelity of visual image display of a near-eye display device according to claim 1, characterized in that: The method of determining the motion distortion of the target area by using the movement information of the target area in the binocular image and the binocular disparity information of each frame includes: The DTW distance between the motion information and the binocular disparity information is used to determine the motion distortion of the target area.

5. The method for optimizing the fidelity of visual image display of a near-eye display device according to claim 1, characterized in that: Determining the matching degree of the target area in the binocular images of adjacent frames includes: Determine the vector description of each pair of feature points of the target area in the binocular images of adjacent frames; The matching degree of the target area in the binocular images of adjacent frames is calculated using the vector description of each pair of feature points and the number of each pair of feature points.

6. The method for optimizing the fidelity of visual image display of a near-eye display device according to claim 1, characterized in that: The method of determining the rendering reference degree of the target area in the rendered monocular image by using the matching degree and the motion distortion degree includes: Determine the relationship coefficient between the matching degree and the motion distortion degree; The rendering reference degree of the target area in the rendered monocular image is determined using the relationship coefficient, matching degree, and motion distortion.

7. The method for optimizing the fidelity of visual image display of a near-eye display device according to claim 6, characterized in that: The method of determining a rendering reference degree of a target area in a rendered monocular image by using the relationship coefficient, the matching degree, and the motion distortion degree includes: The rendering reference degree of the target area in the rendered monocular image is calculated using the mean and standard deviation of the relationship coefficient, matching degree and motion distortion degree.

8. The method for optimizing the fidelity of visual image display of a near-eye display device according to claim 1, characterized in that: The method of determining the pixel adjustment value of the target pixel in the unrendered monocular image by using the rendering reference degree and the pixel value of the target pixel in the target area in the binocular image of the adjacent frame includes: The compensation contribution of the target area in the rendered monocular image is determined using the rendering reference degree; The pixel adjustment value of the target pixel in the unrendered monocular image is determined by using the compensation contribution and the pixel value of the target pixel in the target area in the adjacent frame binocular image.

9. The method for optimizing the visual image display fidelity of a near-eye display device according to claim 8, characterized in that: The method of determining the compensation contribution of the target area in the rendered monocular image by using the rendering reference degree includes: Determine the gaze distance between the target area and the user's gaze point in the unrendered monocular image; The compensation contribution of the target area in the rendered monocular image is calculated using the rendering reference degree and gaze distance.

10. The method for optimizing the visual image display fidelity of a near-eye display device according to claim 1, characterized in that: After determining the pixel adjustment value of the target pixel in the unrendered monocular image, the method further includes: The binocular fusion filtering algorithm is used to perform adaptive weight distribution and smoothing on the overlapping areas of the binocular image after pixel value adjustment to obtain the optimized binocular image.

Images