Visual image display fidelity optimization processing method of near-to-eye display equipment

Abstract

The invention relates to the technical field of image data processing, in particular to a visual image display fidelity optimization processing method for near-to-eye display equipment, which comprises the following steps of: in the head moving process of a user, determining the motion distortion degree of a target area by utilizing the moving information of the target area in a binocular image and the binocular parallax information of each frame; determining the matching degree of the target area in the adjacent frames of binocular images, and determining the rendering reference degree of the target area in the rendered monocular image by using the matching degree and the motion distortion degree; and determining the pixel adjustment value of the target pixel point in the unrendered monocular image by using the rendering reference degree and the pixel value of the target pixel point of the target area in the adjacent frame binocular image. Through the technical scheme of the invention, the binocular image rendering efficiency can be greatly improved, the real-time rendering performance is ensured, the image fusion effect is optimized, the display content is enabled to present higher fidelity, and the virtual or augmented reality experience is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and particularly relates to a method for optimizing the visual image display fidelity of a near-eye display device. Background Art

[0002] A near-eye display device is a wearable display device that projects an image in front of a user's eyes, enabling the user to immerse themselves in virtual or augmented reality content. The fidelity of the visual image is a key factor affecting the user's immersion and experience quality. Optimizing the visual image display fidelity of a near-eye display device is crucial for enhancing the user experience, reducing visual fatigue, and improving the application effect. Currently, generally through personalized rendering with real-time eye tracking and optimization methods combined with the human eye visual perception mechanism, the immersion and authenticity of the visual image are improved.

[0003] In the prior art, high dynamic range rendering is usually adopted to enhance brightness, contrast, and increase color saturation, thereby enhancing the realism of the visual image display. However, high dynamic range rendering requires higher computing resources and storage bandwidth, which may introduce additional latency or reduce the frame rate in real-time rendering scenarios. When the user's head moves, due to the different viewing angles of the left and right eyes, the rendering loads of the left and right eyes are unbalanced and the rendering times are different. The time points at which the left and right eyes receive the images are different, and the binocular disparity information does not match, resulting in deviations in the position and depth perception of virtual objects, and possible drift or jitter, leading to distortion of the visual image display. Summary of the Invention

[0004] In order to solve the technical problem that the rendering loads of the left and right eyes are unbalanced and the rendering times are different during the movement of the user's head, resulting in distortion of the visual image display, the purpose of the present invention is to provide a method for optimizing the visual image display fidelity of a near-eye display device. The specific technical solution adopted is as follows: The present invention provides a method for optimizing the visual image display fidelity of a near-eye display device, the method comprising: During the movement of the user's head, using the movement information of the target region in the binocular images and the binocular disparity information of each frame, determining the motion distortion degree of the target region; Determining the matching degree of the target region in the binocular images of adjacent frames, and using the matching degree and the motion distortion degree to determine the rendering reference degree of the target region in the rendered monocular image; Using the rendering reference degree and the pixel values of the target pixel points in the target region of the binocular images of adjacent frames, determining the pixel adjustment value of the target pixel points in the unrendered monocular image; Wherein, the binocular images of adjacent frames include the rendered monocular image of the previous frame and the unrendered monocular image of the subsequent frame.

[0005] Further, determine the movement information of the target area in the binocular image, including: Determine the first displacement in adjacent frames of the rendered monocular image and the second displacement in adjacent frames of the unrendered monocular image; Use the first displacement and the second displacement to determine the relative movement amplitude of the target area in the binocular image.

[0006] Further, determine the binocular disparity information of each frame of the target area in the binocular image, including: Use the pixel values of the respective target pixel points of the target area in the binocular image to determine the disparity of the target area; Use the disparity of the target area to determine the rate of change of disparity of the target area in each frame relative to the previous frame.

[0007] Further, the determining the motion distortion degree of the target area by using the movement information and the binocular disparity information of each frame of the target area in the binocular image includes: Use the DTW distance between the movement information and the binocular disparity information to determine the motion distortion degree of the target area.

[0008] Further, the determining the matching degree of the target area in adjacent-frame binocular images includes: Determine the vector descriptions of the pairs of feature points of the target area in adjacent-frame binocular images; Use the vector descriptions of the pairs of feature points and the number of pairs of feature points to calculate the matching degree of the target area in adjacent-frame binocular images.

[0009] Further, the determining the rendering reference degree of the target area in the rendered monocular image by using the matching degree and the motion distortion degree includes: Determine the relationship coefficient between the matching degree and the motion distortion degree; Use the relationship coefficient, the matching degree, and the motion distortion degree to determine the rendering reference degree of the target area in the rendered monocular image.

[0010] Further, the determining the rendering reference degree of the target area in the rendered monocular image by using the relationship coefficient, the matching degree, and the motion distortion degree includes: Use the mean values and standard deviations of the relationship coefficient, the matching degree, and the motion distortion degree to calculate the rendering reference degree of the target area in the rendered monocular image.

[0011] Further, the determining the pixel adjustment value of the target pixel points in the unrendered monocular image by using the rendering reference degree and the pixel values of the target pixel points of the target area in adjacent-frame binocular images includes: Use the rendering reference degree to determine the compensation contribution degree of the target area in the rendered monocular image; Determine the pixel adjustment value of the target pixel point in the unrendered monocular image by using the compensation contribution degree and the pixel value of the target pixel point in the binocular images of adjacent frames in the target area.

[0012] Further, the determining the compensation contribution degree of the target area in the rendered monocular image by using the rendering reference degree includes: Determine the fixation distance between the target area in the unrendered monocular image and the user's fixation point; Calculate the compensation contribution degree of the target area in the rendered monocular image by using the rendering reference degree and the fixation distance.

[0013] Further, after determining the pixel adjustment value of the target pixel point in the unrendered monocular image, the method further includes: Use the binocular fusion filtering algorithm to perform adaptive weight assignment and smoothing processing on the overlapping area in the binocular image after pixel value adjustment to obtain an optimized binocular image.

[0014] The present invention has the following beneficial effects: In the process of optimizing the visual image display of a near-eye display device to improve the image display fidelity, due to the need for high rendering display effects when the binocular images move the head, the binocular images receive the picture time points differently, and the parallax information is mismatched, resulting in visual image display distortion. Therefore, the present invention can greatly improve the efficiency of binocular image rendering by real-time tracking the moving trajectories of the binocular fixation points, analyzing the referenceable areas of adjacent-frame binocular images according to the refresh speed of the binocular images during the user's movement, and dynamically compensating for parallax. While ensuring the real-time rendering performance, the image fusion effect is optimized, so that the displayed content presents higher fidelity and enhances the virtual or augmented reality experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0016] Figure 1 It is a flowchart of the steps of a method for optimizing the visual image display fidelity of a near-eye display device provided by an embodiment of the present invention; Figure 2 It is a detailed flowchart of step S1 in a method for optimizing the visual image display fidelity of a near-eye display device provided by an embodiment of the present invention; Figure 3The detailed flowchart of step S2 in the method for optimizing the visual image display fidelity of a near-eye display device provided by an embodiment of the present invention; Figure 4 The detailed flowchart of step S2 in the method for optimizing the visual image display fidelity of a near-eye display device provided by another embodiment of the present invention; Figure 5 The detailed flowchart of step S3 in the method for optimizing the visual image display fidelity of a near-eye display device provided by an embodiment of the present invention; Figure 6 The structural schematic diagram of the hardware operating environment of the device for optimizing the visual image display fidelity of a near-eye display device involved in the solution of the embodiment of the present invention; Figure 7 The framework structural schematic diagram of the system for optimizing the visual image display fidelity of a near-eye display device involved in the solution of the embodiment of the present invention; Figure 8 The schematic diagram before the change of the binocular visual field area when the user wears a mobile near-eye display device from left to right according to the solution of the embodiment of the present invention; Figure 9 The schematic diagram after the change of the binocular visual field area when the user wears a mobile near-eye display device from left to right according to the solution of the embodiment of the present invention. Detailed implementation manners

[0017] In order to further elaborate on the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following, in combination with the accompanying drawings and preferred embodiments, details the specific implementation manners, structures, features and effects of a method for optimizing the visual image display fidelity of a near-eye display device proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs.

[0019] The following specifically describes the specific solution of a method for optimizing the visual image display fidelity of a near-eye display device provided by the present invention with reference to the accompanying drawings.

[0020] Embodiment 1:

[0021] For the method for optimizing the visual image display fidelity of a near-eye display device provided by the present invention, please refer to Figure 1 , which shows the flowchart of the steps of the method for optimizing the visual image display fidelity of a near-eye display device provided by an embodiment of the present invention.

[0022] The method includes: Step S1, during the movement of the user's head, using the movement information of the target area in the binocular images and the binocular disparity information of each frame, determine the motion distortion degree of the target area; In this embodiment, first, during the movement of the user's head, it is necessary to collect binocular images and perform preprocessing: To study the optimization implementation plan for the visual image display fidelity of user movement, static image materials covering more scenarios can be used. The static images are transmitted to the near-eye display device and are split into two images and sent to the micro-displays of the left and right eyes.

[0023] Guide the user to perform head movement along a predetermined path (such as moving left and right), ensure that the movement trajectory covers the dynamic range required for the study, and at the same time use the built-in eye movement tracking system to capture eye movement information. During the movement of the user's head, the device continuously collects consecutive frames of binocular images of the left and right eyes to capture subtle changes during the movement, and synchronously records the timestamp of each frame and the fixation point position.

[0024] Apply adaptive filtering to each frame of the collected binocular images to reduce ambient light interference and sensor noise; use the pre-calibrated camera internal parameters to perform geometric correction on the images to eliminate lens distortion; perform color balance and white balance correction on the left and right eye images to ensure the color and brightness of the two images are consistent.

[0025] For step S1, specifically, in one embodiment, determining the movement information of the target area in the binocular images includes: Determine the first displacement in adjacent frames of the rendered monocular image and the second displacement in adjacent frames of the unrendered monocular image; Use the first displacement and the second displacement to determine the relative movement amplitude of the target area in the binocular images.

[0026] For the sake of easy understanding, here, taking the user's head movement in the left and right direction as an example for illustration. Therefore, the right-eye image in the binocular images is rendered faster than the left-eye image. So, the rendered monocular image is used to refer to the right-eye image, and the unrendered monocular image is called the left-eye image.

[0027] When the user wearing the near-eye display device moves the head, new left and right eye images (binocular images) are directly rendered in the GPU (Graphics Processing Unit), and output to the display screen. If the full-frame rendering calculation amount is large and there may be rendering delays, resulting in images with a large disparity received by the two display screens, the brain cannot fuse the image stimuli of the two eyes, and will perceive a motion trajectory that does not conform to physical reality, ultimately leading to visual confusion or difficulty in focusing, making the scene look blurry or drifting.

[0028] For dynamic fusion distortion, by analyzing the motion trajectories of each region in the binocular images, the influence of head movement on binocular image fusion is obtained, and the possible positions in the display screen image with frame delay are predicted. Furthermore, reprojection compensation is performed based on the rendering results of another display screen image. First, it is necessary to analyze the influence of head movement to determine whether dynamic compensation is required and the intensity of dynamic compensation.

[0029] Analyze the motion trajectories (movement information) of each region (target region) in the binocular images: Different regions in the binocular images contribute differently to visual fusion and overall image display. By performing independent matching and reprojection compensation for each region, and dynamically adjusting the pixel offset according to the motion changes and parallax characteristics of each region, the overall processing efficiency and visual fidelity can be improved while reducing the computational complexity. Therefore, region segmentation is performed on the binocular images.

[0030] Use the graph cut algorithm to perform region segmentation on the consecutive-frame binocular images, and obtain several regions for each frame of the left and right eye images.

[0031] As the head moves, the asynchronous refresh of the left and right eye images becomes more and more serious. The greater the binocular image parallax, the more asynchronous visual information leads to the appearance of "ghosting" or "afterimage" phenomena during the process of visual fusion of the binocular images. It is necessary to analyze the influence of head movement. The position shift of each region in the binocular images caused by head movement results in different speeds of the rendered images of the left and right eyes, increasing the influence on the binocular image parallax. According to the changes in the positions of each region in the consecutive-frame binocular images and the changes in the binocular image parallax, the influence of head movement on the distortion of visual image display is obtained.

[0032] Use the optical flow method to obtain the positions of each region in the consecutive-frame binocular images, and determine the relative movement amplitude of each region according to the offset of the region center:

[0033] Denote the relative movement amplitude of any th region in the consecutive-frame binocular images; Denote the displacement (first displacement) of the th region in the adjacent frames of the right eye image, that is, the distance between the center points of the region in two adjacent frames of images; Denote the displacement (second displacement) of the th region in the adjacent frames of the left eye image; Denote the average displacement of the th region in the binocular images.

[0034] Specifically, please refer to Figure 2, in one embodiment, determining the binocular disparity information of each frame of the target area in the binocular image includes: Step S11, determining the disparity of the target area by using the pixel values of the target pixel points in the target area of the binocular image; Step S12, determining the rate of change of the disparity of the target area with respect to the previous frame by using the disparity of the target area.

[0035] The disparity of the binocular image is the key mechanism for the human brain to perceive three-dimensional space and depth information, and determines the fidelity of the visual image display. Analyze the binocular image disparity and its change caused by head movement:

[0036] Denote the disparity of the th area in the binocular image; Denote the pixel value of the th area at the th pixel point (target pixel point) in the right-eye image; Denote the pixel value of the th area at the same th pixel point (target pixel point) in the left-eye image. Denote the total number of pixel points in the th area.

[0037] Through the rate of change of binocular disparity between adjacent frames, the dynamic change trend of the user's perspective can be captured in real time. According to the change of disparity in the binocular images of adjacent frames, obtain the rate of change of disparity in the binocular image of each frame:

[0038] Denote the rate of change of disparity of the th area in the th frame image, that is, the rate of change of the disparity of each frame of the target area with respect to the previous frame; Denote the disparity of the binocular image of the th area at the moment of the th frame image; Denote the disparity of the binocular image of the th area at the moment of the th frame image.

[0039] Specifically, in one embodiment, the step S1 includes: Determining the motion distortion degree of the target area by using the DTW distance between the motion information and the binocular disparity information.

[0040] To quantify the distortion degree of an image in a dynamic scenario, according to the relative movement amplitude and the disparity change rate of regions in consecutive-frame binocular images, the motion distortion degree of each region is obtained:

[0041] Denote the motion distortion degree of the th region; Denote the sequence composed of the relative movement amplitudes of the th region in consecutive-frame binocular images; Denote the sequence of the disparity change rates of the th region in consecutive-frame binocular images. Denote the DTW (Dynamic Time Warping) similarity distance between the relative movement amplitude and the disparity change rate of the th region, reflecting the influence of motion on the distortion of binocular images.

[0042] Step S2, determine the matching degree of the target region in adjacent-frame binocular images, and use the matching degree and the motion distortion degree to determine the rendering reference degree of the target region in the rendered monocular image; Specifically, please refer to Figure 3 , in one embodiment, the step S2 includes: Step S21, determine the vector descriptions of each pair of feature points of the target region in adjacent-frame binocular images; Step S22, calculate the matching degree of the target region in adjacent-frame binocular images by using the vector descriptions of each pair of feature points and the number of each pair of feature points.

[0043] Due to the rendering delay of binocular images caused by head movement, the update speeds of the left and right eye images on the two side displays are different, which may lead to a large disparity, thus resulting in the phenomenon of binocular image fusion distortion. Since the camera positions of the left and right eye views are close and the seen scenes highly overlap, the previous frame image of the side display that has been rendered can be referred to as a benchmark for compensation. For example, when the head moves to the right, most of the scene rendered by the right eye in the current frame will appear in the field of view of the left eye in the next frame, with a very small disparity and almost identical pixels.

[0044] Please refer to Figure 8 and Figure 9 , Figure 8 is a schematic diagram before the change of the binocular vision area when the user wears the mobile near-eye display device from left to right according to the embodiment scheme of the present invention; Figure 9 is a schematic diagram after the change of the binocular vision area when the user wears the mobile near-eye display device from left to right according to the embodiment scheme of the present invention.

[0045] As shown in the figure, the circles respectively represent the left and right eyes, and the triangles represent the target areas. On the left side of the right-eye view in the first part of the image, as the head moves to the right, it is on the left side of the left-eye view in the second part of the image. The rendered image of the current frame of the left eye can refer to the previous frame image of the right eye.

[0046] Taking the head movement from left to right as an example, the right-eye view first comes into contact with the latest field of view.

[0047] Analyze the renderable reference areas in the binocular images: When the head moves to the right, the rendering of the right-eye image is relatively fast. To prevent excessive parallax between the left and right eye images, the matching effect of each area in the already-rendered right-eye image in the uncompleted-rendered left-eye image can be referred to for reprojection and fast rendering to improve the realism of the image display. Feature matching is performed on adjacent-frame binocular images, and the matching degree of each area in the adjacent-frame binocular images is obtained through the vector description between each pair of feature points in each area.

[0048] Use the SIFT (Scale-invariant feature transform) algorithm to perform feature matching on the current frame left-eye image (the unrendered monocular image of the later frame) and the adjacent previous frame right-eye image (the already-rendered monocular image of the previous frame) to obtain the vector description of each feature point in each area.

[0049] According to the vector descriptions of the feature point pairs in the same area in the current frame left-eye image and the adjacent previous frame right-eye image, obtain the matching degree of each area in the adjacent-frame binocular images:

[0050] Represents the matching degree of the th area in the th frame image, that is, the matching degree of the target area in the adjacent-frame binocular images; Represents the number of pairs of feature points in each area (the number of pairs of each feature point); Represents the th area in the right-eye th frame image, the th vector description of the feature point; Represents the th area in the left-eye th frame image, the same th vector description of the feature point.

[0051] Specifically, please refer to Figure 4 In one embodiment, the step S2 includes: Step S210, determine the relationship coefficient between the matching degree and the motion distortion degree; Step S220: Determine the rendering reference degree of the target area in the rendered monocular image by using the relationship coefficient, matching degree, and motion distortion degree.

[0052] More specifically, step S220 includes: Calculate the rendering reference degree of the target area in the rendered monocular image by using the mean and standard deviation of the relationship coefficient, matching degree, and motion distortion degree respectively.

[0053] When the user's head movement is relatively fast, resulting in a high motion distortion degree, it is usually accompanied by a decrease in the matching degree. Through the variation relationship between the motion distortion degree and the matching degree in consecutive frames of binocular images, the matching degree of the current frame can be predicted. Obtain the relationship coefficient between the motion distortion degree of each area and the matching degree of adjacent-frame binocular images:

[0054] Represents the relationship coefficient between the motion distortion degree of the th area and the matching degree of adjacent-frame binocular images; represents the motion distortion degree of the th area; represents the matching degree of the th area in adjacent-frame binocular images; represents the Pearson correlation coefficient between the motion distortion degree of the

[0055] th area and the matching degree in adjacent-frame binocular images.

[0056] Represents the rendering reference degree of the th area in the right-eye th frame image (the previously rendered monocular image of the previous frame); represents the mean value of the matching degree of the th area in consecutive-frame graphics; represents the relationship coefficient between the motion distortion degree of the th area and the matching degree of adjacent-frame binocular images; represents the standard deviation of the matching degree of the th area in consecutive-frame binocular images; The standard deviation of the motion distortion degree of a region in consecutive-frame binocular images; Indicates the th region's motion distortion degree in the th frame image; Indicates the mean value of the motion distortion degree of the th region in consecutive-frame binocular images. Indicates the linear regression prediction matching degree of the

[0057] th region (for the linear regression prediction formula based on least squares). The larger this formula is, the higher the region matching degree between the right-eye previous-frame image (the rendered monocular image of the previous frame) and the current-frame left-eye image (the rendered monocular image of the subsequent frame), and the higher the rendering reference degree.

[0058] Specifically, please refer to Figure 5 , in one embodiment, the step S3 includes: Step S31, determining the compensation contribution degree of the target region in the rendered monocular image using the rendering reference degree.

[0059] More specifically, the step S31 includes: Determining the fixation distance between the target region in the unrendered monocular image and the user's fixation point; Using the rendering reference degree and the fixation distance to calculate the compensation contribution degree of the target region in the rendered monocular image.

[0060] By analyzing the contributions of each region in the binocular images during the reprojection compensation process, extracting a stable and highly consistent target region on one side of the display screen, here called the reference region, constructing a local projection change in combination with the influence of head movement, dynamically reprojecting the image information of the reference region, and mapping it to the predicted position on the other side of the display screen, so as to achieve precise alignment and natural fusion of the left and right eye images, and finally generate an optimized image with high stereoscopic effect and fidelity, enhancing the immersive experience of the near-eye display device.

[0061] When the rendering reference degree of each region in the binocular images is larger, it indicates that the weight reprojected to the unrendered image is larger; at the same time, when the reference region is near the user's fixation point, the rendering should be completed faster, and the rendered completed image needs to be directly referred to more, which can improve the visual image display fidelity. Analyze the compensation contribution of each region to visual fusion:

[0062] Indicates the compensation contribution degree of the th region in the th frame image of the right eye; Indicates the th region in the th frame image of the left eye and the fixation distance from the user's fixation point. When this value is smaller, it indicates that this region is the focus of the user's attention and should be rendered as soon as possible; Indicates the th region in the th frame image of the right eye for rendering reference.

[0063] Step S32: Determine the pixel adjustment value of the target pixel point in the unrendered monocular image by using the compensation contribution degree and the pixel value of the target pixel point in the target region of the binocular images in adjacent frames.

[0064] Head movement causes dynamic image distortion. It is necessary to use the rendered image in the binocular images as a reference to quickly render the other image. When the compensation contribution of each region is greater, it indicates that this region requires more correction during the distortion compensation process to ensure the accuracy and visual comfort of the image. According to the compensation contribution of each region to visual fusion, obtain the pixel adjustment value:

[0065] Indicates the th region in the th frame image of the left eye for any th pixel point (target pixel point) of the pixel adjustment value; Indicates the th region in the th frame image of the right eye for the compensation contribution degree; Indicates the th region in the th frame image of the left eye for the th pixel point of the pixel value, which is the pixel value of the unrendered received image; Indicates the th region in the th frame image of the right eye for the th pixel point of the pixel value, which is the pixel value of the already rendered pixel. Indicates that according to the compensation contribution of the th region, as the reprojection weight, reproject the pixel value of the previously rendered right eye image to the left eye image and combine it with the original unrendered left eye pixel value to obtain the pixel adjustment value.

[0066] In addition, after the step S3, the method further includes: Using a binocular fusion filtering algorithm, adaptive weight allocation and smoothing processing are performed on the overlapping region in the binocular images after pixel value adjustment to obtain optimized binocular images.

[0067] After dynamic compensation of the binocular images, due to the local reprojection processing of head movement, some pixels in the images may shift, especially in the edge regions or parts with rich details, resulting in jagged distortions or unnatural jumps at the edges. By using the binocular fusion filtering algorithm, through adaptive weight allocation and smoothing processing of the left and right eye images in the overlapping region, the differences between the two images at the edges are effectively fused, thereby repairing local details, reducing image jumps and jaggedness, and making the entire picture present a smoother and more natural visual effect.

[0068] Due to the possible occurrence of motion blur or ghosting caused by user head movement and rendering delay, the asynchronous time warping (ATW) technology is used to eliminate the blur and ghosting caused by rendering delay, making the images in the motion scene clearer and more stable, enabling the near-eye display device to still maintain high-precision synchronization and natural fusion of the binocular images when the user is moving, thereby enhancing the fidelity of the visual image display.

[0069] In the process of optimizing the visual image display of the near-eye display device to improve the image display fidelity, due to the need for high rendering display effects when the binocular images move with the head, the receiving time points of the binocular image frames are different, and the disparity information does not match, resulting in visual image display distortion. Therefore, the present invention can significantly improve the rendering efficiency of the binocular images by real-time tracking the moving trajectories of the binocular fixation points, analyzing the reference regions of adjacent frame binocular images according to the refresh speed of the binocular images during user movement, and dynamically compensating the disparity. While ensuring real-time rendering performance, the image fusion effect is optimized, making the displayed content present a higher fidelity and enhancing the virtual or augmented reality experience.

[0070] By real-time monitoring the moving trajectories of the fixation points in the user's binocular images, the head movement information can be accurately captured, thereby identifying image displacement or distortion caused by head movement; according to the receiving speed of the new images, the front and rear frame images are matched, and stable and reliable reference regions can be extracted, reducing the time error caused by inconsistent frame rates, reusing historical image data to improve the rendering efficiency, and providing a reliable benchmark for disparity compensation; by comparing the positional relationships of the reference regions within the overlapping range of the current frame left and right eye images, the importance of each region in visual fusion can be evaluated, and key compensation is performed on the parts that have a greater impact on depth perception and stereo fusion; the weighted projection algorithm is used to perform pixel-level dynamic adjustment on the slow-refresh side image to ensure fine fusion of the binocular images while achieving efficient rendering, thereby significantly enhancing the visual image realism and viewing comfort of the near-eye display device.

[0071] Embodiment 2:

[0072] An embodiment of the present invention also provides a device for optimizing the visual image display fidelity of a near-eye display device. The device for optimizing the visual image display fidelity of the near-eye display device may be a near-eye display device such as an augmented reality device (AR), a virtual reality device (VR), or a mixed reality device (MR).

[0073] As Figure 6 shown Figure 6 FIG. is a schematic structural diagram of the hardware operating environment of the device for optimizing the visual image display fidelity of the near-eye display device according to the embodiment of the present invention.

[0074] As Figure 6 shown, the device for optimizing the visual image display fidelity of the near-eye display device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to implement connection communication between these components. The user interface 1003 may include a display (Display) and an input unit such as a control panel. Optionally, the user interface 1003 may further include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WIFI interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001. As a computer storage medium, the memory 1005 may include a program for optimizing the visual image display fidelity.

[0075] Those skilled in the art can understand that Figure 6 the hardware structure shown in

[0076] does not constitute a limitation on the device, and may include more or fewer components than shown, or combine some components, or have a different component layout. Figure 6 , Figure 6 continuing to refer to

[0077] In Figure 6 , the network communication module is mainly used to connect to the server and can communicate with the server for data; while the processor 1001 can call the program for optimizing the visual image display fidelity stored in the memory 1005 and execute the steps in the above respective embodiments.

[0078] Based on the hardware structure of the visual image display fidelity optimization processing device for the near-eye display device above, it is used to implement various embodiments of the visual image display fidelity optimization processing method for the near-eye display device of the present invention.

[0079] In addition, the present invention also provides a visual image display fidelity optimization processing system for a near-eye display device. Please refer to Figure 7 , the visual image display fidelity optimization processing system for the near-eye display device includes: A trajectory analysis module A10, which is used to determine the motion distortion degree of the target area by using the movement information of the target area in the binocular images and the binocular disparity information of each frame during the movement of the user's head; A matching reference module A20, which is used to determine the matching degree of the target area in adjacent frame binocular images, and determine the rendering reference degree of the target area in the rendered monocular image by using the matching degree and the motion distortion degree; A pixel optimization module A30, which is used to determine the pixel adjustment value of the target pixel point in the unrendered monocular image by using the rendering reference degree and the pixel value of the target pixel point in the target area of adjacent frame binocular images; Furthermore, the trajectory analysis module A10 is further used for: Determining the first displacement in adjacent frames of the rendered monocular image and the second displacement in adjacent frames of the unrendered monocular image; Using the first displacement and the second displacement to determine the relative movement amplitude of the target area in the binocular images.

[0080] Furthermore, the trajectory analysis module A10 is further used for: Using the pixel values of the respective target pixel points of the target area in the binocular images to determine the disparity of the target area; Using the disparity of the target area to determine the disparity change rate of the target area per frame relative to the previous frame.

[0081] Furthermore, the trajectory analysis module A10 is further used for: Using the DTW distance between the movement information and the binocular disparity information to determine the motion distortion degree of the target area.

[0082] Furthermore, the matching reference module A20 is further used for: Determining the vector descriptions of the pairs of feature points of the target area in adjacent frame binocular images; Calculating the matching degree of the target area in adjacent frame binocular images by using the vector descriptions of the pairs of feature points and the number of pairs of feature points.

[0083] Furthermore, the matching reference module A20 is further used for: Determining the relationship coefficient between the matching degree and the motion distortion degree; Determine the rendering reference degree of the target area in the rendered monocular image by using the relationship coefficient, matching degree, and motion distortion degree.

[0084] Furthermore, the matching reference module A20 is further configured to: Calculate the rendering reference degree of the target area in the rendered monocular image by using the mean and standard deviation of the relationship coefficient, matching degree, and motion distortion degree respectively.

[0085] Furthermore, the pixel optimization module A30 is further configured to: Determine the compensation contribution degree of the target area in the rendered monocular image by using the rendering reference degree; Determine the pixel adjustment value of the target pixel point in the unrendered monocular image by using the compensation contribution degree and the pixel value of the target pixel point in the binocular image of adjacent frames.

[0086] Furthermore, the pixel optimization module A30 is further configured to: Determine the fixation distance between the target area in the unrendered monocular image and the user's fixation point; Calculate the compensation contribution degree of the target area in the rendered monocular image by using the rendering reference degree and the fixation distance.

[0087] Furthermore, the pixel optimization module A30 is further configured to: Use the binocular fusion filtering algorithm to perform adaptive weight allocation and smoothing processing on the overlapping area in the binocular image after pixel value adjustment to obtain an optimized binocular image.

[0088] The specific implementation manner of the visual image display fidelity optimization processing system of the near-eye display device of the present invention is basically the same as each embodiment of the above-mentioned visual image display fidelity optimization processing method of the near-eye display device, and will not be elaborated here.

[0089] In addition, the present invention also provides a computer-readable storage medium. A visual image display fidelity optimization processing program is stored on the computer-readable storage medium of the present invention. When the visual image display fidelity optimization processing program is executed by a processor, the steps of the above-mentioned visual image display fidelity optimization processing method of the near-eye display device are implemented.

[0090] The method implemented when the visual image display fidelity optimization processing program is executed can refer to each embodiment of the visual image display fidelity optimization processing method of the near-eye display device of the present invention, and will not be elaborated here.

[0091] It should be noted that: the above order of the embodiments of the present invention is only for description and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0092] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. The key point of each embodiment is to illustrate the differences from other embodiments.

[0093] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) that contain computer-usable program code.

[0094] The above are only the preferred embodiments of the present invention, and do not limit the protection scope of the present invention. Any equivalent structure / method transformation made by using the content of the specification and drawings of the present invention under the inventive concept of the present invention, or any direct / indirect application in other related technical fields is included in the protection scope of the present invention.

Claims

1. A method for optimizing the fidelity of visual image display of a near-eye display device, characterized in that: The method comprises: When the user's head moves, the motion distortion of the target area is determined by using the motion information of the target area in the binocular image and the binocular disparity information of each frame; Determine the matching degree of the target area in the binocular images of adjacent frames, and determine the rendering reference degree of the target area in the rendered monocular image using the matching degree and the motion distortion degree; Determine the pixel adjustment value of the target pixel in the unrendered monocular image by using the rendering reference degree and the pixel value of the target pixel in the target area in the adjacent frame binocular image; The adjacent frame binocular images include a rendered monocular image of a previous frame and an unrendered monocular image of a subsequent frame.

2. The method for optimizing the fidelity of visual image display of a near-eye display device according to claim 1, characterized in that: Determine the motion information of the target area in the binocular image, including: Determining a first displacement in adjacent frames of the rendered monocular image and a second displacement in adjacent frames of the unrendered monocular image; The relative movement amplitude of the target area in the binocular image is determined by using the first displacement and the second displacement.

3. The method for optimizing the fidelity of visual image display of a near-eye display device according to claim 1, characterized in that: Determine the binocular disparity information of each frame of the target area in the binocular image, including: Determine the disparity of the target area by using the pixel values ​​of the respective target pixel points in the target area in the binocular image; The disparity of the target area is used to determine the disparity change rate of each frame of the target area relative to the previous frame.

4. The method for optimizing the fidelity of visual image display of a near-eye display device according to claim 1, characterized in that: The method of determining the motion distortion of the target area by using the movement information of the target area in the binocular image and the binocular disparity information of each frame includes: The DTW distance between the motion information and the binocular disparity information is used to determine the motion distortion of the target area.

5. The method for optimizing the fidelity of visual image display of a near-eye display device according to claim 1, characterized in that: The determining of the matching degree of the target area in the binocular images of adjacent frames includes: Determine the vector description of each pair of feature points of the target area in the binocular images of adjacent frames; Using the vector description of each pair of feature points and the number of each pair of feature points, the matching degree of the target area in the binocular images of adjacent frames is calculated.

6. The method for optimizing the fidelity of visual image display of a near-eye display device according to claim 1, characterized in that: The method of determining the rendering reference degree of the target area in the rendered monocular image by using the matching degree and the motion distortion degree includes: Determine the relationship coefficient between the matching degree and the motion distortion degree; The rendering reference degree of the target area in the rendered monocular image is determined by using the relationship coefficient, matching degree and motion distortion degree.

7. The method for optimizing the fidelity of visual image display of a near-eye display device according to claim 6, characterized in that: The method of determining the rendering reference degree of the target area in the rendered monocular image by using the relationship coefficient, the matching degree and the motion distortion degree includes: The rendering reference degree of the target area in the rendered monocular image is calculated using the mean and standard deviation of the relationship coefficient, matching degree and motion distortion degree.

8. The method for optimizing the fidelity of visual image display of a near-eye display device according to claim 1, characterized in that: The method of determining the pixel adjustment value of the target pixel in the unrendered monocular image by using the rendering reference degree and the pixel value of the target pixel in the target area in the adjacent frame binocular image comprises: The compensation contribution of the target area in the rendered monocular image is determined using the rendering reference degree; The pixel adjustment value of the target pixel in the unrendered monocular image is determined by using the compensation contribution and the pixel value of the target pixel in the target area in the adjacent frame binocular image.

9. The method for optimizing the fidelity of visual image display of a near-eye display device according to claim 8, characterized in that: The method of determining the compensation contribution of the target area in the rendered monocular image by using the rendering reference degree includes: Determine the gaze distance between the target area and the user's gaze point in the unrendered monocular image; The compensation contribution of the target area in the rendered monocular image is calculated using the rendering reference degree and gaze distance.

10. The method for optimizing the fidelity of visual image display of a near-eye display device according to claim 1, characterized in that: After determining the pixel adjustment value of the target pixel in the unrendered monocular image, the method further includes: The binocular fusion filtering algorithm is used to perform adaptive weight allocation and smoothing on the overlapping areas of the binocular image after pixel value adjustment to obtain the optimized binocular image.

Images