Method for improving the detail display quality of VR devices by using binocular difference mechanism

Abstract

The present invention relates to the technical field of image data processing, and particularly relates to a method for improving the detail display quality of VR devices by using a binocular difference mechanism. The method includes: obtaining the binocular images and the user's gaze behavior characteristics of the current frame of the VR device; the user's gaze behavior characteristics include the gaze area and the fixation point; determining the disparity saliency of the gaze area according to the disparity map of the binocular images; determining the initial brightness interval and the binocular brightness difference degree according to the brightness level of the corresponding point of the fixation point in the binocular images; determining the brightness level adjustment weight according to the disparity saliency, the binocular brightness difference degree and the user's gaze behavior characteristics; adjusting the initial brightness interval according to the brightness level adjustment weight to obtain the adjusted brightness interval; adjusting the brightness of the binocular images according to the adjusted brightness interval, and obtaining the binocular images currently output by the VR device based on the adjusted binocular images. The problem of display delay is avoided, and the display detail quality is guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and particularly relates to a method for improving the detail display quality of VR devices by using a binocular difference mechanism. Background Art

[0002] Currently, the display technologies applied to traditional displays usually improve the display details by increasing the bit depth. For example, the 10-bit display technology that has emerged in recent years has more display details than the traditional 8-bit display technology, including brightness and chromaticity details. However, such an improvement method has three major defects: 1. The color management of the device and the processing of pictures need to be carried out according to 10 bits, which increases the processing time and resource requirements; 2. All pictures and videos need to be encoded using 10 bits, which increases the storage capacity of the files; 3. The bit depth effect is limited by the human eye mechanism (Weber's law), and the bit depth gain is small at low brightness.

[0003] To address the above problems, generally, the color values or brightness values of the pixels around the fixation point are adjusted frame by frame by a fixed amplitude to increase the display details. However, when a VR (Virtual Reality) device is displaying, due to the tunneling effect, the attention degree of the position adjacent to the fixation point is significantly higher than that of the other positions, so a higher adjustment input is required for the position adjacent to the fixation point. Therefore, after the user transfers the fixation point, the details of the displayed content need to be adjusted, which easily leads to the problem of display delay. Summary of the Invention

[0004] In order to solve the technical problem of avoiding display delay while ensuring the display detail quality, the purpose of the present invention is to provide a method for improving the detail display quality of VR devices by using a binocular difference mechanism. The specific technical solution adopted is as follows:

[0005] The present invention provides a method for improving the detail display quality of VR devices by using a binocular difference mechanism. The method includes:

[0006] Obtain the binocular images of the current frame of the VR device and the user's fixation behavior characteristics; the user's fixation behavior characteristics include the fixation area and the fixation point;

[0007] Determine the disparity saliency of the fixation area according to the disparity map of the binocular images;

[0008] Determine the initial brightness interval and the binocular brightness difference degree according to the brightness level of the corresponding point of the fixation point in the binocular images;

[0009] Determine the brightness level adjustment weight according to the disparity saliency, the binocular brightness difference degree, and the user's fixation behavior characteristics;

[0010] Adjust the initial brightness interval according to the adjusted weight of the brightness level to obtain an adjusted brightness interval;

[0011] Adjust the brightness of the binocular images according to the adjusted brightness interval, and obtain the binocular images currently output by the VR device based on the adjusted binocular images.

[0012] According to the method for improving the detail display quality of a VR device by using a binocular difference mechanism provided by the present invention, determining the disparity saliency of the fixation area according to the disparity map of the binocular images includes:

[0013] Calculate the disparity map of the binocular images;

[0014] Calculate the total disparity value corresponding to the fixation area in the disparity map, and calculate the total disparity value of the entire disparity map;

[0015] Determine the disparity saliency of the fixation area according to the ratio of the total disparity value corresponding to the fixation area to the total disparity value of the entire map.

[0016] According to the method for improving the detail display quality of a VR device by using a binocular difference mechanism provided by the present invention, determining the initial brightness interval and the binocular brightness difference degree according to the brightness level of the corresponding point of the fixation point in the binocular images includes:

[0017] Map the fixation point in one of the binocular images to the corresponding point in the other binocular image through the disparity map;

[0018] Use the brightness level of the fixation point in one of the binocular images and the brightness level of the corresponding point in the other binocular image as the interval endpoints respectively to determine the initial brightness interval;

[0019] Determine the binocular brightness difference degree according to the ratio of the order span of the initial brightness interval to the total number of brightness levels of the entire map.

[0020] According to the method for improving the detail display quality of a VR device by using a binocular difference mechanism provided by the present invention, determining the adjusted weight of the brightness level according to the disparity saliency, the binocular brightness difference degree, and the user's fixation behavior characteristics includes:

[0021] Determine the initial rendering weight of each frame of binocular images according to the user's fixation behavior characteristics corresponding to each frame of binocular images;

[0022] Calculate the detail retention degree of each frame of binocular images, and determine the detail display frame segment according to the change of the detail retention degree;

[0023] Determine the intersection binocular images according to the intersection points of the two sequences composed of the initial rendering weights of each frame of binocular images in the detail display frame segment;

[0024] Determine a brightness level adjustment weight based on the parallax saliency, the binocular brightness difference degree, and the difference in the initial rendering weights of the binocular images at the intersection point.

[0025] According to the method for improving the detail display quality of a VR device by using a binocular difference mechanism provided by the present invention, the determining of the initial rendering weights of each frame of binocular images according to the user's gaze behavior characteristics corresponding to each frame of binocular images includes:

[0026] For each eye image in each frame of binocular images, determine the degree of line-of-sight change and the angular difference between the line-of-sight deflection and the device deflection according to the corresponding user's gaze behavior characteristics;

[0027] Determine the proportion of color levels in the gaze area according to the ratio between the number of color levels in the gaze area and the total number of color levels in the eye image;

[0028] Determine the initial rendering weight of the eye image according to the degree of line-of-sight change, the angular difference between the line-of-sight deflection and the device deflection, and the proportion of color levels in the gaze area.

[0029] According to the method for improving the detail display quality of a VR device by using a binocular difference mechanism provided by the present invention, the user's gaze behavior characteristics further include the device deflection angle;

[0030] The determining of the degree of line-of-sight change and the angular difference between the line-of-sight deflection and the device deflection according to the corresponding user's gaze behavior characteristics includes:

[0031] Determine the deflection vectors corresponding to the current frame and the previous frame respectively; the deflection vector is determined according to the change in the gaze points of two adjacent frames;

[0032] According to the deflection vector corresponding to the current frame, determine the line-of-sight change distance, and determine the degree of line-of-sight change according to the ratio between the line-of-sight change distance and the diameter of the gaze area;

[0033] Determine the line-of-sight deflection angle according to the included angle between the deflection vectors corresponding to the current frame and the previous frame respectively;

[0034] Determine the angular difference between the line-of-sight deflection and the device deflection according to the difference between the line-of-sight deflection angle and the device deflection angle.

[0035] According to the method for improving the detail display quality of a VR device by using a binocular difference mechanism provided by the present invention, an eye movement tracking module and an inertial measurement unit are included in the VR device;

[0036] The eye movement tracking module is used to capture the gaze area in real time;

[0037] The inertial measurement unit is used to determine the deflection angle of the device.

[0038] According to the method for improving the detail display quality of a VR device by using a binocular difference mechanism provided by the present invention, calculating the detail retention degree of each frame of binocular images includes:

[0039] For each eye image in each frame of binocular images respectively, determine the color level histogram of the eye image in each channel in the LAB color space;

[0040] According to the separation degree of color levels in the color level histogram, determine the single-channel detail retention degree of the eye image in each channel;

[0041] According to the sum of the single-channel detail retention degrees in each channel, determine the detail retention degree of the eye image.

[0042] According to the method for improving the detail display quality of a VR device by using a binocular difference mechanism provided by the present invention, determining the detail display frame segment according to the change of the detail retention degree includes:

[0043] According to the detail retention degrees of each frame of binocular images, calculate the change rate of the detail retention degree;

[0044] Determine the target frame position closest to the current frame from the frame positions where the positive and negative signs of the change rate change, and determine the frame segment from the target frame position to the current frame as the detail display frame segment.

[0045] According to the method for improving the detail display quality of a VR device by using a binocular difference mechanism provided by the present invention, adjusting the initial brightness interval according to the brightness level adjustment weight to obtain the adjusted brightness interval includes:

[0046] Take the brightness level adjustment weight as the adjustment weight corresponding to the fixation area;

[0047] Iteratively expand the area outward from the fixation area to determine the adjustment weight corresponding to each expanded area; the adjustment weight corresponding to the expanded area decreases with the increase of the iteration times on the basis of the brightness level adjustment weight;

[0048] According to the adjustment weights corresponding to each area, adjust the initial brightness interval to obtain the adjusted brightness intervals corresponding to each area.

[0049] The present invention has the following beneficial effects:

[0050] By determining the disparity saliency of the fixation area based on the disparity map of the binocular images of the current frame, the display details and the matching distortion in the fixation area can be accurately evaluated. By determining the initial brightness interval and the binocular brightness difference degree according to the brightness level of the corresponding point of the fixation point in the binocular images, the brightness difference of the binocular images can be accurately measured. Then, according to the disparity saliency, the binocular brightness difference degree, and the user's fixation behavior characteristics, the brightness level adjustment weight is determined, which can dynamically determine the brightness level adjustment weight by combining the display details and the matching distortion in the fixation area, the brightness difference of the binocular images, and the user's fixation situation, and dynamically determine which areas are preferentially rendered. Then, according to the brightness level adjustment weight, the initial brightness interval is adjusted to obtain the adjusted brightness interval. According to the adjusted brightness interval, the brightness of the binocular images is adjusted, and the current output binocular images of the VR device are obtained based on the adjusted binocular images, avoiding the problem of easy display delay caused by adjusting the pixel points around the fixation point according to a fixed amplitude after the user transfers the fixation point, and ensuring the display detail quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the accompanying drawings required for the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0052] Figure 1 It is a flowchart of a method for improving the detail display quality of a VR device by using a binocular difference mechanism provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0053] In order to further elaborate on the technical means and effects adopted by the present invention to achieve the intended invention purpose, the following describes in detail the specific implementation manner, structure, features, and effects of a method for improving the detail display quality of a VR device by using a binocular difference mechanism according to the present invention in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs.

[0055] The following specifically describes the specific solution of a method for improving the detail display quality of a VR device by using a binocular difference mechanism provided by the present invention in combination with the accompanying drawings.

[0056] Please refer to Figure 1 , which shows a schematic flowchart of a method for improving the detail display quality of a VR device by using a binocular difference mechanism provided in an embodiment of the present invention, including the following steps:

[0057] Step 101, obtain the binocular images and user gaze behavior characteristics of the current frame of the VR device; the user gaze behavior characteristics include the gaze area and the fixation point.

[0058] Among them, the VR device refers to a virtual reality device. The binocular images include a left-eye image and a right-eye image.

[0059] In an embodiment, the user gaze behavior characteristics may further include the device deflection angle in addition to the gaze area and the fixation point. Among them, the fixation point may be the center point of the gaze area. The device deflection angle refers to the deflection angle of the VR device.

[0060] In an embodiment, the VR device may include an eye-tracking module, an inertial measurement unit, a GPU (Graphics Processing Unit), and a display screen. Among them, the eye-tracking module may be an integrated high-precision eye-tracking module, such as: a pupil center corneal reflection (PCCR) sensor, which is used to capture the gaze area in real time. The inertial measurement unit (IMU) is used to measure the real-time posture and movement direction of the VR device, so as to obtain the device deflection angle. The inertial measurement unit may be integrated in the eye-tracking module. The GPU may be a high-performance GPU, which can support 8-bit color image processing and support the VR device. The GPU obtains the left-eye image and the right-eye image with frame bit order alignment in a continuous frame manner, and executes the method for improving the detail display quality of the VR device by using the binocular difference mechanism in each embodiment of the present invention. The display screen may be a high-resolution display screen, which supports at least binocular 4K resolution and an OLED (Organic Light-Emitting Diode) or Micro-LED (Micro-Light-Emitting Diode) screen with a color depth of 10 bits or more, to ensure the accurate presentation of high dynamic range (HDR) content.

[0061] Step 102, determine the disparity saliency of the gaze area according to the disparity map of the binocular images.

[0062] Among them, the disparity map is used to reflect the pixel offset between the left-eye image and the right-eye image. Disparity saliency is used to measure the disparity saliency degree of the fixation area.

[0063] In one embodiment, the binocular images can be matched in the NCC (Normalized Cross-Correlation) manner to obtain a disparity map.

[0064] In one embodiment, the total disparity value corresponding to the fixation area in the disparity map can be calculated, and the total disparity value of the entire disparity map can be calculated. According to the total disparity value corresponding to the fixation area and the total disparity value of the entire map, the disparity saliency of the fixation area can be determined.

[0065] Step 103: Determine the initial brightness interval and the binocular brightness difference degree according to the brightness level of the corresponding point of the fixation point in the binocular images.

[0066] Among them, the binocular brightness difference degree is used to measure the difference size between the brightness levels of the corresponding points of the fixation point in the binocular images.

[0067] In one embodiment, the brightness levels of the corresponding points of the fixation point in the binocular images can be used as the interval endpoints respectively to determine the initial brightness interval.

[0068] In one embodiment, the binocular brightness difference degree can be determined according to the order span of the initial brightness interval and the brightness levels of the entire map.

[0069] Step 104: Determine the brightness level adjustment weight according to the disparity saliency, the binocular brightness difference degree, and the user's fixation behavior characteristics.

[0070] In one embodiment, the initial rendering weight of each frame of binocular images can be determined according to the user's fixation behavior characteristics, the intersection binocular image can be determined according to the initial rendering weights of each frame of images, and the difference in the initial rendering weights of the intersection binocular image can be determined.

[0071] In one embodiment, the brightness level adjustment weight is positively correlated with the disparity saliency. The brightness level adjustment weight is positively correlated with the binocular brightness difference degree. The brightness level adjustment weight is positively correlated with the difference in the initial rendering weights of the intersection binocular image.

[0072] Step 105: Adjust the initial brightness interval according to the brightness level adjustment weight to obtain the adjusted brightness interval.

[0073] In one embodiment, calculate the value of 1 minus the brightness level adjustment weight. The right endpoint of the initial brightness interval can be multiplied by the brightness level adjustment weight, and the left endpoint of the initial brightness interval can be multiplied by the value of 1 minus the brightness level adjustment weight to obtain the adjusted brightness interval.

[0074] In one embodiment, the brightness level adjustment weight can be used as the adjustment weight corresponding to the fixation area, and the adjustment weight corresponding to the area outside the fixation area can be obtained by reducing it on the basis of the brightness level adjustment weight. According to the adjustment weights corresponding to each area respectively, the initial brightness interval is adjusted to obtain the adjusted brightness intervals corresponding to each area respectively.

[0075] Step 106: Adjust the brightness of the binocular images according to the adjusted brightness intervals, and obtain the binocular images currently output by the VR device based on the adjusted binocular images.

[0076] In one embodiment, according to the adjusted brightness intervals, the brightness of the binocular images is adjusted, with the main line-of-sight image having its brightness increased and the secondary line-of-sight image having its brightness decreased.

[0077] In one embodiment, the detail retention degree of the binocular images can be calculated, the magnitudes of the detail retention degrees corresponding to the left-eye image and the right-eye image in the binocular images are compared, and the left-eye image / right-eye image with the larger detail retention degree is used as the main line-of-sight image, and the right-eye image / left-eye image with the smaller detail retention degree is used as the secondary line-of-sight image.

[0078] In one embodiment, after corresponding the image coordinates of the binocular images through the disparity map, they are input into a binocular matching unit restricted by the NCC method to obtain the binocular images currently output by the VR device.

[0079] In the above method for improving the detail display quality of the VR device using the binocular difference mechanism, by determining the disparity saliency of the fixation area according to the disparity map of the binocular images of the current frame, the display details and the matching distortion conditions existing in the fixation area can be accurately evaluated. According to the brightness level of the corresponding point of the fixation point in the binocular images, the initial brightness interval and the binocular brightness difference degree are determined, which can accurately measure the brightness difference of the binocular images. Then, according to the disparity saliency, the binocular brightness difference degree and the user's fixation behavior characteristics, the brightness level adjustment weight is determined, which can dynamically determine the brightness level adjustment weight by combining the display details and the matching distortion conditions existing in the fixation area, the brightness difference of the binocular images and the user's fixation situation, and dynamically determine which areas are preferentially rendered. Then, according to the brightness level adjustment weight, the initial brightness interval is adjusted to obtain the adjusted brightness interval. According to the adjusted brightness interval, the brightness of the binocular images is adjusted, and the binocular images currently output by the VR device are obtained based on the adjusted binocular images, avoiding the problem of easy display delay caused by adjusting the pixel points around the fixation point according to a fixed amplitude after the user transfers the fixation point, and ensuring the display detail quality.

[0080] In one embodiment, determining the disparity saliency of the fixation region based on the disparity map of the binocular images includes: calculating the disparity map of the binocular images; calculating the total disparity value corresponding to the fixation region in the disparity map, and calculating the total disparity value of the entire disparity map; determining the disparity saliency of the fixation region according to the ratio of the total disparity value corresponding to the fixation region to the total disparity value of the entire map.

[0081] In one embodiment, the disparity saliency of the fixation region can be calculated according to the following formula:

[0082]

[0083] where represents the disparity saliency of the fixation region . represents the total disparity value corresponding to the fixation region in the disparity map. represents the total disparity value of the entire disparity map.

[0084] In the above embodiment, determining the disparity saliency of the fixation region according to the ratio of the total disparity value corresponding to the fixation region to the total disparity value of the entire map can accurately evaluate the display details and the matching distortion situation in the fixation region.

[0085] In one embodiment, determining the initial brightness interval and the binocular brightness difference degree according to the brightness level of the corresponding point of the fixation point in the binocular images includes: mapping the fixation point in one of the binocular images to the corresponding point in the other image through the disparity map; using the brightness level of the fixation point in one image and the brightness level of the corresponding point in the other image as the interval endpoints respectively to determine the initial brightness interval; determining the binocular brightness difference degree according to the ratio of the order span of the initial brightness interval to the total brightness order of the entire map.

[0086] For example: the fixation point is mapped to the fixation point in the other image through the disparity map , and the brightness levels , of these two points are extracted, then the initial brightness interval is (assuming that the brightness of the fixation point is greater than that of the fixation point ).

[0087] In one embodiment, the binocular brightness difference degree can be calculated according to the following formula:

[0088]

[0089] where represents the binocular brightness difference degree. represents the order span of the initial brightness interval. represents the total brightness order of the entire map.

[0090] In the above embodiments, according to the ratio of the order span of the initial brightness interval to the brightness order of the entire image, the binocular brightness difference degree is determined, which can accurately measure the brightness difference of binocular images, and thus can be used to adjust the brightness order between corresponding points in the binocular images, so as to avoid distortion while enhancing details.

[0091] In one embodiment, according to the disparity saliency, the binocular brightness difference degree, and the user's fixation behavior characteristics, the brightness order adjustment weight is determined, including: determining the initial rendering weight of each frame of binocular images according to the user's fixation behavior characteristics corresponding to each frame of binocular images; calculating the detail retention degree of each frame of binocular images, and determining the detail display frame segment according to the change of the detail retention degree; determining the intersection binocular image according to the intersection of two sequences composed of the initial rendering weights of each frame of binocular images in the detail display frame segment; and determining the brightness order adjustment weight according to the disparity saliency, the binocular brightness difference degree, and the difference of the initial rendering weights of the intersection binocular image.

[0092] Among them, the detail display frame segment is the frame segment from the target frame position corresponding to the sudden change in the change direction of the detail retention degree last time to the current frame.

[0093] In one embodiment, the initial rendering weights of each frame of left-eye images and each frame of right-eye images in the detail display frame segment form two initial rendering weight sequences, and the target intersection closest to the current frame is selected from the intersections of the two initial rendering weight sequences, and the binocular image corresponding to the target intersection is determined as the intersection binocular image.

[0094] In one embodiment, the brightness order adjustment weight can be calculated according to the following formula:

[0095]

[0096] Among them, represents the brightness order adjustment weight of the fixation area . represents the disparity saliency of the fixation area . represents the binocular brightness difference degree. represents the initial rendering weight of any one eye image in the intersection binocular image, represents the initial rendering weight of the other eye image in the intersection binocular image. represents the difference of the initial rendering weights of the intersection binocular image, reflecting the deviation of the image display details at the actual fixation position.

[0097] In the above embodiments, since the situation where the user's line of sight changes violently often occurs after the display material attracts the user's line of sight to a sufficient extent, and the direction of the user's gaze interacts with the content displayed by the material, the initial rendering weight is used to evaluate the user's perspective movement environment. By extracting the variation period of the user's initial rendering weight, combining the texture complexity represented by the parallax saliency with the binocular brightness difference degree, and jointly setting the restricted range along with the direction of the user's visual line of sight of the image represented by the initial rendering weight, so as to limit the specific scale of brightness adjustment, it can ensure high-quality display details are output without image distortion.

[0098] In one embodiment, according to the user's gaze behavior characteristics corresponding to each frame of binocular images, determining the initial rendering weight of each frame of binocular images includes: for each eye image in each frame of binocular images respectively, determining the degree of line of sight change and the angular difference between the line of sight deflection and the device deflection according to the corresponding user's gaze behavior characteristics; determining the proportion of color levels in the gaze area according to the ratio between the number of color levels in the gaze area and the total number of color levels in one eye image; and determining the initial rendering weight of one eye image according to the degree of line of sight change, the angular difference between the line of sight deflection and the device deflection, and the proportion of color levels in the gaze area.

[0099] In one embodiment, the initial rendering weight is positively correlated with the degree of line of sight change. The initial rendering weight is negatively correlated with the angular difference between the line of sight deflection and the device deflection. The initial rendering weight is negatively correlated with the proportion of color levels in the gaze area.

[0100] In one embodiment, the cosine value of the angular difference between the line of sight deflection and the device deflection can be determined, and the initial rendering weight is determined according to the product of the degree of line of sight change and the cosine value divided by the proportion of color levels in the gaze area.

[0101] In one embodiment, the initial rendering weight can be calculated according to the following formula:

[0102]

[0103] Wherein, represents the initial rendering weight of one eye image of. represents the degree of line of sight change. represents the line of sight deflection angle. represents the device deflection angle. represents the angular difference between the line of sight deflection and the device deflection. represents the cosine function. represents the proportion of color levels in the gaze area.

[0104] In one embodiment, the proportion of color levels in the gaze area can be calculated according to the following formula:

[0105]

[0106] Among them, represents the proportion of the color level in the fixation area. represents the fixation area the number of color levels within. represents the total number of color levels in the single-eye image. It can be understood that the number of color levels in the fixation area can describe the texture richness within the fixation area. By determining the ratio between the number of color levels in the fixation area and the total number of color levels in the single-eye image, the proportion of the color level in the fixation area can be determined, and it can be judged whether the fixation point falls on an important texture area.

[0107] In the above embodiments, the greater the degree of eye movement, the higher the initial rendering weight is required to meet the amplitude adjustment ratio generated by the large-scale change of the user's line of sight. The smaller the angular difference between the line-of-sight deflection and the device deflection, the more active the user's viewing behavior is, and the higher the initial rendering weight is required to meet the purpose of detailed display. The greater the proportion of the color level in the fixation area, the more it indicates that the fixation point falls on an important texture area, and the lower the initial rendering weight is required. Therefore, by determining the initial rendering weight of the single-eye image according to the degree of eye movement, the angular difference between the line-of-sight deflection and the device deflection, and the proportion of the color level in the fixation area, the rendering accuracy can be adaptively optimized and invalid calculations can be avoided.

[0108] In one embodiment, the user's fixation behavior characteristics further include the device deflection angle; according to the corresponding user's fixation behavior characteristics, determining the degree of eye movement and the angular difference between the line-of-sight deflection and the device deflection includes: determining the deflection vectors corresponding to the current frame and the previous frame respectively; the deflection vector is determined according to the change of the fixation points in two adjacent frames; according to the deflection vector corresponding to the current frame, determining the eye movement distance, and according to the ratio between the eye movement distance and the diameter of the fixation area, determining the degree of eye movement; according to the included angle between the deflection vectors corresponding to the current frame and the previous frame respectively, determining the line-of-sight deflection angle; according to the difference between the line-of-sight deflection angle and the device deflection angle, determining the angular difference between the line-of-sight deflection and the device deflection.

[0109] In one embodiment, the deflection vector corresponding to the current frame is a vector pointing from the fixation point of the previous frame to the fixation point of the current frame . The deflection vector corresponding to the previous frame is a vector pointing from the fixation point of the frame before the previous frame to the fixation point of the previous frame . The included angle between the deflection vector corresponding to the current frame and the deflection vector corresponding to the previous frame is used as the line-of-sight deflection angle.

[0110] In one embodiment, the modulus of the deflection vector corresponding to the current frame can be used as the line-of-sight change distance. The degree of line-of-sight change can be calculated according to the following formula:

[0111]

[0112] where represents the degree of line-of-sight change. represents the deflection vector corresponding to the current frame. represents the line-of-sight change distance. represents the diameter of the fixation area of.

[0113] In one embodiment, the device deflection angle can be the deflection angle of the VR device relative to the plane XoZ where the display screen is located . The angular difference between the line-of-sight deflection and the device deflection can be expressed as .

[0114] In the above embodiment, according to the deflection vector corresponding to the current frame, the line-of-sight change distance is determined. According to the ratio between the line-of-sight change distance and the diameter of the fixation area, the degree of line-of-sight change is determined. According to the included angle between the deflection vectors corresponding to the current frame and the previous frame respectively, the line-of-sight deflection angle is determined. According to the difference between the line-of-sight deflection angle and the device deflection angle, the angular difference between the line-of-sight deflection and the device deflection is determined, which can accurately measure the degree of line-of-sight change and the angular difference between the line-of-sight deflection and the device deflection.

[0115] In one embodiment, the VR device includes an eye movement tracking module and an inertial measurement unit; the eye movement tracking module is used to capture the fixation area in real time; the inertial measurement unit is used to determine the device deflection angle.

[0116] In the above embodiment, by capturing the fixation area in real time through the eye movement tracking module and determining the device deflection angle through the inertial measurement unit, it can assist in the execution of the method for improving the detail display quality of the VR device by using the binocular difference mechanism provided by the present invention.

[0117] In one embodiment, calculating the detail retention degree of each frame of binocular images includes: respectively for each eye image in each frame of binocular images, determining the color level histogram of each channel of the eye image in the LAB color space; according to the separation degree of the color levels in the color level histogram, determining the single-channel detail retention degree of the eye image in each channel; according to the sum of the single-channel detail retention degrees in each channel, determining the detail retention degree of the eye image.

[0118] In one embodiment, the Lab color space includes an L channel (luminance channel), an A channel (color channel from dark green to gray to bright pink), and a B channel (color channel from bright blue to gray to yellow).

[0119] In one embodiment, for each channel in each monocular image, the number of pixel points at each color level is determined according to the color level histogram of the image in that channel, and the number of pixel points in the color level with the largest number of pixel points is determined as the maximum value of the color level pixel points, and the standard deviation of the number of pixel points at each color level is determined. According to the difference between the number of pixel points at each color level in the channel and the maximum value of the color level pixel points divided by the standard deviation, the single-channel detail retention degree of the channel is determined.

[0120] In one embodiment, the single-channel detail retention degree can be calculated according to the following formula:

[0121]

[0122] where, represents a monocular image The single-channel detail retention degree under channel E. represents the number of color levels under channel E. represents a single color level in channel E The number of pixel points below. represents the maximum value of the color level pixel points, that is, the number of pixel points in the color level with the largest number of pixel points in channel E. represents the standard deviation of the number of pixel points at each color level in channel E. It can be understood that according to the difference between the number of pixel points at each color level in the channel and the maximum value of the color level pixel points divided by the standard deviation, the separation degree of the color levels in the channel can be reflected. The greater the separation degree, the more likely it is that the channel reflects the detail information of the image with more color levels.

[0123] In one embodiment, the magnitudes of the detail retention degrees corresponding to the left-eye image and the right-eye image in the binocular image can be compared, and the left-eye image / right-eye image with the larger detail retention degree is used as the main line-of-sight image, and the right-eye image / left-eye image with the smaller detail retention degree is used as the secondary line-of-sight image. Thus, the determination of the actual orientation of the important information in the binocular image is achieved.

[0124] In the above embodiments, for each eye image in each frame of binocular images, the color level histogram of each eye image in each channel of the LAB color space is determined, and according to the separation degree of the color levels in the color level histogram, the single-channel detail retention degree of the eye image in each channel is determined, which can accurately measure the richness of the detail information in the image. In addition, since only the image channels and their numbers are extracted for linear operations, the detail retention degree of the binocular images is obtained without incurring additional computational overhead, and the judgment of the actual orientation of the important information in the binocular images is realized.

[0125] In one embodiment, according to the change of the detail retention degree, a detail display frame segment is determined, including: calculating the change rate of the detail retention degree according to the detail retention degrees of each frame of binocular images; determining the target frame position closest to the current frame from the frame positions where the positive and negative signs of the change rate change, and determining the frame segment from the target frame position to the current frame as the detail display frame segment.

[0126] In one embodiment, the change rate of the detail retention degree can be calculated by subtracting 1 from the ratio of the detail retention degrees of two adjacent frames. The change rate of the detail retention degree can be calculated by the following formula:

[0127]

[0128] where represents the change rate of the detail retention degree. represents the detail retention degree of the eye image p, that is, the sum of the single-channel detail retention degrees of the eye image p in each channel. represents the detail retention degree of the next-frame eye image p + 1, that is, the sum of the single-channel detail retention degrees of the eye image p + 1 in each channel.

[0129] In the above embodiments, according to the detail retention degrees of each frame of binocular images, the change rate of the detail retention degree is calculated, the target frame position closest to the current frame is determined from the frame positions where the positive and negative signs of the change rate change, and the frame segment from the target frame position to the current frame is determined as the detail display frame segment, which can accurately determine the detail display frame segment.

[0130] In one embodiment, according to the brightness level adjustment weight, the initial brightness interval is adjusted to obtain an adjusted brightness interval, including: taking the brightness level adjustment weight as the adjustment weight corresponding to the fixation area; iteratively expanding the area outward from the fixation area to determine the adjustment weight corresponding to each expanded area; the degree of reduction of the adjustment weight corresponding to the expanded area on the basis of the brightness level adjustment weight is positively correlated with the number of iterations; adjusting the initial brightness interval according to the adjustment weights corresponding to each area to obtain the adjusted brightness intervals corresponding to each area.

[0131] In one embodiment, in each iteration, the diameter of the expanded area can be doubled centered on the fixation point to obtain a newly expanded area. Each expanded area is an annulus (excluding the fixation area and the previously expanded areas).

[0132] In one embodiment, the adjustment weight corresponding to the expanded area can be determined according to the following formula:

[0133]

[0134] Wherein, represents the adjustment weight corresponding to the currently expanded area. represents the brightness level adjustment weight, that is, the adjustment weight corresponding to the fixation area. represents the current iteration number. represents the total iteration number. represents the degree to which the adjustment weight corresponding to the expanded area decreases based on the brightness level adjustment weight.

[0135] In one embodiment, the adjusted brightness interval corresponding to any area can be expressed as:

[0136]

[0137] Wherein, the initial brightness interval is . represents the adjustment weight corresponding to any area.

[0138] In the above embodiment, the degree to which the adjustment weight corresponding to the expanded area decreases based on the brightness level adjustment weight is positively correlated with the iteration number. Therefore, the adjustment weight corresponding to the expanded area is negatively correlated with the iteration number, such that the adjustment weight of the expanded area closer to the fixation area is larger, and the adjustment weight of the expanded area farther from the fixation area is smaller, thereby enabling accurate adjustment of the brightness of the pixel points in each area.

[0139] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0140] The above-described embodiments merely represent several implementation manners of the present application. Their descriptions are relatively specific and detailed, but they should not be construed as limiting the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several deformations and improvements can still be made, and these all belong to the protection scope of the present application.

[0141] It should be noted that the above-mentioned order of the embodiments of the present invention is only for description and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0142] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments.

Claims

1. A method for improving the detail display quality of VR devices by using the binocular difference mechanism, characterized in that The method includes: Obtaining the binocular images and user gaze behavior characteristics of the current frame of the VR device; the user gaze behavior characteristics include the gaze area and the fixation point; Determining the disparity saliency of the gaze area according to the disparity map of the binocular images; Determining the initial brightness interval and the binocular brightness difference degree according to the brightness level of the corresponding point of the fixation point in the binocular images; Determining the brightness level adjustment weight according to the disparity saliency, the binocular brightness difference degree, and the user gaze behavior characteristics; Adjusting the initial brightness interval according to the brightness level adjustment weight to obtain the adjusted brightness interval; Adjusting the brightness of the binocular images according to the adjusted brightness interval, and obtaining the binocular images currently output by the VR device based on the adjusted binocular images.

2. The method for improving the detail display quality of a VR device by using a binocular difference mechanism according to claim 1, characterized in that, The determining the disparity saliency of the gaze area according to the disparity map of the binocular images includes: Calculating the disparity map of the binocular images; Calculating the total disparity value corresponding to the gaze area in the disparity map, and calculating the total disparity value of the entire disparity map; Determining the disparity saliency of the gaze area according to the ratio of the total disparity value corresponding to the gaze area to the total disparity value of the entire map.

3. The method for improving the detail display quality of a VR device by using a binocular difference mechanism according to claim 1, characterized in that, The determining the initial brightness interval and the binocular brightness difference degree according to the brightness level of the corresponding point of the fixation point in the binocular images includes: Mapping the fixation point in one eye image of the binocular images to the corresponding point in the other eye image through the disparity map; Taking the brightness levels of the fixation point in the one eye image and the corresponding point in the other eye image as the interval endpoints respectively to determine the initial brightness interval; Determining the binocular brightness difference degree according to the ratio of the order span of the initial brightness interval to the total number of brightness levels of the entire map.

4. The method for improving the detail display quality of a VR device by using a binocular difference mechanism according to claim 1, wherein The determining the brightness level adjustment weight according to the disparity saliency, the binocular brightness difference degree, and the user gaze behavior characteristics includes: Determining the initial rendering weight of each frame of binocular images according to the user gaze behavior characteristics corresponding to each frame of binocular images; Calculating the detail retention degree of each frame of binocular images, and determining the detail display frame segment according to the change of the detail retention degree; Determining the intersection binocular image according to the intersection of the two sequences composed of the initial rendering weights of each frame of binocular images in the detail display frame segment; Determining the brightness level adjustment weight according to the disparity saliency, the binocular brightness difference degree, and the difference of the initial rendering weights of the intersection binocular image.

5. The method for improving the detail display quality of a VR device by using a binocular difference mechanism according to claim 4, characterized in that, The determining the initial rendering weight of each frame of binocular images according to the user gaze behavior characteristics corresponding to each frame of binocular images includes: For each eye image in each frame of binocular images respectively, determining the degree of eye movement and the angular difference between the eye deflection and the device deflection according to the corresponding user gaze behavior characteristics; Determining the proportion of color levels in the gaze area according to the ratio between the number of color levels in the gaze area and the total number of color levels in the one eye image; Determining the initial rendering weight of the one eye image according to the degree of eye movement, the angular difference between the eye deflection and the device deflection, and the proportion of color levels in the gaze area.

6. The method for improving the detail display quality of a VR device by using a binocular difference mechanism according to claim 5, wherein, The user's gaze behavior characteristics also include the device deflection angle; Determining the degree of line-of-sight change and the angular difference between line-of-sight deflection and device deflection according to the corresponding user gaze behavior characteristics includes: Determining the deflection vectors corresponding to the current frame and the previous frame respectively; the deflection vector is determined according to the change of the fixation points of two adjacent frames; Determining the line-of-sight change distance according to the deflection vector corresponding to the current frame, and determining the degree of line-of-sight change according to the ratio between the line-of-sight change distance and the diameter of the fixation area; Determining the line-of-sight deflection angle according to the included angle between the deflection vectors corresponding to the current frame and the previous frame respectively; Determining the angular difference between line-of-sight deflection and device deflection according to the difference between the line-of-sight deflection angle and the device deflection angle.

7. The method for improving the detail display quality of a VR device by using a binocular difference mechanism according to claim 6, wherein The VR device includes an eye movement tracking module and an inertial measurement unit; The eye movement tracking module is used to capture the fixation area in real time; The inertial measurement unit is used to determine the device deflection angle.

8. The method for improving the detail display quality of a VR device by using a binocular difference mechanism according to claim 4, characterized in that, Calculating the detail retention degree of each frame of binocular images includes: For each monocular image in each frame of binocular images respectively, determining the color level histogram of each channel of the monocular image in the LAB color space; Determining the single-channel detail retention degree of the monocular image in each channel according to the separation degree of the color levels in the color level histogram; Determining the detail retention degree of the monocular image according to the sum of the single-channel detail retention degrees in each channel.

9. The method for improving the detail display quality of a VR device by using a binocular difference mechanism according to claim 4, characterized in that, Determining the detail display frame segment according to the change of the detail retention degree includes: Calculating the change rate of the detail retention degree according to the detail retention degrees of each frame of binocular images; Determining the target frame position closest to the current frame from the frame positions where the positive and negative signs of the change rate change, and determining the frame segment from the target frame position to the current frame as the detail display frame segment.

10. The method for improving the detail display quality of a VR device by using a binocular difference mechanism according to any one of claims 1 to 9, characterized in that, Adjusting the initial brightness interval according to the brightness level adjustment weight to obtain the adjusted brightness interval includes: Taking the brightness level adjustment weight as the adjustment weight corresponding to the fixation area; Iteratively expanding the area outward from the fixation area to determine the adjustment weights corresponding to the areas expanded each time; the adjustment weights corresponding to the expanded areas decrease with the increase of the iteration times on the basis of the brightness level adjustment weight; Adjusting the initial brightness interval according to the adjustment weights corresponding to each area to obtain the adjusted brightness intervals corresponding to each area.

Images