Method for improving detail display quality of VR equipment by using binocular difference mechanism

Abstract

The invention relates to the technical field of image data processing, in particular to a method for improving the detail display quality of VR equipment by using a binocular difference mechanism. The method comprises the following steps: acquiring a binocular image of a current frame of VR equipment and a user watching behavior feature; the user gazing behavior characteristics comprise a gazing area and a gazing point; according to the disparity map of the binocular image, determining the disparity significance of the fixation area; determining an initial brightness interval and a binocular brightness difference degree according to a brightness order of a corresponding point of the fixation point in the binocular image; determining a brightness order adjustment weight according to the parallax saliency, the binocular brightness difference degree and the user watching behavior characteristics; adjusting the initial brightness interval according to the brightness order adjusting weight to obtain an adjusted brightness interval; and adjusting the brightness of the binocular image according to the adjusted brightness interval, and obtaining a binocular image currently output by the VR equipment based on the adjusted binocular image. The problem of display delay is avoided, and the quality of display details is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to a method for improving detail display quality of a VR device by utilizing a binocular difference mechanism. Background Art

[0002] The display technology currently used in traditional displays usually improves display details by increasing the bit depth. For example, the 10-bit display technology that has emerged in recent years increases display details, including brightness and chromaticity details, compared to the traditional 8-bit display technology. However, this improvement method has three major defects: 1. The color management of the device and the processing of the image must be performed in 10 bits, which increases the processing time and resource requirements; 2. All images and videos need to be encoded using 10 bits, which increases the storage capacity of the file; 3. The bit depth effect is limited by the mechanism of the human eye (Weber's law), and the bit depth gain is small at low brightness.

[0003] To address the above issues, the color or brightness values ​​of the pixels around the gaze point are generally adjusted frame by frame at a fixed amplitude to increase display details. However, when VR (Virtual Reality) devices are displaying, due to the tunneling effect, the attention to the positions near the gaze point is significantly higher than that of other positions, so the adjustment investment in the positions near the gaze point is relatively high. Therefore, after the user shifts the gaze point, the details of the displayed content need to be adjusted, which can easily lead to display delay problems. Summary of the invention

[0004] In order to solve the technical problem of avoiding display delay while ensuring the quality of display details, the purpose of the present invention is to provide a method for improving the detail display quality of VR devices by using a binocular difference mechanism. The technical solution adopted is as follows: The present invention provides a method for improving the detail display quality of a VR device by using a binocular difference mechanism, the method comprising: Obtaining a binocular image of a current frame of a VR device and user gaze behavior characteristics; the user gaze behavior characteristics include a gaze area and a gaze point; Determining the disparity significance of the gaze area according to the disparity map of the binocular image; Determine an initial brightness interval and a binocular brightness difference degree according to the brightness level of the corresponding point of the gaze point in the binocular image; Determining a brightness level adjustment weight according to the parallax significance, the binocular brightness difference degree and the user gaze behavior characteristics; Adjusting the initial brightness range according to the brightness level adjustment weight to obtain an adjusted brightness range; According to the adjusted brightness range, the brightness of the binocular image is adjusted, and the binocular image currently output by the VR device is obtained based on the adjusted binocular image.

[0005] According to the method for improving the detail display quality of VR equipment by using the binocular disparity mechanism provided by the present invention, determining the disparity significance of the gaze area according to the disparity map of the binocular image includes: Calculating a disparity map of the binocular image; Calculating the total disparity value corresponding to the gaze area in the disparity map, and calculating the total disparity value of the entire disparity map; The disparity significance of the gaze area is determined according to a ratio of the total disparity value corresponding to the gaze area to the total disparity value of the entire image.

[0006] According to the method for improving the detail display quality of VR equipment by using the binocular difference mechanism provided by the present invention, the initial brightness range and the binocular brightness difference degree are determined according to the brightness level of the corresponding point of the gaze point in the binocular image, including: The gaze point in one of the binocular images corresponds to a corresponding point in the other binocular image through the disparity map; Taking the brightness level of the gaze point in the first image and the brightness level of the corresponding point in the second image as interval endpoints, respectively, to determine an initial brightness interval; The binocular brightness difference is determined according to the ratio of the order span of the initial brightness interval to the brightness order of the entire image.

[0007] According to the method for improving the detail display quality of VR devices by using the binocular difference mechanism provided by the present invention, the brightness level adjustment weight is determined according to the parallax significance, the binocular brightness difference degree and the user gaze behavior characteristics, including: Determine the initial rendering weight of each binocular image frame according to the user gaze behavior characteristics corresponding to each binocular image frame; Calculating the detail retention of each frame of the binocular image, and determining the detail display frame segment according to the change of the detail retention; Determine an intersection binocular image according to an intersection of two sequences consisting of the initial rendering weights of each binocular image frame in the detail display frame segment; A brightness level adjustment weight is determined according to the parallax significance, the binocular brightness difference degree, and the difference in the initial rendering weights of the intersection binocular images.

[0008] According to the method for improving the detail display quality of VR devices by using the binocular difference mechanism provided by the present invention, the initial rendering weight of each frame of binocular images is determined according to the user gaze behavior characteristics corresponding to each frame of binocular images, including: For each image in each frame of the binocular image, determine the degree of sight change and the angle difference between the sight deflection and the device deflection according to the corresponding user gaze behavior characteristics; Determining a color level ratio of the gaze area according to a ratio between the number of color levels in the gaze area and the total number of color levels in the one-eye image; An initial rendering weight of the eye image is determined according to the degree of change of the eye line, the angle difference between the eye line deflection and the device deflection, and the color level ratio of the gaze area.

[0009] According to the method for improving the detail display quality of VR devices using binocular difference mechanism provided by the present invention, the user gaze behavior characteristics also include a device deflection angle; The determining, according to the corresponding user gaze behavior characteristics, the degree of sight line change and the angle difference between sight line deflection and device deflection includes: Determine the deflection vectors corresponding to the current frame and the previous frame respectively; the deflection vector is determined according to the change of the gaze point of two adjacent frames; Determining a sight line change distance according to the deflection vector corresponding to the current frame, and determining a sight line change degree according to a ratio between the sight line change distance and a diameter of the gaze area; Determining a sight line deflection angle according to an angle between the deflection vectors corresponding to the current frame and the previous frame respectively; According to the difference between the sight line deflection angle and the device deflection angle, the angular difference between the sight line deflection and the device deflection is determined.

[0010] According to the method for improving the detail display quality of a VR device by using a binocular difference mechanism provided by the present invention, the VR device includes an eye tracking module and an inertial measurement unit; The eye tracking module is used to capture the gaze area in real time; The inertial measurement unit is used to determine the device deflection angle.

[0011] According to the method for improving the detail display quality of VR devices by using the binocular difference mechanism provided by the present invention, the calculation of the detail retention of each frame of binocular image includes: For each binocular image in each frame of the binocular image, determine the color level histogram of the binocular image in each channel in the LAB color space; Determining the single-channel detail retention of the first-order image in each channel according to the degree of separation of the color levels in the color level histogram; The detail retention of the one-way image is determined according to the sum of the single-channel detail retention under each channel.

[0012] According to the method for improving the detail display quality of VR devices by using a binocular difference mechanism provided by the present invention, determining the detail display frame segment according to the change of the detail retention degree includes: Calculating the rate of change of the detail retention according to the detail retention of each frame of the binocular image; A target frame position closest to the current frame is determined from the frame positions where the sign of the change rate changes, and a frame segment from the target frame position to the current frame is determined as a detail display frame segment.

[0013] According to the method for improving the detail display quality of VR devices by using a binocular difference mechanism provided by the present invention, the initial brightness interval is adjusted according to the brightness level adjustment weight to obtain an adjusted brightness interval, including: Using the brightness level adjustment weight as the adjustment weight corresponding to the gaze area; Iteratively expand the area outward from the gaze area, and determine the adjustment weight corresponding to the area expanded each time; the degree to which the adjustment weight corresponding to the expanded area is reduced on the basis of the brightness level adjustment weight is positively correlated with the number of iterations; The initial brightness interval is adjusted according to the adjustment weights corresponding to the respective regions to obtain the adjusted brightness intervals corresponding to the respective regions.

[0014] The present invention has the following beneficial effects: By determining the disparity significance of the gaze area according to the disparity map of the binocular image of the current frame, the display details and the matching distortion in the gaze area can be accurately evaluated. According to the brightness level of the corresponding point of the gaze point in the binocular image, the initial brightness range and the degree of binocular brightness difference can be determined, and the brightness difference of the binocular image can be accurately measured. Then, according to the disparity significance, the degree of binocular brightness difference and the user's gaze behavior characteristics, the brightness level adjustment weight is determined. The brightness level adjustment weight can be dynamically determined in combination with the display details and the matching distortion in the gaze area, the brightness difference of the binocular image and the user's gaze situation, and it is dynamically determined which areas are rendered first. Then, according to the brightness level adjustment weight, the initial brightness range is adjusted to obtain the adjusted brightness range. According to the adjusted brightness range, the brightness of the binocular image is adjusted. Based on the adjusted binocular image, the binocular image currently output by the VR device is obtained, thereby avoiding the problem of display delay caused by adjusting the pixels around the gaze point according to a fixed amplitude after the user shifts the gaze point, and ensuring the quality of display details. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 A flowchart of a method for improving the detail display quality of a VR device by using a binocular disparity mechanism is provided in accordance with an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the method for improving the detail display quality of VR equipment using a binocular difference mechanism proposed by the present invention, its specific implementation method, structure, features and effects, in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0018] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0019] The following is a detailed description of a method for improving the detail display quality of a VR device by using a binocular difference mechanism provided by the present invention in conjunction with the accompanying drawings.

[0020] See also Figure 1 , which shows a flow chart of a method for improving the detail display quality of a VR device by using a binocular difference mechanism provided by an embodiment of the present invention, comprising the following steps: Step 101, obtaining a binocular image of a current frame of a VR device and user gaze behavior characteristics; the user gaze behavior characteristics include a gaze area and a gaze point.

[0021] The VR device refers to a virtual reality device. The binocular image includes a left eye image and a right eye image.

[0022] In one embodiment, the user gaze behavior feature may include a device deflection angle in addition to the gaze area and the gaze point. The gaze point may be the center point of the gaze area. The device deflection angle refers to the deflection angle of the VR device.

[0023] In one embodiment, a VR device may include an eye tracking module, an inertial measurement unit, a GPU (Graphics Processing Unit) and a display screen. Among them, the eye tracking module may be an integrated high-precision eye tracking module, such as a pupil center corneal reflection (PCCR, Press Chinese Character Recognition) sensor, which is used to capture the gaze area in real time. The inertial measurement unit (IMU) is used to measure the real-time posture and movement direction of the VR device, so that it can be used to obtain the deflection angle of the device. The inertial measurement unit can be integrated in the eye tracking module. The GPU can be a high-performance GPU that can support 8-bit color image processing and support VR devices. The GPU obtains the left eye image and the right eye image with aligned frame bit order in a continuous frame manner, and executes the method of using the binocular difference mechanism in each embodiment of the present invention to improve the detail display quality of the VR device. The display screen can be a high-resolution display screen, supporting at least 4K resolution for both eyes, and supporting OLED (Organic Light-Emitting Diode) or Micro-LED (Micro-Light-Emitting Diode) screen with a color depth of 10 bits or above to ensure accurate presentation of high dynamic range (HDR) content.

[0024] Step 102: Determine the disparity saliency of the fixation area according to the disparity map of the binocular image.

[0025] The disparity map is used to reflect the pixel offset between the left eye image and the right eye image. The disparity saliency is used to measure the disparity saliency of the fixation area.

[0026] In one embodiment, the binocular images may be matched in a Normalized Cross-Correlation (NCC) manner to obtain a disparity map.

[0027] In one embodiment, the total disparity value corresponding to the gaze area in the disparity map may be calculated, and the total disparity value of the entire disparity map may be calculated. The disparity significance of the gaze area may be determined based on the total disparity value corresponding to the gaze area and the total disparity value of the entire map.

[0028] Step 103, determining the initial brightness range and the degree of binocular brightness difference according to the brightness level of the corresponding point of the gaze point in the binocular image.

[0029] Among them, the binocular brightness difference degree is used to measure the difference between the brightness levels of the corresponding points of the gaze point in the binocular image.

[0030] In one embodiment, the brightness levels of the points corresponding to the gaze point in the binocular image may be used as interval endpoints to determine the initial brightness interval.

[0031] In one embodiment, the binocular brightness difference degree may be determined according to the order span of the initial brightness interval and the brightness order of the entire image.

[0032] Step 104: Determine the brightness level adjustment weight according to the parallax significance, the binocular brightness difference and the user's gaze behavior characteristics.

[0033] In one embodiment, the initial rendering weight of each frame of binocular images can be determined according to the user's gaze behavior characteristics, the intersection binocular images can be determined according to the initial rendering weight of each frame of images, and the difference in the initial rendering weights of the intersection binocular images can be determined.

[0034] In one embodiment, the brightness level adjustment weight is positively correlated with the parallax significance. The brightness level adjustment weight is positively correlated with the degree of binocular brightness difference. The brightness level adjustment weight is positively correlated with the difference in initial rendering weights of the intersection binocular image.

[0035] Step 105 , adjusting the initial brightness range according to the brightness level adjustment weight to obtain an adjusted brightness range.

[0036] In one embodiment, to calculate 1 minus the value of the brightness level adjustment weight, the right endpoint of the initial brightness interval can be multiplied by the brightness level adjustment weight, and the left endpoint of the initial brightness interval can be multiplied by 1 minus the value of the brightness level adjustment weight to obtain the adjusted brightness interval.

[0037] In one embodiment, the brightness level adjustment weight can be used as the adjustment weight corresponding to the gaze area, and the brightness level adjustment weight can be reduced to obtain the adjustment weight corresponding to the area outside the gaze area. According to the adjustment weights corresponding to each area, the initial brightness range is adjusted to obtain the adjusted brightness range corresponding to each area.

[0038] Step 106, adjusting the brightness of the binocular image according to the adjusted brightness range, and obtaining the binocular image currently output by the VR device based on the adjusted binocular image.

[0039] In one embodiment, the brightness of the binocular image is adjusted according to the adjusted brightness range, the brightness of the primary sight image is increased, and the brightness of the secondary sight image is decreased.

[0040] In one embodiment, the detail retention of the binocular image can be calculated, and the detail retention corresponding to the left eye image and the right eye image in the binocular image can be compared. The left eye image / right eye image with a large detail retention degree is used as the main line of sight image, and the right eye image / left eye image with a small detail retention degree is used as the secondary line of sight image.

[0041] In one embodiment, the image coordinates of the binocular images can be matched through the disparity map and then input into a binocular matching unit restricted by the NCC method to obtain the binocular image currently output by the VR device.

[0042] In the above method for improving the detail display quality of VR devices by using the binocular disparity mechanism, by determining the disparity significance of the gaze area according to the disparity map of the binocular image of the current frame, the display details existing in the gaze area and the matching distortion can be accurately evaluated, and the initial brightness range and the degree of binocular brightness difference can be determined according to the brightness level of the corresponding point of the gaze point in the binocular image, so that the brightness difference of the binocular image can be accurately measured, and then the brightness level adjustment weight is determined according to the disparity significance, the degree of binocular brightness difference and the user's gaze behavior characteristics, and the brightness level adjustment weight can be dynamically determined in combination with the display details existing in the gaze area and the matching distortion, the brightness difference of the binocular image and the user's gaze situation, and it is dynamically determined which areas are rendered first, and then the initial brightness range is adjusted according to the brightness level adjustment weight to obtain the adjusted brightness range, and the brightness of the binocular image is adjusted according to the adjusted brightness range, and the binocular image currently output by the VR device is obtained based on the adjusted binocular image, thereby avoiding the problem of display delay caused by adjusting the pixels around the gaze point according to a fixed amplitude after the user shifts the gaze point, thereby ensuring the quality of display details.

[0043] In one embodiment, the disparity significance of the gaze area is determined according to the disparity map of the binocular image, including: calculating the disparity map of the binocular image; calculating the total disparity value corresponding to the gaze area in the disparity map, and calculating the total disparity value of the entire disparity map; determining the disparity significance of the gaze area according to the ratio of the total disparity value corresponding to the gaze area to the total disparity value of the entire image.

[0044] In one embodiment, the disparity saliency of the fixation area may be calculated according to the following formula:

[0045] in, Indicates the gaze area The parallax saliency. Indicates the total disparity value corresponding to the gaze area in the disparity map. Indicates the total disparity value of the entire disparity map.

[0046] In the above embodiment, the disparity significance of the gaze area is determined according to the ratio of the total disparity value corresponding to the gaze area to the total disparity value of the whole image, so that the display details existing in the gaze area and the matching distortion can be accurately evaluated.

[0047] In one embodiment, an initial brightness interval and a degree of binocular brightness difference are determined according to a brightness order of a corresponding point of a gaze point in a binocular image, including: corresponding a gaze point in one binocular image to a corresponding point in another binocular image through a disparity map; determining an initial brightness interval by taking the brightness order of the gaze point in one binocular image and the brightness order of the corresponding point in the other binocular image as interval endpoints; determining the degree of binocular brightness difference according to a ratio of an order span of the initial brightness interval to a brightness order of the entire image.

[0048] For example: fixation point Corresponding to the gaze point in another image through the disparity map , extract the brightness level of these two points , , then the initial brightness range is (Assuming the gaze point Than fixation point The brightness is high).

[0049] In one embodiment, the binocular brightness difference can be calculated according to the following formula:

[0050] in, Indicates the degree of binocular brightness difference. Indicates the order span of the initial brightness interval. Indicates the brightness level of the entire image.

[0051] In the above embodiment, the degree of binocular brightness difference is determined according to the ratio of the order span of the initial brightness interval to the brightness order of the whole image, which can accurately measure the brightness difference of the binocular image and thus be used to adjust the brightness order between corresponding points in the binocular image to avoid distortion while enhancing the details.

[0052] In one embodiment, the brightness level adjustment weight is determined according to the parallax significance, the degree of binocular brightness difference and the user's gaze behavior characteristics, including: determining the initial rendering weight of each binocular image frame according to the user's gaze behavior characteristics corresponding to each frame of the binocular image; calculating the detail retention of each frame of the binocular image, and determining the detail display frame segment according to the change of the detail retention; determining the intersection binocular image according to the intersection of two sequences composed of the initial rendering weights of each frame of the binocular image in the detail display frame segment; determining the brightness level adjustment weight according to the difference in the parallax significance, the degree of binocular brightness difference and the initial rendering weight of the intersection binocular image.

[0053] The detail display frame segment is a frame segment from the target frame corresponding to the sudden change in the direction of the last detail retention to the current frame.

[0054] In one embodiment, the initial rendering weights of the left-eye images of each frame and the right-eye images of each frame in the detail display frame segment form two initial rendering weight sequences, and the target intersection point closest to the current frame is taken from the intersection points of the two initial rendering weight sequences, and the binocular image corresponding to the target intersection point is determined as the intersection binocular image.

[0055] In one embodiment, the brightness level adjustment weight may be calculated according to the following formula:

[0056] in, Indicates the gaze area The brightness level is adjusted by the weight. Indicates the gaze area The parallax saliency. Indicates the degree of binocular brightness difference. represents the initial rendering weight of any image in the intersection binocular image, Represents the initial rendering weight of the other eye image in the intersection binocular image. It represents the difference in the initial rendering weights of the binocular images at the intersection point, reflecting the deviation of the image display details at the actual gaze position.

[0057] In the above embodiment, since the user's line of sight changes dramatically, it is often after the display material attracts the user's line of sight to a large enough extent, and the direction of the user's gaze interacts with the content displayed by the material, the user's perspective movement environment is evaluated through the initial rendering weight. The variation period of the user's initial rendering weight is extracted, and the texture complexity represented by the parallax significance is combined with the degree of binocular brightness difference. The limit range is collaboratively set along the direction of the user's visual line of sight change of the image represented by the initial rendering weight, so as to limit the specific scale of the brightness adjustment, which can ensure the output of high-quality display details without distortion of the image.

[0058] In one embodiment, the initial rendering weight of each binocular image frame is determined according to the user gaze behavior characteristics corresponding to each binocular image frame, including: determining the degree of line of sight change and the angle difference between line of sight deflection and device deflection for each binocular image in each binocular image frame according to the corresponding user gaze behavior characteristics; determining the proportion of color levels in the gaze area according to the ratio between the number of color levels in the gaze area and the total number of color levels in the monocular image; determining the initial rendering weight of the monocular image according to the degree of line of sight change, the angle difference between line of sight deflection and device deflection, and the proportion of color levels in the gaze area.

[0059] In one embodiment, the initial rendering weight is positively correlated with the degree of sight line change. The initial rendering weight is negatively correlated with the angle difference between sight line deflection and device deflection. The initial rendering weight is negatively correlated with the color level ratio of the gaze area.

[0060] In one embodiment, the cosine value of the angle difference between the line of sight deflection and the device deflection can be determined, and the initial rendering weight can be determined according to the product of the line of sight change degree and the cosine value divided by the color level ratio of the gaze area.

[0061] In one embodiment, the initial rendering weight may be calculated according to the following formula:

[0062] in, Represents an image The initial rendering weight of . Indicates the degree of line of sight change. Indicates the sight line deflection angle. Indicates the device deflection angle. Indicates the angular difference between the line of sight deflection and the device deflection. Represents the cosine function. Indicates the color scale ratio of the gaze area.

[0063] In one embodiment, the color scale ratio of the gaze area may be calculated according to the following formula:

[0064] in, Indicates the color scale ratio of the gaze area. Indicates the gaze area The number of color levels within. Represents the total number of color levels in the first-eye image. It can be understood that the number of color levels in the fixation area can describe the richness of the texture in the fixation area. According to the ratio between the number of color levels in the fixation area and the total number of color levels in the first-eye image, the color level ratio of the fixation area is determined, and whether the fixation point falls on the important texture area can be judged.

[0065] In the above embodiment, the greater the degree of change in line of sight, the higher the initial rendering weight needs to be, in order to meet the amplitude adjustment ratio caused by the large change in the user's line of sight. The smaller the angle difference between the line of sight deflection and the device deflection, the more active the user's viewing behavior is, and the higher the initial rendering weight needs to be, in order to meet the purpose of detail display. The greater the proportion of color levels in the gaze area, the more the gaze point falls on the important texture area, and the lower the initial rendering weight needs to be. Therefore, determining the initial rendering weight of an image based on the degree of change in line of sight, the angle difference between the line of sight deflection and the device deflection, and the proportion of color levels in the gaze area can adaptively optimize the rendering accuracy and avoid invalid calculations.

[0066] In one embodiment, the user gaze behavior characteristics also include a device deflection angle; according to the corresponding user gaze behavior characteristics, determining the degree of line of sight change and the angular difference between line of sight deflection and device deflection, including: determining the deflection vector corresponding to the current frame and the previous frame respectively; the deflection vector is determined based on the change of the gaze point between two adjacent frames; determining the line of sight change distance based on the deflection vector corresponding to the current frame, and determining the degree of line of sight change based on the ratio between the line of sight change distance and the diameter of the gaze area; determining the line of sight deflection angle based on the angle between the deflection vectors corresponding to the current frame and the previous frame respectively; determining the angular difference between the line of sight deflection and the device deflection based on the difference between the line of sight deflection angle and the device deflection angle.

[0067] In one embodiment, the current frame corresponds to the deflection vector is the gaze point from the previous frame Point to the gaze point of the current frame The deflection vector corresponding to the previous frame. is the gaze point from the previous frame Point to the gaze point of the previous frame The current frame corresponds to the deflection vector. Deflection vector corresponding to the previous frame The angle between As the sight deflection angle.

[0068] In one embodiment, the model of the deflection vector corresponding to the current frame can be As the sight line change distance. The sight line change degree can be calculated according to the following formula:

[0069] in, Indicates the degree of line of sight change. Indicates the deflection vector corresponding to the current frame. Indicates the distance of line of sight change. Indicates the gaze area diameter.

[0070] In one embodiment, the device deflection angle may be the deflection angle of the VR device relative to the plane XoZ where the display screen is located. The angular difference between the sight line deflection and the device deflection can be expressed as .

[0071] In the above embodiment, the line of sight change distance is determined according to the deflection vector corresponding to the current frame, the line of sight change degree is determined according to the ratio of the line of sight change distance to the diameter of the gaze area, the line of sight deflection angle is determined according to the angle between the deflection vectors corresponding to the current frame and the previous frame respectively, and the angular difference between the line of sight deflection and the device deflection is determined according to the difference between the line of sight deflection angle and the device deflection angle, which can accurately measure the line of sight change degree and the angular difference between the line of sight deflection and the device deflection.

[0072] In one embodiment, the VR device includes an eye tracking module and an inertial measurement unit; the eye tracking module is used to capture the gaze area in real time; and the inertial measurement unit is used to determine the deflection angle of the device.

[0073] In the above embodiment, the gaze area is captured in real time by the eye tracking module, and the device deflection angle is determined by the inertial measurement unit, which can assist the execution of the method provided by the present invention for improving the detail display quality of VR devices by using the binocular difference mechanism.

[0074] In one embodiment, calculating the detail retention of each binocular image frame includes: determining a color level histogram of the monocular image in each channel in the LAB color space for each binocular image in each frame; determining a single-channel detail retention of the monocular image in each channel according to the degree of separation of the color levels in the color level histogram; and determining the detail retention of the monocular image according to the sum of the single-channel detail retention in each channel.

[0075] In one embodiment, the LAB color space includes an L channel (brightness channel), an A channel (a color channel ranging from dark green to gray to bright pink), and a B channel (a color channel ranging from bright blue to gray to yellow).

[0076] In one embodiment, for each channel in each eye image, the number of pixels in each color level is determined according to the color level histogram of the image in the channel, and the number of pixels in the color level with the largest number of pixels is determined as the maximum number of color level pixels, and the standard deviation of the number of pixels in each color level is determined. The single channel detail retention in the channel is determined by dividing the difference between the number of pixels in each color level in the channel and the maximum number of color level pixels by the standard deviation.

[0077] In one embodiment, the single-channel detail retention can be calculated according to the following formula:

[0078] in, Represents an image Single channel detail retention under channel E. Indicates the amount of color levels under channel E. Represents a single color level in channel E The number of pixels below. Indicates the maximum number of color level pixels, that is, the number of pixels in the color level with the largest number of pixels in channel E. Represents the standard deviation of the number of pixels at each color level in channel E. It can be understood that the difference between the number of pixels at each color level in the channel and the maximum number of pixels at that color level divided by the standard deviation can reflect the degree of separation of the color levels in the channel. The greater the degree of separation, the more likely the channel is to reflect the detailed information of the image with more color levels.

[0079] In one embodiment, the detail retention levels of the left eye image and the right eye image in the binocular image can be compared, and the left eye image / right eye image with a large detail retention level is used as the primary sight line image, and the right eye image / left eye image with a small detail retention level is used as the secondary sight line image. This enables the judgment of the actual direction of important information in the binocular image.

[0080] In the above embodiment, for each image in each frame of the binocular image, the color level histogram of the binocular image in each channel in the LAB color space is determined, and the single-channel detail retention of the binocular image in each channel is determined according to the degree of separation of the color levels in the color level histogram, which can accurately measure the richness of the detail information in the image. In addition, since only the image channels and their numbers are extracted for linear operations, the detail retention of the binocular image is obtained without paying additional computing overhead, and the actual direction of important information in the binocular image can be determined.

[0081] In one embodiment, a detail display frame segment is determined according to a change in detail retention, including: calculating a rate of change of detail retention according to the detail retention of each frame of the binocular image; determining a target frame position closest to the current frame from the frame positions where the sign of the rate of change changes, and determining a frame segment from the target frame position to the current frame as a detail display frame segment.

[0082] In one embodiment, the ratio of detail retention between two adjacent frames can be calculated minus 1 to obtain the change rate of detail retention. The change rate of detail retention can be calculated using the following formula:

[0083] in, Indicates the rate of change of detail retention. It represents the detail retention of the first-order image p, that is, the sum of the single-channel detail retention of the first-order image p in each channel. It represents the detail retention of the first-order image p+1 of the next frame, that is, the sum of the single-channel detail retention of the first-order image p+1 in each channel.

[0084] In the above embodiment, the rate of change of the detail retention is calculated based on the detail retention of each frame of the binocular image, and the target frame position closest to the current frame is determined from the frame positions where the sign of the rate of change changes, and the frame segment from the target frame position to the current frame is determined as the detail display frame segment, so that the detail display frame segment can be accurately determined.

[0085] In one embodiment, the initial brightness interval is adjusted according to the brightness level adjustment weight to obtain the adjusted brightness interval, including: taking the brightness level adjustment weight as the adjustment weight corresponding to the gaze area; iteratively expanding the area outward from the gaze area to determine the adjustment weight corresponding to each expanded area; the adjustment weight corresponding to the expanded area, the degree of reduction based on the brightness level adjustment weight is positively correlated with the number of iterations; according to the adjustment weight corresponding to each area, the initial brightness interval is adjusted to obtain the adjusted brightness interval corresponding to each area.

[0086] In one embodiment, in each iteration, the diameter of the expanded area may be doubled with the fixation point as the center to obtain a newly expanded area. Each expanded area is a ring (excluding the fixation area and the previously expanded area).

[0087] In one embodiment, the adjustment weight corresponding to the expanded area may be determined according to the following formula:

[0088] in, Indicates the adjustment weight corresponding to the currently expanded area. Represents the brightness level adjustment weight, that is, the adjustment weight corresponding to the gaze area. Indicates the current iteration number. Indicates the total number of iterations. Indicates the degree to which the adjustment weight corresponding to the expanded area is reduced based on the brightness level adjustment weight.

[0089] In one embodiment, the adjusted brightness interval corresponding to any area can be expressed as:

[0090] The initial brightness range is . Indicates the adjustment weight corresponding to any area.

[0091] In the above embodiment, the degree to which the adjustment weight corresponding to the expanded area is reduced on the basis of the brightness level adjustment weight is positively correlated with the number of iterations. Therefore, the adjustment weight corresponding to the expanded area is negatively correlated with the number of iterations, so that the adjustment weight of the expanded area closer to the gaze area is larger, and the adjustment weight of the expanded area farther away from the gaze area is smaller, thereby accurately adjusting the brightness of the pixels in each area.

[0092] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0093] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application.

[0094] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0095] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. A method for improving the detail display quality of VR equipment using binocular difference mechanism, characterized in that: The method comprises: Obtaining a binocular image of a current frame of a VR device and user gaze behavior characteristics; the user gaze behavior characteristics include a gaze area and a gaze point; Determining the disparity significance of the gaze area according to the disparity map of the binocular image; Determine an initial brightness interval and a binocular brightness difference degree according to the brightness level of the corresponding point of the gaze point in the binocular image; Determining a brightness level adjustment weight according to the parallax significance, the binocular brightness difference degree and the user gaze behavior characteristics; Adjusting the initial brightness range according to the brightness level adjustment weight to obtain an adjusted brightness range; According to the adjusted brightness range, the brightness of the binocular image is adjusted, and the binocular image currently output by the VR device is obtained based on the adjusted binocular image.

2. The method for improving the detail display quality of VR equipment by using binocular difference mechanism according to claim 1, characterized in that: The determining the disparity significance of the gaze area according to the disparity map of the binocular image includes: Calculating a disparity map of the binocular image; Calculating the total disparity value corresponding to the gaze area in the disparity map, and calculating the total disparity value of the entire disparity map; The disparity significance of the gaze area is determined according to a ratio of the total disparity value corresponding to the gaze area to the total disparity value of the entire image.

3. The method for improving the detail display quality of VR equipment by using binocular difference mechanism according to claim 1, characterized in that: The determining of the initial brightness interval and the binocular brightness difference degree according to the brightness level of the corresponding point of the gaze point in the binocular image comprises: The gaze point in one of the binocular images corresponds to a corresponding point in the other binocular image through the disparity map; Taking the brightness level of the gaze point in the first image and the brightness level of the corresponding point in the second image as interval endpoints, respectively, to determine an initial brightness interval; The binocular brightness difference is determined according to the ratio of the order span of the initial brightness interval to the brightness order of the entire image.

4. The method for improving the detail display quality of VR equipment by using binocular difference mechanism according to claim 1, characterized in that: The step of determining the brightness level adjustment weight according to the parallax significance, the binocular brightness difference degree and the user gaze behavior characteristics includes: Determine the initial rendering weight of each binocular image frame according to the user gaze behavior characteristics corresponding to each binocular image frame; Calculating the detail retention of each frame of the binocular image, and determining the detail display frame segment according to the change of the detail retention; Determine an intersection binocular image according to an intersection of two sequences consisting of the initial rendering weights of each binocular image frame in the detail display frame segment; A brightness level adjustment weight is determined according to the parallax significance, the binocular brightness difference degree, and the difference in the initial rendering weights of the intersection binocular images.

5. The method for improving the detail display quality of VR equipment by using binocular difference mechanism according to claim 4, characterized in that: The initial rendering weight of each frame of the binocular image is determined according to the user gaze behavior characteristics corresponding to each frame of the binocular image, including: For each image in each frame of the binocular image, determine the degree of sight change and the angle difference between the sight deflection and the device deflection according to the corresponding user gaze behavior characteristics; Determining a color level ratio of the gaze area according to a ratio between the number of color levels in the gaze area and the total number of color levels in the one-eye image; An initial rendering weight of the eye image is determined according to the degree of change of the eye line, the angle difference between the eye line deflection and the device deflection, and the color level ratio of the gaze area.

6. The method for improving the detail display quality of VR equipment by using binocular difference mechanism according to claim 5, characterized in that: The user gaze behavior characteristics also include a device deflection angle; The determining, according to the corresponding user gaze behavior characteristics, the degree of sight line change and the angle difference between sight line deflection and device deflection includes: Determine the deflection vectors corresponding to the current frame and the previous frame respectively; the deflection vector is determined according to the change of the gaze point of two adjacent frames; Determining a sight line change distance according to the deflection vector corresponding to the current frame, and determining a sight line change degree according to a ratio between the sight line change distance and a diameter of the gaze area; Determining a sight line deflection angle according to an angle between the deflection vectors corresponding to the current frame and the previous frame respectively; According to the difference between the sight line deflection angle and the device deflection angle, the angular difference between the sight line deflection and the device deflection is determined.

7. The method for improving the detail display quality of VR equipment by using binocular difference mechanism according to claim 6, characterized in that: The VR device includes an eye tracking module and an inertial measurement unit; The eye tracking module is used to capture the gaze area in real time; The inertial measurement unit is used to determine the device deflection angle.

8. The method for improving the detail display quality of VR equipment by using binocular difference mechanism according to claim 4, characterized in that: The calculating of the detail retention of each frame of the binocular image includes: For each binocular image in each frame of the binocular image, determine the color level histogram of the binocular image in each channel in the LAB color space; Determining the single-channel detail retention of the first-order image in each channel according to the degree of separation of the color levels in the color level histogram; The detail retention of the one-way image is determined according to the sum of the single-channel detail retention under each channel.

9. The method for improving the detail display quality of VR equipment by using binocular difference mechanism according to claim 4, characterized in that: The determining of the detail display frame segment according to the change of the detail retention degree includes: Calculating the rate of change of the detail retention according to the detail retention of each frame of the binocular image; A target frame position closest to the current frame is determined from the frame positions where the sign of the change rate changes, and a frame segment from the target frame position to the current frame is determined as a detail display frame segment.

10. The method for improving the detail display quality of VR equipment by using binocular difference mechanism according to any one of claims 1 to 9, characterized in that: The adjusting the initial brightness interval according to the brightness level adjustment weight to obtain an adjusted brightness interval includes: Using the brightness level adjustment weight as the adjustment weight corresponding to the gaze area; Iteratively expand the area outward from the gaze area, and determine the adjustment weight corresponding to the area expanded each time; the degree to which the adjustment weight corresponding to the expanded area is reduced on the basis of the brightness level adjustment weight is positively correlated with the number of iterations; The initial brightness interval is adjusted according to the adjustment weights corresponding to the respective regions to obtain the adjusted brightness intervals corresponding to the respective regions.

Images