Color brightness compensation method and system, and MR head-mounted display device

Abstract

The invention provides a color brightness compensation method, a color brightness compensation system and MR head-mounted display equipment. The color brightness compensation method comprises the following steps that gaze point coordinates of a user are obtained so as to determine a corresponding color brightness compensation matrix, the color brightness compensation matrix comprises chromaticity compensation elements and brightness compensation elements of a first color preset in a to-be-compensated image, and the chromaticity compensation elements and the brightness compensation elements of the first color are matched with the chromaticity compensation elements; and chromaticity compensation elements for the other at least one second color, wherein the value of each compensation element is determined according to the brightness value of the target position corresponding to the staring point coordinate relative to the first color; and inputting the color and brightness compensation matrix into a display module so as to perform synchronous pre-compensation of colors and brightness on a plurality of pixel points in the to-be-compensated image through the display module. According to the invention, synchronous pre-compensation of color and brightness non-uniformity can be carried out in real time according to the position of the gaze point of the human eye, so that the optimal effect is presented for the human eye.

Description

Technical Field

[0001] This invention relates to the field of mixed reality technology, and in particular to a color brightness compensation method, a color brightness compensation system, an MR head-mounted display device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] In Mixed Reality (MR) display devices, color and brightness uniformity are critical issues affecting user experience. Silicon-based OLED displays have gradually become the mainstream display technology in the MR field due to their advantages such as low power consumption, small size, fast response time, and high contrast. Simultaneously, the display lenses of MR headsets are gradually shifting from Fresnel lenses to pancake lenses to reduce the size and weight of MR headsets. However, the angular color shift characteristics of silicon-based OLEDs can cause color distortion in MR headsets, and pancake lenses also have a high light loss rate, with more complex optical path designs leading to even higher light loss rates. Therefore, the combination of silicon-based OLED displays and pancake lenses inevitably introduces optical display defects such as color and brightness inhomogeneity into MR headsets.

[0003] Existing color and brightness compensation technologies mostly compensate for the color and brightness of the display separately, which suffers from high data processing load and low real-time performance, making them unsuitable for MR head-mounted displays with complex optical path designs and lens combinations. Furthermore, existing technologies generally fail to consider the strong correlation between color and brightness non-uniformity and the position of the human eye's gaze point, resulting in poor compensation effects.

[0004] In order to overcome the above-mentioned defects in the existing technology, there is an urgent need in the field for an improved color brightness compensation technology, which can synchronously pre-compensate the color and brightness non-uniformity of multiple areas of the whole image in real time according to the user's gaze point position, so as to present the user with the best image display effect in real time and efficiently. Summary of the Invention

[0005] The following provides a brief overview of one or more aspects to offer a basic understanding of them. This overview is not an exhaustive summary of all conceived aspects, nor is it intended to identify key or decisive elements of all aspects, nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed descriptions that follow.

[0006] To overcome the aforementioned deficiencies in existing technologies, this invention provides a color brightness compensation method, a color brightness compensation system, an MR head-mounted display device, a computer-readable storage medium, and a computer program product. This method determines the brightness value of a target location relative to a preset first color based on the user's gaze point coordinates, and accordingly determines a color brightness compensation matrix. This matrix is ​​used to perform chromaticity and brightness compensation on the first color in the image to be compensated, and chromaticity compensation on at least one other second color. Thus, based on this color brightness compensation matrix, this invention can synchronously pre-compensate for color and brightness non-uniformity in multiple regions of the entire image, thereby presenting the user with the optimal image display effect in real time and efficiently.

[0007] Specifically, the color brightness compensation method provided by the first aspect of the present invention includes the following steps: obtaining the coordinates of a user's gaze point to determine a corresponding color brightness compensation matrix. The color brightness compensation matrix includes chromaticity compensation elements and brightness compensation elements for a preset first color in the image to be compensated, and chromaticity compensation elements for at least one other second color. The value of each compensation element is determined based on the brightness value of the target position corresponding to the gaze point coordinates with respect to the first color; and inputting the color brightness compensation matrix into a display module to perform synchronous pre-compensation of color and brightness for multiple pixels in the image to be compensated via the display module.

[0008] Furthermore, in some embodiments of the present invention, the color brightness compensation matrix is ​​pre-calibrated. The step of calibrating the color brightness compensation matrix includes: inputting pure color images of multiple colors into the display module respectively, and parsing its first display image to construct a second color brightness matrix for multiple positions in each of the first display images with respect to the corresponding color in the XYZ color space; selecting a first target position from the multiple positions, and using its Y value with respect to the first color as the target brightness, and performing brightness compensation on the Y values ​​of the remaining positions with respect to the first color to determine a third color brightness matrix; determining a first transformation matrix to convert the third color brightness matrix to the linear RGB color space based on the third color brightness matrix, and combining the average linear RGB values ​​of each of the first display images with respect to the corresponding color to determine a second transformation matrix to convert the third color brightness matrix to the linear RGB color space; and determining the color brightness compensation matrix corresponding to the first target position based on the product of the first transformation matrix and the second transformation matrix.

[0009] Furthermore, in some embodiments of the present invention, before inputting solid color images of multiple colors into the display module respectively, the color brightness compensation method further includes the following steps: inputting a standard stripe image into the display module and parsing its second display image to determine the distortion correction grid of the display module; and pre-correcting each of the solid color images according to the distortion correction grid.

[0010] Furthermore, in some embodiments of the present invention, the step of parsing the first display image to construct a second color luminance matrix of a plurality of positions in each of the first display images with respect to the corresponding color in the XYZ color space includes: parsing the first display image of each color to determine the xyY values ​​of a plurality of positions in each of the first display images with respect to the corresponding color; constructing a first color luminance matrix based on the xyY values ​​of each position with respect to each color; and converting the first color luminance matrix of each position with respect to each color to the XYZ color space to obtain a second color luminance matrix of each position with respect to each color.

[0011] Furthermore, in some embodiments of the present invention, the step of parsing the first display images of each color to determine the xyY values ​​of multiple positions in each first display image with respect to the corresponding color includes: filtering each first display image to remove edge outliers in the xyY values ​​of each color; dividing the xyY values ​​of each color into rows and columns according to the position, and fitting each row and each column of xyY values ​​using a polynomial to determine a first fitting parameter representing the correspondence between the position and the xyY value; and interpolating the blank positions where xyY values ​​were not actually obtained and / or the outlier positions where xyY values ​​are abnormal according to the first fitting parameter to determine the xyY values ​​of multiple positions in each first display image with respect to the corresponding color.

[0012] Further, in some embodiments of the present invention, the step of selecting a first target position from the plurality of positions, using its Y value with respect to the first color as the target brightness, and performing brightness compensation on the Y values ​​of the remaining positions with respect to the first color to determine a third color brightness matrix includes: measuring the gamma value of the display module and the maximum brightness value of the first color at the currently measured grayscale in the second color brightness matrix; determining a standard brightness value for the corresponding position at a preset grayscale based on the gamma value and the maximum brightness value; normalizing the Y values ​​of each position in the second color brightness matrix with respect to each color based on the standard brightness value to determine a fourth color brightness matrix; and using the Y value of the first target position in the fourth color brightness matrix with respect to the first color as the target brightness, performing brightness compensation on the Y values ​​of the remaining positions in the fourth color brightness matrix with respect to the first color to determine the third color brightness matrix.

[0013] Further, in some embodiments of the present invention, the step of selecting a first target position from the plurality of positions, using its Y-value with respect to the first color as the target brightness, and performing brightness compensation on the Y-values ​​of the remaining positions with respect to the first color to determine a third color brightness matrix includes: determining a brightness compensation threshold according to a first preset ratio of the target brightness. The first preset ratio is less than 1; in response to any position having a Y-value with respect to the first color greater than or equal to the brightness compensation threshold, compensating the Y-value of that position with respect to the first color to the target brightness; and in response to any position having a Y-value with respect to the first color less than the brightness compensation threshold, increasing the Y-value of that position with respect to the first color according to a second preset ratio. The product of the first preset ratio and the second preset ratio is less than or equal to 1.

[0014] Furthermore, in some embodiments of the present invention, the step of determining a first transformation matrix for converting the third color luminance matrix to a linear RGB color space based on the third color luminance matrix includes: determining a first transformation matrix for converting the third color luminance matrix to a linear RGB color space based on the XYZ vectors of the first color in the third color luminance matrix and the XYZ matrices of each of the second colors in the third color luminance matrix.

[0015] Further, in some embodiments of the present invention, the step of determining a first transformation matrix for converting the third color luminance matrix to a linear RGB color space based on the XYZ vectors of the third color luminance matrix for the first color and the XYZ matrices of the third color luminance matrix for each of the second colors includes: determining a first equation for the first transformation matrix based on the XYZ matrices of the third color luminance matrix for each of the second colors and the product of the inverse matrix of the XYZ matrices and the diagonal matrix of the product of the first XYZ vectors of the third color luminance matrix for the first colors; determining a second equation for the first transformation matrix based on the first XYZ vectors and the first RGB vectors of the third color luminance matrix for the first colors after conversion to the linear RGB color space; determining at least one third equation for the first transformation matrix based on the second XYZ vectors of the third color luminance matrix for each of the second colors and the second RGB vectors of the third color luminance matrix for each of the second colors after conversion to the linear RGB color space; and concatenating and solving the equations for the first transformation matrix to determine the first transformation matrix.

[0016] Furthermore, in some embodiments of the present invention, the step of determining a second transformation matrix for converting the third color luminance matrix to the linear RGB color space by combining the average linear RGB values ​​of each of the first display images with respect to the corresponding color includes: calculating the average linear RGB values ​​of each of the first display images with respect to the corresponding color, and combining the XYZ vectors of each color in the third color luminance matrix to determine a second transformation matrix for converting the third color luminance matrix to the linear RGB color space.

[0017] Further, in some embodiments of the present invention, the step of calculating the average linear RGB value of each of the first display images with respect to the corresponding color, and combining it with the XYZ vectors of each of the colors in the third color luminance matrix to determine a second transformation matrix for converting the third color luminance matrix to the linear RGB color space includes: determining a first equation for the second transformation matrix based on the third RGB vector of the average linear RGB value of each of the first display images with respect to the first color, and the first XYZ vector of the first color in the third color luminance matrix with respect to the first color; determining at least one second equation for the second transformation matrix based on the fourth RGB vector of the average linear RGB value of each of the first display images with respect to the second color, and the second XYZ vector of the second color in the third color luminance matrix with respect to the second color; determining a third equation for the second transformation matrix based on the third color luminance matrix, a third transformation matrix for converting the third color luminance matrix back to the xyY color luminance space, and the first color luminance matrix; and connecting and solving each of the equations for the second transformation matrix to determine the second transformation matrix.

[0018] Furthermore, in some embodiments of the present invention, each of the first display images is divided into multiple regions according to its position. The step of calculating the average linear RGB value of each of the first display images with respect to the corresponding color, and combining it with the XYZ vectors of the third color luminance matrix with respect to each color, to determine a second transformation matrix for converting the third color luminance matrix to the linear RGB color space includes: calculating the average linear RGB value of each region of each of the first display images with respect to the corresponding color; and determining multiple second transformation matrices for converting the third color luminance matrix of each region to the linear RGB color space based on the average linear RGB value of each region of each of the first display images with respect to the corresponding color, and the XYZ vectors of the third color luminance matrix with respect to each color.

[0019] Furthermore, in some embodiments of the present invention, the step of determining the color brightness compensation matrix corresponding to the first target position based on the product of the first transformation matrix and the second transformation matrix includes: determining the color brightness compensation sub-matrix corresponding to the first target position for each region based on the product of the first transformation matrix and the second transformation matrix of each region.

[0020] Furthermore, in some embodiments of the present invention, after determining the color brightness compensation sub-matrix corresponding to the first target position for each of the regions, the color brightness compensation method further includes the following steps: filtering the color brightness compensation matrix composed of each of the color brightness compensation sub-matrices to remove abnormal values ​​of the compensation elements therein; dividing each of the color brightness compensation sub-matrices into rows and columns according to the position of each of the regions, and using a polynomial to fit the values ​​of the corresponding compensation elements in each row and each column of the color brightness compensation sub-matrix to determine a second fitting parameter representing the correspondence between the position and the value of the compensation element; interpolating the blank compensation elements and / or abnormal elements with abnormal values ​​of the compensation elements in each of the color brightness compensation sub-matrices according to the second fitting parameter to determine the value of each compensation element in each of the color brightness compensation sub-matrices; and compressing and storing the values ​​of each compensation element in each of the color brightness compensation sub-matrices corresponding to the first target position.

[0021] Furthermore, in some embodiments of the present invention, after determining the color brightness compensation matrix corresponding to the first target position, the color brightness compensation method further includes the following steps: reselecting at least one second target position from the plurality of positions, and determining and storing the color brightness compensation matrix corresponding to each second target position respectively.

[0022] Furthermore, in some embodiments of the present invention, the step of reselecting at least one second target position from the plurality of positions and determining and storing the color brightness compensation matrix corresponding to each second target position includes: determining at least one second target position from within the user's eye box space; inputting each second target position into a pre-trained statistical model to predict its corresponding color brightness compensation matrix via the statistical model; and storing the color brightness compensation matrix corresponding to each second target position in the form of a LUT table into the corresponding memory.

[0023] Furthermore, in some embodiments of the present invention, the step of obtaining the user's gaze point coordinates to determine the corresponding color brightness compensation matrix includes: obtaining the user's gaze point coordinates to determine a target position corresponding to the gaze point coordinates, and determining the distance from the target position to the gaze point coordinates; determining a first color brightness compensation matrix corresponding to the target position, and a second color brightness compensation matrix corresponding to at least one adjacent position of the target position, based on the target position; determining a corresponding weight parameter based on the distance from the target position to the gaze point coordinates, wherein the value of the weight parameter decreases as the distance increases; and performing linear interpolation calculation on the first color brightness compensation matrix and each of the second color brightness compensation matrices based on the weight parameter to determine the color brightness compensation matrix corresponding to the gaze point coordinates.

[0024] Furthermore, in some embodiments of the present invention, the image to be compensated is divided into multiple regions according to its position. The step of obtaining the user's gaze point coordinates to determine the corresponding color brightness compensation matrix includes: obtaining the user's gaze point coordinates to determine the corresponding target position; determining a color brightness compensation sub-matrix for each region corresponding to the target position based on the target position; and combining the color brightness compensation sub-matrixes of each region based on the position of each region to determine the color brightness compensation matrix corresponding to the gaze point coordinates.

[0025] Furthermore, the color brightness compensation system provided according to the second aspect of the present invention includes a first memory and a processor. The first memory stores computer instructions. The processor is connected to the first memory and configured to execute the computer instructions stored in the first memory to implement the color brightness compensation method as provided in the first aspect of the present invention.

[0026] Furthermore, in some embodiments of the present invention, the color brightness compensation system further includes a second memory and a display module. The second memory stores a color brightness compensation matrix corresponding to at least one target location. The display module includes a display driver chip, a pancake lens, and a silicon-based OLED display. The processor is also connected to the second memory and the display driver chip, and is used to perform synchronous pre-compensation of color and brightness on multiple pixels in the image to be compensated according to the color brightness compensation matrix stored in the second memory, and then transmit the synchronously pre-compensated image to the display driver chip for outputting the corresponding display image via the pancake lens and the silicon-based OLED display.

[0027] Furthermore, in some embodiments of the present invention, the color brightness compensation system further includes a measurement camera and a multi-color measurement spectrometer. The processor includes a display distortion correction pipeline and a color brightness compensation pipeline. The display distortion correction pipeline is connected to the display driver chip and the measurement camera, and is configured to: input a standard stripe image to the display driver chip, and acquire a second display image output by the display module via the measurement camera; and determine the distortion correction grid of the display module based on the second display image. The color brightness compensation pipeline is connected to the display driver chip, each of the measurement spectrometers, and the second memory, and is configured to: obtain the distortion correction grid from the display distortion correction pipeline to pre-correct pure color images of multiple colors; input each of the pre-corrected pure color images into the display module, and acquire a first display image of the corresponding color output by the display module via the measurement spectrometer of each color; and determine a color brightness compensation matrix corresponding to the target position based on the xyY values ​​of multiple positions in the first display image of each color with respect to the corresponding color, and store it in the second memory.

[0028] Furthermore, the MR head-mounted display device provided according to the third aspect of the present invention is equipped with the color brightness compensation system provided in the second aspect of the present invention.

[0029] Furthermore, the computer-readable storage medium provided according to the fourth aspect of the present invention stores computer instructions thereon. When the computer instructions are executed by a processor, the color brightness compensation method as provided in the first aspect of the present invention is implemented.

[0030] Furthermore, the computer program product provided according to the fifth aspect of the present invention includes computer instructions. When the computer instructions are executed by a processor, they implement the color brightness compensation method as provided in the first aspect of the present invention. Attached Figure Description

[0031] The above-described features and advantages of the present invention will be better understood after reading the following detailed description of embodiments of the present disclosure in conjunction with the accompanying drawings. In the drawings, components are not necessarily drawn to scale, and components having similar related characteristics or features may have the same or similar reference numerals.

[0032] Figure 1 A schematic diagram of the structure of a color brightness compensation system provided according to some embodiments of the present invention is shown.

[0033] Figure 2 A schematic diagram of the structure of a color brightness compensation system provided according to some embodiments of the present invention is shown.

[0034] Figure 3A schematic flowchart of a color brightness compensation method provided according to some embodiments of the present invention is shown.

[0035] Figure 4 A schematic diagram of pre-corrected solid color images provided according to some embodiments of the present invention is shown.

[0036] Figure 5 A schematic diagram of interpolation completion provided according to some embodiments of the present invention is shown.

[0037] Figure 6 A schematic diagram of edge outlier removal provided according to some embodiments of the present invention is shown.

[0038] Figure 7 A schematic diagram of brightness compensation provided according to some embodiments of the present invention is shown.

[0039] Figure 8 A schematic diagram illustrating the principle of statistical modeling for compensation data provided according to some embodiments of the present invention is shown.

[0040] Figure 9 A schematic diagram illustrating the principle of statistical modeling for compensation data provided according to some embodiments of the present invention is shown.

[0041] Figure 10 A schematic flowchart of a color brightness compensation method provided according to some embodiments of the present invention is shown.

[0042] Figure 11 A schematic diagram of the location of the gaze point provided according to some embodiments of the present invention is shown.

[0043] Figure 12 A schematic diagram of synchronous pre-compensation for color and brightness is shown according to some embodiments of the present invention.

[0044] Figure label:

[0045] 11 processors

[0046] 111 Displays distortion correction pipeline

[0047] 112 Color Brightness Compensation Pipeline

[0048] 12 Second Memory

[0049] 13 Display Modules

[0050] 131 Silicon-based OLED display

[0051] 14 Display driver software or firmware computing platform

[0052] 15. Measuring camera

[0053] 16. Measuring Spectrometer

[0054] 21 physical objects

[0055] 22 virtual image

[0056] 23 Eye Box

[0057] 31 Gaze Points

[0058] 32 Other locations Detailed Implementation

[0059] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Although the description of the present invention is presented in conjunction with preferred embodiments, this does not mean that the features of the invention are limited to these embodiments. On the contrary, the purpose of describing the invention in conjunction with embodiments is to cover other options or modifications that may be derived based on the claims of the present invention. To provide a thorough understanding of the invention, many specific details will be included in the following description. The invention may also be implemented without using these details. Furthermore, to avoid confusion or obscuring the focus of the invention, some specific details will be omitted in the description.

[0060] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0061] Furthermore, the terms "upper," "lower," "left," "right," "top," "bottom," "horizontal," and "vertical" used in the following description should be understood as the orientations shown in the relevant paragraphs and accompanying drawings. These relative terms are for illustrative purposes only and do not imply that the described apparatus must be manufactured or operated in a specific orientation, and therefore should not be construed as limiting the invention.

[0062] It is understood that although terms such as "first," "second," and "third" may be used herein to describe various components, regions, layers, and / or parts, these components, regions, layers, and / or parts should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers, and / or parts. Therefore, the first components, regions, layers, and / or parts discussed below may be referred to as second components, regions, layers, and / or parts without departing from some embodiments of the present invention.

[0063] As mentioned above, most existing color and brightness compensation technologies compensate for the color and brightness of the display separately, which results in high data processing load and low real-time performance. Therefore, they are not suitable for MR head-mounted displays with complex optical path designs and lens combinations. In addition, existing technologies generally do not consider the strong correlation between the non-uniformity of color and brightness and the position of the human eye's gaze point, leading to poor compensation effects.

[0064] To overcome the aforementioned deficiencies in existing technologies, this invention provides a color brightness compensation method, a color brightness compensation system, an MR head-mounted display device, a computer-readable storage medium, and a computer program product. This method determines the brightness value of a target location relative to a preset first color based on the user's gaze point coordinates, and accordingly determines a color brightness compensation matrix. This matrix is ​​used to perform chromaticity and brightness compensation on the preset first color in the image to be compensated, and chromaticity compensation on at least one other second color. Thus, based on this color brightness compensation matrix, this invention can synchronously pre-compensate for color and brightness non-uniformity in multiple regions of the entire image, thereby presenting the user with the optimal image display effect in real time and efficiently.

[0065] In some non-limiting embodiments, the color brightness compensation method provided in the first aspect of the present invention can be implemented via the color brightness compensation system provided in the second aspect of the present invention. Please refer to the references for details. Figure 1 and Figure 2 . Figure 1 A schematic diagram of the structure of a color brightness compensation system provided according to some embodiments of the present invention is shown. Figure 2 A schematic diagram of the structure of a color brightness compensation system provided according to some embodiments of the present invention is shown.

[0066] exist Figure 1 In the illustrated embodiment, the color brightness compensation system provided in the second aspect of the present invention can be configured in the MR head-mounted display device provided in the third aspect of the present invention, and includes a first memory (not shown) and a processor 11. Here, the first memory includes, but is not limited to, the computer-readable storage medium provided in the fourth aspect of the present invention, on which the computer program product provided in the fifth aspect of the present invention is stored. The computer program product includes computer instructions. The processor 11 is connected to the first memory and configured to execute the computer instructions in the computer program product stored in the first memory to implement the color brightness compensation method provided in the first aspect of the present invention.

[0067] Furthermore, in some embodiments, the color brightness compensation system provided in the second aspect of the present invention further includes a second memory 12 and a display module 13. The second memory 12 may specifically be an on-chip flash memory, including but not limited to the computer-readable storage medium provided in the third aspect of the present invention, which stores a color brightness compensation matrix corresponding to at least one target location. The display module 13 may be configured with a display driver chip, a pancake lens, and a silicon-based OLED display. Here, the processor 11 may specifically be a display pipeline, which connects to the second memory 12 via display driver software or firmware computing platform 14 to obtain compensation data such as the color brightness compensation matrix provided by the second memory 12. Then, based on the color brightness compensation matrix, it performs synchronous pre-compensation for the color and brightness of multiple pixels in the original image to be compensated, and transmits the synchronously pre-compensated image to the back-end display driver chip for outputting the corresponding display image via the pancake lens and the silicon-based OLED display.

[0068] In addition, Figure 2 In the illustrated embodiment, the color brightness compensation system provided by the second aspect of the present invention may further include a measuring camera 15 and a multi-color measuring spectrometer 16. Correspondingly, the processor 11 may be configured with a display distortion correction pipeline 111 and a color brightness compensation pipeline 112. Here, the display distortion correction pipeline 111 is connected to the display driver chip of the display module 13 and the measuring camera 15, respectively, and is used to calibrate the distortion correction parameters of the pancake lens and the silicon-based OLED display 131 based on preset gaze point coordinates and the original image acquired by the measuring camera 15 from the silicon-based OLED display 131 of the display module 13. The color brightness compensation pipeline 112 is connected to the display distortion correction pipeline 111, the display driver chip, each measuring spectrometer 16, and the aforementioned second memory 12. It is used to calibrate a color brightness compensation matrix based on the preset gaze point coordinates, the image after distortion removal, and the original colors collected by the measuring spectrometer 16 from the silicon-based OLED display screen 131. The matrix is ​​used to perform chromaticity compensation and brightness compensation on the preset first color in the image to be compensated, and to perform chromaticity compensation on at least one other second color. The color brightness compensation matrix is ​​then stored in the second memory 12 for subsequent use.

[0069] The working principle of the above-mentioned color brightness compensation system will be described below with reference to some embodiments of color brightness compensation methods. Those skilled in the art will understand that these embodiments of color brightness compensation methods are merely non-limiting implementations provided by the present invention, intended to clearly demonstrate the main concept of the invention and provide some specific solutions convenient for public implementation, rather than limiting all functions or all working methods of the color brightness compensation system. Similarly, this color brightness compensation system is also only a non-limiting implementation of the present invention and does not constitute a limitation on the executing entity or execution order of the steps in these color brightness compensation methods.

[0070] In some non-limiting embodiments, the color brightness compensation method provided by the present invention can be implemented independently in two stages: offline calibration and online compensation.

[0071] For detailed offline calibration procedures, please refer to the reference. Figures 3-6 . Figure 3 A schematic flowchart of a color brightness compensation method provided according to some embodiments of the present invention is shown. Figure 4 A schematic diagram of pre-corrected solid color images provided according to some embodiments of the present invention is shown. Figure 5 A schematic diagram of interpolation completion provided according to some embodiments of the present invention is shown. Figure 6 A schematic diagram of edge outlier removal provided according to some embodiments of the present invention is shown.

[0072] like Figures 3 to 6 As shown, during the offline calibration process for color brightness compensation, the color brightness compensation system can first input a standard stripe image (e.g., horizontal and vertical stripe images) into the display module 13 via the display distortion correction pipeline 111, and then acquire a second display image output by the display module 13 via the measurement camera 15. The processor 11 then analyzes this second display image to determine the distortion correction grid for compensating for the optical distortion of the display module 13. Afterwards, the color brightness compensation system can input pure color images of multiple colors (e.g., red, green, blue, and white) into the display module 13 via the color brightness compensation pipeline 112, and then acquire a first display image output by the display module 13 via the measurement spectrometer 16 for each corresponding color. The distortion correction grid is then obtained from the display distortion correction pipeline 111 to correct the distortion in each pure color image. The specific process of determining the distortion correction grid and performing distortion pre-correction based on the distortion correction grid does not involve any technical improvement of this application and will not be elaborated here.

[0073] After that, as Figure 3 As shown, the color brightness compensation system can analyze the first display image with distortion removed for each color to determine the xyY values ​​of multiple locations in each first display image with respect to the corresponding color.r ,y r ,Y r ,x g ,y g ,Y g ,x b ,y b ,Y b ,x w ,y w ,Y w Based on this, a first color brightness matrix is ​​constructed. Here, the xyY values ​​are in the xyY color space, where x is the component describing the chromaticity coordinates, representing the color in the International Commission on Illumination (CIE). The x, y, and Y components represent the position of a color on the CIE chromaticity diagram; y is another component of the chromaticity coordinate system, indicating the position of the color on the CIE chromaticity diagram; Y represents the lightness or brightness component, used to describe the lightness intensity of the color. Thus, through the combination of these three components, the xyY values ​​can accurately describe all visible colors.

[0074] Furthermore, in the process of parsing the first display images of each color and constructing the first color brightness matrix, the processor 11 can preferably first filter each first display image to remove edge outliers in the xyY values ​​of each color, then divide the xyY values ​​of each color into rows and columns according to their spatial coordinates, and use a polynomial to fit the xyY values ​​of each row and column to determine the first fitting parameters representing the correspondence between position and xyY values. Afterwards, the processor 11 can interpolate the blank positions where xyY values ​​were not actually obtained and / or the outlier positions where xyY values ​​are abnormal, based on the first fitting parameters, to determine the xyY values ​​of multiple positions in each first display image with respect to the corresponding color, thereby reducing pixel data errors and / or pixel data loss caused by acquisition errors and improving the color brightness compensation effect.

[0075] Next, the processor 11 can convert the first color luminance matrix of each position with respect to each color to the XYZ color space to obtain the second color luminance matrix of each position with respect to each color. Here, the XYZ color space is a mathematical model developed by the CIE to describe color and to standardize the human visual system's perception of all visible colors. Here, X is the stimulus component of color, corresponding to the human eye's perception of red light and very short wavelengths of blue light; Y is the stimulus component of color, representing brightness or lightness, corresponding to the human eye's perception of all visible wavelengths; and Z is the stimulus component of color, corresponding to the human eye's perception of green light and very long wavelengths of red light.

[0076] Specifically, the processor 11 can use the following formula to convert the xyY values ​​of multiple positions in each of the first display images to the XYZ color space with respect to the corresponding colors:

[0077] X = ((x*Y) / y)

[0078] Z = ((1-xy)*Y / y)

[0079] Y = Y.

[0080] Subsequently, the processor 11 can select a first target position (e.g., the center point) from multiple positions of each first display image, and set its Y value with respect to a first color (e.g., white) as the target brightness Y. w Brightness compensation is performed on the Y values ​​of the remaining positions with respect to the first color to determine the brightness matrix of the third color.

[0081] Please refer to the details. Figure 7 , Figure 7 A schematic diagram of brightness compensation provided according to some embodiments of the present invention is shown.

[0082] like Figure 7 As shown, during the brightness compensation process, the processor 11 can first determine the target brightness Y. w The first preset ratio (generally less than 1) is used to determine the brightness compensation threshold. Then, in response to any position in the first displayed image having a Y value with respect to the first color that is greater than or equal to the brightness compensation threshold, the processor 11 can compensate the Y value at that position with respect to the first color to the target brightness Y. w The processor 11 determines the third color luminance matrix to complete luminance compensation. Conversely, if the Y value of any position in the first displayed image with respect to the first color is less than the luminance compensation threshold, the processor 11 can increase the Y value of that position with respect to the first color by a second preset ratio (generally greater than 1) to determine the third color luminance matrix to complete luminance compensation. Here, the product of the first preset ratio and the second preset ratio is generally less than or equal to 1. Compared to applying the same luminance compensation to all pixels of the entire image, the present invention can perform differentiated quantitative compensation based on the actual luminance of each pixel, thereby avoiding image overexposure and distortion caused by excessive luminance compensation in dark areas of the image.

[0083] Furthermore, in some embodiments, the processor 11 may also preferably measure the gamma value of the display module 13 and the maximum brightness value Y of the first color at the currently measured grayscale in the second color brightness matrix before performing brightness compensation. wmax And based on the gamma value and the maximum brightness Y wmax Determine the standard brightness value Y at the corresponding location under a preset grayscale (e.g., grayscale 255).w255 .

[0084] Then, the processor 11 can determine the standard brightness value Y. w255 The Y-values ​​of each position in the second color brightness matrix with respect to each color are normalized to determine the fourth color brightness matrix. Then, the Y-value of the first target position in the fourth color brightness matrix with respect to the first color is used as the target brightness. Brightness compensation is then performed on the Y-values ​​of the remaining positions in the fourth color brightness matrix with respect to the first color to determine the third color brightness matrix after brightness compensation. By performing this normalization process, the present invention can be compatiblely adapted to images of various grayscale levels for universal color brightness compensation.

[0085] Then, the processor 11 can determine the XYZ vector [X] of the first color from the third color luminance matrix. w ,Y w Z w ], and the XYZ matrices of each second color in the third color luminance matrix. Determine the first transformation matrix to convert the third color luminance matrix to the linear RGB color space.

[0086] Specifically, after determining the aforementioned first transformation matrix During the process, the processor 11 can first determine the XYZ matrix of each second color in the third color brightness matrix mentioned above. and the inverse of the XYZ matrix The first XYZ vector [X] of the first color in the third color brightness matrix. w ,Y w Z w The product of the diagonal matrices of the products of the first transformation matrix determines the first equation with respect to the first transformation matrix:

[0087]

[0088] Here, diag(·) is a function that creates a diagonal matrix.

[0089] Furthermore, the processor 11 can also base its decisions on the first XYZ vector [X... w ,Y w Z w ], and the first RGB vector [W] of the first color after the third color brightness matrix is ​​converted to the linear RGB color space. r W g W b Determine the second equation regarding the first transformation matrix:

[0090]

[0091] Furthermore, the processor 11 can also base its calculations on the second XYZ vector [X] of each second color in the third color luminance matrix. r ,Y r Z r ]、[X g ,Y g Z g ]、[X b ,Y b Z b ], and the second RGB vector [R] of each second color after the third color brightness matrix is ​​converted to the linear RGB color space. r ,R g ,R b ]、[G r G g G b ]、[B r B g B b Determine at least one third process regarding the first transformation matrix:

[0092]

[0093] Then, processor 11 can connect and solve the equations concerning the first transformation matrix to determine the first transformation matrix.

[0094] Furthermore, the processor 11 can preferably divide each first display image into multiple (e.g., m*n) regions according to spatial coordinates, and calculate the average linear RGB value of each region of each first display image with respect to the corresponding color. Then, based on the third RGB vector of the average linear RGB values ​​of each region of the first displayed image with respect to the corresponding color... and the first XYZ vector of the first color in the third color luminance matrix. Determine several first equations with respect to several second transformation matrices:

[0095]

[0096] Furthermore, the processor 11 can also use a fourth RGB vector based on the average linear RGB values ​​of the corresponding colors for each region of each first display image.

[0097] And the second XYZ vector [X] of each second color in the third color brightness matrix. r ,Y r Z r ]、[X g ,Y g Z g]、[X b ,Y b Z b Determine at least one second equation with respect to a plurality of second transformation matrices:

[0098]

[0099] Furthermore, the processor 11 can also determine multiple third-party processes related to the second transformation matrix based on the aforementioned third color luminance matrix, the third transformation matrix that converts the third color luminance matrix back to the xyY color luminance space, and the aforementioned first color luminance matrix.

[0100] Then, the processor 11 can connect and solve the equations about the second transformation matrix to determine multiple second transformation matrices that transform the third color luminance matrix of each region to the linear RGB color space. And according to the first transformation matrix mentioned above and the second transformation matrix of each region The product of these factors determines the color brightness compensation sub-matrix for each region corresponding to the first target position:

[0101]

[0102] Furthermore, in some preferred embodiments, after determining the color brightness compensation sub-matrix corresponding to the first target position for each region, the processor 11 can further filter the color brightness compensation matrix composed of the color brightness compensation sub-matrixes to remove outlier values ​​of the compensation elements. Then, the processor 11 can divide each color brightness compensation sub-matrix into rows and columns according to the position of each region, and use a polynomial fitting to determine the values ​​of the corresponding compensation elements in each row and column of the color brightness compensation sub-matrix to determine a second fitting parameter representing the correspondence between the position and the compensation element value. Subsequently, the processor 11 can interpolate the blank compensation elements and / or outlier elements with abnormal compensation element values ​​in each color brightness compensation sub-matrix according to the second fitting parameter, to determine the value of each compensation element in each color brightness compensation sub-matrix, thereby reducing pixel data errors and / or pixel data loss caused by acquisition errors and improving the color brightness compensation effect.

[0103] Next, the processor 11 can also stitch together each color brightness compensation sub-matrix according to the position of each region in the image to obtain a complete second color brightness compensation matrix, and in the form of a parameter LUT table, set the values ​​m of each compensation element in the color brightness compensation matrix of the first target position corresponding to the preset gaze point to the form of a parameter LUT table. x,y,z They are compressed and stored in the second memory (i.e., on-chip flash memory) 12 respectively to reduce data storage requirements.

[0104] Furthermore, in some embodiments, the present invention can also reselect a second target position from the positions of multiple different preset gaze points in three-dimensional space, and repeatedly determine and store o*p*q color brightness compensation matrices corresponding to each second target position as described above, for use in subsequent online compensation stages.

[0105] In addition, please refer to the reference. Figure 8 and Figure 9 . Figure 8 A schematic diagram illustrating the principle of statistical modeling for compensation data provided according to some embodiments of the present invention is shown. Figure 9 A schematic diagram illustrating the principle of statistical modeling for compensation data provided according to some embodiments of the present invention is shown.

[0106] exist Figure 8 and Figure 9 In the illustrated embodiment, the present invention may also preferably extract point data and compensation data within a portion of the optomechanically measured sample eyebox space (i.e., the space where the user's pupil is located when experiencing the best visual effect while wearing the head-mounted display device) 23 to establish and train a statistical model. Subsequently, in response to completing the calibration of the color brightness compensation matrix (or its corresponding LUT table) of the first target position 31 within the eyebox space 23, the present invention may also determine at least one second target position 32 within the user's eyebox space 23, and input each second target position 32 into the pre-trained statistical model to predict its corresponding color brightness compensation matrix. Furthermore, the present invention may also store the color brightness compensation matrices corresponding to each second target position 32 in the form of LUT tables in the corresponding second memory (i.e., on-chip flash memory) 12 for subsequent online compensation stages to call the color brightness compensation matrices of each second target position 32 to calculate its compensated virtual image 22.

[0107] Therefore, by performing the relevant steps in the aforementioned offline calibration phase, this invention can obtain a color and brightness compensation matrix based on the user's gaze point position. Compared to local compensation schemes based on viewing angle, this color and brightness compensation matrix based on the user's gaze point position does not require consideration of the relative angle between the user's eye and the specific pixel to be compensated, thus simplifying the algorithm. Furthermore, this color and brightness compensation matrix based on the user's gaze point position can simultaneously pre-compensate for color and brightness non-uniformity in multiple regions of the entire image, thereby improving the overall compensation effect and maintaining the image's visual quality even when the user switches viewing angles.

[0108] The relevant steps of the online compensation stage of the present invention will be further described below. In the online compensation stage of color brightness compensation, the processor 11 can obtain the user's gaze point coordinates (x, y, z) to determine the corresponding color brightness compensation matrix:

[0109]

[0110] Here, the aforementioned color brightness compensation matrix includes a chromaticity compensation element Δx for a preset first color (e.g., white) in the image to be compensated. w Δy w and brightness compensation element ΔY w , and the chromaticity compensation element Δx for at least one other second color. r Δy r Δx g Δy g Δx b Δy b The values ​​of each compensation element are determined based on the brightness value of the target position relative to the first color at the corresponding gaze point coordinates.

[0111] Please refer to the reference for details. Figure 10 and Figure 11 , Figure 10 A schematic flowchart of a color brightness compensation method provided according to some embodiments of the present invention is shown. Figure 11 A schematic diagram of the location of the gaze point provided according to some embodiments of the present invention is shown.

[0112] like Figure 10 and Figure 11 As shown, in the process of determining the color brightness compensation matrix, the processor 11 can first obtain the coordinates (x, y, z) of the user's actual gaze point m0, so as to determine at least one target position corresponding to the actual gaze point coordinates, and determine the distance from the target position to the actual gaze point m0.

[0113] For example, the processor 11 can first round the coordinates (x, y, z) of the actual gaze point m0 ​​to obtain its corresponding integer coordinates (x′, y′, z′). If the integer coordinates (x′, y′, z′) match any preset gaze point coordinates stored in the second memory (i.e., on-chip flash memory), the processor 11 can determine the position of the integer coordinates (x′, y′, z′) as the target position, and thereby determine the distance from the target position to the actual gaze point m0.

[0114] For example, the processor 11 can also calculate the spatial distance from each preset gaze point stored in the second memory (i.e., on-chip flash memory) to the coordinates (x, y, z) of the actual gaze point m0, and take the preset gaze point with the closest spatial distance as the target position corresponding to the actual gaze point m0.

[0115] Then, the processor 11 can determine the color brightness compensation parameter sub-matrix corresponding to each region based on the target location, and combine the color brightness compensation parameter sub-matrices according to the location of each region to determine the first color brightness compensation parameter matrix m corresponding to the actual gaze point coordinates. x,y,z Then, the processor 11 can select at least one second color brightness compensation parameter matrix m corresponding to adjacent positions m1 to m8 from the preset gaze points stored in the second memory (i.e., on-chip flash memory). x′,y′,z′ Then, combining the distance from the target location to the coordinates of each preset gaze point, the corresponding weight parameter [θ] is determined. x ,θ y ,θ z ], and based on the weight parameter [θ x ,θ y ,θ z Linear interpolation is performed on the first color brightness compensation parameter matrix and each of the second color brightness compensation parameter matrices to determine the color brightness compensation parameter matrix corresponding to the actual gaze point coordinates.

[0116] Here, the second color brightness compensation parameter matrix m corresponding to at least one adjacent position x′,y′,z′ Optionally, it can be a pre-calibrated color brightness compensation parameter matrix for another candidate target location adjacent to the target location along the x, y, or z direction, which is derived from the corresponding second color brightness compensation matrix. The weight parameter [θ] is obtained by compression in the form of a parameter LUT table. x ,θ y ,θ z The value of ] decreases as the distance between the preset gaze point and the actual gaze point in the corresponding direction increases. Thus, this invention can further reduce pixel data errors caused by acquisition errors, thereby improving the color brightness compensation effect.

[0117] Next, the processor can decompress the color and brightness compensation matrix according to the above parameter LUT table format. To obtain the corresponding color brightness compensation matrix And the color brightness compensation matrix Linear interpolation is performed on the m*n*9 parameters to obtain W*H*9 parameters corresponding to the screen resolution of the display module 13. Here, W represents the screen resolution width of the display module 13, and H represents the screen resolution height of the display module 13.

[0118] Those skilled in the art will understand that the above-described embodiments, which combine the color brightness compensation matrix of at least one adjacent gaze point position to interpolate the first color brightness compensation matrix of the actual gaze point coordinates, are merely some non-limiting implementations provided by the present invention. They are intended to clearly demonstrate the main concept of the present invention and provide some preferred solutions to reduce pixel data errors caused by acquisition errors, rather than to limit the scope of protection of the present invention.

[0119] Alternatively, in other embodiments, the processor 11 may also directly perform synchronous pre-compensation of color and brightness for multiple pixels in the image to be compensated based on the first color and brightness compensation matrix of the actual gaze point coordinates, so as to achieve the same effect of improving the compensation efficiency and real-time performance of color and brightness.

[0120] Those skilled in the art will also understand that the above-described embodiments of dividing each first display image into multiple regions according to its position to determine the color brightness compensation sub-matrix corresponding to the first target position of each region are merely some non-limiting implementations provided by the present invention, intended to clearly demonstrate the main concept of the present invention and provide some preferred solutions to improve the accuracy of color brightness compensation, rather than being used to limit the scope of protection of the present invention.

[0121] Optionally, in other embodiments, those skilled in the art may not partition each first display image, but directly calculate the average linear RGB value of each first display image with respect to the corresponding color, and combine it with the XYZ vectors of each color in the third color luminance matrix to determine the second transformation matrix that converts the third color luminance matrix to the linear RGB color space, and determine the color luminance compensation matrix corresponding to the preset gaze point position according to the product of the first transformation matrix and the second transformation matrix as described above.

[0122] Please refer to the following for further information. Figure 12 . Figure 12 A schematic diagram of synchronous pre-compensation for color and brightness is shown according to some embodiments of the present invention.

[0123] like Figure 10 As shown, the processor 11 can convert the aforementioned color brightness compensation matrix based on the gaze point coordinates. The input display module 13 is used to perform synchronous pre-compensation of color and brightness for multiple pixels in the image to be compensated, thereby presenting the best image display effect to the user in real time and efficiently.

[0124] Those skilled in the art will understand that although the relevant steps of the offline calibration stage and the online compensation stage of the present invention are continuously and completely described in the same or related embodiments of this specification, this does not mean that they must be continuously and completely implemented by the same subject or the same device as described in this specification.

[0125] Optionally, in other embodiments, the relevant steps of the offline calibration stage of the present invention can be performed by the first subject or the first device, and the color brightness compensation matrix obtained from the calibration can be stored in the second memory (i.e., on-chip flash memory) 12. Afterwards, the second subject that needs to perform color brightness compensation on the image can purchase or lease the relevant compensation data of the color brightness compensation matrix from the first subject, or store the color brightness compensation matrix in the second memory (i.e., on-chip flash memory) 12. In this way, the second subject can obtain the color brightness compensation matrix via a second device containing only the first memory and the processor 11, and execute the relevant steps of the online compensation stage of the present invention, so as to synchronously pre-compensate for color and brightness non-uniformity in multiple areas of the entire image in real time according to the user's gaze point position, thereby presenting the user with the optimal image display effect in real time and efficiently.

[0126] In summary, the color brightness compensation method, color brightness compensation system, MR head-mounted display device, computer-readable storage medium, and computer program product provided by the present invention can all determine the brightness value of the corresponding target position with respect to a preset first color based on the user's gaze point coordinates, and accordingly determine a color brightness compensation matrix for chromaticity compensation and brightness compensation of the first color in the image to be compensated, and for chromaticity compensation of at least one other second color. Then, based on the color brightness compensation matrix, synchronous pre-compensation for color and brightness non-uniformity is performed, thereby presenting the user with the optimal image display effect in real time and efficiently.

[0127] Although the methods described above are illustrated and depicted as a series of actions for the sake of simplicity, it should be understood and appreciated that these methods are not limited by the order of the actions, as some actions may occur in a different order and / or concurrently with other actions from the illustrations and descriptions herein or not illustrated and described herein but which may be understood by those skilled in the art, according to one or more embodiments.

[0128] Those skilled in the art will understand that information, signals, and data can be represented using any of a variety of different techniques and arts. For example, the data, instructions, commands, information, signals, bits, symbols, and chips described throughout the above description can be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof.

[0129] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps are described above in a generalized manner in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the invention.

[0130] The prior description of this disclosure is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not intended to be limited to the examples and designs described herein, but should be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A color brightness compensation method, characterized in that, Includes the following steps: The coordinates of the user's gaze point are obtained to determine the corresponding color brightness compensation matrix. The color brightness compensation matrix includes chromaticity compensation elements and brightness compensation elements for a preset first color in the image to be compensated, as well as chromaticity compensation elements for at least one other second color. The value of each compensation element is determined based on the brightness value of the target position corresponding to the gaze point coordinates with respect to the first color. as well as The color and brightness compensation matrix is ​​input into the display module so that the display module can perform synchronous pre-compensation of color and brightness for multiple pixels in the image to be compensated.

2. The color brightness compensation method as described in claim 1, characterized in that, The color brightness compensation matrix is ​​pre-calibrated, wherein the steps for calibrating the color brightness compensation matrix include: Multiple pure color images of various colors are input into the display module, and their first display images are parsed to construct second color brightness matrices for multiple positions in each of the first display images in the XYZ color space with respect to the corresponding colors. A first target position is selected from the plurality of positions, and its Y value with respect to the first color is used as the target brightness. The brightness of the remaining positions with respect to the first color is compensated to determine the third color brightness matrix. Based on the third color luminance matrix, a first transformation matrix is ​​determined to convert the third color luminance matrix to the linear RGB color space, and a second transformation matrix is ​​determined by combining the average linear RGB values ​​of each of the first display images with respect to the corresponding color; and The color brightness compensation matrix corresponding to the first target position is determined based on the product of the first transformation matrix and the second transformation matrix.

3. The color brightness compensation method as described in claim 2, characterized in that, Before inputting solid color images of multiple colors into the display module, the color brightness compensation method further includes the following steps: A standard stripe image is input into the display module, and its second display image is parsed to determine the distortion correction grid of the display module; and Each of the solid color images is corrected according to the distortion correction grid.

4. The color brightness compensation method as described in claim 2, characterized in that, The step of parsing the first display image to construct a second color luminance matrix of the corresponding colors at multiple locations in the XYZ color space for each of the first display images includes: The first display image of each color is analyzed to determine the xyY values ​​of multiple positions in each of the first display images with respect to the corresponding color. Construct a first color brightness matrix based on the xyY values ​​of each position with respect to each color; and The first color luminance matrix of each position with respect to each color is converted to the XYZ color space to obtain the second color luminance matrix of each position with respect to each color.

5. The color brightness compensation method as described in claim 4, characterized in that, The step of parsing the first display image of each color to determine the xyY values ​​of multiple positions in each of the first display images with respect to the corresponding color includes: Each of the first display images is filtered to remove edge outliers in the xyY values ​​of each color. The xyY values ​​of each color are divided into rows and columns based on their position, and a polynomial is used to fit the xyY values ​​of each row and column to determine a first fitting parameter representing the correspondence between position and xyY values; and Based on the first fitting parameters, interpolation is performed on blank positions where xyY values ​​were not actually obtained and / or abnormal positions where xyY values ​​are abnormal, so as to determine the xyY values ​​of multiple positions in each of the first display images with respect to the corresponding colors.

6. The color brightness compensation method as described in claim 2, characterized in that, The step of selecting a first target position from the plurality of positions, using its Y-value with respect to the first color as the target brightness, and performing brightness compensation on the Y-values ​​of the remaining positions with respect to the first color to determine the third color brightness matrix includes: Measure the gamma value of the display module, and the maximum brightness value of the first color at the current measured gray level in the second color brightness matrix; Based on the gamma value and the maximum brightness value, determine the standard brightness value of the corresponding position under the preset grayscale. Based on the standard brightness value, the Y value of each position in the second color brightness matrix is ​​normalized with respect to each color to determine the fourth color brightness matrix; and The target brightness is determined by using the Y value of the first target position in the fourth color brightness matrix with respect to the first color, and then performing brightness compensation on the Y values ​​of the remaining positions in the fourth color brightness matrix with respect to the first color to determine the third color brightness matrix.

7. The color brightness compensation method as described in claim 2, characterized in that, The step of selecting a first target position from the plurality of positions, using its Y-value with respect to the first color as the target brightness, and performing brightness compensation on the Y-values ​​of the remaining positions with respect to the first color to determine the third color brightness matrix includes: A brightness compensation threshold is determined based on a first preset ratio of the target brightness, wherein the first preset ratio is less than 1; In response to any position having a Y value with respect to the first color that is greater than or equal to the brightness compensation threshold, the Y value of that position with respect to the first color is compensated to the target brightness; and In response to any position having a Y value of less than the brightness compensation threshold with respect to the first color, the Y value of that position with respect to the first color is increased by a second preset ratio, wherein the product of the first preset ratio and the second preset ratio is less than or equal to 1.

8. The color brightness compensation method as described in claim 2, characterized in that, The step of determining the first transformation matrix for converting the third color luminance matrix to the linear RGB color space based on the third color luminance matrix includes: Based on the XYZ vectors of the first color in the third color luminance matrix and the XYZ matrices of each of the second colors in the third color luminance matrix, a first transformation matrix is ​​determined to convert the third color luminance matrix to the linear RGB color space.

9. The color brightness compensation method as described in claim 8, characterized in that, The step of determining a first transformation matrix to convert the third color luminance matrix to the linear RGB color space based on the XYZ vectors of the first color in the third color luminance matrix and the XYZ matrices of each of the second colors in the third color luminance matrix includes: The first equation regarding the first transformation matrix is ​​determined by multiplying the XYZ matrices of each of the second colors in the third color brightness matrix, and the product of the inverse of the XYZ matrix and the diagonal matrix of the product of the first XYZ vector of the first color in the third color brightness matrix. Based on the first XYZ vector and the first RGB vector of the first color after the third color luminance matrix is ​​transformed into the linear RGB color space, a second equation is determined regarding the first transformation matrix; Based on the second XYZ vectors of each of the second colors in the third color luminance matrix, and the second RGB vectors of each of the second colors after the third color luminance matrix has been transformed to the linear RGB color space, at least one third process is determined regarding the first transformation matrix; and The equations are combined and solved with respect to the first transformation matrix to determine the first transformation matrix.

10. The color brightness compensation method as described in claim 2, characterized in that, The step of determining a second transformation matrix to convert the third color luminance matrix to the linear RGB color space by combining the average linear RGB values ​​of each of the first display images with respect to the corresponding color includes: Calculate the average linear RGB value of each of the first display images with respect to the corresponding color, and combine it with the XYZ vector of each color in the third color luminance matrix to determine the second transformation matrix that transforms the third color luminance matrix to the linear RGB color space.

11. The color brightness compensation method as described in claim 10, characterized in that, The step of calculating the average linear RGB value of each of the first display images with respect to the corresponding color, and combining it with the XYZ vectors of each color in the third color luminance matrix to determine the second transformation matrix for converting the third color luminance matrix to the linear RGB color space includes: Based on the third RGB vector of the average linear RGB value of each of the first display images with respect to the first color, and the first XYZ vector of the third color luminance matrix with respect to the first color, a first equation is determined with respect to the second transformation matrix; Based on the fourth RGB vector of the average linear RGB value of each of the first display images with respect to each of the second colors, and the second XYZ vector of the third color luminance matrix with respect to each of the second colors, at least one second equation is determined with respect to the second transformation matrix; Based on the third color luminance matrix, the third transformation matrix that converts the third color luminance matrix back to the xyY color luminance space, and the first color luminance matrix, determine a third process with respect to the second transformation matrix; and The equations are combined and solved for each of the given equations to determine the second transformation matrix.

12. The color brightness compensation method as described in claim 10, characterized in that, Each of the first display images is divided into multiple regions according to its position. The step of calculating the average linear RGB value of each of the first display images with respect to the corresponding color, and combining it with the XYZ vector of each color in the third color luminance matrix to determine the second transformation matrix for converting the third color luminance matrix to the linear RGB color space includes: Calculate the average linear RGB value of each region of each of the first display images with respect to the corresponding color; and Based on the average linear RGB value of each region of each first display image with respect to the corresponding color, and the XYZ vector of each color in the third color luminance matrix, a plurality of second transformation matrices are determined to transform the third color luminance matrix of each region to the linear RGB color space.

13. The color brightness compensation method as described in claim 12, characterized in that, The step of determining the color brightness compensation matrix corresponding to the first target position based on the product of the first transformation matrix and the second transformation matrix includes: Based on the product of the first transformation matrix and the second transformation matrix of each region, the color brightness compensation sub-matrix corresponding to the first target position of each region is determined.

14. The color brightness compensation method as described in claim 13, characterized in that, After determining the color brightness compensation sub-matrix corresponding to the first target position in each of the aforementioned regions, the color brightness compensation method further includes the following steps: The color brightness compensation matrix composed of the color brightness compensation sub-matrices is filtered to remove outlier values ​​of the compensation elements. The color brightness compensation submatrix is ​​divided into rows and columns according to the position of each region, and the values ​​of the corresponding compensation elements in each row and column of the color brightness compensation submatrix are fitted using a polynomial to determine the second fitting parameter that represents the correspondence between the position and the value of the compensation element. Based on the second fitting parameters, interpolation is performed on blank compensation elements and / or abnormal elements with abnormal compensation element values ​​in each of the color brightness compensation sub-matrices to determine the value of each compensation element in each of the color brightness compensation sub-matrices; and The values ​​of each compensation element in each color brightness compensation submatrix corresponding to the first target position are compressed and stored.

15. The color brightness compensation method as described in claim 2, characterized in that, After determining the color brightness compensation matrix corresponding to the first target position, the color brightness compensation method further includes the following steps: At least one second target position is reselected from the plurality of positions, and the color brightness compensation matrix corresponding to each second target position is determined and stored respectively.

16. The color brightness compensation method as described in claim 15, characterized in that, The step of reselecting at least one second target location from the plurality of locations, and determining and storing the color brightness compensation matrix corresponding to each second target location, includes: Determine at least one second target location from within the user's eye box space; Each of the second target positions is input into a pre-trained statistical model to predict its corresponding color brightness compensation matrix; and The color brightness compensation matrix corresponding to each of the second target positions is stored in the corresponding memory in the form of a LUT table.

17. The color brightness compensation method as described in claim 1, characterized in that, The step of obtaining the user's gaze point coordinates to determine the corresponding color brightness compensation matrix includes: Obtain the coordinates of the user's gaze point to determine the target location corresponding to the gaze point coordinates, and determine the distance from the target location to the gaze point coordinates; Based on the target location, determine the corresponding first color brightness compensation matrix and the second color brightness compensation matrix corresponding to at least one adjacent location of the target location; Based on the distance from the target location to the gaze point coordinates, a corresponding weight parameter is determined, wherein the value of the weight parameter decreases as the distance increases; and Based on the weight parameters, linear interpolation is performed on the first color brightness compensation matrix and each of the second color brightness compensation matrices to determine the color brightness compensation matrix corresponding to the gaze point coordinates.

18. The color brightness compensation method as described in claim 1, characterized in that, The image to be compensated is divided into multiple regions according to its location. The step of obtaining the user's gaze point coordinates to determine the corresponding color and brightness compensation matrix includes: Obtain the coordinates of the user's gaze point to determine the corresponding target location; Based on the target location, determine the color brightness compensation sub-matrix corresponding to the target location for each of the regions; and Based on the location of each region, the color brightness compensation sub-matrices of each region are combined to determine the color brightness compensation matrix corresponding to the gaze point coordinates.

19. A color brightness compensation system, characterized in that, include: The first memory, on which computer instructions are stored; as well as A processor, connected to the first memory, and configured to execute computer instructions stored in the first memory to implement the color brightness compensation method as described in any one of claims 1 to 18.

20. The color brightness compensation system as described in claim 19, characterized in that, Also includes: The second memory stores a color brightness compensation matrix corresponding to at least one target position; as well as The display module includes a display driver chip, a pancake lens, and a silicon-based OLED display. The processor is also connected to the second memory and the display driver chip. It is used to perform synchronous pre-compensation on the color and brightness of multiple pixels in the image to be compensated according to the color and brightness compensation matrix stored in the second memory, and then transmit the image after synchronous pre-compensation to the display driver chip so that it can output the corresponding display image through the pancake lens and the silicon-based OLED display.

21. The color brightness compensation system as described in claim 20, characterized in that, The color brightness compensation system also includes a measuring camera and a multi-color spectrometer. The processor includes a display distortion correction pipeline and a color brightness compensation pipeline. The display distortion correction pipeline connects the display driver chip and the measurement camera, and is configured to: input a standard stripe image to the display driver chip, and acquire a second display image output by the display module via the measurement camera; and determine the distortion correction grid of the display module based on the second display image. The color brightness compensation pipeline connects the display driver chip, each of the measurement spectrometers, and the second memory, and is configured to: obtain the distortion correction grid from the display distortion correction pipeline to pre-correct pure color images of multiple colors; input each of the pre-corrected pure color images into the display module, and acquire the first display image of the corresponding color output by the display module via the measurement spectrometers of each color; and determine the color brightness compensation matrix corresponding to the target position based on the xyY values ​​of multiple positions in the first display image of each color with respect to the corresponding color, and store it in the second memory.

22. An MR head-mounted display device, characterized in that, The MR head-mounted display device is equipped with a color brightness compensation system as described in any one of claims 19 to 21.

23. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the computer instructions are executed by the processor, the color brightness compensation method as described in any one of claims 1 to 18 is implemented.

24. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, the color brightness compensation method as described in any one of claims 1 to 18 is implemented.

Images