Color and luminance compensation method and system, and mr head-mounted display device

Abstract

The present invention provides a color and luminance compensation method and system, and an MR head-mounted display device. The color and luminance compensation method comprises the following steps: acquiring gaze point coordinates of a user to determine a corresponding color and luminance compensation matrix, wherein the color and luminance compensation matrix comprises a chromaticity compensation element and a luminance compensation element for a preset first color in an image to be compensated, and a chromaticity compensation element for at least one other second color, and a value of each compensation element is determined on the basis of a luminance value of the first color at a target position corresponding to the gaze point coordinates; and inputting the color and luminance compensation matrix into a display module, so as to perform, by means of the display module, synchronous color and luminance pre-compensation on a plurality of pixel points in said image. In the present invention, synchronous pre-compensation of color and luminance non-uniformity can be performed in real time on the basis of the gaze point position of human eyes, thereby presenting an optimal effect to the human eyes.

Description

Color brightness compensation method, system, and MR head-mounted display device

[0001] This application claims priority to the patent application with the application date of August 16, 2024, the Chinese application number of 202411133325.7, and the title of "Color brightness compensation method, system, and MR head-mounted display device", and incorporates its entire content by reference into the present application. TECHNICAL FIELD

[0002] The present application relates to the technical field of mixed reality, and in particular to a color brightness compensation method, a color brightness compensation system, an MR head-mounted display device, a computer readable storage medium, and a computer program product. BACKGROUND

[0003] In a mixed reality (MR) display device, the uniformity of color and brightness is a key problem affecting user experience. Silicon-based OLED displays gradually become mainstream displays in the field of mixed reality technology due to their low power consumption, small size, fast response speed, and high contrast ratio. At the same time, the display lens of the MR head-mounted device is gradually changed from a Fresnel lens to a Pancake lens to reduce the volume and weight of the MR head-mounted device. However, the angle color shift characteristics of the silicon-based OLED itself will cause color deviation of the MR head-mounted device, and the Pancake lens also has a high light loss rate, and the more complex the optical path design, the higher the light loss rate. Therefore, the combination of silicon-based OLED displays and Pancake lenses will inevitably introduce optical display defects of color and brightness non-uniformity to the MR head-mounted device.

[0004] Existing color and brightness compensation technologies mostly compensate for the color and brightness of the display respectively, which has the problems of large data operation load and low real-time performance, and thus is not suitable for MR head-mounted devices with complex optical path design of display and lens combination. In addition, the existing technologies also generally do not consider the strong correlation between the non-uniformity of color and brightness and the position of the user's eye gaze point, resulting in poor compensation effect.

[0005] In order to overcome the above-mentioned defects of the prior art, there is an urgent need in the field for an improved color brightness compensation technology for real-time synchronous pre-compensation of color and brightness non-uniformity of multiple regions of the entire image according to the position of the user's eye gaze point, so as to real-time and efficiently present the optimal image display effect to the user. SUMMARY

[0006] The following presents a simplified summary of one or more aspects in order to provide a basic understanding of such aspects. This summary is not an extensive overview of all contemplated aspects, and is intended to neither identify key or critical elements of all aspects nor delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that is presented later.

[0007] To overcome the above-mentioned defects existing in the prior art, the present application provides a color brightness compensation method, a color brightness compensation system, an MR head-mounted display device, a computer readable storage medium, and a computer program product, which can determine the brightness value of a target position corresponding to a user's gaze point coordinate with respect to a preset first color, and determine a color brightness compensation matrix accordingly, so as to perform chroma compensation and brightness compensation on the first color in a to-be-compensated image, and perform chroma compensation on at least one second color. In this way, the present application can perform synchronous pre-compensation on the color and brightness non-uniformity of multiple regions of the entire image based on the color brightness compensation matrix, thereby presenting the user with optimal image display effect in real time and efficiently.

[0008] Specifically, the color brightness compensation method according to the first aspect of the present application comprises the following steps: obtaining the gaze point coordinate of a user to determine a corresponding color brightness compensation matrix. The color brightness compensation matrix comprises chroma compensation elements and brightness compensation elements of a preset first color in a to-be-compensated image, and chroma compensation elements of at least one second color. The values of each compensation element are determined according to the brightness value of a target position corresponding to the gaze point coordinate with respect to the first color; and inputting the color brightness compensation matrix into a display module to perform synchronous pre-compensation on the color and brightness of multiple pixel points in the to-be-compensated image via the display module.

[0009] Further, in some embodiments of the present application, the color brightness compensation matrix is pre-calibrated. The step of calibrating the color brightness compensation matrix comprises: inputting pure color images of multiple colors into the display module respectively, and analyzing first display images thereof to construct second color brightness matrices of the multiple positions in each of the first display images with respect to corresponding colors in XYZ color space respectively; selecting a first target position from the multiple positions, and performing brightness compensation on Y values of the remaining positions with respect to the first color to determine a third color brightness matrix, with a Y value of the first target position with respect to the first color as a target brightness; determining a first conversion matrix converting the third color brightness matrix to a linear RGB color space according to the third color brightness matrix, and determining a second conversion matrix converting the third color brightness matrix to the linear RGB color space in combination with average linear RGB values of each of the first display images with respect to corresponding colors; and determining a color brightness compensation matrix corresponding to the first target position according to a product of the first conversion matrix and the second conversion matrix.

[0010] Further, in some embodiments of the present application, before inputting pure color images of multiple colors into the display module respectively, the color brightness compensation method further comprises the following steps: inputting a standard stripe image into the display module, and analyzing a second display image thereof to determine a distortion correction grid of the display module; and pre-correcting each of the pure color images according to the distortion correction grid.

[0011] Further, in some embodiments of the present application, the step of analyzing the first display images to construct second color brightness matrices of the multiple positions in each of the first display images with respect to corresponding colors in XYZ color space respectively comprises: analyzing the first display images of each of the colors to determine xyY values of the multiple positions in each of the first display images with respect to corresponding colors respectively; constructing first color brightness matrices of each of the positions with respect to each of the colors according to the xyY values of each of the positions with respect to each of the colors; and converting the first color brightness matrices of each of the positions with respect to each of the colors to XYZ color space to obtain the second color brightness matrices of each of the positions with respect to each of the colors respectively.

[0012] Further, in some embodiments of the present application, the step of analyzing each of the first display images of the colors to determine the xyY values of the positions in each of the first display images with respect to the corresponding color includes: filtering each of the first display images to remove edge outliers from the xyY values of each of the colors, respectively; dividing the xyY values of each of the colors by rows and columns according to the positions, and fitting each row and each column of the xyY values with a polynomial to determine first fitting parameters representing the correspondence between the positions and the xyY values; and interpolating blank positions and / or outlier positions with abnormal xyY values according to the first fitting parameters to determine the xyY values of the positions in each of the first display images with respect to the corresponding color, respectively.

[0013] Further, in some embodiments of the present application, the step of selecting a first target position from the positions and performing luminance compensation on the Y values of the remaining positions with respect to the first color to determine a third color luminance matrix includes: measuring a gamma value of the display module and a luminance maximum value of the first color at a currently measured gray scale in the second color luminance matrix; determining standard luminance values of the corresponding positions at a preset gray scale according to the gamma value and the luminance maximum value; performing normalization processing on the Y values of the positions in the second color luminance matrix with respect to the colors according to the standard luminance values to determine a fourth color luminance matrix; and performing luminance compensation on the Y values of the remaining positions in the fourth color luminance matrix with respect to the first color to determine the third color luminance matrix, with the Y value of the first target position in the fourth color luminance matrix with respect to the first color as a target luminance.

[0014] Further, in some embodiments of the present application, the step of selecting a first target position from the positions and performing luminance compensation on the Y values of the remaining positions with respect to the first color to determine a third color luminance matrix includes: determining a luminance compensation threshold according to a first preset proportion of the target luminance. The first preset proportion is less than 1; in response to the Y value of any of the positions with respect to the first color being greater than or equal to the luminance compensation threshold, compensating the Y value of the position with respect to the first color to the target luminance; and in response to the Y value of any of the positions with respect to the first color being less than the luminance compensation threshold, increasing the Y value of the position with respect to the first color according to a second preset proportion. The product of the first preset proportion and the second preset proportion is less than or equal to 1.

[0015] Further, in some embodiments of the present application, the step of determining a first conversion matrix converting the third color luminance matrix to a linear RGB color space according to the third color luminance matrix comprises determining a first conversion matrix converting the third color luminance matrix to a linear RGB color space according to an XYZ vector in the third color luminance matrix regarding the first color, and XYZ matrices in the third color luminance matrix regarding each of the second colors.

[0016] Further, in some embodiments of the present application, the step of determining a first conversion matrix converting the third color luminance matrix to a linear RGB color space according to the third color luminance matrix comprises determining a first conversion matrix converting the third color luminance matrix to a linear RGB color space according to an XYZ vector in the third color luminance matrix regarding the first color, and XYZ matrices in the third color luminance matrix regarding each of the second colors.

[0017] Further, in some embodiments of the present application, the step of determining a second conversion matrix converting the third color luminance matrix to the linear RGB color space in combination with average linear RGB values of each of the first display images regarding corresponding colors comprises calculating the average linear RGB values of each of the first display images regarding corresponding colors respectively, and determining a second conversion matrix converting the third color luminance matrix to the linear RGB color space in combination with XYZ vectors in the third color luminance matrix regarding each of the colors.

[0018] Further, in some embodiments of the present application, the step of determining a second conversion matrix for converting the third colorimetric matrix to the linear RGB color space, based on the average linear RGB values of each of the first display images with respect to each of the colors and the XYZ vectors of each of the colors in the third colorimetric matrix, comprises: determining a first equation for the second conversion matrix based on a third RGB vector of the average linear RGB values of each of the first display images with respect to the first color and a first XYZ vector of each of the second colors in the third colorimetric matrix; determining at least one second equation for the second conversion matrix based on fourth RGB vectors of the average linear RGB values of each of the first display images with respect to each of the second colors and second XYZ vectors of each of the second colors in the third colorimetric matrix; determining a third equation for the second conversion matrix based on the third colorimetric matrix, a third conversion matrix for converting the third colorimetric matrix back to the xyY colorimetric space, and the first colorimetric matrix; and concatenating and solving the equations for the second conversion matrix to determine the second conversion matrix.

[0019] Further, in some embodiments of the present application, each of the first display images is divided into a plurality of regions according to position, and the step of determining a second conversion matrix for converting the third colorimetric matrix to the linear RGB color space, based on the average linear RGB values of each of the first display images with respect to each of the colors and the XYZ vectors of each of the colors in the third colorimetric matrix, comprises: determining the average linear RGB values of each of the regions of each of the first display images with respect to each of the colors; and determining a plurality of second conversion matrices for converting the third colorimetric matrix of each of the regions to the linear RGB color space, based on the average linear RGB values of each of the regions of each of the first display images with respect to each of the colors and the XYZ vectors of each of the colors in the third colorimetric matrix.

[0020] Further, in some embodiments of the present application, the step of determining a colorimetric compensation matrix corresponding to the first target position based on the product of the first conversion matrix and the second conversion matrix comprises: determining a colorimetric compensation sub-matrix corresponding to the first target position for each of the regions based on the product of the first conversion matrix and the second conversion matrix of each of the regions.

[0021] Further, in some embodiments of the present application, after determining the color and brightness compensation sub-matrix corresponding to each of the regions respectively, the color and brightness compensation method further comprises the following steps: filtering the color and brightness compensation matrix composed of each of the color and brightness compensation sub-matrix to remove abnormal values of the compensation elements therein; dividing each of the color and brightness compensation sub-matrix according to the positions of each of the regions, and using polynomial to fit the values of the corresponding compensation elements in each row and each column of the color and brightness compensation sub-matrix to determine the second fitting parameter representing the corresponding relationship between the position and the value of the compensation element; according to the second fitting parameter, interpolating the blank compensation elements and / or abnormal elements with abnormal values of the compensation elements in each of the color and brightness compensation sub-matrix to determine the values of each of the compensation elements in each of the color and brightness compensation sub-matrix respectively; and compressively storing the values of each of the compensation elements in each of the color and brightness compensation sub-matrix corresponding to the first target position.

[0022] Further, in some embodiments of the present application, after determining the color and brightness compensation matrix corresponding to the first target position, the color and brightness compensation method further comprises the following steps: reselecting at least one second target position from the plurality of positions, and respectively determining and storing the color and brightness compensation matrix corresponding to each of the second target positions.

[0023] Further, in some embodiments of the present application, the step of reselecting at least one second target position from the plurality of positions, and respectively determining and storing the color and brightness compensation matrix corresponding to each of the second target positions comprises: determining at least one second target position from within the ocular box space of the user; inputting each of the second target positions into a pre-trained statistical model to respectively estimate the color and brightness compensation matrix corresponding thereto via the statistical model; and storing the color and brightness compensation matrix corresponding to each of the second target positions in the form of a LUT table to the corresponding memory.

[0024] Further, in some embodiments of the present application, the step of obtaining the gaze point coordinates of the user to determine the corresponding color and brightness compensation matrix comprises: obtaining the gaze point coordinates of the user to determine a target position corresponding to the gaze point coordinates, and determining the distance from the target position to the gaze point coordinates; determining, according to the target position, a first color and brightness compensation matrix corresponding to the target position and a second color and brightness compensation matrix corresponding to at least one neighboring position of the target position; determining a weight parameter corresponding to the distance from the target position to the gaze point coordinates according to the distance, wherein the value of the weight parameter decreases as the distance increases; and performing linear interpolation calculation on the first color and brightness compensation matrix and each of the second color and brightness compensation matrices according to the weight parameter to determine the color and brightness compensation matrix corresponding to the gaze point coordinates.

[0025] Further, in some embodiments of the present application, the image to be compensated is divided into a plurality of regions according to position. The step of obtaining the gaze point coordinates of the user to determine the corresponding color and brightness compensation matrix comprises: obtaining the gaze point coordinates of the user to determine a target position corresponding to the gaze point coordinates; determining, according to the target position, a color and brightness compensation sub-matrix corresponding to the target position for each of the regions; and combining the color and brightness compensation sub-matrix of each of the regions according to the position of each of the regions to determine the color and brightness compensation matrix corresponding to the gaze point coordinates.

[0026] In addition, the color and brightness compensation system according to the second aspect of the present application comprises a first memory and a processor. The first memory has computer instructions stored thereon. The processor is connected to the first memory and is configured to execute the computer instructions stored on the first memory to implement the color and brightness compensation method according to the first aspect of the present application.

[0027] Further, in some embodiments of the present application, the color and brightness compensation system further comprises a second memory and a display module. The second memory has color and brightness compensation matrices corresponding to at least one target position stored thereon. The display module comprises a display driving chip, a Pancake lens and a silicon-based OLED display screen. The processor is further connected to the second memory and the display driving chip, and is configured to perform synchronous pre-compensation on the color and brightness of a plurality of pixel points in the image to be compensated according to the color and brightness compensation matrices stored on the second memory, and then transmit the image after the synchronous pre-compensation to the display driving chip for outputting a corresponding display image via the Pancake lens and the silicon-based OLED display screen.

[0028] Further, in some embodiments of the present application, the color brightness compensation system further comprises a measurement camera and a plurality of color measurement spectrometers. The processor comprises a display distortion correction pipeline and a color brightness compensation pipeline. The display distortion correction pipeline is connected to the display driver chip and the measurement camera, and is configured to input a standard fringe image to the display driver chip, and acquire a second display image output by the display module via the measurement camera; and determine a distortion correction grid of the display module according to the second display image. The color brightness compensation pipeline is connected to the display driver chip, each of the measurement spectrometers, and the second memory, and is configured to obtain the distortion correction grid from the display distortion correction pipeline to pre-correct a plurality of color pure images; input each of the pre-corrected color pure images to the display module respectively, and acquire a first display image of a corresponding color output by the display module via the measurement spectrometer of the corresponding color respectively; and determine a color brightness compensation matrix corresponding to the target position according to the xyY values of a plurality of positions in the first display image of each color with respect to the corresponding color, and store the color brightness compensation matrix in the second memory.

[0029] Further, the above-mentioned MR head-mounted display device according to the third aspect of the present application is configured with the above-mentioned color brightness compensation system according to the second aspect of the present application.

[0030] Further, the above-mentioned computer readable storage medium according to the fourth aspect of the present application has computer instructions stored thereon. When the computer instructions are executed by a processor, the color brightness compensation method according to the first aspect of the present application is implemented.

[0031] Further, the above-mentioned computer program product according to the fifth aspect of the present application comprises computer instructions. When the computer instructions are executed by a processor, the color brightness compensation method according to the first aspect of the present application is implemented. BRIEF DESCRIPTION OF DRAWINGS

[0032] The above features and advantages of the present application can be better understood by reading the following detailed description of embodiments of the present application in conjunction with the drawings, in which: In the drawings, components are not necessarily drawn to scale, and components of similar or identical function or structure can be designated with the same or similar reference label.

[0033] FIG. 1 shows a structural schematic diagram of a color brightness compensation system according to some embodiments of the present application.

[0034] FIG. 2 shows a structural schematic diagram of a color brightness compensation system according to some embodiments of the present application.

[0035] FIG. 3 shows a flow schematic diagram of a color brightness compensation method according to some embodiments of the present application.

[0036] FIG. 4 shows a schematic diagram of pre-correction of each pure color image according to some embodiments of the present application.

[0037] FIG. 5 shows a schematic diagram of interpolation completion according to some embodiments of the present application.

[0038] FIG. 6 shows a schematic diagram of removing edge outliers according to some embodiments of the present application.

[0039] FIG. 7 shows a schematic diagram of brightness compensation according to some embodiments of the present application.

[0040] FIG. 8 shows a schematic diagram of statistical model estimation of compensation data according to some embodiments of the present application.

[0041] FIG. 9 shows a schematic diagram of statistical model estimation of compensation data according to some embodiments of the present application.

[0042] FIG. 10 shows a flowchart of a color and brightness compensation method according to some embodiments of the present application.

[0043] FIG. 11 shows a schematic diagram of the position of the gaze point according to some embodiments of the present application.

[0044] FIG. 12 shows a schematic diagram of synchronous pre-compensation of color and brightness according to some embodiments of the present application.

[0045] Reference signs: 11 processor 111 display distortion correction pipeline 112 color and brightness compensation pipeline 12 second memory 13 display module 131 silicon-based OLED display screen 14 display driving software or firmware computing platform 15 measurement camera 16 measurement spectrometer 21 physical object 22 virtual image 23 eyebox 31 gaze point 32 remaining point DETAILED DESCRIPTION

[0046] The advantages and features of the present application will become apparent from the following description of the embodiments of the present application, taken in conjunction with the accompanying drawings. Although the description of the present application will be in the context of its preferred embodiments, the present application can be carried out in a variety of ways and should not be limited to those explicitly set forth herein. In fact, many variations are possible that still fall within the scope of the present application as set forth in the claims. For the purpose of exposition, numerous specific details of the preferred embodiments are set forth in the description that follows. The present application can be practiced without these specific details. In other instances, well-known structures and functions have not been described in detail in order to not obscure the understanding of this description. Also, the description is only to be used to read the structure and functions of the present application, and is not used to limit the scope of the present application.

[0047] In the description of the present application, it is to be understood that the terms "mounting", "connected", "connecting" should be construed broadly in a general sense unless otherwise indicated and limited. For example, it can be fixed connection, or detachable connection, or integral connection; it can be mechanical connection, or electrical connection; it can be direct connection, or indirect connection through an intermediate medium, or internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0048] In addition, "up", "down", "left", "right", "top", "bottom", "horizontal", "vertical" used in the following description should be understood as the orientation shown in the section and the related drawings. Such relative terms are only for the convenience of description, and do not mean that the device described thereby should be manufactured or operated in a particular orientation, and therefore should not be understood as a limitation on the present application.

[0049] It is understood that although the terms "first", "second", "third" and the like can be used herein to describe various components, regions, layers and / or sections, these components, regions, layers and / or sections should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers and / or sections. Therefore, the first components, regions, layers and / or sections discussed below can be referred to as the second components, regions, layers and / or sections without departing from some embodiments of the present application.

[0050] As described above, the existing color and brightness compensation techniques mostly compensate the color and brightness of the display respectively, which has the problems of large data operation load and low real-time performance, and thus is not suitable for the MR head-mounted device with display and lens combination having complex optical path design. In addition, the existing technology does not generally consider the strong correlation between the non-uniformity of color and brightness and the position of the eye fixation point, resulting in poor compensation effect.

[0051] In order to overcome the above-mentioned defects in the prior art, the present application provides a color brightness compensation method, a color brightness compensation system, an MR head-mounted display device, a computer readable storage medium, and a computer program product. The color brightness compensation method can determine the brightness value of a target position corresponding to a user's eye gaze point coordinate with respect to a preset first color, and determine a color brightness compensation matrix based on the brightness value, so as to perform chroma compensation and brightness compensation on the preset first color in a to-be-compensated image, and perform chroma compensation on at least one second color. In this way, the present application can perform synchronous pre-compensation on color and brightness non-uniformity of multiple regions of the entire image based on the color brightness compensation matrix, thereby presenting the user with optimal image display effect in real time and efficiently.

[0052] In some non-limiting embodiments, the color brightness compensation method provided by the first aspect of the present application can be implemented by the color brightness compensation system provided by the second aspect of the present application. For details, please refer to FIG. 1 and FIG. 2. FIG. 1 shows a structural schematic diagram of a color brightness compensation system according to some embodiments of the present application. FIG. 2 shows a structural schematic diagram of a color brightness compensation system according to some embodiments of the present application.

[0053] In the embodiment shown in FIG. 1, the color brightness compensation system provided by the second aspect of the present application can be configured in the MR head-mounted display device provided by the third aspect of the present application, and includes a first storage (not shown) and a processor 11. Here, the first storage includes but is not limited to the computer readable storage medium provided by the fourth aspect of the present application, and the computer program product provided by the fifth aspect of the present application is stored on the computer readable storage medium. The computer program product includes computer instructions. The processor 11 is connected to the first storage, and is configured to execute the computer instructions in the computer program product stored on the first storage, so as to implement the color brightness compensation method provided by the first aspect of the present application.

[0054] In addition, in some embodiments, the color and brightness compensation system provided by the second aspect of the present application further comprises a second memory 12 and a display module 13. The second memory 12 can be embodied as an on-chip flash memory, including but not limited to the above-mentioned computer readable storage medium provided by the third aspect of the present application, and stores a color and brightness compensation matrix corresponding to at least one target position. The display module 13 can be configured with a display driving chip, a Pancake lens, and a silicon-based OLED display screen. Here, the processor 11 can be embodied as a display pipeline, which is connected to the second memory 12 via a display driving software or firmware computing platform 14 to obtain compensation data such as the color and brightness compensation matrix provided by the second memory 12, and then performs synchronous pre-compensation on the color and brightness of a plurality of pixel points in the original image to be compensated according to the color and brightness compensation matrix, and transmits the image after synchronous pre-compensation to the display driving chip at the back end, so that it outputs the corresponding display image via the Pancake lens and the silicon-based OLED display screen.

[0055] In addition, in the embodiment shown in FIG. 2, the color and brightness compensation system provided by the second aspect of the present application can further comprise a measurement camera 15 and a plurality of color measurement spectrometers 16. Correspondingly, the processor 11 can be configured with a display distortion correction pipeline 111 and a color and brightness compensation pipeline 112. Here, the display distortion correction pipeline 111 is connected to the display driving chip of the display module 13 and the measurement camera 15, respectively, for calibrating the distortion correction parameters of the Pancake lens and the silicon-based OLED display screen 131 according to the preset gaze point coordinates and the original image collected by the measurement camera 15 from the silicon-based OLED display screen 131 of the display module 13. The color and brightness compensation pipeline 112 is connected to the display distortion correction pipeline 111, the display driving chip, each measurement spectrometer 16, and the above-mentioned second memory 12, respectively, for calibrating the color and brightness compensation matrix for performing chrominance compensation and brightness compensation on the preset first color in the image to be compensated and performing chrominance compensation on the remaining at least one second color according to the preset gaze point coordinates, the image after removing distortion, and the original color collected by the measurement spectrometer 16 from the silicon-based OLED display screen 131, and then storing the color and brightness compensation matrix into the second memory 12 for subsequent use.

[0056] The working principle of the color and brightness compensation system will be described below in combination with some embodiments of color and brightness compensation methods. Those skilled in the art can understand that the embodiments of the color and brightness compensation methods are only some non-limiting embodiments provided by the present application, which are intended to clearly show the main concept of the present application and provide some specific schemes for facilitating the public to implement, rather than to limit the overall function or overall working mode of the color and brightness compensation system. Similarly, the color and brightness compensation system is also only a non-limiting embodiment provided by the present application, which does not limit the execution subject or execution order of each step in the color and brightness compensation methods.

[0057] In some non-limiting embodiments, the color and brightness compensation method provided by the present application can be independently implemented in two stages of offline calibration and online compensation.

[0058] The specific process of offline calibration will be described in combination with reference to FIGS. 3-6. FIG. 3 shows a flowchart of a color and brightness compensation method according to some embodiments of the present application. FIG. 4 shows a schematic diagram of pre-correction of each pure color image according to some embodiments of the present application. FIG. 5 shows a schematic diagram of interpolation completion according to some embodiments of the present application. FIG. 6 shows a schematic diagram of removal of edge outliers according to some embodiments of the present application.

[0059] As shown in FIGS. 3-6, in the offline calibration process of color and brightness compensation, the color and brightness compensation system can first input a standard stripe image (for example, a horizontal and vertical stripe image) into the display module 13 via the display distortion correction pipeline 111, and collect a second display image output by the display module 13 via the measurement camera 15, and then analyze the second display image via the processor 11 to determine a distortion correction grid for compensating the optical distortion of the display module 13. Then, the color and brightness compensation system can input pure color images of multiple colors (for example, red, green, blue, and white) into the display module 13 via the color and brightness compensation pipeline 112, and collect first display images output by the display module 13 via the measurement spectrometer 16 of each corresponding color, and then obtain the distortion correction grid from the display distortion correction pipeline 111 to correct the distortion in each pure color image according to the distortion correction grid. The specific process of determining the distortion correction grid and performing distortion pre-correction based on the distortion correction grid does not involve the technical improvement of the present application, and will not be described here.

[0060] Then, as shown in FIG. 3, the color and brightness compensation system can analyze the first display images of each color without distortion to determine the xyY values x r ,y r ,Y r ,x g ,y g ,Yg x b y b Y b x w y w Y w , and construct a first color luminance matrix based on the x, y, Y values of each color Here, the x, y, Y values are values in the x, y, Y color space, where x is a component describing chromaticity coordinates, indicating the position of the color on the Commission Internationale de Luminance Commission (CIE) chromaticity diagram; y is another component of the chromaticity coordinates, indicating the position of the color on the CIE chromaticity diagram; Y represents the component of luminance or brightness, used to describe the brightness intensity of the color. In this way, the combination of the three components x, y, Y can accurately describe all visible colors.

[0061] Further, in the process of analyzing the first display images of each color and constructing the first color luminance matrix, the processor 11 can preferably first filter each first display image to remove edge outliers in the x, y, Y values of each color, and then divide the x, y, Y values of each color by spatial coordinate position, and use polynomial fitting to fit each row and each column x, y, Y value to determine the first fitting parameter representing the correspondence between the position and the x, y, Y value. Then, the processor 11 can interpolate the blank position and / or the outlier position with abnormal x, y, Y values that do not actually obtain x, y, Y values, to determine the x, y, Y value of each position about the corresponding color in each first display image, respectively, thereby reducing pixel data errors and / or pixel data missing caused by acquisition errors, and improving the compensation effect of color luminance.

[0062] Further, the processor 11 can convert the first color luminance matrix of each position about each color to the XYZ color space to obtain the second color luminance matrix of each position about each color Here, the XYZ color space is a mathematical model for describing colors established by the CIE, used to standardize the perception of the human visual system for all visible colors, where X is a component of the color stimulus, corresponding to the perception of the human eye to red light and very short wavelength blue light; Y is a component of the color stimulus, representing luminance or brightness, corresponding to the perception of the human eye to all visible wavelengths; Z is a component of the color stimulus, corresponding to the perception of the human eye to green light and very long wavelength red light.

[0063] Specifically, the processor 11 can use the following formula to convert the x, y, Y values of each position about the corresponding color in each first display image to the XYZ color space: X = ((x*Y) / y) Z = ((1-x-y)*Y / y) Y = Y.

[0064] Subsequently, the processor 11 can select a first target position (e.g., the center point) from multiple positions of each first display image, and set its Y value with respect to a first color (e.g., white) as the target brightness Y. w Brightness compensation is performed on the Y values ​​of the remaining positions with respect to the first color to determine the brightness matrix of the third color.

[0065] Please refer to Figure 7 for details. Figure 7 shows a schematic diagram of brightness compensation provided according to some embodiments of the present invention.

[0066] As shown in Figure 7, during the brightness compensation process, the processor 11 can first determine the target brightness Y. w The first preset ratio (generally less than 1) is used to determine the brightness compensation threshold. Then, in response to any position in the first displayed image having a Y value with respect to the first color that is greater than or equal to the brightness compensation threshold, the processor 11 can compensate the Y value at that position with respect to the first color to the target brightness Y. w The processor 11 determines the third color luminance matrix to complete luminance compensation. Conversely, if the Y value of any position in the first displayed image with respect to the first color is less than the luminance compensation threshold, the processor 11 can increase the Y value of that position with respect to the first color by a second preset ratio (generally greater than 1) to determine the third color luminance matrix to complete luminance compensation. Here, the product of the first preset ratio and the second preset ratio is generally less than or equal to 1. Compared to applying the same luminance compensation to all pixels of the entire image, the present invention can perform differentiated quantitative compensation based on the actual luminance of each pixel, thereby avoiding image overexposure and distortion caused by excessive luminance compensation in dark areas of the image.

[0067] Furthermore, in some embodiments, the processor 11 may also preferably measure the gamma value of the display module 13 and the maximum brightness value Y of the first color at the currently measured grayscale in the second color brightness matrix before performing brightness compensation. wmax And based on the gamma value and the maximum brightness Y wmax Determine the standard brightness value Y at the corresponding location under a preset grayscale (e.g., grayscale 255). w255 .

[0068] Then, the processor 11 can determine the standard brightness value Y. w255The Y-values ​​of each position in the second color brightness matrix with respect to each color are normalized to determine the fourth color brightness matrix. Then, the Y-value of the first target position in the fourth color brightness matrix with respect to the first color is used as the target brightness. Brightness compensation is then performed on the Y-values ​​of the remaining positions in the fourth color brightness matrix with respect to the first color to determine the third color brightness matrix after brightness compensation. By performing this normalization process, this invention can be compatiblely adapted to images of various grayscale levels for universal color brightness compensation.

[0069] Then, the processor 11 can determine the XYZ vector [X] of the first color from the third color luminance matrix. w ,Y w Z w ], and the XYZ matrices of each second color in the third color luminance matrix. Determine the first transformation matrix to convert the third color luminance matrix to the linear RGB color space.

[0070] Specifically, after determining the aforementioned first transformation matrix During the process, the processor 11 can first determine the XYZ matrix of each second color in the third color brightness matrix mentioned above. and the inverse of the XYZ matrix The first XYZ vector [X] of the first color in the third color brightness matrix. w ,Y w Z w The product of the diagonal matrices of the products of the first transformation matrix determines the first equation with respect to the first transformation matrix:

[0071] Here, diag(·) is a function that creates a diagonal matrix.

[0072] Furthermore, the processor 11 can also base its decisions on the first XYZ vector [X... w ,Y w Z w ], and the first RGB vector [W] of the first color after the third color brightness matrix is ​​converted to the linear RGB color space. r W g W b Determine the second equation regarding the first transformation matrix:

[0073] Furthermore, the processor 11 can also base its calculations on the second XYZ vector [X] of each second color in the third color luminance matrix. r ,Y r Z r ]、[X g ,Y gZ g ]、[X b ,Y b Z b ], and the second RGB vector [R] of each second color after the third color brightness matrix is ​​converted to the linear RGB color space. r ,R g ,R b ]、[G r G g G b ]、[B r B g B b Determine at least one third process regarding the first transformation matrix:

[0074] Then, processor 11 can connect and solve the equations concerning the first transformation matrix to determine the first transformation matrix.

[0075] Furthermore, the processor 11 can preferably divide each first display image into multiple (e.g., m*n) regions according to spatial coordinates, and calculate the average linear RGB value of each region of each first display image with respect to the corresponding color. Then, based on the third RGB vector of the average linear RGB values ​​of each region of the first displayed image with respect to the corresponding color... and the first XYZ vector of the first color in the third color luminance matrix. Determine several first equations with respect to several second transformation matrices:

[0076] Furthermore, the processor 11 can also use a fourth RGB vector based on the average linear RGB values ​​of the corresponding colors for each region of each first display image. And the second XYZ vector [X] of each second color in the third color brightness matrix. r ,Y r Z r ]、[X g ,Y g Z g ]、[X b ,Y b Z b Determine at least one second equation with respect to a plurality of second transformation matrices:

[0077] Furthermore, the processor 11 can also determine multiple third-party processes related to the second transformation matrix based on the aforementioned third color luminance matrix, the third transformation matrix that converts the third color luminance matrix back to the xyY color luminance space, and the aforementioned first color luminance matrix.

[0078] Then, the processor 11 can connect and solve the equations about the second transformation matrix to determine multiple second transformation matrices that transform the third color luminance matrix of each region to the linear RGB color space. And based on the first transformation matrix mentioned above and the second transformation matrix of each region The product of these factors determines the color brightness compensation sub-matrix for each region corresponding to the first target position:

[0079] Furthermore, in some preferred embodiments, after determining the color brightness compensation sub-matrix corresponding to the first target position for each region, the processor 11 can further filter the color brightness compensation matrix composed of the color brightness compensation sub-matrixes to remove outlier values ​​of the compensation elements. Then, the processor 11 can divide each color brightness compensation sub-matrix into rows and columns according to the position of each region, and use a polynomial to fit the values ​​of the corresponding compensation elements in each row and column of the color brightness compensation sub-matrix to determine a second fitting parameter representing the correspondence between the position and the compensation element value. Subsequently, the processor 11 can interpolate the blank compensation elements and / or outlier elements with abnormal compensation element values ​​in each color brightness compensation sub-matrix according to the second fitting parameter, to determine the value of each compensation element in each color brightness compensation sub-matrix, thereby reducing pixel data errors and / or pixel data loss caused by acquisition errors and improving the color brightness compensation effect.

[0080] Next, the processor 11 can also stitch together each color brightness compensation sub-matrix according to the position of each region in the image to obtain a complete second color brightness compensation matrix, and in the form of a parameter LUT table, set the values ​​m of each compensation element in the color brightness compensation matrix of the first target position corresponding to the preset gaze point to the form of a parameter LUT table. x,y,z They are compressed and stored in the second memory (i.e., on-chip flash memory) 12 respectively to reduce data storage requirements.

[0081] Furthermore, in some embodiments, the present invention can also reselect a second target position from the positions of multiple different preset gaze points in three-dimensional space, and repeatedly determine and store o*p*q color brightness compensation matrices corresponding to each second target position as described above, for use in subsequent online compensation stages.

[0082] Furthermore, please refer to Figures 8 and 9. Figure 8 illustrates a schematic diagram of the principle of a statistical model for predicting compensation data according to some embodiments of the present invention. Figure 9 illustrates a schematic diagram of the principle of a statistical model for predicting compensation data according to some embodiments of the present invention.

[0083] In the embodiments shown in FIG. 8 and FIG. 9, the present application can also preferably extract the point data and compensation data in the eyebox space 23 (i.e. the space where the user wearing the head-mounted device experiences better visual effects when the pupil of the human eye is located) of the partial light machine measured sample to establish and train the statistical model. After the calibration of the color and brightness compensation matrix (or its corresponding LUT table) of the first target position 31 in the eyebox space 23 is completed, the present application can also determine at least one second target position 32 from the user's eyebox 23 space, and input each second target position 32 into the pre-trained statistical model to estimate its corresponding color and brightness compensation matrix through the statistical model. After that, the present application can also store the color and brightness compensation matrix corresponding to each second target position 32 in the form of LUT table into the corresponding second memory (i.e. on-chip flash memory) 12, so as to call the color and brightness compensation matrix of each second target position 32 in the subsequent online compensation stage to calculate the compensated virtual image 22.

[0084] Therefore, by performing the above-mentioned related steps of the offline calibration stage, the present application can obtain the color and brightness compensation matrix based on the gaze point position of the user's eye. Compared with the local compensation scheme based on the observation angle, on the one hand, the color and brightness compensation matrix based on the gaze point position of the user's eye does not need to consider the relative angle of the user's eye to the specific pixel point to be compensated, thereby simplifying the algorithm. On the other hand, the color and brightness compensation matrix based on the gaze point position of the user's eye can simultaneously perform synchronous pre-compensation of color and brightness non-uniformity in multiple regions of the entire image to improve the compensation effect of the entire image, thereby maintaining the visual effect of the image during the user's switching of the viewing angle.

[0085] The related steps of the online compensation stage of the present application will be further described below. In the online compensation stage of color and brightness compensation, the processor 11 can obtain the gaze point coordinates (x, y, z) of the user to determine the corresponding color and brightness compensation matrix:

[0086] Herein, the color and brightness compensation matrix includes the chrominance compensation elements Δx w , Δy w and the brightness compensation element ΔY w of the preset first color (e.g. white) in the image to be compensated, and the chrominance compensation elements Δx r , Δy r , Δx g , Δy g , Δx b , Δy b of at least one second color. The values of each compensation element are determined according to the brightness value of the target position corresponding to the gaze point coordinates with respect to the first color.

[0087] Please refer to FIG. 10 and FIG. 11 for details. FIG. 10 shows a flowchart of a color brightness compensation method according to some embodiments of the present application. FIG. 11 shows a diagram of the positions of gaze points according to some embodiments of the present application.

[0088] As shown in FIG. 10 and FIG. 11, in the process of determining the color brightness compensation matrix, the processor 11 can first obtain the coordinates (x, y, z) of the actual gaze point m0of the user, determine at least one target position corresponding to the actual gaze point coordinates, and determine the distance from the target position to the actual gaze point m0.

[0089] For example, the processor 11 can first round the coordinates (x, y, z) of the actual gaze point m0to obtain the corresponding integer coordinates (x', y', z'). At this time, if the integer coordinates (x', y', z') match any one of the preset gaze point coordinates stored in the second memory (i.e., on-chip flash memory), the processor 11 can determine the position of the integer coordinates (x', y', z') as the target position, and determine the distance from the target position to the actual gaze point m0.

[0090] For another example, the processor 11 can also calculate the spatial distance from each of the preset gaze points stored in the second memory (i.e., on-chip flash memory) to the actual gaze point m0coordinates (x, y, z), and take the preset gaze point with the closest spatial distance as the target position corresponding to the actual gaze point m0.

[0091] Then, the processor 11 can determine the color brightness compensation parameter sub-matrix corresponding to each region according to the target position, and combine the color brightness compensation parameter sub-matrices according to the positions of the regions to determine the first color brightness compensation parameter matrix m x,y,z Then, the processor 11 can select at least one second color brightness compensation parameter matrix m x′,y′,z′ from the second memory (i.e., on-chip flash memory) corresponding to the adjacent positions m1-m8 of the preset gaze points, and determine the weight parameters [θ x , θ y , θ z ] corresponding to each of the preset gaze points according to the distance from the target position to the coordinates of each of the preset gaze points. x y z Then, the processor 11 can perform linear interpolation calculation on the first color brightness compensation parameter matrix and each second color brightness compensation parameter matrix according to the weight parameters [θ

[0092] ​​In this case, the second color brightness compensation parameter matrix m corresponding to the at least one adjacent position x′,y′,z′ Optionally, the color brightness compensation parameter matrix of another candidate target position adjacent to the target position in the x or y or z direction is pre-calibrated, which is also obtained from the corresponding second color brightness compensation matrix According to the parameter LUT table form. The value of the weight parameter [θ x , θ y , θ z ] decreases with the increase of the distance between the preset gaze point and the actual gaze point in the corresponding direction. In this way, the present application can further reduce the pixel data errors caused by acquisition errors, thereby improving the color brightness compensation effect.

[0093] After that, the processor can decompress the color brightness compensation matrix according to the above parameter LUT table form to obtain the corresponding color brightness compensation matrix and perform linear interpolation on the m*n*9 parameters of the color brightness compensation matrix to obtain W*H*9 parameters corresponding to the screen resolution of the display module 13. Here, W represents the screen resolution width of the display module 13, and H represents the screen resolution height of the display module 13.

[0094] Those skilled in the art can understand that the above embodiment of interpolating the first color brightness compensation matrix of the actual gaze point coordinates by combining the color brightness compensation matrix of at least one adjacent gaze point position is only some non-limiting embodiments provided by the present application, which aims to clearly demonstrate the main idea of the present application and provide some preferred schemes for reducing pixel data errors caused by acquisition errors, but not for limiting the protection scope of the present application.

[0095] Optionally, in other embodiments, the processor 11 can also directly perform synchronous pre-compensation on the color and brightness of the plurality of pixel points in the to-be-compensated image based on the first color brightness compensation matrix of the actual gaze point coordinates, so as to achieve the effect of improving the compensation efficiency and real-time of color and brightness.

[0096] Those skilled in the art can also understand that the above embodiment of dividing each first display image into a plurality of regions according to the position to determine the color brightness compensation sub-matrix of each region corresponding to the first target position is only some non-limiting embodiments provided by the present application, which aims to clearly demonstrate the main idea of the present application and provide some preferred schemes for improving the color brightness compensation accuracy, but not for limiting the protection scope of the present application.

[0097] Optionally, in some other embodiments, the first display images can not be partitioned, and the average linear RGB values of the first display images with respect to the corresponding colors can be directly calculated by those skilled in the art. Then, the second conversion matrix for converting the third color luminance matrix to the linear RGB color space can be determined by combining the XYZ vectors with respect to the colors in the third color luminance matrix. Finally, the color luminance compensation matrix corresponding to the preset gaze point position can be determined according to the product of the first conversion matrix and the second conversion matrix as described above.

[0098] Then, please further refer to FIG. 12. FIG. 12 shows a schematic diagram of the synchronization pre-compensation of color and luminance according to some embodiments of the present application.

[0099] As shown in FIG. 10, the processor 11 can determine the color luminance compensation matrix corresponding to the preset gaze point position according to the product of the first conversion matrix and the second conversion matrix as described above. The display module 13 is inputted to perform the synchronization pre-compensation of color and luminance for the plurality of pixel points in the image to be compensated via the display module 13, so as to present the optimal image display effect to the user in real time and efficiently.

[0100] Those skilled in the art can understand that, although the related steps in the offline calibration stage and the online compensation stage of the present application are recorded continuously and completely in the same or related embodiments of the present application, this does not mean that they must be implemented continuously and completely by the same subject or the same device as recorded in the present application.

[0101] Optionally, in some other embodiments, the related steps in the offline calibration stage of the present application can be performed by a first subject or a first device, and the color luminance compensation matrix obtained by calibration can be stored in the second memory (i.e., on-chip flash memory) 12. Then, a second subject who needs to perform color luminance compensation on an image can purchase or lease the related compensation data of the color luminance compensation matrix from the first subject, or store the second memory (i.e., on-chip flash memory) 12 of the color luminance compensation matrix. In this way, the second subject can obtain the color luminance compensation matrix via a second device only comprising the first memory and the processor 11, and perform the related steps in the online compensation stage of the present application to pre-compensate the color and luminance non-uniformity of the plurality of regions in the entire image according to the gaze point position of the user's eye in real time, so as to present the optimal image display effect to the user in real time and efficiently.

[0102] In summary, the color brightness compensation method, the color brightness compensation system, the MR head-mounted display device, the computer readable storage medium and the computer program product provided by the present application can determine the brightness value of the corresponding target position with respect to the preset first color according to the gaze point coordinates of the user, and determine the color brightness compensation matrix for performing chroma compensation and brightness compensation on the first color in the to-be-compensated image and performing chroma compensation on the remaining at least one second color, and then perform synchronous pre-compensation of color and brightness non-uniformity based on the color brightness compensation matrix, so as to present the optimal image display effect to the user in real time and efficiently.

[0103] Although the above-described methods are illustrated and described as a series of acts, it will be appreciated and understood by those skilled in the art that the methods are not limited by the order of acts, as some acts can, in accordance with one or more embodiments, occur simultaneously or in different order than shown and described herein, or can occur with other acts not expressly shown and described herein.

[0104] Those skilled in the art will appreciate that information, signals, and data can be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that can be referenced throughout the above description can be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0105] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality, without reference to a specific

[0106] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the generic principles defined herein can be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the examples and designs described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A color brightness compensation method, characterized by, The method comprises the following steps: obtaining gaze point coordinates of a user to determine a corresponding color and brightness compensation matrix, wherein the color and brightness compensation matrix comprises chrominance compensation elements and brightness compensation elements of a preset first color in a to-be-compensated image and chrominance compensation elements of at least one second color, and the values of the compensation elements are determined according to the target position corresponding to the gaze point coordinates and the brightness value of the first color; and inputting the color and brightness compensation matrix into a display module to synchronously pre-compensate the color and brightness of multiple pixel points in the to-be-compensated image via the display module.

2. The color brightness compensation method of claim 1, wherein, The color and brightness compensation matrix is pre-calibrated, wherein the step of calibrating the color and brightness compensation matrix comprises: inputting pure color images of multiple colors into the display module respectively, and analyzing first display images thereof to respectively construct second color and brightness matrices of multiple positions in each of the first display images in the XYZ color space with respect to the corresponding color; selecting a first target position from the multiple positions, and performing brightness compensation on the Y values of the remaining positions with respect to the first color with the Y value of the first target position with respect to the first color as the target brightness to determine a third color and brightness matrix; determining a first conversion matrix for converting the third color and brightness matrix to a linear RGB color space according to the third color and brightness matrix, and determining a second conversion matrix for converting the third color and brightness matrix to the linear RGB color space in combination with the average linear RGB values of each of the first display images with respect to the corresponding color; and determining the color and brightness compensation matrix corresponding to the first target position according to the product of the first conversion matrix and the second conversion matrix.

3. The color brightness compensation method of claim 2, wherein, Before inputting the pure color images of multiple colors into the display module, the color and brightness compensation method further comprises the following steps: inputting a standard stripe image into the display module, and analyzing a second display image thereof to determine a distortion correction grid of the display module; and correcting each of the pure color images according to the distortion correction grid.

4. The color brightness compensation method of claim 2, wherein, The step of analyzing the first display images of each of the colors to respectively construct second color and brightness matrices of multiple positions in each of the first display images in the XYZ color space with respect to the corresponding color comprises: analyzing the first display images of each of the colors to respectively determine xyY values of multiple positions in each of the first display images with respect to the corresponding color; constructing a first color and brightness matrix according to the xyY values of each of the positions with respect to each of the colors; and converting the first color and brightness matrix of each of the positions with respect to each of the colors to the XYZ color space to respectively obtain second color and brightness matrices of each of the positions with respect to each of the colors.

5. The color brightness compensation method of claim 4, wherein, The step of analyzing the first display images of each of the colors to respectively determine xyY values of multiple positions in each of the first display images with respect to the corresponding color comprises: filtering each of the first display images to respectively remove edge outliers in the xyY values of each of the colors; According to the positions, the xyY values of each of the colors are divided into rows and columns, and polynomial fitting is performed on the xyY values of each row and each column to determine first fitting parameters representing the correspondence between the positions and the xyY values; and According to the first fitting parameters, interpolation is performed on blank positions for which the xyY values are not actually acquired and / or abnormal value positions for which the xyY values are abnormal, to respectively determine the xyY values of a plurality of positions in each of the first display images with respect to the corresponding color.

6. The color brightness compensation method of claim 2, wherein, The step of selecting a first target position from the plurality of positions and performing luminance compensation on the Y values of the remaining positions with respect to the first color with a target luminance of the Y value of the first target position with respect to the first color to determine a third color luminance matrix comprises: measuring a gamma value of the display module and a maximum luminance of the first color in the second color luminance matrix at a currently measured gray scale; determining a standard luminance value of the corresponding position at a preset gray scale according to the gamma value and the maximum luminance; performing normalization processing on the Y values of each of the positions with respect to each of the colors in the second color luminance matrix according to the standard luminance value to determine a fourth color luminance matrix; and performing luminance compensation on the Y values of the remaining positions with respect to the first color in the fourth color luminance matrix with a target luminance of the Y value of the first target position with respect to the first color in the fourth color luminance matrix to determine the third color luminance matrix.

7. The color brightness compensation method of claim 2, wherein, The step of selecting a first target position from the plurality of positions and performing luminance compensation on the Y values of the remaining positions with respect to the first color with a target luminance of the Y value of the first target position with respect to the first color to determine a third color luminance matrix comprises: determining a luminance compensation threshold according to a first preset proportion of the target luminance, wherein the first preset proportion is less than 1; in response to the Y value of any of the positions with respect to the first color being greater than or equal to the luminance compensation threshold, compensating the Y value of the position with respect to the first color to the target luminance; and in response to the Y value of any of the positions with respect to the first color being less than the luminance compensation threshold, increasing the Y value of the position with respect to the first color according to a second preset proportion, wherein the product of the first preset proportion and the second preset proportion is less than or equal to 1.

8. The color brightness compensation method of claim 2, wherein, The step of determining a first conversion matrix for converting the third color luminance matrix to a linear RGB color space according to the third color luminance matrix comprises: determining a first conversion matrix for converting the third color luminance matrix to a linear RGB color space according to the XYZ vector with respect to the first color in the third color luminance matrix and the XYZ matrix with respect to each of the second colors in the third color luminance matrix.

9. The color brightness compensation method of claim 8, wherein, The step of determining a first conversion matrix for converting the third color luminance matrix to a linear RGB color space according to the XYZ vector with respect to the first color in the third color luminance matrix and the XYZ matrix with respect to each of the second colors in the third color luminance matrix comprises: determining a first equation for the first conversion matrix from the first XYZ vector and a first RGB vector for the first color after the third color luminance matrix is converted to the linear RGB color space; determining a second equation for the first conversion matrix from the first XYZ vector and a first RGB vector for the first color after the third color luminance matrix is converted to the linear RGB color space; determining at least one third equation for the first conversion matrix from second XYZ vectors for each of the second colors in the third color luminance matrix and second RGB vectors for each of the second colors after the third color luminance matrix is converted to the linear RGB color space; and solving the equations for the first conversion matrix.

10. The color brightness compensation method of claim 2, wherein, The step of determining a second conversion matrix for converting the third color luminance matrix to the linear RGB color space from the average linear RGB values for each of the first display images for the corresponding colors and the XYZ vectors for each of the colors in the third color luminance matrix includes: calculating the average linear RGB values for each of the first display images for the corresponding colors separately and determining a second conversion matrix for converting the third color luminance matrix to the linear RGB color space from the average linear RGB values for each of the first display images for the corresponding colors and the XYZ vectors for each of the colors in the third color luminance matrix.

11. The color brightness compensation method of claim 10, wherein, The step of calculating the average linear RGB values for each of the first display images for the corresponding colors separately and determining a second conversion matrix for converting the third color luminance matrix to the linear RGB color space from the average linear RGB values for each of the first display images for the corresponding colors and the XYZ vectors for each of the colors in the third color luminance matrix includes: determining a first equation for the second conversion matrix from a third RGB vector of the average linear RGB values for the first color of each of the first display images and a first XYZ vector for the first color in the third color luminance matrix; determining at least one second equation for the second conversion matrix from fourth RGB vectors of the average linear RGB values for each of the second colors of each of the first display images and second XYZ vectors for each of the second colors in the third color luminance matrix; determining a third equation for the second conversion matrix from the third color luminance matrix, a third conversion matrix for converting the third color luminance matrix back to the xyY color luminance space, and the first color luminance matrix; and solving the equations for the second conversion matrix.

12. The color brightness compensation method of claim 10, wherein, The step of calculating the average linear RGB values for each of the first display images for the corresponding colors separately and determining a second conversion matrix for converting the third color luminance matrix to the linear RGB color space from the average linear RGB values for each of the first display images for the corresponding colors and the XYZ vectors for each of the colors in the third color luminance matrix includes: calculating average linear RGB values of each of the regions of each of the first display images with respect to a corresponding color; and determining a plurality of second conversion matrices for converting third color luminance matrices of each of the regions to the linear RGB color space, according to the average linear RGB values of each of the regions of each of the first display images with respect to a corresponding color, and the XYZ vectors in the third color luminance matrix with respect to each of the colors.

13. The color brightness compensation method of claim 12, wherein, The step of determining the color luminance compensation matrix corresponding to the first target position according to the product of the first conversion matrix and the second conversion matrix comprises: determining a color luminance compensation sub-matrix corresponding to the first target position for each of the regions according to the product of the first conversion matrix and the second conversion matrix of each of the regions.

14. The color brightness compensation method of claim 13, wherein, After determining the color luminance compensation sub-matrix corresponding to the first target position for each of the regions, the color luminance compensation method further comprises the following steps: filtering the color luminance compensation matrix composed of the color luminance compensation sub-matrices to remove abnormal values of compensation elements in the color luminance compensation matrix; dividing the color luminance compensation sub-matrices according to positions of each of the regions, and fitting values of corresponding compensation elements in each row and each column of the color luminance compensation sub-matrices by using a polynomial to determine second fitting parameters representing a corresponding relationship between positions and values of compensation elements; interpolating blank compensation elements and / or abnormal elements with abnormal values of compensation elements in the color luminance compensation sub-matrices for which values of compensation elements are not actually obtained according to the second fitting parameters, to determine values of each of the compensation elements in each of the color luminance compensation sub-matrices; and compressively storing the values of each of the compensation elements in each of the color luminance compensation sub-matrices corresponding to the first target position.

15. The color brightness compensation method of claim 2, wherein, After determining the color luminance compensation matrix corresponding to the first target position, the color luminance compensation method further comprises the following steps: reselecting at least one second target position from the plurality of positions, and respectively determining and storing color luminance compensation matrices corresponding to each of the second target positions.

16. The color brightness compensation method of claim 15, wherein, The step of reselecting at least one second target position from the plurality of positions, and respectively determining and storing color luminance compensation matrices corresponding to each of the second target positions comprises: determining at least one second target position from within the eyebox space of the user; inputting each of the second target positions into a pre-trained statistical model to respectively estimate a corresponding color luminance compensation matrix via the statistical model; and storing the color luminance compensation matrices corresponding to each of the second target positions in the form of a LUT table to a corresponding memory.

17. The color brightness compensation method of claim 1, wherein, The step of obtaining the gaze point coordinates of the user to determine a corresponding color luminance compensation matrix comprises: obtaining the gaze point coordinates of the user to determine a target position corresponding to the gaze point coordinates, and determining a distance from the target position to the gaze point coordinates; determining a first color luminance compensation matrix corresponding to the target position, and a second color luminance compensation matrix corresponding to at least one neighboring position of the target position according to the target position; determine a corresponding weight parameter according to a distance from the target position to the gaze point coordinate, wherein the weight parameter decreases as the distance increases; and perform linear interpolation calculation on the first color and brightness compensation matrix and each second color and brightness compensation matrix according to the weight parameter to determine a color and brightness compensation matrix corresponding to the gaze point coordinate.

18. The color brightness compensation method of claim 1, wherein, The image to be compensated is divided into multiple regions according to positions, and the step of obtaining the gaze point coordinate of the user to determine a corresponding color and brightness compensation matrix comprises: obtaining the gaze point coordinate of the user to determine a corresponding target position; determining a color and brightness compensation sub-matrix corresponding to the target position for each region according to the target position; and combining the color and brightness compensation sub-matrices of each region according to the positions of the regions to determine a color and brightness compensation matrix corresponding to the gaze point coordinate.

19. A color brightness compensation system characterized by, comprise: a first memory having computer instructions stored thereon; and a processor connected to the first memory and configured to execute the computer instructions stored on the first memory to implement the color and brightness compensation method according to any one of claims 1-18.

20. The color brightness compensation system of claim 19, wherein, Further comprise: a second memory having color and brightness compensation matrices corresponding to at least one target position stored thereon; and a display module comprising a display driver chip, a Pancake lens, and a silicon-based OLED display screen, wherein the processor is further connected to the second memory and the display driver chip, and is configured to perform synchronous pre-compensation on color and brightness of multiple pixel points in an image to be compensated according to the color and brightness compensation matrices stored on the second memory, and then transmit the image after the synchronous pre-compensation to the display driver chip for outputting a corresponding display image via the Pancake lens and the silicon-based OLED display screen.

21. The color brightness compensation system of claim 20, wherein, The color and brightness compensation system further comprises a measurement camera and a plurality of color spectrometers, and the processor comprises a display distortion correction pipeline and a color and brightness compensation pipeline, wherein the display distortion correction pipeline is connected to the display driver chip and the measurement camera, and is configured to input a standard fringe image to the display driver chip and collect a second display image output by the display module via the measurement camera; and determine a distortion correction grid of the display module according to the second display image, the color and brightness compensation pipeline is connected to the display driver chip, each measurement spectrometer, and the second memory, and is configured to obtain the distortion correction grid from the display distortion correction pipeline to pre-correct a plurality of color pure images; input each pre-corrected color pure image to the display module respectively, and collect first display images of corresponding colors output by the display module via each color measurement spectrometer respectively; and determine a color and brightness compensation matrix corresponding to the target position according to xyY values of multiple positions in each first display image with respect to the corresponding color, and store the color and brightness compensation matrix in the second memory.

22. An MR head-mounted display device, comprising: The MR head-mounted display device is configured with the color brightness compensation system according to any one of claims 19-21.

23. A computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions, when executed by a processor, implement the color brightness compensation method according to any one of claims 1-18.

24. A computer program product comprising computer instructions, characterized in that, The computer instructions, when executed by a processor, implement the color brightness compensation method according to any one of claims 1-18.

Images