Joint pre-compensation method and system for distortion and chromatic aberration, and head-mounted display device

Abstract

Provided in the present invention are a joint pre-compensation method and system for distortion and chromatic aberration, and an MR head-mounted display device. The joint pre-compensation method for distortion and chromatic aberration comprises the following steps: acquiring a first image to be compensated for and gaze point information of a user; on the basis of the gaze point information, determining a first distortion compensation table related to a first color, and a first pixel color offset table of at least one second color relative to the first color; and on the basis of the first distortion compensation table and each first pixel color offset table, respectively processing the first image, so as to obtain a second image on which the joint pre-compensation is completed. In this way, the present invention can realize high-quality visual output at a low cost, and achieve optimal visual effects at different viewing angles.

Description

Joint pre-compensation method, system and head-mounted display device for distortion and chromatic aberration

[0001] This application claims priority to the patent application with the application date of August 16, 2024, the Chinese application number of 202411129948.7, and the title of "Joint pre-compensation method, system and head-mounted display device for distortion and chromatic aberration", and incorporates its entire content by reference into the present application. TECHNICAL FIELD

[0002] The present application relates to the field of mixed reality (MR) display technology, and in particular to a joint pre-compensation method for distortion and chromatic aberration, a joint pre-compensation system for distortion and chromatic aberration, an MR head-mounted display device, a computer readable storage medium, and a computer program product. BACKGROUND

[0003] In the field of MR display, chromatic aberration compensation is a key issue that directly affects the user's visual experience and comfort. Chromatic aberration is a phenomenon caused by the different focal lengths of lenses when refracting light of different wavelengths, resulting in colored halos on the edges of the image. In particular, in MR head-mounted display devices (hereinafter referred to as "headsets"), due to the wide field of view (FOV), the optical elements used often need to have an exceptionally complex design to reduce chromatic aberration, which undoubtedly increases the cost and complexity of the system. In addition, because of the inherent optical properties of the lens, when the user's gaze point or viewing angle changes, the angle relationship with the lens also changes, which means that the effect of chromatic aberration will not always be the same, it will change with the movement of the user's line of sight.

[0004] In existing distortion and chromatic aberration compensation technologies, the main solution is to use multiple optical elements to offset chromatic aberration. However, this method relies on complex optical design, introduces additional optical elements, increases the thickness and weight of the headset device, and it is difficult to take into account factors such as user viewing angle, gaze point, etc. In addition, another prior art obtains red, green and blue distortion grids according to the formula obtained according to the measured chromatic aberration in the rendering pipeline, and renders the red, green and blue pixels in turn according to the distortion grids, and then outputs them to the display end. However, this solution needs to be completed synchronously during rendering, which will occupy the computing power of the graphics processing unit (GPU), and there are problems such as generating more power consumption, reducing rendering effect, etc. In addition, these existing technologies generally do not take into account how to take into account factors such as user viewing angle, gaze point, etc.

[0005] In order to overcome the above-mentioned defects of the prior art, the technical field urgently needs an improved joint pre-compensation method for distortion and chromatic aberration, which is used to reduce the cache requirement of the processor and maintain the simplicity of the optical system design, so as to reduce the complexity and cost of the optical design, reduce or even eliminate the chromatic aberration caused by the gaze angle, and achieve high-quality visual output at a lower cost and optimal visual effect at different observation angles. SUMMARY

[0006] The following presents a simplified summary of one or more aspects in order to provide a basic understanding of such aspects. This summary is not an extensive overview of all contemplated aspects, and is intended to neither identify key or critical elements of all aspects nor delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that is presented later.

[0007] In order to overcome the above-mentioned defects of the prior art, the technical field urgently needs an improved joint pre-compensation method for distortion and chromatic aberration, which is used to reduce the cache requirement of the processor and maintain the simplicity of the optical system design, so as to reduce the complexity and cost of the optical design, reduce or even eliminate the chromatic aberration caused by the gaze angle, and achieve high-quality visual output at a lower cost and optimal visual effect at different observation angles.

[0008] Specifically, according to the first aspect of the present application, the above-mentioned joint pre-compensation method for distortion and chromatic aberration comprises the following steps: obtaining a first image to be compensated and gaze point information of a user; determining a first distortion compensation table for a first color and at least one first pixel color offset table for a second color compared with the first color according to the gaze point information; and processing the first image according to the first distortion compensation table and each first pixel color offset table, respectively, to obtain a second image after the joint pre-compensation.

[0009] Further, in some embodiments of the present application, the first distortion compensation table is determined according to at least one pre-calibrated second distortion compensation table. The step of calibrating the second distortion compensation table comprises: inputting the horizontal and vertical striped image of the first color into the display module, and using a standard camera to capture a second image distorted by the display module at the position of at least one preset gaze point; performing edge detection on the second image to determine a plurality of horizontal striped edge lines and a plurality of vertical striped edge lines therein; sorting each of the horizontal striped edge lines and each of the vertical striped edge lines in the second image, and determining first horizontal coordinates and first vertical coordinates of a plurality of ordered intersection points according to the serial numbers of each of the horizontal striped edge lines and each of the vertical striped edge lines; and constructing a second distortion compensation table corresponding to the preset gaze point according to the first horizontal coordinates and the first vertical coordinates of each of the ordered intersection points.

[0010] Further, in some embodiments of the present application, before performing edge detection on the second image, the joint pre-compensation method further comprises the following steps: inputting a solid image of the first color into the display module, and using the standard camera to capture a third image distorted by the display module, to determine the relative gray scale value of the third image with respect to the first color; obtaining a camera distortion grid of the standard camera, and processing the second image according to the camera distortion grid to obtain a second image without camera distortion; performing first normalization processing on the gray scale of the second image without camera distortion according to the relative gray scale value; and performing binaryzation processing on the second image after the first normalization processing to obtain a second image after the binaryzation processing.

[0011] Further, in some embodiments of the present application, after constructing the second distortion compensation table corresponding to the preset gaze point according to the first horizontal coordinates and the first vertical coordinates of each of the ordered intersection points, the joint pre-compensation method further comprises the following steps: determining a first fitting function of the distortion of the display module according to the first horizontal coordinates and the first vertical coordinates of each of the ordered intersection points in the second distortion compensation table, and the second horizontal coordinates and the second vertical coordinates of the corresponding intersection points in the horizontal and vertical striped image; substituting the second horizontal coordinates and the second vertical coordinates of at least one unknown intersection point in the second distortion compensation table into the first fitting function to determine the first horizontal coordinates and the first vertical coordinates of the at least one unknown intersection point in the second image; and completing the second distortion compensation table according to the first horizontal coordinates and the first vertical coordinates of the at least one unknown intersection point in the second image.

[0012] Further, in some embodiments of the present application, after the second distortion compensation table is completed, the joint pre-compensation method further comprises the following steps: obtaining a target resolution of the second distortion compensation table, and combining the mapping image coordinate values of the plurality of units in the second distortion compensation table to determine a scaling ratio and / or a rotation ratio of the second normalization processing; and performing second normalization processing on the second distortion compensation table according to the scaling ratio and / or the rotation ratio.

[0013] Further, in some embodiments of the present application, after the second distortion compensation table corresponding to the preset gaze point is constructed according to the first horizontal coordinates and the first vertical coordinates of the ordered intersection points, the joint pre-compensation method further comprises the following steps: obtaining a screen resolution of the display module; and performing linear interpolation on the second distortion compensation table according to the screen resolution to obtain a second distortion compensation table conforming to the screen resolution.

[0014] Further, in some embodiments of the present application, the first pixel color offset table is determined according to at least one pre-calibrated second pixel color offset table. The steps of calibrating the second pixel color offset table comprise: inputting the horizontal and vertical striped images of each second color into the display module respectively, and using a standard camera to capture a fourth image distorted by the display module at the position of the preset gaze point, to construct a third distortion compensation table for each second color respectively; obtaining a fifth image simulating the optical path distortion of the first color and each second color by the display module, and using the second distortion compensation table and each third distortion compensation table respectively to map the fifth image corresponding to the color, to generate a sixth image of each color after distortion compensation; performing edge detection on each sixth image respectively, to construct a fourth distortion compensation table corresponding to the color according to the fourth horizontal coordinates and the fourth vertical coordinates of the ordered intersection points of a plurality of horizontal striped edge lines and a plurality of vertical striped edge lines therein; and respectively subtracting the fourth distortion compensation table of each second color from the fourth distortion compensation table of the first color, to determine the second pixel color offset table of each second color compared with the first color corresponding to the preset gaze point.

[0015] Further, in some embodiments of the present application, the step of constructing the third distortion compensation table of each second color respectively comprises: performing edge detection on the fourth image of each second color respectively, to determine a plurality of horizontal striped edge lines and a plurality of vertical striped edge lines therein; sorting each horizontal striped edge line and each vertical striped edge line respectively, to determine the third horizontal coordinates and the third vertical coordinates of the ordered intersection points of each horizontal striped edge line and each vertical striped edge line; and constructing the third distortion compensation table of each second color according to the third horizontal coordinates and the third vertical coordinates of each ordered intersection point.

[0016] Further, in some embodiments of the present application, the step of acquiring the fifth image simulating the optical path distortion of the first color and each of the second colors by the display module comprises: determining a distortion simulation table simulating the optical path distortion of the first color by the display module according to the second distortion compensation table; mapping the horizontal and vertical stripe image of the first color via the second distortion compensation table to generate the fifth image simulating the optical path distortion of the first color by the display module; and mapping the horizontal and vertical stripe image of each of the second colors via the second distortion compensation table to generate the fifth image simulating the optical path distortion of each of the second colors by the display module, respectively.

[0017] Further, in some embodiments of the present application, the step of determining the distortion simulation table simulating the optical path distortion of the first color by the display module according to the second distortion compensation table comprises: determining the maximum width and the maximum height of the second image according to the center point coordinates of the horizontal and vertical stripe image of the first color and the second distortion compensation table; traversing each of the units in the second distortion compensation table within the range of the maximum width and the maximum height, and determining two triangles in combination with the horizontal and vertical coordinates of the three adjacent units thereof; performing barycentric interpolation on the mapping image coordinate values of a plurality of points inside each of the triangles according to the mapping image coordinate values of the endpoints thereof; constructing the distortion simulation table according to each of the units in the second distortion compensation table and the mapping image coordinate values of each of the points obtained by interpolation; and taking the center of the distortion simulation table as the origin to crop it into the same resolution as the horizontal and vertical stripe image of the first color.

[0018] Further, in some embodiments of the present application, the first distortion compensation table is determined according to at least one pre-calibrated second distortion compensation table. The first pixel color offset table is determined according to at least one pre-calibrated second pixel color offset table. The steps of calibrating the second distortion compensation table and the second pixel color offset table comprise: determining a plurality of preset gaze points; and calibrating the corresponding second distortion compensation table and the second pixel color offset table according to the position of each of the preset gaze points, respectively.

[0019] Further, in some embodiments of the present application, after calibrating the second distortion compensation table corresponding to each of the preset gaze points, the joint pre-compensation method further comprises the following steps: determining the second fitting function of the mapping image coordinate value with respect to the horizontal and vertical coordinates of the corresponding unit according to the horizontal and vertical coordinates and the mapping image coordinate value of each of the units in each of the second distortion compensation tables, respectively; and extracting and storing the function relationship parameters representing the second fitting function of each of the second distortion compensation tables, respectively.

[0020] Further, in some embodiments of the present application, the step of determining the first distortion compensation table comprises: determining corresponding function relationship parameters according to the gaze point information; determining a second fitting function of a mapping image coordinate value with respect to a horizontal coordinate and a vertical coordinate of a corresponding unit according to the function relationship parameters; and determining the mapping image coordinate value corresponding to the horizontal coordinate and the vertical coordinate of a plurality of units in the first distortion compensation table according to the second fitting function.

[0021] Further, in some embodiments of the present application, the step of determining the first distortion compensation table comprises: determining a plurality of preset gaze points adjacent to the actual gaze point corresponding thereto according to the gaze point information, and respectively determining a second distortion compensation table of each of the preset gaze points with respect to the first color; respectively determining a weight of each of the preset gaze points according to a distance from the actual gaze point to each of the preset gaze points; and respectively interpolating each of the second distortion compensation tables according to the weight to determine the first distortion compensation table.

[0022] Further, in some embodiments of the present application, the step of determining the first pixel color offset table comprises: respectively determining a second pixel color offset table of at least one second color with respect to the first color for each of the preset gaze points; and interpolating each of the second pixel color offset tables according to the weight to determine the first pixel color offset table.

[0023] Further, in some embodiments of the present application, the step of respectively processing the first image according to the first distortion compensation table and each of the first pixel color offset tables to obtain the second image completed with the joint pre-compensation comprises: mapping the first image using the first distortion compensation table to determine a seventh image that has been distortion compensated; respectively calculating a pixel compensation amount of each of the pixel points in the seventh image with respect to a corresponding second color according to a value of each unit in each of the first pixel color offset tables; and respectively performing chromatic aberration compensation on each of the pixel points in the seventh image according to the pixel compensation amount with respect to each of the second colors to obtain the second image completed with the joint pre-compensation.

[0024] Further, in some embodiments of the present application, before mapping the first image using the first distortion compensation table, the joint pre-compensation method further comprises the following steps: determining a compression format grouping table according to the gaze point information. The gaze point area of the actual gaze point in the compression format grouping table has a minimum first compression ratio, the first non-gaze point area adjacent to the gaze point area has a larger second compression ratio, and the second non-gaze point area away from the gaze point area has a maximum third compression ratio; and interpolating the first distortion compensation table to the image resolution of the first image according to the compression format grouping table.

[0025] Further, in some embodiments of the present application, the step of calculating the pixel compensation amount of each pixel in the seventh image with respect to the corresponding second color according to the value of each cell in each of the first pixel color offset tables comprises: determining the coordinate position of each of the pixels in the seventh image in each of the first pixel color offset tables according to the grouping compression rule of the compressed format grouping table; and interpolating each of the first pixel color offset tables to the image resolution of the first image according to the compressed format grouping table to determine the pixel compensation amount of the corresponding coordinate position of each of the pixels in the seventh image in each of the first pixel color offset tables.

[0026] Further, in some embodiments of the present application, the step of interpolating each of the first pixel color offset tables to the image resolution of the first image according to the compressed format grouping table to determine the pixel compensation amount of the corresponding coordinate position of each of the pixels in the seventh image in each of the first pixel color offset tables comprises: in response to the existence of a compression region change in the coordinate position of any of the pixels in the seventh image in any of the first pixel color offset tables of the second color, calculating the pixel compensation components of a plurality of compression regions before and after the change for the pixel; and summing the pixel compensation components of the plurality of compression regions before and after the change for the pixel to determine the pixel compensation amount of the pixel.

[0027] Further, in some embodiments of the present application, the step of performing chromatic aberration compensation on each of the pixels in the seventh image according to the pixel compensation amount of each of the second colors to obtain the second image with completed joint pre-compensation comprises: judging whether the coordinate position of each of the pixels in the seventh image after the chromatic aberration compensation exceeds the boundary of the second image; and in response to the judgment result that the coordinate position of any of the pixels after the chromatic aberration compensation exceeds the boundary of the second image, filling the gray value of the pixel according to the gray value of the corresponding boundary pixel in the second image.

[0028] Further, in some embodiments of the present application, the step of performing chromatic aberration compensation on each of the pixels in the seventh image according to the pixel compensation amount of each of the second colors to obtain the second image with completed joint pre-compensation comprises: using a filter with a size of M*N to perform weighted summation on the gray values of M*N pixels adjacent to a plurality of target pixels in the seventh image after the chromatic aberration compensation to determine the second image with completed joint pre-compensation.

[0029] Further, in some embodiments of the present application, the first color is green. The at least one second color includes red and blue. Before performing weighted summation on the gray scale values of the M*N pixels adjacent to each target pixel in the seventh image after the color difference compensation, the joint pre-compensation method further includes the following steps: determining the red component position and the blue component position of each pixel in the seventh image after the color difference compensation, respectively, to determine whether the filter area of the red component position and / or the blue component position exceeds the line buffer area divided in advance; and in response to the determination result that the filter area of the red component position and / or the blue component position exceeds the line buffer area, performing linear translation on the red component position and / or the blue component position whose filter area exceeds the line buffer area to move the filter area back to the line buffer area.

[0030] Further, in some embodiments of the present application, after obtaining the second image after the joint pre-compensation is completed, the joint pre-compensation method further includes the following steps: decompressing the second image according to the screen resolution of the display module to obtain an eighth image to be displayed; and inputting the eighth image into the display module to display the image without distortion and color difference on the screen.

[0031] In addition, the joint pre-compensation system of distortion and color difference according to the second aspect of the present application includes a memory and a processor. The memory stores computer instructions. The processor is connected to the memory and is configured to execute the computer instructions stored in the memory to implement the joint pre-compensation method of distortion and color difference according to the first aspect of the present application.

[0032] Further, in some embodiments of the present application, the processor is configured with a display pipeline. The display pipeline is configured to obtain a first image to be compensated and gaze point information of a user; determine a first distortion compensation table for a first color and at least one second color pixel color offset table compared to the first color according to the gaze point information; and process the first image according to the first distortion compensation table and each first pixel color offset table, respectively, to obtain a second image after the joint pre-compensation is completed.

[0033] Further, in some embodiments of the present application, the display pipeline is configured with a distortion compensation circuit, a group compression circuit, a group interpolation circuit, a chromatic aberration compensation circuit and / or a decompression circuit. The distortion compensation circuit is configured to map the first image according to the first distortion compensation table to determine a seventh image that is distortion compensated. The group compression circuit is configured to compress the first distortion compensation table and / or the first pixel color offset table according to a compression format group table corresponding to the gaze point information. The group interpolation circuit is configured to interpolate the first distortion compensation table and / or the first pixel color offset table to an image resolution of the first image according to the compression format group table corresponding to the gaze point information. The chromatic aberration compensation circuit is configured to calculate a pixel compensation amount for each pixel point in the seventh image with respect to a corresponding second color according to a value of each cell in the first pixel color offset table, and to perform chromatic aberration compensation on each of the pixel points in the seventh image according to the pixel compensation amount with respect to each of the second colors to obtain a second image that is completed with the joint pre-compensation. The decompression circuit is configured to decompress the second image to a screen resolution of the display module.

[0034] Further, in some embodiments of the present application, the processor is further configured with a distortion pre-compensation pipeline, a chromatic aberration compensation pipeline, a line of sight calculation pipeline and / or a line of sight interpolation pipeline. The distortion pre-compensation pipeline is configured to construct a second distortion compensation table with respect to a first color and corresponding to a plurality of preset gaze points. The chromatic aberration compensation pipeline is configured to construct at least one second pixel color offset table with respect to each of the preset gaze points and with respect to a second color compared to the first color. The line of sight calculation pipeline is configured to obtain gaze point information of the user. The line of sight interpolation pipeline is configured to perform compression and / or decompression on each of the second distortion compensation tables and / or each of the second pixel color offset tables based on the gaze point information.

[0035] Further, in some embodiments of the present application, the joint pre-compensation system further comprises an on-chip flash memory and / or a display module and / or a standard camera. The on-chip flash memory is configured to store the second distortion compensation table and / or the second pixel color offset table compressed by the line of sight interpolation pipeline. The display module is configured to display an image input therein. The standard camera is configured to capture an image distorted by the display module at a position of each of the preset gaze points.

[0036] In addition, according to the third aspect of the present application, the above-mentioned MR head-mounted display device is provided, wherein the distortion and chromatic aberration joint pre-compensation system of the second aspect of the present application is configured.

[0037] Further, the computer readable storage medium according to the fourth aspect of the present application is provided with computer instructions. The computer instructions are executed by a processor to implement the method of joint pre-compensation of distortion and chromatic aberration according to the first aspect of the present application.

[0038] Further, the computer program product according to the fifth aspect of the present application comprises computer instructions. The computer instructions are executed by a processor to implement the method of joint pre-compensation of distortion and chromatic aberration according to the first aspect of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0039] The above features and advantages of the present application will be better understood through reading the detailed description of embodiments of the present application in conjunction with the following drawings, in which: the components are not necessarily drawn to scale, and components of similar or identical function or features can have the same or similar reference label.

[0040] Fig. 1 shows a structural schematic diagram of a joint pre-compensation system of distortion and chromatic aberration according to some embodiments of the present application.

[0041] Fig. 2 shows a structural schematic diagram of a joint pre-compensation system of distortion and chromatic aberration according to some embodiments of the present application.

[0042] Fig. 3 shows a flow schematic diagram of an offline calibration stage according to some embodiments of the present application.

[0043] Fig. 4 shows a horizontal and vertical striped image of blue, green and red colors according to some embodiments of the present application.

[0044] Fig. 5 shows a second distortion compensation table according to some embodiments of the present application.

[0045] Fig. 6 shows a fifth image of blue, green and red colors according to some embodiments of the present application.

[0046] Fig. 7A shows a third distortion compensation table of blue color according to some embodiments of the present application.

[0047] Fig. 7B shows a third distortion compensation table of green color according to some embodiments of the present application.

[0048] Fig. 7C shows a third distortion compensation table of red color according to some embodiments of the present application.

[0049] Fig. 8 shows a sixth image of blue, green and red colors according to some embodiments of the present application.

[0050] Fig. 9 shows a fourth distortion compensation table according to some embodiments of the present application.

[0051] FIG. 10 shows a second pixel color offset table according to some embodiments of the present application.

[0052] FIG. 11 shows a flowchart of an offline compensation stage according to some embodiments of the present application.

[0053] FIG. 12 shows a gaze point information diagram of a first image according to some embodiments of the present application.

[0054] FIG. 13 shows a gaze point information diagram of a compressed first image according to some embodiments of the present application.

[0055] FIG. 14 shows a first image according to some embodiments of the present application.

[0056] FIG. 15 shows a seventh image according to some embodiments of the present application.

[0057] FIG. 16 shows a second image according to some embodiments of the present application.

[0058] FIG. 17 shows a pixel diagram of a seventh image according to some embodiments of the present application.

[0059] FIG. 18 shows an eighth image according to some embodiments of the present application.

[0060] FIG. 19 shows a principle diagram of a statistical model estimating compensation data according to some embodiments of the present application.

[0061] FIG. 20 shows a principle diagram of a statistical model estimating compensation data according to some embodiments of the present application. DETAILED DESCRIPTION

[0062] The advantages and effects of the present application can be easily understood by those skilled in the art from the contents disclosed in the present specification. Although the present application will be described in conjunction with the preferred embodiments, the features of the present application are not limited to only the embodiments. On the contrary, the purpose of describing the present application in conjunction with the embodiments is to cover other alternatives or modifications which can be extended based on the claims of the present application. In order to provide a thorough understanding of the present application, many specific details will be described in the following description. The present application can also be implemented without using these details. In addition, in order to avoid confusion or ambiguity of the present application, some specific details will be omitted in the description.

[0063] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connection" should be understood in a broad sense, for example, can be fixed connection, can also be detachable connection, or integral connection; can be mechanical connection, can also be electrical connection; can be direct connection, can also be indirect connection through an intermediate medium, can be internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0064] In addition, "upper", "lower", "left", "right", "top", "bottom", "horizontal", "vertical" used in the following description should be understood as the orientation shown in the section and the related drawings. The relative terms are only for the convenience of description, and do not mean that the device described should be manufactured or operated in a particular orientation, so it should not be understood as a limitation on the present application.

[0065] It can be understood that although the terms "first", "second", "third" and the like can be used herein to describe various components, regions, layers and / or parts, these components, regions, layers and / or parts should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers and / or parts. Therefore, the first component, region, layer and / or part discussed below can be called the second component, region, layer and / or part without departing from some embodiments of the present application.

[0066] As described above, in the existing distortion and chromatic aberration compensation technology, the scheme of offsetting chromatic aberration by multiple optical elements is mainly used. However, this method relies on complex optical design, introduces additional optical elements, increases the thickness and weight of the head-mounted device, and it is difficult to consider factors such as user observation angle and gaze point at the same time. In addition, another method obtains the red-green-blue distortion grid according to the formula obtained according to the measured chromatic aberration in the rendering pipeline, and renders the red-green-blue pixels in turn according to the distortion grid, and then outputs them to the display end. However, this method is completed when rendering, which will occupy GPU computing power, and there are problems such as generating more power consumption and reducing rendering effect. In addition, these existing technologies generally do not consider how to consider factors such as user observation angle and gaze point.

[0067] In order to overcome the above-mentioned defects in the prior art, the present application provides a combined pre-compensation method for distortion and chromatic aberration, a combined pre-compensation system for distortion and chromatic aberration, an MR head-mounted display device, a computer readable storage medium, and a computer program product, which can perform chromatic aberration compensation on the basis of a distortion compensation table by using a color offset table, reduce the cache requirement of a processor, and maintain the simplicity of optical system design. In this way, the present application can effectively reduce the complexity and cost of optical design, and reduce or even eliminate the chromatic aberration caused by the gaze angle, so as to realize high-quality visual output at a lower cost and achieve optimal visual effect at different observation angles.

[0068] In some non-limiting embodiments, the combined pre-compensation method for distortion and chromatic aberration provided by the first aspect of the present application can be implemented via the combined pre-compensation system for distortion and chromatic aberration provided by the second aspect of the present application. Specifically, please refer to FIG. 1, which shows a structural schematic diagram of the combined pre-compensation system for distortion and chromatic aberration according to some embodiments of the present application.

[0069] In the embodiment shown in FIG. 1, the combined pre-compensation system for distortion and chromatic aberration provided by the second aspect of the present application can be configured in the MR head-mounted display device provided by the third aspect of the present application, which includes a first memory (not shown) and a processor 11. Here, the first memory includes but is not limited to the computer readable storage medium provided by the fourth aspect of the present application, on which the computer program product provided by the fifth aspect of the present application is stored. The computer program product includes computer instructions. The processor 11 is connected to the first memory and is configured to execute the computer instructions in the computer program product stored on the first memory to implement the combined pre-compensation method for distortion and chromatic aberration provided by the first aspect of the present application.

[0070] Further, in the embodiment shown in FIG. 1, the joint pre-compensation system of distortion and chromatic aberration provided by the second aspect of the present application further comprises a second memory 12 and a display module 13. Here, the second memory 12 can be embodied as an on-chip flash memory, including but not limited to the above-mentioned computer readable storage medium provided by the third aspect of the present application, which stores thereon a first distortion compensation table for the first color, and a first pixel color offset table for at least one second color compared with the first color. The display module 13 can be configured with a display driving chip and a silicon-based OLED display screen. The processor 11 connects the second memory 12 via a display driving software or firmware computing platform 14 to obtain the compensation data such as the first distortion compensation table and the first pixel color offset table provided by the second memory 12, and then processes the first image via the first distortion compensation table and each first pixel color offset table respectively to obtain a second image that has completed joint pre-compensation, and transmits the second image that has completed joint pre-compensation to the display driving chip at the back end for outputting a corresponding display image via the silicon-based OLED display screen.

[0071] Further, in some embodiments, the processor 11 is configured with a display pipeline. The processor 11 can execute the steps of the pre-compensation of distortion and chromatic aberration via the display pipeline. Here, the display pipeline is configured to obtain the first image to be compensated and the gaze point information of the user, and then determine the first distortion compensation table for the first color and the first pixel color offset table for at least one second color compared with the first color according to the gaze point information. Subsequently, the display pipeline processes the first image according to the first distortion compensation table and each first pixel color offset table respectively to obtain a second image that has completed joint pre-compensation.

[0072] Further, in some embodiments, the display pipeline is configured with a de-warping circuit, a group compression circuit, a group interpolation circuit, a chromatic aberration compensation circuit, and / or a decompression circuit. Here, the de-warping circuit is configured to map the first image according to the first warping compensation table to determine a seventh image that is warping compensated. The group compression circuit is configured to compress the first warping compensation table and / or the first pixel color offset table according to a compression format grouping table corresponding to the gaze point information. The group interpolation circuit is configured to interpolate the first warping compensation table and / or the first pixel color offset table to an image resolution of the first image according to the compression format grouping table corresponding to the gaze point information. The chromatic aberration compensation circuit is configured to calculate a pixel compensation amount for each pixel in the seventh image with respect to a corresponding second color according to a value of each cell in each first pixel color offset table, and to perform chromatic aberration compensation on each pixel in the seventh image with respect to the pixel compensation amount for each second color to obtain a second image that is complete joint pre-compensated. The decompression circuit is configured to decompress the second image to a screen resolution of the display module. In this way, the processor 11 can perform each step of de-warping and compensation and chromatic aberration pre-compensation via the display pipeline based on the de-warping circuit, the group compression circuit, the group interpolation circuit, the chromatic aberration compensation circuit, and / or the decompression circuit, etc. hardening circuit, to save the computing power of the central processing unit (CPU) and / or the graphics processing unit (GPU), and to reduce the link delay of data transmission.

[0073] Further, please refer to FIG. 2. FIG. 2 shows a structural schematic diagram of a joint pre-compensation system of distortion and chromatic aberration according to some embodiments of the present application.

[0074] In the embodiment shown in FIG. 2, the processor 11 is further configured with a distortion pre-compensation pipeline 111, a chromatic aberration compensation pipeline 112, a line-of-sight calculation pipeline 113, and / or a line-of-sight interpolation pipeline 114. Here, the distortion pre-compensation pipeline 111 is configured to construct a second warping compensation table with respect to a first color and corresponding to a plurality of preset gaze points. The chromatic aberration compensation pipeline 112 is configured to construct at least one second pixel color offset table with respect to each preset gaze point and corresponding to a second color compared to the first color. The line-of-sight calculation pipeline 113 is configured to obtain gaze point information of a user. The line-of-sight interpolation pipeline 114 is configured to perform compression and / or decompression of each second warping compensation table and / or each second pixel color offset table based on the gaze point information. In this way, the processor 11 can also perform each step of de-warping and compensation and chromatic aberration pre-compensation via the hardening units based on the distortion pre-compensation pipeline 111, the chromatic aberration compensation pipeline 112, the line-of-sight calculation pipeline 113, and / or the line-of-sight interpolation pipeline 114, respectively, to save the computing power of the central processing unit (CPU) and / or the graphics processing unit (GPU), and to reduce the link delay of data transmission.

[0075] In addition, in the embodiment shown in FIG. 2, the joint pre-compensation system of distortion and chromatic aberration provided by the second aspect of the present application can further comprise a standard camera 15. The standard camera 15 is configured to capture images distorted by the display module 13 at the positions of the preset gaze points, and transmit the images to the processor 11 for calculating compensation data such as the second distortion compensation table and / or the second pixel color offset table based on the distorted images, and storing the calculated compensation data in the second memory 12.

[0076] The working principle of the joint pre-compensation system of distortion and chromatic aberration will be described below in combination with some embodiments of the joint pre-compensation method of distortion and chromatic aberration. Those skilled in the art can understand that the embodiments of the joint pre-compensation method of distortion and chromatic aberration are only some non-limiting embodiments provided by the present application, which are intended to clearly demonstrate the main concept of the present application and provide some specific schemes for facilitating the public to implement, rather than for limiting the overall function or overall working mode of the joint pre-compensation system of distortion and chromatic aberration. Similarly, the joint pre-compensation system of distortion and chromatic aberration is also only a non-limiting embodiment provided by the present application, which does not limit the execution subject or execution order of each step in the joint pre-compensation method of distortion and chromatic aberration.

[0077] In some non-limiting embodiments, the joint pre-compensation method of distortion and chromatic aberration provided by the present application can be independently implemented in two stages of offline calibration and online compensation. Those skilled in the art can first calibrate the second distortion compensation table and the second pixel color offset table in the offline calibration stage, and store them in the second memory 12 of the joint pre-compensation system. Then, the joint pre-compensation system can determine the first distortion compensation table for the first color and the first pixel color offset table for the first color compared with the second color based on at least one pre-calibrated second distortion compensation table and at least one pre-calibrated second pixel color offset table in the online detection stage, so as to perform chromatic aberration compensation based on the first pixel color offset table on the basis of the first distortion compensation table.

[0078] In this way, the present application can perform chromatic aberration compensation based on the distortion compensation table by using the pixel color offset table, so as to reduce the cache requirement of the processor, maintain the simplicity of the optical system design, effectively reduce the complexity and cost of the optical design, and reduce or even eliminate the chromatic aberration caused by the gaze angle, so as to realize high-quality visual output in the MR head-mounted display at a lower cost, and achieve optimal visual effect under different observation angles.

[0079] Further, please refer to FIG. 3. FIG. 3 shows a flowchart of the offline calibration stage according to some embodiments of the present application.

[0080] In the embodiment shown in FIG. 3, in the process of offline calibration of the second distortion compensation table, the present application can first input the horizontal and vertical stripe image of the first color (for example, green) into the display module 13, and use the standard camera 15 to capture the second image distorted by the display module 13 at the position of at least one preset gaze point. Then, the present application can perform edge detection on the second image to determine a plurality of horizontal stripe edge lines and a plurality of vertical stripe edge lines therein.

[0081] Further, in some embodiments, before performing edge detection on the second image, the present application can first preferably input the pure color image of the first color into the display module 13, and use the standard camera 15 to capture the third image distorted by the display module to determine the relative gray value of the third image with respect to the first color. Here, the pure color image of the first color is an image of the same gray as the horizontal and vertical stripe image of the first color. Then, the present application can obtain the camera distortion grid of the standard camera 15, and process the obtained second image according to the camera distortion grid to obtain the second image without camera distortion, and then perform first normalization processing on the gray of the second image without camera distortion according to the above relative gray value, and then perform binaryzation processing on the second image after the first normalization processing to obtain the second image after the binaryzation processing, so as to improve the accuracy of edge detection.

[0082] Then, the present application can sort each horizontal stripe edge line and each vertical stripe edge line in the second image after the edge detection, and determine the first horizontal coordinate table_h and the first vertical coordinate table_v of a plurality of ordered intersection points according to the serial numbers of each horizontal stripe edge line and each vertical stripe edge line. Then, the present application can construct the second distortion compensation table corresponding to the preset gaze point according to the first horizontal coordinate and the first vertical coordinate of each ordered intersection point.

[0083] In addition, in some embodiments, after constructing the second distortion compensation table, the present application can preferably use the least square method to determine the first fitting function of the display module distortion according to the first horizontal coordinate table_h and the first vertical coordinate table_v of each ordered intersection point in the second distortion compensation table, and the second horizontal coordinate and the second vertical coordinate of the corresponding intersection point in the horizontal and vertical stripe image. Then, the present application can substitute the second horizontal coordinate and the second vertical coordinate of at least one unknown intersection point in the horizontal and vertical stripe image into the first fitting function to determine the first horizontal coordinate and the first vertical coordinate of the at least one unknown intersection point in the second image. Then, the present application can complete the second distortion compensation table according to the first horizontal coordinate and the first vertical coordinate of the at least one unknown intersection point in the second image to ensure the smoothness and regularity of the second distortion compensation table, so as to improve the subsequent color difference compensation accuracy, and make the second distortion compensation table adaptable to various types of optical machines.

[0084] Further, after the second distortion compensation table is completed, the present application can preferably perform a second normalization process on the second distortion compensation table, so that the resolution and angle of the second distortion compensation table are consistent with the resolution and angle of the image to be compensated. Specifically, in the process of normalization, the present application can first obtain the target resolution of the second distortion compensation table, and determine the scaling ratio and / or rotation ratio of the second normalization process in combination with the mapping image coordinate values of the plurality of units in the second distortion compensation table. Then, the present application can perform a second normalization process of displacement, rotation, scaling and the like on the second distortion compensation table according to the scaling ratio and / or rotation ratio, so as to scale the horizontal and vertical stripe coordinates in the middle of the input horizontal and vertical stripe image to 0-1, and scale the coordinates of other horizontal and vertical stripes at the same ratio, so as to finally generate a second distortion compensation table consistent with the resolution and angle of the image to be compensated.

[0085] In addition, after obtaining the second distortion compensation table, the present application can preferably obtain the screen resolution of the display module 13, and perform linear interpolation on the second distortion compensation table according to the screen resolution, so as to obtain a second distortion compensation table consistent with the screen resolution. Here, the linear interpolation includes but is not limited to various interpolation methods such as bilinear interpolation, bicubic interpolation, spline interpolation, etc.

[0086] In addition, please continue to refer to FIG. 3, in the process of offline calibration of the second pixel color offset table, the present application can input horizontal and vertical stripe images of a plurality of second colors (for example: red, blue) into the display module 13 respectively, and use the standard camera 15 to collect fourth images distorted by the display module 13 at the position of the preset gaze point, so as to construct third distortion compensation tables of each second color respectively.

[0087] Specifically, in the process of constructing the third distortion compensation table of each second color, the present application can perform edge detection on the fourth image of each second color respectively, so as to determine a plurality of horizontal stripe edge lines and a plurality of vertical stripe edge lines. Then, the present application can sort each horizontal stripe edge line and each vertical stripe edge line respectively, so as to determine the third horizontal coordinates and the third vertical coordinates of a plurality of ordered intersection points of each horizontal stripe edge line and each vertical stripe edge line, and construct the third distortion compensation table of each second color according to the third horizontal coordinates and the third vertical coordinates of each ordered intersection point.

[0088] Then, the present application can obtain fifth images simulating the optical path distortion of the display module 13 to the first color and each second color, and use the above-mentioned second distortion compensation table and each third distortion compensation table respectively to map the fifth images of the corresponding colors, so as to generate sixth images of each color after distortion compensation.

[0089] In some embodiments, the present application can first determine the distortion simulation table of the light path distortion of the first color by the simulation display module according to the second distortion compensation table. Specifically, the present application can first determine the horizontal minimum value h_min, the horizontal maximum value h_max, the vertical minimum value v_min and the vertical maximum value v_max in the second distortion compensation table according to the first horizontal coordinate table_h and the first vertical coordinate table_v in the second distortion compensation table. Then, the present application can determine the maximum width width and the maximum height height of the second image according to the center point coordinates (c_h, c_y) of the horizontal and vertical stripe image of the first color and the horizontal and vertical maximum and minimum values of the above-mentioned second distortion compensation table: width = f (h_max, c_h, h_min), height = f (v_max, c_v, v_min).

[0090] Then, the present application can traverse each cell in the second distortion compensation table within the range of the maximum width width and the maximum height height. For each cell (ind_v, ind_h), find its three adjacent cells (ind_v, ind_h+1), (ind_v+1, ind_h) and (ind_v+1, ind_h+1), and obtain the coordinate values (x0, y0), (x1, y1), (x2, y2), (x3, y3) stored in the positions of the four cells in the second distortion compensation table, then determine two triangles according to the horizontal and vertical coordinates of the four cells, and perform barycentric interpolation on the mapping image coordinate values of the multiple points x_index, y_index inside each triangle according to the mapping image coordinate values of the endpoints of each triangle. Then, the present application can construct the distortion simulation table table_barrel_x, table_barrel_y according to each cell in the second distortion compensation table and the mapping image coordinate values of each point obtained by interpolation, and clip it to the same resolution as the horizontal and vertical stripe image of the first color with the center of the distortion simulation table as the origin.

[0091] Thereafter, reference is made to Figs. 4-10. Fig. 4 shows a horizontal and vertical stripe image of blue, green and red colors provided according to some embodiments of the present application. Fig. 5 shows a second distortion compensation table provided according to some embodiments of the present application. Fig. 6 shows a fifth image of blue, green and red colors provided according to some embodiments of the present application. Fig. 7A shows a third distortion compensation table of blue color provided according to some embodiments of the present application. Fig. 7B shows a third distortion compensation table of green color provided according to some embodiments of the present application. Fig. 7C shows a third distortion compensation table of red color provided according to some embodiments of the present application. Fig. 8 shows a sixth image of blue, green and red colors provided according to some embodiments of the present application. Fig. 9 shows a fourth distortion compensation table provided according to some embodiments of the present application. Fig. 10 shows a second pixel color offset table provided according to some embodiments of the present application.

[0092] As shown in Figs. 4-6, after determining the distortion simulation table, the present application can map the horizontal and vertical stripe image of the first color via the second distortion compensation table described above to generate a fifth image simulating the optical path distortion of the first color by the display module, and map the horizontal and vertical stripe images of each second color via the second distortion compensation table described above to generate a fifth image simulating the optical path distortion of each second color by the display module, respectively.

[0093] Thereafter, as shown in Figs. 6-8, the present application can use the second distortion compensation table and each third distortion compensation table to map the fifth image of the corresponding color to generate a sixth image of each color after distortion compensation, and perform edge detection on each sixth image as shown in Fig. 9 to construct a fourth distortion compensation table of the corresponding color according to the fourth horizontal and vertical coordinates of the multiple ordered intersection points of the multiple horizontal stripe edge lines and the multiple vertical stripe edge lines.

[0094] Thereafter, as shown in Fig. 10, the present application can respectively subtract the fourth distortion compensation table of each second color from the fourth distortion compensation table of the first color to determine a second pixel color offset table of each second color compared to the corresponding preset gaze point of the first color, so as to perform color offset of the corresponding pixel while ensuring the accuracy of the color processed by the algorithm.

[0095] Further, in some embodiments, the present application can preferably determine multiple preset gaze points and repeat the above steps to pre-calibrate the corresponding second distortion compensation table and second pixel color offset table according to the position of each preset gaze point. In this way, the joint pre-compensation system can obtain the first distortion compensation table and the first pixel color offset table of the corresponding actual gaze point coordinates by interpolation according to the second distortion compensation table and the second pixel color offset table of each preset gaze point in the offline compensation stage, so as to further improve the compensation accuracy of distortion and color difference.

[0096] Further, after calibrating the second distortion compensation table corresponding to each preset gaze point, the application can preferably fit a second fitting function representing each second distortion compensation table, and then extract and store the function relationship parameters thereof. Specifically, the application can determine a second fitting function of the mapping image coordinate value with respect to the horizontal coordinate and the vertical coordinate of the corresponding unit according to the horizontal coordinate, the vertical coordinate and the mapping image coordinate value of each unit in each second distortion compensation table, and then extract and store the polynomial coefficients and other function relationship parameters of the second fitting function representing each second distortion compensation table, so as to reduce the data storage amount of the second storage 12.

[0097] Then, please refer to FIG. 11. FIG. 11 shows a flowchart of the offline compensation stage provided according to some embodiments of the application.

[0098] As shown in FIG. 11, in the offline compensation stage of distortion and chromatic aberration, the joint pre-compensation system can first acquire a first image to be compensated and the gaze point information of the user.

[0099] Then, the joint pre-compensation system can determine a first distortion compensation table for the first color (for example, green) according to the gaze point information and at least one pre-calibrated second distortion compensation table.

[0100] Specifically, in the process of determining the first distortion compensation table, the joint pre-compensation system can interpolate the first distortion compensation table corresponding to the actual gaze point coordinates according to the second distortion compensation tables of multiple preset gaze points, so as to improve the compensation accuracy. Specifically, the joint pre-compensation system can first determine multiple (for example, 8) preset gaze points adjacent to the actual gaze point corresponding to the gaze point information in space, and then determine the second distortion compensation table for each preset gaze point with respect to the first color. Then, the joint pre-compensation system can determine the weight of each preset gaze point according to the distance from the actual gaze point to each preset gaze point, and then interpolate each second distortion compensation table according to the weight, so as to determine the first distortion compensation table.

[0101] Further, in some embodiments, the joint pre-compensation system can also determine the function relationship parameters corresponding to the gaze point information of the user, and then determine the second fitting function of the mapping image coordinate value with respect to the horizontal coordinate and the vertical coordinate of the corresponding unit according to the function relationship parameters. Then, the joint pre-compensation system can determine the mapping image coordinate value corresponding to the horizontal coordinate and the vertical coordinate of each unit in the first distortion compensation table according to the second fitting function, so as to decompress the second distortion compensation table compressed in the offline calibration stage.

[0102] Afterwards, the joint pre-compensation system can determine the first pixel color offset table of at least one second color (e.g. red, blue) compared with the first color according to the gaze point information of the user and at least one pre-labeled second pixel color offset table. Specifically, the joint pre-compensation system can determine at least one second pixel color offset table of each preset gaze point respectively, and then interpolate each second pixel color offset table according to the weight to determine the first pixel color offset table.

[0103] Please refer to FIG. 12 and FIG. 13. FIG. 12 shows a schematic diagram of the gaze point information of the first image according to some embodiments of the present application. FIG. 13 shows a schematic diagram of the compressed gaze point information of the first image according to some embodiments of the present application.

[0104] In the embodiments shown in FIG. 12 and FIG. 13, the joint pre-compensation system can first determine the compression format grouping table according to the gaze point information. In this case, the gaze point area 21 of the actual gaze point in the compression format grouping table has the smallest first compression ratio, the first non-gaze point area 22 adjacent to the gaze point area has a larger second compression ratio, and the second non-gaze point area 23 far away from the gaze point area has the largest third compression ratio. Afterwards, the joint pre-compensation system can interpolate the first distortion compensation table based on the gaze point coordinates according to the compression format grouping table, and interpolate the first distortion compensation table to the image resolution of the first image to reduce the early data storage amount in the online calibration stage.

[0105] Afterwards, please refer to FIG. 14-FIG. 16. FIG. 14 shows the first image according to some embodiments of the present application. FIG. 15 shows the seventh image according to some embodiments of the present application. FIG. 16 shows the second image according to some embodiments of the present application.

[0106] After determining the first distortion compensation table and the first pixel color offset table, the joint pre-compensation system can process the first image according to the first distortion compensation table and each first pixel color offset table respectively to obtain the second image after joint pre-compensation.

[0107] Specifically, as shown in FIG. 14 and FIG. 15, the joint pre-compensation system can first map the first image using the first distortion compensation table to determine the seventh image after distortion compensation, and then determine the coordinate position of each pixel point in the seventh image in each first pixel color offset table according to the grouping compression rule of the compression format grouping table. Afterwards, the joint pre-compensation system can interpolate each first pixel color offset table to the image resolution of the first image respectively to determine the pixel compensation amount of each pixel point in each first pixel color offset table corresponding to the coordinate position respectively according to the compression format grouping table.

[0108] Further, in the process of grouping compression of the first image, in response to the existence of compression region change of the coordinate position of any pixel point in the seventh image in the first pixel color offset table of any second color, the joint pre-compensation system can calculate the pixel compensation components of the pixel point in multiple compression regions before and after the change respectively, and sum the pixel compensation components of the pixel point in multiple compression regions before and after the change to determine the pixel compensation amount of the pixel point to cope with the situation that grouping compression may cause compression region change.

[0109] After that, as shown in FIG. 16, the joint pre-compensation system can perform chromatic aberration compensation on each pixel point in the seventh image according to the pixel compensation amount of each second color to obtain the second image after joint pre-compensation.

[0110] Specifically, in the process of chromatic aberration compensation of the seventh image, the joint pre-compensation system can first determine whether the coordinate position of each pixel point in the seventh image after chromatic aberration compensation exceeds the boundary of the second image. In response to the determination result that the coordinate position of any pixel point after chromatic aberration compensation exceeds the boundary of the second image, the joint pre-compensation system can fill the gray value of the pixel point according to the gray value of the corresponding boundary pixel point in the second image to cope with the situation that the coordinate position after chromatic aberration compensation exceeds the boundary of the image.

[0111] Further, in some embodiments, the joint pre-compensation system can also use a filter with a size of M*N to perform weighted summation on the gray values of the M*N pixel points adjacent to the target pixel point in the seventh image after chromatic aberration compensation to determine the second image after joint pre-compensation, thereby avoiding the error of joint pre-compensation and further improving the chromatic aberration compensation accuracy.

[0112] For details, please refer to FIG. 17, which shows a pixel point diagram of the seventh image according to some embodiments of the present application.

[0113] In the embodiment shown in FIG. 17, the first color is green, and the second color includes red and blue. The joint pre-compensation system can determine the red component position 31 and the blue component position 32 of each pixel point in the seventh image after chromatic aberration compensation to determine whether the filtering region of the red component position 31 and / or the blue component position 32 exceeds the pre-divided line buffer region. After that, in response to the determination result that the filtering region of the red component position and / or the blue component position exceeds the line buffer region, the joint pre-compensation system can perform linear translation on the red component position and / or the blue component position whose filtering region exceeds the line buffer region to move its filtering region back to the line buffer region. In this way, the joint pre-compensation system can perform distortion correction on the distortion effect of the green image by the display module, and perform chromatic aberration pre-compensation based on the difference in distortion effect of the red / blue image compared with the green image by the display module to reduce the required size of the line buffer region.​

[0114] Those skilled in the art can understand that the above embodiment taking green as the first color and red and blue as the second color is only a non-limiting implementation provided by the present application, which is intended to provide a preferred solution of reducing the color offset of each second color to the first color to reduce the line buffer area requirement, rather than for limiting the protection scope of the present application.

[0115] Alternatively, in other embodiments, those skilled in the art can also select red or blue as the first color and the remaining two colors as the second color based on the above technical concept, and configure a line buffer area with a larger (close to twice) cache capacity to achieve the basic effect of joint pre-compensation of distortion and chromatic aberration.

[0116] Then, please continue to refer to FIG. 18. FIG. 18 shows an eighth image provided according to some embodiments of the present application.

[0117] As shown in FIG. 18, after obtaining the second image with completed joint pre-compensation, the joint pre-compensation system can also decompress the second image according to the screen resolution of the display module to obtain an eighth image to be displayed, and input the eighth image into the display module to display the image with distortion and chromatic aberration removed on the screen. In this way, the joint pre-compensation system can improve the resolution of the compensated image to the screen resolution to improve the subsequent display efficiency.

[0118] In addition, please refer to FIG. 19 and FIG. 20 in combination. FIG. 19 shows a principle diagram of statistical model pre-estimating compensation data provided according to some embodiments of the present application. FIG. 20 shows a principle diagram of statistical model pre-estimating compensation data provided according to some embodiments of the present application.

[0119] As shown in FIG. 19, in the process of compensating the real object 41 for distortion and chromatic aberration, the joint pre-compensation system can also extract part of the optical-mechanical sample to measure the compensation data of the points in the eyebox 43 to establish a statistical model. Here, the eyebox 43 is the space where the pupil of the human eye is located when the user wears the head-mounted device and experiences better visual effect. Different positions of the pupil in the area require different compensation parameters. The form of the compensation data is stored in the form of a parameter LUT table. Then, for other sample measurement of less eyebox point compensation data, the joint pre-compensation system can use the statistical model to pre-estimate the compensation parameters of other points.

[0120] For example, as shown in FIG. 20, for a certain optical-mechanical device, the data of the gaze point 51 (a certain point in the eyebox) is the actually measured compensation data, and the compensation data of the remaining points 52 is pre-estimated by the statistical model. In this way, the processor 11 can use the statistical model to pre-estimate the compensation data of the human eye at different positions in the eyebox with less measured data to obtain the compensated virtual image 42.

[0121] In summary, compared with the prior art of respectively calibrating the distortion compensation table of red, green and blue colors, respectively calibrating the color difference compensation table of each color, respectively performing distortion compensation and color difference compensation of each color, the present application can reduce the original n frames of image n*3 distortion compensation tables to n distortion compensation tables and n*2 color difference compensation tables (the memory occupied by the color difference compensation table is much smaller than the distortion compensation table), thereby greatly reducing the data storage load of the second memory 12, and reducing the original compensation processing of fusing each frame of image with three distortion compensation tables three times to only needing to fuse each frame of image with one distortion compensation table of the first color once, thereby greatly reducing the data processing load of the processor 11. Therefore, the above-mentioned joint pre-compensation method of distortion and color difference, the joint pre-compensation system of distortion and color difference, the MR head-mounted display device, the computer readable storage medium and the computer program product provided by the present application can effectively reduce the complexity and cost of optical design, and reduce or even eliminate the color difference caused by the gaze angle, thereby realizing high-quality visual output at a lower cost and achieving optimal visual effect under different observation angles.

[0122] Although the above-described methods are illustrated and described as a series of acts, it will be appreciated and understood by those skilled in the art that the methods are not limited by the order of acts, as some acts can, in accordance with one or more embodiments, occur simultaneously or in different orders than those illustrated and described herein or those described in the claims.

[0123] Those skilled in the art will appreciate that information, signals, and data can be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that can be referenced throughout the above description can be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0124] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0125] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of joint pre-compensation of distortion and chromatic aberration, characterized in that, The method comprises the following steps: obtaining a first image to be compensated and gaze point information of a user; determining a first distortion compensation table for a first color and a first pixel color offset table of at least one second color compared with the first color according to the gaze point information; and processing the first image according to the first distortion compensation table and each first pixel color offset table to obtain a second image after the joint pre-compensation. The first distortion compensation table is determined according to at least one pre-calibrated second distortion compensation table, wherein the step of calibrating the second distortion compensation table comprises:

2. The method of joint pre-compensation of claim 1, wherein, inputting a horizontal and vertical stripe image of the first color into a display module and using a standard camera to capture a second image distorted by the display module at the position of at least one preset gaze point; performing edge detection on the second image to determine a plurality of horizontal stripe edge lines and a plurality of vertical stripe edge lines in the second image; sorting each horizontal stripe edge line and each vertical stripe edge line in the second image, and determining first horizontal coordinates and first vertical coordinates of a plurality of ordered intersection points according to the serial numbers of each horizontal stripe edge line and each vertical stripe edge line; and constructing a second distortion compensation table corresponding to the preset gaze point according to the first horizontal coordinates and the first vertical coordinates of each ordered intersection point. Before performing edge detection on the second image, the joint pre-compensation method further comprises the following steps:

3. The method of joint pre-compensation of claim 2, wherein, inputting a pure color image of the first color into the display module and using the standard camera to capture a third image distorted by the display module to determine the relative gray value of the third image with respect to the first color; obtaining a camera distortion grid of the standard camera, and processing the second image according to the camera distortion grid to obtain a second image without camera distortion; performing first normalization processing on the gray scale of the second image without camera distortion according to the relative gray value; and performing binary processing on the second image after the first normalization processing to obtain a second image after the binary processing. After constructing the second distortion compensation table corresponding to the preset gaze point according to the first horizontal coordinates and the first vertical coordinates of each ordered intersection point, the joint pre-compensation method further comprises the following steps:

4. The method of joint pre-compensation of claim 3, wherein, determining a first fitting function of the display module distortion according to the first horizontal coordinates and the first vertical coordinates of each ordered intersection point in the second distortion compensation table and the second horizontal coordinates and the second vertical coordinates of the corresponding intersection points in the horizontal and vertical stripe image; substituting the second horizontal coordinates and the second vertical coordinates of at least one unknown intersection point in the second distortion compensation table into the first fitting function to determine the first horizontal coordinates and the first vertical coordinates of the at least one unknown intersection point in the second image; and completing the second distortion compensation table according to the first horizontal coordinates and the first vertical coordinates of the at least one unknown intersection point in the second image. After completing the second distortion compensation table, the joint pre-compensation method further comprises the following steps:

5. The method of joint pre-compensation of claim 4, wherein, ​ acquire a target resolution of the second distortion compensation table, and combine mapping image coordinate values of a plurality of cells in the second distortion compensation table to determine a scaling ratio and / or a rotation ratio of a second normalization processing; and perform the second normalization processing on the second distortion compensation table according to the scaling ratio and / or the rotation ratio.

6. The method of joint pre-compensation of claim 3, wherein, After constructing the second distortion compensation table corresponding to the preset gaze point according to the first horizontal coordinates and the first vertical coordinates of the ordered intersection points, the joint pre-compensation method further comprises the following steps: acquire a screen resolution of the display module; and perform linear interpolation on the second distortion compensation table according to the screen resolution to obtain a second distortion compensation table conforming to the screen resolution.

7. The method of joint pre-compensation of claim 2, wherein, The first pixel color offset table is determined according to at least one pre-calibrated second pixel color offset table, wherein the step of calibrating the second pixel color offset table comprises: input the horizontal and vertical stripe images of each second color into the display module respectively, and use a standard camera to capture fourth images after distortion of the display module at the position of the preset gaze point to construct third distortion compensation tables of each second color respectively; acquire fifth images simulating optical path distortion of the first color and each second color by the display module, and use the second distortion compensation table and each third distortion compensation table respectively to map the fifth images of the corresponding colors to generate sixth images of each color after distortion compensation; perform edge detection on each sixth image respectively to construct fourth distortion compensation tables of the corresponding colors according to fourth horizontal coordinates and fourth vertical coordinates of a plurality of ordered intersection points of a plurality of horizontal stripe edge lines and a plurality of vertical stripe edge lines therein; and subtract each fourth distortion compensation table of each second color from the fourth distortion compensation table of the first color to determine second pixel color offset tables of each second color compared with the first color corresponding to the preset gaze point respectively.

8. The method of joint pre-compensation of claim 7, wherein, The step of constructing the third distortion compensation table of each second color respectively comprises: perform edge detection on each fourth image of each second color respectively to determine a plurality of horizontal stripe edge lines and a plurality of vertical stripe edge lines therein; sort each horizontal stripe edge line and each vertical stripe edge line respectively to determine third horizontal coordinates and third vertical coordinates of a plurality of ordered intersection points of each horizontal stripe edge line and each vertical stripe edge line; and construct the third distortion compensation table of each second color respectively according to the third horizontal coordinates and the third vertical coordinates of the ordered intersection points.

9. The method of joint pre-compensation of claim 7, wherein, The step of acquiring the fifth images simulating optical path distortion of the first color and each second color by the display module comprises: determine a distortion simulation table simulating optical path distortion of the first color by the display module according to the second distortion compensation table; map the horizontal and vertical stripe images of the first color via the second distortion compensation table to generate the fifth images simulating optical path distortion of the first color by the display module; and mapping each of the second color horizontal and vertical fringe images via the second distortion compensation table to generate a fifth image simulating optical path distortion of each of the second color by the display module.

10. The method of joint pre-compensation of claim 9, wherein, The step of determining the distortion simulation table simulating optical path distortion of the first color by the display module according to the second distortion compensation table comprises: determining a maximum width and a maximum height of the second image according to the center point coordinates of the first color horizontal and vertical fringe images and the second distortion compensation table; traversing each of the units in the second distortion compensation table within the maximum width and the maximum height, and determining two triangles in combination with the horizontal and vertical coordinates of three adjacent units thereof; performing barycentric interpolation on mapping image coordinate values of a plurality of points inside each of the triangles according to mapping image coordinate values of endpoints of each of the triangles; constructing the distortion simulation table according to each of the units in the second distortion compensation table and the mapping image coordinate values of each of the points obtained by interpolation; and taking the center of the distortion simulation table as an origin, and cutting it into a resolution equal to that of the first color horizontal and vertical fringe images.

11. The method of joint pre-compensation of claim 1, wherein, The first distortion compensation table is determined according to at least one pre-calibrated second distortion compensation table, and the first pixel color offset table is determined according to at least one pre-calibrated second pixel color offset table, wherein the step of calibrating the second distortion compensation table and the second pixel color offset table comprises: determining a plurality of preset gaze points; and calibrating corresponding second distortion compensation tables and second pixel color offset tables respectively according to positions of each of the preset gaze points.

12. The method of joint pre-compensation of claim 11, wherein, After calibrating the second distortion compensation table corresponding to each of the preset gaze points, the joint pre-compensation method further comprises the following steps: determining a second fitting function of the mapping image coordinate value with respect to the horizontal and vertical coordinates of the corresponding unit according to the horizontal and vertical coordinates and the mapping image coordinate value of each of the units in each of the second distortion compensation tables; and extracting and storing function relationship parameters representing the second fitting function of each of the second distortion compensation tables respectively.

13. The method of joint pre-compensation of claim 12, wherein, The step of determining the first distortion compensation table comprises: determining corresponding function relationship parameters according to the gaze point information; determining a second fitting function of the mapping image coordinate value with respect to the horizontal and vertical coordinates of the corresponding unit according to the function relationship parameters; and determining the mapping image coordinate value corresponding to the horizontal and vertical coordinates of a plurality of units in the first distortion compensation table according to the second fitting function.

14. The method of joint pre-compensation of claim 11, wherein, The step of determining the first distortion compensation table comprises: determining a plurality of preset gaze points adjacent to the actual gaze point corresponding thereto according to the gaze point information, and determining a second distortion compensation table of each of the preset gaze points with respect to the first color respectively; determining weights of each of the preset gaze points according to distances from the actual gaze point to each of the preset gaze points; and performing interpolation on each of the second distortion compensation tables according to the weights to determine the first distortion compensation table.

15. The method of joint pre-compensation of claim 14, wherein, The step of determining the first pixel color offset table comprises: determine at least one second pixel color offset table for each of the preset gaze points, respectively, wherein the at least one second pixel color offset table is determined based on a second color of each of the preset gaze points compared with the first color of the second pixel; and interpolate each of the second pixel color offset tables according to the weight to determine the first pixel color offset table.

16. The method of joint precompensation of claim 1, wherein, The step of processing the first image according to the first distortion compensation table and each of the first pixel color offset tables to obtain the second image with the joint pre-compensation comprises: mapping the first image using the first distortion compensation table to determine a seventh image with distortion compensation; calculating a pixel compensation amount of each pixel point in the seventh image with respect to a corresponding second color according to a value of each cell in each of the first pixel color offset tables, respectively; and performing chromatic aberration compensation on each of the pixel points in the seventh image according to the pixel compensation amount with respect to each of the second colors to obtain the second image with the joint pre-compensation.

17. The method of joint pre-compensation of claim 16, wherein, Before mapping the first image using the first distortion compensation table, the joint pre-compensation method further comprises the following steps: determining a corresponding compression format grouping table according to the gaze point information, wherein a gaze point area of an actual gaze point in the compression format grouping table has a smallest first compression ratio, a first non-gaze point area adjacent to the gaze point area has a larger second compression ratio, and a second non-gaze point area away from the gaze point area has a largest third compression ratio; and interpolating the first distortion compensation table to an image resolution of the first image according to the compression format grouping table.

18. The method of joint pre-compensation of claim 17, wherein, The step of calculating a pixel compensation amount of each pixel point in the seventh image with respect to a corresponding second color according to a value of each cell in each of the first pixel color offset tables, respectively, comprises: determining a coordinate position of each of the pixel points in the seventh image in each of the first pixel color offset tables according to a grouping compression rule of the compression format grouping table; and interpolating each of the first pixel color offset tables to the image resolution of the first image according to the compression format grouping table to determine a pixel compensation amount of a corresponding coordinate position of each of the pixel points in the seventh image in each of the first pixel color offset tables, respectively.

19. The method of joint pre-compensation of claim 18, wherein, The step of interpolating each of the first pixel color offset tables to the image resolution of the first image according to the compression format grouping table to determine a pixel compensation amount of a corresponding coordinate position of each of the pixel points in the seventh image in each of the first pixel color offset tables, respectively, comprises: in response to a change in a compression area of a coordinate position of any of the pixel points in the seventh image in any of the first pixel color offset tables of any of the second colors, calculating pixel compensation components of a plurality of compression areas before and after the change of the pixel point, respectively; and summing the pixel compensation components of the plurality of compression areas before and after the change of the pixel point to determine the pixel compensation amount of the pixel point.

20. The method of joint precompensation of claim 16, wherein, The step of performing chromatic aberration compensation on each of the pixel points in the seventh image according to the pixel compensation amount with respect to each of the second colors to obtain the second image with the joint pre-compensation, comprises: determining whether the coordinate positions of the pixels in the seventh image after the chromatic aberration compensation exceed the boundary of the second image; and in response to a determination that the coordinate position of any of the pixels after the chromatic aberration compensation exceeds the boundary of the second image, filling the gray value of the pixel according to the gray value of the corresponding boundary pixel in the second image.

21. The method of joint precompensation of claim 16, wherein, The step of compensating the pixels in the seventh image according to the pixel compensation amount of each of the second colors to obtain the second image after the joint pre-compensation comprises: performing weighted summation on the gray values of the M*N pixels adjacent to the target pixels in the seventh image after the chromatic aberration compensation using a filter of size M*N to determine the second image after the joint pre-compensation.

22. The method of joint precompensation of claim 21, wherein, The first color is green, and the at least one second color includes red and blue. Before performing the weighted summation on the gray values of the M*N pixels adjacent to the target pixels in the seventh image after the chromatic aberration compensation using a filter of size M*N, the joint pre-compensation method further comprises the following steps: determining the red component position and the blue component position of each of the pixels in the seventh image after the chromatic aberration compensation to determine whether the filter regions of the red component positions and / or the blue component positions exceed the line buffer region; and in response to a determination that the filter region of the red component position and / or the blue component position exceeds the line buffer region, performing linear translation on the red component position and / or the blue component position whose filter region exceeds the line buffer region to move the filter region back to the line buffer region.

23. The method of joint precompensation of claim 1, wherein, After obtaining the second image after the joint pre-compensation, the joint pre-compensation method further comprises the following steps: decompressing the second image according to the screen resolution of the display module to obtain an eighth image to be displayed; and inputting the eighth image to the display module to display a distortion-free and chromatic aberration-free image on the screen.

24. A combined distortion and chromatic aberration pre-compensation system, characterized by, comprises: a memory having computer instructions stored thereon; a processor connected to the memory and configured to execute the computer instructions stored on the memory to implement the joint pre-compensation method of distortion and chromatic aberration according to any one of claims 1-23.

25. The combined precompensation system of claim 24, wherein, The processor is configured with a display pipeline, wherein the display pipeline is configured to obtain a first image to be compensated and gaze point information of a user, determine a first distortion compensation table for a first color and at least one second color pixel color offset table relative to the first color according to the gaze point information, and process the first image according to the first distortion compensation table and each of the first pixel color offset tables to obtain the second image after the joint pre-compensation.

26. The combined precompensation system of claim 25, wherein, The display pipeline is configured with a distortion removal circuit, a packet compression circuit, a packet interpolation circuit, a chromatic aberration compensation circuit, and / or a decompression circuit, wherein, The distortion compensation circuit is configured to map the first image according to the first distortion compensation table to determine a seventh image subjected to distortion compensation, The group compression circuit is configured to compress the first distortion compensation table and / or the first pixel color offset table according to a compression format grouping table corresponding to the gaze point information, The group interpolation circuit is configured to interpolate the first distortion compensation table and / or the first pixel color offset table to an image resolution of the first image according to a compression format grouping table corresponding to the gaze point information, The chromatic aberration compensation circuit is configured to calculate a pixel compensation amount of each pixel point in the seventh image with respect to a corresponding second color according to a value of each cell in each of the first pixel color offset tables, and perform chromatic aberration compensation on each of the pixel points in the seventh image according to the pixel compensation amount with respect to each of the second colors to obtain a second image subjected to the joint pre-compensation, The decompression circuit is configured to decompress the second image to a screen resolution of the display module.

27. The combined precompensation system of claim 24, wherein, The processor is further configured with a distortion pre-compensation pipeline, a chromatic aberration compensation pipeline, a line-of-sight calculation pipeline, and / or a line-of-sight interpolation pipeline, wherein, The distortion pre-compensation pipeline is configured to construct a second distortion compensation table with respect to a first color and corresponding to a plurality of preset gaze points, The chromatic aberration compensation pipeline is configured to construct at least one second pixel color offset table with respect to each of the preset gaze points and with respect to a second color different from the first color, The line-of-sight calculation pipeline is configured to obtain gaze point information of the user, The line-of-sight interpolation pipeline is configured to perform compression and / or decompression on each of the second distortion compensation tables and / or each of the second pixel color offset tables based on the gaze point information.

28. The combined precompensation system of claim 27, wherein, Further comprising: On-chip flash memory for storing the second distortion compensation tables and / or the second pixel color offset tables compressed by the line-of-sight interpolation pipeline; And / or A display module for displaying images input thereto; and / or A standard camera for capturing images distorted by the display module at positions of the preset gaze points.

29. An MR head-mounted display device, comprising: The MR head-mounted display device is configured with the joint distortion and chromatic aberration pre-compensation system of any one of claims 24-28.

30. A computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions, when executed by the processor, implement the MR head-mounted display joint distortion and chromatic aberration pre-compensation method of any one of claims 1-23.

31. A computer program product comprising computer instructions therein, wherein the computer instructions, when executed by a computer, cause the computer to perform the steps of claim 30. The computer instructions, when executed by the processor, implement the MR head-mounted display joint distortion and chromatic aberration pre-compensation method of any one of claims 1-23.

Images