Camera image display method, display device and augmented reality display system

Abstract

The invention provides a camera image display method and device, an augmented reality display system and a computer readable storage medium. The camera image display method comprises the following steps: acquiring a to-be-corrected original image through a live-action camera; an optical machine distortion grid representing optical machine distortion characteristics of the display optical machine is obtained, and the optical machine distortion grid is obtained by performing complementation, displacement, rotation and / or fixed-point processing on a preliminary distortion grid obtained through shooting according to the result of an optical machine calibration experiment; and processing the original image according to the optical machine distortion grid so as to generate a pre-corrected image, and obtaining a display image offsetting optical distortion at the rear end of the display optical machine. By performing complementation, displacement, rotation and / or fixed-point processing on the preliminary distortion grid calibrated by the experiment, the method can obtain a more accurate distortion grid to compensate for the distortion of the optical machine, thereby improving the quality of the display image and improving the user experience.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of extended reality display, and in particular to a camera image display method, a camera image display device, an extended reality display system, and a computer readable storage medium. BACKGROUND

[0002] Extended reality (XR) display technology is an immersive display technology that brings a seamless transition between the virtual world and the real world to the experimenter by creating a real and virtual combined digital environment with modern high-tech means centered on computers, including virtual reality (VR) display, augmented reality (AR) display, mixed reality (MR) display and other implementation modes.

[0003] In virtual reality (VR) display applications, it is usually necessary to place display optical machines and other optical elements between the human eye and the high-resolution display screen to increase the field of view, thereby providing an immersive experience for the user. However, the display optical machine introduces optical machine distortion while increasing the field of view, resulting in distortion of the display image and affecting the user experience. In addition, in augmented reality (AR) and mixed reality (MR) display applications, the real scene image collected by the real scene camera from the real world also needs to be processed through the superposition of the camera lens and the display optical machine before entering the user's eye, thus also producing superimposed distortion effects and affecting the user experience.

[0004] Therefore, with the continuous development of extended reality (XR) display systems towards large field of view and wide angle display, there is an urgent need in the art for a camera image display technology to compensate for the optical machine distortion introduced by the display optical machine, thereby improving the quality of the display image and improving the user experience. SUMMARY

[0005] The following presents a simplified summary of one or more aspects in order to provide a basic understanding of such aspects. This summary is not an extensive overview of all contemplated aspects, and is intended to neither identify key or critical elements of all aspects nor delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that is presented later.

[0006] In order to adapt to the development demand of an extended reality (XR) display system to a large field of view and a wide angle display, the present application provides a camera image display method, a camera image display device, an extended reality display system, and a computer readable storage medium, which can obtain a more accurate distortion grid by completing, shifting, rotating and / or pinpointing a preliminary distortion grid obtained from an experiment calibration, to compensate for optical and mechanical distortion, thereby improving the quality of the display image and the user experience.

[0007] Specifically, the camera image display method according to the first aspect of the present application comprises the following steps: acquiring an original image to be corrected via a real scene camera; acquiring an optical and mechanical distortion grid representing the optical and mechanical distortion characteristics of a display optical and mechanical system, wherein the optical and mechanical distortion grid is obtained by completing, shifting, rotating and / or pinpointing a preliminary distortion grid obtained from a shooting according to the results of an optical and mechanical calibration experiment; and processing the original image according to the optical and mechanical distortion grid to generate a pre-corrected image, and obtaining a display image offsetting optical distortion at the rear end of the display optical and mechanical system.

[0008] Further, in an embodiment of the present application, the step of determining the optical and mechanical distortion grid comprises: inputting a standard horizontal and vertical stripe image into the display optical and mechanical system, and collecting a distortion image output by the display optical and mechanical system via a standard camera; detecting the horizontal and vertical stripe edges of the distortion image, and calculating the intersection coordinates of each horizontal stripe edge line and each vertical stripe edge line to generate a first distortion table indicating the preliminary distortion grid; performing polynomial fitting and weighted averaging on the first distortion table to determine a completed second distortion table; and performing normalization processing on the second distortion table according to a target resolution of the optical and mechanical distortion grid to generate the optical and mechanical distortion grid.

[0009] Further, in an embodiment of the present application, the step of detecting the horizontal and vertical stripe edges of the distortion image comprises: converting the distortion image into a grayscale image; performing binaryzation processing on the grayscale image based on the size of a preset window, wherein the size of the preset window is determined according to the size of the distortion image and the target size of the optical and mechanical distortion table; and performing horizontal and vertical stripe edge detection on the binaryzation-processed distortion image to determine a plurality of horizontal stripe edge lines and a plurality of vertical stripe edge lines therein.

[0010] Further, in an embodiment of the present application, the step of detecting the horizontal and vertical stripe edges of the distortion image comprises: detecting the horizontal and vertical stripe edges of the distortion image, and connecting a plurality of connected domains belonging to the same stripe by using a sliding window; and marking the midpoints of the edge lines represented by a plurality of the connected domains, and sorting the midpoints to obtain a plurality of ordered horizontal stripe edge lines and a plurality of ordered vertical stripe edge lines.

[0011] Further, in an embodiment of the present application, the step of performing polynomial fitting and weighted averaging on the first distortion table to determine a second distortion table includes: performing a first polynomial fitting on the first distortion table with respect to a distortion model to determine a third distortion table; and performing weighted averaging on the first distortion table and the third distortion table to determine the second distortion table.

[0012] Further, in an embodiment of the present application, the step of performing weighted averaging on the first distortion table and the third distortion table to determine the second distortion table includes: performing a second polynomial fitting on the first distortion table with respect to horizontal and vertical stripes, and performing extrapolation on positions without intersection to determine a fourth distortion table; and performing weighted averaging on the fourth distortion table and the third distortion table to determine the second distortion table.

[0013] Further, in an embodiment of the present application, the step of performing normalization on the second distortion table according to a target resolution of the optical-mechanical distortion grid to generate the optical-mechanical distortion grid includes: determining a scaling ratio and / or a rotation ratio of the normalization according to the target resolution of the optical-mechanical distortion grid and values of a plurality of cells in the second distortion table; and performing normalization on the second distortion table according to the scaling ratio and / or the rotation ratio to generate the optical-mechanical distortion grid.

[0014] Further, in an embodiment of the present application, the step of processing the original image according to the optical-mechanical distortion grid to generate a pre-corrected image includes: performing backward unwrapping on the original image according to the optical-mechanical distortion grid to determine a corresponding input position of each output pixel in the pre-corrected image in the original image; and performing weighted averaging on values of a plurality of neighboring pixels near the input position in the original image to determine a value of the output pixel in the pre-corrected image.

[0015] Further, in an embodiment of the present application, after the optical-mechanical distortion grid is acquired, the display method further includes the following steps: storing the acquired optical-mechanical distortion grid to an on-chip flash memory of a processor, wherein the processor is further configured with a backward unwrapping processing circuit, and the step of performing backward unwrapping on the original image according to the optical-mechanical distortion grid includes: reading the optical-mechanical distortion grid from the on-chip flash memory; and inputting the original image and the optical-mechanical distortion grid to the backward unwrapping processing circuit to perform the backward unwrapping.

[0016] Further, in an embodiment of the present application, the step of processing the original image according to the optical-mechanical distortion grid to generate a pre-corrected image comprises: obtaining a camera distortion grid representing camera distortion characteristics of the real scene camera; obtaining a field of view angle conversion grid representing relative poses of a camera view angle of the real scene camera and a human eye view angle of a user; fusing the camera distortion grid, the optical-mechanical distortion grid and the field of view angle conversion grid to generate the fused grid; and processing the original image according to the fused grid to generate a pre-corrected image that offsets camera distortion and optical-mechanical distortion and implements field of view angle conversion.

[0017] Further, in an embodiment of the present application, the step of obtaining a camera distortion grid representing camera distortion characteristics of the real scene camera comprises: collecting a calibration image via the real scene camera; processing the calibration image by using Zhang Zhengyou calibration method based on a distortion model function of a fish-eye-like camera to determine distortion parameters of the real scene camera; and generating the camera distortion grid according to the distortion parameters.

[0018] Further, in an embodiment of the present application, the step of obtaining a field of view angle conversion grid representing relative poses of a camera view angle of the real scene camera and a human eye view angle of a user comprises: obtaining eye movement signals of the user; calculating scaling amount, translation amount and / or rotation amount for converting the camera view angle into the human eye view angle according to the eye movement signals; and generating the field of view angle conversion grid according to the scaling amount, the translation amount and / or the rotation amount.

[0019] Further, in an embodiment of the present application, the step of fusing the camera distortion grid, the optical-mechanical distortion grid and the field of view angle conversion grid to generate the fused grid comprises: performing first fusion processing on the camera distortion grid and the field of view angle conversion grid to generate a field of view angle fused grid; and performing second fusion processing on the field of view angle fused grid and the optical-mechanical distortion grid to generate the fused grid.

[0020] Further, in an embodiment of the present application, before performing the second fusion processing, the correction method further comprises the following steps: performing scaling processing on the optical-mechanical distortion grid according to a required display field of view; and / or performing alignment processing on a display field of view of the display optical-mechanical and a camera field of view of the real scene camera.

[0021] Further, in an embodiment of the present application, the step of performing the second fusion processing on the field of view fusion grid and the optical-mechanical distortion grid to generate the fusion grid comprises: calculating the width and height of each camera distortion grid cell according to the size of the camera distortion grid; and traversing the optical-mechanical distortion grid to find the position and weight proportion of each optical-mechanical distortion grid cell in the camera distortion grid according to the width and height of each camera distortion grid cell to perform interpolation fusion.

[0022] In addition, the camera image display device according to the second aspect of the present application comprises a memory and a processor. The memory stores computer instructions. The processor is connected to the memory and is configured to execute the computer instructions stored in the memory to implement the camera image display method according to the first aspect of the present application.

[0023] Further, in an embodiment of the present application, the processor comprises: an on-chip flash memory for storing an optical-mechanical distortion grid representing the optical-mechanical distortion characteristics of the display optical engine, a camera distortion grid representing the camera distortion characteristics of the real-world camera, and / or a field of view conversion grid representing the relative pose between the camera view angle of the real-world camera and the human eye view angle of the user; a software processing unit configured with a distortion fusion algorithm for fusing the camera distortion grid, the optical-mechanical distortion grid and / or the field of view conversion grid to generate a fusion grid; and a display pipeline configured with a back-warping processing circuit for performing a back-warping hardening processing on the original image according to the fusion grid to generate a pre-corrected image that offsets optical distortion.

[0024] In addition, the extended reality display system according to the third aspect of the present application comprises: a real-world camera for capturing real-world images; the camera image display device according to the second aspect of the present application for obtaining the real-world images from the real-world camera and performing a pre-correction of optical distortion thereon to generate a pre-corrected image that offsets optical distortion; and a display for displaying the pre-corrected image to obtain a display image that offsets optical distortion at the back end of the display optical engine.

[0025] In addition, the computer readable storage medium according to the fourth aspect of the present application stores computer instructions. The computer instructions are executed by a processor to implement the camera image display method according to the first aspect of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0026] The above features and advantages of the present application can be better understood by reading the following detailed description of embodiments of the present application in conjunction with the drawings, in which: the components are not necessarily drawn to scale, and components of similar or identical function or structure can be designated with identical or similar reference numerals.

[0027] Figure 1 A hardware architecture diagram of an extended reality display system provided according to some embodiments of the present application is shown.

[0028] Figure 2 A flowchart of a camera image display method provided according to some embodiments of the present application is shown.

[0029] Figure 3 A schematic of determining a fusion grid provided according to some embodiments of the present application is shown.

[0030] Figure 4 A flowchart of determining an optomechanical distortion grid provided according to some embodiments of the present application is shown.

[0031] Figure 5 A schematic of a binarization process provided according to some embodiments of the present application is shown.

[0032] Figure 6 A schematic of generating a first distortion table provided according to some embodiments of the present application is shown.

[0033] Figure 7 A schematic of completing a distortion table provided according to some embodiments of the present application is shown.

[0034] Figure 8 A schematic of an optomechanical distortion grid provided according to some embodiments of the present application is shown.

[0035] Figure 9 A schematic of a first fusion process provided according to some embodiments of the present application is shown.

[0036] Figure 10 A schematic of a second fusion process provided according to some embodiments of the present application is shown.

[0037] Figure 11 A schematic of scaling an optomechanical distortion table provided according to some embodiments of the present application is shown.

[0038] Figure 12 A schematic of aligning a display field of view of a display optomechanical with a camera field of view of a real scene camera provided according to some embodiments of the present application is shown. DETAILED DESCRIPTION

[0039] The following detailed description is presented to enable any person skilled in the art to make and use the application. Various modifications to the embodiments described herein will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the scope of the application. Thus, the present application is not intended to be limited to the embodiments described herein but is to be accorded the widest scope consistent with the principles and features disclosed herein. To the extent that section headings are used, these should not be taken as limiting the subject matter described therein but are instead used merely for convenience. Further, the descriptions used and the examples provided are intended to be control examples, and as such are provided by way of example only. Nothing in this section should be construed as a limitation on the overall scope of the application.

[0040] In the description of the present application, it is to be understood that the specific structural components, such as "mounting", "connected", "connection", should be interpreted broadly, for example, can be fixedly connected, can be detachably connected, or integrally connected; can be mechanically connected, can be electrically connected; can be directly connected, can be indirectly connected through an intermediate medium, can be internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0041] In addition, "upper", "lower", "left", "right", "top", "bottom", "horizontal", "vertical" used in the following description should be understood as the orientation shown in the section and the related drawings. The relative terms are only for the convenience of description, and do not mean that the device described should be manufactured or operated in a particular orientation, so it should not be understood as a limitation on the application.

[0042] It can be understood that although the terms "first", "second", "third" and the like can be used herein to describe various components, regions, layers and / or parts, these components, regions, layers and / or parts should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers and / or parts. Therefore, the first component, region, layer and / or part discussed below can be referred to as the second component, region, layer and / or part without departing from some embodiments of the application.

[0043] As mentioned above, in the Virtual Reality (VR) display application, it is usually required to place display optics and other optical elements between the human eye and the high-resolution display screen to increase the field of view, so as to provide the user with an immersive experience. However, the display optics will introduce optical distortion while increasing the field of view, thereby causing distortion of the displayed image and affecting the user experience. In addition, in the Augmented Reality (AR) and Mixed Reality (MR) display applications, the real scene image collected by the real scene camera from the real world also needs to be processed through the superposition of the camera lens and the display optics before entering the user's eye, and thus the superimposed distortion effect will also be generated and affect the user experience.

[0044] In order to adapt to the development needs of the Extended Reality (XR) display system to large field of view and wide angle display, the present application provides a camera image display method, a camera image display device, an extended reality display system, and a computer readable storage medium, which can obtain a more accurate distortion grid by completing, displacing, rotating and / or fixing the preliminary distortion grid of the experimental calibration, to compensate for the optical distortion, thereby improving the quality of the displayed image and improving the user experience.

[0045] In some non-limiting embodiments, the above-mentioned camera image display method provided by the first aspect of the present application can be implemented via the above-mentioned camera image display device provided by the second aspect of the present application. Specifically, the above-mentioned camera image display device can be configured in the above-mentioned extended reality display system provided by the third aspect of the present application, wherein the storage and the processor are configured. The storage includes but is not limited to the above-mentioned computer readable storage medium provided by the fourth aspect of the present application, and the computer instructions are stored on the storage. The processor is connected to the storage and is configured to execute the computer instructions stored on the storage to implement the above-mentioned camera image display method provided by the first aspect of the present application.

[0046] First, please refer to Figure 1 , Figure 1 The architecture diagram of the extended reality display system provided by some embodiments of the present application is shown.

[0047] As Figure 1As shown, the third aspect of the present application provides the above-mentioned extended reality display system, which is configured with the real scene camera 10, the camera image display device 20 and the display 30 provided by the second aspect of the present application. In some embodiments, the real scene camera 10 can be a fisheye camera, which is used to collect real scene images of the real world in a wide angle. The display device 20 is used to obtain the real scene images from the real scene camera 10 and pre-correct the optical distortion of the real scene images to generate pre-corrected images that offset the optical distortion. The display 30 is configured with a display driving chip and a screen, which is used to display the pre-corrected images to obtain display images that offset the optical distortion at the user's eyes 40 behind the display optical engine.

[0048] Further, in Figure 1 In the embodiments shown, the camera image display device 20 is configured with a display pipeline 21, an on-chip flash 22, a software processing unit 23 and other modules 24 such as a memory. The on-chip flash 22 is used to store the optical engine distortion grid representing the optical engine distortion characteristics of the display optical engine, the camera distortion grid representing the camera distortion characteristics of the real scene camera 10, and / or the field of view angle conversion grid representing the relative pose between the camera view angle of the real scene camera 10 and the user's eye view angle. The software processing unit 23 includes but is not limited to a firmware computing platform, which is configured with display driving software and / or distortion fusion algorithms for compensating camera distortion, optical engine distortion, field of view angle conversion. The display pipeline 21 can be used as the above-mentioned processor to process the original images according to the fusion grid generated by the distortion fusion algorithms to generate pre-corrected images that offset various optical distortions such as camera distortion, optical engine distortion, field of view angle conversion, etc.

[0049] By pre-determining and storing the optical engine distortion grid representing the optical engine distortion characteristics of the display optical engine, the camera distortion grid representing the camera distortion characteristics of the real scene camera 10, and the field of view angle conversion grid representing the relative pose between the camera view angle of the real scene camera 10 and the user's eye view angle, and fusing one or more of them according to actual needs, and then performing de-distortion processing based on the fusion grid, the present application can compensate for various optical distortions that may be involved in the extended reality (XR) display system in one step, thereby comprehensively coordinating the effects of various optical distortion corrections to improve image quality and overall distortion correction efficiency.

[0050] The working principles of the camera image display device 20 and the extended reality display system will be described below in combination with some embodiments of camera image display methods. Those skilled in the art can understand that the embodiments of the camera image display methods are only some non-limiting embodiments provided by the present application, which are intended to clearly show the main concept of the present application and provide some specific schemes for facilitating the public to implement, rather than for limiting the overall functions or overall working modes of the camera image display device 20. Similarly, the camera image display device 20 is also only a non-limiting embodiment provided by the present application, which does not limit the execution subject and execution order of each step in the camera image display methods.

[0051] Please refer to Figure 1 , Figure 2 and Figure 3 . Figure 2 A flowchart of a camera image display method according to some embodiments of the present application is shown. Figure 3 A schematic diagram of determining a fusion grid according to some embodiments of the present application is shown.

[0052] As shown in Figure 1 and Figure 2 , in the process of displaying the camera image, the live camera 10 can collect the original live image from the real world in real time and transmit it to the display pipeline 21 of the camera image display device 20 for image correction processing to generate a pre-corrected image that offsets one or more optical distortions. Then, the display pipeline 21 can transmit the pre-corrected image to the display 30 at the back end for display, so as to obtain a display image that offsets the optical distortion at the user's eye 40 at the back end of the display light engine.

[0053] As shown in Figure 2 and Figure 3 , for the light engine distortion caused by the display light engine, the skilled person can set a wide-angle standard camera 31 at the position of the user's eye 40 (i.e. the back end of the display light engine), align the center of the standard camera 31 with the center of the screen of the display 30, and make the image plane of the standard camera 31 parallel to the screen, so as to use the standard camera 31 to capture the distorted image output by the display 30 via the display light engine. Then, the standard camera 31 can input the captured distorted image into the light engine distortion grid generation unit 321 in the compensation grid generation algorithm module 32 to generate a light engine distortion grid that characterizes the light engine distortion characteristics of the display light engine.

[0054] Please refer to Figures 4 to 8 . Figure 4 A flowchart of determining a light engine distortion grid according to some embodiments of the present application is shown. Figure 5 A schematic diagram of binarization processing according to some embodiments of the present application is shown. Figure 6A diagram illustrating generation of a first distortion table according to some embodiments of the present application is shown. Figure 7 A diagram illustrating distortion table optimization according to some embodiments of the present application is shown. Figure 8 A diagram illustrating an optical-mechanical distortion grid according to some embodiments of the present application is shown.

[0055] As shown in Figure 4 the process of determining the optical-mechanical distortion grid, the optical-mechanical distortion grid generating unit 321 can first control the display 30 to display a standard horizontal-vertical stripe image, and simulate a human eye to capture a distortion image outputted by the display optical-mechanical system via the standard camera 31. Then, the optical-mechanical distortion grid generating unit 321 can take the standard horizontal-vertical stripe image, the coordinates of the display optical-mechanical center on the acquired distortion image, and the number of horizontal-vertical stripes as inputs, and perform optical-mechanical distortion calculation to determine the optical-mechanical distortion grid.

[0056] Specifically, as shown in Figure 5 the process of optical-mechanical distortion calculation, the optical-mechanical distortion grid generating unit 321 can first convert the acquired distortion image into a grayscale image, and then perform binaryzation processing on the grayscale image based on a preset window size to obtain a black-and-white binary image describing the shape and position of the horizontal-vertical stripes. Then, the optical-mechanical distortion grid generating unit 321 can perform edge detection on the binary image to determine a plurality of horizontal stripe edge lines and a plurality of vertical stripe edge lines. Here, the preset window size is determined according to the size of the distortion image and the target size of the optical-mechanical distortion table to be generated, and the specific value does not involve the technical improvement of the present application, so it is not described.

[0057] Further, in some embodiments, after detecting the plurality of horizontal stripe edge lines and the plurality of vertical stripe edge lines in the distortion image, the optical-mechanical distortion grid generating unit 321 can preferably connect a plurality of connected domains belonging to the same stripe using a sliding window, mark the midpoints of the edge lines represented by the plurality of connected domains, and sort the midpoints to obtain a plurality of ordered horizontal stripe edge lines and a plurality of ordered vertical stripe edge lines. Specifically, in the process of sorting the edge lines, for each horizontal stripe edge line, the midpoint is the point where the middle value of all x coordinates of the horizontal stripe edge line is located, and the optical-mechanical distortion grid generating unit 321 can sort the midpoints according to the y coordinates of the midpoints of the horizontal stripe edge lines. Similarly, for each vertical stripe edge line, the midpoint is the point where the middle value of all y coordinates of the vertical stripe edge line is located, and the optical-mechanical distortion grid generating unit 321 can sort the midpoints according to the x coordinates of the midpoints of the vertical stripe edge lines. Then, the optical-mechanical distortion grid generating unit 321 can determine the serial numbers of the intersection points of the horizontal-vertical stripe edge lines according to the serial numbers of the horizontal-vertical stripe edge lines, and use the serial numbers as the basis for configuring the register serial numbers.

[0058] After that, as shown in Figure 4 , Figure 6 and Figure 7 , the optical-mechanical distortion grid generating unit 321 can calculate the intersection coordinates of each horizontal fringe edge line and each vertical fringe edge line respectively to generate a first distortion table indicating a preliminary distortion grid obtained by detecting the distortion image, and perform polynomial fitting and weighted averaging on the first distortion table to determine a completed second distortion table.

[0059] Specifically, since the first distortion table shown in Figure 6 is a preliminary table obtained by detecting the distortion image, it only includes visible grid points (i.e. non-nan values in the table) that can be obtained by detection, and cannot cover invisible grid points (i.e. nan values in the table) that cannot be obtained by detection. Therefore, in some embodiments, the optical-mechanical distortion grid generating unit 321 can first perform first polynomial fitting (e.g. fourth-order polynomial fitting) on the above-mentioned first distortion table with respect to the distortion model to determine a third distortion table representing the pillow-shaped distortion of the optical-mechanical system, and then perform second polynomial fitting on the above-mentioned first distortion table with respect to the horizontal and vertical fringes, and perform extrapolation processing on the positions without intersection to determine a fourth distortion table supplemented with invisible grid point coordinates. After that, the optical-mechanical distortion grid generating unit 321 can perform weighted averaging on the horizontal and vertical coordinate values of each grid point in the third distortion table and the fourth distortion table with respect to the center coordinates of the images before and after distortion, so as to determine the completed second distortion table as shown in Figure 7 .

[0060] After that, as shown in Figure 4 and Figure 8 , the optical-mechanical distortion grid generating unit 321 can perform normalization processing on the second distortion table according to the target resolution of the optical-mechanical distortion grid to finally generate the optical-mechanical distortion grid representing the optical-mechanical distortion characteristics of the display optical-mechanical system.

[0061] Specifically, in the process of normalization processing, the optical-mechanical distortion grid generating unit 321 can first determine the scaling ratio and / or rotation ratio of the normalization processing according to the target resolution of the optical-mechanical distortion grid and the values of the plurality of cells in the second distortion table. After that, the optical-mechanical distortion grid generating unit 321 can perform fixed-point normalization processing such as displacement, rotation, scaling, etc. on the second distortion table according to the scaling ratio and / or rotation ratio, so as to scale the horizontal and vertical fringe coordinates in the middle of the input horizontal and vertical fringe image to 0-1, and scale the coordinates of other horizontal and vertical fringes according to the same ratio, to finally generate the optical-mechanical distortion grid representing the optical-mechanical distortion characteristics of the display optical-mechanical system.

[0062] Please continue to refer to Figure 1 , Figure 2 and Figure 3After generating the optical-mechanical distortion mesh, the optical-mechanical distortion mesh generation unit 321 can store the generated optical-mechanical distortion mesh into the on-chip flash memory 22 of the camera image display device 20. Subsequently, during the display of the camera image, in response to acquiring the original image to be corrected from the real-view camera 10, the display pipeline 21 can read the optical-mechanical distortion mesh from the on-chip flash memory 22 via the distortion fusion algorithm in the software processing unit 23, and process the original image according to the acquired optical-mechanical distortion mesh to generate a pre-corrected image that can compensate for optical-mechanical distortion at the user's eye 40 at the back end of the display optical engine.

[0063] Specifically, the display pipeline 21 may preferably be configured with a backward warp processing circuit. In response to acquiring the original image to be corrected from the real-view camera 10 and acquiring the optical-mechanical distortion grid from the software processing unit 23, the display pipeline 21 can input the original image and the optical-mechanical distortion grid together into the backward warp processing circuit. The backward warp processing circuit performs backward warp processing on the original image according to the optical-mechanical distortion grid to determine the input position of each output pixel in the pre-corrected image in the original image. Then, the values ​​of multiple adjacent pixels near the input position in the original image are weighted and averaged to determine the value of each output pixel in the pre-corrected image.

[0064] In addition, Figure 3 In the illustrated embodiment, for camera distortion caused by the camera lens, the camera distortion mesh generation unit 322 configured in the compensation mesh generation algorithm module 32 can pre-acquire calibration images via the real-world camera 10, and then, based on the polynomial function R of the fisheye camera distortion model, generate the mesh. d =f(σ,I), and the calibration image is processed using Zhang Zhengyou's calibration method to determine the distortion parameters I1~I4 of the real-scene camera 10. Here, R d σ represents the distance from the image point to the center of the image after camera distortion occurs, σ is the incident angle, which can be calculated from the horizontal and vertical coordinates of the image before camera distortion, and I1~I4 are the polynomial coefficients to be solved.

[0065] Then, the camera distortion mesh generation unit 322 can generate the camera distortion mesh corresponding to the camera field of view according to the obtained distortion parameters I1 to I4, and store the generated camera distortion mesh into the on-chip flash memory 22 of the camera image display device 20.

[0066] Furthermore, in response to the difference in field of view caused by the inconsistency between the camera view of the real-view camera 10 and the human eye view of the user's eye 40, the field of view conversion grid generation unit 323 configured in the compensation grid generation algorithm module 32 can obtain the user's eye movement signal in advance via an eye tracker or other devices, and then calculate the scaling, translation and / or rotation of the camera view into the human eye view based on the eye movement signal, thereby generating a field of view conversion grid based on the scaling, translation and / or rotation, and storing the generated field of view conversion grid into the on-chip flash memory 22 of the camera image display device 20.

[0067] After that, as Figure 1 As shown, in response to receiving the original image to be corrected provided by the real-view camera 10, the distortion fusion algorithm of the software processing unit 23 can first obtain the above-mentioned optical-mechanical distortion grid, camera distortion grid and field of view conversion grid from the on-chip flash memory 22 of the camera image display device 20, and then fuse them to generate a fused grid.

[0068] Please refer to the details. Figures 9 to 12 . Figure 9 A schematic diagram of a first fusion process provided according to some embodiments of the present invention is shown. Figure 10 A schematic diagram of a second fusion process provided according to some embodiments of the present invention is shown. Figure 11 A schematic diagram of a scaled optical engine distortion table provided according to some embodiments of the present invention is shown. Figure 12 A schematic diagram of the display field of view of an aligned display optical engine and the camera field of view of a real-view camera provided according to some embodiments of the present invention is shown.

[0069] like Figure 9 and Figure 10 As shown, in the process of fusing the camera distortion mesh, optomechanical distortion mesh and field-of-view transformation mesh, the distortion fusion algorithm can first perform a first fusion process on the camera distortion mesh and the field-of-view transformation mesh to generate a field-of-view fused mesh, and then perform a second fusion process on the field-of-view fused mesh and the optomechanical distortion mesh to generate the aforementioned fused mesh.

[0070] Furthermore, such as Figure 11 and Figure 12 As shown, in order to overcome the problem of inconsistent field of view and / or table size between the field of view fusion grid and the optomechanical distortion grid, the distortion fusion algorithm can preferably scale the optomechanical distortion grid according to the maximum and minimum values ​​of the horizontal and vertical coordinates of the optomechanical distortion grid and the required display field of view before performing the second fusion process, and / or perform field of view alignment according to the camera field of view of the real scene camera 10 and the field of view of the optomechanical display, so as to improve the correction accuracy of the fusion grid.

[0071] Further, in the process of performing the second fusion processing on the field of view angle fusion grid and the optical-mechanical distortion grid, the distortion fusion algorithm can preferably calculate the width and height of each camera distortion grid unit according to the size of the camera distortion grid, and traverse the optical-mechanical distortion grid according to the width and height of each camera distortion grid unit to find the position and weight proportion of each optical-mechanical distortion grid unit in the camera distortion grid, so as to perform the interpolation fusion of the field of view angle fusion grid and the optical-mechanical distortion grid according to the position and weight proportion.

[0072] As shown in Figure 1 After the fusion grid is generated, the software processing unit 23 can input the fusion grid to the display pipeline 21, which performs the backward unwrapping processing on the original image according to the fusion grid to generate a pre-corrected image that can simultaneously offset the optical-mechanical distortion and the camera distortion at the user's eye 40 at the rear end of the display optical-mechanical and realize the field of view angle conversion from the camera view angle to the human eye view angle, thereby realizing the optical distortion compensation required by various extended reality (XR) display systems such as mixed reality (MR) headsets in one station.

[0073] Those skilled in the art can understand that although Figure 1 The embodiment shown in the figure describes the camera image display device 20 as a multi-element electronic device containing the display pipeline 21, the on-chip flash memory 22, the software processing unit 23 and other modules 24, which is only one non-limiting embodiment provided by the present application, which is intended to clearly demonstrate the main idea of the present application and provide a specific scheme for the public to implement, rather than to limit the protection scope of the present application.

[0074] Alternatively, in other embodiments, those skilled in the art can also integrate the display pipeline 21, the on-chip flash memory 22, the software processing unit 23 and other modules into an extended reality display processor chip based on the above idea through integrated circuit technology and / or chip processing technology, to achieve the functions of the above optical correction and camera image display, and further realize the compactness and lightweight design of the device, which is beneficial to its application in various wearable extended reality (XR) display systems such as mixed reality (MR) headsets.

[0075] In summary, the camera image display method, camera image display device, extended reality display system and computer readable storage medium provided by the present application not only can obtain more accurate distortion grid to compensate for optical-mechanical distortion by completing, displacing, rotating and / or fixing the preliminary distortion grid obtained through experimental calibration, but also can further use a general fusion distortion grid to describe various types of optical distortion such as camera distortion, optical-mechanical distortion and field of view angle conversion, thereby solving various types of optical distortion at the user's eye 40 at the rear end of the display optical-mechanical in one station and improving the user experience.

[0076] Although the above methods have been illustrated and described as a series of actions, it will be understood and appreciated that the methods are not limited by the order of actions, as some actions can, in accordance with one or more embodiments, occur simultaneously or in different order than shown and described herein, or can be omitted entirely, depending on the embodiment.

[0077] Those skilled in the art will appreciate that information, signals, and data can be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that can be referenced throughout the above description can be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0078] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0079] The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein can be implemented or performed with a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor can be a microprocessor, but in the alternative, the processor can be any conventional processor, controller, microcontroller, or state machine. A processor can also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0080] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A display method of a camera image, characterized by, The method comprises the following steps: acquiring an original image to be corrected via a real scene camera; acquiring a light engine distortion grid representing the light engine distortion characteristics of a display light engine, wherein the light engine distortion grid is obtained by completing, displacing, rotating and / or pinpointing a preliminary distortion grid obtained from a light engine calibration experiment; and processing the original image according to the light engine distortion grid to generate a pre-corrected image, and obtaining a display image that offsets optical distortion at the back end of the display light engine.

2. The display method according to claim 1, wherein The step of determining the light engine distortion grid comprises: inputting a standard horizontal and vertical stripe image into the display light engine, and acquiring a distortion image output by the display light engine via a standard camera; detecting the horizontal and vertical stripe edges of the distortion image, and calculating the intersection coordinates of each horizontal stripe edge line and each vertical stripe edge line to generate a first distortion table indicating the preliminary distortion grid; performing polynomial fitting and weighted averaging on the first distortion table to determine a completed second distortion table; and performing normalization processing on the second distortion table according to a target resolution of the light engine distortion grid to generate the light engine distortion grid.

3. The display method according to claim 2, wherein The step of detecting the horizontal and vertical stripe edges of the distortion image comprises: converting the distortion image into a grayscale image; performing binary processing on the grayscale image based on the size of a preset window, wherein the size of the preset window is determined according to the size of the distortion image and the target size of the light engine distortion table; and performing horizontal and vertical stripe edge detection on the distortion image subjected to the binary processing to determine a plurality of horizontal stripe edge lines and a plurality of vertical stripe edge lines.

4. The display method according to claim 3, wherein The step of detecting the horizontal and vertical stripe edges of the distortion image comprises: detecting the horizontal and vertical stripe edges of the distortion image, and connecting a plurality of connected domains belonging to the same stripe by using a sliding window; and marking the midpoints of the edge lines represented by a plurality of the connected domains, and sorting the midpoints to obtain a plurality of ordered horizontal stripe edge lines and a plurality of ordered vertical stripe edge lines.

5. The display method according to claim 2, wherein The step of performing polynomial fitting and weighted averaging on the first distortion table to determine a completed second distortion table comprises: performing first polynomial fitting on the first distortion table with respect to a distortion model to determine a third distortion table; and performing weighted averaging on the first distortion table and the third distortion table to determine the second distortion table.

6. The display method according to claim 5, wherein The step of performing weighted averaging on the first distortion table and the third distortion table to determine the second distortion table comprises: performing second polynomial fitting on the first distortion table with respect to horizontal and vertical stripes, and performing extrapolation processing on positions without intersection to determine a fourth distortion table; and performing weighted averaging on the fourth distortion table and the third distortion table to determine the second distortion table.

7. The display method according to claim 2, wherein The step of performing normalization processing on the second distortion table according to a target resolution of the light engine distortion grid to generate the light engine distortion grid comprises: determine a scaling ratio and / or a rotation ratio of the normalization processing according to a target resolution of the optical-mechanical distortion grid and values of a plurality of cells in the second distortion table; and perform normalization processing on the second distortion table according to the scaling ratio and / or the rotation ratio to generate the optical-mechanical distortion grid.

8. The display method according to claim 1 or 2, wherein The step of processing the original image according to the optical-mechanical distortion grid to generate a pre-corrected image comprises: performing backward unwrapping processing on the original image according to the optical-mechanical distortion grid to determine a corresponding input position of each output pixel in the pre-corrected image in the original image; and performing weighted average on values of a plurality of adjacent pixels near the input position in the original image to determine a value of the output pixel in the pre-corrected image.

9. The display method according to claim 8, wherein After the optical-mechanical distortion grid is acquired, the display method further comprises the following steps: storing the acquired optical-mechanical distortion grid to an on-chip flash memory of a processor, wherein the processor is further configured with a backward unwrapping processing circuit, The step of performing backward unwrapping processing on the original image according to the optical-mechanical distortion grid comprises: reading the optical-mechanical distortion grid from the on-chip flash memory; and inputting the original image and the optical-mechanical distortion grid into the backward unwrapping processing circuit to perform the backward unwrapping processing.

10. The display method according to claim 1 or 2, wherein The step of processing the original image according to the optical-mechanical distortion grid to generate a pre-corrected image comprises: acquiring a camera distortion grid representing camera distortion characteristics of the real scene camera; acquiring a field of view angle conversion grid representing a relative pose of a camera view angle of the real scene camera and a human eye view angle of a user; fusing the camera distortion grid, the optical-mechanical distortion grid and the field of view angle conversion grid to generate the fusion grid; and processing the original image according to the fusion grid to generate a pre-corrected image that offsets camera distortion and optical-mechanical distortion and realizes field of view angle conversion.

11. The display method according to claim 10, wherein The step of acquiring a camera distortion grid representing camera distortion characteristics of the real scene camera comprises: acquiring a calibration image via the real scene camera; processing the calibration image by using Zhang Zhengyou calibration method based on a distortion model function of a fisheye-like camera to determine distortion parameters of the real scene camera; and generating the camera distortion grid according to the distortion parameters.

12. The display method according to claim 10, wherein The step of acquiring a field of view angle conversion grid representing a relative pose of a camera view angle of the real scene camera and a human eye view angle of a user comprises: acquiring eye movement signals of the user; calculating a scaling amount, a translation amount and / or a rotation amount for converting the camera view angle into the human eye view angle according to the eye movement signals; and generating the field of view angle conversion grid according to the scaling amount, the translation amount and / or the rotation amount.

13. The display method according to claim 10, wherein The step of fusing the camera distortion grid, the optical-mechanical distortion grid and the field of view angle conversion grid to generate the fusion grid comprises: performing first fusion processing on the camera distortion grid and the field of view angle conversion grid to generate a field of view angle fusion grid; and performing second fusion processing on the field of view angle fusion grid and the optical-mechanical distortion grid to generate the fusion grid.

14. The display method according to claim 13, wherein Before the second fusion processing is performed, the correction method further comprises the following steps: scaling the optical-mechanical distortion grid according to a required display field of view; and / or aligning the display field of view of the display optical-mechanical system with the camera field of view of the real scene camera.

15. The display method according to claim 13, wherein The step of performing the second fusion processing on the field of view angle fusion grid and the optical-mechanical distortion grid to generate the fusion grid comprises: calculating the width and height of each camera distortion grid cell according to the size of the camera distortion grid; and traversing the optical-mechanical distortion grid according to the width and height of each camera distortion grid cell to find the position and weight proportion of each optical-mechanical distortion grid cell in the camera distortion grid for interpolation fusion.

16. A display device of a camera image, characterized by comprise: a memory having computer instructions stored thereon; and a processor connected to the memory and configured to execute the computer instructions stored on the memory to implement the display method of the camera image according to any one of claims 1-15.

17. The display device of claim 16, wherein, The processor comprises: on-chip flash memory for storing an optical-mechanical distortion grid representing optical-mechanical distortion characteristics of a display optical-mechanical system, a camera distortion grid representing camera distortion characteristics of a real scene camera, and / or a field of view angle conversion grid representing the relative pose between the camera view angle of the real scene camera and the human eye view angle of a user; a software processing unit configured with a distortion fusion algorithm for fusing the camera distortion grid, the optical-mechanical distortion grid, and / or the field of view angle conversion grid to generate a fusion grid; and a display pipeline configured with a backward warping processing circuit for performing a hardening processing of backward warping on the original image according to the fusion grid to generate a pre-corrected image that offsets optical distortion.

18. An extended reality display system, comprising: comprise: a real scene camera for capturing real scene images of the real world; the display device of the camera image according to claim 16 or 17 for obtaining the real scene images from the real scene camera and pre-correcting optical distortion thereof to generate a pre-corrected image that offsets optical distortion; and a display for displaying the pre-corrected image to obtain a display image that offsets optical distortion at the back end of a display optical-mechanical system.

19. A computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are executed by the processor to implement the display method of the camera image according to any one of claims 1-15.

Images