Camera image display method and apparatus, and extended reality display system

Abstract

The present invention provides a camera image display method and apparatus, an extended reality display system, and a computer readable storage medium. The camera image display method comprises: acquiring, via a reality capture camera, an original image to be corrected; acquiring an optomechanical distortion grid representing optomechanical distortion characteristics of a display optomechanical system, wherein the optomechanical distortion grid is obtained by performing completion, translation, rotation and / or fixed-point processing on a captured initial distortion grid on the basis of the result of an optomechanical calibration experiment; and processing the original image on the basis of the optomechanical distortion grid to generate a pre-corrected image, and obtaining, at the rear end of the display optomechanical system, a display image with optical distortion compensated. By performing completion, translation, rotation and / or fixed-point processing on the initial distortion grid obtained by experimental calibration, the present invention can obtain a more accurate distortion grid to compensate for the optomechanical distortion, so that the quality of the display image is improved, and the user experience is improved.

Description

Methods for displaying camera images, display devices, and extended reality display systems

[0001] This application claims priority to Chinese patent application No. 202410558996.1, filed on May 7, 2024, entitled "Method for displaying camera images, display device and extended reality display system", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This invention relates to the field of extended reality display, and more particularly to a method for displaying camera images, a device for displaying camera images, an extended reality display system, and a computer-readable storage medium. Background Technology

[0003] Extended Reality (XR) display technology is an immersive display technology that uses modern high-tech methods centered on computers to create a digital environment that combines real and virtual elements, providing users with a seamless transition between the virtual and real worlds. It includes various implementation methods such as Virtual Reality (VR) display, Augmented Reality (AR) display, and Mixed Reality (MR) display.

[0004] In virtual reality (VR) display applications, optical components such as a display optical engine are typically placed between the user's eye and the high-resolution display to increase the field of view, thereby providing an immersive experience. However, while increasing the field of view, the display optical engine introduces optical-mechanical distortion, leading to image distortion and impacting the user experience. Furthermore, in augmented reality (AR) and mixed reality (MR) display applications, real-world images captured by cameras must be superimposed by the camera lens and the display optical engine before reaching the user's eye, resulting in superimposed distortion and further affecting the user experience.

[0005] Therefore, as extended reality (XR) display systems continue to develop towards large field of view and wide-angle display, there is an urgent need in this field for a camera image display technology to compensate for the optical-mechanical distortion introduced by the display optical engine, thereby improving the quality of the displayed image and enhancing the user experience. Summary of the Invention

[0006] The following provides a brief overview of one or more aspects to offer a basic understanding of them. This overview is not an exhaustive summary of all conceived aspects, nor is it intended to identify key or decisive elements of all aspects, nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed descriptions that follow.

[0007] To meet the growing demand for extended reality (XR) display systems with larger field of view and wider angles, this invention provides a method for displaying camera images, a device for displaying camera images, an extended reality display system, and a computer-readable storage medium. These methods can compensate for optomechanical distortion by completing, shifting, rotating, and / or punctuating a preliminary distorted mesh calibrated in an experiment, thereby improving the quality of the displayed image and enhancing the user experience.

[0008] Specifically, the method for displaying the camera image according to the first aspect of the present invention includes the following steps: acquiring an original image to be corrected via a real-view camera; acquiring an optical-mechanical distortion grid characterizing the optical-mechanical distortion properties of the display optical engine, wherein the optical-mechanical distortion grid is obtained by completing, shifting, rotating and / or punctuating a preliminary distortion grid obtained by shooting based on the results of an optical-mechanical calibration experiment; and processing the original image according to the optical-mechanical distortion grid to generate a pre-corrected image, and obtaining a display image that compensates for optical distortion at the rear end of the display optical engine.

[0009] Further, in one embodiment of the present invention, the step of determining the optical-mechanical distortion mesh includes: inputting a standard horizontal and vertical stripe image into the display optical engine, and acquiring a distorted image output by the display optical engine via a standard camera; detecting the edges of the horizontal and vertical stripes in the distorted image, and calculating the coordinates of the intersection points of each horizontal stripe edge line and each vertical stripe edge line to generate a first distortion table indicating the preliminary distortion mesh; performing polynomial fitting and weighted averaging on the first distortion table to determine a complete second distortion table; and normalizing the second distortion table according to the target resolution of the optical-mechanical distortion mesh to generate the optical-mechanical distortion mesh.

[0010] Furthermore, in one embodiment of the present invention, the step of detecting the horizontal and vertical stripe edges of the distorted image includes: converting the distorted image into a grayscale image; performing binarization processing on the grayscale image based on the size of a preset window, wherein the size of the preset window is determined according to the size of the distorted image and the target size of the optical-mechanical distortion table; and performing horizontal and vertical stripe edge detection on the distorted image after binarization processing to determine multiple horizontal stripe edge lines and multiple vertical stripe edge lines therein.

[0011] Furthermore, in one embodiment of the present invention, the step of detecting the horizontal and vertical stripe edges of the distorted image includes: detecting the horizontal and vertical stripe edges of the distorted image, and connecting multiple connected regions belonging to the same stripe using a sliding window; and marking the midpoints of the edge lines represented by the multiple connected regions, and sorting them according to the midpoints to obtain the multiple ordered horizontal stripe edge lines and the multiple ordered vertical stripe edge lines.

[0012] Furthermore, in one embodiment of the present invention, the step of performing polynomial fitting and weighted averaging on the first distorted table to determine the completed second distorted table includes: performing a first polynomial fitting on the first distorted table with respect to the distortion model to determine a third distorted table; and performing a weighted averaging on the first distorted table and the third distorted table to determine the second distorted table.

[0013] Furthermore, in one embodiment of the present invention, the step of performing a weighted average of the first distorted table and the third distorted table to determine the second distorted table includes: performing a second polynomial fitting of horizontal and vertical stripes on the first distorted table, and performing extrapolation processing on the positions where there is no intersection to determine a fourth distorted table; and performing a weighted average of the fourth distorted table and the third distorted table to determine the second distorted table.

[0014] Furthermore, in one embodiment of the present invention, the step of normalizing the second distortion table according to the target resolution of the optical-mechanical distortion mesh to generate the optical-mechanical distortion mesh includes: determining the scaling ratio and / or rotation ratio of the normalization process according to the target resolution of the optical-mechanical distortion mesh and the values ​​of a plurality of cells in the second distortion table; and normalizing the second distortion table according to the scaling ratio and / or the rotation ratio to generate the optical-mechanical distortion mesh.

[0015] Furthermore, in one embodiment of the present invention, the step of processing the original image according to the optomechanical distortion grid to generate a pre-corrected image includes: performing backward curling processing on the original image according to the optomechanical distortion grid to determine the input position corresponding to each output pixel in the pre-corrected image in the original image; and performing a weighted average of the values ​​of multiple neighboring pixels near the input position in the original image to determine the value of the output pixel in the pre-corrected image.

[0016] Furthermore, in one embodiment of the present invention, after acquiring the optical-mechanical distortion mesh, the display method further includes the following steps: storing the acquired optical-mechanical distortion mesh into the on-chip flash memory of the processor, wherein the processor is further configured with a backward curling processing circuit, and the step of performing backward curling processing on the original image based on the optical-mechanical distortion mesh includes: reading the optical-mechanical distortion mesh from the on-chip flash memory; and inputting the original image and the optical-mechanical distortion mesh into the backward curling processing circuit to perform the backward curling processing.

[0017] Furthermore, in one embodiment of the present invention, the step of processing the original image according to the optomechanical distortion mesh to generate a pre-corrected image includes: obtaining a camera distortion mesh characterizing the camera distortion characteristics of the real-world camera; obtaining a field-of-view transformation mesh characterizing the relative pose of the camera viewpoint of the real-world camera and the user's human eye viewpoint; fusing the camera distortion mesh, the optomechanical distortion mesh, and the field-of-view transformation mesh to generate the fused mesh; and processing the original image according to the fused mesh to generate a pre-corrected image that cancels out camera distortion and optomechanical distortion and achieves field-of-view transformation.

[0018] Furthermore, in one embodiment of the present invention, the step of obtaining a camera distortion mesh characterizing the camera distortion characteristics of the real-world camera includes: acquiring a calibration image via the real-world camera; processing the calibration image using the Zhang Zhengyou calibration method based on a distortion model function similar to a fisheye camera to determine the distortion parameters of the real-world camera; and generating the camera distortion mesh according to the distortion parameters.

[0019] Furthermore, in one embodiment of the present invention, the step of obtaining a field-of-view conversion mesh representing the relative pose of the camera viewpoint of the real-world camera and the user's human eye viewpoint includes: obtaining the user's eye movement signal; calculating, based on the eye movement signal, the scaling, translation, and / or rotation amount that converts the camera viewpoint into the human eye viewpoint; and generating the field-of-view conversion mesh based on the scaling, translation, and / or rotation amount.

[0020] Furthermore, in one embodiment of the present invention, the step of fusing the camera distortion mesh, the optomechanical distortion mesh, and the field-of-view conversion mesh to generate the fused mesh includes: performing a first fusion process on the camera distortion mesh and the field-of-view conversion mesh to generate a field-of-view fused mesh; and performing a second fusion process on the field-of-view fused mesh and the optomechanical distortion mesh to generate the fused mesh.

[0021] Furthermore, in one embodiment of the present invention, before performing the second fusion process, the display method further includes the following steps: scaling the optical engine distortion grid according to the required display field of view; and / or aligning the display field of view of the display optical engine with the camera field of view of the real-world camera.

[0022] Furthermore, in one embodiment of the present invention, the step of performing a second fusion process on the field-of-view fusion mesh and the optomechanical distortion mesh to generate the fusion mesh includes: calculating the width and height of each camera distortion mesh unit according to the size of the camera distortion mesh; and traversing the optomechanical distortion mesh according to the width and height of each camera distortion mesh unit to find the position and weight ratio of each optomechanical distortion mesh unit in the camera distortion mesh for interpolation fusion.

[0023] Furthermore, the camera image display apparatus provided according to the second aspect of the present invention includes a memory and a processor. The memory stores computer instructions. The processor is connected to the memory and configured to execute the computer instructions stored in the memory to implement the camera image display method provided in the first aspect of the present invention.

[0024] Further, in one embodiment of the present invention, the processor includes: on-chip flash memory for storing an optical-mechanical distortion grid characterizing the optical-mechanical distortion characteristics of the display optical engine, a camera distortion grid characterizing the camera distortion characteristics of the real-world camera, and / or a field-of-view conversion grid characterizing the relative pose of the camera viewpoint of the real-world camera and the user's human eye viewpoint; a software processing unit configured with a distortion fusion algorithm for fusing the camera distortion grid, the optical-mechanical distortion grid, and / or the field-of-view conversion grid to generate a fused grid; and a display pipeline configured with a backward curling processing circuit for performing backward curling hardening processing on the original image according to the fused grid to generate a pre-corrected image that compensates for optical distortion.

[0025] Furthermore, the extended reality display system provided according to the third aspect of the present invention includes: a real-world camera for acquiring real-world images; a display device for the camera images provided in the second aspect of the present invention for acquiring the real-world images from the real-world camera and pre-correcting their optical distortion to generate a pre-corrected image that compensates for optical distortion; and a display for displaying the pre-corrected image to obtain a display image that compensates for optical distortion at the back end of the display optical engine.

[0026] Furthermore, the computer-readable storage medium provided according to the fourth aspect of the present invention stores computer instructions. When the computer instructions are executed by a processor, the camera image display method provided in the first aspect of the present invention is implemented. Attached Figure Description

[0027] The above-described features and advantages of the present invention will be better understood after reading the following detailed description of embodiments of the present disclosure in conjunction with the accompanying drawings. In the drawings, components are not necessarily drawn to scale, and components having similar related characteristics or features may have the same or similar reference numerals.

[0028] Figure 1 shows a hardware architecture diagram of an extended reality display system provided according to some embodiments of the present invention.

[0029] Figure 2 shows a flowchart of a camera image display method provided according to some embodiments of the present invention.

[0030] Figure 3 illustrates a schematic diagram of determining the fused mesh according to some embodiments of the present invention.

[0031] Figure 4 shows a flowchart of determining the optomechanical distortion grid according to some embodiments of the present invention.

[0032] Figure 5 shows a schematic diagram of binarization processing provided according to some embodiments of the present invention.

[0033] Figure 6 shows a schematic diagram of generating a first distortion table according to some embodiments of the present invention.

[0034] Figure 7 shows a schematic diagram of a completed distortion table provided according to some embodiments of the present invention.

[0035] Figure 8 shows a schematic diagram of an optomechanical distortion grid provided according to some embodiments of the present invention.

[0036] Figure 9 shows a schematic diagram of a first fusion process provided according to some embodiments of the present invention.

[0037] Figure 10 shows a schematic diagram of a second fusion process provided according to some embodiments of the present invention.

[0038] Figure 11 shows a schematic diagram of a scaled optical engine distortion table provided according to some embodiments of the present invention.

[0039] Figure 12 shows a schematic diagram of the display field of view of an aligned display optical engine and the camera field of view of a real-view camera according to some embodiments of the present invention. Detailed Implementation

[0040] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Although the description of the present invention is presented in conjunction with preferred embodiments, this does not mean that the features of the invention are limited to these embodiments. On the contrary, the purpose of describing the invention in conjunction with embodiments is to cover other options or modifications that may be derived based on the claims of the present invention. To provide a thorough understanding of the invention, many specific details will be included in the following description. The invention may also be implemented without using these details. Furthermore, to avoid confusion or obscuring the focus of the invention, some specific details will be omitted in the description.

[0041] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0042] Furthermore, the terms "upper," "lower," "left," "right," "top," "bottom," "horizontal," and "vertical" used in the following description should be understood as the orientations shown in the relevant paragraphs and accompanying drawings. These relative terms are for illustrative purposes only and do not imply that the described apparatus must be manufactured or operated in a specific orientation, and therefore should not be construed as limiting the invention.

[0043] It is understood that although terms such as "first," "second," and "third" may be used herein to describe various components, regions, layers, and / or parts, these components, regions, layers, and / or parts should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers, and / or parts. Therefore, the first components, regions, layers, and / or parts discussed below may be referred to as second components, regions, layers, and / or parts without departing from some embodiments of the present invention.

[0044] As mentioned above, in Virtual Reality (VR) display applications, optical components such as a display optical engine are typically placed between the user's eye and the high-resolution display screen to increase the field of view, thereby providing the user with an immersive experience. However, while increasing the field of view, the display optical engine introduces optical engine distortion, resulting in image distortion and affecting the user experience. Furthermore, in Augmented Reality (AR) and Mixed Reality (MR) display applications, real-world images captured by cameras from the real world also need to be superimposed by the camera lens and the display optical engine before reaching the user's eye, thus also producing superimposed distortion effects and impacting the user experience.

[0045] To meet the evolving needs of Extended Reality (XR) display systems for larger field-of-view and wider-angle displays, this invention provides a method for displaying camera images, a device for displaying camera images, an Extended Reality display system, and a computer-readable storage medium. These methods can compensate for optomechanical distortion by completing, shifting, rotating, and / or punctuating a preliminary distorted mesh calibrated in an experiment, thereby improving the quality of the displayed image and enhancing the user experience.

[0046] In some non-limiting embodiments, the camera image display method provided in the first aspect of the present invention can be implemented via the camera image display device provided in the second aspect of the present invention. Specifically, the camera image display device can be configured in the extended reality display system provided in the third aspect of the present invention, wherein a memory and a processor are configured. The memory includes, but is not limited to, the computer-readable storage medium provided in the fourth aspect of the present invention, on which computer instructions are stored. The processor is connected to the memory and configured to execute the computer instructions stored in the memory to implement the camera image display method provided in the first aspect of the present invention.

[0047] First, please refer to Figure 1, which shows an architecture diagram of an extended reality display system provided according to some embodiments of the present invention.

[0048] As shown in Figure 1, the extended reality display system provided in the third aspect of the present invention includes a real-world camera 10, a camera image display device 20 provided in the second aspect of the present invention, and a display 30. In some embodiments, the real-world camera 10 may be a fisheye-like camera for capturing real-world images with a wide angle. The display device 20 is used to acquire real-world images from the real-world camera 10 and pre-correct their optical distortion to generate a pre-corrected image that compensates for optical distortion. The display 30 is equipped with a display driver chip and a screen for displaying the pre-corrected image to obtain a display image that compensates for optical distortion at the user's eye 40 at the rear end of the display optical engine.

[0049] Further, in the embodiment shown in FIG1, the camera image display device 20 is configured with a display pipeline 21, on-chip flash memory 22, software processing unit 23, and other modules 24 such as the aforementioned memory. The on-chip flash memory 22 is used to store an optomechanical distortion grid characterizing the optomechanical distortion characteristics of the display optomechanical system, a camera distortion grid characterizing the camera distortion characteristics of the real-view camera 10, and / or a field-of-view transformation grid characterizing the relative pose of the camera viewpoint of the real-view camera 10 and the user's human eye viewpoint. The software processing unit 23 includes, but is not limited to, a firmware computing platform, which is configured with display driver software, and / or a distortion fusion algorithm for compensating for camera distortion, optomechanical distortion, and field-of-view transformation. The display pipeline 21 can be used as the aforementioned processor to process the original image according to the fusion grid generated by the distortion fusion algorithm to generate a pre-corrected image that compensates for various optical distortions such as camera distortion, optomechanical distortion, and field-of-view transformation.

[0050] By pre-determining and storing an optical-mechanical distortion grid characterizing the optical-mechanical distortion characteristics of the display optical engine, a camera distortion grid characterizing the camera distortion characteristics of the real-view camera 10, and a field-of-view conversion grid characterizing the relative pose of the camera viewpoint of the real-view camera 10 and the user's human eye viewpoint, and fusing one or more of these grids according to actual needs, and then performing distortion correction processing based on the fused grid, this invention can compensate for various optical distortions that may be involved in extended reality (XR) display systems in one stop, thereby comprehensively coordinating the effects of various optical distortion corrections to improve image quality and overall distortion correction efficiency.

[0051] The working principle of the camera image display device 20 and the extended reality display system will be described below with reference to some embodiments of camera image display methods. Those skilled in the art will understand that these embodiments of camera image display methods are merely non-limiting implementations provided by the present invention, intended to clearly demonstrate the main concepts of the invention and provide specific solutions convenient for public implementation, rather than limiting all functions or operating modes of the camera image display device 20. Similarly, the camera image display device 20 is also only one non-limiting implementation provided by the present invention, and does not constitute a limitation on the executing entity and execution order of the steps in these camera image display methods.

[0052] Please refer to Figures 1, 2, and 3. Figure 2 shows a flowchart of a camera image display method according to some embodiments of the present invention. Figure 3 shows a schematic diagram of determining a fusion mesh according to some embodiments of the present invention.

[0053] As shown in Figures 1 and 2, during the display of camera images, the real-world camera 10 can acquire raw real-world images in real time and transmit them to the display pipeline 21 of the camera image display device 20 for image correction processing to generate a pre-corrected image that compensates for one or more optical distortions. Then, the display pipeline 21 can transmit this pre-corrected image to the rear-end display 30 for display, thereby obtaining a display image that compensates for optical distortion at the user's eye 40 at the rear end of the display optical engine.

[0054] As shown in Figures 2 and 3, to address the optical-mechanical distortion caused by the display optical engine, technicians can pre-position a wide-angle standard camera 31 at the user's eye position 40 (i.e., the rear end of the display optical engine). This standard camera 31 is aligned with the center of the display screen 30, and its image plane is parallel to the screen. The standard camera 31 then captures the distorted image output from the display optical engine by the display 30. The standard camera 31 can then input the captured distorted image into the optical-mechanical distortion mesh generation unit 321 within the compensation mesh generation algorithm module 32 to generate an optical-mechanical distortion mesh characterizing the optical-mechanical distortion properties of the display optical engine.

[0055] Please refer to Figures 4 through 8. Figure 4 shows a flowchart of determining the optomechanical distortion mesh according to some embodiments of the present invention. Figure 5 shows a schematic diagram of binarization processing according to some embodiments of the present invention. Figure 6 shows a schematic diagram of generating a first distortion table according to some embodiments of the present invention. Figure 7 shows a schematic diagram of distortion table optimization according to some embodiments of the present invention. Figure 8 shows a schematic diagram of the optomechanical distortion mesh according to some embodiments of the present invention.

[0056] As shown in Figure 4, in the process of determining the optomechanical distortion grid, the optomechanical distortion grid generation unit 321 can first control the display 30 to display a standard horizontal and vertical stripe image, and then use a standard camera 31 to simulate the human eye to capture and output the distorted image after it has been processed by the display optomechanical system. Afterwards, the optomechanical distortion grid generation unit 321 can use the standard horizontal and vertical stripe image, the coordinates of the display optomechanical center on the acquired distorted image, and the number of horizontal and vertical stripes as inputs to calculate the optomechanical distortion and determine the optomechanical distortion grid.

[0057] Specifically, as shown in Figure 5, during the optical-mechanical distortion calculation, the optical-mechanical distortion mesh generation unit 321 first converts the acquired distorted image into a grayscale image, and then performs binarization processing on the grayscale image based on a preset window size to obtain a black-and-white binary image describing the shape and position of the horizontal and vertical stripes. Afterwards, the optical-mechanical distortion mesh generation unit 321 performs edge detection on the horizontal and vertical stripes of the binarized image to determine multiple horizontal stripe edge lines and multiple vertical stripe edge lines. Here, the preset window size is determined based on the size of the distorted image and the target size of the optical-mechanical distortion table to be generated; its specific value does not involve any technical improvement of the present invention and therefore will not be described in detail.

[0058] Furthermore, in some embodiments, after detecting and determining multiple horizontal and vertical stripe edge lines in the distorted image, the optomechanical distortion mesh generation unit 321 can preferably use a sliding window to connect multiple connected regions belonging to the same stripe, then mark the midpoints of the edge lines represented by the multiple connected regions, and sort them according to the midpoints to obtain multiple ordered horizontal and vertical stripe edge lines. Specifically, in the process of sorting each edge line, for each horizontal stripe edge line, the midpoint is the point where the median value of all x-coordinates of the horizontal stripe edge line is located, and the optomechanical distortion mesh generation unit 321 can sort them according to the y-coordinate of the midpoint of each horizontal stripe edge line. Similarly, for each vertical stripe edge line, the midpoint is the point where the median value of all y-coordinates of the vertical stripe edge line is located, and the optomechanical distortion mesh generation unit 321 can sort them according to the x-coordinate of the midpoint of each vertical stripe edge line. Subsequently, the optomechanical distortion grid generation unit 321 can determine the sequence number of the intersection point of each horizontal and vertical stripe edge line according to the sequence number of each horizontal and vertical stripe edge line, so as to serve as the basis for configuring the register sequence number.

[0059] Subsequently, as shown in Figures 4, 6 and 7, the optomechanical distortion mesh generation unit 321 can calculate the intersection coordinates of each horizontal stripe edge line and each vertical stripe edge line to generate a first distortion table indicating the preliminary distortion mesh obtained from the detected distortion image, and perform polynomial fitting and weighted averaging on the first distortion table to determine the completed second distortion table.

[0060] Specifically, since the first distortion table shown in Figure 6 is a preliminary table obtained by detecting distorted images, it only includes visible grid points that can be detected (i.e., non-NaN values ​​in the table), and cannot cover invisible grid points that cannot be detected (i.e., NaN values ​​in the table). Therefore, in some embodiments, the optomechanical distortion grid generation unit 321 can first perform a first polynomial fitting (e.g., fourth-order polynomial fitting) on ​​the first distortion table with respect to the distortion model to determine a third distortion table characterizing optomechanical pincushion distortion, and then perform a second polynomial fitting on the first distortion table with horizontal and vertical stripes, and perform extrapolation processing on the non-intersecting positions to determine a fourth distortion table supplemented with the coordinates of invisible grid points. Afterwards, the optomechanical distortion grid generation unit 321 can perform a weighted average of the horizontal and vertical coordinate values ​​of each grid point in the third and fourth distortion tables with respect to the coordinates of the image center before and after distortion, thereby determining the completed second distortion table as shown in Figure 7.

[0061] Then, as shown in Figures 4 and 8, the optomechanical distortion grid generation unit 321 can normalize the second distortion table according to the target resolution of the optomechanical distortion grid to finally generate an optomechanical distortion grid that characterizes the optomechanical distortion properties of the optomechanical system.

[0062] Specifically, during the normalization process, the optomechanical distortion mesh generation unit 321 can first determine the scaling and / or rotation ratio of the normalization process based on the target resolution of the optomechanical distortion mesh and the values ​​of multiple cells in the second distortion table. Then, the optomechanical distortion mesh generation unit 321 can perform normalization processes such as displacement, rotation, scaling, and fixed-point normalization on the second distortion table according to the scaling and / or rotation ratio. This scales the coordinates of the middle horizontal and vertical stripes in the input horizontal and vertical stripe image to between 0 and 1, and scales the coordinates of other horizontal and vertical stripes according to the same ratio, ultimately generating an optomechanical distortion mesh that characterizes the optomechanical distortion properties of the display optomechanical system. Here, the fixed-point normalization process refers to converting the values ​​recorded in each cell of the table from the original floating-point number format to a fixed-point number format to facilitate adaptation to the data format requirements of various software and hardware algorithms in the chip.

[0063] Please continue referring to Figures 1, 2, and 3. After generating the optical-mechanical distortion mesh, the optical-mechanical distortion mesh generation unit 321 can store the generated optical-mechanical distortion mesh into the on-chip flash memory 22 of the camera image display device 20. Subsequently, during the display of the camera image, in response to acquiring the original image to be corrected from the real-view camera 10, the display pipeline 21 can read the optical-mechanical distortion mesh from the on-chip flash memory 22 via the distortion fusion algorithm in the software processing unit 23, and process the original image according to the acquired optical-mechanical distortion mesh to generate a pre-corrected image that can compensate for optical-mechanical distortion at the user's eye 40 at the back end of the display optical engine.

[0064] Specifically, the display pipeline 21 may preferably be configured with a backward warp processing circuit. In response to acquiring the original image to be corrected from the real-view camera 10 and acquiring the optical-mechanical distortion grid from the software processing unit 23, the display pipeline 21 can input the original image and the optical-mechanical distortion grid together into the backward warp processing circuit. The backward warp processing circuit performs backward warp processing on the original image according to the optical-mechanical distortion grid to determine the input position of each output pixel in the pre-corrected image in the original image. Then, the values ​​of multiple adjacent pixels near the input position in the original image are weighted and averaged to determine the value of each output pixel in the pre-corrected image.

[0065] Furthermore, in the embodiment shown in Figure 3, regarding camera distortion caused by the camera lens, the camera distortion mesh generation unit 322 configured in the compensation mesh generation algorithm module 32 can pre-acquire calibration images via the real-world camera 10, and then, based on the polynomial function R of the fisheye camera distortion model, generate the mesh. d =f(σ, I), and the calibration image is processed using Zhang Zhengyou's calibration method to determine the distortion parameters I1~I4 of the real-scene camera 10. Here, R d σ represents the distance from the image point to the center of the image after camera distortion occurs, σ is the incident angle, which can be calculated from the horizontal and vertical coordinates of the image before camera distortion, and I1~I4 are the polynomial coefficients to be solved.

[0066] Then, the camera distortion mesh generation unit 322 can generate the camera distortion mesh corresponding to the camera field of view according to the obtained distortion parameters I1 to I4, and store the generated camera distortion mesh into the on-chip flash memory 22 of the camera image display device 20.

[0067] Furthermore, in response to the difference in field of view caused by the inconsistency between the camera view of the real-view camera 10 and the human eye view of the user's eye 40, the field of view conversion grid generation unit 323 configured in the compensation grid generation algorithm module 32 can obtain the user's eye movement signal in advance via an eye tracker or other devices, and then calculate the scaling, translation and / or rotation of the camera view into the human eye view based on the eye movement signal, thereby generating a field of view conversion grid based on the scaling, translation and / or rotation, and storing the generated field of view conversion grid into the on-chip flash memory 22 of the camera image display device 20.

[0068] Subsequently, as shown in Figure 1, in response to receiving the original image to be corrected provided by the real-view camera 10, the distortion fusion algorithm of the software processing unit 23 can first obtain the aforementioned optical-mechanical distortion grid, camera distortion grid and field-of-view conversion grid from the on-chip flash memory 22 of the camera image display device 20, and then fuse them to generate a fused grid.

[0069] Please refer to Figures 9 to 12 for details. Figure 9 shows a schematic diagram of a first fusion process provided according to some embodiments of the present invention. Figure 10 shows a schematic diagram of a second fusion process provided according to some embodiments of the present invention. Figure 11 shows a schematic diagram of a scaling optical engine distortion table provided according to some embodiments of the present invention. Figure 12 shows a schematic diagram of the display field of view of the aligned display optical engine and the camera field of view of the real-view camera provided according to some embodiments of the present invention.

[0070] As shown in Figures 9 and 10, in the process of fusing the camera distortion mesh, optomechanical distortion mesh and field-of-view transformation mesh, the distortion fusion algorithm can first perform a first fusion process on the camera distortion mesh and the field-of-view transformation mesh to generate a field-of-view fused mesh, and then perform a second fusion process on the field-of-view fused mesh and the optomechanical distortion mesh to generate the aforementioned fused mesh.

[0071] Furthermore, as shown in Figures 11 and 12, in order to overcome the problem of inconsistent field of view and / or table size between the field of view fusion grid and the optomechanical distortion grid, the distortion fusion algorithm can preferably scale the optomechanical distortion grid according to the maximum and minimum values ​​of the horizontal and vertical coordinates of the optomechanical distortion grid and the required display field of view before performing the second fusion process, and / or perform field of view alignment according to the camera field of view of the real-view camera 10 and the field of view of the optomechanical display, so as to improve the correction accuracy of the fusion grid.

[0072] Furthermore, during the second fusion process of the field-view fusion mesh and the optomechanical distortion mesh, the distortion fusion algorithm can preferably calculate the width and height of each camera distortion mesh unit according to the size of the camera distortion mesh, and traverse the optomechanical distortion mesh according to the width and height of each camera distortion mesh unit to find the position and weight ratio of each optomechanical distortion mesh unit in the camera distortion mesh, and then perform interpolation fusion of the field-view fusion mesh and the optomechanical distortion mesh according to the position and weight ratio.

[0073] As shown in Figure 1, after generating the fused mesh, the software processing unit 23 can input the fused mesh into the display pipeline 21, which will then perform backward curling processing on the original image based on the fused mesh to generate a pre-corrected image that can simultaneously cancel optical engine distortion and camera distortion at the user's eye 40 at the back end of the display optical engine, and realize the conversion of the field of view from the camera viewpoint to the user's eye viewpoint. This allows for one-stop realization of various optical distortion compensations required by various extended reality (XR) display systems such as mixed reality (MR) head-mounted displays.

[0074] Those skilled in the art will understand that although the embodiment shown in FIG1 describes the camera image display device 20 as a multi-functional electronic device including display pipeline 21, on-chip flash memory 22, software processing unit 23 and other modules 24, this is only a non-limiting implementation provided by the present invention, intended to clearly illustrate the main concept of the present invention and provide a specific solution that is easy for the public to implement, rather than being used to limit the scope of protection of the present invention.

[0075] Alternatively, in other embodiments, those skilled in the art can also, based on the above concept, integrate various modules such as the display pipeline 21, on-chip flash memory 22, and software processing unit 23 into a single extended reality display processor chip using integrated circuit technology and / or chip fabrication technology, in order to achieve the same optical correction and camera image display functions, and further realize the compact and lightweight design of the device, so as to facilitate its application in various wearable extended reality (XR) display systems such as mixed reality (MR) headsets.

[0076] In summary, the camera image display method, camera image display device, extended reality display system, and computer-readable storage medium provided by this invention can not only obtain a more accurate distortion mesh to compensate for optical-mechanical distortion by completing, shifting, rotating, and / or punctuating the experimentally calibrated preliminary distortion mesh, but also further use a universal fused distortion mesh to describe various types of optical distortion such as camera distortion, optical-mechanical distortion, and field-of-view transformation. This allows for a one-stop solution to various types of optical distortion at 40 points on the user's eye at the back end of the display optical mechanism, thereby improving the user experience.

[0077] Although the methods described above are illustrated and depicted as a series of actions for the sake of simplicity, it should be understood and appreciated that these methods are not limited by the order of the actions, as some actions may occur in a different order and / or concurrently with other actions from the illustrations and descriptions herein or not illustrated and described herein but which may be understood by those skilled in the art, according to one or more embodiments.

[0078] Those skilled in the art will understand that information, signals, and data can be represented using any of a variety of different techniques and arts. For example, the data, instructions, commands, information, signals, bits, symbols, and chips described throughout the above description can be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof.

[0079] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps are described above in a generalized manner in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the invention.

[0080] The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or performed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but in alternatives, it may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.

[0081] The prior description of this disclosure is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not intended to be limited to the examples and designs described herein, but should be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for displaying camera images, characterized in that, Includes the following steps: The original image to be corrected is acquired via a live camera; Obtain an optomechanical distortion mesh characterizing the optomechanical distortion properties of the display optomechanical system. This mesh is obtained by completing, shifting, rotating, and / or puncturing a preliminary distortion mesh acquired through imaging, based on the results of optomechanical calibration experiments. The original image is processed according to the optical-mechanical distortion grid to generate a pre-corrected image, and a display image that compensates for optical distortion is obtained at the back end of the display optical engine.

2. The display method as described in claim 1, characterized in that, The steps for determining the optomechanical distortion grid include: The standard horizontal and vertical stripe images are input into the display optical engine, and the distorted image output by the display optical engine is acquired via a standard camera; The edges of the horizontal and vertical stripes in the distorted image are detected, and the coordinates of the intersection points of each horizontal stripe edge line and each vertical stripe edge line are calculated to generate a first distortion table indicating the preliminary distortion grid. Polynomial fitting and weighted averaging are performed on the first distorted table to determine the completed second distorted table; and The second distortion table is normalized according to the target resolution of the optical-mechanical distortion grid to generate the optical-mechanical distortion grid.

3. The display method as described in claim 2, characterized in that, The step of detecting the edges of the horizontal and vertical stripes in the distorted image includes: Convert the distorted image to a grayscale image; The grayscale image is binarized based on a preset window size, wherein the preset window size is determined according to the size of the distorted image and the target size of the optomechanical distortion table; and The distorted image after binarization is subjected to horizontal and vertical stripe edge detection to determine multiple horizontal stripe edge lines and multiple vertical stripe edge lines.

4. The display method as described in claim 3, characterized in that, The step of detecting the edges of the horizontal and vertical stripes in the distorted image includes: Detect the edges of horizontal and vertical stripes in the distorted image, and connect multiple connected regions belonging to the same stripe using a sliding window; and Mark the midpoints of the edge lines represented by the multiple connected regions, and sort them according to the midpoints to obtain the multiple ordered horizontal stripe edge lines and the multiple ordered vertical stripe edge lines.

5. The display method as described in claim 2, characterized in that, The step of performing polynomial fitting and weighted averaging on the first distorted table to determine the completed second distorted table includes: A first polynomial fit is performed on the first distortion table with respect to the distortion model to determine the third distortion table; and The second distorted table is determined by taking a weighted average of the first distorted table and the third distorted table.

6. The display method as described in claim 5, characterized in that, The step of taking a weighted average of the first distorted table and the third distorted table to determine the second distorted table includes: A second polynomial fitting of horizontal and vertical stripes is performed on the first distorted table, and extrapolation is performed on the non-intersecting positions to determine the fourth distorted table; and The second distortion table is determined by taking a weighted average of the fourth distortion table and the third distortion table.

7. The display method as described in claim 2, characterized in that, The step of normalizing the second distortion table according to the target resolution of the optomechanical distortion grid to generate the optomechanical distortion grid includes: Based on the target resolution of the optomechanical distortion mesh and the values ​​of multiple cells in the second distortion table, the scaling and / or rotation ratio of the normalization process are determined; and The second distortion table is normalized according to the scaling ratio and / or the rotation ratio to generate the optomechanical distortion mesh.

8. The display method as described in claim 1 or 2, characterized in that, The step of processing the original image according to the optomechanical distortion grid to generate a pre-corrected image includes: The original image is backward-curled according to the optical-mechanical distortion grid to determine the input position of each output pixel in the pre-corrected image in the original image; and The values ​​of multiple neighboring pixels near the input position in the original image are weighted and averaged to determine the value of the output pixel in the pre-corrected image.

9. The display method as described in claim 8, characterized in that, After acquiring the optical-mechanical distortion mesh, the display method further includes the following step: storing the acquired optical-mechanical distortion mesh into the on-chip flash memory of the processor, wherein the processor is further configured with a backward curl-up processing circuit. The step of performing backward curling processing on the original image based on the optical-mechanical distortion grid includes: reading the optical-mechanical distortion grid from the on-chip flash memory; and inputting the original image and the optical-mechanical distortion grid into the backward curling processing circuit to perform the backward curling processing.

10. The display method as described in claim 1 or 2, characterized in that, The step of processing the original image according to the optomechanical distortion grid to generate a pre-corrected image includes: Obtain a camera distortion mesh that characterizes the camera distortion properties of the real-world camera; Obtain a field-of-view transformation mesh representing the relative pose of the real-world camera's viewpoint and the user's human eye viewpoint; The camera distortion mesh, the optomechanical distortion mesh, and the field-of-view transformation mesh are fused to generate the fused mesh; and The original image is processed according to the fused mesh to generate a pre-corrected image that compensates for camera distortion and optomechanical distortion and achieves field-of-view conversion.

11. The display method as described in claim 10, characterized in that, The step of obtaining the camera distortion mesh characterizing the camera distortion properties of the real-world camera includes: The calibration image is acquired via the real-view camera; Based on the distortion model function of a fisheye camera, the calibration image is processed using the Zhang Zhengyou calibration method to determine the distortion parameters of the real-scene camera; and The camera distortion mesh is generated based on the distortion parameters.

12. The display method as described in claim 10, characterized in that, The step of obtaining the field-of-view transformation mesh representing the relative pose of the camera viewpoint of the real-world camera and the user's human eye viewpoint includes: Acquire the user's eye movement signals; Based on the eye-tracking signals, calculate the scaling, translation, and / or rotation amounts that convert the camera's field of view into the human eye's field of view; and The field-of-view transformation mesh is generated based on the scaling amount, the translation amount, and / or the rotation amount.

13. The display method as described in claim 10, characterized in that, The step of fusing the camera distortion mesh, the optomechanical distortion mesh, and the field-of-view transformation mesh to generate the fused mesh includes: A first fusion process is performed on the camera distortion mesh and the field-of-view transformation mesh to generate a field-of-view fused mesh; and A second fusion process is performed on the field-view fusion mesh and the optomechanical distortion mesh to generate the fused mesh.

14. The display method as described in claim 13, characterized in that, Before performing the second fusion process, the display method further includes the following steps: The optomechanical distortion grid is scaled according to the required display field of view; and / or The display field of view of the display optical engine is aligned with the camera field of view of the real-scene camera.

15. The display method as described in claim 13, characterized in that, The step of performing a second fusion process on the field-view fusion mesh and the optomechanical distortion mesh to generate the fused mesh includes: Based on the dimensions of the camera distortion mesh, calculate the width and height of each camera distortion mesh cell; and Based on the width and height of each camera distortion mesh unit, the optical-mechanical distortion mesh is traversed to find the position and weight ratio of each optical-mechanical distortion mesh unit in the camera distortion mesh, so as to perform interpolation fusion.

16. A method for displaying camera images, characterized in that, Includes the following steps: The original image to be corrected is acquired via a live camera; Obtain an optomechanical distortion mesh characterizing the optomechanical distortion properties of the display optomechanic; Obtain a camera distortion mesh that characterizes the camera distortion properties of the real-world camera; Obtain a field-of-view transformation mesh representing the relative pose of the real-world camera's viewpoint and the user's human eye viewpoint; The camera distortion mesh, the optomechanical distortion mesh, and the field-of-view transformation mesh are fused to generate the fused mesh; as well as The original image is processed according to the fused mesh to generate a pre-corrected image that compensates for camera distortion and optomechanical distortion and achieves field-of-view conversion.

17. The display method according to claim 16, characterized in that, The step of obtaining the field-of-view transformation mesh representing the relative pose of the camera viewpoint of the real-world camera and the user's human eye viewpoint includes: Acquire the user's eye movement signals; Based on the eye-tracking signals, calculate the scaling, translation, and / or rotation amounts that convert the camera's field of view into the human eye's field of view; and The field-of-view transformation mesh is generated based on the scaling amount, the translation amount, and / or the rotation amount.

18. The display method as described in claim 16, characterized in that, The step of fusing the camera distortion mesh, the optomechanical distortion mesh, and the field-of-view transformation mesh to generate the fused mesh includes: A first fusion process is performed on the camera distortion mesh and the field-of-view transformation mesh to generate a field-of-view fused mesh; and A second fusion process is performed on the field-view fusion mesh and the optomechanical distortion mesh to generate the fused mesh.

19. The display method as described in claim 18, characterized in that, Before performing the second fusion process, the display method further includes the following steps: The optomechanical distortion grid is scaled according to the required display field of view; and / or The display field of view of the display optical engine is aligned with the camera field of view of the real-scene camera.

20. The display method as described in claim 18, characterized in that, The step of performing a second fusion process on the field-view fusion mesh and the optomechanical distortion mesh to generate the fused mesh includes: Based on the dimensions of the camera distortion mesh, calculate the width and height of each camera distortion mesh cell; and Based on the width and height of each camera distortion mesh unit, the optical-mechanical distortion mesh is traversed to find the position and weight ratio of each optical-mechanical distortion mesh unit in the camera distortion mesh, so as to perform interpolation fusion.

21. A display device for camera images, characterized in that, include: Memory, on which computer instructions are stored; as well as A processor, connected to the memory, and configured to execute computer instructions stored in the memory to implement the method for displaying camera images as claimed in any one of claims 1 to 20.

22. The display device as claimed in claim 21, characterized in that, The processor includes: On-chip flash memory is used to store an optomechanical distortion grid characterizing the optomechanical distortion characteristics of the display optomechanical system, a camera distortion grid characterizing the camera distortion characteristics of the real-world camera, and / or a field-of-view conversion grid characterizing the relative pose of the camera viewpoint of the real-world camera and the user's human eye viewpoint. A software processing unit, configured with a distortion fusion algorithm, is used to fuse the camera distortion mesh, the optomechanical distortion mesh, and / or the field-of-view transformation mesh to generate a fused mesh; and The display pipeline includes a backward curling processing circuit for hardening the original image backward according to the fusion grid to generate a pre-corrected image that compensates for optical distortion.

23. An extended reality display system, characterized in that, include: Reality cameras are used to capture real-world images. The camera image display device as described in claim 21 or 22 is configured to acquire the real-scene image from the real-scene camera and pre-correct its optical distortion to generate a pre-corrected image that compensates for the optical distortion. as well as A display for displaying the pre-corrected image to obtain a display image that compensates for optical distortion at the rear end of the display optical engine.

24. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the computer instructions are executed by the processor, the method for displaying camera images as described in any one of claims 1 to 20 is implemented.

Images