Image processor, processing method, storage medium and extended reality display device

Abstract

The present invention provides an image processor, an image processing method, a computer-readable storage medium, and an extended reality display device. The image processor includes a display pipeline integrated with a software processing unit and at least one image processing hardening unit, configured to: acquire a user's eye movement signal and motion sensing information; acquire a virtual rendered image to be processed; optically correct the virtual rendered image based on the eye movement signal and the motion sensing information using first software disposed in the software processing unit and the at least one image processing hardening unit; and display-compensate and correct the virtual rendered image based on the eye movement signal and the motion sensing information using second software disposed in the software processing unit and the at least one image processing hardening unit.

Description

[Technical Field]

[0001] This application claims priority to a patent application filed on September 27, 2022, with Chinese application number 202211184496.3 and titled "Image processor, processing method, storage medium and extended reality display device."

[0002] The present application relates to extended reality display technology, and more particularly to an image processor, an image processing method, a computer-readable storage medium, and an extended reality display device. [Background technology]

[0003] Extended reality (XR) display technology refers to the technology that fuses reality and virtuality through computers to create a virtual environment where humans and machines can interact, and includes, but is not limited to, augmented reality (AR), virtual reality (VR), and mixed reality (MR). By combining these three visual interaction technologies, extended reality display technology can provide users with a sense of seamless transition between the virtual and real worlds.

[0004] In response to the demands of image compensation in the XR field, conventional display compensation algorithms are often unable to support complex and fine compensation, resulting in limited compensation effectiveness. To address the above issues in conventional technology, there is a need for image processing technology that combines the user's eye movement signal and motion sensing information to perform optical compensation and display compensation, and installs and multiplexes image processing hardening means to improve the efficiency of data storage, processing, and transmission, thereby improving the image display quality of XR display devices with limited software and hardware resources. Summary of the Invention [Means for solving the problem]

[0005] The following is a simplified summary of one or more aspects to provide a basic understanding of these aspects. This summary is not an extensive overview of all contemplated aspects, and is not intended to identify key or critical elements of all aspects or to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description provided later.

[0006] To solve the above problems in the prior art, the present invention provides an image processor, an image processing method, a computer-readable storage medium, and an extended reality display device, which combine a user's eye movement signal and motion sensing information to perform optical correction and display compensation correction, and installs and multiplexes an image processing hardening unit to improve the efficiency of data storage, processing, and transmission, thereby improving the image display quality of an XR display device with limited software and hardware resources.

[0007] Specifically, the image processor according to the first aspect of the present invention includes a display pipeline, which integrates a software processing unit and at least one image processing hardening unit, and is configured to: acquire a user's eye movement signal and motion sensing information, acquire a virtual rendered image to be processed, optically correct the virtual rendered image based on the eye movement signal and the motion sensing information using first software arranged in the software processing unit and the at least one image processing hardening unit, and display-compensate-correct the virtual rendered image based on the eye movement signal and the motion sensing information using second software arranged in the software processing unit and the at least one image processing hardening unit.

[0008] Furthermore, in some embodiments of the present invention, the at least one image processing hardening unit includes a first memory, which stores pixel point data at multiple positions in a current frame and / or pixel point data at multiple positions in at least one previous frame. The step of performing the optical correction and / or the display compensation correction on the virtually rendered image includes the steps of: acquiring the pixel point data at multiple positions in the current frame and / or the pixel point data at multiple positions in at least one previous frame from the first memory; and performing the optical correction on the virtually rendered image by the first software and / or performing the display compensation correction on the virtually rendered image by the second software based on the eye movement signal, the motion sensing information, and the pixel point data at multiple positions in the current frame and / or the pixel point data at multiple positions in the at least one previous frame.

[0009] Furthermore, in some embodiments of the present invention, the at least one image processing hardening unit further comprises at least one hardening calculation circuit, and the step of performing the optical correction and / or the display compensation correction on the virtually rendered image further comprises: performing the optical correction on the virtually rendered image by the first software and the at least one hardening calculation circuit based on the eye movement signal, the motion sensing information, and pixel point data at a plurality of positions in the current frame and / or pixel point data at a plurality of positions in the at least one previous frame; and performing the display compensation correction on the virtually rendered image by the second software and the at least one hardening calculation circuit based on the eye movement signal, the motion sensing information, and pixel point data at a plurality of positions in the current frame and / or pixel point data at a plurality of positions in the at least one previous frame.

[0010] Furthermore, in some embodiments of the present invention, the step of performing the display compensation correction on the virtual rendering image by the second software and the at least one sensing calculation circuit based on the eye movement signal, the motion sensing information, and pixel point data at multiple positions in the current frame and / or pixel point data at multiple positions in the at least one past frame comprises the steps of: determining a target pixel point based on the eye movement signal and the motion sensing information; obtaining raw data of the target pixel point and at least one associated pixel point from the first memory; performing hardening calculation on the raw data of the target pixel point and at least one associated pixel point by the at least one hardening calculation circuit; and performing display compensation correction on the virtual rendering image by the second software to remove distortion, correct uniformity, remove color separation, and / or compensate color accuracy based on a result of the hardening calculation.

[0011] Furthermore, in some embodiments of the present invention, the at least one image processing hardening unit further includes a second memory, wherein the second memory stores calibration data of an optical module, a display panel, and / or a camera. The step of performing the optical correction on the virtual rendered image further includes the steps of: acquiring the calibration data from the second memory; and performing lens optical correction on the virtual rendered image according to the calibration data using the first software and the at least one hardening calculation circuit. The step of performing the display compensation correction on the virtual rendered image further includes the steps of acquiring the calibration data from the second memory; and performing screen display compensation correction on the virtual rendered image according to the calibration data using the second software and the at least one hardening calculation circuit.

[0012] Furthermore, in some embodiments of the present invention, the step of performing display compensation correction on the virtual rendering image includes the steps of: acquiring a real scene image to be processed; layer-blending the optically corrected virtual rendering image and the real scene image by third software disposed in the software processing unit and the at least one image processing hardening unit based on the eye movement signal and the motion sensing information to obtain a mixed reality image; and performing the display compensation correction on the mixed reality image by the second software and the at least one image processing sensing unit based on the eye movement signal and the motion sensing information.

[0013] Furthermore, in some embodiments of the present invention, the step of layer-blending the optically corrected virtual rendering image and the real scene image by third software disposed in the software processing unit and the at least one image processing hardening unit based on the eye movement signal and the motion sensing information includes the steps of: acquiring interaction information between the real scene image and the virtual rendering image; and performing the layer-blending on the optically corrected virtual rendering image and the real scene image by the third software and the at least one image processing hardening unit based on the eye movement signal, the motion sensing information, and the interaction information.

[0014] Furthermore, in some embodiments of the present invention, the display pipeline is further configured to: color enhance the optically corrected virtual rendering image or the mixed reality image by a fourth software disposed in the software processing unit and the at least one image processing hardening unit based on the eye movement signal and the motion sensing information, thereby obtaining a color-enhanced image; and perform the display compensation correction on the color-enhanced image by the second software and the at least one image processing hardening unit based on the eye movement signal and the motion sensing information.

[0015] Furthermore, in some embodiments of the present invention, the at least one hardening calculation circuit further comprises at least one of a weighting addition circuit, an average calculation circuit, a filter circuit, and a pixel position relationship mapping circuit. The display pipeline is further configured to perform spatial distortion correction on the virtually rendered image by fifth software disposed in the software processing unit and the at least one image processing hardening unit based on the eye movement signal, and / or perform recompression processing on the virtually rendered image by sixth software disposed in the software processing unit and the at least one image processing hardening unit based on the eye movement signal.

[0016] In addition, the image processing method according to a second aspect of the present invention includes the steps of acquiring a user's eye movement signal and motion sensing information, and acquiring a virtual rendering image to be processed; optically correcting the virtual rendering image using first software arranged in the software processing unit and the at least one image processing hardening unit based on the eye movement signal and the motion sensing information; and display compensation correcting the virtual rendering image using second software arranged in the software processing unit and the at least one image processing hardening unit based on the eye movement signal and the motion sensing information.

[0017] The computer-readable storage medium according to the third aspect of the present invention stores computer instructions that, when executed by a processor, implement the image processing method according to the second aspect of the present invention.

[0018] Furthermore, the extended reality display device according to a fourth aspect of the present invention comprises an eye tracker, a motion sensor, a main processor, a coprocessor, and a display terminal. The eye tracker collects a user's eye movement signal. The motion sensor collects user movement sensing information. The main processor outputs a virtual rendering image to be processed. The coprocessor can select the image processor according to the first aspect of the present invention. The image processor is connected to each of the eye tracker, the motion sensor, and the main processor, and acquires the eye movement signal, the movement sensing information, and the virtual rendering image. The display terminal is connected to the image processor, and acquires and displays a corrected image that has been optically corrected and displayed compensation corrected by the image processor.

[0019] In some embodiments of the present invention, the extended reality display device further includes a camera, wherein the image processor is connected to the camera and configured to acquire a real scene image to be processed via the camera, layer-blend the optically corrected virtual rendering image and the real scene image using a software processing unit and at least one image processing hardening unit disposed in the image processor based on the eye movement signal and the motion sensing information to acquire a mixed reality image, and perform display compensation correction on the mixed reality image using the software processing unit and the at least one image processing hardening unit based on the eye movement signal and the motion sensing information. [Brief explanation of the drawings]

[0020] The above features and advantages of the present invention can be better understood by reading the detailed description of the embodiments of the present disclosure in conjunction with the following drawings, in which components are not necessarily drawn to scale and components having similar associated properties or characteristics may be labeled with the same or similar reference numerals.

[0021] [Figure 1]1 is a schematic diagram of an extended reality display device provided in accordance with some embodiments of the present invention. [Figure 2] 1 is a flow diagram of an image processing method provided in accordance with some embodiments of the present invention. [Figure 3] 3 is a schematic diagram of an image correction process performed on a frame-by-frame basis provided in accordance with some embodiments of the present invention; [Figure 4] 1 is a schematic diagram of distortion removal correction provided in accordance with some embodiments of the present invention. [Figure 5] FIG. 2 is a schematic diagram of uniformity correction provided in accordance with some embodiments of the present invention. [Figure 6] FIG. 2 is a schematic diagram of color separation removal correction provided in accordance with some embodiments of the present invention. [Figure 7] 1 is a schematic diagram of a color accuracy compensation correction provided in accordance with some embodiments of the present invention. [Figure 8] 2 is a schematic diagram of an image correction process performed by each device provided in accordance with some embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0022] Hereinafter, embodiments of the present invention will be described with reference to specific examples. However, those skilled in the art can easily understand other advantages and effects of the present invention based on the contents disclosed herein. The present invention will be described in connection with a preferred embodiment, but this does not mean that the features of the present invention are limited to this embodiment. On the contrary, the purpose of introducing the invention in connection with the embodiment is to cover other options or variations that may extend based on the scope of the claims of the present invention. The following description includes many specific details to provide a thorough understanding of the present invention. The present invention can also be practiced without these details. Furthermore, specific details will be omitted in the description to avoid confusing or obscuring the focus of the present invention.

[0023] In the description of the present invention, unless otherwise expressly defined and limited, the terms "attached," "coupled," and "connected" should be understood in a broad sense, and may mean, for example, a fixed connection, a detachable connection, an integral connection, a mechanical connection, an electrical connection, a direct connection, an indirect connection via an intermediate medium, or internal communication between two elements. The specific meanings of the above terms in the present invention can be specifically understood by those skilled in the art.

[0024] Additionally, the terms "upper," "lower," "left," "right," "top," "bottom," "horizontal," and "vertical" used in the following description should be understood as referring to the orientations shown in this paragraph and in the associated drawings. These relative terms are used for convenience of explanation only and do not imply that the described devices must be manufactured or operated in any particular orientation, and therefore should not be understood as limiting the present invention.

[0025] Although terms such as "first," "second," and "third" may be used herein to describe various components, regions, layers, and / or portions, it is understood that these components, regions, layers, and / or portions should not be limited by these terms, and that these terms are used only to distinguish between different components, regions, layers, and / or portions. Thus, a first component, region, layer, and / or portion described below may be referred to as a second component, region, layer, and / or portion without departing from some embodiments of the present invention.

[0026] As described above, to meet the image compensation requirements for XR, most conventional display compensation algorithms are implemented as hardware modules in the display driver IC (DDIC) on the screen side, while most conventional optical compensation algorithms are implemented as software in the graphics processing unit (GPU). However, display algorithms implemented through DDIC hardware often cannot support complex and detailed compensation due to limitations in the DDIC manufacturing process and hardware resources, limiting the compensation effectiveness. Optical distortion compensation methods implemented through GPU software suffer from high power consumption and latency. In addition, there are several new display defects that require the design of compensation algorithms when combining a micro-OLED display with a pancake lens optical component in an XR display device or an LCD display with a pancake lens optical component in a 0D / 1D / 2D backlight module.

[0027] In order to solve the above problems in the prior art, the present invention provides an image processor, an image processing method, a computer-readable storage medium, and an extended reality display device that can perform optical correction and display compensation correction by combining a user's eye movement signal and movement sensing information, and can improve the image display quality of an XR display device with limited software and hardware resources by installing and multiplexing an image processing hardening unit to improve the efficiency of data storage, processing, and transmission.

[0028] In some non-limiting examples, the image processing method according to the second aspect of the present invention can be implemented via the image processor according to the first aspect of the present invention. Specifically, the image processor may be disposed separately in the extended reality display device according to the fourth aspect of the present invention as a co-processor chip, or may be integrated as a software program and hardware unit into a main processor such as a central processing unit (CPU) or a graphics processing unit (GPU) of the extended reality display device according to the fourth aspect of the present invention.

[0029] Furthermore, the image processor according to the first aspect of the present invention may comprise or be connected to a processing unit and a storage unit of a software program, including but not limited to a computer-readable storage medium having stored thereon the computer instructions according to the third aspect of the present invention. The processing unit is connected to the storage unit and performs the image processing method according to the second aspect of the present invention by executing the computer instructions stored in the storage unit.

[0030] The operating principles of the above-mentioned image processor and extended reality display device will be described below in conjunction with several embodiments of an image processing method. In some non-limiting embodiments, the extended reality display device may adopt a coprocessor system architecture. Those skilled in the art will understand that these embodiments of the image processing method are merely some non-limiting embodiments provided by the present invention, intended to clearly illustrate the main concept of the present invention and provide some specific solutions to facilitate public implementation, and are not intended to limit all functions or all operating modes of the image processor and extended reality display device. Similarly, the image processor and extended reality display device are also merely non-limiting embodiments provided by the present invention, and do not constitute limitations on the entities that perform each step in these image processing methods.

[0031] Reference is now made in conjunction with Figures 1 and 2. Figure 1 shows a schematic diagram of an extended reality display device provided in accordance with some embodiments of the present invention. Figure 2 shows a flow diagram of an image processing method provided in accordance with some embodiments of the present invention.

[0032] As shown in FIG. 1 , in some embodiments of the present invention, an extended reality display device may include an eye tracker 10, a motion sensor 20, a main processor 30, a coprocessor 40, and a display terminal 50. The eye tracker 10 is for collecting eye movement signals from a user. The motion sensor 20 is for collecting user movement sensing information. The main processor 30 is for outputting a virtual rendering image to be processed. The coprocessor 40 may select and use the image processor described above according to the first aspect of the present invention. The coprocessor 40 is connected to the eye tracker 10, the motion sensor 20, and the main processor 30, respectively, and acquires the eye movement signals, the movement sensing information, and the virtual rendering image. The display terminal 50 is connected to the coprocessor 40 and acquires and displays an image that has been optically corrected and display-compensated by the coprocessor 40.

[0033] Furthermore, the coprocessor 40 may be provided with a display pipeline, which integrates a software processing unit and at least one image processing hardening unit and is configured to perform image processing flows such as optical correction and display compensation correction of virtual rendering images using different software programs and the same image processing hardening unit, thereby improving the efficiency of data storage, processing, and transmission and improving the image display quality of the XR display device with limited software and hardware resources.

[0034] Specifically, the software processing units disposed in the display pipeline of the coprocessor 40 include, but are not limited to, an optical correction unit and a display compensation correction unit. The image processing hardening unit disposed in the display pipeline is at least one selected from a transistor-level cache memory, a weighted summation circuit, an average calculation circuit, a filter circuit, and a pixel position relationship mapping circuit, and can cache multiple pixel data in a first image of a current frame and / or several previous frames, and / or perform hardening calculations such as weighted summation, average calculation, filtering, and / or pixel position relationship mapping on the pixel data.

[0035] As shown in FIG. 2, during image processing, the display pipeline can first obtain the user's eye movement signal, motion sensing information, and a virtual rendering image to be processed.

[0036] Here, the eye movement signal includes data such as the user's eye deflection angle, gaze position, and gaze direction, but is not limited to these. The display pipeline may be directly connected to the eye tracker 10 and directly acquire the eye movement signal from the eye tracker 10, or may be indirectly connected to the eye tracker 10 via an image processing unit such as a GPU of the main processor 30 and synchronously acquire the user's eye movement signal via the main processor 30. In some embodiments, a display driving software and / or firmware computing platform is preferably disposed in the coprocessor 40. This display driving software and / or firmware computing platform is connected to each of the eye tracker 10, the motion sensor 20, and the display pipeline. When acquiring the user's eye movement signal, the coprocessor 40 may first acquire the user's eye movement signal from the eye tracker 10 via the display driving software and / or firmware computing platform, perform eye tracking calculations, and determine the gaze position. The display driver software and / or firmware computing platform can then update the viewpoint information within the system based on this viewpoint position, build an updated compression model, and forward the compression parameters of the updated compression model to the display pipeline for subsequent image processing.

[0037] The motion sensing information may be selected from three degrees of freedom (X / Y / Z) of the user's head and correspondingly calculated yaw, pitch, and roll information, or six degrees of freedom (forward / backward, up / down, left / right, yaw, pitch, and roll) of the user's head. The virtual rendered image may be a raw image generated by an image processing unit such as a GPU of the main processor 30. In some embodiments, the main processor 30 is connected to the eye tracker 10, acquires the user's eye movement signals from the eye tracker 10, performs viewpoint rendering compression on the generated raw virtual rendered image based on the eye movement signals, and transmits the viewpoint rendering compressed compressed image to the coprocessor 40, thereby reducing the data transmission load and data processing load of the entire system.

[0038] As shown in Figures 1 and 2, after obtaining a user's eye movement signal and motion sensing information and obtaining a virtual rendering image to be processed, an optical correction unit (i.e., first software) arranged in the display pipeline and at least one integrated image processing hardening unit can perform optical correction processing on the obtained virtual rendering image.

[0039] Specifically, the at least one image processing hardening unit includes a first memory. The first memory may store pixel point data at multiple positions in a current frame and / or pixel point data at multiple positions in at least one previous frame. As shown in FIG. 3, during optical correction, the optical correction unit (i.e., first software) disposed in the display pipeline may first determine a target pixel point and at least one related pixel point corresponding to a viewpoint based on a user's eye movement signal and motion sensing information, and then obtain pixel point data at multiple corresponding positions in the current frame and / or pixel point data at multiple corresponding positions in at least one previous frame from the first memory. Then, the optical correction unit (i.e., first software) may perform optical correction processing on the virtual rendering image based on the obtained eye movement signal, motion sensing information, and pixel point data at multiple positions in the current frame and / or pixel point data at multiple positions in at least one previous frame.

[0040] Furthermore, the at least one image processing hardening unit further includes at least one hardening calculation circuit, which is at least one selected from a weighted summation circuit, an average calculation circuit, a filter circuit, and a pixel position relationship mapping circuit. In optically correcting the virtually rendered image, the display pipeline can use the first software and the at least one hardening calculation circuit to optically correct the virtually rendered image based on the eye movement signal, the motion sensing information, and pixel point data at multiple positions in the current frame and / or pixel point data at multiple positions in at least one past frame.

[0041] For example, the display pipeline can sequentially input the original data of the target pixel point and at least one associated pixel point (i.e., pixel point data at multiple positions in the current frame, and / or pixel point data at multiple positions in the at least one past frame) into at least one hardening calculation circuit, such as a weighted addition circuit, an average value calculation circuit, a filter circuit, and a pixel position relationship mapping circuit, to perform hardening calculation, obtain corresponding hardening calculation results, and then optically correct the virtual rendering image based on the hardening calculation results using the first software.

[0042] In this way, in the present invention, when optical correction is performed for each frame, pixel point data for multiple positions in images of multiple frames can be multiplexed via the first memory, which significantly reduces the demands on hardware storage resources and eliminates the need to repeatedly calculate and cache each pixel point data, thereby improving the image display quality of XR display devices with limited software and hardware resources.

[0043] Also, as shown in Figures 1 and 2, after acquiring the user's eye movement signal and motion sensing information and acquiring the virtual rendering image to be processed, the display pipeline may perform display compensation correction processing on the acquired virtual rendering image using the arranged display compensation correction unit (i.e., second software) and at least one integrated image processing hardening unit.

[0044] As described above, the at least one image processing hardening unit includes a first memory. The first memory can store pixel point data at multiple positions in a current frame and / or pixel point data at multiple positions in at least one previous frame. As shown in FIG. 3, when performing display compensation correction, the display compensation correction unit (i.e., second software) disposed in the display pipeline can determine a target pixel point corresponding to a viewpoint and at least one associated pixel point thereof based on a user's eye movement signal and motion sensing information, and obtain pixel point data at multiple corresponding positions in the current frame and / or pixel point data at multiple corresponding positions in at least one previous frame from the first memory. Then, the display compensation correction unit (i.e., second software) can perform display compensation correction processing on the virtually rendered image based on the obtained eye movement signal, motion sensing information, and pixel point data at multiple positions in the current frame and / or pixel point data at multiple positions in at least one previous frame.

[0045] Furthermore, the at least one image processing hardening unit further includes at least one hardening calculation circuit, such as the above-mentioned weighting addition circuit, average value calculation circuit, filter circuit, pixel position relationship mapping circuit, etc. When performing display compensation correction processing on the virtually rendered image, the display pipeline can use the second software and the at least one hardening calculation circuit to perform display compensation correction processing on the virtually rendered image based on the eye movement signal, the motion sensing information, and pixel point data at multiple positions in the current frame and / or pixel point data at multiple positions in at least one past frame.

[0046] For example, the display pipeline may first sequentially input the original data of a target pixel point and at least one associated pixel point (i.e., pixel point data at multiple positions in a current frame, and / or pixel point data at multiple positions in the at least one past frame) into at least one hardening calculation circuit, such as a weighted summation circuit, an average value calculation circuit, a filter circuit, or a pixel position relationship mapping circuit, to perform hardening calculation, obtain corresponding hardening calculation results, and then use second software to perform display compensation correction processes, such as distortion removal, uniformity correction, color separation removal, and / or color accuracy compensation, on the virtual rendering image based on the hardening calculation results.

[0047] Specifically, please refer to Figures 4 to 7. Figure 4 shows a schematic diagram of distortion removal correction provided in accordance with some embodiments of the present invention. Figure 5 shows a schematic diagram of uniformity correction provided in accordance with some embodiments of the present invention. Figure 6 shows a schematic diagram of color separation removal correction provided in accordance with some embodiments of the present invention. Figure 7 shows a schematic diagram of color accuracy compensation correction provided in accordance with some embodiments of the present invention.

[0048] As shown in Figure 4, distortion removal correction is based on pixel coordinate correspondence. Specifically, the pixel point data (r xout ,g xout ,b xout ) is the viewpoint information, weight setting, and pixel position information of the original image (r xin ,g xin ,b xin ) 3×3 dimensional correspondence mapping relation M x,3×3 and is implemented mainly based on a pixel mapping circuit to improve problems such as image distortion.

[0049] As shown in Fig. 5, the uniformity correction is performed by using the pixel position information r xin and uniformity compensation gain demura , temperature compensation gain temperature , life compensation gain life The target pixel value r of each pixel is calculated based on the gain coefficients for the three dimensions of xoutThe principle is to determine the uniformity of the sub-pixels and mainly correct the uniformity of the sub-pixels.

[0050] As shown in Fig. 6, the color separation removal correction is performed by using the pixel position information (r xin ,g xin ,b xin ) and weighted average the data of neighboring pixels of the target pixel point to make the RGB three colors correspond to the data of the target pixel point. This is mainly used to deal with color separation of the RGB three colors due to image distortion.

[0051] As shown in Figure 7, the color accuracy compensation correction is performed using a 3 × 3 dimensional matrix f 3dlut The ratio of the RGB three colors of the target pixel point is calculated by xin ,g xin ,b xin ) and the pixel data of the target pixel point (r xout ,g xout ,b xout ) and is mainly used to compensate for the color accuracy of the display panel.

[0052] As described above, one aspect of the present invention is that when performing display compensation correction for each frame, pixel point data for multiple positions in images of multiple frames is multiplexed via a first memory, thereby significantly reducing the demands on hardware storage resources and eliminating the need to repeatedly calculate and cache each pixel point data. At the same time, by using hardware resources such as a weighted addition circuit, an average calculation circuit, a filter circuit, and a pixel position relationship mapping circuit in conjunction with optical correction, the software processing load for display compensation correction can be reduced, and ultimately the image display quality of an XR display device can be improved despite the limited software and hardware resources.

[0053] Furthermore, as shown in FIG. 1 , in some embodiments of the present invention, the extended reality display device preferably further includes a camera 60. Here, the camera 60 is at least one selected from a binocular fisheye camera, a monochrome IR camera, a structured light depth camera, and a laser ToF depth camera, and is connected to the coprocessor 40 via an image signal processing (ISP) module. The display pipeline disposed in the coprocessor 40 preferably acquires a real scene image to be processed via the camera 60, and, based on the acquired eye movement signal and motion sensing information, layer-blends the optically corrected virtual rendering image and the real scene image using a locally disposed layer blending unit (i.e., third software) and at least one image processing hardening unit, such as the first memory, weighted summation circuit, average calculation circuit, filter circuit, and pixel position relationship mapping circuit, to obtain a mixed reality image. Here, the real scene image may be a real scene image optically corrected by the ISP module, or may be an original real scene image without optical correction. Then, the display pipeline can perform the above-mentioned display compensation correction on the mixed reality image by the software processing unit and at least one image processing hardening unit based on the acquired eye movement signal and motion sensing information.

[0054] Furthermore, in some embodiments of the present invention, the at least one image processing hardening unit preferably further includes a second memory. The second memory is for storing calibration data of an optical module (e.g., a lens module), a display panel, and / or a camera. As shown in FIG. 8 , when optically correcting a virtual rendering image, the display pipeline can obtain calibration data for each device from the second memory one by one, and use the optical correction unit (i.e., first software) to perform lens optical correction on the virtual rendering image based on the calibration data, in combination with at least one image processing hardening unit, such as the first memory, a weighting and summing circuit, an average value calculation circuit, a filter circuit, and a pixel position relationship mapping circuit. Furthermore, when performing display compensation correction on a virtual rendering image, the display pipeline can obtain calibration data for each device from the second memory one by one, and use the display compensation correction unit (i.e., second software) to perform screen display compensation correction on the virtual rendering image based on the calibration data, in combination with at least one image processing hardening unit, such as the first memory, a weighting and summing circuit, an average value calculation circuit, a filter circuit, and a pixel position relationship mapping circuit. In this way, the present invention can further improve the image display quality of XR display devices with limited software and hardware resources.

[0055] In addition, in performing display compensation correction on the mixed reality image, the display pipeline preferably acquires interaction information, such as occlusion information between the real scene image and the virtual rendering image and transparency of related layers, from the graphics processing unit (GPU) of the main processor 30. Then, based on the eye movement signal, the motion sensing information, and the interaction information, the display pipeline performs layer blending on the optically corrected virtual rendering image and the real scene image using the layer blending unit (i.e., the third software) in combination with at least one image processing hardening unit, such as the first memory, weighting addition circuit, average calculation circuit, filter circuit, and pixel position relationship mapping circuit. The display pipeline may then perform display compensation correction on the aliased mixed reality image using the display compensation correction unit (i.e., the second software) and at least one image processing hardening unit, such as the first memory, weighting addition circuit, average calculation circuit, filter circuit, and pixel position relationship mapping circuit. Specific aspects of display compensation correction on the mixed reality image are similar to those in the above-described embodiment and are not described here.

[0056] In some embodiments of the present invention, it is preferable to further arrange a color enhancement unit (i.e., fourth software) in the display pipeline. In image processing, the display pipeline uses the color enhancement unit (i.e., fourth software) to color enhance the optically corrected virtual rendering image or mixed reality image based on the acquired eye movement signal and motion sensing information, in combination with at least one image processing hardening unit, such as the first memory, weighted summation circuit, average calculation circuit, filter circuit, and pixel position relationship mapping circuit, to obtain a color-enhanced image. Furthermore, based on the eye movement signal and motion sensing information, the display compensation correction means (i.e., second software) and at least one image processing hardening unit, such as the first memory, weighted summation circuit, average calculation circuit, filter circuit, and pixel position relationship mapping circuit, perform display compensation correction on the acquired color-enhanced image. The specific aspects of the display compensation correction for the color-enhanced image are similar to those in the above-described embodiments, and therefore will not be described here.

[0057] In some embodiments of the present invention, the display pipeline may further include a spatial distortion correction unit (i.e., fifth software). In image processing, the display pipeline may perform spatial distortion correction on the virtual rendering image based on the acquired eye movement signal by using the spatial distortion correction unit (i.e., fifth software) in combination with at least one image processing hardening unit, such as the first memory, weighted summation circuit, average value calculation circuit, filter circuit, and pixel position relationship mapping circuit.

[0058] Specifically, when performing spatial distortion correction, upon acquiring a virtual rendering image to be processed, the spatial distortion correction unit (i.e., the fifth software) can first determine pixel data and processing parameters required for the spatial distortion correction process. In some embodiments, the pixel data is preferably determined based on the user's eye movement signal. The spatial distortion correction unit (i.e., the fifth software) then acquires processing parameters from the corresponding memories and pixel cache data required for the image distortion correction process from the corresponding first memories. The processing parameters and pixel cache data are then sequentially input to one or more of the weighting and summation circuit, average calculation circuit, filter circuit, and pixel position relationship mapping circuit described above, and performs weighting and summation, average calculation, filtering, and / or pixel position relationship mapping hardening calculations on the pixel cache data to obtain corresponding hardening calculation results. The spatial distortion correction unit (i.e., the fifth software) can then obtain a distortion-corrected image through software operations such as data arrangement and allocation, thereby improving the image display quality of an XR display device with limited software and hardware resources.

[0059] In some embodiments of the present invention, it is preferable to further arrange a recompression unit (i.e., sixth software) in the display pipeline. In image processing, the display pipeline can perform recompression processing on the virtually rendered image based on the acquired eye movement signal by this recompression unit (i.e., sixth software) in combination with at least one image processing hardening unit such as the first memory, weighted summation circuit, average value calculation circuit, filter circuit, and pixel position relationship mapping circuit.

[0060] Specifically, this recompression process can be divided into two processes: an upsampling process and a downsampling process. In the downsampling process, after obtaining compression parameters associated with the user's eye movement signal and a virtual rendered image to be processed from the display driving software and / or firmware computing platform, the recompression unit (i.e., the sixth software) can first divide the virtual rendered image into multiple sections according to the user's gaze position based on the compression parameters of the updated compression model, and determine the coordinate range of each section and the downsampling magnification of a non-attention section away from the gaze position. Then, the recompression unit (i.e., the sixth software) can obtain processing parameters for the downsampling operation from each corresponding memory based on the coordinates of each pixel in each section, and obtain pixel cache data required for the downsampling process from each corresponding first memory. The recompression unit (i.e., the sixth software) then sequentially inputs the acquired processing parameters and pixel cache data of each pixel to one or more of the above-mentioned weighting and summing circuits, average value calculation circuits, filter circuits, and pixel position relationship mapping circuits, and performs weighting and summing, average value calculation, filtering, and / or pixel position relationship mapping hardening calculations on the pixel cache data to obtain a first hardening calculation result based on the downsampling process.The recompression unit (i.e., the sixth software) then performs software operations such as data arrangement and allocation to obtain a downsampling-compressed image.

[0061] Furthermore, in the upsampling process, the recompression unit (i.e., the sixth software) can also determine an upsampling magnification for each attention section including / near the gaze position based on the compression parameters of the updated compression model after obtaining the compression parameters associated with the user's eye movement signal and the virtual rendering image to be processed. Then, the recompression unit (i.e., the sixth software) can obtain processing parameters for the upsampling operation from each corresponding memory based on the coordinates of each pixel in each section, and obtain pixel cache data required for the upsampling process from each corresponding first memory. Here, the pixel cache data includes, but is not limited to, cache data of at least one neighboring pixel of the current frame and cache data of at least one previous frame of the neighboring pixels. The recompression unit (i.e., the sixth software) then sequentially inputs the acquired processing parameters and pixel cache data for each pixel into one or more of the weighting and summing circuits, averaging circuits, filtering circuits, and pixel positional relationship mapping circuits, and performs hardening calculations (weighting and summing, averaging, filtering, and / or pixel positional relationship mapping) on ​​the pixel cache data to obtain a second hardening calculation result based on super-resolution processing. The recompression unit (i.e., the sixth software) then combines the second hardening calculation result with the first hardening calculation result, and performs software operations such as data placement and allocation to obtain a recompressed image that achieves a resolution equivalent to a 4K display with 40 pixels per degree in the focus area including / near the viewpoint position, and downsamples and compresses at least one non-focus area away from the viewpoint position. In this way, the present invention can further improve the equivalent resolution and real-time performance of an XR display device using limited software and hardware resources.

[0062] As described above, compared to the conventional technology in which a display driver IC (DDIC) on the screen side performs display compensation in a hardware module and a graphics processor (GPU) performs optical compensation in software, the present invention performs optical compensation and display compensation by combining a user's eye movement signal and motion sensing information with the display pipeline of the image processor 40, and improves data storage, processing, and transmission efficiency by installing and multiplexing an image processing hardening unit, thereby improving the image display quality of an XR display device with limited software and hardware resources. Furthermore, the present invention introduces multiple compensation algorithms, such as display compensation for distortion removal, uniformity correction, color separation removal, and / or color accuracy compensation, to the combination of a Micro-OLED display and a pancake lens optical component in an XR display device, and the combination of an LCD display and a pancake lens optical component in a 0D / 1D / 2D backlight module, providing a convenient solution for next-generation VR head-mounted display devices that use a Micro-OLED + pancake lens and an LCD + pancake lens as display optical solutions.

[0063] Furthermore, after obtaining a corrected image that has been subjected to optical correction and display compensation correction processing by the coprocessor 40, the coprocessor 40 may transmit the corrected image to the display terminal 50 to display an extended reality image with high quality.

[0064] Those skilled in the art can understand that the above-described system architecture of the extended reality display device employing the coprocessor 40 is merely a non-limiting embodiment provided by the present invention, which is intended to clearly illustrate the main concept of the present invention and provide a concrete scheme for facilitating public implementation.

[0065] In other embodiments, the image processor according to the first aspect of the present invention may be integrated into a main processor unit such as a central processing unit (CPU) or a graphics processing unit (GPU) of the extended reality display device according to the fourth aspect of the present invention via a software program and a hardware unit to achieve similar technical effects, so further details are omitted here.

[0066] For ease of explanation, the methods above are illustrated and described as a series of acts; however, it is to be understood and appreciated that these methods are not limited to the order of acts, as, according to one or more embodiments, some acts may occur in a different order and / or may occur concurrently with other acts not illustrated or described herein but which one of ordinary skill in the art would understand.

[0067] Those skilled in the art will appreciate that information, signals, and data may be represented using any of a variety of different technologies and techniques. For example, the data, instructions, commands, information, signals, bits, symbols, and chips referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof.

[0068] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been described generally in terms of their functional form. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints applied to the overall system. Engineers may implement the described functionality in different ways for each particular application, but such implementation decisions should not be interpreted as a departure from the scope of the present invention.

[0069] The various logic modules and circuits described in connection with the embodiments disclosed herein may be implemented or performed with a general-purpose processor, an NPU AI network model calculation acceleration processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0070] The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integrated into the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete structural elements in a user terminal.

[0071] In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functions may be stored on or transferred by one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media, including any medium that facilitates transfer of a computer program from one place to another. A storage medium may be any available medium that can be accessed by a computer. For example, and not as a limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Any connection is also properly termed a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair cable, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, the coaxial cable, fiber optic cable, twisted pair cable, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, disk and disc include compressed disks (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs; while disks often reproduce data magnetically, discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0072] The preceding description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications of the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other modifications without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein. [Explanation of symbols]

[0073] 10. Eye Tracker 20 sensors 30 Main Processor 40 Coprocessors 50 Display edge 60 cameras

Claims

1. 1. An image processor comprising a display pipeline integrated with a software processing unit and at least one image processing hardening unit, Acquire a user's eye movement signal and movement sensing information, and acquire a virtual rendering image to be processed; optically correcting the virtual rendering image by first software disposed in the software processing unit and the at least one image processing hardening unit based on the eye movement signal and the motion sensing information; and performing display compensation correction of the virtual rendering image by a second software disposed in the software processing unit and the at least one image processing hardening unit based on the eye movement signal and the motion sensing information. An image processor configured to:

2. the at least one image processing hardening unit comprises a first memory for storing pixel point data of a plurality of positions of a current frame and / or pixel point data of a plurality of positions of at least one previous frame; The step of performing the optical correction and / or the display compensation correction on the virtual rendering image includes: Obtaining pixel point data at a plurality of positions of a current frame and / or pixel point data at a plurality of positions of at least one previous frame from the first memory; performing the optical correction on the virtual rendering image by the first software and / or performing the display compensation correction on the virtual rendering image by the second software based on the eye movement signal, the motion sensing information, and pixel point data at multiple positions in the current frame and / or pixel point data at multiple positions in the at least one previous frame; 2. The image processor of claim 1, comprising:

3. the at least one image processing hardening unit further comprises at least one hardening calculation circuit; The step of performing the optical correction and / or the display compensation correction on the virtual rendering image includes: performing the optical correction on the virtually rendered image by the first software and the at least one hardening calculation circuit based on the eye movement signal, the motion sensing information, and pixel point data at multiple positions in the current frame and / or pixel point data at multiple positions in the at least one previous frame; and / or performing the display compensation correction on the virtually rendered image by the second software and the at least one hardening calculation circuit based on the eye movement signal, the motion sensing information, and pixel point data at multiple positions of the current frame and / or pixel point data at multiple positions of the at least one previous frame; The image processor of claim 2 further comprising:

4. The step of performing display compensation correction on the virtually rendered image by the second software and the at least one hardening calculation circuit based on the eye movement signal, the motion sensing information, and pixel point data at a plurality of positions of the current frame and / or pixel point data at a plurality of positions of the at least one past frame includes: determining a target pixel point based on the eye movement signal and the motion sensing information; obtaining raw data of the target pixel point and at least one associated pixel point from the first memory; performing a hardening calculation on the original data of the target pixel point and at least one associated pixel point by the at least one hardening calculation circuit; performing display compensation correction of the virtual rendering image by the second software based on the result of the hardening calculation, such as distortion removal, uniformity correction, color separation removal, and / or color accuracy compensation; 4. The image processor of claim 3, comprising:

5. the at least one image processing hardening unit further comprises a second memory for storing calibration data of an optical module, a display panel, and / or a camera; and / or wherein the step of performing the optical correction on the virtually rendered image further comprises: retrieving the calibration data from the second memory; and performing, by the first software and the at least one hardening calculation circuit, lens optical correction on the virtually rendered image in response to the calibration data; The step of performing the display compensation correction on the virtual rendering image includes the steps of obtaining the calibration data from the second memory, and performing the screen display compensation correction on the virtual rendering image according to the calibration data using the second software and the at least one hardening calculation circuit.

4. An image processor according to claim 3.

6. The step of performing display compensation correction on the virtual rendering image includes: acquiring a real scene image to be processed; and layer-blending the optically corrected virtual rendering image and the real scene image by a third software disposed in the software processing unit and the at least one image processing hardening unit based on the eye movement signal and the motion sensing information to obtain a mixed reality image. performing the display compensation correction on the mixed reality image by the second software and the at least one image processing hardening unit based on the eye movement signal and the motion sensing information; 2. The image processor of claim 1, comprising:

7. the step of layer-blending the optically corrected virtual rendering image and the real scene image by third software disposed in the software processing unit and the at least one image processing hardening unit based on the eye movement signal and the motion sensing information, acquiring interaction information between the real scene image and the virtual rendering image; performing the layer blending on the optically corrected virtual rendering image and the real scene image by the third software and the at least one image processing hardening unit based on the eye movement signal, the motion sensing information, and the interaction information; 7. The image processor of claim 6, comprising:

8. The display pipeline further comprises: color-enhancing the optically corrected virtual rendering image or the mixed reality image by a fourth software disposed in the software processing unit and the at least one image processing hardening unit based on the eye movement signal and the motion sensing information to obtain a color-enhanced image; The second software and the at least one image processing hardening unit perform the display compensation correction on the color-enhanced image based on the eye movement signal and the motion sensing information.

10. The image processor according to claim 1, wherein the image processor is configured to:

9. the at least one hardening calculation circuit further comprises at least one of a weighting addition circuit, an average calculation circuit, a filter circuit, and a pixel position relationship mapping circuit; The display pipeline further comprises: performing spatial distortion correction on the virtually rendered image by a fifth software located in the software processing unit and the at least one image processing hardening unit based on the eye movement signal; and / or and performing a recompression process on the virtual rendering image by a sixth software disposed in the software processing unit and the at least one image processing hardening unit based on the eye movement signal.

3. The image processor according to claim 1, wherein the image processor is configured to:

10. acquiring a user's eye movement signal and motion sensing information, and obtaining a virtual rendering image to be processed; optically correcting the virtual rendering image by first software disposed in the software processing unit and the at least one image processing hardening unit based on the eye movement signal and the motion sensing information; display-compensating and correcting the virtual rendering image by second software disposed in the software processing unit and the at least one image processing hardening unit based on the eye movement signal and the motion sensing information; An image processing method comprising:

11. A computer-readable storage medium having computer instructions stored thereon, The computer instructions, when executed by a processor, perform the image processing method of claim 10. A computer-readable storage medium comprising:

12. an eye tracker that collects eye movement signals from a user; a motion sensor for collecting motion sensing information of a user; a main processor that outputs a virtual rendering image of the object to be processed; an image processor according to any one of claims 1 to 10, connected to the eye tracker, the motion sensor, and the main processor, respectively, for acquiring the eye movement signal, the motion sensing information, and the virtual rendering image; a display terminal connected to the image processor, for acquiring and displaying a corrected image that has been optically corrected and displayed by the image processor; An extended reality display device comprising:

13. further comprising a camera coupled to the image processor; Acquire a real scene image to be processed via the camera; Based on the eye movement signal and the motion sensing information, a software processing unit disposed in the image processor and the at least one image processing hardening unit layer-blend the optically corrected virtual rendering image and the real scene image to obtain a mixed reality image; The software processing unit and the at least one image processing hardening unit perform display compensation correction on the mixed reality image based on the eye movement signal and the motion sensing information.

13. The extended reality display device according to claim 12, wherein the extended reality display device is configured to:

Images