Image processing system, method and storage medium based on the fovea principle

Abstract

An image processing system based on the foveal principle, comprising: a display pipeline that acquires a user's eye movement signal and performs first foveal processing based on the eye movement signal; and an image processing pipeline, wherein the display pipeline further generates a frame synchronization signal based on the eye movement signal and transmits the frame synchronization signal to the image processing pipeline so that the image processing pipeline performs second foveal processing based on the frame synchronization signal.

Description

[Technical Field]

[0001] This application claims priority to a patent application filed on August 26, 2022, with Chinese application number 202211032460.3 and titled "Image processing system, method and storage medium based on the foveal principle."

[0002] The present invention relates to an augmented reality display technology, and in particular to an image processing system based on the fovea principle, an image processing method based on the fovea principle, an augmented reality display device, and a computer-readable storage medium. [Background technology]

[0003] Extended reality (XR) display technology refers to the technology that combines reality and virtuality through computers to create a virtual environment that allows human-computer interaction, and includes, but is not limited to, augmented reality (AR), virtual reality (VR), and mixed reality (MR). By combining these three visual interaction technologies, augmented reality display technology can provide users with an immersive experience of seamless transition between the virtual and real worlds.

[0004] To meet the demand for high frame rate display in the XR field, conventional technologies generally use a display pipeline to perform image processing based on the foveal principle on virtual rendering images or composite images after image mixing. However, this technology lacks interaction and multiplexing with the image signal processing (ISP) pipeline, which can improve image processing quality and user perception, but has the disadvantages of high power consumption and long latency. If the foveal image processing operations are simply transferred to the display panel driver circuit (Display Drive Integrated Circuit, DDIC) of the ISP pipeline for processing, the ISP pipeline and DDIC lack the display pipeline processing capabilities based on the gaze point, and are unable to support complex and detailed compensation, resulting in limited effectiveness.

[0005] To overcome the above-mentioned deficiencies in the prior art, the field needs an augmented reality display technology that comprehensively improves image processing effects in various aspects, such as image quality, power consumption, real-time performance, and user sensation, thereby comprehensively remedying the current situation of insufficient computing power for XR high frame rate display. Summary of the Invention [Means for solving the problem]

[0006] The following presents a simplified summary of one or more aspects to provide a basic understanding of these aspects. This summary is not a detailed overview of every conceivable aspect, and is not intended to identify key or critical elements of every aspect, nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that is presented later.

[0007] In order to overcome the above-mentioned drawbacks of the prior art, the present invention provides an image processing system based on the fovea principle, an image processing method based on the fovea principle, an augmented reality display device, and a computer-readable storage medium, which comprehensively improves the image quality, power consumption, real-time performance, and user experience of image processing through the data linkage interaction and multiplexing processing architecture and processing flow of the display pipeline and image processing pipeline, thereby comprehensively improving the current situation in which the computing power for high frame rate XR display is insufficient.

[0008] Specifically, the image processing system based on the fovea principle provided by one aspect of the present invention includes a display pipeline and an image processing pipeline, wherein the display pipeline acquires a user's eye movement signal and performs first foveal processing based on the eye movement signal, and the display pipeline further generates a frame synchronization signal based on the eye movement signal and transmits the frame synchronization signal to the image processing pipeline so that the image processing pipeline performs second central concave processing based on the frame synchronization signal.

[0009] Furthermore, in some embodiments of the present invention, the frame synchronization signal comprises the eye movement signal, fixation point region data, and / or mesh matrix weight map data, where the fixation point region data indicates a coordinate position of a user's fixation point and / or its surrounding pixels with respect to an image, and the mesh matrix weight map data indicates the user's level of interest in multiple regions of the image.

[0010] Furthermore, in some embodiments of the present invention, the display pipeline is configured to identify the user's gaze point area data based on the eye movement signal, generate the mesh matrix weight map data based on the gaze point area data, perform the first foveation processing based on the mesh matrix weight map data, and generate the frame synchronization signal based on the eye movement signal, the gaze point area data, and / or the mesh matrix weight map data.

[0011] Furthermore, in some embodiments of the present invention, the display pipeline comprises a display hardening unit and a first software processing unit, the display hardening unit is connected to an eye tracker via the first software processing unit to acquire the eye movement signal, and performs the first foveation processing based on the eye movement signal, the first software processing unit is connected to the image processing pipeline and the display hardening unit, and synchronously aligns frame synchronization signals of the same frame image in the image processing pipeline and the display hardening unit.

[0012] Furthermore, in some embodiments of the present invention, the image processing pipeline includes an image processing hardening unit and a second software processing unit, the image processing hardening unit is connected to the display pipeline and receives the frame synchronization signal via the display pipeline to perform second foveation processing by at least one hardening calculation circuit based on the frame synchronization signal, and the second software processing unit is connected to the display pipeline and receives the frame synchronization signal via the display pipeline to perform second foveation processing by at least one software program based on the frame synchronization signal.

[0013] Furthermore, in some embodiments of the present invention, the image processing hardening unit includes a plurality of the hardening calculation circuits, at least one of which is provided with an independent switch, and the hardening calculation circuit performs an independent second foveation processing based on the switching signal of the corresponding switch.

[0014] Furthermore, in some embodiments of the present invention, the second software processing unit extracts image features of a current frame image for each frame and sets the switching signals of each of the switches in real time based on the image features, so as to control one or more of the hardening calculation circuits in the image processing hardening unit to perform the second foveation processing for each frame, or the switches of each of the hardening calculation circuits in the image processing hardening unit are preset to corresponding switch states based on the image processing function of the image processing system, so as to fixedly control one or more of the hardening calculation circuits in the image processing hardening unit to perform the second foveation processing.

[0015] Furthermore, in some embodiments of the present invention, the display pipeline is further connected to an image rendering module, acquires a virtual rendered image via the image rendering module, and performs the first foveation processing on the virtual rendered image based on the eye movement signal. The image processing pipeline is further connected to a camera, acquires a real scene image via the camera, and performs the second foveation processing on the real scene image based on the frame synchronization signal. The image processing pipeline further transmits the second foveated real scene image to the display pipeline so as to be layer stacked with the first foveated or unfoveated virtual rendered image.

[0016] In addition, the image processing method based on the foveal principle provided by the second aspect of the present invention includes the steps of acquiring a user's eye movement signal, transmitting the eye movement signal to a display pipeline of an image processing system so as to perform first foveal processing and generating a frame synchronization signal, and transmitting the frame synchronization signal to the image processing pipeline of the image processing system so that the image processing pipeline performs second foveal processing based on the frame synchronization signal.

[0017] Furthermore, in some embodiments of the present invention, the step of transmitting the eye movement signal to a display pipeline of an image processing system to perform the first foveal processing and generating a frame synchronization signal comprises the steps of identifying the user's gaze point based on the eye movement signal, generating the mesh matrix weight map data based on the gaze point, performing the first foveal processing based on the mesh matrix weight map data, and generating the frame synchronization signal based on the eye movement signal and / or the mesh matrix weight map data.

[0018] Furthermore, in some embodiments of the present invention, the image processing pipeline includes an image processing hardening unit and a software processing unit, and at least one hardening calculation circuit is disposed in the image processing hardening unit. The step of transmitting the frame synchronization signal to the image processing pipeline of the image processing system so that the image processing pipeline performs second foveation processing based on the frame synchronization signal includes the steps of obtaining a switching signal via the image processing hardening unit and the software processing unit, identifying a hardening calculation circuit and / or software processing unit that requires the second foveation processing based on the switching signal, and transmitting the frame synchronization signal to the at least one hardening calculation circuit and / or software processing unit that requires the second foveation processing so that the at least one hardening calculation circuit and / or software processing unit independently performs the second foveation processing.

[0019] Furthermore, in some embodiments of the present invention, the image processing hardening unit is provided with a plurality of the hardening calculation circuits. The step of identifying the hardening calculation circuit and / or software processing unit requiring the second foveal processing based on the switching signal comprises: extracting image features of a current frame image for each frame via the software processing unit and setting switching signals of each of the hardening calculation circuits in real time based on the image features, in order to control one or more of the hardening calculation circuits in the image processing hardening unit to perform the second foveal processing for each frame; or presetting the switching states of each of the hardening calculation circuits in the image processing hardening unit based on the image processing function of the image processing system, and fixedly controlling one or more of the hardening calculation circuits in the image processing hardening unit to perform the second foveal processing based on the switching states of each of the hardening calculation circuits.

[0020] Furthermore, in some embodiments of the present invention, the display pipeline is further connected to an image rendering module, acquires a virtual rendered image via the image rendering module, and performs the first foveation processing on the virtual rendered image based on the eye movement signal. The image processing pipeline is further connected to a camera, acquires a real scene image via the camera, and performs the second foveation processing on the real scene image based on the frame synchronization signal. After the second foveation processing is performed, the image processing method further includes transmitting the second foveated real scene image from the image processing pipeline to the display pipeline so as to be layer stacked with the first virtual rendered image that may or may not have been foveated.

[0021] The augmented reality display device provided in a third aspect of the present invention includes the image processing system provided in the first aspect of the present invention.

[0022] The fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer commands which, when executed by a processor, implement the image processing method provided in the second aspect of the present invention.

[0023] The above features and advantages of the present invention can be better understood after reading the detailed description of the embodiments of the present disclosure in conjunction with the following drawings, in which components are not necessarily drawn to scale, and components having similar interrelated properties or characteristics may have the same or similar reference numerals. [Brief explanation of the drawings]

[0024] [Figure 1] 1 shows a schematic architecture diagram of an image processing system provided in accordance with some embodiments of the present invention; [Figure 2] 1 shows a flow diagram of an image processing method provided by some embodiments of the present invention; [Figure 3] 1 shows a schematic diagram of a mesh matrix weight map provided in accordance with some embodiments of the present invention; [Figure 4] 1 shows a schematic architecture diagram of an image processing pipeline provided by some embodiments of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0025] Hereinafter, embodiments of the present invention will be described with reference to specific examples, and those skilled in the art will be able to easily understand other advantages and effects of the present invention based on the contents disclosed herein. Although the present invention will be described with reference to a preferred embodiment, this does not mean that the features of the present invention are limited to this embodiment. On the contrary, the purpose of introducing the invention in connection with the embodiment is to cover other options or modifications that may extend based on the claims of the present invention. In order to provide a deep understanding of the present invention, the following description includes many specific details. The present invention can also be practiced without these details. Also, some specific details are omitted from the description so as not to confuse or obscure the gist of the present invention.

[0026] In the description of the present invention, unless otherwise clearly specified and limited, the terms "attach," "contact," and "connect" should be understood in a broad sense, and may refer to, for example, a fixed connection, a detachable connection, an integral connection, a mechanical connection, an electrical connection, a direct connection, an indirect connection via an intermediate medium, or internal communication between two elements. Those skilled in the art can specifically understand the specific meaning of the above terms in the present invention.

[0027] Additionally, the terms "upper," "lower," "left," "right," "ceiling," "bottom," "horizontal," and "vertical" used in the following description should be understood as referring to the orientations depicted in the segments and associated drawings. These relative terms are merely for convenience of explanation and should not be understood as a limitation on the present invention, as they do not imply that the described devices must be manufactured or operated in any particular orientation.

[0028] Although the terms "first," "second," "third," etc. may be used herein to describe various components, regions, layers, and / or portions, it will be understood that these components, regions, layers, and / or portions should not be limited by these terms, but are merely used to distinguish between different components, regions, layers, and / or portions. Thus, a first component, region, layer, and / or portion discussed below could be referred to as a second component, region, layer, and / or portion without departing from some embodiments of the present invention.

[0029] As described above, to meet the demand for high frame rate display in the XR field, conventional technologies generally use a display pipeline to perform image processing based on the foveal principle on virtual rendering images or composite images after image mixing. However, this technology lacks interaction and multiplexing with the image signal processing (ISP) pipeline, which can improve image processing quality and user perception, but has the disadvantages of high power consumption and long latency. If the foveal image processing operations are simply transferred to the display drive integrated circuit (DDIC) of the ISP pipeline for processing, the ISP pipeline and DDIC lack the display pipeline processing capabilities based on the gaze point, and are unable to support complex and detailed compensation, resulting in limited effectiveness.

[0030] In order to overcome the above-mentioned drawbacks of the prior art, the present invention provides an image processing system based on the fovea principle, an image processing method based on the fovea principle, an augmented reality display device, and a computer-readable storage medium, which comprehensively improves the image quality, power consumption, real-time performance, and user experience of image processing through the data linkage interaction and multiplexing processing architecture and processing flow of the display pipeline and image processing pipeline, thereby comprehensively improving the current situation in which the computing power for high frame rate XR display is insufficient.

[0031] In some non-limiting examples, the image processing method provided by the second aspect of the present invention can be implemented via the image processing system provided by the first aspect of the present invention. Specifically, the image processing system includes a memory and a processor. The memory includes, but is not limited to, the computer-readable storage medium provided by the fourth aspect of the present invention, and has computer commands stored therein. The processor is coupled to the memory and configured to execute the computer commands stored in the memory to implement the image processing method provided by the second aspect of the present invention.

[0032] Furthermore, in some embodiments, the image processing system provided by the first aspect of the present invention can be arranged in the augmented reality display device provided by the third aspect of the present invention to multiplex image processing data of the display pipeline and / or the image processing pipeline when hardware resources are limited, thereby saving hardware computing power, ensuring the central imaging effect, and improving the hardware resource utilization of the augmented reality display device.

[0033] The operating principles of the image processing system and the augmented reality display device will be described below in relation to several examples of image processing methods. Those skilled in the art will understand that these image processing methods are merely a few non-limiting embodiments provided by the present invention, are intended to clearly illustrate the main concept of the present invention, and are intended to provide several specific methods that are generally easy to implement, and are not intended to limit all functions or all operations of the image processing system and the augmented reality display device. Similarly, these image processing systems and the augmented reality display device are merely a few non-limiting embodiments provided by the present invention, and do not limit the entities that perform each step in these image processing methods.

[0034] Please refer to Figures 1 and 2. Figure 1 shows a schematic architecture diagram of an image processing system provided by some embodiments of the present invention. Figure 2 shows a schematic flow diagram of an image processing method provided by some embodiments of the present invention.

[0035] 1 and 2, in some embodiments of the present invention, an image processing system 10 may preferably include a display pipeline 11 and an image processing pipeline (ISP pipeline) 12. The display pipeline 11 is connected to an external device, such as an eye tracker 20, to acquire a user's eye movement signal and performs first foveated processing on a local first image based on the acquired eye movement signal. At the same time, the display pipeline 11 is also connected to the image processing pipeline 12 and is further configured to generate a frame synchronization signal based on the acquired eye movement signal and transmit the frame synchronization signal to the image processing pipeline 12, so that the image processing pipeline 12 performs second foveated processing on a local second image based on the frame synchronization signal.

[0036] 1, the image processing system 10 may further include at least one software module, such as a mesh grid generation module 13. The mesh grid generation module 13 may be selected from at least one of a digital signal processor (DSP), a microcontroller unit (MCU), and a microprocessor, and may be used to generate mesh grid weight map data based on input eye movement signals and / or perform software calculations and / or processes on input signals and / or data. The display pipeline 11 may be connected to the eye tracker 20 via the mesh grid generation module 13 to acquire fovea-related data, such as gaze position and mesh grid weight map data, and may further perform first foveation processing on the local first image based on the acquired fovea-related data.

[0037] 1, the display pipeline 11 may preferably include a display hardening unit 111 and a first software processing unit 112. The display hardening unit 111 includes one or more transistor-level hardening calculation circuits and / or memories for performing hardening calculation processing and storing input signals. The first software processing unit 112 is selected from at least one of a digital signal processor (DSP), a microcontroller (MCU), and a microprocessor, and is mainly used to perform software calculations and / or software processing on input signals and / or data, such as calculation of gain values ​​when adjusting pixel brightness, calculation of pixel mapping logical relationships, location information of brightness relative relationships between the three RGB colors, and calculation of the format of called image gamma data, for each hardening calculation circuit and / or memory location register and data processing parameters in the display hardening unit 111.

[0038] Furthermore, in some embodiments, the mesh matrix generation module 13 may be disposed in the first software processing unit 112 to similarly achieve the effect of generating first mesh matrix weight map data based on an input eye movement signal. Here, the display hardening unit 111 is connected to the external eye tracker 20 via the first software processing unit 112 to acquire the eye movement signal and can perform first foveation processing based on the eye movement signal. In addition, the first software processing unit 112 is connected to the image processing pipeline 12 and the display hardening unit 111, respectively, and can process the frame synchronization signals of the same frame images in the image processing pipeline 12 and the display hardening unit 111 to synchronously align them.

[0039] Specifically, the mesh matrix generation module 13 or the first software processing unit 112 having the mesh matrix generation program installed therein can first identify the user's gaze area data based on the eye movement signal received from the eye tracker 20, in response to receiving the eye movement signal indicating information such as the user's eyeball deflection angle, gaze direction, and gaze coordinates, and then generate first mesh matrix weight map data based on the gaze area data. Here, the gaze area data indicates the coordinate positions of the user's gaze point on the image and / or one or more pixels surrounding the gaze point. The first mesh matrix weight map data indicates the user's level of interest in each of multiple regions of the image.

[0040] Please further refer to FIG. 3, which shows a schematic diagram of a mesh matrix weight map provided by some embodiments of the present invention.

[0041] As shown in FIG. 3, in the mesh matrix weight map, the user's gaze point may have the highest weight (e.g., 3), several neighboring pixel points nearby may have higher weights (e.g., 2), and several edge pixel points farther from the user's gaze point may have lower weights (e.g., 1). The weight of each pixel point may gradually decrease uniformly or unevenly from the gaze point, with pixel points close to the gaze point having higher weights and a relatively dense mesh. Conversely, pixel points farther from the gaze point have lower weights and a relatively sparse mesh. In this way, the display hardening unit 111 can perform a first foveation process on the local first image based on this first mesh matrix weight map data. For example, all pixel points with a weight of 3 may be processed one by one. Alternatively, for example, one of N pixel points close to a pixel point with a weight of 2 may be processed, and the other N-1 pixel points may be compensated for by interpolation, where N is a multiple of 1 or 4. Also, for example, one of 4N pixel points nearest to a pixel point with a weight of 1 is processed, and the other 4N-1 pixel points are compensated for by interpolation, where N is 1 or a multiple of 4. The present invention performs first foveation processing on a local first image in the display pipeline 11 to reduce the data amount of the first image, thereby improving the quality of image processing and the user's sensation, and improving the hardware resource utilization rate of the image processing system 10, on the premise of ensuring the central imaging effect of the user's gaze area.

[0042] The first software processing unit 112 may also generate a frame synchronization signal based on the acquired eye movement signal, the identified fixation point region data, and / or the generated first mesh matrix weight map data, and transmit the frame synchronization signal to the image processing pipeline 12 so that the image processing pipeline 12 performs second foveation processing on the local second image based on the frame synchronization signal. Taking the eye movement signal as an example of synchronously aligning the eye movement signal, the first software processing unit 112 may add a timestamp to the acquired eye movement signal and transmit the eye movement signal with the added timestamp to the image processing pipeline 12 to generate the frame synchronization signal.

[0043] 1, the image processing pipeline 12 may preferably include an image processing hardening unit 121 and a second software processing unit 122. The image processing hardening unit 121 includes one or more transistor-level hardening calculation circuits and / or memories for performing hardening calculations on and storing input signals. The second software processing unit 122 is selected from at least one of a digital signal processor (DSP), a microcontroller (MCU), and a microprocessor, and has at least one software program, such as an auto exposure (AE), auto white balance (AWB), auto focus (AF), anti-flickering, auto lens shade correction (auto lens shade correction) (hereinafter referred to as the 5A module of the ISP), and a mesh matrix generation module, configured to generate second mesh matrix weight map data based on an input frame synchronization signal, and / or to perform software calculations and / or processing used for the foveal characteristics of the 5A module, such as mesh setting for statistically analyzing image information related to auto exposure and / or auto white balance, and adjusting the algorithm logic for applying statistical information, for input signals and / or data, such as the location registers and data processing parameters of each hardening calculation circuit and / or memory in the image processing hardening unit 121.

[0044] Specifically, the second software processing unit 122 is connected to the first software processing unit 112 of the display pipeline 11, and can acquire a frame synchronization signal, such as an eye movement signal with a timestamp added, and / or foveal data, such as first mesh matrix weight map data with a timestamp added, via the first software processing unit 112. The second software processing unit 122 can then generate second mesh matrix weight map data based on the eye movement signal, as described above, and perform second foveal processing on a second image of a local corresponding frame based on the timestamp and the generated second mesh matrix weight map data using at least one software program (e.g., a 5A module) therein. In some embodiments, the resolution and weights of the second mesh matrix weight map data generated by the second software processing unit 122 can be set based on parameters such as the resolution of the second image and can be different from the first mesh matrix weight map data generated by the first software processing unit 112, thereby better adapting to the processing needs of the second image and improving the image processing quality of the image processing pipeline 12.

[0045] In addition, the image processing hardening unit 121 can be directly connected to the first software processing unit 112 of the display pipeline 11 or can be connected to the first software processing unit 112 via the second software processing unit 122, so as to receive a frame synchronization signal directly or indirectly via the display pipeline 11. Here, the frame synchronization signal may be first mesh matrix weight map data generated by the first software processing unit 112, or second mesh matrix weight map data generated by the second software processing unit 122. Each hardening calculation circuit in the image processing hardening unit 121 can perform second foveal processing by the hardening calculation circuit based on the received first and / or second mesh matrix weight map data, respectively, to reduce the data volume of the second image, thereby reducing the power consumption and delay of the image processing system and improving the hardware resource utilization of the image processing system 10, while ensuring the central imaging effect of the user's gaze area.

[0046] Furthermore, by using the first mesh matrix weight map data as a frame synchronization signal, the image processing hardening unit 121 can directly use the acquired first mesh matrix weight map data to perform the second foveation processing, thereby further saving software computing power, saving hardware resources of the hardening calculation circuit, and improving the real-time performance of image processing. In particular, in a mixed reality (MR) display embodiment, by selecting and synchronizing the foveation processing of the virtual rendering image (i.e., the first image) and the real scene image (i.e., the second image) using the same mesh matrix weight map data, the present invention effectively improves the compatibility between the virtual object and the ISP real scene, ensuring that the over-resolution of both is similar, thereby achieving natural over-resolution of the virtual rendering image and the real scene image.

[0047] Please refer further to FIG. 4, which shows a schematic architecture diagram of an image processing pipeline provided by some embodiments of the present invention.

[0048] 4, the image processing hardening unit 121 may preferably include one or more transistor-level hardening calculation circuits, such as defective pixel correction (DPCC), demosaic (Demosaic), black level correction (BLC), automatic lens shadow correction (LSC), noise reduction (NR), sharpening (SHP), color calibration (CCM), gamma correction, and color space conversion (CSC). In addition, the second software processing unit 122 and / or at least one hardening calculation circuit may each be provided with an independent switch. Then, in response to a locally input second image, the image processing pipeline 12 receives switching signals from the image processing hardening unit 121 and the software processing unit 122, respectively, and can identify the hardening calculation circuit and / or software processing unit 122 that require second foveation processing based on the switching signals. Thereafter, when the switching signal of the software processing unit 122 and / or any one or more hardening calculation circuits is ON, the image processing pipeline 12 can determine that a second foveation processing needs to be performed, and thereby a frame synchronization signal is transmitted to at least one hardening calculation circuit and / or the software processing unit 122 to independently perform the second foveation processing. Conversely, when the switching signal of the software processing unit 122 and / or any one or more hardening calculation circuits is OFF, the image processing pipeline 12 can determine that a second foveation processing does not need to be performed, and thereby skip the operation of this second foveation processing, thereby providing a personalized foveation processing function.

[0049] In some embodiments, the switching signal may be preset based on specific functions of the image processing system 10 before the image processing system 10 is shipped or before the device is put into operation. In this way, the image processing system 10 can be fixedly controlled so that one or more hardening calculation circuits in the image processing hardening unit 121 perform preset or customized second foveation processing on a local second image based on a preset or customized processing scheme.

[0050] Furthermore, in some preferred embodiments, the switching signal may be set for each frame by the second software processing unit 122. Specifically, in the XR display process, the image processing pipeline 12 can use a pre-trained artificial intelligence (AI) model to extract image features of the second image of the current frame for each frame, and determine whether a foveal processing operation is required and specific information, such as the mesh matrix, that needs to be used. Then, based on the determination result that a foveal processing operation is required for the second image of the current frame, the image processing pipeline 12 can set the switching signal of each hardening calculation circuit in real time and synchronously obtain and / or generate specific information regarding the foveal processing operation, such as the mesh matrix. In this way, each hardening calculation circuit needs to perform personalized foveal processing for the second image of the current frame based on the switching signal set in real time by the second software processing unit 122. In this way, the image processing pipeline 12 can control one or more hardening calculation circuits in the image processing hardening unit 121 for each frame, thereby dynamically performing personalized second foveal processing for the second image of each frame.

[0051] Those skilled in the art can understand that the architectures of the display pipeline 11 and the image processing pipeline 12, which are respectively configured as independent software calculation units 112 and 122, shown in FIG. 1, are merely provided as non-limiting embodiments for the purpose of clearly illustrating the main concept of the present invention and providing a concrete method for facilitating public implementation, and do not limit the scope of protection of the present invention.

[0052] Alternatively, in another embodiment, the display pipeline and the image processing pipeline can each have a different hardening unit and share the same software processing unit. In this way, this software processing unit is connected to the display hardening unit and the image processing hardening unit, respectively, and can synchronously transmit eye movement signals, frame synchronization signals such as mesh matrix weight map data, and / or foveal data between the display hardening unit and the image processing hardening unit, thereby realizing data cooperation interaction and multiplexing between the display pipeline and the image processing pipeline. The specific operation of this software processing unit is similar to that of the first software processing unit and the second software processing unit, and will not be further described here.

[0053] Also, please continue to refer to FIG. 1. In some mixed reality (MR) display embodiments, the display pipeline 11 is further connected to an image rendering module 30, such as a central processing unit (CPU) or a graphics processing unit (GPU) of the mixed reality display device, to acquire a virtual rendered image through the image rendering module 30 and perform a first foveated processing on the virtual rendered image based on eye movement signals. The image processing pipeline 12 is also connected to an image acquisition device, such as a binocular camera 40, to acquire a real scene image through the binocular camera 40 and perform a second foveated processing on the real scene image based on a frame synchronization signal provided by the display pipeline 11. Here, since there is usually a large difference in resolution between the virtual rendered image and the real scene image, the first mesh matrix weight map for performing the first foveated processing and the second mesh matrix weight map for performing the second foveated processing may be different, which relates to the large difference in resolution.

[0054] Furthermore, after performing the second foveation processing, the image processing pipeline 12 can add a timestamp to the real scene image that has undergone the second foveation processing so that it can be layer stacked by the display pipeline 11, and then synchronously transmit the image from the image processing pipeline 12 to the display pipeline 11, and output it to the subsequent module 14 for mixed reality output display. In this way, the present invention can further reduce the amount of data of the real scene image transmitted between pipelines, thereby saving the computing power of the hardware.

[0055] Furthermore, the display pipeline 11 can find a virtual rendering image of a corresponding frame based on the timestamp of the real scene image, and perform a layer stack and mixed reality output display together with the real scene image. Here, the virtual rendering image may be a compressed image that has undergone first foveation processing, and after completing the layer stack, the display pipeline 11 can send the layer stack image directly to the subsequent module 14 to perform a mixed reality output display.

[0056] Alternatively, in some other embodiments, the virtual rendered image may be an original virtual rendered image without the first foveal processing. In this way, after completing the layer stack, the display pipeline 11 can perform the first foveal processing on the layer stack image based on the user's eye movement signal, and then send it to the subsequent module 14 for mixed reality output display.

[0057] As described above, the image processing system, image processing method, augmented reality display device, and computer-readable storage medium provided by the present invention comprehensively improve the image quality, power consumption, real-time performance, and user experience of image processing through the data linkage interaction and multiplexing processing architecture and processing flow between the display pipeline and the image processing pipeline, thereby comprehensively improving the current situation in which the computing power for XR high frame rate display is insufficient.

[0058] Although the methods described above have been illustrated and described as a series of acts for ease of interpretation, it should be understood that these methods are not limited by the order of acts, as, depending on one or more embodiments, some acts may occur in a different order and / or concurrently with other acts shown and described herein or not shown and described herein but which one of ordinary skill in the art would understand.

[0059] Those skilled in the art should understand that information, signals, and data may be represented using any of a variety of different technologies and techniques. For example, the data, instructions, commands, information, signals, bits, symbols, and chips described above may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0060] Those skilled in the art should further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in their functional form. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Those skilled in the art may employ different methods to implement the described functionality for various particular applications, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

[0061] The various example logic modules and circuits described in connection with the embodiments disclosed herein may be implemented with or performed by a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in cooperation with a DSP core, or any other such configuration.

[0062] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processing system comprising: a display pipeline that acquires a user's eye movement signal and performs first foveation processing based on the eye movement signal; and an image processing pipeline, The display pipeline further generates a frame synchronization signal based on the eye movement signal and transmits the frame synchronization signal to the image processing pipeline so that the image processing pipeline performs second foveal processing based on the frame synchronization signal.

2. The image processing system of claim 1, characterized in that the frame synchronization signal includes the eye movement signal, gaze point area data indicating the coordinate position of the user's gaze point and / or its surrounding pixels relative to the image, and / or mesh matrix weight map data indicating the user's degree of interest in multiple areas of the image.

3. The display pipeline: Identifying fixation point area data of the user based on the eye movement signal; generating the mesh matrix weight map data based on the fixation point region data; performing the first foveation processing based on the mesh matrix weight map data; 3. An image processing system according to claim 2, arranged to generate the frame synchronization signal based on the eye movement signal, the fixation point area data, and / or the mesh matrix weight map data.

4. the display pipeline comprises: a first software processing unit; and a display hardening unit connected to an eye tracker via the first software processing unit to acquire the eye movement signal, and performing the first foveation processing based on the eye movement signal; 2. The image processing system of claim 1, wherein the first software processing unit is connected to the image processing pipeline and the display hardening unit and synchronously aligns frame synchronization signals of identical frame images in the image processing pipeline and the display hardening unit.

5. the image processing pipeline comprises a second software processing unit; and an image processing hardening unit connected to the display pipeline, the image processing hardening unit receiving the frame synchronization signal through the display pipeline to perform second foveation processing by at least one hardening calculation circuit based on the frame synchronization signal; 2. The image processing system of claim 1, wherein the second software processing unit is connected to the display pipeline and receives the frame synchronization signal through the display pipeline so as to perform second foveation processing by at least one software program based on the frame synchronization signal.

6. 6. The image processing system according to claim 5, wherein the image processing hardening unit includes a plurality of the hardening calculation circuits, at least one of the hardening calculation circuits includes an independent switch, and the hardening calculation circuit performs an independent second foveal processing based on a switching signal of the corresponding switch.

7. The second software processing unit extracts image features of a current frame image for each frame and arranges switching signals of each of the switches in real time based on the image features, so as to control the one or more hardening calculation circuits in the image processing hardening unit to perform the second foveation processing for each frame; 7. The image processing system of claim 6, wherein the switches of each of the hardening calculation circuits in the image processing hardening unit are preset to corresponding switch states based on the image processing function of the image processing system, so that one or more of the hardening calculation circuits in the image processing hardening unit are fixedly controlled to perform the second foveation processing.

8. the display pipeline is further connected to an image rendering module, and receives a virtual rendered image through the image rendering module; and further performs the first foveation processing on the virtual rendered image based on the eye movement signal; the image processing pipeline is further connected to a camera, acquires a real scene image via the camera, and performs the second foveation processing on the real scene image based on the frame synchronization signal; 2. The image processing system of claim 1, wherein the image processing pipeline further transmits the second foveated real scene image to the display pipeline so that it is layer stacked with the first foveated or unfoveated virtual rendering image.

9. acquiring an eye movement signal of a user; transmitting the eye movement signal to a display pipeline of an image processing system to perform a first foveation process and generate a frame synchronization signal; and transmitting the frame synchronization signal to the image processing pipeline of the image processing system so that the image processing pipeline performs second foveation processing based on the frame synchronization signal.

10. transmitting the eye movement signal to a display pipeline of an image processing system to perform the first foveation processing and generate a frame synchronization signal; identifying a fixation point of the user based on the eye movement signal; generating the mesh matrix weight map data based on the fixation point; performing the first foveation processing based on the mesh matrix weight map data; 10. The image processing method according to claim 9, further comprising the step of generating the frame synchronization signal based on the eye movement signal and / or the mesh matrix weight map data.

11. the image processing pipeline comprises an image processing hardening unit and a software processing unit, and at least one hardening calculation circuit is disposed in the image processing hardening unit; and the step of transmitting the frame synchronization signal to the image processing pipeline of the image processing system so that the image processing pipeline performs second foveation processing based on the frame synchronization signal comprises: obtaining a switching signal through the image processing hardening unit and the software processing unit; identifying a hardening calculation circuit and / or software processing unit that requires the second foveation processing based on the switching signal; and transmitting the frame synchronization signal to at least one hardening calculation circuit and / or software processing unit that requires the second foveation processing so that the at least one hardening calculation circuit and / or software processing unit performs the second foveation processing independently.

12. The image processing hardening unit is provided with a plurality of the hardening calculation circuits, and the step of identifying the hardening calculation circuit and / or software processing unit that requires the second foveal processing based on the switching signal includes: extracting image features of a current frame image for each frame via the software processing unit and arranging switching signals of each of the hardening calculation circuits in real time based on the image features, so as to control one or more of the hardening calculation circuits in the image processing hardening unit to perform the second foveation processing for each frame; or 12. The image processing method according to claim 11, further comprising a step of: setting in advance the switching states of each of the hardening calculation circuits in the image processing hardening unit based on the image processing function of the image processing system; and fixedly controlling one or more of the hardening calculation circuits in the image processing hardening unit to perform the second foveation processing based on the switching states of each of the hardening calculation circuits.

13. the display pipeline is further connected to an image rendering module, acquires a virtual rendered image through the image rendering module, and performs the first foveation processing on the virtual rendered image based on the eye movement signal; the image processing pipeline is further connected to a camera, acquires a real scene image through the camera, and performs the second foveation processing on the real scene image based on the frame synchronization signal; and after performing the second foveation processing, the image processing method includes:

10. The image processing system of claim 9, further comprising transmitting the second foveated real scene image from the image processing pipeline to the display pipeline so as to be layer stacked with the first foveated or unfoveated virtual rendering image.

14. An augmented reality display device comprising the image processing system according to any one of claims 1 to 8.

15. 14. A computer readable storage medium having stored thereon computer commands which, when executed by a processor, perform an image processing method according to any one of claims 9 to 13.

Images