Image processing system, method and storage medium based on fovea principle
In extended reality display, using the data interaction between the display pipeline and the image processing pipeline and eye movement signal processing, the image quality, power consumption and real-time improvement are achieved, and the problem of insufficient image processing in the prior art is solved, and the improvement is improved. User experience and hardware resource utilization.
Patent Information
- Application Number
- CN202211032460.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-26
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-08-26
AI Technical Summary
In the extended reality display, the image processing quality, power consumption and real-time performance are insufficient, especially under the demand for high frame rate display, the lack of effective joint interaction between image signal processing and display pipelines, resulting in poor user experience and Computing power is short.
By establishing a processing architecture for joint data interaction and multiplexing between the display pipeline and the image processing pipeline, the user's eye movement signal is used to perform foveal processing, generate frame synchronization signals, realize the first and second foveal processing, and optimize image processing Harden the switching control of the computing circuit and software processing unit to improve image quality and hardware resource utilization.
It significantly improves the image quality, power consumption and real-time performance of image processing, improves user experience, and effectively solves the problem of insufficient computing power for XR high frame rate display.
Smart Images

Figure CN117670640B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an extended reality display technology, and in particular to an image processing system based on the fovea principle, an image processing method based on the fovea principle, an extended reality display device, and a computer-readable storage medium. Background Art
[0002] Extended Reality (XR) display technology refers to the technology that combines the real and the virtual through computers to create a virtual environment for human-computer interaction, including but not limited to Augmented Reality (AR) display technology, Virtual Reality (VR) display technology, and Mixed Reality (MR) display technology. By integrating these three visual interaction technologies, extended reality display technology can bring the experiencer an immersive feeling of seamless transition between the virtual world and the real world.
[0003] In response to the high frame rate display requirements in the XR field, the existing technology generally uses the display pipeline to perform image processing based on the fovea principle on the virtual rendered image or the composite image after image aliasing, but lacks joint interaction and multiplexing with the image signal processing (ISP) pipeline. Although it can improve the image processing quality and user experience, it has the defects of high power consumption and large delay. If the fovea image processing operation is simply transferred to the display panel drive circuit (DDIC) of the ISP pipeline for processing, there is a problem that both the ISP pipeline and the DDIC lack the display pipeline processing capability based on the gaze point, and cannot support complex and sophisticated compensation and the effect is limited.
[0004] In order to overcome the above-mentioned defects of the existing technology, this field is in urgent need of an extended reality display technology to comprehensively improve the image processing effect in terms of image quality, power consumption, real-time performance, user experience, etc., so as to comprehensively improve the current shortage of XR high frame rate display computing power. Summary of the invention
[0005] A brief summary of one or more aspects is given below to provide a basic understanding of these aspects. This summary is not an exhaustive overview of all conceived aspects, and is neither intended to identify the key or decisive elements of all aspects nor to define the scope of any or all aspects. Its only purpose is to give some concepts of one or more aspects in a simplified form as a prelude to a more detailed description that will be given later.
[0006] In order to overcome the above-mentioned defects of the prior art, the present invention provides an image processing system based on the fovea principle, an image processing method based on the fovea principle, an extended reality display device, and a computer-readable storage medium, which can comprehensively improve the image quality, power consumption, real-time performance and user experience of image processing through a processing architecture and processing flow of data joint interaction and reuse of the display pipeline and the image processing pipeline, thereby comprehensively improving the current shortage of XR high frame rate display computing power.
[0007] Specifically, the image processing system based on the fovea principle provided according to the first aspect of the present invention includes a display pipeline and an image processing pipeline. The display pipeline obtains an eye movement signal of a user and performs a first fovea processing according to the eye movement signal. The display pipeline also generates a frame synchronization signal according to the eye movement signal and transmits the frame synchronization signal to the image processing pipeline, so that the image processing pipeline performs a second fovea processing according to the frame synchronization signal.
[0008] Further, in some embodiments of the present invention, the frame synchronization signal includes the eye movement signal, gaze point area data and / or grid matrix weight map data. The gaze point area data indicates the coordinate position of the user's gaze point on the image and / or its surrounding pixels. The grid matrix weight map data indicates the user's attention to multiple areas of the image.
[0009] Further, in some embodiments of the present invention, the display pipeline is configured to: determine the user's gaze point area data based on the eye movement signal; generate the grid matrix weight map data based on the gaze point area data; perform the first fovea processing based on the grid matrix weight map data; and generate the frame synchronization signal based on the eye movement signal, the gaze point area data and / or the grid matrix weight map data.
[0010] Further, in some embodiments of the present invention, the display pipeline includes a display hardening unit and a first software processing unit. The display hardening unit is connected to an eye tracker via the first software processing unit to obtain the eye movement signal, and performs the first foveal processing according to the eye movement signal. The first software processing unit is connected to the image processing pipeline and the display hardening unit, and synchronously aligns the frame synchronization signals of the same frame image in the image processing pipeline and the display hardening unit.
[0011] Further, in some embodiments of the present invention, the image processing pipeline includes an image processing hardening unit and a second software processing unit. The image processing hardening unit is connected to the display pipeline and obtains the frame synchronization signal via the display pipeline to perform a second fovea processing based on at least one hardened calculation circuit according to the frame synchronization signal. The second software processing unit is connected to the display pipeline and obtains the frame synchronization signal via the display pipeline to perform a second fovea processing based on at least one software program according to the frame synchronization signal.
[0012] Furthermore, in some embodiments of the present invention, the image processing hardening unit is configured with a plurality of the hardening calculation circuits, at least one of which is configured with an independent switch, and the hardening calculation circuit performs independent second fovea processing according to a switch signal of a corresponding switch.
[0013] Furthermore, in some embodiments of the present invention, the second software processing unit extracts image features of the current frame image frame by frame, and configures the switch signals of each of the switches in real time according to the image features, so as to control one or more of the hardening calculation circuits in the image processing hardening unit to perform the second fovea processing frame by frame. Alternatively, the switches of each of the hardening calculation circuits in the image processing hardening unit are pre-set to corresponding switch states based on the image processing function of the image processing system, so as to fixedly control one or more of the hardening calculation circuits in the image processing hardening unit to perform the second fovea processing.
[0014] Furthermore, in some embodiments of the present invention, the display pipeline is also connected to an image rendering module, a virtual rendering image is obtained through the image rendering module, and the first fovea processing is performed on the virtual rendering image according to the eye movement signal. The image processing pipeline is also connected to a camera, a real scene image is obtained through the camera, and the second fovea processing is performed on the real scene image according to the frame synchronization signal. The image processing pipeline also transmits the real scene image that has undergone the second fovea processing to the display pipeline to perform layer aliasing with the virtual rendering image that has undergone or has not undergone the first fovea processing.
[0015] In addition, the above-mentioned image processing method based on the fovea principle provided according to the second aspect of the present invention includes the following steps: obtaining the user's eye movement signal; transmitting the eye movement signal to the display pipeline of the image processing system to perform a first fovea processing and generate a frame synchronization signal; and transmitting the frame synchronization signal to the image processing pipeline of the image processing system, so that the image processing pipeline performs a second fovea processing according to the frame synchronization signal.
[0016] Further, in some embodiments of the present invention, the step of transmitting the eye movement signal to the display pipeline of the image processing system to perform the first fovea processing and generate a frame synchronization signal includes: determining the user's gaze point based on the eye movement signal; generating the grid matrix weight map data based on the gaze point; performing the first fovea processing based on the grid matrix weight map data; and generating the frame synchronization signal based on the eye movement signal and / or the grid matrix weight map data.
[0017] Further, in some embodiments of the present invention, the image processing pipeline includes an image processing hardening unit and a software processing unit. At least one hardening calculation circuit is configured in the image processing hardening unit. The step of transmitting the frame synchronization signal to the image processing pipeline of the image processing system so that the image processing pipeline performs the second fovea processing according to the frame synchronization signal comprises: obtaining a switch signal via the image processing hardening unit and the software processing unit; determining the hardening calculation circuit and / or software processing unit that needs to perform the second fovea processing according to the switch signal; and transmitting the frame synchronization signal to at least one hardening calculation circuit and / or software processing unit that needs to perform the second fovea processing so that the at least one hardening calculation circuit and / or software processing unit independently performs the second fovea processing.
[0018] Further, in some embodiments of the present invention, a plurality of the hardening calculation circuits are configured in the image processing hardening unit. The step of determining the hardening calculation circuit and / or software processing unit that needs to perform the second fovea processing according to the switch signal includes: extracting the image features of the current frame image frame by frame via the software processing unit, and configuring the switch signals of each hardening calculation circuit in real time according to the image features, so as to control one or more hardening calculation circuits in the image processing hardening unit to perform the second fovea processing frame by frame; or presetting the switch state of each hardening calculation circuit in the image processing hardening unit based on the image processing function of the image processing system, and fixedly controlling one or more hardening calculation circuits in the image processing hardening unit to perform the second fovea processing according to the switch state of each hardening calculation circuit.
[0019] Furthermore, in some embodiments of the present invention, the display pipeline is also connected to an image rendering module, a virtual rendering image is obtained through the image rendering module, and the first fovea processing is performed on the virtual rendering image according to the eye movement signal. The image processing pipeline is also connected to a camera, a real scene image is obtained through the camera, and the second fovea processing is performed on the real scene image according to the frame synchronization signal. After performing the second fovea processing, the image processing method further includes the following steps: transmitting the real scene image that has undergone the second fovea processing from the image processing pipeline to the display pipeline to perform layer aliasing with the virtual rendering image that has undergone or has not undergone the first fovea processing.
[0020] In addition, the above-mentioned extended reality display device provided according to the third aspect of the present invention includes the above-mentioned image processing system provided by the first aspect of the present invention.
[0021] In addition, the computer-readable storage medium provided according to the fourth aspect of the present invention stores computer instructions, which, when executed by a processor, implement the image processing method provided by the second aspect of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] The above features and advantages of the present invention can be better understood after reading the detailed description of the embodiments of the present disclosure in conjunction with the following drawings. In the drawings, the components are not necessarily drawn to scale, and components with similar related properties or features may have the same or similar reference numerals.
[0023] Figure 1 A schematic diagram of the architecture of an image processing system provided according to some embodiments of the present invention is shown.
[0024] Figure 2 A schematic flow chart of an image processing method provided according to some embodiments of the present invention is shown.
[0025] Figure 3 A schematic diagram of a grid matrix weight map provided according to some embodiments of the present invention is shown.
[0026] Figure 4 A schematic diagram of the architecture of an image processing pipeline provided according to some embodiments of the present invention is shown. DETAILED DESCRIPTION
[0027] The following specific embodiments illustrate the implementation of the present invention, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Although the description of the present invention will be introduced in conjunction with the preferred embodiment, this does not mean that the features of this invention are limited to this implementation. On the contrary, the purpose of introducing the invention in conjunction with the implementation is to cover other options or modifications that may be extended based on the claims of the present invention. In order to provide a deep understanding of the present invention, the following description will include many specific details. The present invention can also be implemented without using these details. In addition, in order to avoid confusion or blurring the focus of the present invention, some specific details will be omitted in the description.
[0028] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0029] In addition, the terms "upper", "lower", "left", "right", "top", "bottom", "horizontal" and "vertical" used in the following description should be understood as the directions shown in the paragraph and the related drawings. Such relative terms are only used for the convenience of description and do not mean that the device described therein must be manufactured or operated in a specific direction, and therefore should not be understood as limiting the present invention.
[0030] It is understood that although the terms "first", "second", "third", etc. may be used herein to describe various components, regions, layers and / or parts, these components, regions, layers and / or parts should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers and / or parts. Therefore, the first component, region, layer and / or part discussed below may be referred to as a second component, region, layer and / or part without departing from some embodiments of the present invention.
[0031] As mentioned above, in response to the high frame rate display requirements in the XR field, the existing technology generally uses the display pipeline to perform image processing based on the fovea principle on the virtual rendered image or the composite image after image aliasing, but lacks joint interaction and multiplexing with the image signal processing (ISP) pipeline. Although it can improve the image processing quality and user experience, it has the defects of high power consumption and large delay. If the fovea image processing operation is simply transferred to the display panel drive circuit (Display Drive Integrated Circuit, DDIC) of the ISP pipeline for processing, there is a problem that both the ISP pipeline and the DDIC lack the display pipeline processing capability based on the gaze point, and cannot support complex and sophisticated compensation and the effect is limited.
[0032] In order to overcome the above-mentioned defects of the prior art, the present invention provides an image processing system based on the fovea principle, an image processing method based on the fovea principle, an extended reality display device, and a computer-readable storage medium, which can comprehensively improve the image quality, power consumption, real-time performance and user experience of image processing through a processing architecture and processing flow of data joint interaction and reuse of the display pipeline and the image processing pipeline, thereby comprehensively improving the current shortage of XR high frame rate display computing power.
[0033] In some non-limiting embodiments, the image processing method provided in the second aspect of the present invention can be implemented via the image processing system provided in the first aspect of the present invention. Specifically, the image processing system is configured with a memory and a processor. The memory includes but is not limited to the computer-readable storage medium provided in the fourth aspect of the present invention, on which computer instructions are stored. The processor is connected to the memory and is configured to execute the computer instructions stored on the memory to implement the image processing method provided in the second aspect of the present invention.
[0034] Furthermore, in some embodiments, the image processing system provided in the first aspect of the present invention may be configured in the extended reality display device provided in the third aspect of the present invention, and is used to reuse image processing data of the display pipeline and / or image processing pipeline when hardware resources are limited, thereby saving hardware computing power and improving the hardware resource utilization of the extended reality display device while ensuring the central imaging effect.
[0035] The following will describe the working principles of the above-mentioned image processing system and extended reality display device in conjunction with some embodiments of the image processing method. Those skilled in the art will understand that these image processing methods are only some non-limiting implementation methods provided by the present invention, which are intended to clearly demonstrate the main concept of the present invention and provide some specific solutions that are convenient for the public to implement, rather than to limit all functions or all working modes of the image processing system and the extended reality display device. Similarly, the image processing system and the extended reality display device are also only some non-limiting implementation methods provided by the present invention, and do not constitute a limitation on the execution subject of each step in these image processing methods.
[0036] Please refer to Figure 1 and Figure 2 . Figure 1 A schematic diagram of the architecture of an image processing system provided according to some embodiments of the present invention is shown. Figure 2 A schematic flow chart of an image processing method provided according to some embodiments of the present invention is shown.
[0037] like Figure 1 and Figure 2 As shown, in some embodiments of the present invention, the image processing system 10 may preferably be configured with a display pipeline (Display Pipeline) 11 and an image processing pipeline (ISP Pipeline) 12. The display pipeline 11 is connected to an external device such as an eye tracker 20 to obtain the user's eye movement signal, and performs a first foveal processing on the local first image according to the obtained eye movement signal. At the same time, the display pipeline 11 is also connected to the image processing pipeline 12, and is also configured to generate a frame synchronization signal according to the obtained eye movement signal, and transmit the frame synchronization signal to the image processing pipeline 12, so that the image processing pipeline 12 performs a second foveal processing on the local second image according to the frame synchronization signal.
[0038] Further, in Figure 1In the illustrated embodiment, the image processing system 10 may also be configured with at least one software module such as a grid matrix (Matirx Grid) generation module 13. The grid matrix generation module 13 is selected from at least one of a digital signal processor (Data Signal Processor, DSP), a microcontroller unit (Microcontroller Unit, MCU), and a microprocessor (Microprocessor), and is used to generate grid matrix weight map data according to the input eye movement signal, and / or perform software calculation and / or processing on the input signal and / or data. The display pipeline 11 can be connected to the eye tracker 20 via the grid matrix generation module 13 to obtain fovea-related data such as gaze position (Gaze Position), grid matrix weight map data, and then perform the first fovea processing on the local first image according to the obtained fovea-related data.
[0039] In addition, Figure 1 In the embodiment shown, the display pipeline 11 may be preferably configured with a display hardening unit 111 and a first software processing unit 112. The display hardening unit 111 is configured with one or more transistor-level hardening calculation circuits and / or memories for hardening calculation processing and storage of input signals. The first software processing unit 112 is selected from at least one of a digital signal processor (DSP), a microcontroller (MCU), and a microprocessor, and is mainly used for calculating the gain value when adjusting the pixel brightness, calculating the pixel mapping logic relationship, calculating the configuration information of the relative relationship of the brightness of the RGB three colors, and the format of the called image gamma data, etc., for the input signals and / or data such as the configuration registers and data processing parameters of each hardening calculation circuit and / or memory in the display hardening unit 111. Software calculation and / or software processing.
[0040] Further, in some embodiments, the above-mentioned grid matrix generation module 13 can also be configured in the first software processing unit 112 to also achieve the effect of generating the first grid matrix weight map data according to the input eye movement signal. Here, the display hardening unit 111 can be connected to the external eye tracker 20 via the first software processing unit 112 to obtain the eye movement signal, and perform the first fovea processing according to the eye movement signal. In addition, the first software processing unit 112 can be connected to the image processing pipeline 12 and the display hardening unit 111 respectively, and perform synchronization alignment processing on the frame synchronization signals of the same frame image in the image processing pipeline 12 and the display hardening unit 111.
[0041] Specifically, in response to the eye movement signal indicating the user's eye deflection angle, line of sight direction, gaze point coordinates and other information obtained from the eye tracker 20, the grid matrix generation module 13 or the first software processing unit 112 configured with the grid matrix generation program can first determine the user's gaze point area data according to the eye movement signal, and then generate the first grid matrix weight map data according to the gaze point area data. Here, the gaze point area data indicates the coordinate position of the user's gaze point on the image and / or one or more pixels around it. The first grid matrix weight map data respectively indicates the user's attention to multiple areas of the image.
[0042] Please refer to further Figure 3 , Figure 3 A schematic diagram of a grid matrix weight map provided according to some embodiments of the present invention is shown.
[0043] like Figure 3 As shown, in the grid matrix weight map, the user's gaze point can have the highest weight (for example: 3), multiple adjacent pixels near it can have higher weights (for example: 2), and multiple edge pixels far from the user's gaze point can have lower weights (for example: 1). The weight of each pixel can be uniformly or unevenly decreased from the gaze point, and the weight of the pixel close to the gaze point is higher, and its grid is relatively dense. Conversely, the weight of the pixel far from the gaze point is lower, and its grid is relatively sparse. In this way, the display hardening unit 111 can perform the first fovea processing on the local first image according to the first grid matrix weight map data. For example, all pixels with a weight of 3 are processed one by one. For another example, one of the nearly N pixels of a pixel with a weight of 2 is processed, and another N-1 pixels are compensated by interpolation, where N is a multiple of 1 or 4. For another example, one of the nearly 4N pixels of a pixel with a weight of 1 is processed, and another 4N-1 pixels are compensated by interpolation, where N is a multiple of 1 or 4. By performing a first fovea processing on the local first image, the present invention can perform a first fovea processing on the local first image in the display pipeline 11 to reduce the data volume of the first image, thereby improving the image processing quality and user experience while ensuring the central imaging effect of the user's gaze area, and improving the hardware resource utilization of the image processing system 10.
[0044] In addition, the first software processing unit 112 can also generate a frame synchronization signal according to the acquired eye movement signal, the determined gaze point area data and / or the generated first grid matrix weight map data, and transmit the frame synchronization signal to the image processing pipeline 12, so that the image processing pipeline 12 can perform the second fovea processing on the local second image according to the frame synchronization signal. Taking the synchronous alignment of the eye movement signal as an example, the first software processing unit 112 can add a timestamp to the acquired eye movement signal to generate a frame synchronization signal, and transmit the eye movement signal with the timestamp to the image processing pipeline 12.
[0045] like Figure 1 As shown, the image processing pipeline 12 may be preferably configured with an image processing hardening unit 121 and a second software processing unit 122. The image processing hardening unit 121 is configured with one or more transistor-level hardening calculation circuits and / or memories for hardening calculation processing and storage of input signals. The second software processing unit 122 is selected from at least one of a digital signal processor (DSP), a microcontroller (MCU), and a microprocessor, wherein at least one software program such as automatic exposure (AE), automatic white balance (AWB), automatic focus (AF), anti-flickering, lens shading correction (auto lens shade correction) (hereinafter referred to as ISP's 5A module), and a grid matrix generation module is configured, which is used to generate second grid matrix weight map data according to the input frame synchronization signal, and / or perform software calculation and / or processing for setting the grid of image information involved in statistical automatic exposure and / or automatic white balance, adjusting the algorithm logic of applying statistical information, and other 5A module foveal characteristics according to the input frame synchronization signal, and / or the input signals and / or data such as the configuration registers and data processing parameters of each hardening calculation circuit and / or memory in the image processing hardening unit 121.
[0046] Specifically, the second software processing unit 122 can be connected to the first software processing unit 112 of the display pipeline 11, and obtain frame synchronization signals such as eye movement signals with timestamps and / or foveal data such as first grid matrix weight map data with timestamps through the first software processing unit 112. Afterwards, the second software processing unit 122 can generate second grid matrix weight map data according to the eye movement signal as described above, and perform second foveal processing based on at least one software program (e.g., 5A module) on the second image of the local corresponding frame according to the timestamp and the generated second grid matrix weight map data. In some embodiments, the resolution and weight of the second grid matrix weight map data generated by the second software processing unit 122 are set according to parameters such as the resolution of the second image, which can be different from the first grid matrix weight map data generated by the first software processing unit 112, so as to further adapt to the processing requirements of the second image and improve the image processing quality of the image processing pipeline 12.
[0047] In addition, the image processing hardening unit 121 may also be directly connected to the first software processing unit 112 of the display pipeline 11, or connected to the first software processing unit 112 via the second software processing unit 122, so as to directly or indirectly obtain a frame synchronization signal via the display pipeline 11. Here, the frame synchronization signal may be the first grid matrix weight map data generated by the first software processing unit 112, or the second grid matrix weight map data generated by the second software processing unit 122. Each hardening calculation circuit in the image processing hardening unit 121 may perform a second fovea processing based on the hardening calculation circuit according to the acquired first and / or second grid matrix weight map data, so as to reduce the data volume of the second image, thereby reducing the power consumption and delay of the image processing system while ensuring the central imaging effect of the user's gaze area, and improving the hardware resource utilization of the image processing system 10.
[0048] Furthermore, by using the first grid matrix weight map data as a frame synchronization signal, the image processing hardening unit 121 can directly use the acquired first grid matrix weight map data to perform the second fovea processing, thereby further saving software computing power, saving hardware resources of the hardened computing circuit, and improving the real-time performance of image processing. Especially in the embodiment of mixed reality (MR) display, by selecting the same grid matrix weight map data to synchronously perform fovea processing of the virtual rendered image (i.e., the first image) and the real scene image (i.e., the second image), the present invention can effectively improve the adaptability of the virtual object and the ISP real scene, and ensure that the resolution transition between the two is similar, thereby achieving a natural transition between the virtual rendered image and the real scene image.
[0049] Please refer to further Figure 4 , Figure 4A schematic diagram of the architecture of an image processing pipeline provided according to some embodiments of the present invention is shown.
[0050] exist Figure 4 In the illustrated embodiment, the image processing hardening unit 121 may preferably be configured with one or more transistor-level hardening calculation circuits such as defective pixel compensation (DPCC), demosaicing (Demosaic), black level correction (BLC), lens shading correction (LSC), noise reduction (NR), sharpening (SHP), color calibration (CCM), gamma correction, and color space conversion (CSC). Moreover, the second software processing unit 122 and / or at least one hardening calculation circuit may be respectively configured with independent switches. Afterwards, in response to the second image input locally, the image processing pipeline 12 may obtain switch signals via the image processing hardening unit 121 and the software processing unit 122, respectively, and determine the hardening calculation circuit and / or software processing unit 122 that needs to perform the second fovea processing according to the switch signal. Afterwards, if the switch signal of the software processing unit 122 and / or any one or more hardened calculation circuits is on, the image processing pipeline 12 may determine that it needs to perform the second fovea processing, and thus transmit the frame synchronization signal to it, so that the at least one hardened calculation circuit and / or the software processing unit 122 can independently perform the second fovea processing. Conversely, if the switch signal of the software processing unit 122 and / or any one or more hardened calculation circuits is off, the image processing pipeline 12 may determine that it does not need to perform the second fovea processing, and thus skip the operation of the second fovea processing, so as to provide a personalized fovea processing function.
[0051] In some embodiments, the switch signal can be preset before the image processing system 10 leaves the factory or is installed and operated according to the specific functions of the image processing system 10. In this way, the image processing system 10 can fixedly control one or more hardening calculation circuits in the image processing hardening unit 121 according to a preset or customized processing method to perform a preset or customized second fovea processing on the local second image.
[0052] Further, in some preferred embodiments, the above-mentioned switch signal can also be set frame by frame by the second software processing unit 122. Specifically, during the XR display process, the image processing pipeline 12 can extract the image features of the second image of the current frame frame by frame through the second software processing unit 122 using a pre-trained artificial intelligence (AI) model to determine whether it needs to perform a fovea processing operation, as well as specific information such as the grid matrix that needs to be used. Afterwards, the image processing pipeline 12 can configure the switch signals of each hardened computing circuit in real time according to the judgment result that the second image of the current frame needs to perform a fovea processing operation, and synchronously acquire and / or generate specific information related to the fovea processing operation such as the grid matrix. In this way, each hardened computing circuit will perform personalized fovea processing on the second image of the current frame according to the switch signal configured in real time by the second software processing unit 122. By analogy, the image processing pipeline 12 can control one or more hardened computing circuits in the image processing hardening unit 121 frame by frame to dynamically perform personalized second fovea processing on the second image of each frame.
[0053] Those skilled in the art can understand that Figure 1 The display pipeline 11 and the image processing pipeline 12 shown are respectively configured with independent software computing units 112 and 122. This is just a non-restrictive implementation method provided by the present invention, which aims to clearly demonstrate the main concept of the present invention and provide a specific solution that is easy for the public to implement, rather than to limit the scope of protection of the present invention.
[0054] Optionally, in other embodiments, the display pipeline and the image processing pipeline may be configured with different hardening units respectively, and share the same software processing unit. In this way, the software processing unit may be connected to the display hardening unit and the image processing hardening unit respectively, and synchronously transmit eye movement signals, frame synchronization signals such as grid matrix weight map data, and / or foveal data between the display hardening unit and the image processing hardening unit to achieve data joint interaction and multiplexing between the display pipeline and the image processing pipeline. The specific working mode of the software processing unit is similar to that of the first software processing unit and the second software processing unit, and will not be repeated here.
[0055] In addition, please continue to refer to Figure 1In some mixed reality (MR) display embodiments, the display pipeline 11 is also connected to the image rendering modules 30 such as the central processing unit (CPU) and the graphics processing unit (GPU) of the mixed reality display device, and the virtual rendering image is obtained through these image rendering modules 30, and the first fovea processing is performed on the virtual rendering image according to the eye movement signal. In addition, the image processing pipeline 12 is also connected to an image acquisition device such as a binocular camera 40, and the real scene image is obtained through the binocular camera 40, and the second fovea processing is performed on the real scene image according to the frame synchronization signal provided by the display pipeline 11. Here, since the resolution of the virtual rendering image and the real scene image is usually quite different, the first grid matrix weight map for the first fovea processing and the second grid matrix weight map for the second fovea processing may also be different, and involve a large resolution difference.
[0056] Furthermore, after the second fovea processing is performed, the image processing pipeline 12 can also add a timestamp to the real scene image that has undergone the second fovea processing, and synchronously transmit it from the image processing pipeline 12 to the display pipeline 11, so that the display pipeline 11 can perform layer stacking, and output it to the subsequent module 14 for mixed reality output display. In this way, the present invention can further reduce the amount of data of the real scene image transmitted between pipelines to save hardware computing power.
[0057] In addition, the display pipeline 11 can find the virtual rendering image of the corresponding frame according to the timestamp of the real scene image, and perform layer aliasing and mixed reality output display on it and the real scene image. Here, the virtual rendering image can be a compressed image that has undergone the first fovea processing. After completing the layer aliasing, the display pipeline 11 can directly send the aliased image to the subsequent module 14 for mixed reality output display.
[0058] Optionally, in some other embodiments, the virtual rendered image may also be an original virtual rendered image that has not been subjected to the first fovea processing. In this way, after completing the layer aliasing, the display pipeline 11 may perform the first fovea processing on the aliased image according to the user's eye movement signal, and then send it to the subsequent module 14 for mixed reality output display.
[0059] In summary, the above-mentioned image processing system, image processing method, extended reality display device and computer-readable storage medium provided by the present invention can comprehensively improve the image quality, power consumption, real-time performance and user experience of image processing through the processing architecture and processing flow of data joint interaction and reuse of the display pipeline and the image processing pipeline, thereby comprehensively improving the current shortage of XR high frame rate display computing power.
[0060] Although the above methods are illustrated and described as a series of actions for simplicity of explanation, it should be understood and appreciated that these methods are not limited by the order of the actions, because according to one or more embodiments, some actions may occur in a different order and / or concurrently with other actions from those illustrated and described herein or not illustrated and described herein but understandable to those skilled in the art.
[0061] Those skilled in the art will appreciate that information, signals, and data may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips cited throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or optical particles, or any combination thereof.
[0062] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technician may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention.
[0063] The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in cooperation with a DSP core, or any other such configuration.
[0064] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein, but should be granted the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. An image processing system based on the fovea principle, comprising a display pipeline and an image processing pipeline, wherein: The display pipeline obtains the user's eye movement signal and performs a first foveal processing according to the eye movement signal, characterized in that: The display pipeline also generates a frame synchronization signal based on the eye movement signal, and transmits the frame synchronization signal to the image processing pipeline, so that the image processing pipeline performs a second fovea processing based on the frame synchronization signal, wherein the frame synchronization signal includes the eye movement signal, gaze point area data and / or grid matrix weight map data, wherein the gaze point area data indicates the coordinate position of the user's gaze point on the image and / or its surrounding pixels, and the grid matrix weight map data indicates the user's attention to multiple areas of the image.
2. The image processing system according to claim 1, wherein: The display pipeline is configured as: Determining gaze point area data of the user according to the eye movement signal; Generating the grid matrix weight map data according to the gaze point area data; performing the first fovea processing according to the grid matrix weight map data; as well as The frame synchronization signal is generated according to the eye movement signal, the gaze point area data and / or the grid matrix weight map data.
3. The image processing system according to claim 1, wherein: The display pipeline includes a display hardening unit and a first software processing unit, wherein the display hardening unit is connected to an eye tracker via the first software processing unit to obtain the eye movement signal, and performs the first foveal processing according to the eye movement signal. The first software processing unit is connected to the image processing pipeline and the display hardening unit, and synchronously aligns frame synchronization signals of the same frame image in the image processing pipeline and the display hardening unit.
4. The image processing system according to claim 1, wherein: The image processing pipeline includes an image processing hardening unit and a second software processing unit, wherein the image processing hardening unit is connected to the display pipeline and obtains the frame synchronization signal through the display pipeline to perform a second fovea processing based on at least one hardening calculation circuit according to the frame synchronization signal. The second software processing unit is connected to the display pipeline and obtains the frame synchronization signal via the display pipeline to perform a second fovea processing based on at least one software program according to the frame synchronization signal.
5. The image processing system according to claim 4, characterized in that: The image processing hardening unit is provided with a plurality of the hardening calculation circuits, wherein at least one of the hardening calculation circuits is provided with an independent switch, and the hardening calculation circuit performs an independent second fovea processing according to a switch signal of a corresponding switch.
6. The image processing system according to claim 5, characterized in that The second software processing unit extracts image features of the current frame image frame by frame, and configures the switch signals of each of the switches in real time according to the image features, so as to control one or more hardening calculation circuits in the image processing hardening unit to perform the second fovea processing frame by frame, or The switches of each of the hardening calculation circuits in the image processing hardening unit are pre-set to corresponding switch states based on the image processing function of the image processing system to fixedly control one or more of the hardening calculation circuits in the image processing hardening unit to perform the second fovea processing.
7. The image processing system according to claim 1, wherein: The display pipeline is also connected to an image rendering module, obtains a virtual rendering image through the image rendering module, and performs the first fovea processing on the virtual rendering image according to the eye movement signal. The image processing pipeline is also connected to a camera, acquires a real scene image through the camera, and performs the second fovea processing on the real scene image according to the frame synchronization signal. The image processing pipeline also transmits the real scene image processed by the second fovea processing to the display pipeline to perform layer aliasing with the virtual rendering image processed or not processed by the first fovea processing.
8. An image processing method based on the fovea principle, characterized in that: The following steps are involved: Obtain user's eye movement signals; Transmitting the eye movement signal to a display pipeline of an image processing system to perform a first fovea processing and generate a frame synchronization signal, wherein the frame synchronization signal includes the eye movement signal, gaze point region data and / or grid matrix weight map data, the gaze point region data indicates the coordinate position of the user's gaze point on the image and / or its surrounding pixels, and the grid matrix weight map data indicates the user's attention to multiple regions of the image; as well as The frame synchronization signal is transmitted to an image processing pipeline of the image processing system, so that the image processing pipeline performs a second fovea processing according to the frame synchronization signal.
9. The image processing method according to claim 8, characterized in that: The step of transmitting the eye movement signal to a display pipeline of an image processing system to perform a first fovea processing and generate a frame synchronization signal comprises: Determining a gaze point of the user according to the eye movement signal; Generating the grid matrix weight map data according to the gaze point; performing the first fovea processing according to the grid matrix weight map data; and The frame synchronization signal is generated according to the eye movement signal, the gaze point area data and / or the grid matrix weight map data.
10. The image processing method according to claim 8, characterized in that: The image processing pipeline includes an image processing hardening unit and a software processing unit, wherein the image processing hardening unit is provided with at least one hardening calculation circuit, and the step of transmitting the frame synchronization signal to the image processing pipeline of the image processing system so that the image processing pipeline performs the second fovea processing according to the frame synchronization signal includes: Acquiring a switch signal via the image processing hardening unit and the software processing unit; Determining, according to the switch signal, a hardening calculation circuit and / or a software processing unit that needs to perform the second fovea processing; and The frame synchronization signal is transmitted to at least one hardened calculation circuit and / or software processing unit that needs to perform the second fovea processing, so that the at least one hardened calculation circuit and / or the software processing unit independently performs the second fovea processing.
11. The image processing method according to claim 10, characterized in that: The image processing hardening unit is provided with a plurality of the hardening calculation circuits, and the step of determining the hardening calculation circuit and / or software processing unit that needs to perform the second fovea processing according to the switch signal comprises: Extracting image features of the current frame image frame by frame via the software processing unit, and configuring the switch signals of each hardening calculation circuit in real time according to the image features, so as to control one or more hardening calculation circuits in the image processing hardening unit to perform the second fovea processing frame by frame; or Based on the image processing function of the image processing system, the switch state of each hardening calculation circuit in the image processing hardening unit is preset, and according to the switch state of each hardening calculation circuit, one or more hardening calculation circuits in the image processing hardening unit are fixedly controlled to perform the second fovea processing.
12. The image processing method according to claim 8, characterized in that: The display pipeline is also connected to an image rendering module, a virtual rendering image is obtained through the image rendering module, and the first fovea processing is performed on the virtual rendering image according to the eye movement signal. The image processing pipeline is also connected to a camera, a real scene image is obtained through the camera, and the second fovea processing is performed on the real scene image according to the frame synchronization signal. After performing the second fovea processing, the image processing method further includes the following steps: The real scene image that has undergone the second fovea processing is transmitted from the image processing pipeline to the display pipeline to be layered with the virtual rendering image that has undergone or has not undergone the first fovea processing.
13. An extended reality display device, characterized in that: The invention comprises the image processing system according to any one of claims 1 to 7.
14. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the image processing method according to any one of claims 8 to 12 is implemented.
