Image processor, processing method, storage medium, and mixed reality display system

By using the color management module of the image processor to perform front-end and back-end processing on the signals of virtual and real layers, the problem of the inability to process HDR data in existing technologies is solved, and the color processing requirements of mixed reality display are realized.

CN119012023AActive Publication Date: 2024-11-22GRAVITYXR ELECTRONICS & TECH CO LTD
View PDF 10 Cites 0 Cited by

Patent Information

Application Number
CN202310598911.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-22
Publication Date
2024-11-22
Estimated Expiration
2043-05-22

AI Technical Summary

Technical Problem

Existing MR devices cannot input and process high dynamic range (HDR) data, and existing image processing devices cannot meet the front-end and back-end processing requirements of color gamut, hue, and color temperature of virtual reality (VR) and reality layer signals in mixed reality displays.

Method used

An image processor is provided, comprising a first, a second, and a third color management module, which respectively perform front-end and back-end processing on signals of virtual layers, real layers, and blended layers with different color gamuts, hues, and color temperatures. The module exchanges register data and statistical information with a color processing firmware unit through a communication interface to achieve flexible color processing.

Benefits of technology

It enables flexible color processing of VST signals and VR layer signals, meeting the application requirements of mixed reality displays and improving the flexibility and efficiency of image processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119012023A_ABST
    Figure CN119012023A_ABST
Patent Text Reader

Abstract

The invention provides an image processor, an image processing method, a storage medium and a mixed reality display system. The image processor comprises a first color management module of which the input end is connected with a rendering module and which is used for acquiring virtual layer data from the rendering module and performing first color processing on the virtual layer data; the input end of the second color management module is connected with the camera module, and the second color management module is used for acquiring real layer data from the camera module and performing second color processing on the real layer data; the input end of the image layer mixing module is connected with the first color management module and the second color management module, and the image layer mixing module is used for carrying out image layer mixing processing on the virtual image layer data and the real image layer data which are subjected to color processing so as to determine mixed image data; and the input end of the third color management module is connected with the image layer mixing module, and the third color management module is used for acquiring the mixed image data from the image layer mixing module, performing third color processing on the mixed image data and outputting the mixed image data subjected to the third color processing to the mixed reality display.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of mixed reality display, and in particular to an image processor, an image processing method, a mixed reality display system, and a computer-readable storage medium. Background Art

[0002] In the field of Mixed Reality (MR) display, existing MR devices can only perform simple processing on the color and brightness of input images, but do not support the input and processing of High Dynamic Range (HDR) data. Although the existing color processing pipelines of mobile phones and other image processing devices can decode single or multiple HDR signals and adapt to the Standard Dynamic Range (SDR), they cannot meet the application requirements of mixed reality display, which requires the front-end processing of the color gamut, hue, and color temperature of the video see through (VST) signal and the virtual reality (VR) layer signal, and then the back-end processing of the color gamut, hue, and color temperature of the mixed layer signal after the layers are mixed.

[0003] In order to overcome the above-mentioned defects of the prior art, the art is in urgent need of an image processing technology for mixed reality display, which can provide flexible color processing functions in different color spaces of virtual layers, real layers and mixed layers, so as to meet the application requirements of mixed reality display. Summary of the invention

[0004] A brief summary of one or more aspects is given below to provide a basic understanding of these aspects. This summary is not an exhaustive overview of all conceived aspects, and is neither intended to identify the key or decisive elements of all aspects nor to define the scope of any or all aspects. Its only purpose is to give some concepts of one or more aspects in a simplified form as a prelude to a more detailed description that will be given later.

[0005] In order to overcome the above-mentioned defects of the prior art, the present invention provides an image processor, an image processing method, a mixed reality display system, and a computer-readable storage medium, which can support the front-end color processing of different color gamuts, different hues and different color temperatures for VST signals and VR layer signals respectively, and the back-end color processing of corresponding color gamuts, corresponding hues and corresponding color temperatures for mixed layer signals after layer mixing, thereby meeting the application requirements of mixed reality display.

[0006] Specifically, the above-mentioned image processor provided according to the first aspect of the present invention includes: a first color management module, whose input end is connected to the rendering module, for obtaining virtual layer data from the rendering module and performing a first color processing on it; a second color management module, whose input end is connected to the camera module, for obtaining real layer data from the camera module and performing a second color processing on it; a layer mixing module, whose input end is respectively connected to the first color management module and the second color management module, for performing layer mixing processing on the virtual layer data after the first color processing and the real layer data after the second color processing to determine mixed image data; and a third color management module, whose input end is connected to the layer mixing module, for obtaining the mixed image data from the layer mixing module, performing a third color processing on it, and outputting the mixed image data after the third color processing to the mixed reality display.

[0007] Furthermore, in one embodiment of the present invention, the image processor further includes a communication interface. The communication interface connects the color processing firmware unit and each of the color management modules. The first color management module is configured to: obtain first register data from the color processing firmware unit via the communication interface according to the requirements of the first color processing, and return first statistical information to the color processing firmware unit. The second color management module is configured to: obtain second register data from the color processing firmware unit via the communication interface according to the requirements of the second color processing, and return second statistical information to the color processing firmware unit. The third color management module is configured to: obtain third register data from the color processing firmware unit via the communication interface according to the requirements of the third color processing, and return third statistical information to the color processing firmware unit.

[0008] Furthermore, in one embodiment of the present invention, each of the color management modules includes a tone mapping unit, a color gamut mapping unit and a color temperature processing unit, and is configured to: obtain register data related to tone mapping, color gamut mapping and / or color temperature processing from the color processing firmware unit via the communication interface according to color processing requirements, and return statistical information related to the tone mapping, the color gamut mapping and / or the color temperature processing to the color processing firmware unit.

[0009] Furthermore, in one embodiment of the present invention, each of the color management modules is also configured to: perform histogram equalization processing on the corresponding virtual layer data, real layer data or mixed image data according to the register data related to the tone mapping to determine the statistical information related to the tone mapping; and / or in response to the color processing requirements of converting from a large color gamut to a small color gamut, perform step-by-step mapping processing on the corresponding virtual layer data, real layer data or mixed image data according to the register data related to the color gamut mapping to determine the statistical information related to the color gamut mapping.

[0010] Furthermore, in one embodiment of the present invention, each of the color management modules also includes a ReGamma unit and / or a DeGamma unit, and is configured to: obtain register data related to ReGamma processing and / or DeGamma processing from the color processing firmware unit via the communication interface according to the color processing requirements, and return statistical information related to the ReGamma processing and / or the DeGamma processing to the color processing firmware unit.

[0011] Furthermore, in one embodiment of the present invention, the rendering module also inputs HDR metadata into the color processing firmware unit. Each of the color management modules also includes an HDR unit, and is configured to: obtain register data related to HDR processing from the color processing firmware unit via the communication interface according to the color processing requirements, and return statistical information related to the HDR processing to the color processing firmware unit, wherein the register data related to HDR processing is updated and determined according to the HDR metadata and the returned statistical information related to HDR processing.

[0012] Furthermore, in an embodiment of the present invention, the image processor further comprises the color processing firmware unit. The color processing firmware unit is configured to: obtain external light parameter information; and generate a tone curve, generate a color gamut mapping function, perform color temperature adjustment operation and / or perform brightness adjustment operation according to the light parameter information and at least one statistical information returned by the color management module, so as to update the register data.

[0013] Furthermore, in an embodiment of the present invention, the data volume of the first register data and the second register data is smaller than the data volume of the third register data. The data volume of the first statistical information and the second statistical information is smaller than the data volume of the third statistical information.

[0014] Furthermore, in an embodiment of the present invention, the rendering module is also used to render and generate one or more UI layer data. The first color management module is further configured to: obtain the one or more UI layer data from the rendering module, and perform one or more corresponding fourth color processing on the one or more UI layer data; and input the UI layer data after the one or more fourth color processing into the layer blending module for layer blending processing to determine the corresponding blended image data.

[0015] Furthermore, in an embodiment of the present invention, the image processor further comprises an image signal processing module. The second color management module is connected to the camera module via the image signal processing module. The image signal processing module is configured to: obtain the video stream data shot by the camera module; and perform AWB processing and / or AE processing on the video stream data to generate the reality layer data.

[0016] In addition, the above-mentioned mixed reality display system provided according to the second aspect of the present invention includes: a rendering module, used to generate virtual layer data; a camera module, used to collect real layer data; and the above-mentioned image processor provided by the first aspect of the present invention, connected to the rendering module and the camera module to obtain the virtual layer data and the real layer data, and perform image processing based on the virtual layer data and the real layer data to output mixed image data to be displayed.

[0017] In addition, the above-mentioned image processing method provided according to the third aspect of the present invention includes the following steps: obtaining virtual layer data from a rendering module and inputting it into a first color management module; obtaining real layer data from a camera module and inputting it into a second color management module; obtaining color processing requirements; according to the color processing requirements, controlling the first color management module to perform a first color processing on the virtual layer data, and / or controlling the second color management module to perform a second color processing on the real layer data; performing layer mixing processing on the layer data of the first color management module and the second color management module via a layer mixing module to determine mixed image data; inputting the mixed image data into a third color management module, and performing a third color processing on it according to the color processing requirements; and transmitting the mixed image data output by the third color management module to a mixed reality display.

[0018] In addition, the computer-readable storage medium provided according to the fourth aspect of the present invention stores computer instructions, which, when executed by a processor, implement the image processing method provided by the third aspect of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above features and advantages of the present invention can be better understood after reading the detailed description of the embodiments of the present disclosure in conjunction with the following drawings. In the drawings, the components are not necessarily drawn to scale, and components with similar related properties or features may have the same or similar reference numerals.

[0020] Figure 1 A schematic structural diagram of a mixed reality display system provided according to some embodiments of the present invention is shown.

[0021] Figure 2 A flowchart of an image processing method provided according to some embodiments of the present invention is shown.

[0022] Figure 3 A schematic diagram of a process for processing an image in a typical data format according to some embodiments of the present invention is shown.

[0023] Figure 4 A schematic diagram of a process for processing an image in a typical data format according to some embodiments of the present invention is shown. DETAILED DESCRIPTION

[0024] The following specific embodiments illustrate the implementation of the present invention, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Although the description of the present invention will be introduced in conjunction with the preferred embodiment, this does not mean that the features of this invention are limited to this implementation. On the contrary, the purpose of introducing the invention in conjunction with the implementation is to cover other options or modifications that may be extended based on the claims of the present invention. In order to provide a deep understanding of the present invention, the following description will include many specific details. The present invention can also be implemented without using these details. In addition, in order to avoid confusion or blurring the focus of the present invention, some specific details will be omitted in the description.

[0025] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0026] In addition, the terms "upper", "lower", "left", "right", "top", "bottom", "horizontal" and "vertical" used in the following description should be understood as the directions shown in the paragraph and the related drawings. Such relative terms are only used for the convenience of description and do not mean that the device described therein must be manufactured or operated in a specific direction, and therefore should not be understood as limiting the present invention.

[0027] It is understood that although the terms "first", "second", "third", etc. may be used herein to describe various components, regions, layers and / or parts, these components, regions, layers and / or parts should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers and / or parts. Therefore, the first component, region, layer and / or part discussed below may be referred to as a second component, region, layer and / or part without departing from some embodiments of the present invention.

[0028] As mentioned above, in the field of Mixed Reality (MR) display, existing MR devices can only perform simple processing on the color and brightness of input images, but do not support the input and processing of High Dynamic Range (HDR) data. Although the existing color processing pipelines of mobile phones and other image processing devices can decode single-channel or multi-channel HDR signals and adapt to the Standard Dynamic Range (SDR), they cannot meet the application requirements of mixed reality display, which requires the front-end processing of the color gamut, hue, and color temperature of the reality layer (Video See Through, VST) signal and the virtual reality (Virtual Reality, VR) layer signal, and then the back-end processing of the color gamut, hue, and color temperature of the mixed layer signal after the layers are mixed.

[0029] In order to overcome the above-mentioned defects of the prior art, the present invention provides an image processor, an image processing method, a mixed reality display system, and a computer-readable storage medium, which can support the front-end color processing of different color gamuts, different hues and different color temperatures for VST signals and VR layer signals respectively, and the back-end color processing of corresponding color gamuts, corresponding hues and corresponding color temperatures for mixed layer signals after layer mixing, thereby meeting the application requirements of mixed reality display.

[0030] In some non-limiting embodiments, the image processing method provided in the third aspect of the present invention may be implemented via the mixed reality display system provided in the second aspect of the present invention. Figure 1 , Figure 1 A schematic structural diagram of a mixed reality display system provided according to some embodiments of the present invention is shown.

[0031] exist Figure 1 In the illustrated embodiment, the mixed reality display system is configured with a main processor 10, a display pipeline 20, an image signal processing (ISP) pipeline 30, a camera module 40, and a memory (not shown).

[0032] The memory includes but is not limited to the computer-readable storage medium provided in the fourth aspect of the present invention, on which are stored computer instructions for executing one or more steps of the image processing method provided in the third aspect of the present invention.

[0033] The main processor 10 is configured with a rendering module such as a graphics processing unit (GPU) and a video codec module for generating virtual layer data and transmitting the virtual layer data to a first color management module 21 in the display pipeline 20 for first color processing.

[0034] The display pipeline 20 is configured as the above-mentioned image processor provided by the first aspect of the present invention, which is configured with a first color management module 21, a second color management module 22, a third color management module 23, a layer blending module 24 and a post-stage module 25, and can be used as a coprocessor to implement the above-mentioned image processing method provided by the third aspect of the present invention by reading and executing computer instructions stored in the above-mentioned memory, thereby cooperating with the main processor 10 to perform mixed reality image processing.

[0035] The camera module 40 can be connected to the display pipeline 20 via the ISP pipeline 30 to collect real layer data and transmit it to the second color management module 22 in the display pipeline 20 for second color processing.

[0036] The following will describe the working principles of the above-mentioned display pipeline 20 and mixed reality display system in conjunction with some embodiments of the image processing method. Those skilled in the art will understand that these embodiments of the image processing method are only some non-restrictive implementation methods provided by the present invention, which are intended to clearly demonstrate the main concept of the present invention and provide some specific solutions that are convenient for the public to implement, rather than to limit all functions or all working modes of the display pipeline 20 and the mixed reality display system. Similarly, the display pipeline 20 and the mixed reality display system are also only a non-restrictive implementation method provided by the present invention, and do not constitute a limitation on the execution subject and execution order of each step in the image processing method.

[0037] Please refer to Figure 1 and Figure 2 , Figure 2A flowchart of an image processing method provided according to some embodiments of the present invention is shown.

[0038] like Figure 1 and Figure 2 As shown, during the mixed reality image processing, the display pipeline 20 may first obtain virtual layer data from a rendering module such as a GPU of the main processor 10, and input the virtual layer data into the first color management module 21. Here, the virtual layer data may preferably include color-related information such as color temperature information generated by GPU rendering.

[0039] In addition, the display pipeline 20 can also obtain the real layer data collected by the camera module 40 via the ISP pipeline 30, and input it into the second color management module 22. Figure 1 In the illustrated embodiment, the ISP pipeline 30 may preferably be configured with an automatic white balance (AWB) unit and / or an automatic exposure (AE) unit. The ISP pipeline 30 may first obtain the video stream data captured by the camera module, and then perform AWB processing on the obtained video stream data via the AWB unit, and / or perform AE processing on the obtained video stream data via the AE unit, so as to generate the real layer data. In this way, the real layer data may also preferably include the AWB information and / or AE information generated by the ISP pipeline 30.

[0040] Afterwards, the display pipeline 20 can obtain color processing requirements for the virtual layer data and the real layer data from the mixed reality display system, and control the first color management module 21 to perform first color processing on the virtual layer data according to the color processing requirements, and / or control the second color management module 22 to perform second color processing on the real layer data according to the color processing requirements.

[0041] Specifically, in Figure 1 In the illustrated embodiment, the mixed reality display system may also be preferably configured with a color processing firmware unit 50. The color processing firmware unit 50 may collect external ambient light intensity information via a sensor, obtain color temperature information generated by GPU rendering via the main processor 10, obtain AWB information and / or AE information generated by the ISP pipeline 30, and generate a tone curve, generate a color gamut mapping function, perform color temperature adjustment operations, and / or brightness adjustment operations in combination with statistical information such as statistical histograms returned by each color management module 21 to 23 in the display pipeline 20, so as to generate and update register data.

[0042] In response to a request instruction for the first color processing, the first color management module 21 can obtain the corresponding first register data from the color processing firmware unit 50 via the communication interface (not shown) of the display pipeline 20 to perform the first color processing on the above-mentioned virtual layer data, and return the first statistical information generated by the processing to the color processing firmware unit 50 for it to regenerate and update the corresponding register data.

[0043] Further, in Figure 1 In the illustrated embodiment, the first color management module 21 may preferably be configured with a tone mapping unit, a color gamut mapping unit, and a color temperature processing unit. In the process of performing the first color processing, the first color management module 21 may first obtain the first register data involving the first color gamut, the first tone, and the first color temperature from the color processing firmware unit 50 via the communication interface to perform the first color processing on the virtual layer data regarding tone mapping, color gamut mapping, and / or color temperature processing, and then return the first statistical information involving tone mapping, color gamut mapping, and / or color temperature processing generated by the processing to the color processing firmware unit 50, so that it can regenerate and update the corresponding register data.

[0044] Specifically, when determining the statistical information related to tone mapping, the first color management module 21 can perform histogram equalization processing on the virtual layer data according to the first register data related to tone mapping to determine the statistical information related to tone mapping. In addition, when determining the statistical information related to color gamut mapping, in response to the color processing requirements of converting from a large color gamut to a small color gamut, the first color management module 21 can perform step-by-step mapping processing on the virtual layer data according to the first register data related to color gamut mapping to determine the statistical information related to color gamut mapping.

[0045] In addition, Figure 1 In the illustrated embodiment, a ReGamma unit and / or a DeGamma unit may also be preferably configured in the first color management module 21. During the first color processing, the first color management module 21 may also obtain first register data related to the ReGamma processing and / or the DeGamma processing from the color processing firmware unit 50 via the communication interface to perform corresponding ReGamma processing and / or DeGamma processing on the virtual layer data, and then return the first statistical information related to the ReGamma processing and / or DeGamma processing generated by the processing to the color processing firmware unit 50 for it to regenerate and update the corresponding register data.

[0046] In addition, Figure 1In the illustrated embodiment, the first color management module 21 may also be preferably configured with an HDR unit. Correspondingly, the color processing firmware unit 50 may also preferably obtain HDR metadata from a rendering module such as a GPU of the main processor 10, and generate and update corresponding register data in combination with the HDR metadata. During the first color processing, the first color management module 21 may also obtain first register data related to HDR processing from the color processing firmware unit 50 via a communication interface to perform HDR processing on the virtual layer data, and then return the statistical information related to HDR processing generated by the processing to the color processing firmware unit 50 for it to regenerate and update the corresponding register data.

[0047] In this way, the present invention can flexibly perform HDR processing, tone mapping, color gamut mapping, color temperature processing and / or ReGamma processing on the virtual layer data according to the actually received first color processing request instruction, and bypass the processing units not involved in the request instruction to obtain the virtual layer data required for layer mixing.

[0048] Similarly, in Figure 1 In the illustrated embodiment, the second color management module 22 may also be preferably configured with an HDR unit, a tone mapping unit, a color gamut mapping unit, a color temperature processing unit, a ReGamma unit and / or a DeGamma unit. In response to the request instruction for the second color processing, the second color management module 22 may also obtain the second register data related to HDR processing, tone mapping, color gamut mapping, color temperature processing, ReGamma processing and / or DeGamma processing from the color processing firmware unit 50 via the communication interface (not shown) of the display pipeline 20 as described above, so as to perform HDR processing, tone mapping, color gamut mapping, color temperature processing, ReGamma processing and / or DeGamma processing on the above-mentioned real layer data, and bypass processing is performed on the processing units not involved in the request instruction to obtain the real layer data required for layer mixing. Afterwards, the second color management module 22 may also return the second statistical information generated by the processing to the color processing firmware unit 50 as described above, so that it can regenerate and update the corresponding register data. The specific steps of obtaining register data, performing front-end color processing and returning statistical information are similar to those of the embodiment of the first color management module 21, and will not be elaborated here.

[0049] By configuring two front-end color management modules 21-22 for virtual layer data and real layer data respectively, and providing bypass processing function, the above-mentioned display pipeline 20 provided by the present invention can provide flexible color processing functions for virtual layer data and real layer data in different color spaces according to the actual needs of mixed reality display, thereby meeting the application needs of mixed reality display.

[0050] Please continue to refer to Figure 1 and Figure 2 , the input end of the layer blending module 24 is respectively connected to the first color management module 21 and the second color management module 22. After completing the first color processing and the second color processing respectively, the display pipeline 20 can perform layer blending processing on the virtual layer data after the first color processing and the real layer data after the second color processing through the layer blending module 24 to determine the blended image data, and then input the blended image data into the third color management module 23 to perform the third color processing according to the color processing requirements.

[0051] Here, the specific solution for layer mixing processing of multiple layer data does not involve the technical improvement of the present invention, and thus will not be elaborated in detail.

[0052] The input end of the third color management module 23 is connected to the output end of the layer blending module 24, which is preferably configured with a tone mapping unit, a color gamut mapping unit, a color temperature processing unit, a ReGamma unit and / or a DeGamma unit. In the process of performing the third color processing, in response to the request instruction of the third color processing, the third color management module 22 can obtain the mixed image data from the layer blending module, and obtain the third register data involving tone mapping, color gamut mapping, color temperature processing, ReGamma processing and / or DeGamma processing from the color processing firmware unit 50 via the communication interface (not shown) of the display pipeline 20 as described above, so as to perform tone mapping, color gamut mapping, color temperature processing, ReGamma processing and / or DeGamma processing on the above-mentioned mixed image data, and bypass the processing units not involved in the request instruction to obtain the display image data required for mixed reality display. Afterwards, the third color management module 23 can also return the third statistical information generated by the third color processing to the color processing firmware unit 50 as described above, so that it can regenerate and update the corresponding register data, and output the mixed image data (i.e., display image data) after the third color processing to the mixed reality display 60 such as the VR display screen via the post-stage module 25 for mixed reality display. Here, the specific steps of obtaining register data, performing front-end color processing, and returning statistical information are similar to those of the embodiment of the first color management module 21, and thus will not be described in detail.

[0053] Furthermore, in some embodiments of the present invention, the display pipeline 20 may preferentially configure the processing subunit involved in the third color processing according to the color processing requirements of the mixed reality display, and correspondingly configure the third register data, and then differentially configure the processing subunits and register data involved in the first and second color processing according to the requirements of the layer mixing processing, so that the data volume of the first register data and the second register data involved in the above-mentioned front-end color management is smaller than the data volume of the third register data involved in the back-end color management, and the data volume of the above-mentioned first statistical information and the second statistical information is smaller than the data volume of the third statistical information. In this way, by configuring the data volume of the front-end color management to be smaller than the data volume of the back-end color management, the present invention can effectively save the hardware resources of the display pipeline 20 for storing and calculating repeated data, thereby improving the color management capability of the image processor under the condition of the same hardware resources, or saving hardware resources under the condition of the same color management capability.

[0054] Those skilled in the art will appreciate that the above-mentioned embodiment of configuring the data volume of the front-end color management to be smaller than the data volume of the back-end color management is only a preferred solution provided by the present invention, which aims to provide a specific solution that is easy for the public to implement, and is not used to limit the scope of protection of the present invention.

[0055] Optionally, in other embodiments, those skilled in the art may also use other methods to respectively configure the register data of the first, second, and third color processing. Even if the data amount of the first register data and the second register data involved in the front-end color management is greater than or equal to the data amount of the third register data involved in the back-end color management, the normal operation of the display pipeline 20 is not affected.

[0056] In addition, for some mixed reality display applications involving human-computer interaction graphic interfaces, videos and other contents, the mixed reality display system can also generate corresponding one or more UI layer data through rendering by rendering modules such as the GPU of the main processor 10. Here, the UI layer data can be HDR layer data or SDR layer data. The first color management module 21 of the display pipeline 20 can preferably obtain the one or more UI layer data from the rendering module, and perform one or more corresponding fourth color processing on them according to the requirements of the fourth color processing. Afterwards, the first color management module 21 can also input the UI layer data that has undergone the one or more fourth color processing into the layer mixing module 24 for layer mixing processing to determine the corresponding mixed image data, thereby meeting the requirements of the mixed reality display applications involving human-computer interaction graphic interfaces, videos and other contents.

[0057] In this way, by separately configuring a back-end color management module 23 for the mixed image data and providing a bypass processing function, the above-mentioned display pipeline 20 provided by the present invention can provide flexible color processing functions for the mixed image data in the corresponding color space according to the actual needs of the mixed reality display, thereby further meeting the application needs of the mixed reality display.

[0058] The following will continue to provide some non-limiting specific embodiments, and accordingly illustrate the specific working methods of the display pipeline 20 and the mixed reality display system provided by the present invention. Figure 3 and Figure 4 , Figure 3 and Figure 4 Schematic diagrams of the processes for processing images in typical data formats according to some embodiments of the present invention are respectively shown.

[0059] exist Figure 3 In the illustrated embodiment, the display pipeline 20 may first obtain virtual layer data of B11 bit, HDR, gamma, wide color gamut (rendering), and warm color temperature generated by the rendering of the GPU of the main processor 10, and input the virtual layer data into the first color management module 21 for first color processing to obtain first data to be mixed of B12 bit, linear, wide color gamut, and D65. At the same time, the display pipeline 20 may also obtain the real layer data of B13 bit, SDR, gamma 2.2, camera sensor color gamut, and cold color temperature collected by the camera module 40 via the ISP pipeline 30, and input the real layer data into the second color management module 22 for second color processing to obtain second data to be mixed of B12 bit, linear, wide color gamut, and D65.

[0060] Afterwards, the display pipeline 20 can perform layer mixing processing on the first data to be mixed and the second data to be mixed via the layer mixing module 24 to determine the mixed image data of B12 bit, linear, wide color gamut, and D65, and input it into the third color management module 23 for third color processing to obtain the image data to be displayed of B14 bit, SDR, gamma, screen color gamut, and warm color temperature. It can be understood that B11 to B14 represent the bit depth of each image, and their specific values ​​can be set manually according to the actual accuracy requirements, which does not involve the technical improvement of the present application and will not be expanded here.

[0061] exist Figure 4In the illustrated embodiment, the display pipeline 20 may first obtain virtual layer data of B21 bit, sRGB, gamma, sRGB color gamut, and warm color temperature generated by rendering from the GPU of the main processor 10, and input the virtual layer data into the first color management module 21 for first color processing to obtain first data to be mixed of B22 bit, sRGB, gamma, sRGB color gamut, and D65. At the same time, the display pipeline 20 may also obtain real layer data of B23 bit, sRGB, gamma, sRGB color gamut, and cold color temperature collected from the camera module 40 via the ISP pipeline 30, and input the real layer data into the second color management module 22 for second color processing to obtain second data to be mixed of B22 bit, sRGB, gamma, sRGB color gamut, and D65.

[0062] Afterwards, the display pipeline 20 can perform layer mixing processing on the first data to be mixed and the second data to be mixed via the layer mixing module 24 to determine the mixed image data of B22 bit, sRGB, gamma, sRGB color gamut, and D65, and input it into the third color management module 23 for third color processing to obtain the image data to be displayed of B24 bit, sRGB, gamma, sRGB color gamut, and warm color temperature. It can be understood that B21 to B24 represent the bit depth of each image, and their specific values ​​can be set manually according to the actual accuracy requirements, which does not involve the technical improvement of the present application and will not be expanded here.

[0063] It can be seen that by configuring a color management module 21~23 for virtual layer data, real layer data and mixed image data respectively, and providing a bypass processing function, the above-mentioned display pipeline 20 provided by the present invention can provide flexible color processing functions for virtual layer data, real layer data and mixed image data in different color spaces according to the actual needs of mixed reality display, thereby meeting the application needs of mixed reality display.

[0064] Those skilled in the art will understand that Figure 1 The display pipeline 20 shown is only a non-limiting implementation of the above-mentioned image processor provided by the first aspect of the present invention, and is intended to clearly demonstrate the main concept of the present invention and provide a specific solution that is easy for the public to implement, rather than to limit the scope of protection of the present invention.

[0065] Optionally, in other embodiments, those skilled in the art may also integrate the display pipeline 20 with any one or more of the main processor 10, the ISP pipeline 30, and the color processing firmware unit 50 based on the above concept provided by the present invention to construct a composite image processor with corresponding multiple functions, which will not be described one by one here.

[0066] In summary, the above-mentioned image processor, mixed reality display system, image processing method, and computer-readable storage medium provided by the present invention not only support the input of external light parameter information, but also can replace multiple simple display pipelines with fewer and more complex display pipelines 20, thereby flexibly optimizing the display effects of virtual layer data and real layer data respectively, and allowing layers to be mixed under different conditions, so as to meet the needs of various mixed reality applications for color adjustment and management.

[0067] Although the above methods are illustrated and described as a series of actions for simplicity of explanation, it should be understood and appreciated that these methods are not limited by the order of the actions, because according to one or more embodiments, some actions may occur in a different order and / or concurrently with other actions from those illustrated and described herein or not illustrated and described herein but understandable to those skilled in the art.

[0068] Those skilled in the art will appreciate that information, signals, and data may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips cited throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or optical particles, or any combination thereof.

[0069] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technician may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention.

[0070] The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in cooperation with a DSP core, or any other such configuration.

[0071] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein, but should be granted the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processor, characterized in that: include: A first color management module, whose input end is connected to the rendering module, for obtaining virtual layer data from the rendering module and performing a first color processing on the virtual layer data; A second color management module, whose input end is connected to the camera module, is used to obtain the real layer data from the camera module and perform a second color processing on it; A layer blending module, whose input end is connected to the first color management module and the second color management module respectively, and is used to perform layer blending processing on the virtual layer data after the first color processing and the real layer data after the second color processing to determine blended image data; as well as A third color management module, whose input end is connected to the layer blending module, is used to obtain the mixed image data from the layer blending module, perform third color processing on the mixed image data, and output the mixed image data after the third color processing to the mixed reality display.

2. The image processor according to claim 1, wherein: It also includes a communication interface, wherein the communication interface is connected to the color processing firmware unit and each of the color management modules, The first color management module is configured to: obtain first register data from the color processing firmware unit via the communication interface according to the first color processing requirement, and return first statistical information to the color processing firmware unit, The second color management module is configured to: obtain second register data from the color processing firmware unit via the communication interface according to the second color processing requirement, and return second statistical information to the color processing firmware unit, The third color management module is configured to: obtain third register data from the color processing firmware unit via the communication interface according to the third color processing requirement, and return third statistical information to the color processing firmware unit.

3. The image processor according to claim 2, wherein: Each of the color management modules includes a tone mapping unit, a color gamut mapping unit and a color temperature processing unit, and is configured as follows: According to color processing requirements, register data related to tone mapping, color gamut mapping and / or color temperature processing is obtained from the color processing firmware unit via the communication interface, and statistical information related to the tone mapping, the color gamut mapping and / or the color temperature processing is returned to the color processing firmware unit.

4. The image processor according to claim 3, characterized in that Each of the color management modules is further configured to: According to the register data related to the tone mapping, a histogram equalization process is performed on the corresponding virtual layer data, the real layer data or the mixed image data to determine the statistical information related to the tone mapping; and / or In response to the color processing requirements of converting from a large color gamut to a small color gamut, the corresponding virtual layer data, real layer data or mixed image data is mapped step by step according to the register data related to the color gamut mapping to determine the statistical information related to the color gamut mapping.

5. The image processor according to claim 3, characterized in that: Each of the color management modules further includes a ReGamma unit and / or a DeGamma unit, and is configured as follows: According to the color processing requirements, register data related to ReGamma processing and / or DeGamma processing is obtained from the color processing firmware unit via the communication interface, and statistical information related to the ReGamma processing and / or the DeGamma processing is returned to the color processing firmware unit.

6. The image processor according to claim 3, characterized in that The rendering module also inputs HDR metadata into the color processing firmware unit. Each of the color management modules also includes an HDR unit and is configured as follows: According to the color processing requirements, register data related to HDR processing is obtained from the color processing firmware unit via the communication interface, and statistical information related to the HDR processing is returned to the color processing firmware unit, wherein the register data related to HDR processing is updated and determined based on the HDR metadata and the returned statistical information related to the HDR processing.

7. The image processor according to claim 3, characterized in that: Also included is the color processing firmware unit, wherein the color processing firmware unit is configured to: Acquiring external light parameter information; and According to the light parameter information and the statistical information returned by at least one of the color management modules, a tone curve is generated, a color gamut mapping function is generated, a color temperature adjustment operation is performed, and / or a brightness adjustment operation is performed to update the register data.

8. The image processor according to claim 2, wherein: The data amounts of the first register data and the second register data are smaller than the data amount of the third register data, and the data amounts of the first statistical information and the second statistical information are smaller than the data amount of the third statistical information.

9. The image processor according to claim 1, wherein: The rendering module is further used to render and generate one or more UI layer data, and the first color management module is further configured to: Acquire the one or more UI layer data from the rendering module, and perform one or more corresponding fourth color processing on the one or more UI layer data; as well as The UI layer data processed by the one or more fourth colors is input into the layer blending module for layer blending processing to determine corresponding blended image data.

10. The image processor according to claim 1, wherein: It also includes an image signal processing module, wherein the second color management module is connected to the camera module via the image signal processing module, and the image signal processing module is configured as follows: Acquiring video stream data captured by the camera module via the camera module; and The video stream data is subjected to AWB processing and / or AE processing to generate the reality layer data.

11. A mixed reality display system, characterized in that: include: Rendering module, used to generate virtual layer data; Camera module, used to collect real-world layer data; as well as The image processor as described in any one of claims 1 to 10 is connected to the rendering module and the camera module to obtain the virtual layer data and the real layer data, and performs image processing based on the virtual layer data and the real layer data to output mixed image data to be displayed.

12. An image processing method, characterized in that: The following steps are involved: Acquire virtual layer data from the rendering module and input it into the first color management module; Acquire the reality layer data from the camera module and input it into the second color management module; Get color processing requirements; According to the color processing requirement, controlling the first color management module to perform a first color processing on the virtual layer data, and / or controlling the second color management module to perform a second color processing on the real layer data; Performing layer blending processing on the layer data of the first color management module and the second color management module via a layer blending module to determine blended image data; Inputting the mixed image data into a third color management module, and performing third color processing on the mixed image data according to the color processing requirement; as well as The mixed image data output by the third color management module is transmitted to the mixed reality display.

13. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the image processing method according to claim 12 is implemented.

Citation Information

Patent Citations

  • Efficient tone-mapping of high-bit-depth video to low-bit-depth display

    CN103327323A

  • LED intelligent controller achieving layer blending and color edit on the basis of DMX512

    CN103491688A

  • Method and system of camera control and image processing with a multi-frame-based window for image data statistics

    CN109040576A

  • Systems and methods for tone mapping of high dynamic range images for high-quality deep learning based processing

    CN112085664A

  • Gaze-Based Exposure

    CN112584127A