Image processor, processing method, storage medium, and mixed reality display system

By designing multiple color management modules and communication interfaces in the image processor, flexible color processing of virtual reality and real-life layer signals is solved, and the problem that the existing technology cannot process HDR data and meet mixed reality display needs is improved. Image processor's color management capabilities and saves hardware resources.

WO2024239904A1PCT designated stage expired Publication Date: 2024-11-28GRAVITYXR ELECTRONICS & TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/089820
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-22
Filing Date
2024-04-25
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Existing mixed reality display devices cannot effectively process high dynamic range (HDR) data, and cannot meet the front-end processing of color gamut, tone, and color temperature of virtual reality (VR) and real layer signals respectively, and then mix layers. The back-end processing requirement is required.

Method used

An image processor is designed, including multiple color management modules, which are used to perform front-end and back-end color processing of virtual layers, real layers and mixed layers in different color gamuts, different tones and different color temperatures, through communication The interface interacts with the color processing firmware unit, generates tone curves, color gamut mapping functions, and performs brightness adjustments to achieve flexible color processing.

Benefits of technology

It realizes flexible color processing of virtual reality and real layer signals, meets the application needs of mixed reality display, improves the color management capabilities of the image processor, and saves hardware resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024089820_28112024_PF_FP_ABST
    Figure CN2024089820_28112024_PF_FP_ABST
Patent Text Reader

Abstract

Provided in the present invention are an image processor, a processing method, a storage medium, and a mixed reality display system. The image processor comprises: a first color management module, of which an input end is connected to a rendering module, configured to acquire virtual layer data from the rendering module and perform first color processing on the virtual layer data; a second color management module, of which an input end is connected to a camera module, configured to acquire real layer data from the camera module and perform second color processing on the real layer data; a layer mixing module, of which an input end is separately connected to the first color management module and the second color management module, configured to perform layer mixing processing on the virtual layer data and the real layer data that have undergone the color processing to determine mixed image data; and a third color management module, of which an input end is connected to the layer mixing module, configured to acquire the mixed image data from the layer mixing module, perform third color processing on the mixed image data, and output the mixed image data that have undergone the third color processing to a mixed reality display.
Need to check novelty before this filing date? Find Prior Art

Description

Image processor, processing method, storage medium and mixed reality display system

[0001] This invention claims priority to a patent application with a filing date of May 22, 2023, Chinese application number 202310598911.8, and titled “Image Processor, Processing Method, Storage Medium, and Mixed Reality Display System.” Technical Field

[0002] The present invention relates to the field of mixed reality display, and in particular to an image processor, an image processing method, a mixed reality display system, and a computer-readable storage medium. Background Art

[0003] In the field of Mixed Reality (MR) displays, existing MR devices can only perform simple processing on the color and brightness of input images, but do not support the input and processing of High Dynamic Range (HDR) data. While the existing color processing pipelines of image processing devices such as mobile phones can decode and adapt single- or multi-channel HDR signals to Standard Dynamic Range (SDR), they cannot meet the application requirements of MR displays, which require front-end processing of the color gamut, hue, and color temperature of the video see-through (VST) and virtual reality (VR) layer signals, respectively, and then back-end processing of the color gamut, hue, and color temperature of the mixed layer signal after the layers are mixed.

[0004] In order to overcome the above-mentioned defects of the existing technology, the field urgently needs an image processing technology for mixed reality display, which is used to provide flexible color processing functions in different color spaces of virtual layers, real layers and mixed layers, so as to meet the application requirements of mixed reality display.

[0005] Summary of the Invention

[0006] The following is a brief summary of one or more aspects to provide a basic understanding of these aspects. This summary is not an exhaustive overview of all conceivable aspects and is neither intended to identify key or critical elements of all aspects nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that will be provided later.

[0007] In order to overcome the above-mentioned defects of the prior art, the present invention provides an image processor, an image processing method, a mixed reality display system, and a computer-readable storage medium, which can support front-end color processing of different color gamuts, different hues and different color temperatures for VST signals and VR layer signals respectively, and back-end color processing of corresponding color gamuts, corresponding hues and corresponding color temperatures for mixed layer signals after layer mixing, thereby meeting the application requirements of mixed reality display.

[0008] Specifically, the above-mentioned image processor provided according to the first aspect of the present invention includes: a first color management module, whose input end is connected to the rendering module, for obtaining virtual layer data from the rendering module and performing a first color processing on it; a second color management module, whose input end is connected to the camera module, for obtaining real layer data from the camera module and performing a second color processing on it; a layer blending module, whose input end is respectively connected to the first color management module and the second color management module, for performing layer blending processing on the virtual layer data after the first color processing and the real layer data after the second color processing to determine the mixed image data; and a third color management module, whose input end is connected to the layer blending module, for obtaining the mixed image data from the layer blending module, performing a third color processing on it, and outputting the mixed image data after the third color processing to the mixed reality display.

[0009] Furthermore, in one embodiment of the present invention, the image processor further includes a communication interface. The communication interface connects the color processing firmware unit and each of the color management modules. The first color management module is configured to: obtain first register data from the color processing firmware unit via the communication interface based on the requirements of the first color processing, and return first statistical information to the color processing firmware unit. The second color management module is configured to: obtain second register data from the color processing firmware unit via the communication interface based on the requirements of the second color processing, and return second statistical information to the color processing firmware unit. The third color management module is configured to: obtain third register data from the color processing firmware unit via the communication interface based on the requirements of the third color processing, and return third statistical information to the color processing firmware unit.

[0010] Furthermore, in one embodiment of the present invention, each of the color management modules includes a tone mapping unit, a color gamut mapping unit and a color temperature processing unit, and is configured to: obtain register data related to tone mapping, color gamut mapping and / or color temperature processing from the color processing firmware unit via the communication interface according to color processing requirements, and return statistical information related to the tone mapping, the color gamut mapping and / or the color temperature processing to the color processing firmware unit.

[0011] Furthermore, in one embodiment of the present invention, each of the color management modules is also configured to: perform histogram equalization processing on the corresponding virtual layer data, real layer data or mixed image data according to the register data related to the tone mapping to determine the statistical information related to the tone mapping; and / or in response to the color processing requirements of converting from a large color gamut to a small color gamut, perform step-by-step mapping processing on the corresponding virtual layer data, real layer data or mixed image data according to the register data related to the color gamut mapping to determine the statistical information related to the color gamut mapping.

[0012] Furthermore, in one embodiment of the present invention, each of the color management modules also includes a ReGamma unit and / or a DeGamma unit, and is configured to: obtain register data related to ReGamma processing and / or DeGamma processing from the color processing firmware unit via the communication interface according to the color processing requirements, and return statistical information related to the ReGamma processing and / or the DeGamma processing to the color processing firmware unit.

[0013] Furthermore, in one embodiment of the present invention, the rendering module also inputs HDR metadata into the color processing firmware unit. Each of the color management modules further includes an HDR unit, and is configured to: obtain register data related to HDR processing from the color processing firmware unit via the communication interface according to the color processing requirements, and return statistical information related to the HDR processing to the color processing firmware unit, wherein the register data related to HDR processing is updated and determined based on the HDR metadata and the returned statistical information related to HDR processing.

[0014] Furthermore, in one embodiment of the present invention, the image processor further includes the color processing firmware unit. The color processing firmware unit is configured to: obtain external light parameter information; and generate a tone curve, generate a color gamut mapping function, perform a color temperature adjustment operation, and / or perform a brightness adjustment operation based on the light parameter information and statistical information returned by at least one of the color management modules, so as to update the register data.

[0015] Furthermore, in an embodiment of the present invention, the data size of the first register data and the second register data is smaller than the data size of the third register data. The data size of the first statistical information and the second statistical information is smaller than the data size of the third statistical information.

[0016] Furthermore, in one embodiment of the present invention, the rendering module is further configured to render and generate one or more UI layer data. The first color management module is further configured to: obtain the one or more UI layer data from the rendering module, perform one or more corresponding fourth color processing on the one or more UI layer data; and input the UI layer data subjected to the one or more fourth color processing into the layer blending module for layer blending processing to determine corresponding blended image data.

[0017] Furthermore, in one embodiment of the present invention, the image processor further includes an image signal processing module. The second color management module is connected to the camera module via the image signal processing module. The image signal processing module is configured to: obtain video stream data captured by the camera module; and perform AWB processing and / or AE processing on the video stream data to generate the reality layer data.

[0018] In addition, the above-mentioned mixed reality display system provided according to the second aspect of the present invention includes: a rendering module for generating virtual layer data; a camera module for collecting real layer data; and the above-mentioned image processor provided by the first aspect of the present invention, connected to the rendering module and the camera module to obtain the virtual layer data and the real layer data, and perform image processing based on the virtual layer data and the real layer data to output mixed image data to be displayed.

[0019] In addition, the above-mentioned image processing method provided according to the third aspect of the present invention includes the following steps: obtaining virtual layer data from a rendering module and inputting it into a first color management module; obtaining real layer data from a camera module and inputting it into a second color management module; obtaining color processing requirements; according to the color processing requirements, controlling the first color management module to perform first color processing on the virtual layer data, and / or controlling the second color management module to perform second color processing on the real layer data; performing layer mixing processing on the layer data of the first color management module and the second color management module via a layer mixing module to determine mixed image data; inputting the mixed image data into a third color management module, and performing third color processing on it according to the color processing requirements; and transmitting the mixed image data output by the third color management module to a mixed reality display.

[0020] Furthermore, the computer-readable storage medium provided in accordance with the fourth aspect of the present invention stores computer instructions, which, when executed by a processor, implement the image processing method provided in accordance with the third aspect of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above features and advantages of the present invention will be better understood after reading the detailed description of the embodiments of the present disclosure in conjunction with the following drawings. In the drawings, the components are not necessarily drawn to scale, and components with similar related properties or characteristics may have the same or similar reference numerals.

[0022] FIG1 shows a schematic structural diagram of a mixed reality display system according to some embodiments of the present invention.

[0023] FIG2 shows a flowchart of an image processing method according to some embodiments of the present invention.

[0024] FIG3 shows a schematic flow chart of processing an image in a typical data format according to some embodiments of the present invention.

[0025] FIG4 shows a schematic flow chart of processing an image in a typical data format according to some embodiments of the present invention. DETAILED DESCRIPTION

[0026] The following specific embodiments illustrate the embodiments of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Although the description of the present invention will be introduced in conjunction with the preferred embodiment, this does not mean that the features of this invention are limited to this embodiment. On the contrary, the purpose of introducing the invention in conjunction with the embodiment is to cover other options or modifications that may be extended based on the claims of the present invention. In order to provide a deep understanding of the present invention, the following description will include many specific details. The present invention can also be implemented without using these details. In addition, in order to avoid confusion or blurring the focus of the present invention, some specific details will be omitted in the description.

[0027] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0028] Furthermore, the terms "upper," "lower," "left," "right," "top," "bottom," "horizontal," and "vertical" used in the following description should be understood to refer to the orientations depicted in that section and the accompanying drawings. These relative terms are used solely for convenience of description and do not necessarily imply that the devices described herein must be manufactured or operated in a specific orientation. Therefore, they should not be construed as limiting the present invention.

[0029] It will be understood that although the terms "first," "second," "third," etc. may be used herein to describe various components, regions, layers, and / or portions, these components, regions, layers, and / or portions should not be limited by these terms, and these terms are merely used to distinguish different components, regions, layers, and / or portions. Thus, a first component, region, layer, and / or portion discussed below may be referred to as a second component, region, layer, and / or portion without departing from some embodiments of the present invention.

[0030] As mentioned above, in the field of Mixed Reality (MR) display, existing MR devices can only perform simple processing on the color and brightness of input images, but do not support the input and processing of High Dynamic Range (HDR) data. Although the existing color processing pipelines of image processing devices such as mobile phones can decode single-channel or multi-channel HDR signals and adapt to Standard Dynamic Range (SDR), they cannot meet the application requirements of Mixed Reality display, which requires first performing front-end processing of the color gamut, hue, and color temperature of the video see-through (VST) layer signal and the virtual reality (VR) layer signal, and then performing back-end processing of the color gamut, hue, and color temperature of the mixed layer signal after the layers are mixed.

[0031] In order to overcome the above-mentioned defects of the prior art, the present invention provides an image processor, an image processing method, a mixed reality display system, and a computer-readable storage medium, which can support front-end color processing of different color gamuts, different hues and different color temperatures for VST signals and VR layer signals respectively, and back-end color processing of corresponding color gamuts, corresponding hues and corresponding color temperatures for mixed layer signals after layer mixing, thereby meeting the application requirements of mixed reality display.

[0032] In some non-limiting embodiments, the image processing method provided in the third aspect of the present invention can be implemented via the mixed reality display system provided in the second aspect of the present invention. Specifically, please refer to Figure 1, which shows a schematic diagram of the structure of a mixed reality display system provided in accordance with some embodiments of the present invention.

[0033] In the embodiment shown in FIG. 1 , the mixed reality display system is configured with a main processor 10 , a display pipeline 20 , an image signal processing (ISP) pipeline 30 , a camera module 40 , and a memory (not shown).

[0034] The memory includes but is not limited to the computer-readable storage medium provided in the fourth aspect of the present invention, on which are stored computer instructions for executing one or more steps in the image processing method provided in the third aspect of the present invention.

[0035] The main processor 10 is configured with a rendering module such as a graphics processing unit (GPU) and a video codec module for generating virtual layer data and transmitting the virtual layer data to the first color management module 21 in the display pipeline 20 for first color processing.

[0036] The display pipeline 20 is configured as the above-mentioned image processor provided by the first aspect of the present invention, which is equipped with a first color management module 21, a second color management module 22, a third color management module 23, a layer blending module 24 and a post-stage module 25. It can be used as a coprocessor to implement the above-mentioned image processing method provided by the third aspect of the present invention by reading and executing the computer instructions stored in the above-mentioned memory, thereby cooperating with the main processor 10 to perform mixed reality image processing.

[0037] The camera module 40 can be connected to the display pipeline 20 via the ISP pipeline 30 to collect real layer data and transmit it to the second color management module 22 in the display pipeline 20 for second color processing.

[0038] The following will describe the working principles of the above-mentioned display pipeline 20 and mixed reality display system in conjunction with some embodiments of the image processing method. Those skilled in the art will understand that these embodiments of the image processing method are only some non-limiting implementation methods provided by the present invention, which are intended to clearly demonstrate the main concept of the present invention and provide some specific solutions that are convenient for the public to implement, rather than to limit all functions or all working modes of the display pipeline 20 and the mixed reality display system. Similarly, the display pipeline 20 and the mixed reality display system are also only a non-limiting implementation method provided by the present invention, and do not constitute a limitation on the execution subject and execution order of each step in the image processing method.

[0039] Please refer to FIG. 1 and FIG. 2 . FIG. 2 shows a flow chart of an image processing method according to some embodiments of the present invention.

[0040] As shown in Figures 1 and 2, during mixed reality image processing, the display pipeline 20 may first obtain virtual layer data from a rendering module such as the GPU of the main processor 10 and input the data into the first color management module 21. The virtual layer data may preferably include color-related information such as color temperature information generated by GPU rendering.

[0041] In addition, the display pipeline 20 can also obtain the real-world layer data collected by the camera module 40 via the ISP pipeline 30 and input it into the second color management module 22. Specifically, in the embodiment shown in FIG1 , the ISP pipeline 30 can preferably be configured with an automatic white balance (AWB) unit and / or an automatic exposure (AE) unit. The ISP pipeline 30 can first obtain the video stream data captured by the camera module, and then perform AWB processing on the obtained video stream data via the AWB unit, and / or perform AE processing on the obtained video stream data via the AE unit to generate the real-world layer data. In this way, the real-world layer data can also preferably include the AWB information and / or AE information generated by the ISP pipeline 30.

[0042] Afterwards, the display pipeline 20 can obtain the color processing requirements for the virtual layer data and the real layer data from the mixed reality display system, and control the first color management module 21 to perform the first color processing on the virtual layer data according to the color processing requirements, and / or control the second color management module 22 to perform the second color processing on the real layer data according to the color processing requirements.

[0043] Specifically, in the embodiment shown in FIG1 , the mixed reality display system may also be preferably configured with a color processing firmware unit 50. The color processing firmware unit 50 may collect external ambient light intensity information via sensors, obtain color temperature information generated by GPU rendering via the main processor 10, obtain AWB information and / or AE information generated by the ISP pipeline 30, and combine statistical information such as statistical histograms returned by each color management module 21 to 23 in the display pipeline 20 to generate a tone curve, generate a color gamut mapping function, perform color temperature adjustment operations, and / or brightness adjustment operations to generate and update register data.

[0044] In response to the request instruction for the first color processing, the first color management module 21 can obtain the corresponding first register data from the color processing firmware unit 50 via the communication interface (not shown) of the display pipeline 20 to perform the first color processing on the above-mentioned virtual layer data, and return the first statistical information generated by the processing to the color processing firmware unit 50 for it to regenerate and update the corresponding register data.

[0045] Furthermore, in the embodiment shown in FIG1 , the first color management module 21 may preferably be configured with a tone mapping unit, a color gamut mapping unit, and a color temperature processing unit. During the first color processing, the first color management module 21 may first obtain first register data related to the first color gamut, the first hue, and the first color temperature from the color processing firmware unit 50 via the communication interface to perform the first color processing related to tone mapping, color gamut mapping, and / or color temperature processing on the virtual layer data. The first statistical information related to the tone mapping, color gamut mapping, and / or color temperature processing generated by the processing may then be returned to the color processing firmware unit 50 for regeneration and updating of the corresponding register data.

[0046] Specifically, when determining statistical information related to tone mapping, the first color management module 21 may perform histogram equalization on the virtual layer data based on the first register data related to tone mapping to determine the statistical information related to tone mapping. Furthermore, when determining statistical information related to gamut mapping, in response to a color processing requirement for transitioning from a large color gamut to a small color gamut, the first color management module 21 may perform step-by-step mapping on the virtual layer data based on the first register data related to gamut mapping to determine the statistical information related to gamut mapping.

[0047] Furthermore, in the embodiment shown in FIG1 , the first color management module 21 may also preferably be configured with a ReGamma unit and / or a DeGamma unit. During the first color processing, the first color management module 21 may also obtain first register data related to the ReGamma processing and / or DeGamma processing from the color processing firmware unit 50 via the communication interface to perform the corresponding ReGamma processing and / or DeGamma processing on the virtual layer data. The module then returns the generated first statistical information related to the ReGamma processing and / or DeGamma processing to the color processing firmware unit 50 for regeneration and updating of the corresponding register data.

[0048] Furthermore, in the embodiment shown in FIG1 , an HDR unit may also be preferably configured in the first color management module 21. Accordingly, the color processing firmware unit 50 may also preferably obtain HDR metadata from a rendering module such as the GPU of the main processor 10, and generate and update corresponding register data in conjunction with the HDR metadata. During the first color processing, the first color management module 21 may also obtain first register data related to HDR processing from the color processing firmware unit 50 via a communication interface to perform HDR processing on the virtual layer data, and then return the statistical information related to HDR processing generated by the processing to the color processing firmware unit 50 for regeneration and updating of the corresponding register data.

[0049] In this way, the present invention can flexibly perform HDR processing, tone mapping, color gamut mapping, color temperature processing and / or ReGamma processing on the virtual layer data according to the actual first color processing request instruction received, and bypass the processing units not involved in the request instruction to obtain the virtual layer data required for layer mixing.

[0050] Similarly, in the embodiment shown in FIG1 , the second color management module 22 may also preferably be configured with an HDR unit, a tone mapping unit, a gamut mapping unit, a color temperature processing unit, a ReGamma unit, and / or a DeGamma unit. In response to a second color processing request, the second color management module 22 may also, as described above, retrieve second register data related to HDR processing, tone mapping, gamut mapping, color temperature processing, ReGamma processing, and / or DeGamma processing from the color processing firmware unit 50 via a communication interface (not shown) of the display pipeline 20 to perform HDR processing, tone mapping, gamut mapping, color temperature processing, ReGamma processing, and / or DeGamma processing on the above-mentioned real layer data, and bypass processing units not involved in the request to obtain the real layer data required for layer blending. Subsequently, the second color management module 22 may also, as described above, return the second statistical information generated by the processing to the color processing firmware unit 50 for regeneration and updating of the corresponding register data. The specific steps of acquiring register data, performing front-end color processing, and returning statistical information are similar to those in the embodiment of the first color management module 21 and will not be further described here.

[0051] By configuring two front-end color management modules 21-22 for virtual layer data and real layer data respectively, and providing bypass processing functions, the above-mentioned display pipeline 20 provided by the present invention can provide flexible color processing functions for virtual layer data and real layer data in different color spaces according to the actual needs of mixed reality display, thereby meeting the application requirements of mixed reality display.

[0052] Continuing with Figures 1 and 2 , the input terminals of the layer blending module 24 are connected to the first color management module 21 and the second color management module 22, respectively. After completing the first color processing and the second color processing, the display pipeline 20 can perform layer blending on the virtual layer data that has undergone the first color processing and the real layer data that has undergone the second color processing via the layer blending module 24 to determine blended image data. This blended image data is then input into the third color management module 23 for third color processing according to color processing requirements.

[0053] Here, the specific solution for layer blending processing of multiple layer data does not involve the technical improvement of the present invention and is therefore not elaborated on.

[0054] The input end of the third color management module 23 is connected to the output end of the layer blending module 24, which is preferably equipped with a tone mapping unit, a color gamut mapping unit, a color temperature processing unit, a ReGamma unit and / or a DeGamma unit. During the third color processing, in response to a request instruction for the third color processing, the third color management module 22 can obtain mixed image data from the layer blending module, and obtain third register data related to tone mapping, color gamut mapping, color temperature processing, ReGamma processing and / or DeGamma processing from the color processing firmware unit 50 via the communication interface (not shown) of the display pipeline 20 as described above, so as to perform tone mapping, color gamut mapping, color temperature processing, ReGamma processing and / or DeGamma processing on the above-mentioned mixed image data, and bypass processing of the processing units not involved in the request instruction to obtain the display image data required for mixed reality display. Afterwards, the third color management module 23 can also return the third statistical information generated by the third color processing to the color processing firmware unit 50 as described above, so that the third statistical information can be regenerated and updated in the corresponding register data. The mixed image data (i.e., display image data) that has undergone the third color processing can be output to the mixed reality display 60, such as a VR display screen, via the post-stage module 25 for mixed reality display. Here, the specific steps of obtaining register data, performing front-end color processing, and returning statistical information are similar to those of the embodiment of the first color management module 21, and thus will not be further described.

[0055] Furthermore, in some embodiments of the present invention, the display pipeline 20 may preferentially configure the processing subunits involved in the third color processing according to the color processing requirements of the mixed reality display, and correspondingly configure the third register data, and then differentially configure the processing subunits and register data involved in the first and second color processing according to the requirements of the layer blending processing, so that the data volume of the first register data and the second register data involved in the above-mentioned front-end color management is smaller than the data volume of the third register data involved in the back-end color management, and the data volume of the above-mentioned first statistical information and the second statistical information is smaller than the data volume of the third statistical information. In this way, by configuring the data volume of the front-end color management to be smaller than the data volume of the back-end color management, the present invention can effectively save the hardware resources of the display pipeline 20 for storing and calculating duplicate data, thereby improving the color management capability of the image processor under the condition of the same hardware resources, or saving hardware resources under the condition of the same color management capability.

[0056] Those skilled in the art will understand that the above-mentioned embodiment of configuring the data volume of the front-end color management to be smaller than the data volume of the back-end color management is only a preferred solution provided by the present invention, which aims to provide a specific solution that is easy for the public to implement, and is not used to limit the scope of protection of the present invention.

[0057] Alternatively, in other embodiments, those skilled in the art may also use other methods to separately configure the register data for the first, second, and third color processing. Even if the data volume of the first register data and the second register data involved in the front-end color management is greater than or equal to the data volume of the third register data involved in the back-end color management, the normal operation of the display pipeline 20 is not affected.

[0058] In addition, for some mixed reality display applications involving human-computer interaction graphic interfaces, videos and other contents, the mixed reality display system can also generate one or more corresponding UI layer data through rendering by rendering modules such as the GPU of the main processor 10. Here, the UI layer data can be HDR layer data or SDR layer data. The first color management module 21 of the display pipeline 20 can preferably obtain the one or more UI layer data from the rendering module, and perform one or more corresponding fourth color processing on it according to the requirements of the fourth color processing. Afterwards, the first color management module 21 can also input the UI layer data that has undergone the one or more fourth color processing into the layer mixing module 24 for layer mixing processing to determine the corresponding mixed image data, thereby meeting the requirements of the mixed reality display application involving human-computer interaction graphic interfaces, videos and other contents.

[0059] In this way, by separately configuring a back-end color management module 23 for the mixed image data and providing a bypass processing function, the above-mentioned display pipeline 20 provided by the present invention can provide flexible color processing functions for the mixed image data in the corresponding color space according to the actual needs of mixed reality display, thereby further meeting the application needs of mixed reality display.

[0060] The following will continue to provide some non-limiting specific embodiments to illustrate the specific working methods of the display pipeline 20 and mixed reality display system provided by the present invention. Please refer to Figures 3 and 4, respectively, which are schematic diagrams of the process flow of processing images in typical data formats according to some embodiments of the present invention.

[0061] In the embodiment shown in FIG3 , the display pipeline 20 may first obtain virtual layer data generated by rendering from the GPU of the main processor 10, which has B11 bits, HDR, gamma, wide color gamut (rendering), and warm color temperature, and input the virtual layer data into the first color management module 21 for first color processing to obtain first data to be mixed, which has B12 bits, linearity, wide color gamut, and D65. Simultaneously, the display pipeline 20 may also obtain real layer data collected from the camera module 40 via the ISP pipeline 30, which has B13 bits, SDR, gamma 2.2, camera sensor color gamut, and cool color temperature, and input the real layer data into the second color management module 22 for second color processing to obtain second data to be mixed, which has B12 bits, linearity, wide color gamut, and D65.

[0062] The display pipeline 20 then performs layer blending processing on the first and second mixed data via the layer blending module 24 to determine mixed image data with B12 bits, linearity, wide color gamut, and D65. This mixed image data is then input into the third color management module 23 for third color processing to obtain image data to be displayed with B14 bits, SDR, gamma, screen color gamut, and warm color temperature. It will be appreciated that B11-B14 represent the bit depth of each image, and their specific values ​​can be manually set based on actual accuracy requirements. This does not involve the technical improvements of this application and is not discussed here.

[0063] In the embodiment shown in FIG4 , the display pipeline 20 may first obtain virtual layer data generated by rendering from the GPU of the main processor 10, which has B21 bits, sRGB, gamma, sRGB color gamut, and warm color temperature, and input the virtual layer data into the first color management module 21 for first color processing to obtain first data to be mixed, which has B22 bits, sRGB, gamma, sRGB color gamut, and D65. Simultaneously, the display pipeline 20 may also obtain real layer data collected from the camera module 40 via the ISP pipeline 30, which has B23 bits, sRGB, gamma, sRGB color gamut, and cold color temperature, and input the real layer data into the second color management module 22 for second color processing to obtain second data to be mixed, which has B22 bits, sRGB, gamma, sRGB color gamut, and D65.

[0064] The display pipeline 20 can then perform layer blending processing on the first and second mixed data via the layer blending module 24 to determine mixed image data with B22 bits, sRGB, gamma, sRGB color gamut, and D65. This mixed image data is then input into the third color management module 23 for third color processing to obtain image data to be displayed with B24 bits, sRGB, gamma, sRGB color gamut, and warm color temperature. It will be understood that B21-B24 represent the bit depth of each image, and their specific values ​​can be manually set based on actual accuracy requirements. This does not involve the technical improvements of the present application and is not elaborated on here.

[0065] It can be seen that by configuring a color management module 21~23 for virtual layer data, real layer data and mixed image data respectively, and providing a bypass processing function, the above-mentioned display pipeline 20 provided by the present invention can provide flexible color processing functions for virtual layer data, real layer data and mixed image data in different color spaces according to the actual needs of mixed reality display, thereby meeting the application needs of mixed reality display.

[0066] Those skilled in the art will understand that the display pipeline 20 shown in Figure 1 is only a non-limiting implementation of the above-mentioned image processor provided by the first aspect of the present invention, which is intended to clearly demonstrate the main concept of the present invention and provide a specific solution that is easy for the public to implement, rather than to limit the scope of protection of the present invention.

[0067] Optionally, in other embodiments, those skilled in the art may also integrate the display pipeline 20 with any one or more of the main processor 10, the ISP pipeline 30, and the color processing firmware unit 50 based on the above concept provided by the present invention to construct a composite image processor with corresponding multiple functions, which will not be described one by one here.

[0068] In summary, the above-mentioned image processor, mixed reality display system, image processing method, and computer-readable storage medium provided by the present invention not only support the input of external light parameter information, but also can replace multiple simple display pipelines with fewer and more complex display pipelines 20, thereby flexibly optimizing the display effects of virtual layer data and real layer data respectively, and allowing layers to be mixed under different conditions, so as to meet the color adjustment and management needs of various mixed reality applications.

[0069] Although the above methods are illustrated and described as a series of acts for simplicity of explanation, it is to be understood and appreciated that these methods are not limited by the order of the acts, as some acts may occur in a different order and / or concurrently with other acts from those illustrated and described herein or not illustrated and described herein but understandable to those skilled in the art according to one or more embodiments.

[0070] Those skilled in the art will appreciate that information, signals, and data may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips cited throughout the foregoing description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0071] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. A skilled person may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention.

[0072] The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0073] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processor, characterized in that: include: A first color management module, whose input end is connected to the rendering module, for obtaining virtual layer data from the rendering module and performing a first color processing on the virtual layer data; A second color management module, whose input end is connected to the camera module, is used to obtain the real layer data from the camera module and perform a second color processing on it; A layer blending module, whose input end is connected to the first color management module and the second color management module respectively, and is used to perform layer blending processing on the virtual layer data after the first color processing and the real layer data after the second color processing to determine blended image data; as well as A third color management module, whose input end is connected to the layer blending module, is used to obtain the mixed image data from the layer blending module, perform third color processing on the mixed image data, and output the mixed image data after the third color processing to the mixed reality display.

2. The image processor according to claim 1, wherein: It also includes a communication interface, wherein the communication interface is connected to the color processing firmware unit and each of the color management modules, The first color management module is configured to: obtain first register data from the color processing firmware unit via the communication interface according to the first color processing requirement, and return first statistical information to the color processing firmware unit, The second color management module is configured to: obtain second register data from the color processing firmware unit via the communication interface according to the second color processing requirement, and return second statistical information to the color processing firmware unit, The third color management module is configured to: obtain third register data from the color processing firmware unit via the communication interface according to the third color processing requirement, and return third statistical information to the color processing firmware unit.

3. The image processor according to claim 2, wherein: Each of the color management modules includes a tone mapping unit, a color gamut mapping unit and a color temperature processing unit, and is configured as follows: According to color processing requirements, register data related to tone mapping, color gamut mapping and / or color temperature processing is obtained from the color processing firmware unit via the communication interface, and statistical information related to the tone mapping, the color gamut mapping and / or the color temperature processing is returned to the color processing firmware unit.

4. The image processor according to claim 3, characterized in that Each of the color management modules is further configured to: According to the register data related to the tone mapping, a histogram equalization process is performed on the corresponding virtual layer data, the real layer data or the mixed image data to determine the statistical information related to the tone mapping; and / or In response to the color processing requirements of converting from a large color gamut to a small color gamut, the corresponding virtual layer data, real layer data or mixed image data is mapped step by step according to the register data related to the color gamut mapping to determine the statistical information related to the color gamut mapping.

5. The image processor according to claim 3, characterized in that: Each of the color management modules further includes a ReGamma unit and / or a DeGamma unit, and is configured as follows: According to the color processing requirements, register data related to ReGamma processing and / or DeGamma processing is obtained from the color processing firmware unit via the communication interface, and statistical information related to the ReGamma processing and / or the DeGamma processing is returned to the color processing firmware unit.

6. The image processor according to claim 3, characterized in that The rendering module also inputs HDR metadata into the color processing firmware unit. Each of the color management modules also includes an HDR unit and is configured as follows: According to the color processing requirements, register data related to HDR processing is obtained from the color processing firmware unit via the communication interface, and statistical information related to the HDR processing is returned to the color processing firmware unit, wherein the register data related to HDR processing is updated and determined based on the HDR metadata and the returned statistical information related to the HDR processing.

7. The image processor according to claim 3, characterized in that: Also included is the color processing firmware unit, wherein the color processing firmware unit is configured to: Acquiring external light parameter information; and According to the light parameter information and the statistical information returned by at least one of the color management modules, a tone curve is generated, a color gamut mapping function is generated, a color temperature adjustment operation is performed, and / or a brightness adjustment operation is performed to update the register data.

8. The image processor according to claim 2, wherein: The data amounts of the first register data and the second register data are smaller than the data amount of the third register data, and the data amounts of the first statistical information and the second statistical information are smaller than the data amount of the third statistical information.

9. The image processor according to claim 1, wherein: The rendering module is further used to render and generate one or more UI layer data, and the first color management module is further configured to: Acquire the one or more UI layer data from the rendering module, and perform one or more corresponding fourth color processing on the one or more UI layer data; as well as The UI layer data processed by the one or more fourth colors is input into the layer blending module for layer blending processing to determine corresponding blended image data.

10. The image processor according to claim 1, wherein: It also includes an image signal processing module, wherein the second color management module is connected to the camera module via the image signal processing module, and the image signal processing module is configured as follows: Acquiring video stream data captured by the camera module via the camera module; and The video stream data is subjected to AWB processing and / or AE processing to generate the reality layer data.

11. A mixed reality display system, characterized in that: include: Rendering module, used to generate virtual layer data; Camera module, used to collect real-world layer data; as well as The image processor as described in any one of claims 1 to 10 is connected to the rendering module and the camera module to obtain the virtual layer data and the real layer data, and performs image processing based on the virtual layer data and the real layer data to output mixed image data to be displayed.

12. An image processing method, characterized in that: The following steps are involved: Acquire virtual layer data from the rendering module and input it into the first color management module; Acquire the reality layer data from the camera module and input it into the second color management module; Get color processing requirements; According to the color processing requirement, controlling the first color management module to perform a first color processing on the virtual layer data, and / or controlling the second color management module to perform a second color processing on the real layer data; Performing layer blending processing on the layer data of the first color management module and the second color management module via a layer blending module to determine blended image data; Inputting the mixed image data into a third color management module, and performing third color processing on the mixed image data according to the color processing requirement; as well as The mixed image data output by the third color management module is transmitted to the mixed reality display.

13. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the image processing method according to claim 12 is implemented.

Citation Information

Patent Citations

  • Image processors, processing methods, storage media, and mixed reality display systems

    CN119012023B

  • Low latency mixed reality head wearable device

    CN108924540A

  • Data integration platform for virtual reality server

    CN113709520A

  • Creation and providing system of mixed reality using distance of reality image

    KR101921706B1

  • Augmented reality apparatus and method

    US20140240354A1