Image processors, processing methods, storage media, and mixed reality display systems

Abstract

This invention provides an image processor, a processing method, a storage medium, and a mixed reality display system. The image processor includes: a first color management module, whose input is connected to a rendering module, for acquiring virtual layer data from the rendering module and performing first color processing on it; a second color management module, whose input is connected to a camera module, for acquiring real-world layer data from the camera module and performing second color processing on it; a layer blending module, whose inputs are respectively connected to the first and second color management modules, for performing layer blending processing on the color-processed virtual layer data and real-world layer data to determine mixed image data; and a third color management module, whose input is connected to the layer blending module, for acquiring mixed image data from the layer blending module, performing third color processing on it, and outputting the third-color-processed mixed image data to a mixed reality display.

Description

Technical Field

[0001] This invention relates to the field of mixed reality display, and more particularly to an image processor, an image processing method, a mixed reality display system, and a computer-readable storage medium. Background Technology

[0002] In the field of Mixed Reality (MR) displays, existing MR devices can only perform simple processing on the color and brightness of input images, but do not support the input and processing of High Dynamic Range (HDR) data. Although the existing color processing pipelines of image processing devices such as mobile phones can decode single or multiple HDR signals and adapt them to Standard Dynamic Range (SDR), they cannot meet the application requirements of mixed reality displays, which require front-end processing of the Video See Through (VST) and Virtual Reality (VR) layer signals separately in terms of color gamut, hue, and color temperature, and then back-end processing of the mixed layer signal in terms of color gamut, hue, and color temperature.

[0003] In order to overcome the above-mentioned defects of the existing technology, there is an urgent need in the field for an image processing technology for mixed reality display, which can provide flexible color processing functions in different color spaces of virtual layer, reality layer and mixed layer, so as to meet the application requirements of mixed reality display. Summary of the Invention

[0004] The following provides a brief overview of one or more aspects to offer a basic understanding of them. This overview is not an exhaustive summary of all conceived aspects, nor is it intended to identify key or decisive elements of all aspects, nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed descriptions that follow.

[0005] To overcome the aforementioned deficiencies in the existing technology, the present invention provides an image processor, an image processing method, a mixed reality display system, and a computer-readable storage medium, which can support front-end color processing of VST signals and VR layer signals with different color gamuts, different hues, and different color temperatures, and back-end color processing of the mixed layer signals after layer mixing with corresponding color gamuts, corresponding hues, and corresponding color temperatures, thereby meeting the application requirements of mixed reality display.

[0006] Specifically, the image processor provided according to the first aspect of the present invention includes: a first color management module, the input of which is connected to a rendering module, for acquiring virtual layer data from the rendering module and performing a first color processing on it; a second color management module, the input of which is connected to a camera module, for acquiring real-world layer data from the camera module and performing a second color processing on it; a layer blending module, the inputs of which are respectively connected to the first color management module and the second color management module, for performing layer blending processing on the virtual layer data processed by the first color processing and the real-world layer data processed by the second color processing to determine blended image data; and a third color management module, the input of which is connected to the layer blending module, for acquiring the blended image data from the layer blending module, performing a third color processing on it, and outputting the blended image data processed by the third color processing to a mixed reality display.

[0007] Furthermore, in one embodiment of the present invention, the image processor further includes a communication interface. The communication interface connects the color processing firmware unit and each of the color management modules. The first color management module is configured to: obtain first register data from the color processing firmware unit via the communication interface according to the needs of the first color processing, and return first statistical information to the color processing firmware unit. The second color management module is configured to: obtain second register data from the color processing firmware unit via the communication interface according to the needs of the second color processing, and return second statistical information to the color processing firmware unit. The third color management module is configured to: obtain third register data from the color processing firmware unit via the communication interface according to the needs of the third color processing, and return third statistical information to the color processing firmware unit.

[0008] Furthermore, in one embodiment of the present invention, each of the color management modules includes a tone mapping unit, a gamut mapping unit, and a color temperature processing unit, and is configured to: obtain register data related to tone mapping, gamut mapping, and / or color temperature processing from the color processing firmware unit via the communication interface according to color processing requirements, and return statistical information related to tone mapping, gamut mapping, and / or color temperature processing to the color processing firmware unit.

[0009] Furthermore, in one embodiment of the present invention, each of the color management modules is further configured to: perform histogram equalization processing on the corresponding virtual layer data, real layer data, or mixed image data according to register data related to the tone mapping, in order to determine statistical information related to the tone mapping; and / or, in response to the color processing requirement of transitioning from a large color gamut to a small color gamut, perform step-by-step mapping processing on the corresponding virtual layer data, real layer data, or mixed image data according to register data related to the color gamut mapping, in order to determine statistical information related to the color gamut mapping.

[0010] Furthermore, in one embodiment of the present invention, each of the color management modules further includes a ReGamma unit and / or a DeGamma unit, and is configured to: obtain register data related to ReGamma processing and / or DeGamma processing from the color processing firmware unit via the communication interface according to the color processing requirements, and return statistical information related to the ReGamma processing and / or the DeGamma processing to the color processing firmware unit.

[0011] Furthermore, in one embodiment of the present invention, the rendering module also inputs HDR metadata into the color processing firmware unit. Each of the color management modules further includes an HDR unit, configured to: obtain register data related to HDR processing from the color processing firmware unit via the communication interface according to the color processing requirements, and return statistical information related to the HDR processing to the color processing firmware unit, wherein the register data related to HDR processing is updated and determined based on the HDR metadata and the returned statistical information related to the HDR processing.

[0012] Furthermore, in one embodiment of the present invention, the image processor further includes the color processing firmware unit. The color processing firmware unit is configured to: acquire external light parameter information; and, based on the light parameter information and statistical information returned by at least one of the color management modules, generate a hue curve, generate a color gamut mapping function, perform color temperature adjustment calculations and / or perform brightness adjustment calculations to update the register data.

[0013] Furthermore, in one embodiment of the present invention, the amount of data in the first register data and the second register data is less than the amount of data in the third register data. The amount of data in the first statistical information and the second statistical information is less than the amount of data in the third statistical information.

[0014] Furthermore, in one embodiment of the present invention, the rendering module is further configured to render and generate one or more UI layer data. The first color management module is further configured to: obtain the one or more UI layer data from the rendering module and perform corresponding one or more fourth color processing on it; and input the UI layer data processed by the one or more fourth colors into the layer blending module for layer blending processing to determine the corresponding blended image data.

[0015] Furthermore, in one embodiment of the present invention, the image processor further includes an image signal processing module. The second color management module is connected to the camera module via the image signal processing module. The image signal processing module is configured to: acquire video stream data captured by the camera module; and perform AWB processing and / or AE processing on the video stream data to generate the real-world layer data.

[0016] Furthermore, the mixed reality display system provided according to the second aspect of the present invention includes: a rendering module for generating virtual layer data; a camera module for acquiring real layer data; and the image processor provided according to the first aspect of the present invention, connected to the rendering module and the camera module, for acquiring the virtual layer data and the real layer data, and performing image processing based on the virtual layer data and the real layer data to output mixed image data to be displayed.

[0017] Furthermore, the image processing method provided by the third aspect of the present invention includes the following steps: obtaining virtual layer data from a rendering module and inputting it into a first color management module; obtaining real-world layer data from a camera module and inputting it into a second color management module; obtaining color processing requirements; controlling the first color management module to perform first color processing on the virtual layer data and / or controlling the second color management module to perform second color processing on the real-world layer data according to the color processing requirements; performing layer blending processing on the layer data of the first color management module and the second color management module via a layer blending module to determine blended image data; inputting the blended image data into a third color management module and performing third color processing on it according to the color processing requirements; and transmitting the blended image data output by the third color management module to a mixed reality display.

[0018] Furthermore, the computer-readable storage medium provided according to the fourth aspect of the present invention stores computer instructions. When the computer instructions are executed by a processor, the image processing method provided according to the third aspect of the present invention is implemented. Attached Figure Description

[0019] The above-described features and advantages of the present invention will be better understood after reading the following detailed description of embodiments of the present disclosure in conjunction with the accompanying drawings. In the drawings, components are not necessarily drawn to scale, and components having similar related characteristics or features may have the same or similar reference numerals.

[0020] Figure 1 A schematic diagram of the structure of a mixed reality display system provided according to some embodiments of the present invention is shown.

[0021] Figure 2 A flowchart of an image processing method provided according to some embodiments of the present invention is shown.

[0022] Figure 3 A schematic flowchart illustrating the processing of images in typical data formats according to some embodiments of the present invention is shown.

[0023] Figure 4 A schematic flowchart illustrating the processing of images in typical data formats according to some embodiments of the present invention is shown. Detailed Implementation

[0024] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Although the description of the present invention is presented in conjunction with preferred embodiments, this does not mean that the features of the invention are limited to these embodiments. On the contrary, the purpose of describing the invention in conjunction with embodiments is to cover other options or modifications that may be derived based on the claims of the present invention. To provide a thorough understanding of the invention, many specific details will be included in the following description. The invention may also be implemented without using these details. Furthermore, to avoid confusion or obscuring the focus of the invention, some specific details will be omitted in the description.

[0025] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0026] Furthermore, the terms "upper," "lower," "left," "right," "top," "bottom," "horizontal," and "vertical" used in the following description should be understood as the orientations shown in the relevant paragraphs and accompanying drawings. These relative terms are for illustrative purposes only and do not imply that the described apparatus must be manufactured or operated in a specific orientation, and therefore should not be construed as limiting the invention.

[0027] It is understood that although terms such as "first," "second," and "third" may be used herein to describe various components, regions, layers, and / or parts, these components, regions, layers, and / or parts should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers, and / or parts. Therefore, the first components, regions, layers, and / or parts discussed below may be referred to as second components, regions, layers, and / or parts without departing from some embodiments of the present invention.

[0028] As mentioned above, in the field of Mixed Reality (MR) displays, existing MR devices can only perform simple processing on the color and brightness of input images, but do not support the input and processing of High Dynamic Range (HDR) data. Although the existing color processing pipelines of image processing devices such as mobile phones can decode single or multiple HDR signals and adapt them to Standard Dynamic Range (SDR), they cannot meet the application requirements of mixed reality displays, which require first performing front-end processing (color gamut, hue, and color temperature) on the Video See Through (VST) layer signal and the Virtual Reality (VR) layer signal respectively, and then performing back-end processing (color gamut, hue, and color temperature) on the mixed layer signal after layer blending.

[0029] To overcome the aforementioned deficiencies in the existing technology, the present invention provides an image processor, an image processing method, a mixed reality display system, and a computer-readable storage medium, which can support front-end color processing of VST signals and VR layer signals with different color gamuts, different hues, and different color temperatures, and back-end color processing of the mixed layer signals after layer mixing with corresponding color gamuts, corresponding hues, and corresponding color temperatures, thereby meeting the application requirements of mixed reality display.

[0030] In some non-limiting embodiments, the image processing method provided in the third aspect of the present invention can be implemented via the mixed reality display system provided in the second aspect of the present invention. Please refer to [reference needed] for details. Figure 1 , Figure 1 A schematic diagram of the structure of a mixed reality display system provided according to some embodiments of the present invention is shown.

[0031] exist Figure 1 In the illustrated embodiment, the mixed reality display system includes a main processor 10, a display pipeline 20, an image signal processing (ISP) pipeline 30, a camera module 40, and a memory (not shown).

[0032] The memory includes, but is not limited to, the computer-readable storage medium provided in the fourth aspect of the present invention, having stored thereon computer instructions for performing one or more steps of the image processing method provided in the third aspect of the present invention.

[0033] The main processor 10 is equipped with a graphics processing unit (GPU) and a video codec module to generate virtual layer data and transmit it to the first color management module 21 in the display pipeline 20 for first color processing.

[0034] The display pipeline 20 is configured as the image processor provided in the first aspect of the present invention, wherein it is equipped with a first color management module 21, a second color management module 22, a third color management module 23, a layer blending module 24 and a post-processor 25, and can be used as a coprocessor to implement the image processing method provided in the third aspect of the present invention by reading and executing computer instructions stored in the memory, thereby cooperating with the main processor 10 to perform mixed reality image processing.

[0035] The camera module 40 can be connected to the display pipeline 20 via the ISP pipeline 30 to acquire real-world layer data and transmit it to the second color management module 22 in the display pipeline 20 for second color processing.

[0036] The working principle of the display pipeline 20 and the mixed reality display system described below will be described with reference to some embodiments of image processing methods. Those skilled in the art will understand that these embodiments of image processing methods are merely non-limiting implementations provided by the present invention, intended to clearly demonstrate the main concepts of the invention and provide specific solutions convenient for public implementation, rather than limiting all functions or all operating methods of the display pipeline 20 and the mixed reality display system. Similarly, the display pipeline 20 and the mixed reality display system are also only one non-limiting implementation provided by the present invention, and do not limit the executing entity or execution order of each step in the image processing method.

[0037] Please refer to the reference. Figure 1 and Figure 2 , Figure 2A flowchart of an image processing method provided according to some embodiments of the present invention is shown.

[0038] like Figure 1 and Figure 2 As shown, during mixed reality image processing, the display pipeline 20 can first obtain virtual layer data from the rendering module such as the GPU of the main processor 10 and input it into the first color management module 21. Here, the virtual layer data can preferably include color-related information such as color temperature information generated by GPU rendering.

[0039] Furthermore, the display pipeline 20 can also acquire the real-world layer data captured by the camera module 40 via the ISP pipeline 30 and input it into the second color management module 22. Specifically, in Figure 1 In the illustrated embodiment, the ISP pipeline 30 may preferably be configured with an Automatic White Balance (AWB) unit and / or an Automatic Exposure (AE) unit. The ISP pipeline 30 may first acquire video stream data captured by the camera module, then perform AWB processing on the acquired video stream data via the AWB unit, and / or perform AE processing on the acquired video stream data via the AE unit, to generate the aforementioned reality layer data. Thus, the reality layer data may also preferably include AWB information and / or AE information generated by the ISP pipeline 30.

[0040] Subsequently, the display pipeline 20 can obtain the color processing requirements for the virtual layer data and the real layer data from the mixed reality display system, and control the first color management module 21 to perform first color processing on the virtual layer data according to the color processing requirements, and / or control the second color management module 22 to perform second color processing on the real layer data according to the color processing requirements.

[0041] Specifically, in Figure 1 In the illustrated embodiment, the mixed reality display system may also preferably be configured with a color processing firmware unit 50. This color processing firmware unit 50 can acquire ambient light intensity information from a sensor, obtain color temperature information generated by GPU rendering via the main processor 10, and obtain AWB information and / or AE information generated by the ISP pipeline 30. It then combines this information with statistical information such as statistical histograms returned by the color management modules 21-23 in the display pipeline 20 to generate hue curves, generate color gamut mapping functions, perform color temperature adjustment calculations, and / or brightness adjustment calculations to generate and update register data.

[0042] In response to the request command for the first color processing, the first color management module 21 can obtain the corresponding first register data from the color processing firmware unit 50 through the communication interface (not shown) of the display pipeline 20 to perform the first color processing on the virtual layer data, and return the first statistical information generated by the processing to the color processing firmware unit 50 so that it can regenerate and update the corresponding register data.

[0043] Furthermore, in Figure 1 In the illustrated embodiment, the first color management module 21 may preferably be configured with a tone mapping unit, a gamut mapping unit, and a color temperature processing unit. During the first color processing, the first color management module 21 may first obtain first register data involving the first color gamut, the first hue, and the first color temperature from the color processing firmware unit 50 via a communication interface, so as to perform first color processing on the virtual layer data regarding tone mapping, gamut mapping, and / or color temperature processing, and then return the first statistical information involving tone mapping, gamut mapping, and / or color temperature processing generated by the processing to the color processing firmware unit 50, so that it can regenerate and update the corresponding register data.

[0044] Specifically, when determining statistical information related to tone mapping, the first color management module 21 can perform histogram equalization on the virtual layer data based on the first register data related to tone mapping to determine the statistical information related to tone mapping. Furthermore, when determining statistical information related to color gamut mapping, in response to the color processing requirement of transitioning from a large color gamut to a small color gamut, the first color management module 21 can perform step-by-step mapping processing on the virtual layer data based on the first register data related to color gamut mapping to determine the statistical information related to color gamut mapping.

[0045] In addition, Figure 1 In the illustrated embodiment, the first color management module 21 may preferably be configured with a ReGamma unit and / or a DeGamma unit. During the first color processing, the first color management module 21 may also obtain first register data involving ReGamma processing and / or DeGamma processing from the color processing firmware unit 50 via a communication interface, so as to perform corresponding ReGamma processing and / or DeGamma processing on the virtual layer data, and then return the first statistical information involving ReGamma processing and / or DeGamma processing generated by the processing to the color processing firmware unit 50, so that it can regenerate and update the corresponding register data.

[0046] In addition, Figure 1In the illustrated embodiment, the first color management module 21 may preferably also be configured with an HDR unit. Correspondingly, the color processing firmware unit 50 may also preferably obtain HDR metadata from the rendering module such as the GPU of the main processor 10, and combine the HDR metadata to generate and update the corresponding register data. During the first color processing, the first color management module 21 may also obtain the first register data related to HDR processing from the color processing firmware unit 50 via the communication interface to perform HDR processing on the virtual layer data, and then return the statistical information related to HDR processing generated by the processing to the color processing firmware unit 50 for it to regenerate and update the corresponding register data.

[0047] Thus, the present invention can flexibly perform HDR processing, tone mapping, color gamut mapping, color temperature processing and / or ReGamma processing on the virtual layer data according to the actual received first color processing request instruction, and perform bypass processing on the processing units not involved in the request instruction, so as to obtain the virtual layer data required for layer blending.

[0048] Similarly, in Figure 1 In the illustrated embodiment, the second color management module 22 may preferably be configured with an HDR unit, a tone mapping unit, a color gamut mapping unit, a color temperature processing unit, a ReGamma unit, and / or a DeGamma unit. In response to a request command for second color processing, the second color management module 22 may, as described above, obtain second register data relating to HDR processing, tone mapping, color gamut mapping, color temperature processing, ReGamma processing, and / or DeGamma processing from the color processing firmware unit 50 via the communication interface (not shown) of the display pipeline 20. This data is used to perform HDR processing, tone mapping, color gamut mapping, color temperature processing, ReGamma processing, and / or DeGamma processing on the aforementioned real-world layer data, and to bypass processing units not involved in the request command, thereby obtaining the real-world layer data required for layer blending. Afterwards, the second color management module 22 may, as described above, return the second statistical information generated during processing to the color processing firmware unit 50 for regeneration and updating of the corresponding register data. The specific steps of obtaining register data, performing front-end color processing, and returning statistical information are similar to those in the embodiment of the first color management module 21, and will not be elaborated further here.

[0049] By configuring two front-end color management modules 21-22 for virtual layer data and real layer data respectively, and providing bypass processing functionality, the display pipeline 20 provided by the present invention can provide flexible color processing functions for virtual layer data and real layer data in different color spaces according to the actual needs of mixed reality display, thereby meeting the application requirements of mixed reality display.

[0050] Please continue to refer to this. Figure 1 and Figure 2 The input terminals of the layer blending module 24 are connected to the first color management module 21 and the second color management module 22, respectively. After completing the first color processing and the second color processing, the display pipeline 20 can perform layer blending processing on the virtual layer data processed by the first color and the real layer data processed by the second color through the layer blending module 24 to determine the blended image data. Then, the blended image data is input into the third color management module 23 for third color processing according to the color processing requirements.

[0051] The specific scheme for layer blending of multiple layers of data does not involve the technical improvements of this invention, and therefore will not be elaborated upon here.

[0052] The input of the third color management module 23 is connected to the output of the layer blending module 24, and preferably includes a tone mapping unit, a gamut mapping unit, a color temperature processing unit, a ReGamma unit, and / or a DeGamma unit. During the third color processing, in response to a request command for third color processing, the third color management module 22 can obtain blended image data from the layer blending module, and as described above, obtains third register data involving tone mapping, gamut mapping, color temperature processing, ReGamma processing, and / or DeGamma processing from the color processing firmware unit 50 via the communication interface (not shown) of the display pipeline 20. This data is then used to perform tone mapping, gamut mapping, color temperature processing, ReGamma processing, and / or DeGamma processing on the blended image data, and to bypass processing units not involved in the request command, thereby obtaining the display image data required for the mixed reality display. Subsequently, the third color management module 23 can also return the third statistical information generated by the third color processing to the color processing firmware unit 50 as described above, so that it can regenerate and update the corresponding register data, and output the mixed image data (i.e., display image data) processed by the third color processing to the mixed reality display 60 such as a VR display screen via the back-end module 25 for mixed reality display. Here, the specific steps of obtaining register data, performing front-end color processing and returning statistical information are similar to those in the embodiment of the first color management module 21, and therefore will not be described in detail.

[0053] Furthermore, in some embodiments of the present invention, the display pipeline 20 can preferentially configure the processing sub-units involved in the third color processing according to the color processing requirements of mixed reality display, and correspondingly configure the third register data. Then, according to the requirements of layer blending processing, the processing sub-units and register data involved in the first and second color processing are configured differentially. This results in the amount of data in the first and second registers involved in the front-end color management being less than the amount of data in the third registers involved in the back-end color management, and the amount of data in the first and second statistical information being less than the amount of data in the third statistical information. In this way, by configuring the amount of data in the front-end color management to be less than the amount of data in the back-end color management, the present invention can effectively save hardware resources for the display pipeline 20 to store and process repetitive data, thereby improving the color management capability of the image processor under the same hardware resource conditions, or saving hardware resources under the same color management capability conditions.

[0054] Those skilled in the art will understand that the above-described embodiment, which configures the amount of data for front-end color management to be less than the amount of data for back-end color management, is merely a preferred embodiment provided by the present invention and is intended to provide a specific solution that is easy for the public to implement, rather than being used to limit the scope of protection of the present invention.

[0055] Alternatively, in other embodiments, those skilled in the art can also use other methods to configure the register data for the first, second, and third color processing respectively. Even if the amount of data in the first and second registers involved in the aforementioned front-end color management is greater than or equal to the amount of data in the third register involved in the back-end color management, it will not affect the normal operation of the display pipeline 20.

[0056] Furthermore, for mixed reality display applications involving human-computer interaction graphical interfaces, videos, and other content, the mixed reality display system can also render and generate one or more corresponding UI layer data via the rendering module such as the GPU of the main processor 10. Here, the UI layer data can be HDR layer data or SDR layer data. The first color management module 21 of the display pipeline 20 can preferably obtain the one or more UI layer data from the rendering module and perform one or more fourth color processing according to the requirements of fourth color processing. Afterwards, the first color management module 21 can also input the UI layer data processed by the one or more fourth color processing into the layer blending module 24 for layer blending processing to determine the corresponding blended image data, thereby meeting the requirements of the mixed reality display application involving human-computer interaction graphical interfaces, videos, and other content.

[0057] Thus, by configuring a separate backend color management module 23 for the mixed image data and providing bypass processing functionality, the display pipeline 20 provided by the present invention can provide flexible color processing functionality for the mixed image data in the corresponding color space according to the actual needs of mixed reality display, thereby further meeting the application requirements of mixed reality display.

[0058] The following will continue to provide some non-limiting specific embodiments, thereby demonstrating the specific operation of the display pipeline 20 and mixed reality display system provided by the present invention. Please refer to the following for details. Figure 3 and Figure 4 , Figure 3 and Figure 4 Schematic flowcharts illustrating the processing of images in typical data formats according to some embodiments of the present invention are shown respectively.

[0059] exist Figure 3 In the illustrated embodiment, the display pipeline 20 can first obtain virtual layer data (B11 bit, HDR, gamma, wide color gamut, and warm color temperature) generated by the GPU of the main processor 10, and input it into the first color management module 21 for first color processing to obtain first data to be mixed (B12 bit, linear, wide color gamut, and D65). Simultaneously, the display pipeline 20 can also obtain real-world layer data (B13 bit, SDR, gamma 2.2, camera sensor color gamut, and cool color temperature) collected by the camera module 40 via the ISP pipeline 30, and input it into the second color management module 22 for second color processing to obtain second data to be mixed (B12 bit, linear, wide color gamut, and D65).

[0060] Subsequently, the display pipeline 20 can perform layer blending processing on the first and second data to be blended via the layer blending module 24 to determine the blended image data with B12 bits, linearity, wide color gamut, and D65, and input it into the third color management module 23 for third color processing to obtain the image data to be displayed with B14 bits, SDR, gamma, screen color gamut, and warm color temperature. It is understood that B11 to B14 represent the bit depth of each image, and their specific values ​​can be manually set according to actual precision requirements, without involving any technical improvement of this application, and will not be elaborated upon here.

[0061] exist Figure 4In the illustrated embodiment, the display pipeline 20 can first obtain virtual layer data (B21-bit, sRGB, gamma, sRGB color gamut, and warm color temperature) generated by the GPU of the main processor 10, and input it into the first color management module 21 for first color processing to obtain first data to be mixed (B22-bit, sRGB, gamma, sRGB color gamut, and D65). Simultaneously, the display pipeline 20 can also obtain real-world layer data (B23-bit, sRGB, gamma, sRGB color gamut, and cool color temperature) captured by the camera module 40 via the ISP pipeline 30, and input it into the second color management module 22 for second color processing to obtain second data to be mixed (B22-bit, sRGB, gamma, sRGB color gamut, and D65).

[0062] Subsequently, the display pipeline 20 can perform layer blending processing on the first and second data to be blended via the layer blending module 24 to determine the mixed image data with B22 bits, sRGB, gamma, sRGB color gamut, and D65, and input it into the third color management module 23 for third color processing to obtain the image data to be displayed with B24 bits, sRGB, gamma, sRGB color gamut, and warm color temperature. It is understood that B21 to B24 represent the bit depth of each image, and their specific values ​​can be manually set according to actual precision requirements, without involving any technical improvement of this application, and will not be elaborated upon here.

[0063] Therefore, by configuring a color management module 21-23 for virtual layer data, real layer data, and mixed image data respectively, and providing bypass processing functionality, the display pipeline 20 provided by the present invention can provide flexible color processing functions for virtual layer data, real layer data, and mixed image data in different color spaces according to the actual needs of mixed reality display, thereby meeting the application requirements of mixed reality display.

[0064] Those skilled in the art will understand that Figure 1 The display pipeline 20 shown is only a non-limiting embodiment of the image processor described above in the first aspect of the present invention, intended to clearly illustrate the main concept of the present invention and provide a specific solution that is easy for the public to implement, rather than intended to limit the scope of protection of the present invention.

[0065] Alternatively, in other embodiments, those skilled in the art can also integrate the display pipeline 20 with any one or more of the main processor 10, ISP pipeline 30, and color processing firmware unit 50 based on the above-mentioned concept provided by the present invention to construct a composite image processor with multiple corresponding functions, which will not be described in detail here.

[0066] In summary, the image processor, mixed reality display system, image processing method, and computer-readable storage medium provided by this invention not only support the input of external light parameter information, but also replace multiple simple display pipelines with fewer and more complex display pipelines 20, thereby flexibly optimizing the display effects of virtual layer data and real layer data respectively, and allowing the layers to be mixed under different conditions, so as to meet the color adjustment and management needs of various mixed reality applications.

[0067] Although the methods described above are illustrated and depicted as a series of actions for the sake of simplicity, it should be understood and appreciated that these methods are not limited by the order of the actions, as some actions may occur in a different order and / or concurrently with other actions from the illustrations and descriptions herein or not illustrated and described herein but which may be understood by those skilled in the art, according to one or more embodiments.

[0068] Those skilled in the art will understand that information, signals, and data can be represented using any of a variety of different techniques and arts. For example, the data, instructions, commands, information, signals, bits, symbols, and chips described throughout the above description can be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof.

[0069] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps are described above in a generalized manner in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the invention.

[0070] The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or performed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but in alternatives, it may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.

[0071] The prior description of this disclosure is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not intended to be limited to the examples and designs described herein, but should be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processor, characterized in that, include: The first color management module has its input end connected to the rendering module, and is used to obtain virtual layer data from the rendering module and perform first color processing on it; The second color management module has its input end connected to the camera module, and is used to obtain the real-world layer data from the camera module and perform second color processing on it; A layer blending module, whose input terminals are respectively connected to the first color management module and the second color management module, is used to perform layer blending processing on the virtual layer data processed by the first color and the real layer data processed by the second color, so as to determine the blended image data; as well as The third color management module has its input end connected to the layer blending module. It is used to obtain the blended image data from the layer blending module, perform third color processing on it, and output the blended image data after the third color processing to the mixed reality display. Each of the color management modules includes a tone mapping unit, a color gamut mapping unit, and a color temperature processing unit.

2. The image processor as described in claim 1, characterized in that, It also includes a communication interface, wherein the communication interface connects the color processing firmware unit and each of the color management modules. The first color management module is configured to: obtain first register data from the color processing firmware unit via the communication interface according to the requirements of the first color processing, and return first statistical information to the color processing firmware unit. The second color management module is configured to: obtain second register data from the color processing firmware unit via the communication interface according to the requirements of the second color processing, and return second statistical information to the color processing firmware unit. The third color management module is configured to: obtain third register data from the color processing firmware unit via the communication interface according to the requirements of the third color processing, and return third statistical information to the color processing firmware unit.

3. The image processor as described in claim 2, characterized in that, Each of the aforementioned color management modules is further configured as follows: According to the color processing requirements, register data involving tone mapping, gamut mapping and / or color temperature processing is obtained from the color processing firmware unit via the communication interface, and statistical information involving tone mapping, gamut mapping and / or color temperature processing is returned to the color processing firmware unit.

4. The image processor as described in claim 3, characterized in that, Each of the aforementioned color management modules is further configured as follows: Based on the register data related to the tone mapping, histogram equalization is performed on the corresponding virtual layer data, real layer data, or mixed image data to determine the statistical information related to the tone mapping; and / or In response to the color processing requirement of shifting from a large color gamut to a small color gamut, step-by-step mapping processing is performed on the corresponding virtual layer data, real layer data, or mixed image data based on the register data related to the color gamut mapping, in order to determine the statistical information related to the color gamut mapping.

5. The image processor as described in claim 3, characterized in that, Each of the aforementioned color management modules further includes a ReGamma unit and / or a DeGamma unit, and is configured as follows: According to the color processing requirements, register data related to ReGamma processing and / or DeGamma processing is obtained from the color processing firmware unit via the communication interface, and statistical information related to the ReGamma processing and / or DeGamma processing is returned to the color processing firmware unit.

6. The image processor as described in claim 3, characterized in that, The rendering module also inputs HDR metadata into the color processing firmware unit. Each color management module further includes an HDR unit, which is configured as follows: According to the color processing requirements, register data related to HDR processing is obtained from the color processing firmware unit via the communication interface, and statistical information related to HDR processing is returned to the color processing firmware unit. The register data related to HDR processing is updated and determined based on the HDR metadata and the returned statistical information related to HDR processing.

7. The image processor as claimed in claim 3, characterized in that, It also includes the color processing firmware unit, wherein the color processing firmware unit is configured to: Obtain external light parameter information; and Based on the light parameter information and statistical information returned by at least one of the color management modules, a hue curve is generated, a color gamut mapping function is generated, color temperature adjustment calculations and / or brightness adjustment calculations are performed to update the register data.

8. The image processor as described in claim 2, characterized in that, The amount of data in the first register and the second register is less than the amount of data in the third register, and the amount of data in the first statistical information and the second statistical information is less than the amount of data in the third statistical information.

9. The image processor as claimed in claim 1, characterized in that, The rendering module is also used to render and generate one or more UI layer data, and the first color management module is further configured to: The rendering module obtains one or more UI layer data and performs corresponding one or more fourth color processing on them; as well as The UI layer data that has undergone one or more fourth color processing is input into the layer blending module for layer blending processing to determine the corresponding blended image data.

10. The image processor as claimed in claim 1, characterized in that, It also includes an image signal processing module, wherein the second color management module is connected to the camera module via the image signal processing module, and the image signal processing module is configured to: The video stream data captured by the camera module is acquired via the camera module; and The video stream data is processed by AWB and / or AE to generate the real-world layer data.

11. A mixed reality display system, characterized in that, include: The rendering module is used to generate virtual layer data; The camera module is used to capture data from the real-world layers. as well as The image processor according to any one of claims 1 to 10 connects the rendering module and the camera module to obtain the virtual layer data and the real layer data, and performs image processing based on the virtual layer data and the real layer data to output mixed image data to be displayed.

12. An image processing method, characterized in that, Includes the following steps: Obtain virtual layer data from the rendering module and input it into the first color management module; The camera module acquires the real-world layer data and inputs it into the second color management module; Obtain color processing requirements; According to the color processing requirements, the first color management module is controlled to perform first color processing on the virtual layer data, and / or the second color management module is controlled to perform second color processing on the real layer data; The layer data of the first color management module and the second color management module are processed by the layer blending module to determine the blended image data; The mixed image data is input into a third color management module, and third color processing is performed on it according to the color processing requirements. Each color management module includes a tone mapping unit, a color gamut mapping unit, and a color temperature processing unit. The mixed image data output by the third color management module is transmitted to the mixed reality display.

13. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the computer instructions are executed by the processor, the image processing method as described in claim 12 is implemented.

Images