High dynamic range (HDR) image processing with adaptive color volume data mapping
By performing tone mapping and inverse tone mapping in perceptually uniform Jzazbz color space, the problem of HDR images reproduction on SDR display devices is solved, reducing the delay and calculation load of image processing, suitable for limited resource devices, and achieving high efficiency image processing.
Patent Information
- Application Number
- CN202410638165.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-24
- Filing Date
- 2024-05-22
- Publication Date
- 2024-11-26
AI Technical Summary
Prior art When processing high dynamic range (HDR) images, it is difficult to accurately reproduce the color, brightness, or contrast of an image on a standard dynamic range (SDR) display device, and the nonorthogonality of the YUV color space results in tone mapping and inverse tone. Mapping operations are complex, resource-consuming, and are not suitable for limited resource equipment.
The perceptually uniform Jzazbz color space is used for tone mapping and inverse tone mapping operations. Through the back-photoelectric transfer function and color volume data mapping technology, the image data is converted from RGB or YUV color space to Jzazbz color space, and based on the dynamics of the display device. Mapping ranges or gamuts reduces processing overhead and computational load.
Significantly reduce the delay and computational load of tone mapping and inverse tone mapping without losing accuracy, suitable for limited resource devices, supports high data accuracy and wide range programming options for video processing, suitable for low latency and low power application.
Smart Images

Figure CN119031107A_ABST
Abstract
Description
Technical Field
[0001] The present implementation relates generally to image processing, and in particular to high dynamic range (HDR) image processing with adaptive tone and color volume mapping. Background Art
[0002] Display devices (such as televisions, set-top boxes, computers, and mobile phones) may use imaging technologies different from those used by image capture devices (such as cameras and video recorders). Advances in display technology have led to improved capabilities, such as high dynamic range, wider color gamut, and migration from high-definition (HD) displays to ultra-high-definition (UHD) display technologies. As a result, image processing can be used to properly render images captured by devices with different system capabilities and standards on a given display. For example, a display device that can only display standard dynamic range (SDR) content may not be able to reproduce the full range of color, brightness, or contrast of an image captured in a high dynamic range (HDR) format. An image processor can modify the raw image data generated by an image capture device so that the image can be reproduced more accurately or realistically on a display device (such as to utilize the full dynamic range of the display).
[0003] Some image processing techniques (also known as "tone mapping") can weaken the color, brightness, or contrast of an HDR image to be rendered on an SDR display. Some other image processing techniques (also known as "inverse tone mapping") can enhance the color, brightness, or contrast of an SDR image to be rendered on an HDR display. Even when both the image capture device and the display device support HDR, image processing can further improve the quality of the displayed image due to the difference between the display environment (such as a television with electronically limited brightness, color, contrast, and resolution) and the image capture environment (such as a natural environment with unlimited brightness, color, contrast, and resolution). As image capture and display technology continues to advance, new image processing techniques are needed to bridge the capabilities (such as dynamic range or color gamut) between image capture devices and display devices. Summary of the invention
[0004] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.
[0005] One innovative aspect of the subject matter of the present disclosure can be implemented in a method of image processing. The method includes: receiving first image data representing an image generated by an image capture device based on an optoelectronic transfer function (OETF); transforming the first image data into second image data based on an inverse of the OETF, wherein the second image data includes a red (R) channel, a green (G) channel, and a blue (B) channel associated with an RGB color space; and converting the second image data into a color space having a color space associated with the RGB color space. z a z b z The brightness associated with the color space (J z ) channel, red-green (a z ) channel and the yellow-blue (b z ) channel; converting the J of the third image data based at least in part on a dynamic range or color gamut associated with the display device z Channel, a z Channel and b z Channel mapping to display data J z Channel, a z Channel and b z channel; and processing display data for display by a display device.
[0006] Another innovative aspect of the subject matter of the present disclosure can be implemented in an image processor, the image processor comprising a processing system, the processing system comprising hardware processes and software processes, the hardware processes and software processes causing the image processor to: receive first image data representing an image generated by an image capture device based on the OETF; transform the first image data into second image data based on the inverse of the OETF, wherein the second image data includes a red (R) channel, a green (G) channel, and a blue (B) channel associated with an RGB color space; and convert the second image data into a color space having a color space associated with the RGB color space. z a z b z The brightness associated with the color space (J z ) channel, red-green (a z ) channel and the yellow-blue (b z ) channel; converting the J of the third image data based at least in part on a dynamic range or color gamut associated with the display device z Channel, a z Channel and b z Channel mapping to display data J z Channel, a z Channel and b z channel; and processing display data for display by a display device. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] The present implementations are illustrated by way of example and are not intended to be limited by the figures of the accompanying drawings.
[0008] Figure 1 A block diagram of an example image capture and display system is shown in accordance with some implementations.
[0009] Figure 2 A block diagram of an example image processing system is shown in accordance with some implementations.
[0010] Figure 3 A block diagram of an example color volume data mapping (CVM) system is shown in accordance with some implementations.
[0011] Figure 4 A block diagram of an example system for global CVM is shown in accordance with some implementations.
[0012] Figure 5 A block diagram of an example system for local CVM is shown in accordance with some implementations.
[0013] Figure 6 An example of mapping global pixel tone values to local pixel tone values using adaptive histogram equalization is shown.
[0014] Figure 7 A block diagram illustrating an example adaptive detail processor according to some implementations.
[0015] Figure 8 Another block diagram of an example image processing system is shown in accordance with some implementations.
[0016] Fig. 9 An illustrative flow chart depicting example operations for image processing according to some implementations is shown. DETAILED DESCRIPTION
[0017] In the following description, many specific details, such as examples of specific components, circuits and processes, are set forth to provide a thorough understanding of the present disclosure. As used herein, the term "coupling" represents being directly connected to or connected through one or more intermediate components or circuits. The terms "electronic system" and "electronic device" can be used interchangeably to refer to any system capable of electronically processing information. Moreover, in the following description, and for the purpose of explanation, specific terms are set forth to provide a thorough understanding of aspects of the present disclosure. However, it will be apparent to those skilled in the art that these specific details may not be required to practice the exemplary embodiments. In other instances, well-known circuits and devices are shown in block diagram form to avoid making the present disclosure difficult to understand. The subsequent parts of the detailed description are presented in accordance with other symbolic representations of procedures, logic blocks, processing and the operation of data bits in a computer memory.
[0018] These descriptions and representations are the means by which the technical personnel in the data processing field use to most effectively communicate the essence of their work to other technical personnel in the field. In the present disclosure, procedures, logic blocks, processes, etc. are considered to be the self-consistent sequence of steps or instructions that lead to a desired result. Steps are those steps that require the physical operation of physical quantities. Usually, although not necessarily, these quantities take the form of electrical or magnetic signals that can be stored, transferred, combined, compared, and otherwise operated in a computer system. However, it should be remembered that all of these terms and similar terms are associated with appropriate physical quantities and are merely convenient labels applied to these quantities.
[0019] Unless otherwise specifically stated, as will be apparent from the following discussion, it is appreciated that throughout this application, discussions utilizing terms such as "access," "receive," "send," "use," "select," "determine," "normalize," "multiply," "average," "monitor," "compare," "apply," "update," "measure," "derive," and the like refer to the actions and processes of a computer system or similar electronic computing device to manipulate and transform data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission, or display devices.
[0020] In the accompanying drawings, a single block may be described as implementing one or more functions; however, in actual practice, one or more functions implemented by the block may be implemented in a single component or across multiple components, and / or may be implemented using hardware, using software, or using a combination of hardware and software. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described below in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technician may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as departing from the scope of the present disclosure. Moreover, the example input device may include components different from those shown, including known components such as processors, memories, etc.
[0021] Unless the techniques described herein are specifically described as being implemented in a particular manner, the techniques described herein may be implemented in hardware, software, firmware, or any combination thereof. Any features described as modules or components may also be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be implemented at least in part by a non-transitory processor-readable storage medium that includes instructions that, when executed, implement one or more of the methods described above. The non-transitory processor-readable data storage medium may form part of a computer program product, which may include packaging materials.
[0022] Non-transitory processor-readable storage media may include random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, other known storage media, etc. Additionally or alternatively, the technology may be at least partially implemented by a processor-readable communication medium that carries or communicates code in the form of instructions or data structures and can be accessed, read, and / or executed by a computer or other processor.
[0023] The various illustrative logical blocks, modules, circuits, and instructions described in conjunction with the embodiments disclosed herein may be executed by one or more processors (or processing systems). The term "processor" as used herein may refer to any general purpose processor, special purpose processor, conventional processor, controller, microcontroller, and / or state machine capable of executing scripts or instructions of one or more software programs stored in memory.
[0024] As described above, image processing can be used to properly render images captured by devices with different system capabilities and standards on a given display. More specifically, an image processor can modify the raw image data generated by an image capture device so that the image can be reproduced more accurately or realistically on a display device (such as to utilize the full dynamic range of the display). Many existing image capture and display devices are configured to process image data having a brightness (Y) color channel and a chroma (UV) color channel. As such, many existing image processors are designed to perform tone mapping operations or inverse tone mapping operations in a YUV color space. However, the Y color channel and the UV color channel are not completely orthogonal. In other words, changing the dynamic range of the Y component of a given image also changes the saturation of any distinguishable chromaticity by a variable amount. Thus, performing tone mapping or inverse tone mapping in a YUV color space may impose significant adverse consequences on quality, latency, memory, and processing resources, which may not be suitable for many image processing applications (such as not suitable for processing video or not suitable for implementation in battery-powered devices or other edge devices with limited resources).
[0025] Aspects of the present disclosure recognize that by performing tone mapping and inverse tone mapping operations in a perceptually uniform color space (UCS), image processing overhead can be significantly reduced without losing accuracy. A UCS is any color space in which the geometric distance between any two points in the color space reflects an equal amount of change in perceived color. An example suitable UCS for tone mapping and inverse tone mapping is a luminance (U)-valued space. z ), red-green(a z ) color components and yellow-blue (b z ) Color component definition J z a z b z Color space.
[0026] Various aspects relate generally to image processing, and more particularly to z a z b z In some aspects, an image processing system may receive image data having color channels associated with a perceptually non-uniform color space (such as RGB or YUV) and may convert the image data to a J z a z b z In some implementations, the image processing system can perform a global CVM operation that converts the J color space supported by the image source (such as an image capture device) into a color space supported by the image source (such as an image capture device). z value, a z Value and bz The range of values is mapped to the j supported by the graphics target (such as a display device). z value, a z Value and b z In some other implementations, the image processing system may further implement a local CVM operation that adaptively tunes the J of the image data based on various properties of the received image. z Channel, a z Channel and b z Furthermore, in some implementations, the image processing system can adaptively blend the J at the edge of the object. z values and other features to preserve high frequency details in low contrast areas of the received image.
[0027] Particular implementations of the subject matter described in this disclosure can be implemented to achieve one or more of the following potential advantages. z a z b z ), aspects of the present disclosure can substantially reduce the latency and computational load of CVM operations (such as tone mapping and inverse tone mapping) without losing accuracy. For example, unlike the YUV color space, J z a z b z The color space is a perceptually uniform color space. As such, tone mapping operations and inverse tone mapping operations can use J z a z b z The CVM technology of the present implementation can be implemented using a two-dimensional (2D) lookup table (LUT) in a color space (compared to a 3D LUT that is generally used for tone mapping operations and inverse tone mapping operations in a YUV color space). In addition, aspects of the present disclosure support high data accuracy for video processing and provide a wide range of programming options and hardware flexibility. Thus, the CVM technology of the present implementation can be suitable for limited area, low latency applications (such as set-top boxes, digital televisions, streaming devices, and other video applications) or devices with limited memory or processing resources (such as low-power edge devices).
[0028] Figure 1A block diagram of an example image capture and display system 100 according to some implementations is shown. System 100 includes an image capture device 110 (also referred to as an "image source"), an image processor 120, and a display device 130 (also referred to as an "image target"). Image capture device 110 captures light pattern 101 (also referred to as "scene light") and converts the captured light into a digital image. For example, image capture device 110 can be a camera. Display device 130 displays the digital image by reproducing the light pattern as display light 104 on a corresponding display surface. Example suitable display devices include, among other examples, televisions, computer monitors, laptops, tablet computers, and smart phones.
[0029] Image capture device 110 includes sensor 112, optoelectronic transfer function (OETF) 114, color space converter (CSC) 116, and encoder 118. Sensor 112 converts scene light 101 into an electrical signal (E C ). In some implementations, sensor 112 may include an array of optical sensing elements (such as charge coupled device (CCD) cells or complementary metal oxide semiconductor (CMOS) cells), each of which is configured to sample a corresponding pixel of scene light 101. OETF 114 converts the electrical signal E C Transformed into decoded image data (RGB C ), the decoded image data (RGB C ) has a red (R) color channel, a green (G) color channel, and a blue (B) color channel associated with the RGB color space. More specifically, OETF 114 converts the RGB information from the analog domain to the digital domain. For example, OETF 114 may convert the analog electrical signal E C Converted into digital R values, digital G values, and digital B values representing the primary color components associated with the sensor 112 .
[0030] CSC 116 changes the decoded image data RGB C In some aspects, the CSC 116 may convert the decoded image data into an RGB color space. C The CSC 116 may convert the decoded image data RGB to another color space (such as a YUV color space) that may be more easily compressed or otherwise encoded by the encoder 118 for transmission to the display device 130. C Converted into image data with a luminance (Y) channel and a chrominance (UV) channel associated with the YUV color space (YUV C ). The encoder 118 converts the converted image data YUV CEncoded into image capture data 102 for transmission to image processor 120 or display device 130. For example, encoder 118 may apply data compression or signal modulation to the converted image data YUV based at least in part on various communication standards or protocols supported by the communication medium (not shown for simplicity) or display device 130. C .
[0031] Image processor 120 processes image capture data 102 to generate display data 103, which can be used to reproduce original scene light 101 more accurately or realistically on display device 130 (taking into account the differences in the dynamic range or color gamut supported by image capture device 110 and display device 130). For example, a display device that can only display standard dynamic range (SDR) content may not be able to reproduce the full range of color, brightness, or contrast of an image captured in a high dynamic range (HDR) format. On the other hand, an image captured in an SDR format may not utilize the full range of color, brightness, or contrast supported by a display device that can display HDR content. Therefore, image processor 120 can bridge the image capture capabilities of image capture device 110 and the image display capabilities of display device 130. In some aspects, image processor 120 can be incorporated into image capture device 110. In some other aspects, image processor 120 can be incorporated into image display device 130.
[0032] The display device 130 includes a decoder 132, a CSC 134, an electro-optical transfer function (EOTF) 136, and a display 138. The decoder 132 receives the display data 103 from the image processor 120 and decodes the received data to restore the display data having a Y channel, a U channel, and a V channel associated with a YUV color space (YUV D ). In some implementations, the decoder 132 may reverse the encoding performed by the encoder 118 of the image capture device 110. The CSC 134 changes the YUV D In some aspects, the CSC 134 may convert the restored image data into a YUV color space. D Convert the image data from the YUV color space to another color space (such as the RGB color space) used by the display 138 to render and display the image. In some implementations, the CSC 134 may convert the image data from the YUV color space to another color space (such as the RGB color space) used by the display 138 to render and display the image. D Converted into image data having R color channel, G color channel and B color channel associated with RGB color space (RGB D ).
[0033] EOTF 136 converts the image data RGB Dinto an electrical signal (E) that can be used to illuminate the pixels of display 138 D ). More specifically, EOTF 136 converts RGB information from a digital domain to an analog domain. For example, EOTF 136 can convert digital image data RGB D The display 138 converts the electrical signal E into an analog brightness value (or "nit") associated with the display 138. D , is converted into display light 104. For example, display 138 may include an array of pixel elements that are each configured to display a corresponding pixel of a corresponding image (such as using a cathode ray tube (CRT), a liquid crystal display (LCD), an organic light emitting diode (OLED), or other display technology). More specifically, the color and brightness of the light output by each pixel element can be determined by the electrical signal E D The characteristics of a are defined or otherwise indicated.
[0034] The OETF 114 of the image capture device 110 reacts to the electrical signal E C More specifically, OETF 114 converts the electrical signal E C From the linear domain to the non-linear domain. As a result, the image capture data 102 received by the image processor 120 is non-linear. Aspects of the present disclosure recognize that many image processing operations are more complex to perform on non-linear image data than on linear image data. Thus, in some aspects, the image processor 120 may include an inverse electro-optical transfer function (IOETF) 122, a color volume data mapping (CVM) component 123, and an inverse electro-optical transfer function (IEOTF) 124. The IOETF 122 transfers the image capture data 102 back to the linear domain for processing as display data 103. Thus, the IOETF 122 may be the inverse of the OETF 114. The IEOTF 124 transfers the display data 103 back to the non-linear domain for display by the display device 130. Thus, the IEOTF 124 may be the inverse of the EOTF 136.
[0035] The CVM component 123 is configured to map the linear image capture data 102 to the linear display data 103. In some implementations, the CVM component 123 may attenuate the color, brightness, or contrast of the HDR image for display on an SDR display device (also referred to as "tone mapping"). In some other implementations, the CVM component 123 may enhance the color, brightness, or contrast of the SDR image for display on an HDR display device (also referred to as "inverse tone mapping"). Still further, in some implementations, the CVM component 123 may adjust the color, brightness, or contrast of the HDR image data for display on an HDR display device (such as to compensate for differences in the dynamic range or color gamut supported by the image source and the image target).
[0036] As described above, the image capture data 102 is based on converted image data YUV having a Y color channel, a U color channel, and a V color channel. C To encode. As such, the image capture data 102 is also associated with the YUV color space. However, the YUV color space is not a perceptually uniform color space. As such, the brightness plane, the saturation plane, and the chromaticity plane are not orthogonal to each other. In other words, changing the brightness range will result in a change in the amount of change in the saturation of different chromatic colors. Similarly, changing the saturation range will result in a change in the amount of change in the brightness level of different colors. Thus, implementing tone mapping or inverse tone mapping in the YUV color space is more prone to artifacts, and may impose significant adverse consequences on latency, memory, and processing resources, which may not be suitable for many image processing applications (such as not suitable for processing video or not suitable for implementation in battery-powered devices or other edge devices with limited resources).
[0037] Aspects of the present disclosure recognize that by using a perceptually uniform color space (UCS) such as, for example, a color space composed of luminance (J z ), red-green(a z ) color components and yellow-blue (b z ) Color component definition J z a z b z Performing tone mapping and inverse tone mapping operations in a J color space can significantly reduce image processing overhead without losing accuracy. In some aspects, the image processor 120 can convert the image capture data 102 to J z a z b z More specifically, the image processor 120 may convert the Y channel, the U channel, and the V channel of the image capture data 102 into a color space corresponding to the J color space. z a z b z Color space associated with J z Channel, a z Channel and b z The CVM component 123 may further convert the J of the image capture data 102 based on the dynamic range or color gamut supported by each of the image capture device 110 and the display device 130. z Channel, a z Channel and b z Channel mapping to display data 103 J z Channel, a z Channel and b z aisle.
[0038] Figure 2 2 is a block diagram of an example image processing system 200 according to some implementations. The image processing system 200 is configured to convert image data (YUV) captured by a source device (such as an image capture device) into in ) is converted into display data (YUV out ) to be displayed by a target device (such as a display device). In some implementations, the image processing system 200 may be Figure 1 An example of a CVM component 123. For example, refer to Figure 1 , image data YUV in may be an example of image capture data 102, and display data YUV out may be an example of display data 103 .
[0039] The image processing system 200 includes a color space converter (CSC) 202, a full range expander 204, an IOETF 206, an RGB to long, medium, short (LMS) CSC 208, a perceptual quantizer (PQ) IEOTF 210, an LMS to J z a z b z CSC 212, source to target (ST) color volume data mapper (CVM) 214, J z a z b z The CSC 202 converts the received image data into YUV. in Converted into image data having R color channel, G color channel and B color channel associated with RGB color space (RGB in ). In some implementations, CSC202 may receive the image data YUV in Convert from YUV color space to RGB color space. In some other implementations, the image processing system 200 may directly receive image data in RGB color space. in In such an implementation, the CSC 202 may be bypassed or omitted from the image processing system 200 .
[0040] The full range expander 204 expands the converted image data RGB in The maximum range to generate expanded image data (RGB exp ). For example, the range of digital YUV values may be limited between an artificial minimum value ("min") and an artificial maximum value ("max") in order to retain some encoding for timing reference purposes. Thus, if the received image data YUVin has been limited (such as between 256 and 3760, within a 12-bit range), then full range extension can make the expanded image data RGB exp falls within a predetermined maximum range associated with the RGB color space (such as between 0 and 4095). in When converted to RGB color space, the converted image data RGB in The associated range of values may remain the same.Thus, each of the R, G and B components may be extended using only a limited set of parameters.
[0041] IOETF 206 will extend the image data RGB exp Transformed into linearly interpolated image data (RGB lin i ). In some implementations, IOETF 206 may be the inverse of the OETF implemented by the source device. Figure 1 , IOETF 206 may be an example of IOETF 122 of image processor 120. Thus, IOETF 206 may be the inverse of OETF 114 of image capture device 110. To achieve high-precision IOETF curves while reducing memory requirements for lookup table (LUT) storage, IOETF 206 may use a 128-segment piecewise linear interpolation technique to convert 16-bit RGB components into 32-bit RGB components in the linear domain.
[0042] RGB to LMS CSC 208 converts linear image data to RGB lin i Converted into image data having a long (L) channel, a medium (M) channel, and a short (S) channel associated with the LMS color space (LMS lin i ). The L channel, M channel, and S channel represent the responses of three types of cones in the human eye. Thus, the converted image data LMS lin i This may also be referred to herein as “perceptual image data.” In some implementations, the RGB to LMS CSC 208 may convert the linear image data RGB to LMS by intermediate conversion to the tristimulus (XYZ) space defined by the International Commission on Illumination (CIE). lin i Converted into perceptual image data LMS lin i : Where XD65 , Y D65 and Z D65 belongs to the CIE XYZ tristimulus space, where the CIE standard illuminant D65 is the white point, b=1.15, and g=0.66. Aspects of the present disclosure recognize that any RGB color space can be transformed into the CIE XYZ tristimulus space. Thus, the transformation matrix used to transform image data from the RGB color space to the XYZ color space can depend on the RGB color space implemented by the image processing system 200.
[0043] PQ IEOTF 210 will perceive the image data LMS lin i Transform from the linear domain to the PQ domain. The PQ IEOTF 210 is a nonlinear transfer function adopted by the Society of Motion Picture and Television Engineers (SMPTE) and the International Telecommunication Union (ITU) standards for HDR display. For example, the PQ domain supports brightness levels in the range of 0.0001 to 10,000 nits. Thus, the PQ IEOTF 210 can convert the perceived image data LMS lin i The L channel, M channel and S channel are transformed into PQ image data (LMS PQ i )'s L' channel, M' channel, and S' channel: where c 1 =3424 / 2 12 , c 2 =2413 / 2 7 , c 3 =2392 / 2 7 , n = 2610 / 2 14 , and p = 1.7x2523 / 2 5 .
[0044] LMS to J z a z b z CSC 212 converts the PQ image data to LMS PQ i Converted into a z a z b z The brightness associated with the color space (J z ) channel, red-green (a z ) channel and the yellow-blue (b z ) channel image data (J z a z b z i ). As reference Figure 1 To describe, J z a z b z The color space is perceptually uniform. z a z b z i It may also be referred to herein as “perceptually uniform image data”. In some implementations, LMS to J z a z b z The CSC 212 may convert the PQ image data LMS into PQ i Transformed into perceptually uniform image data J z a z b z i : where d = -0.56, and d 0 =1.6295499532821566x10 -11 .
[0045] The ST CVM 214 is configured to convert the perceptually uniform image data J z a z b z i Mapped to perceptually uniform display data that is more suitable for presentation on the target device (J z a z b z d ). More specifically, the ST CVM 214 may z value, a z Value and b z The range of values is mapped to the J z value, a z Value and b z In some implementations, ST CVM 214 can z a z b z i J z Channels are mapped to narrower J z value range (also known as "tone mapping") to reduce the perceptually uniform image data J z a z b zi In some other implementations, ST CVM 214 can be implemented by z a z b z i J z Channels are mapped to wider J z value range (also known as "inverse tone mapping") to increase the perceptually uniform image data J z a z b z i The associated dynamic range.
[0046] In some implementations, ST CVM 214 can be configured to z a z b z i a z Channel and b z Channels are mapped to a z Value and b z A narrower range of values (also called a "saturation map") is used to reduce the distortion of perceptually uniform image data. z a z b z i In some implementations, the ST CVM 214 can be configured to generate a color gamut associated with the J z a z b z i a z Channel and b z Channels are mapped to a z Value and b z A wider range of values (also called "inverse saturation mapping") is used to increase the perceptually uniform image data J z a z b z i The associated color gamut. z Channel and a z Channel and b z The channels are orthogonal, so the ST CVM 214 can adjust the perceptually uniform image data J z a z b z i The color tone (or J z The ST CVM 214 is also able to adjust the perceptually uniform image data J z a z b z iThe saturation (or a z Channel and b z channels) without affecting the image hue.
[0047] J z a z b z The LMS CSC 216 will display the data in a perceptually uniform manner. z a z b z d Converted into perceptual display data having an L' channel, an M' channel, and an S' channel associated with an LMS color space (LMS PQ d ). More specifically, J z a z b z To LMSCSC 216 can be transferred from LMS to J z a z b z The color space conversion performed by CSC 212 is inverted. In some implementations, J z a z b z The LMS CSC 216 can convert the perceptually uniform display data J into the display data J according to equations 3 and 4 below. z a z b z d Convert to perceptual display data LMS PQ d : where d = -0.56, and d 0 =1.6295499532821566x10 -11 .
[0048] PQ EOTF 218 will perceive the display data LMS PQ d Transform from the PQ domain back to the linear domain. More specifically, the PQ EOTF 218 may be the inverse of the PQ IEOTF 210. Thus, the PQ EOTF 218 may transform the perceptual display data LMS PQ d The L' channel, M' channel and S' channel are converted into linear display data (LMS lin d )'s L channel, M channel, and S channel: where c 1=3424 / 2 12 , c 2 =2413 / 2 7 , c 3 =2392 / 2 7 , n = 2610 / 2 14 , and p = 1.7x2523 / 2 5 .
[0049] LMS to RGB CSC 220 converts linear display data LMS lin d Converted into display data having R channel, G channel and B channel associated with RGB color space (RGB lin d ). More specifically, the LMS to RGB CSC 220 may invert the color space conversion performed by the RGB to LMS CSC 208. In some implementations, the LMS to RGB CSC 220 may convert the linear display data LMS to RGB by an intermediate conversion to the CIE XYZ tristimulus space. lin d Convert to display data RGB lin d : Where b=1.15, and g=0.66. Aspects of the present disclosure recognize that the CIE XYZ tristimulus space can be transformed into any RGB color space. Thus, the transformation matrix used to transform display data from the XYZ color space to the RGB color space may depend on the RGB color space implemented by the image processing system 200.
[0050] IEOTF 222 converts the converted display data RGB lin d Transformed into non-linear display data (RGB out ). In some implementations, the IEOTF 222 may be the inverse of the EOTF implemented by the target device. Figure 1 , IEOTF 222 may be an example of IEOTF 124 of image processor 120. Thus, IEOTF 222 may be the inverse of EOTF 136 of image capture device 130. To achieve a high precision IEOTF curve while reducing memory requirements for LUT storage, IEOTF 222 may use a 128-segment piecewise linear interpolation technique to convert 32-bit RGB components into 16-bit RGB components in a non-linear domain.
[0051] CSC 224 will display data RGB non-linearly out Convert to display data (YUV out ) for output to the target device. More specifically, CSC 224 may invert the color space conversion performed by CSC 202. In some implementations, CSC 224 may convert the display data RGB out Convert from RGB color space to YUV color space. In some other implementations, the image processing system 200 may directly output nonlinear display data RGB in RGB color space. out In such an implementation, the CSC 224 may be bypassed or omitted from the image processing system 200 .
[0052] Figure 3 1 shows a block diagram of an example CVM system 300 according to some implementations. The CVM system 300 is configured to transform image data 302 representing an image captured by a source device into display data 308 that can be used to reproduce the image at a display device. In some implementations, the CVM system 300 may be Figure 1 CVM component 123 or Figure 2 An example of the ST CVM 214. For example, see Figure 2 , the image data 302 may be perceptually uniform image data J z a z b z i And, the display data 308 may be perceptually uniform display data J z a z b z d Thus, each of the image data 302 and the display data 308 includes z a z b z Color space associated with J z Channel, a z Channel and b z aisle.
[0053] The CVM system 300 includes a global color volume data mapper 310, a local color volume data mapper 320, and an adaptive detail processor 330. The global color volume data mapper 310 is configured to map the received image data 302 to a color space corresponding to the color space based on the dynamic range and color gamut supported by each of the source device and the target device. z a z b z A set of global color values 304 associated with a color space. For example, J can be included in the image data 302.z The range of values is defined or limited by the dynamic range of the source device, and the a z Value and b z The range of values is defined or limited by the color gamut of the source device. In contrast, the J that can be included in the global color value 304 is z The range of values is defined or limited by the dynamic range of the target device, and the a that can be included in the global color value 304 z Value and b z The range of values is bounded or limited by the color gamut supported by the target device.
[0054] In some aspects, the global color volume data mapper 310 may map J supported by the source device to z value, a z Value and b z The range of values is mapped to the J z value, a z Value and b z The global color volume data mapper 310 may map the J values of the image data 302 to the image data 304 based on a “global” mapping between the source device and the destination device. In some implementations, the global color volume data mapper 310 may map the J values of the image data 302 to the image data 304 based on a tone mapping operation or an inverse tone mapping operation. z Channel J mapped to global color value 304 z In some other implementations, the global color volume data mapper 310 may map the a of the image data 302 based on a saturation mapping operation or an inverse saturation mapping operation. z Channel and b z Channel maps to a global color value of 304 z Channel and b z aisle.
[0055] The local color volume data mapper 320 is configured to remap the set of global color values 304 to a set of local color values 306 based on various properties of the corresponding image. Aspects of the present disclosure recognize that the human visual system is able to perceive more detail in darker and lighter areas of an image than in mid-tones. However, as the image data 302 is remapped to a set of local color values 306 by global expansion (or densification) of the dynamic range, the image data 302 is remapped to a set of local color values 306 by global expansion (or densification) of the dynamic range. z Channel J mapped to global color value 304 z As a result of the color channel, some image details may be faded in the darker or lighter areas of the image. In some aspects, the local color volume data mapper 320 can dynamically tune the global color value 304 so that details are preserved in the darker and lighter areas of the image. For example, the J of the local color value 306 zA channel may span a wider range of dark tones than midtones, and may span the widest range of light (or "highlight") tones.
[0056] Aspects of the present disclosure also recognize that a global spread of saturation values may not produce optimized colors for various objects in any given scene. More specifically, different saturation levels may be more pleasing to the human eye for different types of content. For example, a bluer sky and greener grass may be visually appealing in the context of an outdoor sports scene, but may be distracting in the context of a movie scene (for at least some types of films). In some aspects, the local color volume data mapper 320 may dynamically tune the global color values 304 based on contextual information about the image. For example, the local color volume data mapper 320 may determine classifications associated with various objects or features in the image, and may adjust the a of the global color value 304 based on the determined classifications. z Channel and b z Channels are remapped to local color values a z Channel and b z aisle.
[0057] The adaptive detail processor 330 may further remap the local color values 306 to the display data 308 based on the details detected in the corresponding image. Aspects of the present disclosure recognize that, as a result of stretching dark and highlight tones, high-frequency details may be lost in low-contrast regions of an image depicted by the local color values 306. For example, individual blades of grass on a soccer field may be difficult to discern due to the stretching of tones at the edges of the grass. In some aspects, the adaptive detail processor 330 may adaptively tune the local color values 306 so that high-frequency details are preserved in low-contrast regions of an image depicted by the display data 308. More specifically, the adaptive detail processor 330 may detect edges of objects or features in the image and adaptively blend the local color values 306 at the detected edges based on the contrast of the surrounding area. z aisle.
[0058] Figure 4 4 is a block diagram of an example system 400 for global CVM according to some implementations. The system 400 is configured to convert image data ({J z [x,y] in 、a z [x,y] in 、b z [x,y] in}, where x and y refer to the horizontal pixel coordinate and the vertical pixel coordinate) z Channel, a z Channel and b z Channels are mapped to global display data ({Jz [x,y] GM 、a z [x,y] GM 、b z [x,y] GM}) z Channel, a z Channel and b z In some implementations, the system 400 may be Figure 3 An example of a global color volume data mapper 310. For example, refer to Figure 3 , image data {J z [x,y] in 、a z [x,y] in 、b z [x,y] in} may be an example of the image data 302, and the display data {J z [x,y] GM 、a z [x,y] GM 、b z [x,y] GM} may be an example of a global color value 304 .
[0059] System 400 includes a global tone mapping component 410 and a global saturation mapping component 420. In some aspects, system 400 may map image data {J z [x,y] in 、a z [x,y] in 、b z [x,y] in} is converted into intensity (i in ) value and saturation (s in ) value, where i in =J z [x,y] in , and s in =a z [xy] in *b z [x,y] in The global tone mapping component 410 is configured to convert the received image data into a color image based on the source image metadata 402 associated with the received image data and the target capabilities 404 associated with the target device. in Value and s in The value is mapped to the global intensity value (i GM The global saturation mapping component 420 is configured to map i based on the image metadata 402 and the target capability 404. in Value and s inThe value is mapped to the global saturation value (s GM ).
[0060] Target capabilities 404 may include any known capabilities of the target device, such as, for example, the dynamic range or color gamut supported by the target device. In contrast, source image metadata 402 may be received with the image data and may indicate the dynamic range or color gamut of the master display (used to create or "master" image data). Suitable metadata for the example may also include, among other examples, color calibration data associated with the master display, maximum frame average light level (MaxFALL), maximum content light level (MaxCLL), RGB primary colors, white point or display maximum light level, and display minimum light level. In some implementations, source image metadata 402 may include static metadata that is broadly applicable to a series of images (or multiple video frames). In some other implementations, source image metadata 402 may include dynamic metadata specific to the received image data.
[0061] In some implementations, the global tone mapping component 410 may convert i to a color based on a source-to-target tone mapping operator in an intensity axis (TI) lookup table (LUT) and a source-to-target tone mapping operator in a saturation axis (TS) LUT. in Value and s in Values are mapped to i GM Value. z a z b z The color space is a perceptually uniform color space, and J z Channel and a z Channel and b z The channels are completely orthogonal, so the color volume data mapping operation is isotropic and can be implemented by a one-dimensional (1D) LUT associated with each hue and saturation axis. For example, the global tone mapping component 410 can look up the intensity value i in the TI LUT. in The tone mapping can be done by looking up the saturation value s in the TS LUT. in Tone mapping, where TI LUT and TSLUT are 1D LUTs, and where i GM =TI[i in ]*TS[s in ]. In some implementations, the global tone mapping component 410 may determine or generate the TI LUT and TS LUT based on the source image metadata 402, the target capabilities 404, or any combination thereof. In some aspects, the system 400 may further convert the global intensity value i GM Converted into global display data J z Channel, where J z [x,y]GM =i GM .
[0062] In some implementations, the global saturation mapping component 420 can convert i to a source-to-destination color saturation mapping operator based on a source-to-destination color saturation mapping operator in an intensity axis (SI) LUT and a source-to-destination color saturation mapping operator in a saturation axis (SS) LUT. in Value and s in Values are mapped to s GM Value. z a z b z The color space is a perceptually uniform color space, so the global saturation mapping operation is essentially isotropic and requires only 1D operators associated with each intensity and saturation axis. For example, the global saturation mapping component 420 can look up the SI LUT for the intensity value i in The saturation map can be found in the SS LUT for the saturation value s in Saturation mapping, where SI LUT and SS LUT are 2D LUTs, and where s GM =SI[i in ]*SS[s in ]. In some implementations, the global saturation mapping component 420 can determine or generate the SI LUT and the SS LUT based on the source image metadata 402, the target capabilities 404, or any combination thereof. In some aspects, the system 400 can further convert the global saturation value s GM Converted into a global display data z Channel and b z Channel, where a z [xy] GM =a z [xy] in *s GM , and b z [x,y] GM =b z [x,y] in *s GM .
[0063] Figure 5 1 is a block diagram of an example system 500 for local CVM according to some implementations. The system 500 is configured to display global display data ({J z [x,y] GM 、a z [x,y] GM , b z [x,y] GM}) z Channel, a z Channel and bz Channel mapping to local display data ({J z [x,y] LM 、a z [x,y] LM 、b z [x,y] LM}) z Channel, a z Channel and b z In some implementations, the system 500 may be Figure 3 An example of a local color volume data mapper 320. For example, refer to Figure 3 , global display data {J z [x,y] GM 、a z [x,y] GM 、b z [x,y] GM} may be an example of a global color value 304, and the local display data {J z [x,y] LM 、a z [x,y] LM 、b z [x,y] LM} may be an example of a local color value 306 .
[0064] The system 500 includes a local tone mapping component 510 and a local saturation mapping component 520. The local tone mapping component 510 is configured to convert the global display data J z [x,y] GM J z Channel mapping to local display data J z [x,y] LM J z In some aspects, the local tone mapping component 510 can be based at least in part on J z [x,y] GM More specifically, the mapping can improve the contrast of the image while preserving details in both the darker and brighter areas of the image. In some implementations, the local tone mapping component 510 can map the global display data J z [x,y] GM J z The channel undergoes histogram equalization. Histogram equalization is a technique used to increase the contrast of an image by spreading or redistributing a relatively narrow range of tonal values over a wider range.
[0065] Aspects of the present disclosure recognize that the human visual system is able to perceive more detail in darker and lighter areas of an image than in mid-tones. Thus, in some implementations, the local tone mapping component 510 can perform histogram equalization on independent tone bands. As used herein, a "band" refers to a range of tones that does not overlap with any other band. For example, with global display data J z [x,y] GM J z Each tonal value associated with a channel may fall within a spectrum that is divided or subdivided into a “dark” band, a “middle” band, and a “highlight” band. The local tone mapping component 510 may perform a separate histogram equalization operation on the tonal values within each of the dark, middle, and highlight bands so that tonal values in a given band do not cross over into another band due to the histogram equalization.
[0066] The local saturation mapping component 520 is configured to convert the global display data a z [x,y] GM and b z [x,y] GM a z Channel and b z Channels are mapped to local display data a z [x,y] LM and b z [x,y] LM a z Channel and b z Aspects of the present disclosure recognize that the brightness of an image may affect the perception of colors in the image. Thus, in some implementations, the local saturation mapping component 520 may be based at least in part on the local display data J z [x,y] LM J z For example, the local saturation mapping component 520 can be based on J z [x,y] LM The value of chromaticity (h z ) or chroma (C z ),in:
[0067] In some implementations, the local saturation mapping component 520 can further map the global display data a based at least in part on the classification 502 of one or more objects or features in the image. z [x,y] GM and b z [x,y] GM a z Channel and bz Channels are mapped to local display data a z [x,y] LM and b z [x,y] LM a z Channel and b z The classification 502 provides contextual information about an object or feature that can be used to determine the best color for displaying such an object or feature on a target device. For example, if the classification 502 associated with a particular object indicates that the object is a human face, the local saturation mapping component 520 can tune the chromaticity h of the object. z and chroma C z (Using Equation 5 and Equation 6) to match skin tones that are known to look more natural on the target device.
[0068] In some aspects, the classification 502 can be inferred by a neural network 530. Example suitable neural networks can include various types of deep neural networks (DNNs), such as convolutional neural networks (CNNs) or recurrent neural networks (RNNs), among other examples. In some implementations, the neural network 530 can be based on the global display data {J z [x,y] GM 、a z [x,y] GM 、b z [x,y] GM}(such as in Figure 5 502 based on one or more channels of the image (shown in FIG. 50 ). In some other implementations, neural network 530 may infer classification 502 based on other data associated with the image (not shown for simplicity). Example suitable data may include metadata (such as Figure 4 402) or image data from which global display data is derived (such as Figure 3 of image data 302).
[0069] Figure 6 An example mapping 600 is shown for mapping global pixel hue (or intensity) values to local pixel hue (or intensity) values using adaptive histogram equalization. In some implementations, the mapping 600 may be composed of Figure 5 The local tone mapping component 510 is implemented. For example, refer to Figure 5 , the global pixel tone value can be the global display data J z [x,y] GM J z The local pixel tone value may be an example of a channel, and the local pixel tone value may be a local display data J z [x,y] LM J zAn example of a channel.
[0070] exist Figure 6 In the example of z [x,y] GM The values of ) are plotted along the vertical axis, and the local pixel tone values (such as J z [x,y] LM The mapping 600 is represented by a curve extending between a minimum intensity threshold (TH0) and a maximum intensity threshold (TH3) along the horizontal axis. Figure 6 As shown in , the range of intensity values between TH0 and TH3 represents a spectrum subdivided into a "dark" band, an "intermediate" band, and a "highlight" band. More specifically, the dark band is defined by a minimum intensity threshold TH0 and a first intensity threshold (TH1), the intermediate band is defined by a first intensity threshold TH1 and a second intensity threshold (TH2), and the highlight band is defined by a second intensity threshold TH2 and a maximum intensity threshold TH3.
[0071] As reference Figure 5 As described above, the local tone mapping component 510 can perform histogram equalization on the global pixel tone values within each of the dark band, the middle band, and the highlight band. In other words, the mapping 600 can redistribute or "stretch" the pixel tone values within each band. For example, within the dark band, the local tone mapping component 510 can remap J z [x,y] GM The value of J z [x,y] LM The resulting values of are more evenly distributed, but remain bounded by thresholds TH0 and TH1. Similarly, within the middle band, the local tone mapping component 510 may remap J z [x,y] GM The value of J z [x,y] LM The resulting values of are more evenly distributed, but remain bounded by thresholds TH1 and TH2. Moreover, within the highlight band, the local tone mapping component 510 may remap J z [x,y] GM The value of J z [x,y] LM The resulting values of are more evenly distributed but remain bounded by thresholds TH2 and TH3.
[0072] In some implementations, guard bands may be used to maintain a sufficient amount of separation between local pixel tone values in each of the dark band, the middle band, and the highlight band (such as to avoid destroying pixel intensity due to histogram stretching). For example, a first guard band (GB0) provides a buffer between black pixel values and the next darkest local pixel tone values; a second guard band (GB1) provides a buffer between local pixel tone values in the dark band and local pixel tone values in the middle band; a third guard band (GB2) provides a buffer between local pixel tone values in the middle band and local pixel tone values in the highlight band; and a fourth guard band (GB3) provides a buffer between white pixel values and the next highest local pixel tone values.
[0073] Aspects of the present disclosure recognize that the human visual system can perceive more detail in dark areas of an image than in mid-tones. Thus, in some implementations, the dark bands can span a wider range of pixel intensities than the mid-tones (such that TH0–TH1>TH1–TH2). As such, the dark bands can support a greater amount of histogram stretching than in the mid-tones. Aspects of the present disclosure further recognize that the human visual system can perceive even more detail in bright areas of an image than in dark areas. Thus, in some implementations, the highlight bands can span a wider range of pixel intensities than the dark bands (such that TH2–TH3>TH0–TH1). As such, the highlight bands can support a greater amount of histogram stretching than the dark and mid-tones.
[0074] Figure 7 1 is a block diagram of an example system 700 for adaptive detail processing according to some implementations. The system 700 is configured to convert local display data ({J z [x,y] LM 、a z [x,y] LM 、b z [x,y] LM}) z Channel, a z Channel and b z Channel mapping to output display data ({J z [x,y] out 、a z [x,y] out 、b z [x,y] out}) z Channel, a z Channel and b z In some implementations, the system 700 may be Figure 3 An example of an adaptive detail processor 330 is shown in FIG. Figure 3 , local display data {J z [x,y]LM 、a z [x,y] LM 、b z [x,y] LM} may be an example of a local color value 306, and the output display data {J z [x,y] out 、a z [x,y] out 、b z [x,y] out} may be an example of display data 308.
[0075] The system 700 includes a hue variance estimation component 710, a detail extraction component 720, a hue blending component 730, and a saturation correction component 740. The hue variance estimation component 710 is configured to determine the local display data J z [x,y] LM J z Channel and the original image data from which the local display data is derived (J z [x,y] in ) z The difference between channels (ΔJ z [x,y])(where ΔJ z [x,y]=|J z [x,y] LM –J z [x,y] in |). For example, the original image data can be Figure 4 The image data {J z [x,y] in 、a z [x,y] in 、b z [x,y] in}or Figure 3 An example of image data 302 is shown.
[0076] The detail extraction unit 720 is configured based on ΔJ z [x, y] values to detect edges or textures in the image. In some aspects, detail extraction component 720 may generate control signal 702 based at least in part on the detected edges or textures. More specifically, control signal 702 may indicate whether each pixel value is associated with a flat area of the image or a textured area of the image. In some implementations, in the case where the pixel value is associated with a textured area of the image, control signal 702 may further indicate the amount of contrast associated with the textured area.
[0077] The tone mixing unit 730 is configured to z[x,y] in J z Channel, local display dataJ z [x,y] LM J z channel and control signal 702 to generate output display data J z [x,y] out J z More specifically, J z [x,y] out Each value of can be J z [x,y] in The corresponding values and J z [x,y] LM In some implementations, the hue blending component 730 can control or adjust the blending amount based on the control signal 702 to preserve high-frequency details in low-contrast areas of the image.
[0078] For example, if control signal 702 indicates that the pixel value is associated with a flat area of the image, hue blending component 730 may avoid adding J z [x,y] LM The value of J z [x,y] in As a result, for pixel values associated with flat areas of the image, J z [x,y] out The value can be equal to J z [x,y] LM The value of (J z [x,y] out =J z [x,y] LM ). In contrast, if the control signal 702 indicates that the pixel value is associated with a textured region of the image, the tone blending component 730 may set J z [x,y] LM The value of J z [x,y] in As a result, for pixel values associated with textured regions of the image, J z [x,y] out The value can be J z [x,y] LM and J z [x,y] in The combination of values (J z [x,y] out =αJ z [x,y] LM +βJ z [x,y] in, where α and β depend on the amount of contrast in the region).
[0079] The saturation correction component 740 is configured to output the display data J based on the z [x,y] out J z Channel to display local data a z [x,y] LM and b z [x,y] LM a z Channel and b z Channel mapping to output display data a z [x,y] out and b z [x,y] out a z Channel and b z Channel. As reference Figure 5 As described above, the brightness of an image may affect the perception of colors in the image. Thus, in some implementations, the local saturation mapping component 520 may adjust the chromaticity h of a given pixel. z and chroma C z (Using Equation 5 and Equation 6) to compensate for any changes made to the brightness of the pixel (such as by the hue blending component 730).
[0080] Figure 8 Another block diagram of an example image processing system 800 according to some implementations is shown. The image processing system 800 is configured to convert image data captured by a source device (such as an image capture device) into display data for display by a target device (such as a display device). In some implementations, the image processing system 800 may be Figure 1 The image processor 120 or Figure 2 An example of an image processing system 200 is shown.
[0081] The image processing system 800 includes a device interface 810, a processing system 820, and a memory 830. The device interface 810 is configured to communicate with a source device and a target device. In some implementations, the device interface 810 may include an image source interface (I / F) 812 and an image target interface 814. The image source interface 812 is configured to receive image data from a source device. In some implementations, the image source interface 812 may receive first image data representing an image generated by an image capture device based on an optoelectronic transfer function (OETF). The image target interface 814 is configured to provide display data to a target device.
[0082] The memory 830 includes an image data buffer 831 configured to store received image data and any intermediate data resulting from image processing operations. The memory 830 also includes a non-transitory computer-readable medium (including one or more non-volatile memory elements, such as EPROM, EEPROM, flash memory, or hard drive, among other examples) that can store at least the following software (SW) modules: an inverse optical-electrical transfer function (IOETF) SW module 832 to transform the first image data into second image data based on an inverse of an OETF associated with an image capture device, wherein the second image data includes a red (R) channel, a green (G) channel, and a blue (B) channel associated with an RGB color space; A uniform color space (UCS) conversion SW module 833 is used to convert the second image data into a color space having the same color space as J. z a z b z The brightness associated with the color space (J z ) channel, red-green (a z ) channel and the yellow-blue (b z ) channel's third image data; A color volume data mapping (CVM) SW module 834 to convert the J of the third image data into a color volume data map based at least in part on a dynamic range or color gamut associated with a display device. z Channel, a z Channel and b z Channel mapping to the first display data J z Channel, a z Channel and b z Channels; and A target rendering SW module 835 is used to process the first display data so as to be displayed by a display device. Each software module includes instructions that, when executed by the processing system 820 , cause the image processing system 800 to perform a corresponding function.
[0083] The processing system 820 may include any suitable one or more processors capable of executing scripts or instructions of one or more software programs stored in the image processing system 800 (such as in the memory 830). For example, the processing system 820 may execute the IOETF SW module 832 to transform the first image data into the second image data based on the inverse of the OETF associated with the image capture device, wherein the second image data includes an R channel, a G channel, and a B channel associated with the RGB color space. The processing system 820 may also execute the UCS conversion SW module 833 to convert the second image data into a second image data having an R channel, a G channel, and a B channel associated with the J color space. z a z bz Color space associated with J z Channel, a z Channel and b z The processing system 820 may further execute the CVM SW module 834 to convert the J of the third image data into a color channel based at least in part on a dynamic range or color gamut associated with the display device. z Channel, a z Channel and b z Channel mapping to the first display data J z Channel, a z Channel and b z Furthermore, the processing system 820 may execute the target rendering SW module 835 to process the first display data so as to be displayed by the display device.
[0084] Fig. 9 An illustrative flow chart depicting example operations 900 for image processing according to some implementations is shown. In some implementations, the example operations 900 may be performed by an image processing system such as Figure 1 The image processor 120 or respectively Figure 2 and Figure 8 The image processing system 200 or 800 is implemented in any of the above embodiments.
[0085] The image processing system receives first image data representing an image generated by an image capture device based on the OETF (910). The image processing system transforms the first image data into second image data based on the inverse of the OETF, wherein the second image data includes a red (R) channel, a green (G) channel, and a blue (B) channel associated with an RGB color space (920). The image processing system further converts the second image data into a color space having a color space associated with the OETF. z a z b z The brightness associated with the color space (J z ) channel, red-green (a z ) channel and the yellow-blue (b z ) channel of the third image data (930). The image processing system converts the J channel of the third image data to a color channel based at least in part on a dynamic range or color gamut associated with the display device. z Channel, a z Channel and b z Channel mapping to the first display data J z Channel, a z Channel and b z The image processing system processes the first display data so as to be displayed by the display device (950).
[0086] In some implementations, converting the second image data into the third image data may include: converting the second image data into fourth image data having a long (L) channel, a medium (M) channel, and a short (S) channel associated with an LMS color space; transforming the fourth image data into fifth image data based on a perceptual quantizer (PQ) transfer function; and converting the fifth image data into the third image data in a PQ domain associated with the PQ transfer function.
[0087] In some aspects, the image processing system may further receive metadata associated with the first image data, wherein the metadata indicates a dynamic range or color gamut associated with the master display. In some implementations, the metadata may include color calibration data associated with the master display, a maximum frame average light level, a maximum content light level, RGB primaries, a white point, or a display maximum light level and a display minimum light level.
[0088] In some implementations, the J of the third image data is z Channel, a z Channel and b z Channel mapping to the first display data J z Channel, a z Channel and b z The channel may include: generating a plurality of lookup tables (LUTs) based at least in part on the received metadata and a dynamic range or color gamut associated with the display device; converting the J values of the third image data to the LUT based at least in part on one or more first LUTs of the plurality of LUTs; z Channel, a z Channel and b z Channel mapping to the first display data J z channel; and converting the J of the third image data based at least in part on one or more second LUTs in the plurality of LUTs z Channel, a z Channel and b z Channel is mapped to a of the first display data z Channel and b z aisle.
[0089] In some implementations, the J of the third image data is z Channel, a z Channel and b z Channel is mapped to a of the first display data z Channel and b z The channel may include: converting the J of the third image data based on one or more second LUTs z Channel, a z Channel and b zThe first display data is mapped to a global saturation value; one or more features in the image are classified based on the neural network model; and the global saturation value is remapped to a first display data based at least in part on the classification of the one or more features. z Channel and b z aisle.
[0090] In some implementations, the J of the third image data is z Channel, a z Channel and b z Channel mapping to the first display data J z The channel may include: converting the J of the third image data based on the one or more first LUTs z Channel, a z Channel and b z The channel is mapped to a global tonal value within a spectrum of values, wherein the spectrum is divided into a plurality of bands; and the global tonal values in each of the plurality of bands are remapped to corresponding equalized tonal values within the band based on histogram equalization of the global tonal values.
[0091] In some implementations, the J of the third image data is z Channel, a z Channel and b z Channel mapping to the first display data J z The channel may further include: a J based on the equalized tone value and the third image data z differences between the channels to extract details in the image; and adjusting the equalized tonal values based on the extracted details.
[0092] In some implementations, the J of the third image data is z Channel, a z Channel and b z Channel is mapped to a of the first display data z Channel and b z The channel may include: converting the J of the third image data based on one or more second LUTs z Channel, a z Channel and b z The channels are mapped to a global saturation value; and the global saturation value is dynamically adjusted based at least in part on the equalized hue value.
[0093] In some implementations, processing of the first display data may include: converting the first display data into second display data having an R channel, a G channel, and a B channel associated with an RGB color space; and transforming the second display data into third display data based on an inverse of an EOTF associated with a display device.
[0094] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.
[0095] Moreover, it will be appreciated by those skilled in the art that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the aspects disclosed herein can be implemented as electronic hardware, computer software, or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technician can implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as departing from the scope of the present disclosure.
[0096] The methods, sequences or algorithms described in conjunction with the aspects disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or in any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative, the storage medium may be integrated with the processor.
[0097] In the foregoing description, embodiments have been described with reference to specific examples thereof. However, it will be apparent that various modifications and changes may be made thereto without departing from the broader scope of the present disclosure as set forth in the appended claims. Accordingly, the description and drawings are to be regarded in an illustrative rather than a restrictive sense.
Claims
1. A method for image processing, the method comprising: receiving first image data representing an image generated by an image capture device based on an optical-to-electrical transfer function (OETF); transforming the first image data into second image data based on an inverse of the OETF, the second image data having a red (R) channel, a green (G) channel, and a blue (B) channel associated with an RGB color space; The second image data is converted into a z a z b z The brightness associated with the color space (J z ) channel, red-green (a z ) channel and the yellow-blue (b z ) channel's third image data; The J of the third image data is based at least in part on a dynamic range or color gamut associated with a display device. z Channel, the a z Channel and the b z Channel mapping to the first display data J z Channel, a z Channel and b z aisle; as well as The first display data is processed for display by the display device.
2. The method according to claim 1, wherein the conversion of the second image data into the third image data comprises: converting the second image data into fourth image data having a long (L) channel, a medium (M) channel, and a short (S) channel associated with an LMS color space; transforming the fourth image data into fifth image data based on a perceptual quantizer (PQ) transfer function; as well as The fifth image data is converted into the third image data in a PQ domain associated with the PQ transfer function.
3. The method according to claim 1, further comprising: Metadata associated with the first image data is received, the metadata indicating a dynamic range or color gamut associated with a master display.
4. The method of claim 3, wherein the metadata comprises color calibration data, maximum frame average light level, maximum content light level, RGB primaries, white point, or display maximum light level and display minimum light level associated with the master display.
5. The method according to claim 3, wherein the J of the third image data z Channel, the a z Channel and the b z The J channel to the first display data z Channel, the a z Channel and the b z The mapping of channels includes: generating a plurality of lookup tables (LUTs) based at least in part on the received metadata and the dynamic range or color gamut associated with the display device; The J of the third image data is converted based at least in part on one or more first LUTs of the plurality of LUTs. z Channel, the a z Channel and the b z The J channel is mapped to the first display data z aisle; as well as The J of the third image data is converted based at least in part on one or more second LUTs of the plurality of LUTs. z Channel, the a z Channel and the b z Channel is mapped to the a of the first display data z Channel and the b z aisle.
6. The method according to claim 5, wherein the J of the third image data z Channel, the a z Channel and the b z channel to the first display data of the a z Channel and the b z The mapping of channels includes: The J of the third image data is converted based on the one or more second LUTs z Channel, the a z Channel and the b z Channels are mapped to global saturation values; classifying one or more features in the image based on a neural network model; and remapping the global saturation value to the a of the first display data based at least in part on the classification of the one or more features z Channel and the b z aisle.
7. The method according to claim 5, wherein the J of the third image data z Channel, the a z Channel and the b z The J channel to the first display data z The mapping of channels includes: The J of the third image data is converted based on the one or more first LUTs z Channel, the a z Channel and the b z The channels are mapped to global hue values within a spectrum of values, the spectrum being divided into a plurality of bands; and The global tonal values in each of the plurality of bands are remapped to corresponding equalized tonal values within the band based on histogram equalization of the global tonal values.
8. The method according to claim 7, wherein the J of the third image data z Channel, the a z Channel and the b z The J channel to the first display data z The mapping of channels also includes: Based on the equalized tone value and the J of the third image data z Differences between channels to extract details in the image; and The equalized tonal values are adjusted based on the extracted details.
9. The method of claim 8, wherein the adjusting of the equalized hue value comprises: The equalized tone value is selectively combined with the J value of the third image data. z Channel mixing.
10. The method according to claim 9, further comprising: It is determined whether each pixel of the image is associated with a flat region or a textured region, the selective blending of the equalized tonal values being based on whether each pixel of the image is associated with a flat region or a textured region.
11. The method according to claim 7, wherein the J of the third image data z Channel, the a z Channel and the b z channel to the first display data of the a z Channel and the b z The mapping of channels includes: The J of the third image data is converted based on the one or more second LUTs z Channel, the a z Channel and the b z Channels are mapped to global saturation values; and The global saturation value is dynamically adjusted based at least in part on the equalized hue value.
12. The method of claim 1, wherein the processing of the first display data comprises: converting the first display data into second display data having an R channel, a G channel, and a B channel associated with the RGB color space; as well as The second display data is transformed into third display data based on an inverse of an electro-optical transfer function (EOTF) associated with the display device.
13. An image processor, comprising: Processing systems; as well as a memory storing instructions that, when executed by the processing system, cause the image processor to: receiving first image data representing an image generated by an image capture device based on an optical-to-electrical transfer function (OETF); transforming the first image data into second image data based on an inverse of the OETF, the second image data having a red (R) channel, a green (G) channel, and a blue (B) channel associated with an RGB color space; The second image data is converted into a z a z b z The brightness associated with the color space (J z ) channel, red-green (a z ) channel and the yellow-blue (b z ) channel's third image data; The J of the third image data is based at least in part on a dynamic range or color gamut associated with a display device. z Channel, the a z Channel and the b z Channel mapping to the first display data J z Channel, a z Channel and b z aisle; as well as The first display data is processed for display by the display device.
14. The image processor of claim 13, wherein the conversion of the second image data into the third image data comprises: converting the second image data into fourth image data having a long (L) channel, a medium (M) channel, and a short (S) channel associated with an LMS color space; transforming the fourth image data into fifth image data based on a perceptual quantizer (PQ) transfer function; as well as The fifth image data is converted into the third image data in a PQ domain associated with the PQ transfer function.
15. The image processor of claim 13, wherein execution of the instructions further causes the image processor to: Metadata associated with the first image data is received, the metadata indicating a dynamic range or color gamut associated with a master display.
16. The image processor according to claim 13, wherein the J of the third image data z Channel, the a z Channel and the b z The J channel to the first display data z Channel, the a z Channel and the b z The mapping of channels includes: generating a plurality of lookup tables (LUTs) based at least in part on the received metadata and the dynamic range or color gamut associated with the display device; The J of the third image data is converted based at least in part on one or more first LUTs of the plurality of LUTs. z Channel, the a z Channel and the b z The J channel is mapped to the first display data z aisle; as well as The J of the third image data is converted based at least in part on one or more second LUTs of the plurality of LUTs. z Channel, the a z Channel and the b z Channel is mapped to the a of the first display data z Channel and the b z aisle.
17. The image processor according to claim 15, wherein the J of the third image data z Channel, the a z Channel and the b z channel to the first display data of the a z Channel and the b z The mapping of channels includes: The J of the third image data is converted based on the one or more second LUTs z Channel, the a z Channel and the b z Channels are mapped to global saturation values; classifying one or more features in the image based on a neural network model; as well as remapping the global saturation value to the a of the first display data based at least in part on the classification z Channel and the b z aisle.
18. The image processor according to claim 15, wherein the J of the third image data z Channel, the a z Channel and the b z The J channel to the first display data z The mapping of channels includes: The J of the third image data is converted based on the one or more first LUTs z Channel, the a z Channel and the b z The channels are mapped to global hue values within a spectrum of values, the spectrum being divided into a plurality of bands; and The global tonal values in each of the plurality of bands are remapped to corresponding equalized tonal values in the band based on histogram equalization of the global tonal values.
19. The image processor according to claim 17, wherein the J of the third image data z Channel, the a z Channel and the b z The J channel to the first display data z The mapping of channels also includes: Based on the equalized tone value and the J of the third image data z Differences between channels to extract details in the image; and The equalized tonal values are adjusted based on the extracted details.
20. The image processor according to claim 17, wherein the J of the third image data z Channel, the a z Channel and the b z channel to the first display data of the a z Channel and the b z The mapping of channels includes: The J of the third image data is converted based on the one or more second LUTs z Channel, the a z Channel and the b z Channels are mapped to global saturation values; and The global saturation value is dynamically adjusted based at least in part on the equalized hue value.
