High dynamic range (HDR) image processing with adaptive color volume mapping
Adaptive tone mapping in a perceptually uniform color space addresses the challenge of rendering images with varying dynamic ranges and color gamuts, enhancing image quality and reducing computational overhead in image processing.
Patent Information
- Application Number
- JP2024075695
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-24
- Filing Date
- 2024-05-08
- Publication Date
- 2024-12-05
AI Technical Summary
Existing image processing techniques struggle to accurately render images captured by devices with different dynamic ranges and color gamuts on displays with varying capabilities, leading to loss of quality and inefficiencies in tone mapping operations, particularly in YUV color spaces.
Perform tone mapping and inverse tone mapping operations in a perceptually uniform color space (Jz, az, bz) to reduce latency and computational burden without loss of accuracy, using 2D lookup tables and adaptive histogram equalization to enhance image detail and color gamut adaptation.
Significantly reduces latency and computational overhead in image processing while maintaining accuracy, making it suitable for resource-constrained devices and improving image quality on displays with varying dynamic ranges and color gamuts.
Smart Images

Figure 2024169330000001_ABST
Abstract
Description
[Technical field]
[0001] The present embodiments relate generally to image processing, and more particularly to high dynamic range (HDR) image processing with adaptive tone and color volume mapping. [Background technology]
[0002] Display devices (such as televisions, set-up boxes, computers, and mobile phones) may use different image technology than that used by imaging devices (such as cameras and video recorders). Advances in display technology have resulted in increased capabilities, such as higher dynamic range, wider color gamut, and the transition from high definition (HD) to ultra high definition (UHD) display technologies. As a result, image processing may be used to properly render images captured by devices of different system capabilities and specifications on a given display. For example, a display device that can only display standard dynamic range (SDR) content may not be able to reproduce the full range of color, brightness, or contrast of an image captured in a high dynamic range (HDR) format. An image processor may modify the raw image data generated by the imaging device so that the image can be reproduced more accurately or realistically on the display device (e.g., to use the full dynamic range of the display).
[0003] Some image processing techniques (called "tone mapping") reduce the color, brightness, or contrast of an HDR image so that it is rendered on an SDR display. Other image processing techniques (also called "inverse tone mapping") increase the color, brightness, or contrast of an SDR image so that it is rendered on an HDR display. Even if both the imaging device and the display device support HDR, image processing can be used to further improve the quality of the displayed image because of the differences between the display environment (e.g., a television where brightness, color, contrast, and resolution are electronically constrained) and the imaging environment (e.g., a natural environment where brightness, color, contrast, and resolution are unlimited). As imaging and display technologies continue to develop, new image processing techniques are needed to bridge the performance (e.g., dynamic range and color gamut) between the imaging device and the display device. Summary of the Invention
[0004] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.
[0005] One innovative aspect of the subject matter of this disclosure can be implemented in a method of image processing, the method including: receiving first image data representing an image generated by an imaging device based on an optical-to-electrical transfer function (OETF); converting the first image data into second image data including red (R), green (G) and blue (B) channels associated with an RGB color space based on an inverse of the OETF; and converting the second image data into a second image data including red (R), green (G) and blue (B) channels associated with an RGB color space based on an inverse of the OETF. z a z b z Lightness (J) related to color space z ), red-green (a z ) and yellow-blue (b z) channels into third image data; and based at least in part on a dynamic range or color gamut associated with the display device, z , a z and b z Display data for channel J z , a z and b z This includes mapping the display data to a channel and processing the display data for display by a display device.
[0006] Another innovative aspect of the subject matter of this disclosure can be implemented in an image processor including a processing system with hardware and software processes that cause the image processor to receive first image data representing an image generated by an imaging device based on an OETF, convert the first image data into second image data including red (R), green (G) and blue (B) channels associated with an RGB color space based on an inverse function of the OETF, and convert the second image data into a ... z a z b z The brightness (J z ), red-green (a z ) and yellow-blue (b z ) channels into third image data, and based at least in part on a dynamic range or color gamut associated with the display device, z , a z and b z Display data for channel J z , a z and b z The display data is then processed for display by the display device. [Brief description of the drawings]
[0007] The present embodiments are illustrated by way of example and are not intended to be limited by the forms shown in the accompanying drawings.
[0008] [Figure 1]FIG. 1 illustrates a block diagram of an example imaging and display system, according to some embodiments.
[0009] [Diagram 2] FIG. 2 illustrates a block diagram of an example image processing system, according to some embodiments.
[0010] [Diagram 3] FIG. 3 illustrates an example of a color volume mapping (CVM) system, according to some embodiments.
[0011] [Figure 4] FIG. 4 illustrates a block diagram of an example system for global CVM, according to some embodiments.
[0012] [Diagram 5] FIG. 5 illustrates a block diagram of an example system for local CVM, according to some embodiments.
[0013] [Figure 6] FIG. 6 illustrates an example of mapping global pixel tone values to local pixel tone values with adaptive histogram equalization.
[0014] [Figure 7] FIG. 7 illustrates a block diagram of an example adaptive detail processor, according to some embodiments.
[0015] [Figure 8] FIG. 8 illustrates another block diagram of an example image processing system, according to some embodiments.
[0016] [Figure 9] FIG. 9 illustrates an exemplary flow chart showing example operations for image processing, according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0017] In the following description, numerous specific details are presented, such as examples of specific components, circuits, and processes, to provide a thorough understanding of the present disclosure. As used herein, the term "coupled" means directly connected or connected through one or more intervening components or circuits. The terms "electronic system" and "electronic device" may be used interchangeably to refer to any system capable of electronically processing information. Also, in the following description, for purposes of explanation, specific nomenclature is presented to provide a thorough understanding of aspects of the present disclosure. However, it will be apparent to one skilled in the art that these specific details may not be required to practice the exemplary embodiments. As another example, well-known circuits and devices are illustrated in block diagram form in order to avoid obscuring the present disclosure. Some portions of the detailed description that follows are presented in the form of procedures, logic blocks, processes, and other symbolic representations of operations on data bits within a computer memory.
[0018] These descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. In this disclosure, a procedure, logic block, process, etc., are conceived to be a self-consistent sequence of steps or instructions leading to a desired result. The steps are those that require physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals that can be stored, transferred, combined, compared, and otherwise manipulated in a computer system. It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities.
[0019] Unless specifically stated differently, and as will be apparent from the discussion below, throughout this application discussions using terms such as "accessing," "receiving," "transmitting," "using," "selecting," "determining," "normalizing," "multiplying," "averaging," "monitoring," "comparing," "applying," "updating," "measuring," "deriving," and the like, are understood to refer to operations or processes of a computer system or similar electronic computing device that manipulate and convert data represented as (electronic) physical quantities in the registers and memory of a computer system into other data similarly represented as physical quantities in the memory or registers of a computer system or other device that stores, transmits, or displays such information.
[0020] In the figures, a single block may be described as performing a function or functions. However, in actual implementation, the function or functions performed by the block may be implemented in a single component, across multiple components, and / or using hardware, software, or a combination of hardware and software. To clearly illustrate this interchangeability of hardware and software, various example components, blocks, modules, circuits, and steps are described below generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, and such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure. Additionally, the example input device may include components other than those shown, including well-known components such as a processor, memory, etc.
[0021] The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof, unless specifically described as being implemented in a particular manner. Also, any features described as modules or components may be implemented together in an integrated logic device or separately as separate, but cooperable, logic devices. If implemented in software, the techniques may be realized at least in part by a non-transitory processor-readable storage medium that includes instructions that, when executed, perform one or more of the methods described above. The non-transitory processor-readable storage medium may form part of a computer program product, which may include packaging materials.
[0022] The non-transitory processor-readable storage medium may comprise random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), read only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read only memory (EEPROM), flash memory, and other known storage media, etc. The techniques may additionally or alternatively be realized at least in part by a processor-readable communications medium that carries or communicates code in the form of instructions or data structures that can be accessed, read, and / or executed by a computer or other processor.
[0023] The various illustrative logic blocks, modules, circuits, and instructions described in connection with the embodiments disclosed herein may be executed by one or more processors (or processing systems). As used herein, the term "processor" may refer to any general-purpose processor, special purpose processor, conventional processor, controller, microcontroller, and / or state machine capable of executing scripts or instructions of one or more software programs stored in a memory.
[0024] As mentioned above, image processing may be used to properly render images captured by devices of different system capabilities and specifications on a given display. More specifically, an image processor may modify the original image data generated by the image capture device so that the image can be reproduced more accurately or realistically on the display device (e.g., to use the full dynamic range of the display). Many existing image capture and display devices are configured to process image data having luminance (Y) and chrominance (UV) color channels. Thus, many existing image processors are designed to perform tone mapping or inverse tone mapping operations in the YUV color space. However, the Y and UV color channels are not completely orthogonal. In other words, changing the dynamic range of the Y component of a given image changes the saturation of any distinguishable hue by different amounts. Thus, tone mapping or inverse tone mapping in the YUV color space may incur significant penalties in quality, latency, memory and processing resources, which may be inappropriate for many image processing applications (e.g., video processing or implementation in battery-powered devices and other resource-limited advanced devices).
[0025] Aspects of the present disclosure recognize that image processing overhead can be significantly reduced without loss of accuracy by performing tone mapping and inverse tone mapping operations in a perceptual uniform color space (UCS). A UCS is any color space in which the geometric distance between any two points in the color space reflects an equal amount of change in perceived color. An example of a suitable UCS for tone mapping and inverse tone mapping is the UCS for lightness (J z ), red-green (a z ) and yellow-blue (b z ) defined by color components J z a z b z It is a color space.
[0026] Various aspects relate generally to image processing, and more specifically to J z a z bz In some embodiments, an image processing system may receive image data having color channels associated with a color space that is not perceptually uniform (e.g., RGB or YUV), and may convert the image data into a color volume mapping (CVM) technique that operates in a color space. z a z b z In some embodiments, the image processing system may convert the input color space to a J color space supported by the image source (e.g., an image capture device). z , a z and b z Specifies the range of values supported by the image target (e.g., a display device). z , a z and b z In another embodiment, the image processing system may further perform a global CVM operation that maps the J-values of the image data based on various characteristics of the received image. z , a z and b z In some embodiments, the image processing system performs a local CVM operation that adaptively adjusts the channels to preserve high frequency detail at the edges of objects or other features in low contrast areas of the received image. z Values may be blended.
[0027] Particular embodiments of the subject matter described in this disclosure can be implemented to realize one or more of the following potential advantages. z a z b z By processing image data in a perceptually uniform color space (such as RGB), aspects of the present disclosure may significantly reduce the latency and computational burden of CVM operations (such as tone mapping and inverse tone mapping) without losing accuracy. z a z b zThe color space is a perceptually uniform color space. Therefore, tone mapping and inverse tone mapping operations are performed in a J-dimensional space (as compared to three-dimensional (3D) look-up tables (LUTs) that are commonly used for tone mapping and inverse tone mapping operations in YUV color space). z a z b z In color space, it can be implemented using a two-dimensional (2D) lookup table (LUT). In addition, the aspects of the present disclosure provide high data accuracy for video processing, as well as a wide range of programming options and hardware flexibility. Therefore, the CVM technique of the present embodiment can be suitable for area-constrained low-latency applications (such as setup boxes, digital televisions, streaming devices and other video applications), or devices with limited memory and processing resources (such as low-power advanced devices).
[0028] 1 is a block diagram of an example of an imaging and display system 100 according to some embodiments. The system 100 includes an imaging device (also referred to as an "image source") 110, an image processor 120, and a display device (also referred to as an "image target") 130. The imaging device 110 captures a pattern of light 101 (also referred to as "scene light") and converts the captured light into a digital image. For example, the imaging device 110 can be a camera. The display device 130 displays the digital image by reproducing the light pattern as display light 104 on a corresponding display surface. Examples of suitable display devices include televisions, computer monitors, laptops, tablets, smartphones, etc.
[0029] The imaging device 110 includes a sensor 112, an optical-to-electrical transfer function (OETF) 114, a color space converter (CSC) 116, and an encoder 118. The sensor 112 converts the scene light 101 into an electrical signal Ec. In some embodiments, the sensor 112 may include an array of optical sensing elements (such as charge-coupled device (CCD) cells or complementary metal-oxide semiconductor (CMOS) cells) each configured to sample an individual pixel of the scene light 101. The OETF 114 converts the electrical signal Ec into coded image data (RGBc) having red (R), green (G), and blue (B) color channels associated with an RGB color space. More specifically, the OETF 114 converts the RGB information from an analog domain to a digital domain. For example, the OETF 114 may convert the analog electrical signal Ec into digital R, G, and B values representing the primary color components associated with the sensor 112.
[0030] The CSC 116 modifies the color space associated with the encoded image data RGBc. In some aspects, the CSC 116 may convert the encoded image data RGBc from the RGB color space to another color space (such as the YUV color space) that the encoder 118 may easily compress or encode for transmission to the display device 130. In some embodiments, the CSC 116 may convert the encoded image data RGBc to image data (YUVc) having luminance (Y) and chrominance (UV) channels associated with the YUV color space. The encoder 118 encodes the converted image data YUVc as imaging data 102 for transmission to the image processor 120 or the display device 130. For example, the encoder 118 may perform data compression or signal modulation on the converted image data YUVc based at least in part on a communication medium (not shown for simplicity) or various communication standards or protocols supported by the display device 130.
[0031] The image processor 120 processes the imaging data 102 and generates display data 103 that can be used to more accurately or realistically reproduce the original scene light 101 on the display device 130 (given the difference in dynamic range or color gamut supported by the imaging device 110 and the display device 130). For example, a display device capable of displaying only standard dynamic range (SDR) content may not be able to reproduce the full range of color, brightness, or contrast of an image captured in a high dynamic range (HDR) format. On the other hand, an image captured in an SDR format may not use the full range of color, brightness, or contrast supported by a display device capable of displaying HDR content. Thus, the image processor 120 may bridge the image acquisition capabilities of the imaging device 110 and the image display capabilities of the display device 130. In some embodiments, the image processor 120 may be incorporated into the imaging device 110. In other embodiments, the image processor 120 may be incorporated into the image display device 130.
[0032] The display device 130 includes a decoder 132, a CSC 134, an electro-mechanical transfer function (EOFT) 136, and a display 138. The decoder 132 receives the display data 103 from the image processor 120 and decodes the received data into display data having Y, U, and V channels associated with the YUV color space (YUV D In some embodiments, the decoder 132 reverses the encoding performed by the encoder 118 of the image capture device 110. The CSC 134 recovers the recovered image data YUV D In some embodiments, the CSC 134 modifies the color space associated with the restored image data YUV D In some embodiments, the CSC 134 may convert the image data YUV from the YUV color space to another color space (e.g., RGB color space) used by the display 138 to render and display the image. D is expressed as image data having R, G and B color channels associated with the RGB color space (RGB D) may be converted to
[0033] EOTF136 is the converted image data RGB D , an electrical signal E that can be used to light up pixels of the display 138. D More specifically, the EOTF 136 converts RGB information from the digital domain to the analog domain. For example, the EOTF 136 converts digital image data RGB D The display 138 may convert the electrical signal E D The display 138 converts the color of the light output by each pixel element into a display light 104. For example, the display 138 may comprise an array of pixel elements, each configured to display an individual pixel of a corresponding image (e.g., using a cathode ray tube (CRT), liquid crystal display (LCD), organic light emitting diode (OLED), or other display technology). More specifically, the color and brightness of the light output by each pixel element is represented by an electrical signal E D may be determined or otherwise dictated by the characteristics of
[0034] The OETF 114 of the imaging device 110 outputs an electrical signal E C More specifically, the OETF 114 performs a nonlinear transformation on the electrical signal E Cfrom a linear domain to a nonlinear domain. As a result, the imaging data 102 received by the image processor 120 is nonlinear. Aspects of the present disclosure recognize that many image processing operations are more complex to perform on nonlinear image data than on linear image data. Thus, in some embodiments, the image processor 120 may include an inverse optical-to-electrical transfer function (IOETF) 122, a color volume mapping (CVM) component 123, and an inverse electrical-to-optical transfer function (IEOTF) 124. The IOETF 122 returns the imaging data 102 to a linear domain for processing as the display data 103. Thus, the IOETF 122 may be an inverse function of the OETF 114. The IEOTF 124 returns the display data 103 to a nonlinear domain for display by the display device 130. Thus, the IEOTF 124 may be an inverse function of the EOTF 136.
[0035] CVM component 123 is configured to map linear imaging data 102 to linear display data 103. In some embodiments, CVM component 123 may reduce the color, brightness, or contrast of an HDR image for display on an SDR display device (also referred to as "tone mapping"). In other embodiments, CVM component 123 may increase the color, brightness, or contrast of an SDR image for display on an HDR display device (also referred to as "inverse tone mapping"). Furthermore, in some embodiments, CVM component 123 may adjust the color, brightness, or contrast of HDR image data for display on an HDR display device (e.g., to compensate for differences in dynamic range or color gamut supported by the image source and image target).
[0036] As described above, the image data 102 may be converted into image data YUV having Y, U and V color channels. CThe image data 102 is therefore also associated with the YUV colour space. However, the YUV colour space is not a perceptually uniform colour space. Thus, the lightness, saturation and hue planes are not orthogonal to each other. In other words, changing the luminance range will cause different amounts of saturation change for colours of different hues. Similarly, changing the saturation range will cause different amounts of luminance level change for different colours. Thus, tone mapping or inverse tone mapping in the YUV colour space is prone to artifacts and may incur significant penalties in latency, memory and processing resources. This may not be suitable for many image processing applications, such as video processing or implementation in other advanced devices that are battery-powered or resource-constrained.
[0037] The present disclosure relates to, for example, a method for controlling brightness (J z ), red-green (a z ) and yellow-blue (b z ) defined by color components J z a z b z It has been recognized that image processing overhead can be significantly reduced without loss of accuracy by performing tone mapping and inverse tone mapping operations in a perceptually uniform color space (UCS), such as a color space. In some embodiments, the image processor 120 converts the imaging data 102 into a J-image data set for image processing. z a z b z More specifically, the image processor 120 may convert the Y, U, and V channels of the imaging data 102 into a J z a z b z J related to color space z , a z and b z The CVM component 123 may further convert the J-channel image data 102 into a J-channel image based on the dynamic range or color gamut supported by the image capture device 110 and the display device 130, respectively. z , a z and b zDisplay channel data 103 J z , a z and b z It may be mapped to a channel.
[0038] 2 illustrates an example block diagram of an image processing system 200 according to some implementations. The image processing system 200 processes image data (YUV in ) to be displayed by a target device (e.g., a display device) as display data (YUV out ) in some embodiments, the image processing system 200 may be an example of the CVM component 123 of FIG. 1. For example, with reference to FIG. 1, image data YUV in may be an example of the imaging data 102, and the display data YUV out is an example of the display data 103.
[0039] The image processing system 200 includes a color space converter (CSC) 202, a gamut expander 204, an IEOTF 206, an RGB-longwave, midwave, shortwave (LMS) CSC 208, a perceptual quantizer (PQ) IEOTF 210, and an LMS-J z a z b z CSC 212, Source-to-Target (ST) Color Volume Mapper (CVM) 214, and J z a z b z The CSC 202 includes a LMS CSC 216, a PQ EOTF 218, a LMS-RGB CSC 220, an IEOTF 222, and a CSC 224. The CSC 202 converts the received image data YUV in is expressed as image data having R, G, and B color channels associated with the RGB color space (RGB in In some embodiments, the CSC 202 converts the received image data into YUV in In another embodiment, the image processing system 200 may directly convert the image data in the RGB color space from the YUV color space to the RGB color space. inIn such an embodiment, the CSC 202 may be bypassed or omitted from the image processing system 200.
[0040] The full range expander 204 converts the converted image data RGB in Expand the maximum range of the enlarged image data (RGB exp For example, the range of digital YUV values may be restricted to an artificial minimum value ("min") and an artificial maximum value ("max") to reserve some code for timing reference purposes. Thus, the received image data YUV in is limited (for example, to 256 to 3760 in a 12-bit range), full gamut expansion results in the expanded image data RGB exp The received image data may be constrained to be within a certain maximum range associated with the RGB color space (e.g., from 0 to 4095). in Even if the image is converted to the RGB color space, the converted image data RGB in The range of values associated with may remain the same. Thus, each of the R, G and B components may be extended with only a limited set of parameters.
[0041] IOETF 206 is an extended image data RGB exp Linearly interpolated image data (RGB lin i ) in some embodiments, the IOETF 206 may be an inverse function of the OETF implemented by the source device. For example, with reference to FIG. 1, the IOETF 206 may be an example of the IOETF 122 of the image processor 120. Thus, the IOETF 206 may be an inverse function of the OETF 114 of the image capture device 110. To obtain a highly accurate IOETF curve while reducing memory requirements for storing a look-up table (LUT), the IOETF 206 may convert the 16-bit RGB components to 32-bit RGB components in the linear domain using a 128-way piecewise linear interpolation method.
[0042] RGB-LMS CSC 208 is a linear image data RGB lin i is expressed as image data having long wavelength (L), mid wavelength (M) and short wavelength (S) channels associated with the LMS color space (LMS lin i ) where the L, M and S channels represent the responses of the three types of cones in the human eye. lin i is sometimes referred to herein as “perceptual image data.” In some embodiments, the RGB-LMS CSC 208 converts the linear image data RGB through an intermediate transformation to the tristimulus (XYZ) space defined by the International Commission on Illumination (CIE). lin i Perception image data LMS lin i may be converted to
number
[0043] PQ IEOTF 210 is a perceptual image data LMS lin i from the linear domain to the PQ domain. The PQ IEOTF 210 is a nonlinear transfer function adopted by the Society of Motion Picture and Television Engineers (SMPTE) and International Telecommunications Union (ITU) standards for HDR displays. For example, the PQ domain supports luminance levels ranging from 0.0001 to 10000 nits. Thus, the PQ IEOTF 210 converts the perceptual image data LMS lin iThe L, M and S channels of the PQ image data (LMS PQ i ) may be converted to the L', M' and S' channels.
number
[0044] LMS-J z a z b z CSC 212 is a PQ image data LMS PQ i J z a z b z Lightness (J) related to color space z ), red-green (a z ) and yellow-blue (b z ) channel image data (J z a z b z i ) as explained with reference to Figure 1. z a z b z The color space is perceptually uniform. Therefore, in this specification, the converted image data J z a z b z i In some embodiments, LMS-J is also called "perceptually uniform image data." z a z b z The CSC 212 calculates the PQ image data LMS according to the following Equations 1 and 2. PQ i Perceptually uniform image data J z a z b z i may be converted to
number
[0045] ST CVM 214 is a perceptually uniform image data z a z b z i By presenting the image data on the target device, the system can obtain uniform image data. z a z b z d More specifically, the ST CVM 214 is configured to map the J z Value, a z value and b z The range of values is supported by the target device. z Value, a z value and b z In some embodiments, the ST CVM 214 may map the J z a z b z i J z Perceptually uniform image data J is obtained by mapping the channels to a narrower range of Jz values (also known as "tone mapping"). z a z b z i In another embodiment, the ST CVM 214 may reduce the dynamic range associated with J z a z b z i J z Perceptually uniform image data J is obtained by mapping the J channels to a wider range of J values (also known as "inverse tone mapping"). z a z b z i Some also increase the dynamic range associated with
[0046] In some embodiments, the ST CVM 214 z a z b z i A z and b z Channel, a z value and b z Perceptually uniform image data by mapping it to a narrower range of values (also called "saturation mapping") z a z b z i In some embodiments, the ST CVM 214 may reduce the color gamut associated with J z a z b z i A z and b z Channel, a z value and b z Perceptually uniform image data by mapping it to a wider range of values (also known as "inverse chroma mapping") z a z b z i This may expand the color gamut associated with J. z The channel is a z Channel and b z Since the channels are orthogonal, the ST CVM 214 produces perceptually uniform image data without affecting image saturation. z a z b z i The light and dark of (or J z The ST CVM 214 can adjust the perceptually uniform image data (J) without affecting the brightness and darkness of the image. z a z b z i saturation (or a z and b z You can also adjust the channel.
[0047] J z a z b z -LMS CSC 216 is a perceptually uniform display dataz a z b z d The LMS color space is represented by perceptual display data (LMS PQ d ) More specifically, J z a z b z -LMS CSC 216 is the LMS-J z a z b z In some embodiments, the JSC 212 may perform an inverse color space conversion. z a z b z -LMS CSC 216 provides perceptual uniformity display data J z a z b z d is calculated as the perceptual display data LMS according to the following Equations 3 and 4. PQ d may be converted to:
number
[0048] PQ EOTF 218 is a perceptual display data LMS PQ d , from the PQ domain back to the linear domain. More specifically, the PQ EOTF 218 may be an inverse function of the PQ IEOTF 210. Thus, the PQ EOTF 218 converts the perceptual display data LMS PQ d The L', M' and S' channels are linear display data (LMS lin d ) may be converted to the L, M and S channels.
number
[0049] The LMS-RGB CSC 220 is a linear display data LMS lin d The display data has R, G and B channels associated with the RGB color space (RGB lin d ) to the linear display data LMS CSC 208 via an intermediate conversion to the CIE XYZ tristimulus space. lin d Display data RGB lin d may be converted to
number
[0050] IOETF 222 is the converted display data RGB lin d Display data nonlinearly RGB outIn some embodiments, IEOTF 222 may be an inverse function of the EOTF implemented by the target device. For example, with reference to FIG. 1, IEOTF 222 may be an example of IEOTF 124 of image processor 120. Thus, IEOTF 222 may be an inverse function of EOTF 136 of image capture device 130. To obtain a highly accurate IEOTF curve while reducing memory requirements for storing the LUT, IEOTF 222 may convert the 32-bit RGB components to 16-bit RGB components in the nonlinear domain using a 128-way piecewise linear interpolation method.
[0051] CSC 224 is a non-linear display data RGB out Display data YUV for output to the target device out More specifically, CSC 224 may perform the inverse of the color space conversion performed by CSC 202. In some embodiments, CSC 224 converts display data RGB out In another embodiment, the image processing system 200 may convert the non-linear display data RGB in the RGB color space to a YUV color space. out In some implementations, the CSC 224 may be bypassed or omitted from the image processing system 200.
[0052] 3 illustrates an example of a CVM system 300, according to some embodiments. The CVM system 300 is configured to convert image data 302 representing an image captured by a source device into display data 308 usable to reproduce the image on a display device. In some embodiments, the CVM system 300 may be an example of the CVM component 123 of FIG. 1 or the ST CVM 214 of FIG. 2. For example, with reference to FIG. 2, the image data 302 may be perceptually uniform image data J. z a z b z i The display data 308 may be an example of perceptually uniform image data J. z a z b zb Thus, each of the image data 302 and the display data 308 may be an example of J z a z b z J related to color space z , a z , b z Contains channels.
[0053] The CVM system 300 includes a global color volume mapper 310, a local color volume mapper 320, and an adaptive detail processor 330. The global color volume mapper 310 maps received image data 302 to a local color volume mapper 320 based on the dynamic range and color gamut supported by the source and target devices, respectively. z a z b z The image data 302 may include a set of global color values 304 that may be mapped to the set of global color values 304 associated with the color space. z The range of values that may be contained in the image data 302 is constrained or limited by the dynamic range of the source device. z value and b z The range of values is constrained or limited by the color gamut of the source device. z The range of values that can be included in the global color values 304 is constrained or limited by the dynamic range of the target device. z value and b z The range of values is constrained or limited by the color gamut supported by the target device.
[0054] In some embodiments, the global color volume mapper 310 may be configured to use a JPG or JPEG 2000 RGB color space supported by the source device. z Value, a z value and b z The range of values may be expanded or contracted to accommodate the J2E supported by the target device. z Value, a z value and b zIn some embodiments, the global color volume mapper 310 may map the J-values of the image data 302 to a range of values and may determine a set of global color values 304 based on a "global" mapping between the source device and the target device. ... z Channel J with global color value 304 z In another embodiment, the global color volume mapper 310 may map a saturation of the image data 302 to a channel based on a saturation mapping operation or an inverse saturation mapping operation. z and b z Channel a with global color value 304 z and b z Some map to channels.
[0055] The local color volume mapper 320 remaps the set of global color values 304 to a set of local color values 306 based on various characteristics of the corresponding image. Aspects of the present disclosure recognize that the human visual system is capable of perceiving more detail in dark and light regions of an image compared to mid-tones. However, the J-values of the image data 302 can be reduced through a global expansion (or compression) of the dynamic range. z Channel J with global color value 304 z As a result of mapping to the J channel, some image detail may be lost in the darker or lighter areas of the image. In some embodiments, the local color volume mapper 320 may dynamically adjust the global color values 304 to preserve detail in the darker and lighter areas of the image. For example, the J of the local color values 306 z A channel may span a wider range of dark tones than mid-tones, and may span the widest range of light (or "highlight") tones.
[0056] Aspects of the present disclosure also recognize that a global expansion of saturation values may not produce optimal colors for various objects in any given scene. More specifically, different levels of saturation for different types of content may be more aesthetically pleasing to the human eye. For example, bluer skies and greener grass may be visually appealing in outdoor sports scene content, but distracting (at least for some movie genres) in movie scene content. In some aspects, the local color volume mapper 320 may dynamically adjust the global color values 304 based on contextual information about the image. For example, the local color volume mapper 320 may determine classifications associated with various objects or features in the image and scale the a of the global color values 304 based on the determined classifications. z Channel and b z channel to the a of the local color value z Channel and b z Remap to a channel.
[0057] The adaptive detail processor 330 may further remap the local color values 306 to the display data 308 based on the details detected in the corresponding image. Aspects of the present disclosure recognize that the stretching of dark and highlight tones may result in a loss of high frequency detail in low contrast areas of the image represented by the local color values 306. For example, individual blades of grass on a football field may become difficult to distinguish at the edges of the grass due to the stretching of tones. In some aspects, the adaptive detail processor 330 may adaptively adjust the local color values 306 such that high frequency detail is preserved in low contrast areas of the image represented by the display data 308. More specifically, the adaptive detail processor 330 detects edges of objects or features in the image and adjusts the J of the local color values 306 at the detected edges based on the contrast of the surrounding areas. z The channels may be adaptively blended.
[0058] FIG. 4 illustrates an example block diagram of a system 400 for global CVM, according to some implementations. The system 400 receives image data ({J z [x,y] in ,a z [x,y] in ,b z [x,y] in}, where x and y indicate the horizontal and vertical pixel coordinates. z , a z and b z Channels are global display data ({J z [x,y] GM ,a z [x,y] GM ,b z [x,y] GM})'s J z , a z and b z In some embodiments, the system 400 may be an example of the global color volume mapper 310 of FIG. 3. For example, referring to FIG. 3, the image data {J z [x,y] in ,a z [x,y] in ,b z [x,y] in} may be an example of image data 302, and display data {J z [x,y] GM ,a z [x,y] GM ,b z [x,y] GM} may be an example of a global color value 304.
[0059] The system 400 includes a global tone mapping component 410 and a global saturation mapping component 420. In some embodiments, the system 400 converts image data {J z [x,y] in ,a z [x,y] in ,b z [x,y] in} to luminance(i in) value and saturation (s in ) value, where i in =J z [x,y] in and s in =a z [x,y] in ×b z [x,y] in The global tone mapping component 410 determines the tone of the received image based on the source image metadata 402 associated with the received image data and the target capabilities 404 associated with the target device. in Value and s in Set the value to the global brightness value (i GM ) based on the image metadata 402 and the target performance 404. in Value and s in Set the value to the global saturation value (S GM )
[0060] The target capabilities 404 may include any known capabilities of the target device, such as the dynamic range or color gamut supported by the target device. Meanwhile, the source image metadata 402 may be received along with the image data and may indicate the dynamic range or color gamut of the mastering display (used to generate or "master" the image data). Examples of suitable metadata include color calibration data associated with the mastering display, maximum frame average light (MaxFALL), maximum content light (MaxCLL), RGB primaries, white point, or display maximum and minimum light, etc. In some embodiments, the source image metadata 402 may include static metadata generally applicable to a series of images (or multiple frames of a video). In other embodiments, the source image metadata 402 may include dynamic metadata specific to the received image data.
[0061] In some embodiments, the global tone mapping component 410 performs a global tone mapping operation based on a source-target tone mapping operator in a lightness axis (TI) lookup table (LUT) and a source-target tone mapping operator in a saturation axis (TS) LUT. in Value and s in value and i GM It may be mapped to a value. z a z b z The color space is a perceptually uniform color space, and J z Channel a z and b z Since the color volume mapping operation is fully orthogonal to the channels, it is isotropic and can be implemented by a one-dimensional (1D) LUT associated with each tonal and chroma axis. For example, the global tone mapping component 410 may use a TI LUT to map lightness values i in We may look up the tone map for the saturation value s in the TS LUT. in Here, the TI and TS LUTs are 1D LUTs, and i GM =TI[i in ]×TS[s in In some embodiments, the global tone mapping component 410 may determine or generate the TI LUT and the TS LUT based on the source image metadata 402, the target performance 404, or any combination thereof. In some aspects, the system 400 may further include a global lightness value i GM Global display data J z channel, where J z [x,y] GM =i GM It is.
[0062] In some embodiments, the global saturation mapping component 420 performs a global saturation mapping operation based on a source-target color saturation mapping operator in the lightness axis (SI) LUT and a source-target color saturation mapping operator in the saturation axis (SS) LUT. in Value and s inValue and s GM It may be mapped to a value. z a z b z Because the color space is a perceptually uniform color space, the global saturation mapping operation is isotropic in nature and requires only 1D operators associated with each lightness and chroma axis. For example, the global saturation mapping component 420 maps lightness values i in Retrieve the saturation mapping for s in the SS LUT. in where the SI and SS LUTs are 2D LUTs and s GM =SI[i in ]×SS[s in In some embodiments, the global saturation mapping component 420 may determine or generate the SI LUT and the SS LUT based on the source image metadata 402, the target capabilities 404, or any combination thereof. In some aspects, the system 400 may further include a global saturation value s GM The global display data of a z and b z channel, where a z [xy] GM =a z [xy] in ×s GM , b z [x,y] GM =b z [x,y] in ×s GM It is.
[0063] FIG. 5 illustrates an example block diagram of a system 500 for local CVM, according to some implementations. The system 500 receives global view data ({J z [x,y] GM ,a z [x,y] GM ,b z [x,y] GM})'s J z , a z and b z Channel to local display data ({J z[x,y] LM ,a z [x,y] LM ,b z [x,y] LM})'s J z , a z and b z In some embodiments, the system 500 may be an example of the local color volume mapper 320 of FIG. 3. For example, with reference to FIG. 3, the global view data ({J z [x,y] GM ,a z [x,y] GM ,b z [x,y] GM}) may be an example of a global color value 304, and local display data ({J z [x,y] LM ,a z [x,y] LM ,b z [x,y] LM}) may be an example of a local color value 306.
[0064] The system 500 includes a local tone mapping component 510 and a local saturation mapping component 520. The local tone mapping component 510 maps the J of the global display data to a local saturation mapping component 520. z Channel J z [x,y] GM Local display data J z Channel J z [x,y] LM In some embodiments, the local tone mapping component 510 is configured to map J z [x,y] GM The mapping may be based at least in part on a distribution of values of J of the global display data. More specifically, the mapping may improve the contrast of the image while preserving details in dark and light areas of the image. In some embodiments, the tone mapping component 510 may perform a mapping based at least in part on a distribution of values of J of the global display data. z Channel J z [x,y]GM Histogram equalization is sometimes based on the histogram equalization technique, which increases the contrast of an image by stretching or redistributing a relatively narrow range of tone values over a wider range.
[0065] Aspects of the present disclosure recognize that the human visual system can perceive more detail in darker and lighter regions of an image compared to mid-tones. Thus, in some embodiments, the local tone mapping component 510 may perform histogram equalization on separate bands of tones. As used herein, a "band" refers to a range of tones that does not overlap with any other band. For example, the J of the global display data may be z Channel J z [x,y] GM Each tonal value associated with lies in a spectrum that is partitioned or divided into a "dark," "mid," and "highlight" band. The local tone mapping component 510 may perform separate histogram equalization operations on the tonal values within each of the dark, mid, and highlight bands, such that the tonal values in a given band do not spill over into other bands as a result of the histogram equalization.
[0066] The local saturation mapping component 520 maps the a z and b z Channel A z [x,y] GM , b z [x,y] GM a of the local display data z and b z Channel A z [x,y] LM , b z [x,y] LM Aspects of the present disclosure recognize that the luminance of an image can affect the perception of color in the image. Thus, in some embodiments, the local saturation mapping component 520 maps the J of the local display data to z Channel J z [x,y]LM For example, the local saturation mapping component 520 may perform the above mapping based at least in part on the hue (h Z ) or chroma (C Z ) to J z [x,y] LM may be adjusted based on the value of
number
[0067] In some embodiments, the local saturation mapping component 520 further comprises: z and b z Channel A z [x,y] GM , b z [x,y] GM The local display data of a z and b z Channel A z [x,y] LM , b z [x,y] LM The classification 502 provides contextual information about the object or feature that can be used to determine the optimal color for displaying such object or feature on a target device. For example, if the classification 502 corresponding to an object indicates that the object is a human face, the local saturation mapping component 520 may map the hue of the object to z and chroma C z may be adjusted (using Equations 5 and 6) to match skin tones that are known to look more natural on the target device.
[0068] In some aspects, the classification 502 may be inferred by a neural network 530. Examples of suitable neural networks may include various types of deep neural networks, such as, for example, convolutional neural networks (CNNs), recurrent neural networks (RNNs), etc. In some embodiments, the neural network 530 may receive global representation data {J z [x,y] GM ,a z [x,y] GM ,b z [x,y] GM 4 , 502. In some embodiments, the neural network 530 infers the classification 502 based on one or more channels of the image. In other embodiments, the neural network 530 infers the classification 502 based on other data associated with the image (not shown for simplicity). Examples of suitable data may include metadata (such as image metadata 402 of FIG. 4 ) or image data from which global display data (such as image data 302 of FIG. 3 ) is extracted.
[0069] 6 illustrates an example mapping 600 of global pixel tone (or lightness) values to local pixel tone (or lightness) values using adaptive histogram equalization. In some embodiments, the mapping 600 may be performed by the local tone mapping component 510 of FIG. 5. For example, with reference to FIG. 5, the global pixel tone values are calculated by multiplying the J of the global display data by 100. z Channel J z [x,y] GM where the local pixel tone value is the J of the local display data. z Channel J z [x,y] LM This may be an example of.
[0070] In the example of Figure 6, (e.g. J z [x,y] GM Global pixel tone values (such as the value of J) are shown along the vertical axis. z [x,y] LM6. Local pixel tone values (such as values of TH0 and TH3) are shown along the horizontal axis. The mapping 600 is represented by a curve extending along the horizontal axis between a minimum lightness threshold (TH0) and a maximum lightness threshold (TH3). As illustrated in FIG. 6, the range of lightness values between TH0 and TH3 represents a spectrum divided into a "dark", "mid" and "highlight" band. More specifically, the dark band is bounded by a minimum lightness threshold TH0 and a first lightness threshold (TH1), the mid band is bounded by a first lightness threshold TH1 and a second lightness threshold (TH2), and the highlight band is bounded by a second lightness threshold TH2 and a maximum lightness threshold TH3.
[0071] 5, the local tone mapping component 510 may perform histogram equalization on the global pixel tone values in each of the dark, mid, and highlight bands. In other words, the mapping 600 may redistribute or "stretch" the pixel tone values within each band. For example, in the dark band, the local tone mapping component 510 may perform histogram equalization on the global pixel tone values in each of the dark, mid, and highlight bands. z [x,y] LM The values of J are more evenly distributed but still bounded by the thresholds TH0 and TH1. z [x,y] GM Similarly, in the mid band, the local tone mapping component 510 remaps the values of the resulting J z [x,y] LM so that the values of J are more evenly distributed but still bounded by the thresholds TH1 and TH2. z [x,y] GM Also, in the highlight band, the local tone mapping component 510 remaps the values of the resulting J z [x,y] LM so that the values of J are more evenly distributed but still bounded by the thresholds TH2 and TH3. z [x,y] GMRemap the values of
[0072] In some embodiments, guard bands may be used to maintain sufficient separation of local pixel tone values in each of the dark, mid, and highlight bands (e.g., to avoid squashing pixel brightness as a result of histogram stretching). For example, a first guard band (GB0) provides a buffer between a black pixel value and the next darkest local pixel tone value, a second guard band (GB1) provides a buffer between the local pixel tone values of the dark band and the mid band, a third guard band (GB2) provides a buffer between the local pixel tone values of the mid band and the highlight band, and a fourth guard band (GB3) provides a buffer between a white pixel value and the next highest local pixel tone value.
[0073] Aspects of the present disclosure recognize that the human visual system can perceive more detail in dark regions of an image compared to mid-tones. Thus, in some embodiments, the dark band may span a wider range of pixel brightness than the mid-band (such that TH0-TH1>TH1-TH2). Thus, the dark band may experience more histogram stretching than the mid-band. Aspects of the present disclosure further recognize that the human visual system can perceive more detail in bright regions of an image compared to dark regions. Thus, in some embodiments, the highlight band may span a wider range of pixel brightness than the dark band (such that TH2-TH3>TH0-TH1). Thus, the highlight band may experience more histogram stretching than the dark and mid-bands.
[0074] FIG. 7 illustrates a block diagram of an example system 700 for adaptive detail processing, according to some implementations. The system 700 receives local display data ({J z [x,y] LM ,a z [x,y] LM ,bz [x,y] LM})'s J z , a z , b z Channel output display data ({J z [x,y] out ,a z [x,y] out ,b z [x,y] out})'s J z , a z , b z In some embodiments, the system 700 is an example of the adaptive detail processor 330 of FIG. 3. Referring to FIG. 3, the local display data {J z [x,y] LM ,a z [x,y] LM ,b z [x,y] LM} may be an example of a local color value 306, and the output display data {J z [x,y] out ,a z [x,y] out ,b z [x,y] out} may be an example of display data 308.
[0075] The system 700 includes a tone change estimation component 710, a detail extraction component 720, a tone blending component 730, and a saturation correction component 740. The tone change estimation component 710 estimates the J channels J of the local display data. z [x,y] LM and J of the original image data from which the local display data was derived. z Channel (J z [x,y] in ) between z [x,y]), where ΔJ z [x,y]=|J z [x,y] LM -J z [x,y] inThe original image data is, for example, the image data {J z [x,y] in ,a z [x,y] in ,b z [x,y] in} or may be an example of image data 302 in FIG.
[0076] The detail extraction component 720 calculates ΔJ z The detail extraction component 720 is configured to detect edges or texture in the image based on the [x,y] values. In some aspects, the detail extraction component 720 may generate the control signal 702 based at least in part on the detected edges or texture. More specifically, the control signal 702 may indicate whether each pixel value corresponds to a flat region of the image or to a region in which texture is present. In some embodiments, if the pixel value corresponds to a region in which texture is present, the control signal 702 may further indicate a magnitude of contrast associated with the region in which texture is present.
[0077] The tone blending component 730 performs J of the image data. z Channel J z [x,y] in and J of local display data z Channel J z [x,y] LM and the J of the output display data based on the control signal 702. z Channel J z [x,y] out More specifically, J z [x,y] out Each value of J z [x,y] in Individual values of and J z [x,y] LMIn some embodiments, the tone blending component 730 may control or adjust the amount of blending based on the control signal 702 to preserve high frequency detail in low contrast areas of the image.
[0078] For example, the tone blending component 730 may select J when the control signal 702 indicates that the pixel value corresponds to a flat region of the image. z [x,y] LM The value of J z [x,y] in As a result, for pixel values that correspond to flat regions of the image, z [x,y] out The value of J z [x,y] LM may be the same as the value of J z [x,y] out =J z [x,y] LM On the other hand, the tone blending component 730 may use J if the control signal 702 indicates that the pixel value corresponds to a portion of the image where texture is present. z [x,y] LM The value of J z [x,y] in As a result, for pixel values that relate to textured areas of the image, z [x,y] out The value of J z [x,y] LM Value of and J z [x,y] in It may be a combination of values of J z [x,y] out =αJ z [x,y] LM +βJ z [x,y] in , where α and β depend on the amount of contrast in the region.
[0079] The saturation correction component 740 adjusts the J of the output display data.z Channel J z [x,y] out Based on the local display data, z and b z Channel A z [x,y] LM , b z [x,y] LM Output the display data of a z and b z Channel A z [x,y] out , b z [x,y] out 5, the luminance of an image can affect the perception of color in an image. Thus, in some embodiments, the local saturation mapping component 520 maps the hue of a given pixel to a saturation value h , in order to compensate for any changes to the luminance of the pixel (e.g., by the tone blending component 730). z and saturation C z may be adjusted (using equations 5 and 6).
[0080] 8 illustrates another block diagram of an example image processing system 800 according to some implementations. Image processing system 800 is configured to convert image data captured by a source device (e.g., an imaging device) into display data for display by a target device (e.g., a display device). In some embodiments, image processing system 800 may be an example of image processor 120 of FIG. 1 or image processing system 200 of FIG. 2.
[0081] The image processing system 800 comprises a device interface 810, a processing system 820, and a memory 830. The device interface 810 is configured to communicate with a source device and a target device. In some embodiments, the device interface 810 comprises an image source interface (I / F) 812 and an image target interface 814. The image source interface 812 is configured to receive image data from a source device. In some embodiments, the image source interface 812 receives first image data representing an image generated by an imaging device based on an optical-to-electrical transfer function (OETF). The image target interface 814 is configured to provide display data to the target device.
[0082] The memory 830 comprises an image data buffer 831 configured to store the received image data and any intermediate data resulting from the image processing operations. The memory 830 also comprises a non-transitory computer readable medium (such as one or more non-volatile memory elements, e.g., EPROM, EEPROM, flash memory, hard drive) that may store at least the following software (SW) modules: an inverse optical-to-electrical transfer function (IOETF) SW module 832 for converting the first image data based on an inverse function of an OETF associated with the imaging device into second image data including red (R), green (G) and blue (B) channels associated with an RGB color space; The second image data is z a z b z Lightness (J) related to color space z ), red-green (a z ) and yellow-blue (b z a Uniform Color Space (UCS) conversion SW module 833 for converting the third image data having 1-channels; of the third image data based at least in part on a dynamic range or color gamut corresponding to the display device. z , a z and b zChannel J of the first display data z , a z and b z A Color Volume Mapping (CVM) SW module 834 for mapping to the channels; and A target drawing SW module 835 for processing the first display data so that the first display data is suitable for display by the display device. Each software module contains instructions that, when executed by processing system 820, cause image processing system 800 to perform a corresponding function.
[0083] The processing system 820 may comprise any suitable processor or processors capable of executing scripts or instructions of one or more software programs stored in the image processing system 800 (e.g., in the memory 830). For example, the processing system 820 may execute an IOETF SW module 832 to convert the first image data into second image data based on an inverse function of an OETF associated with the imaging device, where the second image data includes R, G, and B channels associated with an RGB color space. The processing system 820 may execute a UCS conversion SW module 833 to convert the second image data into J z a z b z J related to color space z , a z and b z The processing system 820 may also execute a CVM SW module 834 to convert the J of the third image data based at least in part on the dynamic range or color gamut corresponding to the display device. z , a z and b z Channel J of the first display data z , a z and b z Additionally, the processing system 820 may execute a target rendering SW module 835 to process the first display data so that it is suitable for display by a display device.
[0084] 9 illustrates an example flowchart showing an example operation 900 for image processing according to some implementations. In some embodiments, the example operation 900 may be performed by an image processing system (such as, for example, image processor 120 of FIG. 1 or image processing systems 200 or 800 of FIGS. 2 and 8, respectively).
[0085] The image processing system receives 910 first image data representative of an image generated by an imaging device based on the OETF. The image processing system converts 920 the first image data into second image data based on the inverse of the OETF, where the second image data includes red (R), green (G), and blue (B) channels associated with an RGB color space. The image processing system further converts the second image data into J z a z b z Lightness (J) related to color space z ), red-green (a z ) and yellow-blue (b z The image processing system converts the third image data into third image data having J (930) channels based at least in part on a dynamic range or color gamut associated with the display device. z , a z and b z Channel J of the first display data z , a z and b z The first representation data is then mapped to the channel (940). The image processing system processes the first representation data to make it suitable for display by the display device (950).
[0086] In some embodiments, converting the second image data to the third image data may include converting the second image data to fourth image data having long wavelength (L), mid wavelength (M) and short wavelength (S) channels associated with an LMS color space, converting the fourth image data to fifth image data based on a perceptual quantizer (PQ) transfer function, and converting the fifth image data to third image data in a PQ domain associated with the PQ transfer function.
[0087] In some aspects, the image processing system may further receive metadata associated with the first image data, where the metadata is indicative of a dynamic range or color gamut associated with the mastering display. In some embodiments, the metadata may include color calibration data, maximum frame average light, maximum content light, RGB primaries, white point, or display maximum and minimum light associated with the mastering display.
[0088] In some embodiments, the third image data J z , a z and b z Channel J of the first display data z , a z and b z The mapping to the channel may further include generating a plurality of look-up tables (LUTs) based at least in part on the received metadata and a dynamic range or color gamut associated with the display device, and generating a J-channel of the third image data based at least in part on one or more first LUTs of the plurality of LUTs. z , a z and b z Channel J of the first display data z and calculating a J of the third image data based at least in part on one or more second LUTs of the plurality of LUTs. z , a z and b z The channel is a of the first display data z and b z and mapping the signal to a channel.
[0089] In some embodiments, the third image data J z , a z and b z a of the first display data of the channel z and b z The mapping of the third image data to the J channel is based on the one or more second LUTs. z , a z and b za first representation data representation of a first feature based on the first channel of the first image; a second representation data representation of a first feature based on the first channel of the first image; a third representation data representation of a first feature based on the first channel of the first image; z and b z and remapping the received signal to a channel.
[0090] In some embodiments, the third image data J z , a z and b z a of the first display data of the channel z and b z The mapping to the J channel of the third image data z , a z and b z The method may include mapping the channels to global tone values within a spectrum of values partitioned into a plurality of bands based on one or more first LUTs, and remapping the global tone value in each of the plurality of bands to a respective equalized tone value within the band based on a histogram equalization of the global tone values.
[0091] In some embodiments, the third image data J z , a z and b z J of the first display data of the channel z The mapping to the channel may further include extracting image detail based on a difference between the equalized tone value and the Jz channel of the third image data, and adjusting the equalized tone value based on the extracted detail.
[0092] In some embodiments, the third image data J z , a z and b z a of the first display data of the channel z and b z The mapping to the J channel of the third image data z , a z and b zThe method may include mapping the channels to a global saturation value based on one or more second LUTs, and dynamically adjusting the global saturation value based at least in part on the equalized tone value.
[0093] In some embodiments, processing the first display data may include converting the first display data to second display data having R, G, and B channels associated with an RGB color space, and converting the second display data to third display data based on an inverse function of an EOTF associated with the display device.
[0094] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different technologies or techniques. For example, the data, instructions, commands, information, signals, bits, symbols and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.
[0095] Furthermore, those skilled in the art will appreciate that the various exemplary logic blocks, modules, circuits, and algorithmic steps described in connection with the aspects disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various exemplary components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, and such implementation decisions should not be interpreted as causing a departure from the technical scope of the present disclosure.
[0096] The methods, sequences or algorithms described in connection with the aspects disclosed herein may be embodied directly in hardware, in software executed by a processor, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. Alternatively, the storage medium may be integral to the processor.
[0097] In the foregoing specification, the embodiments have been described with reference to specific examples thereof. It will be apparent, however, that various modifications and changes can be made therein without departing from the broader scope of the disclosure as set forth in the appended claims. The specification and drawings are therefore to be regarded in an illustrative rather than a restrictive sense.
Claims
1. 1. A method of image processing, comprising the steps of: receiving first image data representative of an image generated by an imaging device based on an optical-to-electrical transfer function (OETF); converting the first image data into second image data having red (R), green (G) and blue (B) channels associated with an RGB color space based on an inverse function of the OETF; The second image data is z a z b z Lightness (J) related to color space z ), red-green (a z ) and yellow-blue (b z ) converting the image data into third image data having three channels; The J of the third image data based at least in part on a dynamic range or color gamut associated with a display device. z , a z and b z Channel J of the first display data z , a z and b z Mapping the signal to a channel; processing the first display data for display by the display device; The method includes:
2. Converting the second image data into the third image data, converting the second image data into fourth image data having long wavelength (L), mid wavelength (M) and short wavelength (S) channels associated with an LMS color space; converting the fourth image data into fifth image data based on a perceptual quantizer (PQ) transfer function; transforming the fifth image data into the third image data in a PQ domain associated with the PQ transfer function; Includes The method of claim 1.
3. receiving metadata associated with the first image data; The metadata indicates a dynamic range or color gamut associated with the mastering display. The method of claim 1.
4. the metadata includes color calibration data associated with the mastering display, maximum frame average light, maximum content light, RGB primaries, white point, or display maximum and minimum light; The method according to claim 3.
5. The J of the third image data z , a z and b z The Jth channel of the first display data z , a z and b z Mapping to a channel generating a plurality of look-up tables (LUTs) based at least in part on the received metadata and the dynamic range or color gamut associated with the display device; The J of the third image data based at least in part on one or more first LUTs of the plurality of LUTs. z , a z and b z The channel is J of the first display data. z Mapping the signal to a channel; The J of the third image data based at least in part on one or more second LUTs of the plurality of LUTs. z , a z and b z The channel is the a z and b z Mapping the signal to a channel; Includes The method according to claim 3.
6. The J of the third image data z , a z and b z The channel is the a z and b zMapping to a channel The J of the third image data based on the one or more second LUTs. z , a z and b z Mapping the channels to a global saturation value; classifying one or more features in the image based on a neural network model; based at least in part on the classification of the one or more features, the global saturation value being z and b z remapping to a channel; Includes The method according to claim 5.
7. The J of the third image data z , a z and b z The Jth channel of the first display data z Mapping to a channel The J of the third image data based on the one or more first LUTs. z , a z and b z Mapping the channels to global tone values within a spectrum of values partitioned into a number of bands; remapping a global tone value for each band of the plurality of bands based on a histogram equalization of the global tone value to a distinct equalized tone value within the band; Includes The method according to claim 5.
8. The J of the third image data z , a z and b z The Jth channel of the first display data z The mapping to the channel may further include: The equalized tone value and the J of the third image data z extracting detail in the image based on differences between the channels; adjusting the equalization tone value based on the extracted detail; and Includes The method according to claim 7.
9. The adjusting of the equalization tone value adjusts the equalization tone value to the J of the third image data. z Includes selective blending with channels The method according to claim 8.
10. determining whether each pixel of the image corresponds to a flat region or to a region in which texture is present; The selective blending of the equalized tone values is based on whether each pixel of the image corresponds to a flat area or to an area in which texture is present.
10. The method of claim 9.
11. The J of the third image data z , a z and b z The channel is the a z and b z Mapping to a channel Based on the second LUT, the J of the third image data z , a z and b z Mapping the channels to a global saturation value; dynamically adjusting the global saturation value based at least in part on the equalized tone value; Includes The method according to claim 7.
12. Processing the first display data converting the first display data into second display data having R, G, and B channels associated with the RGB color space; converting the second display data into third display data based on an inverse of an electro-optical transfer function (EOTF) associated with the display device; Includes The method of claim 1.
13. 1. An image processor comprising: A processing system; a memory that, when executed by the processing system, causes the image processor to: receiving first image data representative of an image generated by an imaging device based on an optical-to-electrical transfer function (OETF); converting the first image data into second image data having red (R), green (G) and blue (B) channels associated with an RGB color space based on an inverse function of the OETF; The second image data is z a z b z Lightness (J) related to color space z ), red-green (a z ) and yellow-blue (b z ) into third image data having three channels; The J of the third image data based at least in part on a dynamic range or color gamut associated with a display device. z , a z and b z Channel J of the first display data z , a z and b z Map it to a channel, a memory storing instructions for processing the first display data for display by the display device; Equipped Image processor.
14. converting the second image data into the third image data; converting the second image data into fourth image data having long wavelength (L), mid wavelength (M) and short wavelength (S) channels associated with an LMS color space; converting the fourth image data into fifth image data based on a perceptual quantizer (PQ) transfer function; transforming the fifth image data into the third image data in a PQ domain associated with the PQ transfer function; Includes 12. An image processor according to claim 11.
15. Execution of the instructions further causes the image processor to receive metadata associated with the first image data; The metadata indicates a dynamic range or color gamut associated with the mastering display.
12. An image processor according to claim 11.
16. The J of the third image data z , a z and b z The Jth channel of the first display data z , a z and b z Mapping to a channel generating a plurality of look-up tables (LUTs) based at least in part on the received metadata and the dynamic range or color gamut associated with the display device; The J of the third image data based at least in part on one or more first LUTs of the plurality of LUTs. z , a z and b z The channel is J of the first display data. z Mapping the signal to a channel; The J of the third image data based at least in part on one or more second LUTs of the plurality of LUTs. z , a z and b z The channel is the a z and b z Mapping the signal to a channel; Includes 14. An image processor according to claim 13.
17. The J of the third image data z , a z and b zThe channel is the a z and b z Mapping to a channel The J of the third image data based on the one or more second LUTs. z , a z and b z Mapping the channels to a global saturation value; classifying one or more features in the image based on a neural network model; based at least in part on the classification, the global saturation value being calculated based on the a z and b z remapping to a channel; Includes 16. An image processor according to claim 15.
18. The J of the third image data z , a z and b z The Jth channel of the first display data z Mapping to a channel The J of the third image data based on the one or more first LUTs. z , a z and b z Mapping the channels to global tone values within a spectrum of values partitioned into a number of bands; remapping a global tone value for each band of the plurality of bands based on a histogram equalization of the global tone value to a distinct equalized tone value within the band; Includes 16. An image processor according to claim 15.
19. The J of the third image data z , a z and b z The Jth channel of the first display data z Mapping to a channel The equalized tone value and the J of the third image data zextracting detail in the image based on differences between the channels; adjusting the equalization tone value based on the extracted detail; and Further includes 20. An image processor as claimed in claim 17.
20. The J of the third image data z , a z and b z The channel is the a z and b z Mapping to a channel The J of the third image data based on the one or more second LUTs. z , a z and b z Mapping the channels to a global saturation value; dynamically adjusting the global saturation value based at least in part on the equalized tone value; Includes 20. An image processor as claimed in claim 17.
