Display device and image bright color enhancement method

Abstract

The invention relates to a display device and an image bright color enhancement method. A display device includes a display and a controller. Wherein the display is configured to display a user interface; the controller is configured to: a display configured to display an image and / or a user interface; the controller is configured to convert the down-sampling RGB image to obtain an ICtCp image; inputting the ICtCp image into a first convolutional neural network, and outputting a one-dimensional lookup table of a brightness component I and a two-dimensional lookup table of a chrominance component CtCp; performing ICtCp color space inverse conversion to generate a first RGB three-dimensional lookup table; fixing network parameters of the first convolutional neural network, and inputting the downsampling RGB image into a second convolutional neural network to generate a residual RGB three-dimensional lookup table; and according to the first RGB three-dimensional lookup table and the residual RGB three-dimensional lookup table, performing lookup mapping operation on the original RGB image to generate a target RGB image. By adopting the method, the accuracy and computing power requirements are considered, so that the computing power of the television end can be better adapted.

Description

Technical Field

[0001] This application relates to the field of image brightness and color processing technology, and in particular to a display device and an image brightness and color enhancement method. Background Technology

[0002] The emergence of SDR to HDR technology stems from upgrades in display technology and users' demand for higher image quality. With the widespread adoption of HDR (High Dynamic Range) displays, the limitations of traditional SDR (Standard Dynamic Range) content in terms of brightness, contrast, and color performance have become increasingly apparent. SDR content typically has a narrow dynamic range (typical brightness 0-100 nits) and a smaller color space (e.g., Rec.709), while HDR offers higher peak brightness (up to several thousand nits), a wider color gamut (e.g., Rec.2020), and more delicate details in both light and dark areas. To improve the dynamic range and color performance of images and allow existing SDR content to display better on HDR devices, SDR to HDR technology was developed. However, the application scope of SDR to HDR in brightness expansion, color enhancement, and detail preservation still needs further development.

[0003] Therefore, there is an urgent need for a display device and image brightness enhancement method that has more efficient computing power than the algorithm used in traditional SDR to HDR conversion, and balances accuracy and computing power requirements to better adapt to the computing power of TVs. Summary of the Invention

[0004] This application provides a display device and an image brightness and color enhancement method, which has more efficient computing power than the algorithm used in traditional SDR to HDR conversion, and balances accuracy and computing power requirements to better adapt to the computing power of TV terminals.

[0005] In a first aspect, some embodiments provide a display device, including a display and a controller. The display is configured to display a user interface; the controller is configured to:

[0006] The display is configured to show images and / or a user interface;

[0007] The controller is configured as follows:

[0008] The downsampled RGB image is converted from the RGB color space to the ICtCp color space to obtain the ICtCp image;

[0009] The ICtCp image is input into a preset first convolutional neural network, which outputs a one-dimensional lookup table for the luminance component I and a two-dimensional lookup table for the chrominance component CtCp.

[0010] Based on the one-dimensional lookup table of the luminance component I and the two-dimensional lookup table of the chrominance component CtCp, perform inverse conversion of the ICtCp color space to generate the first RGB three-dimensional lookup table.

[0011] The network parameters of the first convolutional neural network are fixed, and the downsampled RGB image is input into a preset second convolutional neural network to generate a residual RGB three-dimensional lookup table based on the RGB color space;

[0012] Based on the first RGB three-dimensional lookup table and the residual RGB three-dimensional lookup table, a lookup mapping operation is performed on the original RGB image to generate the target RGB image.

[0013] The above technical solution has the following advantages or beneficial effects: By first converting the image to the ICtCp color space, which has better perceptual uniformity, and then using a first convolutional neural network to process the luminance and chrominance components separately, it is possible to effectively separate and independently enhance luminance and color information, avoiding the hue distortion or detail loss caused by mutual interference between luminance and color in the traditional RGB space, thereby improving the accuracy of the interpretation from the source. By fixing the network parameters of the ICtCp path and introducing a second residual network to learn and generate a residual lookup table in the RGB space, it is possible to perform refined color correction and detail compensation on the basis of the obtained primary enhancement results. This dual-path collaborative mechanism, combined with the idea of ​​residual learning, enables the model to not only have powerful feature extraction and mapping capabilities. Finally, by fusing the two lookup tables and performing efficient lookup mapping on the original high-resolution image, it is possible to concentrate the complex computational load on the low-resolution analysis stage at the front end while ensuring top-level visual enhancement effects, thereby greatly reducing the real-time computing overhead of the terminal device, enabling this high-quality SDR-to-HDR conversion technology to be deployed and run in real time on devices with limited computing power, such as televisions.

[0014] In one embodiment, the controller performs a lookup mapping operation on the original RGB image based on the first RGB three-dimensional lookup table and the residual RGB three-dimensional lookup table to generate the target RGB image, and is configured as follows:

[0015] The first RGB three-dimensional lookup table and the residual RGB three-dimensional lookup table are superimposed element by element, and the enhanced features from the ICtCp color space and the RGB color space are combined to obtain the final RGB three-dimensional lookup table.

[0016] Using the final RGB 3D lookup table, a 3D lookup table mapping operation is performed on the original RGB image to generate the target RGB image.

[0017] The above technical solution has the following advantages or beneficial effects: by combining the two lookup tables element by element, the precise luminance-color separation relationship preserved by the ICp path and the fine color residual correction learned by the RGB path can be effectively fused, thereby generating a final RGB 3D lookup table with more complete information and more accurate mapping. Furthermore, by using this fused lookup table to perform mapping operations on the original image, the expansion of the dynamic range of brightness, the enhancement of color saturation, and the color correction of local details can be completed simultaneously in a single efficient lookup calculation, ultimately presenting a high-quality conversion effect in the output image that is closer to the native HDR content.

[0018] In one embodiment, the controller performs the conversion of a downsampled RGB image from the RGB color space to the ICtCp color space to obtain an ICtCp image, and is configured as follows:

[0019] The downsampled RGB image is converted from a gamma-coded nonlinear RGB color space to a linear RGB color space;

[0020] Using the first color conversion matrix, the downsampled RGB image in the linear RGB color space is converted to the LMS color space to obtain an LMS color space image;

[0021] The LMS color space image is subjected to a nonlinear transfer function transformation, wherein the nonlinear transfer function includes at least a perceptual quantization curve or a mixed log-gamma curve.

[0022] Using the second color conversion matrix, the LMS color space image after nonlinear transfer function transformation is converted to the ICtCp color space to obtain the ICtCp image.

[0023] The above technical solution has the following advantages or beneficial effects: By converting the image to the linear RGB space, the influence of gamma correction on the light signal can be eliminated, ensuring that color calculation is based on real physical brightness. By converting to the LMS space and performing PQ / HLG nonlinear transformation, efficient data compression can be achieved while preserving high dynamic range information. Finally, by converting to the ICtCp space, a perceptually uniform brightness-color separation representation can be obtained, effectively avoiding the brightness-color crosstalk problem of traditional color spaces, and significantly improving the accuracy and effect quality of subsequent independent brightness and color enhancement.

[0024] In one embodiment, the controller is configured to perform the following actions before inputting the ICtCp image into a preset first convolutional neural network:

[0025] Calculate the pixel statistics of each channel in the ICtCp image, wherein the pixel statistics include the pixel mean and pixel variance;

[0026] Based on the pixel statistics, the ICtCp image is subjected to zero-centering and variance normalization.

[0027] The above technical solution has the following advantages or beneficial effects: by calculating the pixel mean and variance of each channel of the image, the distribution characteristics of the input data can be accurately quantified. Furthermore, by performing zero-centering and variance normalization based on this statistical value, the distribution of each channel of the input data can be uniformly adjusted to a standard normal distribution with a mean of zero and a variance of one, effectively eliminating the internal covariate shift problem caused by differences in brightness or color distribution of the input data.

[0028] In one embodiment, the first convolutional neural network includes parallel and structurally independent luminance component processing subnetworks and chrominance component processing subnetworks; the controller executes inputting the ICtCp image into the preset first convolutional neural network and outputting a one-dimensional lookup table for the luminance component I and a two-dimensional lookup table for the chrominance component CtCp, configured as follows:

[0029] The ICtCp image is input into the luminance component processing sub-network. Hierarchical feature extraction is performed using multiple consecutive convolutional and pooling layers in the luminance component processing sub-network. The extracted high-dimensional feature vector is mapped into a one-dimensional lookup table representing the luminance mapping relationship through a fully connected layer at the end. The one-dimensional lookup table is a vector containing a first preset number of lookup points.

[0030] The ICtCp image is input into the chroma component processing sub-network. Feature extraction and size adjustment are performed using multiple consecutive convolutional layers, pooling layers and at least one fill layer in the chroma component processing sub-network. The feature map is mapped into a two-dimensional lookup table representing the chroma mapping relationship through the convolutional layer at the end. The two-dimensional lookup table is a two-dimensional array with a structure of a second preset number of points multiplied by a third preset number of points.

[0031] The above technical solution has the following advantages or beneficial effects: By utilizing convolutional and pooling layers in the luminance sub-network for hierarchical feature extraction and mapping it to a one-dimensional lookup table through fully connected layers, it can accurately learn and represent the complex luminance mapping relationship from standard dynamic range to high dynamic range, ensuring that the details of image brightness and darkness are fully expanded and preserved. Simultaneously, by performing feature extraction and size adjustment through a structure containing fill layers in the chroma sub-network and outputting a two-dimensional lookup table through convolutional layers, it can effectively capture the spatial context information of color components and accurately establish a nonlinear transformation model for chroma components. This parallel processing mechanism of luminance and color separation fundamentally avoids the mutual interference caused by the mixing of luminance and color in traditional RGB space.

[0032] In one embodiment, the controller executes an inverse ICtCp color space conversion based on a one-dimensional lookup table of the luminance component I and a two-dimensional lookup table of the chrominance component CtCp to generate a first RGB three-dimensional lookup table, which is configured as follows:

[0033] Trilinear interpolation is performed on the one-dimensional lookup table of the luminance component I and the two-dimensional lookup table of the chrominance component CtCp to generate an ICtCp spatial lookup table with a three-dimensional coordinate structure.

[0034] The ICtCp space lookup table is sequentially transformed to the LMS color space through the first inverse transformation matrix, restored to the linear RGB color space through nonlinear inverse transformation processing, and the first RGB three-dimensional lookup table is generated by applying the second inverse transformation matrix.

[0035] The above technical solution has the following advantages or beneficial effects: By performing trilinear interpolation on the one-dimensional luminance lookup table and the two-dimensional chrominance lookup table, a complete three-dimensional ICtCp space lookup table can be quickly reconstructed, fully restoring the three-dimensional color mapping relationship while maintaining data compactness. By sequentially performing inverse matrix transformation and nonlinear inverse processing, the enhanced effect of luminance-color separation can be accurately maintained during the inverse color space transformation, ensuring that color distortion or loss of detail is avoided when converting to RGB space.

[0036] In one embodiment, the controller executes the input of the downsampled RGB image into a preset second convolutional neural network to generate a residual RGB three-dimensional lookup table based on the RGB color space, configured as follows:

[0037] The downsampled RGB image is input into the intermediate layer of the first convolutional neural network with fixed network parameters and the input layer of the second convolutional neural network;

[0038] Using the intermediate layer of the first convolutional neural network, at least one intermediate feature map is extracted and used as an auxiliary feature to characterize the enhanced prior knowledge in the ICtCp space.

[0039] Deep features of the RGB color space are extracted from the downsampled RGB image through the front-end layer of the second convolutional neural network.

[0040] The auxiliary features and the deep features are fused across color spaces to obtain a fused feature map;

[0041] The fused feature map is input into the subsequent residual block group of the second convolutional neural network for forward propagation calculation. Through the output layer of the second convolutional neural network, a residual RGB three-dimensional lookup table is generated for fine correction of the first RGB three-dimensional lookup table.

[0042] The above technical solution has the following advantages or beneficial effects: by fusing the enhanced prior features extracted from the ICtCp space with the deep features of the RGB space, a complementary feature representation that takes into account the characteristics of different color spaces can be constructed. By generating a residual lookup table based on these features using a residual network, subtle differences between the initial result and the ideal state can be accurately captured.

[0043] In one embodiment, the controller is configured to perform a three-dimensional lookup table mapping operation on the original RGB image using the final RGB three-dimensional lookup table to generate the target RGB image.

[0044] For each pixel in the original RGB image, the values ​​of the R, G, and B channels are extracted and used to construct three-dimensional input coordinates;

[0045] The three-dimensional input coordinates are mapped to a three-dimensional discrete grid composed of the final RGB three-dimensional lookup table. By performing trilinear interpolation calculation, the output RGB value corresponding to the three-dimensional input coordinates and after dynamic range and color gamut enhancement is obtained.

[0046] The target RGB image is formed by iterating through all pixels in the original RGB image and collecting all calculated output RGB values.

[0047] The above technical solution has the following advantages or beneficial effects: by mapping the three-dimensional RGB coordinates of each pixel to the enhanced color space, a global nonlinear transformation of the relationship between color and brightness can be achieved. By performing trilinear interpolation calculations in a three-dimensional discrete grid, the dynamic range expansion and color gamut mapping of each pixel can be accurately completed while maintaining a continuous and smooth color transition.

[0048] Secondly, some embodiments also provide an image brightness and color enhancement method, applied to a display device provided by various possible implementations of the first aspect, the method comprising:

[0049] The downsampled RGB image is converted from the RGB color space to the ICtCp color space to obtain the ICtCp image;

[0050] The ICtCp image is input into a preset first convolutional neural network, which outputs a one-dimensional lookup table for the luminance component I and a two-dimensional lookup table for the chrominance component CtCp.

[0051] Based on the one-dimensional lookup table of the luminance component I and the two-dimensional lookup table of the chrominance component CtCp, perform inverse conversion of the ICtCp color space to generate the first RGB three-dimensional lookup table.

[0052] The network parameters of the first convolutional neural network are fixed, and the downsampled RGB image is input into a preset second convolutional neural network to generate a residual RGB three-dimensional lookup table based on the RGB color space;

[0053] Based on the first RGB three-dimensional lookup table and the residual RGB three-dimensional lookup table, a lookup mapping operation is performed on the original RGB image to generate the target RGB image.

[0054] In one embodiment, the step of performing a lookup mapping operation on the original RGB image based on the first RGB three-dimensional lookup table and the residual RGB three-dimensional lookup table to generate the target RGB image includes:

[0055] The first RGB three-dimensional lookup table and the residual RGB three-dimensional lookup table are superimposed element by element, and the enhanced features from the ICtCp color space and the RGB color space are combined to obtain the final RGB three-dimensional lookup table.

[0056] Using the final RGB 3D lookup table, a 3D lookup table mapping operation is performed on the original RGB image to generate the target RGB image. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application;

[0059] Figure 2 This is a schematic diagram of the hardware configuration of a display device provided in some embodiments of this application;

[0060] Figure 3 This is a schematic diagram of the hardware configuration of the control device provided in some embodiments of this application;

[0061] Figure 4 This is a schematic diagram of the software configuration of a display device provided in some embodiments of this application;

[0062] Figure 5 This is a schematic diagram of the SDR to HDR conversion process in traditional technology.

[0063] Figure 6 This is an architectural diagram of the image brightness enhancement method in some embodiments of this application;

[0064] Figure 7 This is a flowchart illustrating the image brightness enhancement method in some embodiments of this application;

[0065] Figure 8 This is a flowchart illustrating the interaction between the controller and the display in some embodiments of this application;

[0066] Figure 9 This is a structural block diagram of an image brightness enhancement device in some embodiments of this application. Detailed Implementation

[0067] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0068] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0069] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0070] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0071] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0072] In this embodiment, the display device 200 generally refers to a device with screen display and data processing capabilities. For example, the display device 200 includes, but is not limited to, smart TVs, mobile terminals, computers, monitors, advertising screens, wearable devices, virtual reality devices, augmented reality devices, etc.

[0073] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application. For example... Figure 1As shown, users can operate the display device 200 via touch operation, mobile terminal 300, and control device 100. For example, control device 100 can be a remote control, stylus, gamepad, etc.

[0074] The mobile terminal 300 can function as a control device for human-computer interaction between the user and the display device 200. It can also function as a communication device for establishing a communication connection with the display device 200 and exchanging data. In some embodiments, the mobile terminal 300 can have software applications installed on it and communicate with the display device 200 via network communication protocols to achieve one-to-one control and data communication. Furthermore, it can transmit audio and video content displayed on the mobile terminal 300 to the display device 200 for synchronized display.

[0075] like Figure 1 The diagram also shows that the display device 200 communicates with the server 400 via various communication methods. This allows the display device 200 to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0076] Display device 200 can provide broadcast television reception function, and can also be equipped with intelligent network television function that provides computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

[0077] Figure 2 Provided for some embodiments of this application Figure 1 Hardware configuration block diagram of display device 200.

[0078] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0079] In some embodiments, detector 230 is used to acquire signals from the external environment or to interact with the outside world. For example, detector 230 includes a light receiver, a sensor for acquiring ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to acquire external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0080] In some embodiments, the display 260 includes display function components for presenting an image and driving components for driving the image display. The display 260 is used to receive and display image signals output from the controller 250. The display 260 can connect to the remote control via infrared, Bluetooth, or other wireless methods, and can respond to button operations initiated by the user on the remote control. For example, the display 260 can be used to display video content, image content, menu control interface components, and user-controlled UI interfaces.

[0081] In some embodiments, the communication device 220 is a component used to communicate with external devices or the server 400 according to various communication protocol types. The display device 200 may have multiple communication devices 220 depending on the supported communication methods. For example, when the display device 200 supports wireless network communication, it may have a communication device 220 with WiFi functionality. When the display device 200 supports Bluetooth connectivity, it needs to have a communication device 220 with Bluetooth functionality.

[0082] The communication device 220 enables the display device 200 to communicate with external devices or the server 400 via wireless or wired connections. Wired connections utilize data cables, interfaces, or other components to connect the display device 200 to external devices. Wireless connections utilize wireless signals or wireless networks. The display device 200 can directly establish a connection with external devices or indirectly through gateways, routers, or other connection devices.

[0083] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200.

[0084] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0085] In some embodiments, a user can input user commands through a graphical user interface (GUI) displayed on a display 260, and the user input interface receives user input commands through the graphical user interface (GUI).

[0086] In some embodiments, the audio output device 270 can be a built-in speaker of the display device 200 or an external audio output device connected to the display device 200. For the external audio output device connected to the display device 200, the display device 200 may also be provided with an external audio output terminal, through which the audio output device can be connected to the display device 200 to output sound from the display device 200.

[0087] In some embodiments, the user input interface 280 can be used to receive instructions from user input.

[0088] Figure 3 Provided for some embodiments of this application Figure 1 Hardware configuration block diagram of the central control device. (Example) Figure 3 As shown, the control device 100 may include: a controller 110, a communication interface 130, a user input / output interface, a memory, and a power supply.

[0089] The control device 100 is configured to control the display device 200, and to receive user input operation commands and convert the operation commands into commands that the display device 200 can recognize and respond to, thus acting as an intermediary for interaction between the user and the display device 200.

[0090] In some embodiments, the control device 100 may be an intelligent device. For example, the control device 100 may be equipped with various applications for controlling the display device 200 according to user needs.

[0091] In some embodiments, such as Figure 1 As shown, the mobile terminal 300 or other smart electronic devices can perform similar functions to the control device 100 after installing the application of the control display device 200.

[0092] The controller 110 includes a processor 112, RAM 113, ROM 114, a communication interface 130, and a communication bus. The controller 110 is used to control the operation of the control device 100, as well as the communication and cooperation between internal components and the external and internal data processing functions.

[0093] Under the control of the controller 110, the communication interface 130 enables communication of control signals and data signals with the display device 200. The communication interface 130 may include at least one of other near-field communication modules such as WiFi chip 131, Bluetooth module 132, and NFC module 133.

[0094] User input / output interface 140, wherein the input interface includes at least one of other input interfaces such as microphone 141, touchpad 142, sensor 143, and button 144.

[0095] In some embodiments, the control device 100 includes at least one of a communication interface 130 and an input / output interface 140. The control device 100 is configured with the communication interface 130, such as a WiFi, Bluetooth, or NFC module, which can encode user input commands via WiFi, Bluetooth, or NFC protocols and send them to the display device 200.

[0096] The memory 190 is used to store various operating programs, data, and applications for driving and controlling the control device 100 under the control of the controller. The memory 190 can also store various control signal instructions input by the user.

[0097] The power supply 180 is used to provide operating power support for the various components of the control device 100 under the control of the controller.

[0098] In order to perform user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program used to manage and control the hardware and software resources in the display device 200. The operating system can (control the display device) provide a user interface, allowing users to interact with the display device 200 and supporting the running of various applications.

[0099] It should be noted that the operating system can be a native operating system based on a specific operating platform, a third-party operating system that is deeply customized based on a specific operating platform, or an independent operating system specifically developed for display devices.

[0100] An operating system can be divided into different modules or levels based on the functions it implements, for example... Figure 4 As shown, in some embodiments, the system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the System Library layer, and the Kernel layer.

[0101] In some embodiments, the application layer provides services and interfaces for applications, enabling the display device 200 to run applications and interact with the user based on the applications. The application layer may contain at least one application, which may be a built-in Windows program, system settings program, or clock program of the operating system; or it may be an application developed by a third-party developer. In specific implementations, the application packages in the application layer are not limited to the examples above.

[0102] The framework layer provides application programming interfaces (APIs) and a programming framework for applications. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications within the application layer. Through the API, applications can access system resources and obtain system services during execution.

[0103] like Figure 4 As shown, the application framework layer in this embodiment includes a view system, managers, and content providers. The view system designs and implements the application's interface and interactions, and includes lists, grids, text boxes, and buttons. The managers include at least one of the following modules: an activity manager for interacting with all running activities in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to application packages currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0104] In some embodiments, the Activity Manager manages the lifecycle of individual applications and common navigation and back functions, such as controlling application exit, opening, and back actions. The Window Manager manages all window programs, such as obtaining the screen size, determining if a status bar is present, locking the screen, capturing the screen, and controlling changes to the display window, such as shrinking the display window, shaking the display, or distorting the display.

[0105] In some embodiments, the system runtime library layer can provide support for the framework layer. When the framework layer is used, the operating system runs the instruction library contained in the system runtime library layer, such as the C / C++ instruction library, to implement the functions to be performed by the framework layer.

[0106] In some embodiments, the kernel layer is a functional layer situated between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. For example, ... Figure 4As shown, hardware drivers can be configured in the kernel layer. The kernel layer can contain at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0107] It should be noted that the above examples are merely a simple division of operating system functions and do not limit the specific form of the operating system of the display device 200 in this application embodiment. Depending on the function of the display device, the type of operating system, and other factors, the number of levels and the specific level type of the operating system may be expressed in other forms.

[0108] The emergence of SDR to HDR technology stems from upgrades in display technology and users' demand for higher image quality. With the widespread adoption of HDR (High Dynamic Range) displays, the limitations of traditional SDR (Standard Dynamic Range) content in terms of brightness, contrast, and color performance have become increasingly apparent. SDR content typically has a narrow dynamic range (typical brightness 0-100 nits) and a smaller color space (e.g., Rec.709), while HDR offers higher peak brightness (up to several thousand nits), a wider color gamut (e.g., Rec.2020), and more delicate details in both light and dark areas. To improve the dynamic range and color performance of images and allow existing SDR content to display better on HDR devices, SDR to HDR technology was developed. However, the application scope of SDR to HDR in brightness expansion, color enhancement, and detail preservation still needs further development.

[0109] More specifically, common SDR to HDR conversion methods, such as Figure 5 As shown, it is usually converted from RGB to Yxy format and then enhanced.

[0110] To address the aforementioned issues, this invention provides a technical solution for SDR to HDR conversion, utilizing a TV-side SOC processor. The specific steps are as follows: First, the TV acquires a 4K resolution SDR signal, which is preprocessed by the SOC processor to obtain a 4K image. Next, the 4K image is downsampled to 256×256 resolution and input into a model to calculate the SDR to HDR luminance and chrominance LUT tables. Color space conversion is then performed to obtain a 3DLut table in RGB space. Adding an RGB3DLut residual network and optimizing RGB3DLut improves the SDR to HDR conversion effect. Then, a 3DLut table lookup is performed on the original 4K signal to convert the SDR signal to an HDR signal. Finally, Gamma mapping and CM processing are performed on the 4K HDR signal to achieve real-time display.

[0111] Based on the foregoing content, embodiments of this application provide a display device, including:

[0112] The display is configured to show images and / or a user interface;

[0113] The controller is configured as follows:

[0114] The downsampled RGB image is converted from the RGB color space to the ICtCp color space to obtain the ICtCp image;

[0115] The ICtCp image is input into a preset first convolutional neural network, which outputs a one-dimensional lookup table for the luminance component I and a two-dimensional lookup table for the chrominance component CtCp.

[0116] Based on a one-dimensional lookup table for the luminance component I and a two-dimensional lookup table for the chrominance component CtCp, an inverse transformation of the ICtCp color space is performed to generate a first RGB three-dimensional lookup table.

[0117] The network parameters of the first convolutional neural network are fixed, and the downsampled RGB image is input into the preset second convolutional neural network to generate a residual RGB three-dimensional lookup table based on the RGB color space;

[0118] Based on the first RGB three-dimensional lookup table and the residual RGB three-dimensional lookup table, a lookup mapping operation is performed on the original RGB image to generate the target RGB image.

[0119] Specifically, firstly, downsampled RGB images refer to low-resolution (e.g., 256×256) images obtained from the original 4K SDR images in the TV video stream through downsampling operations to adapt to the limited computing power of the TV. The purpose is to significantly reduce the amount of data that the subsequent neural network needs to process while ensuring sufficient information. This is a key preprocessing step for the entire method to run in real time on the TV chip.

[0120] like Figure 6As shown, converting a downsampled RGB image from the RGB color space to the ICtCp color space is a precise color space transformation process. Specifically, it's not a one-step process but involves a series of standardized computational steps: First, the common, gamma-encoded, non-linear RGB image is converted back to the linear RGB space to obtain accurate brightness information. Then, a specific transformation matrix is ​​applied to convert the linear RGB to the LMS color space, which more closely approximates the response of cone cells in the human eye. Next, the signal in the LMS space is non-linearly compressed, using either a PQ (Perceptual Quantization) curve or an HLG (Hybrid Log-Gamma) transfer function, to efficiently encode high dynamic range brightness information. Finally, another transformation matrix is ​​used to convert the processed signal to the final ICtCp color space. The ICtCp space is chosen because it has better perceived uniformity of brightness and chromaticity than the traditional YCbCr space. This means that subsequent processing in this space can effectively reduce crosstalk between bright colors, laying the foundation for high-quality brightness-color separation enhancement.

[0121] The ICtCp image is input into a pre-trained first convolutional neural network (CNN). This network is specifically designed for feature extraction and mapping and includes two parallel and structurally independent branches: a luminance component processing sub-network and a chrominance component processing sub-network. The luminance component processing sub-network (e.g., through convolutional, pooling, and finally fully connected layers) processes the ICtCp image and outputs a 1×33 one-dimensional lookup table defining the mapping from input luminance to output enhanced luminance. Simultaneously, the chrominance component processing sub-network (e.g., through convolutional, pooling, padding, and finally 1×1 convolutional layers) outputs a 17×17 two-dimensional lookup table defining the joint mapping between the Ct and Cp chrominance components. This separate design perfectly matches the characteristics of the ICtCp space itself, ensuring that luminance enhancement and color expansion can be performed independently and accurately.

[0122] Based on a one-dimensional lookup table for the luminance component I and a two-dimensional lookup table for the chrominance component CtCp, an inverse ICtCp color space transformation is performed to generate a first RGB three-dimensional lookup table. Since the direct output consists of one-dimensional and two-dimensional tables, they cannot be directly used to process three-dimensional RGB images. Therefore, trilinear interpolation is first used to reconstruct or upsample these two low-dimensional lookup tables into a complete three-dimensional ICtCp space lookup table (e.g., 33×33×33). Then, the inverse process is executed: this three-dimensional ICtCp lookup table is transformed back to LMS space using an inverse transformation matrix, then restored to linear RGB space using a nonlinear inverse transformation, and finally, the final first RGB three-dimensional lookup table is generated using a color transformation matrix. This table carries the initial enhancement effect of luminance-color separation learned in the ICtCp space.

[0123] The parameters of the first convolutional neural network (CNN) are fixed, and the downsampled RGB image is input into a pre-defined second CNN. A residual RGB 3D lookup table is generated based on the RGB color space. Fixing the network parameters means that the weights and biases of the first CNN will not be updated during subsequent training; its network parameters are frozen. At this point, a second processing path is initiated: the same downsampled RGB image is input into another pre-trained second CNN with a residual network structure (e.g., ResNet18). This network's task is not to learn the entire color map from scratch, but to learn a residual or difference. More importantly, to fuse dual-path information, this network extracts deep features in the RGB space while also fusing intermediate layer features extracted by the first CNN with fixed parameters. In this way, the second CNN can refer to the prior knowledge of the ICp path to generate a residual RGB 3D lookup table in the RGB space. This table is used to finely correct the basic RGB 3DLUT obtained in the first step, compensating for any subtle color deviations.

[0124] Finally, based on the first RGB 3D lookup table and the residual RGB 3D lookup table, a lookup mapping operation is performed on the original RGB image to generate the target RGB image. Using the first RGB 3D lookup table obtained in the first step and the residual RGB 3D lookup table obtained in the second step, a final RGB 3D lookup table with more complete information and more accurate mapping is obtained. Then, using this final 3D lookup table, an efficient lookup operation is performed on the initial unsampled, full-resolution original 4K SDR RGB image. Specifically, the (R, G, B) value of each pixel in the original image is extracted as a 3D coordinate, the corresponding position is found in the 3D grid formed by the final RGB 3D lookup table, and the new (R, G, B) value of the pixel after enhancement is calculated through trilinear interpolation. After traversing all pixels, a high dynamic range (HDR) target RGB image with more vibrant colors, richer brightness levels, and clearer details is generated, thus greatly improving the user's viewing experience.

[0125] In this embodiment, by first converting the image to the ICtCp color space, which offers better perceptual uniformity, and then using a first convolutional neural network to process the luminance and chrominance components separately, effective separation and independent enhancement of luminance and chrominance information can be achieved. This avoids the hue distortion or detail loss caused by mutual interference between luminance and chrominance in the traditional RGB space, thereby improving the accuracy of the image from the source. By fixing the network parameters of the ICtCp path and introducing a second residual network to learn and generate a residual lookup table in the RGB space, fine-grained color correction and detail compensation can be performed on top of the obtained primary enhancement results. This dual-path collaborative mechanism, combined with the idea of ​​residual learning, enables the model to not only possess powerful feature extraction and mapping capabilities. Finally, by fusing the two lookup tables and performing efficient lookup mapping on the original high-resolution image, top-level visual enhancement effects can be ensured while concentrating the complex computational load on the low-resolution analysis stage at the front end. This significantly reduces the real-time computational overhead of the terminal device, enabling this high-quality SDR-to-HDR conversion technology to be deployed and run in real time on devices with limited computing power, such as televisions.

[0126] In one embodiment, the controller performs a lookup mapping operation on the original RGB image based on the first RGB three-dimensional lookup table and the residual RGB three-dimensional lookup table to generate the target RGB image, and is configured as follows:

[0127] The first RGB three-dimensional lookup table and the residual RGB three-dimensional lookup table are superimposed element by element, and the enhancement features from the ICtCp color space and the RGB color space are combined to obtain the final RGB three-dimensional lookup table.

[0128] Using the final RGB 3D lookup table, a 3D lookup table mapping operation is performed on the original RGB image to generate the target RGB image.

[0129] Specifically, the first RGB 3D lookup table originates from the ICTCp path and carries the basic effect of luminance-color separation enhancement; while the residual RGB 3D lookup table originates from the RGB path and learns the fine-grained color correction amounts for the basic effect. Adding the two element-wise (i.e., adding corresponding values) essentially performs deep fusion, generating a final RGB 3D lookup table with more complete information and more accurate mapping, combining the accuracy advantages of luminance-color separation with the detail compensation capabilities of residual learning.

[0130] Then, using the final RGB 3D lookup table, a 3D lookup table mapping operation is performed on the original RGB image. Here, the original RGB image refers to the full-resolution 4K SDR source image without downsampling. Specifically, the (R, G, B) values ​​of each pixel in the image are extracted as 3D coordinates, located in the 3D grid formed by the final RGB 3D lookup table, and the enhanced HDR output value of that pixel is obtained in real time through trilinear interpolation.

[0131] Ultimately, by traversing all pixels and aggregating the results, the target HDR image can be generated. All the complex calculations (neural network inference) are condensed into the early lookup table generation stage, while the actual conversion of massive pixels is transformed into an extremely efficient lookup table and interpolation operation, thus balancing image quality and the real-time processing capability of the TV chip.

[0132] In this embodiment, by element-wise superimposing and combining the two lookup tables, the precise luminance-color separation relationship preserved by the ICp path and the fine-grained color residual correction learned by the RGB path can be effectively fused, thereby generating a final RGB 3D lookup table with more complete information and more accurate mapping. Furthermore, by using this fused lookup table to perform mapping operations on the original image, the expansion of the dynamic range of brightness, the enhancement of color saturation, and the color correction of local details can be completed simultaneously in a single efficient lookup calculation, ultimately presenting a high-quality conversion effect in the output image that is closer to the native HDR content.

[0133] In one embodiment, the controller performs a conversion of the downsampled RGB image from the RGB color space to the ICtCp color space to obtain an ICtCp image, which is configured as follows:

[0134] Convert downsampled RGB images from a gamma-coded non-linear RGB color space to a linear RGB color space;

[0135] Using the first color conversion matrix, the downsampled RGB image in the linear RGB color space is converted to the LMS color space to obtain the LMS color space image;

[0136] Perform a nonlinear transfer function transformation on the LMS color space image. The nonlinear transfer function includes at least a perceptual quantization curve or a mixed log-gamma curve.

[0137] Using the second color transformation matrix, the LMS color space image after nonlinear transfer function transformation is converted to the ICtCp color space to obtain the ICtCp image.

[0138] Specifically, converting an image from gamma-coded non-linear RGB to linear RGB is to eliminate artificial non-linear distortions and restore pixel values ​​to a linear relationship proportional to the brightness of the real scene. This is a prerequisite for all subsequent color calculations to be accurate; otherwise, all enhancement algorithms will be performed on distorted data.

[0139] Then, the linear RGB color space is converted to the LMS color space using the first color conversion matrix. The LMS color space simulates the response characteristics of the three cone cells in the human retina, and it is closer to the physiological basis of human vision. Next, performing a nonlinear transfer function transformation on the LMS image (using PQ or HLG curves) is the core of processing high dynamic range information. The dynamic range (i.e., the ratio of the brightest to the darkest) of linear light signals is very large, making direct processing inefficient. PQ (Perceptual Quantization) or HLG (Hybrid Log-Gamma) are encoding curves that can compress high dynamic range light information more efficiently, preserving more brightness and darkness details while reducing the burden of data storage and transmission.

[0140] Finally, the processed data is converted to the ICtCp color space using a second color transformation matrix. ICtCp is a new color space designed for HDR and wide color gamut video. Its biggest advantage is that the luminance and chrominance components are perceptibly uniformly separated. This means that operating on the luminance (I component) in this space will not cause changes in the chrominance (CtCp component), which is sensitive to human vision. This fundamentally avoids the luminance-chrominance crosstalk problem common in traditional spaces such as YCbCr, providing an ideal working environment for subsequent neural networks to independently and accurately enhance luminance and color.

[0141] This involves converting RGB values ​​to the CIE XYZ color space using a linear matrix. Taking the sRGB color gamut as an example, the conversion matrix is ​​as follows:

[0142] ;

[0143] Transform the XYZ space into the LMS space using the following matrix:

[0144] ;

[0145] Logarithmic compression is applied to the LMS space to simulate the nonlinear LMS space of human eye response to brightness:

[0146] ;

[0147] The nonlinear LMS space is transformed into the ICtCp space using the following matrix:

[0148] ;

[0149] Component range: I∈[0,1] (luminance component, covering the full luminance range of BT.2020); Ct, Cp∈[-0.5,+0.5] (chrominance component, orthogonal direction).

[0150] In this embodiment, by converting the image to the linear RGB space, the influence of gamma correction on the light signal can be eliminated, ensuring that color calculations are based on true physical brightness. By converting to the LMS space and performing PQ / HLG nonlinear transformation, efficient data compression can be achieved while preserving high dynamic range information. Finally, by converting to the ICtCp space, a perceptibly uniform luminance-color separation representation can be obtained, effectively avoiding the luminance-color crosstalk problem of traditional color spaces, and significantly improving the accuracy and quality of subsequent independent luminance and color enhancement.

[0151] In one embodiment, the controller is configured to perform the following actions before inputting the ICtCp image into a preset first convolutional neural network:

[0152] Calculate the pixel statistics for each channel in the ICtCp image, including the pixel mean and pixel variance.

[0153] Based on pixel statistics, the ICtCp image is subjected to zero-centering and variance normalization.

[0154] Specifically, calculating the pixel statistics (mean and variance) of each channel in the ICtCp image is crucial for obtaining the distribution characteristics of the current input data. The mean represents the average brightness level of the pixel values ​​in that channel, while the variance measures the dispersion of pixel values ​​around the mean. By calculating these two statistics, the overall location and scale of the input data are quantified.

[0155] Zero-centering refers to subtracting the mean of each channel from the pixel value in each channel of an ICtCp image. This shifts the entire data distribution to a zero-centered position, resulting in a new mean of zero. Variance normalization, following zero-centering, involves dividing each pixel value in each channel by the standard deviation of that channel (the square root of the variance). This step scales the data, making its variance one.

[0156] After these two steps, the data distribution of each channel in the ICTCp image will be adjusted to a standard normal distribution with a mean of zero and a variance of one.

[0157] In this embodiment, by calculating the pixel mean and variance of each channel of the image, the distribution characteristics of the input data can be accurately quantified. Furthermore, by performing zero-centering and variance normalization based on these statistical values, the distribution of each channel of the input data can be uniformly adjusted to a standard normal distribution with a mean of zero and a variance of one, effectively eliminating the internal covariate shift problem caused by differences in brightness or color distribution of the input data.

[0158] In one embodiment, the first convolutional neural network includes parallel and structurally independent luminance component processing subnetworks and chrominance component processing subnetworks; the controller executes inputting the ICtCp image into the preset first convolutional neural network and outputting a one-dimensional lookup table for the luminance component I and a two-dimensional lookup table for the chrominance component CtCp, configured as follows:

[0159] The ICtCp image is input into the luminance component processing sub-network. Multiple consecutive convolutional and pooling layers in the luminance component processing sub-network are used to extract hierarchical features. The extracted high-dimensional feature vector is mapped into a one-dimensional lookup table representing the luminance mapping relationship through a fully connected layer at the end. The one-dimensional lookup table is a vector containing a first preset number of lookup points.

[0160] The ICtCp image is input into the chroma component processing sub-network. Feature extraction and size adjustment are performed using multiple consecutive convolutional layers, pooling layers and at least one padding layer in the chroma component processing sub-network. The feature map is mapped into a two-dimensional lookup table representing the chroma mapping relationship through the convolutional layer set at the end. The two-dimensional lookup table is a two-dimensional array with a structure of a second preset number of points multiplied by a third preset number of points.

[0161] Specifically, firstly, the luminance component processing subnetwork is specifically responsible for learning the luminance mapping relationship. Its input is an ICtCp image, and its workflow is a standard feature extraction and regression process: multi-level abstract features are extracted from the luminance component I through multiple successive convolutional and pooling layers. These features capture key information from local contrast to global luminance distribution. After feature extraction, the network ends with a fully connected layer that compresses and maps these high-dimensional feature vectors into a fixed-size one-dimensional vector, i.e., a one-dimensional lookup table. This table (e.g., a 1×33 grid) defines a complete, continuous, non-linear transformation curve from the input SDR luminance to the output HDR luminance. The role of the fully connected layer is to summarize the learned complex features into a global, precise mapping rule.

[0162] Meanwhile, a chroma component processing subnetwork is dedicated to learning the chroma mapping in parallel. Its input is the same ICtCp image, but it focuses on processing the Ct and Cp chroma components. A key detail in its structure is the inclusion of at least one padding layer. The purpose of this padding layer is to finely control the size of the feature map, ensuring that after a series of convolutions and pooling operations, the feature map can be resized to the desired output size. Finally, the network terminates with a convolutional layer (typically a 1×1 convolution) that directly maps the final two-dimensional feature map into a two-dimensional array, i.e., a two-dimensional lookup table. This table (e.g., a 17×17 grid) defines the transformation relationship from input SDR chroma coordinates to output HDR chroma coordinates within the Ct-Cp chroma plane. Using convolutional layers instead of fully connected layers better preserves the relative positions and contextual relationships of chroma information in two-dimensional space.

[0163] The neural network structure layers of the luminance component processing subnetwork include:

[0164]

[0165] The neural network structure layers of the chroma component processing subnetwork include:

[0166]

[0167] In this embodiment, by utilizing convolutional and pooling layers in the luminance sub-network for hierarchical feature extraction and mapping it to a one-dimensional lookup table via fully connected layers, the complex luminance mapping relationship from standard dynamic range to high dynamic range can be accurately learned and represented, ensuring that the details of image brightness and darkness are fully expanded and preserved. Simultaneously, by performing feature extraction and size adjustment through a structure including fill layers in the chroma sub-network and outputting a two-dimensional lookup table through convolutional layers, the spatial context information of color components can be effectively captured, accurately establishing a nonlinear transformation model for the chroma components. This parallel processing mechanism of luminance and color separation fundamentally avoids the mutual interference caused by the mixing of luminance and color in traditional RGB space.

[0168] In one embodiment, the controller executes a one-dimensional lookup table based on the luminance component I and a two-dimensional lookup table based on the chrominance component CtCp, performs an inverse conversion of the ICtCp color space, and generates a first RGB three-dimensional lookup table, which is configured as follows:

[0169] Trilinear interpolation is performed on the one-dimensional lookup table of luminance component I and the two-dimensional lookup table of chromaticity component CtCp to generate an ICtCp spatial lookup table with a three-dimensional coordinate structure.

[0170] The ICtCp color space lookup table is sequentially transformed to the LMS color space using the first inverse transformation matrix, then restored to the linear RGB color space through nonlinear inverse transformation processing, and finally a first RGB three-dimensional lookup table is generated using the second inverse transformation matrix.

[0171] Specifically, firstly, trilinear interpolation is performed on the one-dimensional luminance lookup table and the two-dimensional chrominance lookup table. The neural network directly outputs a one-dimensional luminance table (such as a 1×33 vector) and a two-dimensional chrominance table (such as a 17×17 matrix), which cannot be directly used to process three-dimensional RGB pixels. Trilinear interpolation is a mathematical method that uses these two low-dimensional tables to intelligently calculate the output value corresponding to each coordinate point (I, Ct, Cp) in the complete three-dimensional ICtCp color space. This process is equivalent to filling in the flesh and blood based on the known skeleton data, thereby reconstructing a complete three-dimensional mapping relationship, such as a 33×33×33 ICtCp space lookup table. This step reintegrates the separated luminance and chrominance information into a unified, three-dimensional color transformation model. Then, an inverse color space conversion is performed to transfer the enhancement effect from the ICtCp space to the RGB space.

[0172] The inverse process is the precise reverse operation of the previous forward conversion: First, the three-dimensional ICtCp lookup table is converted back to the LMS color space using the first inverse transformation matrix; then, a nonlinear inverse transformation (i.e., the inverse function of PQ or HLG) is performed to decompress the efficiently encoded signal and restore it to a linear luminance representation; finally, the second inverse transformation matrix is ​​applied to convert the data from the LMS space back to the linear RGB color space. After this series of precise inverse transformations, the first RGB three-dimensional lookup table is finally generated.

[0173] First, the ICtCp space is transformed back into the nonlinear LMS space using an inverse matrix. The inverse matrix used is:

[0174] ;

[0175] The nonlinear LMS signal is subjected to inverse nonlinear processing to recover the linear LMS value. The corresponding inverse function is selected based on the nonlinear function used in the original conversion.

[0176] If logarithmic compression is used:

[0177] ;

[0178] If using PQ nonlinearity (BT.2100):

[0179] .

[0180] Transform the linear LMS back to the XYZ space using the inverse matrix:

[0181] ;

[0182] Verify the correctness of the matrix; this matrix should be the inverse of the LMS matrix in the original RGB to XYZ conversion.

[0183] To convert XYZ values ​​back to RGB space, you need to select the corresponding inverse matrix based on the target color gamut (such as sRGB, BT.2020). Taking sRGB as an example:

[0184] ;

[0185] The range is left unused to ensure that the calculated RGB linear values ​​are within the range of [0,1]. If they exceed this range, they need to be clipped.

[0186] If the target RGB is non-linearly encoded (such as gamma correction for sGB), a forward gamma function (EOTF) must be applied:

[0187] The same applies to G and B.

[0188] Logarithmic compression is applied to the LMS space to simulate the nonlinear LMS space of human eye response to brightness:

[0189] ;

[0190] The nonlinear LMS space is transformed into the ICtCp space using the following matrix:

[0191] .

[0192] In this embodiment, by performing trilinear interpolation on the one-dimensional luminance lookup table and the two-dimensional chrominance lookup table, a complete three-dimensional ICtCp space lookup table can be quickly reconstructed, fully restoring the three-dimensional color mapping relationship while maintaining data compactness. By sequentially performing inverse matrix transformation and nonlinear inverse processing, the enhanced effect of luminance-color separation can be accurately maintained during the inverse color space transformation, ensuring that color distortion or loss of detail is avoided when converting to RGB space.

[0193] In one embodiment, the controller executes the input of a downsampled RGB image to a preset second convolutional neural network to generate a residual RGB three-dimensional lookup table based on the RGB color space, configured as follows:

[0194] The downsampled RGB image is input into the intermediate layer of the first convolutional neural network with fixed network parameters and the input layer of the second convolutional neural network;

[0195] By utilizing the intermediate layers of the first convolutional neural network, at least one intermediate feature map is extracted and used as an auxiliary feature to represent enhanced prior knowledge in the ICtCp space.

[0196] Deep features of the RGB color space are extracted from downsampled RGB images through the front-end layer of the second convolutional neural network.

[0197] Auxiliary features and deep features are fused across color spaces to obtain a fused feature map;

[0198] The fused feature map is input into the subsequent residual block group of the second convolutional neural network for forward propagation calculation. Through the output layer of the second convolutional neural network, a residual RGB three-dimensional lookup table is generated for fine correction of the first RGB three-dimensional lookup table.

[0199] Specifically, first, the downsampled RGB image is fed into a second convolutional neural network (a residual network, such as ResNet18), and simultaneously fed into a first convolutional neural network whose parameters have been fixed. This fixing means that the weights of the first network are no longer updated; its knowledge has been frozen and it is treated as a stable feature extractor.

[0200] Next, crucial feature fusion is performed. Intermediate layer feature maps are extracted from the fixed first network. These feature maps carry prior knowledge from the luminance-color separation enhancement process in the ICtCp space, such as which areas need focused luminance enhancement and which color relationships need to be preserved. Simultaneously, the second convolutional neural network extracts deep features from the RGB space from its input image. Fusing these two sets of features from different color spaces and processing stages essentially compares and combines the experience gained from the ICtCp path with the observations from the RGB path, allowing the second convolutional neural network to more accurately determine where corrections are needed.

[0201] Finally, residual learning is performed based on the fused features. The fused feature map is fed into the subsequent residual blocks of the second convolutional neural network for computation. The structural characteristics of the residual network make it very good at learning the tiny residual (i.e., difference) between a target value and an initial value. Here, its task is not to output a complete color conversion table, but to learn a correction factor, that is, to accurately calculate the subtle color mapping relationship missing between the first RGB 3D lookup table obtained in the first step and the ideal effect.

[0202] In the residual network structure design, the input layer receives image data in RGB space, and its size can be set according to actual needs, such as 256×256×3, and undergoes data preprocessing such as normalization and standardization. The backbone network adopts the ResNet18 architecture, which includes standard convolutional layers, residual blocks, pooling layers, and fully connected layers, and can effectively extract multi-level features in RGB space to capture complex color mapping relationships. The output layer finally generates a residual RGB three-dimensional lookup table with a size of N×N×N×3, such as 33×33×33×3, to describe the color transformation residuals in RGB space. The output typically uses activation functions such as ReLU, tanh, or linear.

[0203] In the ICtCp branch parameter processing, the trained ICtCp branch network parameters are frozen during training and do not participate in the parameter updates of the main network. The input to this branch is the same as that of the main network, which is an RGB image, and its output is a three-dimensional lookup table generated in the ICtCp space. To achieve cross-path information complementarity, the output of the ICtCp branch network is used as an auxiliary feature and fused with the intermediate features extracted by the ResNet18 backbone network. The fusion method is usually weighted fusion.

[0204] In the generation and application stage of the residual RGB 3D lookup table, the residual lookup table output by the residual network is first added element-wise to the RGB 3D lookup table trained by the ICTCp path, resulting in a final RGB 3D lookup table with fused dual-path enhancement effects. This superposition operation effectively supplements and optimizes the color mapping capability of the original lookup table. Subsequently, the parameters of the residual network are gradually optimized through the backpropagation algorithm, making the superimposed 3D lookup table gradually approach the target color mapping relationship, ultimately obtaining the optimal parameters. The generated final RGB 3D lookup table will be applied to subsequent color mapping tasks, such as image color correction and display device color management, achieving higher color accuracy and visual fidelity through an efficient lookup table mapping mechanism.

[0205] In this embodiment, by fusing enhanced prior features extracted from the ICtCp space with deep features from the RGB space, a complementary feature representation that takes into account the characteristics of different color spaces can be constructed. By generating a residual lookup table based on these features using a residual network, subtle differences between the initial result and the ideal state can be accurately captured.

[0206] In one embodiment, the controller performs a 3D lookup table mapping operation on the original RGB image using the final RGB 3D lookup table to generate the target RGB image, and is configured as follows:

[0207] For each pixel in the original RGB image, extract the values ​​of the R, G, and B channels and construct three-dimensional input coordinates;

[0208] The three-dimensional input coordinates are mapped to a three-dimensional discrete grid composed of the final RGB three-dimensional lookup table. By performing trilinear interpolation calculation, the output RGB values ​​corresponding to the three-dimensional input coordinates and after dynamic range and color gamut enhancement are obtained.

[0209] Iterate through all pixels in the original RGB image, collect all calculated output RGB values, and construct the target RGB image.

[0210] In this embodiment, by mapping the three-dimensional RGB coordinates of each pixel to the enhanced color space, a global nonlinear transformation of the relationship between color and brightness can be achieved. By performing trilinear interpolation calculations in a three-dimensional discrete grid, the dynamic range expansion and color gamut mapping of each pixel can be accurately completed while maintaining a continuous and smooth color transition.

[0211] The above mainly describes the display device. In an exemplary embodiment, an image brightness and color enhancement method is also provided, applied to the aforementioned display device. (Reference) Figure 7 The method includes:

[0212] Step S702: Convert the downsampled RGB image from the RGB color space to the ICtCp color space to obtain the ICtCp image;

[0213] Step S704: Input the ICtCp image into a preset first convolutional neural network and output a one-dimensional lookup table for the luminance component I and a two-dimensional lookup table for the chrominance component CtCp.

[0214] Step S706: Based on the one-dimensional lookup table of luminance component I and the two-dimensional lookup table of chrominance component CtCp, perform inverse conversion of ICtCp color space to generate the first RGB three-dimensional lookup table.

[0215] Step S708: Fix the network parameters of the first convolutional neural network, and input the downsampled RGB image into the preset second convolutional neural network to generate a residual RGB three-dimensional lookup table based on the RGB color space;

[0216] Step S710: Based on the first RGB three-dimensional lookup table and the residual RGB three-dimensional lookup table, perform a lookup mapping operation on the original RGB image to generate the target RGB image.

[0217] In one embodiment, a lookup mapping operation is performed on the original RGB image based on a first RGB three-dimensional lookup table and a residual RGB three-dimensional lookup table to generate a target RGB image, including:

[0218] The first RGB three-dimensional lookup table and the residual RGB three-dimensional lookup table are superimposed element by element, and the enhancement features from the ICtCp color space and the RGB color space are combined to obtain the final RGB three-dimensional lookup table.

[0219] Using the final RGB 3D lookup table, a 3D lookup table mapping operation is performed on the original RGB image to generate the target RGB image.

[0220] In one embodiment, converting a downsampled RGB image from the RGB color space to the ICtCp color space to obtain an ICtCp image includes:

[0221] Convert downsampled RGB images from a gamma-coded non-linear RGB color space to a linear RGB color space;

[0222] Using the first color conversion matrix, the downsampled RGB image in the linear RGB color space is converted to the LMS color space to obtain the LMS color space image;

[0223] Perform a nonlinear transfer function transformation on the LMS color space image. The nonlinear transfer function includes at least a perceptual quantization curve or a mixed log-gamma curve.

[0224] Using the second color transformation matrix, the LMS color space image after nonlinear transfer function transformation is converted to the ICtCp color space to obtain the ICtCp image.

[0225] In one embodiment, before inputting the ICtCp image into a preset first convolutional neural network, the following steps are included:

[0226] Calculate the pixel statistics for each channel in the ICtCp image, including the pixel mean and pixel variance.

[0227] Based on pixel statistics, the ICtCp image is subjected to zero-centering and variance normalization.

[0228] In one embodiment, the first convolutional neural network includes parallel and structurally independent luminance component processing subnetworks and chrominance component processing subnetworks; the ICtCp image is input into the preset first convolutional neural network, and the output is a one-dimensional lookup table for the luminance component I and a two-dimensional lookup table for the chrominance component CtCp, including:

[0229] The ICtCp image is input into the luminance component processing sub-network. Multiple consecutive convolutional and pooling layers in the luminance component processing sub-network are used to extract hierarchical features. The extracted high-dimensional feature vector is mapped into a one-dimensional lookup table representing the luminance mapping relationship through a fully connected layer at the end. The one-dimensional lookup table is a vector containing a first preset number of lookup points.

[0230] The ICtCp image is input into the chroma component processing sub-network. Feature extraction and size adjustment are performed using multiple consecutive convolutional layers, pooling layers and at least one padding layer in the chroma component processing sub-network. The feature map is mapped into a two-dimensional lookup table representing the chroma mapping relationship through the convolutional layer set at the end. The two-dimensional lookup table is a two-dimensional array with a structure of a second preset number of points multiplied by a third preset number of points.

[0231] In one embodiment, based on a one-dimensional lookup table for the luminance component I and a two-dimensional lookup table for the chrominance component CtCp, an inverse transformation of the ICtCp color space is performed to generate a first RGB three-dimensional lookup table, including:

[0232] Trilinear interpolation is performed on the one-dimensional lookup table of luminance component I and the two-dimensional lookup table of chromaticity component CtCp to generate an ICtCp spatial lookup table with a three-dimensional coordinate structure.

[0233] The ICtCp color space lookup table is sequentially transformed to the LMS color space using the first inverse transformation matrix, then restored to the linear RGB color space through nonlinear inverse transformation processing, and finally a first RGB three-dimensional lookup table is generated using the second inverse transformation matrix.

[0234] In one embodiment, a downsampled RGB image is input into a preset second convolutional neural network to generate a residual RGB three-dimensional lookup table based on the RGB color space, including:

[0235] The downsampled RGB image is input into the intermediate layer of the first convolutional neural network with fixed network parameters and the input layer of the second convolutional neural network;

[0236] By utilizing the intermediate layers of the first convolutional neural network, at least one intermediate feature map is extracted and used as an auxiliary feature to represent enhanced prior knowledge in the ICtCp space.

[0237] Deep features of the RGB color space are extracted from downsampled RGB images through the front-end layer of the second convolutional neural network.

[0238] Auxiliary features and deep features are fused across color spaces to obtain a fused feature map;

[0239] The fused feature map is input into the subsequent residual block group of the second convolutional neural network for forward propagation calculation. Through the output layer of the second convolutional neural network, a residual RGB three-dimensional lookup table is generated for fine correction of the first RGB three-dimensional lookup table.

[0240] In one embodiment, a three-dimensional lookup table mapping operation is performed on the original RGB image using the final RGB three-dimensional lookup table to generate the target RGB image, including:

[0241] For each pixel in the original RGB image, extract the values ​​of the R, G, and B channels and construct three-dimensional input coordinates;

[0242] The three-dimensional input coordinates are mapped to a three-dimensional discrete grid composed of the final RGB three-dimensional lookup table. By performing trilinear interpolation calculation, the output RGB values ​​corresponding to the three-dimensional input coordinates and after dynamic range and color gamut enhancement are obtained.

[0243] Iterate through all pixels in the original RGB image, collect all calculated output RGB values, and construct the target RGB image.

[0244] In some embodiments, a schematic diagram illustrating the interaction flow between the controller, display, and server, in conjunction with the foregoing content, may be referenced. Figure 8 .

[0245] The controller acquires an RGB image from the display and downsamples it, converting the downsampled RGB image from the RGB color space to the ICtCp color space to obtain an ICtCp image. The ICtCp image is then input into a preset first convolutional neural network, which outputs a one-dimensional lookup table for the luminance component I and a two-dimensional lookup table for the chrominance component CtCp. Based on these tables, an inverse ICtCp color space conversion is performed to generate a first RGB three-dimensional lookup table. The network parameters of the first convolutional neural network are fixed, and the downsampled RGB image is input into a preset second convolutional neural network to generate a residual RGB three-dimensional lookup table based on the RGB color space. According to the first and residual RGB three-dimensional lookup tables, a lookup mapping operation is performed on the original RGB image to generate the target RGB image, which is then displayed on the display.

[0246] Based on the same inventive concept, this application also provides an image brightness enhancement apparatus for implementing the image brightness enhancement method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more image brightness enhancement apparatus embodiments provided below can be found in the limitations of the image brightness enhancement method described above, and will not be repeated here.

[0247] In one exemplary embodiment, such as Figure 9 As shown, an image brightness enhancement device is provided, comprising:

[0248] The color space conversion module 902 is used to convert the downsampled RGB image from the RGB color space to the ICtCp color space to obtain the ICtCp image.

[0249] The first lookup table generation module 904 is used to input the ICtCp image into a preset first convolutional neural network and output a one-dimensional lookup table for the luminance component I and a two-dimensional lookup table for the chrominance component CtCp.

[0250] The color space inverse conversion module 906 is used to perform ICtCp color space inverse conversion based on a one-dimensional lookup table of luminance component I and a two-dimensional lookup table of chrominance component CtCp, and generate a first RGB three-dimensional lookup table.

[0251] The second lookup table generation module 908 is used to fix the network parameters of the first convolutional neural network and input the downsampled RGB image into the preset second convolutional neural network to generate a residual RGB three-dimensional lookup table based on the RGB color space.

[0252] The image enhancement module 910 is used to perform a lookup mapping operation on the original RGB image based on the first RGB three-dimensional lookup table and the residual RGB three-dimensional lookup table to generate the target RGB image.

[0253] Each module in the aforementioned image brightness enhancement device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0254] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0255] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A display device, characterized by comprising: The method comprises the following steps: a display configured to display images and / or user interfaces; a controller configured to: convert a down-sampled RGB image from an RGB color space to an ICtCp color space to obtain an ICtCp image; input the ICtCp image into a preset first convolutional neural network to output a one-dimensional lookup table of a luminance component I and a two-dimensional lookup table of a chroma component CtCp; perform ICtCp color space inverse conversion based on the one-dimensional lookup table of the luminance component I and the two-dimensional lookup table of the chroma component CtCp to generate a first RGB three-dimensional lookup table; fix network parameters of the first convolutional neural network, input the down-sampled RGB image into a preset second convolutional neural network, and generate a residual RGB three-dimensional lookup table based on the RGB color space; perform lookup mapping operation on an original RGB image according to the first RGB three-dimensional lookup table and the residual RGB three-dimensional lookup table to generate a target RGB image.

2. The display device of claim 1, wherein, The controller configured to perform lookup mapping operation on an original RGB image according to the first RGB three-dimensional lookup table and the residual RGB three-dimensional lookup table to generate a target RGB image is configured to: perform element-by-element superposition on the first RGB three-dimensional lookup table and the residual RGB three-dimensional lookup table to combine enhanced features from the ICtCp color space and the RGB color space to obtain a final RGB three-dimensional lookup table; perform three-dimensional lookup table mapping operation on the original RGB image by using the final RGB three-dimensional lookup table to generate the target RGB image.

3. The display device of claim 1, wherein, The controller configured to perform conversion of a down-sampled RGB image from an RGB color space to an ICtCp color space to obtain an ICtCp image is configured to: convert the down-sampled RGB image from a gamma-encoded nonlinear RGB color space to a linear RGB color space; convert the down-sampled RGB image in the linear RGB color space to an LMS color space by using a first color conversion matrix to obtain an LMS color space image; perform nonlinear transfer function conversion on the LMS color space image, wherein the nonlinear transfer function at least includes a perceptual quantization curve or a hybrid logarithmic gamma curve; convert the LMS color space image after the nonlinear transfer function conversion to the ICtCp color space by using a second color conversion matrix to obtain the ICtCp image.

4. The display device of claim 1, wherein, The controller configured to perform the following before inputting the ICtCp image into a preset first convolutional neural network is configured to: calculate pixel statistical values of each channel in the ICtCp image, wherein the pixel statistical values include a pixel mean value and a pixel variance value; perform zero-centering and variance normalization processing on the ICtCp image based on the pixel statistical values.

5. The display device of claim 1, wherein, The first convolutional neural network comprises a luminance component processing subnetwork and a chroma component processing subnetwork which are arranged in parallel and independent in structure; and the controller configured to input the ICtCp image into a preset first convolutional neural network to output a one-dimensional lookup table of a luminance component I and a two-dimensional lookup table of a chroma component CtCp is configured to: The ICtCp image is input into the luminance component processing sub-network. Hierarchical feature extraction is performed using multiple consecutive convolutional and pooling layers in the luminance component processing sub-network. The extracted high-dimensional feature vector is mapped into a one-dimensional lookup table representing the luminance mapping relationship through a fully connected layer at the end. The one-dimensional lookup table is a vector containing a first preset number of lookup points. The ICtCp image is input into the chroma component processing sub-network. Feature extraction and size adjustment are performed using multiple consecutive convolutional layers, pooling layers and at least one fill layer in the chroma component processing sub-network. The feature map is mapped into a two-dimensional lookup table representing the chroma mapping relationship through the convolutional layer at the end. The two-dimensional lookup table is a two-dimensional array with a structure of a second preset number of points multiplied by a third preset number of points.

6. The display device of claim 5, wherein, The controller executes a one-dimensional lookup table based on the luminance component I and a two-dimensional lookup table based on the chrominance component CtCp, performs an inverse conversion of the ICtCp color space, and generates a first RGB three-dimensional lookup table, which is configured as follows: Trilinear interpolation is performed on the one-dimensional lookup table of the luminance component I and the two-dimensional lookup table of the chrominance component CtCp to generate an ICtCp spatial lookup table with a three-dimensional coordinate structure. The ICtCp space lookup table is sequentially transformed to the LMS color space through the first inverse transformation matrix, restored to the linear RGB color space through nonlinear inverse transformation processing, and the first RGB three-dimensional lookup table is generated by applying the second inverse transformation matrix.

7. The display device of claim 1, wherein, The controller executes the input of the downsampled RGB image into a preset second convolutional neural network, generating a residual RGB three-dimensional lookup table based on the RGB color space, and is configured as follows: The downsampled RGB image is input into the intermediate layer of the first convolutional neural network with fixed network parameters and the input layer of the second convolutional neural network; Using the intermediate layer of the first convolutional neural network, at least one intermediate feature map is extracted and used as an auxiliary feature to characterize the enhanced prior knowledge in the ICtCp space. Deep features of the RGB color space are extracted from the downsampled RGB image through the front-end layer of the second convolutional neural network. The auxiliary features and the deep features are fused across color spaces to obtain a fused feature map; The fused feature map is input into the subsequent residual block group of the second convolutional neural network for forward propagation calculation. Through the output layer of the second convolutional neural network, a residual RGB three-dimensional lookup table is generated for fine correction of the first RGB three-dimensional lookup table.

8. The display device of claim 2, wherein, The controller executes a three-dimensional lookup table mapping operation on the original RGB image using the final RGB three-dimensional lookup table to generate the target RGB image, and is configured as follows: For each pixel in the original RGB image, the values ​​of the R, G, and B channels are extracted and used to construct three-dimensional input coordinates; The three-dimensional input coordinates are mapped to a three-dimensional discrete grid composed of the final RGB three-dimensional lookup table. By performing trilinear interpolation calculation, the output RGB value corresponding to the three-dimensional input coordinates and after dynamic range and color gamut enhancement is obtained. The target RGB image is formed by iterating through all pixels in the original RGB image and collecting all calculated output RGB values.

9. An image light color enhancement method, characterized by, Applied to a display device as described in any one of claims 1-8, the method comprises: The downsampled RGB image is converted from the RGB color space to the ICtCp color space to obtain the ICtCp image; The ICtCp image is input into a preset first convolutional neural network, which outputs a one-dimensional lookup table for the luminance component I and a two-dimensional lookup table for the chrominance component CtCp. Based on the one-dimensional lookup table of the luminance component I and the two-dimensional lookup table of the chrominance component CtCp, perform inverse conversion of the ICtCp color space to generate the first RGB three-dimensional lookup table. The network parameters of the first convolutional neural network are fixed, and the downsampled RGB image is input into a preset second convolutional neural network to generate a residual RGB three-dimensional lookup table based on the RGB color space; Based on the first RGB three-dimensional lookup table and the residual RGB three-dimensional lookup table, a lookup mapping operation is performed on the original RGB image to generate the target RGB image.

10. The image lightness enhancement method of claim 9, wherein, The step of performing a lookup mapping operation on the original RGB image based on the first RGB three-dimensional lookup table and the residual RGB three-dimensional lookup table to generate the target RGB image includes: The first RGB three-dimensional lookup table and the residual RGB three-dimensional lookup table are superimposed element by element, and the enhanced features from the ICtCp color space and the RGB color space are combined to obtain the final RGB three-dimensional lookup table. Using the final RGB 3D lookup table, a 3D lookup table mapping operation is performed on the original RGB image to generate the target RGB image.

Images