High dynamic range image processing method and model training method, device, equipment, medium and product

Abstract

Embodiments of the present application provide a high dynamic range image processing method, a model training method, an apparatus, a device, a medium and a product. The method comprises: obtaining source image data to be processed, inputting the source image data into an image processing model, processing the source image data by the image processing model, obtaining an HDR image corresponding to the source image data and LED brightness data according to the processing of the image processing model, determining liquid crystal transmittance of a display device according to the HDR image, determining backlight brightness of the display device according to the LED brightness data, and determining HDR display content of the HDR image in the display device according to the liquid crystal transmittance and the backlight brightness. The method is used to realize effective conversion of an image with a lower dynamic range to a high dynamic range image, and improve the display quality of the high dynamic range image.

Description

Technical Field

[0001] This application relates to the field of digital image processing, and in particular to a high dynamic range image processing method, model training method, apparatus, device, medium and product. Background Technology

[0002] Currently, it is often necessary to convert SDR (Standard Dynamic Range) images and / or LDR (Low Dynamic Range) images to HDR (High Dynamic Range) images for display on HDR-enabled display devices.

[0003] Existing technologies employ HDR mapping algorithms to map SDR and / or LDR content into the HDR space, enabling SDR and / or LDR content to exhibit more natural and realistic colors and details with an expanded color gamut and higher brightness.

[0004] However, in the mapping process of existing technologies, improper selection of the mapping curve can easily lead to the neglect or over-brightening of highlight areas in the HDR space, reducing image quality and affecting the picture effect. Summary of the Invention

[0005] This application provides high dynamic range image processing methods, model training methods, apparatuses, devices, media, and products to effectively convert images with low dynamic range to HDR images and improve the display quality of HDR images.

[0006] In a first aspect, embodiments of this application provide a high dynamic range image processing method, applied to a server, comprising:

[0007] Obtain the source image data to be processed;

[0008] The source image data is input into the image processing model, and the image processing model processes the source image data to obtain the HDR image and LED brightness data corresponding to the source image data;

[0009] The liquid crystal transmittance of the display device is determined based on the HDR image, and the backlight brightness of the display device is determined based on the LED brightness data;

[0010] The HDR display content of the HDR image in the display device is obtained based on the liquid crystal transmittance and backlight brightness.

[0011] Optionally, the image processing model includes an encoder, a decoder, and an LED layer network, with the LED layer network located between the encoder and the decoder;

[0012] The source image data is processed using an image processing model to obtain the corresponding HDR image and LED brightness data, including:

[0013] The source image data is processed by an encoder to obtain the encoded feature data corresponding to the source image data;

[0014] The coded feature data is decoded by a decoder to obtain an HDR image of the source image data;

[0015] LED brightness data is obtained by processing the encoded feature data through the LED layer network.

[0016] Secondly, embodiments of this application provide a model training method, including:

[0017] Obtain the training dataset, which includes multiple training source image data and the corresponding training LED brightness data and training HDR display content for each training source image data;

[0018] The training model is iteratively trained using the training dataset to obtain the trained image processing model.

[0019] The trained image processing model is used to process the source image data to obtain the HDR image and LED brightness data corresponding to the source image data. Based on the HDR image, the liquid crystal transmittance of the display device is determined, based on the LED brightness data, the backlight brightness of the display device is determined, and based on the liquid crystal transmittance and backlight brightness, the HDR display content of the HDR image in the display device is determined.

[0020] Optionally, the training model is trained using a training dataset, including:

[0021] Input the training source image data into the training model;

[0022] The training model processes the training source images and outputs the predicted HDR image and predicted LED brightness data corresponding to the training source image data.

[0023] The training model is trained in its first iteration using predicted HDR images, predicted LED brightness data, and training LED brightness data and training HDR display content corresponding to the training source image data.

[0024] Optionally, the first iteration of training uses a first loss function, which is determined based on predicted LED brightness data, predicted HDR display content, training LED brightness data, and training HDR display content; wherein,

[0025] The training LED brightness data is determined in advance based on the corresponding training source image data; the training HDR display content is determined in advance based on the training HDR image corresponding to the training source image data and the training LED brightness data.

[0026] The predicted HDR display content is determined based on the predicted HDR image output by the trained model and the predicted LED brightness data.

[0027] Optionally, the predicted HDR display content is determined based on the predicted HDR image output by the training model and the predicted LED brightness data, including:

[0028] Backlight simulation processing is performed based on the predicted LED brightness data and the corresponding LED light distribution data to obtain the predicted backlight brightness corresponding to the training source image data.

[0029] The predicted HDR display content is determined based on the predicted HDR image and the predicted backlight brightness.

[0030] Among them, the LED light distribution data is the light distribution information corresponding to each value of the predicted LED brightness data.

[0031] Optionally, the trained model includes a generator and a discriminator;

[0032] The training source image data is input into the generator, which processes the training source image data and outputs the predicted HDR display content corresponding to the training source image data.

[0033] The predicted HDR display content and the training HDR display content corresponding to the training source image are input into the discriminator, and the generator is trained in the second iteration by the discriminator.

[0034] Optionally, the second iteration of training uses a second loss function, which is determined based on the mask image data corresponding to the training source image data, the training HDR display content, and the predicted HDR display content;

[0035] The mask image data is obtained by performing highlight extraction on the training source image data. The mask image data is used to indicate areas in the training source image data whose brightness exceeds a preset threshold.

[0036] Thirdly, embodiments of this application provide a high dynamic range image processing apparatus, applied to a server, comprising:

[0037] The first acquisition module is used to acquire the source image data to be processed;

[0038] The first processing module is used to input the source image data into the image processing model, and process the source image data through the image processing model to obtain the HDR image and LED brightness data corresponding to the source image data;

[0039] The first processing module is also used to determine the liquid crystal transmittance of the display device based on the HDR image and to determine the backlight brightness of the display device based on the LED brightness data.

[0040] The first processing module is also used to obtain the HDR display content of the HDR image in the display device based on the liquid crystal transmittance and backlight brightness.

[0041] Optionally, the image processing model includes an encoder, a decoder, and an LED layer network, with the LED layer network located between the encoder and the decoder;

[0042] The first processing module is also used to process the source image data through the encoder to obtain the encoded feature data corresponding to the source image data;

[0043] The coded feature data is decoded by a decoder to obtain an HDR image of the source image data;

[0044] LED brightness data is obtained by processing the encoded feature data through the LED layer network.

[0045] Fourthly, embodiments of this application provide a model training apparatus, comprising:

[0046] The second acquisition module is used to acquire the training dataset, which includes multiple training source image data and training LED brightness data and training HDR display content corresponding to each training source image data.

[0047] The second processing module is used to iteratively train the training model using the training dataset to obtain the trained image processing model.

[0048] The trained image processing model is used to process the source image data to obtain the HDR image and LED brightness data corresponding to the source image data. Based on the HDR image, the liquid crystal transmittance of the display device is determined, based on the LED brightness data, the backlight brightness of the display device is determined, and based on the liquid crystal transmittance and backlight brightness, the HDR display content of the HDR image in the display device is determined.

[0049] Optionally, the second processing module is also used to input the training source image data into the training model;

[0050] The training model processes the training source images and outputs the predicted HDR image and predicted LED brightness data corresponding to the training source image data.

[0051] The training model is trained in its first iteration using predicted HDR images, predicted LED brightness data, and training LED brightness data and training HDR display content corresponding to the training source image data.

[0052] Optionally, the second processing module is further configured to perform a first iteration of training using a first loss function, the first loss function being determined based on predicted LED brightness data, predicted HDR display content, trained LED brightness data, and trained HDR display content;

[0053] Among them, the training LED brightness data is determined in advance based on the corresponding training source image data; the training HDR display content is determined in advance based on the training HDR image and training LED brightness data corresponding to the training source image data.

[0054] The predicted HDR display content is determined based on the predicted HDR image output by the trained model and the predicted LED brightness data.

[0055] Optionally, the second processing module is further configured to perform backlight simulation processing based on the predicted LED brightness data and the corresponding LED light distribution data to obtain the predicted backlight brightness corresponding to the training source image data.

[0056] The predicted HDR display content is determined based on the predicted HDR image and the predicted backlight brightness.

[0057] Among them, the LED light distribution data is the light distribution information corresponding to each value of the predicted LED brightness data.

[0058] Optionally, the trained model includes a generator and a discriminator;

[0059] The second processing module is also used to input the training source image data into the generator, process the training source image data through the generator, and output the predicted HDR display content corresponding to the training source image data.

[0060] The predicted HDR display content and the training HDR display content corresponding to the training source image are input into the discriminator, and the generator is trained in the second iteration by the discriminator.

[0061] Optionally, the second processing module is further configured to perform a second iteration of training using a second loss function, wherein the second loss function is determined based on the mask image data corresponding to the training source image data, the training HDR display content, and the predicted HDR display content;

[0062] The mask image data is obtained by performing highlight extraction on the training source image data. The mask image data is used to indicate areas in the training source image data whose brightness exceeds a preset threshold.

[0063] Fifthly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0064] The memory stores instructions that the computer executes;

[0065] The processor executes computer execution instructions stored in memory, causing the processor to perform various possible implementations of the first and / or second aspects described above.

[0066] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement various possible implementations of the first and / or second aspects described above.

[0067] In a seventh aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements various possible implementations of the first and / or second aspects described above.

[0068] The high dynamic range (HDR) image processing method, model training method, apparatus, device, medium, and product provided in this application acquire source image data to be processed, input the source image data into an image processing model, process the source image data through the image processing model to obtain an HDR image and LED brightness data corresponding to the source image data, determine the liquid crystal transmittance of the display device based on the HDR image, determine the backlight brightness of the display device based on the LED brightness data, and determine the HDR display content of the HDR image on the display device based on the liquid crystal transmittance and backlight brightness. This application achieves an effective conversion of images with low dynamic range to high dynamic range images and improves the display quality of high dynamic range images. Attached Figure Description

[0069] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0070] Figure 1 This application provides an illustration of the application scenario.

[0071] Figure 2 Flowchart of the high dynamic range image processing method provided in this application Figure 1 ;

[0072] Figure 3 A schematic diagram of the structure of the image processing model provided in this application;

[0073] Figure 4 Flowchart of the high dynamic range image processing method provided in this application Figure 2 ;

[0074] Figure 5 Flowchart of the model training method provided in this application Figure 1 ;

[0075] Figure 6Flowchart of the model training method provided in this application Figure 2 ;

[0076] Figure 7 Flowchart of the model training method provided in this application Figure 3 ;

[0077] Figure 8 Flowchart of the model training method provided in this application Figure 4 ;

[0078] Figure 9 A schematic diagram of the high dynamic range image processing apparatus provided in this application;

[0079] Figure 10 A schematic diagram of the structure of the model training device provided in this application;

[0080] Figure 11 A schematic diagram of the structure of the electronic device provided in this application.

[0081] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0082] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0083] Because SDR and / or LDR images typically have lower peak brightness and contrast, they struggle to display higher brightness and deeper shadow details. Therefore, when watching movies, TV shows, and games in everyday life, SDR format images are prone to insufficient contrast, uneven brightness, and low color saturation, making it difficult to present a more vivid and realistic picture.

[0084] Currently, HDR images can enhance image quality through richer details in both bright and dark areas and a wider color gamut. Therefore, converting SDR and / or LDR images to HDR images can effectively improve image quality. Figure 1 The application scenario diagram provided in this application is as follows: Figure 1As shown, by performing image processing on SDR and / or LDR images, processed image data is obtained, and the processed image data is displayed on a liquid crystal display device, so as to realize the display of SDR and / or LDR images in HDR format on the display device.

[0085] Existing technologies for converting SDR images to HDR images employ HDR mapping algorithms to map SDR content into the HDR space, enabling the SDR content to display more natural and realistic colors and details within an expanded color gamut and higher brightness. However, due to the lower dynamic range of SDR and / or LDR images, they lack sufficient highlight details mapped into the HDR space. To accommodate the brightness range of HDR image display, existing mapping technologies require brightness compression or expansion of the SDR and / or LDR images. Mismatches in mapping curves can easily lead to the neglect or over-brightening of highlight details in the HDR space, thereby reducing image quality and affecting the overall picture effect.

[0086] The high dynamic range (HDR) image processing method provided in this application inputs the source image data to be processed into a trained image processing model. The encoder in the image processing model processes the source image data to obtain coded feature data. The decoder in the image processing model processes the coded feature data to obtain the corresponding HDR image. The LED layer network in the image processing model then processes the coded feature data to obtain the corresponding LED brightness data. Based on the LED brightness data, the corresponding backlight brightness is determined. The liquid crystal transmittance of the display device is determined based on the HDR image. HDR display content corresponding to the source image data to be processed is generated based on the backlight brightness and liquid crystal transmittance and displayed on the display device, effectively reducing detail loss and improving the image display effect. The image processing model is generated by training a training model. During training, highlight extraction processing is performed on the training source image data to obtain the corresponding mask image data. A loss function is determined based on the mask image data. The generator is iteratively trained using the discriminator of the training model to optimize the generator's model parameters, further enhancing the display quality of the HDR image and improving the accuracy of converting source image data to HDR images.

[0087] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0088] Figure 2Flowchart of the high dynamic range image processing method provided in this application Figure 1 ,like Figure 2 As shown, this high dynamic range (HDR) image processing method can be used in a server. The execution subject of this method can be a HDR image processing device, a server integrating a HDR image processing device, or a display device integrating a HDR image processing device. The following description uses a server integrating a HDR image processing device as the execution subject as an example. The method includes:

[0089] S201. Obtain the source image data to be processed.

[0090] More specifically, the source image data to be processed is obtained, which is the image data of any image in the linear domain.

[0091] Optionally, the LDR image and / or SDR image to be displayed can be linearly processed to obtain the corresponding source image data.

[0092] S202. The source image data is processed by an image processing model to obtain the HDR image and LED brightness data corresponding to the source image data.

[0093] More specifically, the image processing model includes an encoder, a decoder, and an LED layer network, with the LED layer network located between the encoder and the decoder. The encoder processes the source image data to obtain the coded feature data corresponding to the source image data. The decoder decodes the coded feature data to obtain the HDR image of the source image data. The LED layer network processes the coded feature data to obtain the LED brightness data.

[0094] In one possible embodiment, Figure 3 A schematic diagram of the structure of the image processing model provided in this application is shown below. Figure 3 As shown, the image processing model structure includes an encoding base network, a decoding base network, and an LED layer network. The encoding base network represents the encoder, the decoding base network represents the decoder, and the LED layer network includes convolutional layer 1, convolutional layer 2, and a GAP module. The aforementioned base network is the UNet network. For details on building the encoder (encoding base network) and decoder (decoding base network) based on the UNet network, please refer to relevant technologies; further details will not be elaborated here.

[0095] Optionally, the encoding base network is used to process the received source image data and output encoded feature data. The encoded feature data is then output to the decoding base network and the LED layer network. The decoding base network processes the encoded feature data to output an HDR image. The width and height of the matrix corresponding to the encoded feature data are then adjusted sequentially through convolutional layer 1 and convolutional layer 2 of the LED layer network to obtain adjusted encoded feature data. The adjusted encoded feature data is then subjected to GAP (Global Average Pooling) processing to obtain LED brightness data. This LED brightness data includes the average LED brightness of each channel in the encoded feature map. The matrix size of the adjusted encoded feature data matches that of the LED brightness data.

[0096] This embodiment adds an LED layer network to the decoder and encoder to effectively compress the spatial information of the encoded feature data into a matrix of LED brightness data based on the encoded feature data output by the encoder. At the same time, it retains the global information of each channel in the encoded feature data, improves the accuracy of the high dynamic range image converted from the source image data, reduces information loss, and further improves the quality and display effect of the high dynamic range image.

[0097] S203. Determine the liquid crystal transmittance of the display device based on the HDR image, and determine the backlight brightness of the display device based on the LED brightness data.

[0098] More specifically, the backlight chip is driven based on LED brightness data to obtain the required backlight brightness for the display panel. Backlight brightness refers to the intensity of light emitted by the backlight source of the display device (such as an LCD screen), which is the light source behind the display panel and consists of LEDs. Liquid crystal transmittance includes the transmittance values ​​for each pixel of the HDR image to be displayed. Liquid crystal configuration information includes, but is not limited to, resolution, brightness, contrast ratio, and color gamut.

[0099] S204. Obtain the HDR display content of the HDR image in the display device based on the liquid crystal transmittance and backlight brightness.

[0100] More specifically, the transmittance value of each pixel on the display device screen is determined based on the transmittance of the liquid crystal, and the transmittance of each pixel is adjusted according to the transmittance value of each pixel. In this way, the HDR image content displayed on the display device is adjusted by the backlight brightness and the transmittance of each pixel, thereby realizing the display of high-brightness details of the HDR image.

[0101] In existing technologies, backlight brightness is not only a significant factor contributing to increased energy consumption in display devices, but it also easily affects the screen's overall brightness when displaying bright images. This embodiment addresses this by dynamically adjusting the brightness of HDR images using both backlight brightness and liquid crystal transmittance. This effectively solves the problem of high energy consumption caused by using high backlight brightness in existing technologies, while simultaneously enhancing image contrast.

[0102] The high dynamic range image processing method provided in this application, when converting source image data into an HDR image for display, determines the corresponding matching LED brightness data based on the source image data and obtains the backlight brightness of the display device. It also determines the corresponding liquid crystal transmittance based on the HDR image corresponding to the source image data, thereby obtaining the transmittance of each pixel displayed in the display device. This achieves the restoration of high-brightness details in the HDR image, enhances the contrast of the image display, and reduces the display power consumption of the high dynamic range image.

[0103] In one possible embodiment, Figure 4 Flowchart of the HDR image display method provided in this application Figure 2 ,like Figure 4 As shown, the SDR image to be processed undergoes linear processing to obtain the corresponding source image data. This source image data is then input to an encoder, which generates coded feature data corresponding to the source image data and sends it to the decoder and LED layer network. The LED layer network processes the coded feature data to obtain LED brightness data. Based on the LED brightness data, the backlight brightness of the display device is determined, and the backlight chip of the display device is driven. The decoder decodes the coded feature data to obtain an HDR image. Based on the HDR image, the liquid crystal transmittance of the display device is determined. The liquid crystal screen of the display device is configured according to this transmittance to display the HDR content corresponding to the SDR image on the display device.

[0104] The high dynamic range image processing method provided in this application adjusts the backlight brightness of the display device to obtain a more accurate basic light source brightness, and displays HDR images by adjusting the transmittance of each pixel, thereby effectively converting images with low dynamic range to high dynamic range images and improving the display effect of high dynamic range images.

[0105] Figure 5 Flowchart of the model training method provided in this application Figure 1 ,like Figure 5 As shown in this embodiment, the model training method is described in detail, and the method includes:

[0106] S501. Obtain the training dataset.

[0107] More specifically, a training dataset is obtained, which includes multiple training source image data, as well as training LED brightness data and training HDR display content corresponding to each training source image data. The training HDR display content is determined in advance based on the training source image data and training LED brightness data.

[0108] S502. Use the training dataset to iteratively train the training model to obtain the trained image processing model.

[0109] More specifically, the training model is iteratively trained using a training dataset to obtain a trained image processing model. This trained model processes the source image data to obtain the corresponding HDR image and LED brightness data. Based on the HDR image, the liquid crystal transmittance of the display device is determined; based on the LED brightness data, the backlight brightness of the display device is determined; and based on the liquid crystal transmittance and backlight brightness, the HDR display content of the HDR image on the display device is determined. This application enhances the generalization ability of the image processing model and the accuracy of the output image processing results by iteratively training the training model, thereby improving the display effect of high dynamic range images.

[0110] Optionally, the training source image data is input into the training model; the training model processes the training source image and outputs the predicted HDR image and predicted LED brightness data corresponding to the training source image data; the training model is trained in the first iteration using the predicted HDR image, predicted LED brightness data, training LED brightness data and training HDR display content corresponding to the training source image data.

[0111] Optionally, the first iteration of training uses a first loss function, which is determined based on the predicted LED brightness data, the predicted HDR display content, the training LED brightness data, and the training HDR display content. The training LED brightness data is determined in advance based on the corresponding training source image data; the training HDR display content is determined in advance based on the training HDR image corresponding to the training source image data and the training LED brightness data; and the predicted HDR display content is determined based on the predicted HDR image and the predicted LED brightness data output by the training model.

[0112] In one possible embodiment, such as Figure 6As shown, training LED brightness data is obtained by partitioning the training source image data. Backlight simulation processing is then performed based on the training LED brightness data and the corresponding LED light distribution data to obtain the training backlight brightness corresponding to the training source image data. Finally, tone mapping processing is performed on the training source image data and the training backlight brightness to obtain the training HDR display content corresponding to the training source image data. The LED light distribution data refers to the light distribution information corresponding to each value in the training LED brightness data.

[0113] For example, training HDR display content I GT =TM(I in / BL)⊙BL, where I in Here, TM represents the matrix corresponding to the training source image data, BL represents the matrix corresponding to the training backlight brightness obtained from backlight simulation, and the training backlight brightness is determined based on the training LED brightness data and LED light distribution data. The aforementioned training LED brightness data is obtained by partitioning the training source image data. The symbol ⊙ is used to represent element-wise dot product processing (e.g., Hardman product operation). The detailed calculation process corresponding to the tone mapping processing, and the steps for partitioning the training source image data to obtain the corresponding LED brightness data, can be found in relevant technologies and will not be elaborated further here.

[0114] Optionally, backlight simulation processing is performed based on the predicted LED brightness data and the corresponding LED light distribution data to obtain the predicted backlight brightness corresponding to the training source image data; the predicted HDR display content is determined based on the predicted HDR image and the predicted backlight brightness; wherein, the LED light distribution data is the light distribution information corresponding to each value of the predicted LED brightness data.

[0115] In one possible embodiment, such as Figure 7 As shown, the training model includes an image processing model and a discriminator. Training source image data is input into the image processing model, which generates and outputs predicted HDR images and predicted LED brightness data. Backlight simulation processing is performed on the predicted LED brightness data and the corresponding LED light distribution data to obtain the predicted backlight brightness corresponding to the training source image data. The predicted HDR display content is obtained by performing a Haldman product operation on the predicted HDR image and the predicted backlight brightness. This predicted HDR display content and... Figure 6 The training HDR display content obtained in the example is input into the discriminator. The discriminator compares the predicted HDR display content with the training HDR display content to determine the authenticity of the predicted HDR display content. If it is determined that the predicted HDR display content is not real data, a first loss function is generated. The parameters of the image processing model are optimized by minimizing the first loss function using the gradient descent method.

[0116] For example, the first loss function L stage1 The formula is shown below:

[0117] L stage1 =w1*L2(LED,LED) GT )+w2*L1(I,I GT )+w3*L perceptual (I,I GT )

[0118] Among them, L perceptual For the perceived loss, L1 is the absolute error loss, L2 is the mean squared error loss, w1 is 0.4, which is the weighting coefficient used to characterize the mean squared error loss, w2 is 0.4, which is the weighting coefficient used to characterize the absolute error loss, and w3 is 0.2, which is the weighting coefficient used to characterize the perceived loss.

[0119] This embodiment performs backlight simulation processing on training LED brightness data and LED light distribution data obtained by partitioning training source image data to obtain training backlight brightness. Based on the training backlight brightness, training HDR display content is generated so that the image processing model can learn high-quality image features. The discriminator then identifies HDR display content with backlight brightness, improving the reliability and accuracy of converting low dynamic range images into high dynamic range images and displaying them on the display device.

[0120] Optionally, the training model includes a generator and a discriminator; the training source image data is input into the generator, the generator processes the training source image data, and outputs the predicted HDR display content corresponding to the training source image data; the predicted HDR display content and the training HDR display content corresponding to the training source image are input into the discriminator, and the discriminator performs a second iteration of training on the generator.

[0121] Optionally, the second iteration of training uses a second loss function, which is determined based on the mask image data corresponding to the training source image data, the training HDR display content, and the predicted HDR display content; wherein, the mask image data is obtained by performing highlight extraction processing on the training source image data, and the mask image data is used to indicate the areas in the training source image data whose brightness exceeds a preset threshold.

[0122] In one possible embodiment, such as Figure 8As shown, after the training source image data is input into the encoder, encoded feature data is generated and then input into the decoder and LED layer network respectively. The decoder generates a predicted HDR image, and the LED layer network generates predicted LED brightness data. Backlight simulation processing is performed based on the LED brightness data and LED light distribution data to obtain the predicted backlight brightness. The predicted backlight brightness and the predicted HDR image are then subjected to a Haldman product operation to obtain the predicted HDR display content. The predicted HDR display content and... Figure 6 The training HDR display content obtained in the embodiment is input to the discriminator. After the discriminator compares the predicted HDR display content with the training HDR display content, it generates a second loss function value corresponding to the training HDR display content based on the second loss function. The second loss function value is then used to update and iterate the encoder and decoder parameters in the generator. The parameters of the LED layer network are preset parameter values.

[0123] For example, the second loss function L stage2 The formula is shown below:

[0124] L stage2 =mask⊙[H(1,D(G(I)] in )))+(H(1,D(I GT ))+H(0,D(G(I in ))))]

[0125] Where G() is the output of the image processing model, D() is the judgment result of the discriminant network on the authenticity of the training HDR display content, H() is the cross-entropy loss function, and mask is the matrix corresponding to the mask image data. The formula for calculating the mask image data is as follows:

[0126]

[0127] Where i is the x-coordinate of a pixel in the mask image, j is the y-coordinate of a pixel in the mask image, and TH is the brightness threshold, which is determined based on the source image data to be processed.

[0128] In one possible embodiment, the value of TH is determined according to TH = (0.5, th), that is, the larger value between th and 0.5 is taken as the TH brightness threshold. Here, th is the brightness value corresponding to the pixel that is still ranked first after removing some pixels. The removed pixels are the pixels whose brightness values ​​are in the top 20% after arranging the pixels in the source image data in descending order of brightness value.

[0129] In one possible embodiment, the TH luminance threshold is determined to be a fixed value.

[0130] This embodiment uses masked image data to weight the second loss function and iteratively trains the generator based on the second loss function, thereby improving the accuracy of the image processing model in processing highlight details and further enhancing the reliability of the image processing model.

[0131] Alternatively, methods for generating training LED brightness data based on training source image data can also be found in relevant prior art, which will not be elaborated here in this embodiment.

[0132] Figure 9 This is a schematic diagram of the high dynamic range image processing apparatus provided in this application, applied to a server, such as... Figure 9 As shown, the high dynamic range image processing apparatus 90 provided in this embodiment includes:

[0133] The first acquisition module 901 is used to acquire the source image data to be processed;

[0134] The first processing module 902 is used to input the source image data into the image processing model, process the source image data through the image processing model, and obtain the HDR image and LED brightness data corresponding to the source image data.

[0135] The first processing module 902 is also used to determine the liquid crystal transmittance of the display device based on the HDR image, and to determine the backlight brightness of the display device based on the LED brightness data;

[0136] The first processing module 902 is also used to obtain the HDR display content of the HDR image in the display device based on the liquid crystal transmittance and backlight brightness.

[0137] Optionally, the image processing model includes an encoder, a decoder, and an LED layer network, with the LED layer network located between the encoder and the decoder;

[0138] The first processing module 902 is also used to process the source image data through the encoder to obtain the encoded feature data corresponding to the source image data;

[0139] The coded feature data is decoded by a decoder to obtain an HDR image of the source image data;

[0140] LED brightness data is obtained by processing the encoded feature data through the LED layer network.

[0141] The high dynamic range image processing apparatus provided in this embodiment can execute the high dynamic range image processing method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0142] Figure 10 A schematic diagram of the structure of the model training device provided in this application is shown below. Figure 10As shown, the model training device 100 provided in this embodiment includes:

[0143] The second acquisition module 1001 is used to acquire a training dataset, which includes multiple training source image data and training LED brightness data and training HDR display content corresponding to each training source image data.

[0144] The second processing module 1002 is used to iteratively train the training model using the training dataset to obtain the trained image processing model.

[0145] The trained image processing model is used to process the source image data to obtain the HDR image and LED brightness data corresponding to the source image data. Based on the HDR image, the liquid crystal transmittance of the display device is determined, based on the LED brightness data, the backlight brightness of the display device is determined, and based on the liquid crystal transmittance and backlight brightness, the HDR display content of the HDR image in the display device is determined.

[0146] Optionally, the second processing module 1002 is also used to input training source image data into the training model;

[0147] The training model processes the training source images and outputs the predicted HDR image and predicted LED brightness data corresponding to the training source image data.

[0148] The training model is trained in its first iteration using predicted HDR images, predicted LED brightness data, and training LED brightness data and training HDR display content corresponding to the training source image data.

[0149] Optionally, the second processing module 1002 is further configured to perform a first iteration of training using a first loss function, the first loss function being determined based on predicted LED brightness data, predicted HDR display content, training LED brightness data, and training HDR display content;

[0150] Among them, the training LED brightness data is determined in advance based on the corresponding training source image data; the training HDR display content is determined in advance based on the training HDR image and training LED brightness data corresponding to the training source image data.

[0151] The predicted HDR display content is determined based on the predicted HDR image output by the trained model and the predicted LED brightness data.

[0152] Optionally, the second processing module 1002 is further configured to perform backlight simulation processing based on the predicted LED brightness data and the corresponding LED light distribution data to obtain the predicted backlight brightness corresponding to the training source image data.

[0153] The predicted HDR display content is determined based on the predicted HDR image and the predicted backlight brightness.

[0154] Among them, the LED light distribution data is the light distribution information corresponding to each value of the predicted LED brightness data.

[0155] Optionally, the trained model includes a generator and a discriminator;

[0156] The second processing module 1002 is also used to input the training source image data into the generator, process the training source image data through the generator, and output the predicted HDR display content corresponding to the training source image data.

[0157] The predicted HDR display content and the training HDR display content corresponding to the training source image are input into the discriminator, and the generator is trained in the second iteration by the discriminator.

[0158] Optionally, the second processing module 1002 is further configured to perform a second iteration of training using a second loss function, wherein the second loss function is determined based on the mask image data corresponding to the training source image data, the training HDR display content, and the predicted HDR display content;

[0159] The mask image data is obtained by performing highlight extraction on the training source image data. The mask image data is used to indicate areas in the training source image data whose brightness exceeds a preset threshold.

[0160] The model training device provided in this embodiment can execute the model training method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0161] Figure 11 A schematic diagram of the structure of the electronic device provided in this application. Figure 11 As shown, the electronic device 110 provided in this embodiment includes at least one processor 1101 and a memory 1102. Optionally, the device 110 further includes a communication component 1103. The processor 1101, the memory 1102, and the communication component 1103 are connected via a bus 1104.

[0162] In a specific implementation, at least one processor 1101 executes computer execution instructions stored in memory 1102, causing at least one processor 1101 to perform the above-described method.

[0163] The specific implementation process of processor 1101 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0164] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0165] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0166] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0167] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0168] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0169] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0170] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0171] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0172] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0173] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0174] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0175] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0176] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A high dynamic range image processing method, characterized in that, Applied to servers, including: Obtain the source image data to be processed; The source image data is input into an image processing model, and the source image data is processed by the image processing model to obtain the HDR image and LED brightness data corresponding to the source image data; The liquid crystal transmittance of the display device is determined based on the HDR image, and the backlight brightness of the display device is determined based on the LED brightness data; The HDR display content of the HDR image in the display device is obtained based on the liquid crystal transmittance and the backlight brightness.

2. The method according to claim 1, characterized in that, The image processing model includes an encoder, a decoder, and an LED layer network, with the LED layer network located between the encoder and the decoder. The source image data is processed using the image processing model to obtain the corresponding HDR image and LED brightness data, including: The source image data is processed by the encoder to obtain the encoded feature data corresponding to the source image data; The decoder decodes the encoded feature data to obtain an HDR image of the source image data; The LED brightness data is obtained by processing the encoded feature data through the LED layer network.

3. A model training method, characterized in that, include: Obtain a training dataset, which includes multiple training source image data and training LED brightness data and training HDR display content corresponding to each training source image data; The training model is iteratively trained using the training dataset to obtain the trained image processing model; The trained image processing model is used to process the source image data to be processed, and obtain the HDR image and LED brightness data corresponding to the source image data. Based on the HDR image, the liquid crystal transmittance of the display device is determined, based on the LED brightness data, the backlight brightness of the display device is determined, and based on the liquid crystal transmittance and the backlight brightness, the HDR display content of the HDR image in the display device is determined.

4. The method according to claim 3, characterized in that, Training the training model using the aforementioned training dataset includes: Input the training source image data into the training model; The training model processes the training source image and outputs the predicted HDR image and predicted LED brightness data corresponding to the training source image data. The training model is trained in the first iteration using the predicted HDR image, the predicted LED brightness data, the training LED brightness data corresponding to the training source image data, and the training HDR display content.

5. The method according to claim 4, characterized in that, The first iteration of training uses a first loss function, which is determined based on predicted LED brightness data, predicted HDR display content, training LED brightness data, and training HDR display content; where, The training LED brightness data is determined in advance based on the corresponding training source image data; the training HDR display content is determined in advance based on the training HDR image corresponding to the training source image data and the training LED brightness data; The predicted HDR display content is determined based on the predicted HDR image output by the training model and the predicted LED brightness data.

6. The method according to claim 5, characterized in that, Determining the predicted HDR display content based on the predicted HDR image output by the training model and the predicted LED brightness data includes: Based on the predicted LED brightness data and the corresponding LED light distribution data, backlight simulation processing is performed to obtain the predicted backlight brightness corresponding to the training source image data. The predicted HDR display content is determined based on the predicted HDR image and the predicted backlight brightness. The LED light distribution data refers to the light distribution information corresponding to each value of the predicted LED brightness data.

7. The method according to claim 4, characterized in that, The training model includes a generator and a discriminator; The generator inputs the training source image data, processes the training source image data, and outputs the predicted HDR display content corresponding to the training source image data. The predicted HDR display content and the training HDR display content corresponding to the training source image are input into the discriminator, and the generator is trained in a second iteration by the discriminator.

8. The method according to claim 7, characterized in that, The second iteration of training uses the second loss function, which is determined based on the mask image data corresponding to the training source image data, the training HDR display content, and the predicted HDR display content. The mask image data is obtained by performing highlight extraction processing on the training source image data, and the mask image data is used to indicate the areas in the training source image data whose brightness exceeds a preset threshold.

9. A high dynamic range image processing apparatus, characterized in that, Applied to servers, including: The first acquisition module acquires the source image data to be processed; The first processing module is used to input the source image data into an image processing model, and process the source image data through the image processing model to obtain the HDR image and LED brightness data corresponding to the source image data; The first processing module is further configured to determine the liquid crystal transmittance of the display device based on the HDR image, and to determine the backlight brightness of the display device based on the LED brightness data; The first processing module is further configured to obtain the HDR display content of the HDR image in the display device based on the liquid crystal transmittance and the backlight brightness.

10. A model training device, characterized in that, include: The second acquisition module is used to acquire a training dataset, which includes multiple training source image data and training LED brightness data and training HDR display content corresponding to each training source image data. The second processing module is used to iteratively train the training model using the training dataset to obtain the trained image processing model. The trained image processing model is used to process the source image data to be processed, and obtain the HDR image and LED brightness data corresponding to the source image data. Based on the HDR image, the liquid crystal transmittance of the display device is determined, based on the LED brightness data, the backlight brightness of the display device is determined, and based on the liquid crystal transmittance and the backlight brightness, the HDR display content of the HDR image in the display device is determined.

11. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-8.

13. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-8.

Images