An image processing method, apparatus and storage medium

Abstract

This application provides an image processing method, apparatus, and storage medium. The method involves acquiring a standard dynamic range (SDR) image; inputting the SDR image into a color mapping model; and having the color mapping model perform color mapping and color gamut mapping on each pixel in the SDR image based on the color and color gamut mapping relationship between the SDR image and the high dynamic range (HDR) image, thereby obtaining the color and color gamut information of the HDR image. Based on the color and color gamut information of the HDR image, an HDR image is generated. By converting the SDR image to an HDR image, it is not necessary to collect the original data according to the acquisition requirements of HDR images, reducing the hardware configuration requirements of electronic devices and effectively saving costs. Furthermore, the method of converting SDR images to HDR images based on a color mapping model trained with a large amount of training data reduces reliance on manual processing, improving efficiency while saving labor costs.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus and storage medium. Background Technology

[0002] With the development of image processing technology, people have increasingly higher requirements for image display effects. Currently, methods can be used to generate High Dynamic Range (HDR) images to improve image display quality. HDR is an image processing technology that can improve image brightness and contrast; HDR images can provide a wider dynamic range and retain more image details.

[0003] In practical applications, electronic devices can acquire raw data according to the requirements for HDR image acquisition, and then process the raw data through color correction, encoding, and other methods to generate HDR images. Because generating HDR images requires high-quality raw data, electronic devices need advanced hardware configurations to acquire raw data that meets the HDR image acquisition requirements, resulting in high costs. Summary of the Invention

[0004] This application provides an image processing method, apparatus, and storage medium, the purpose of which is to solve the problem of how to improve efficiency while saving costs.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] Firstly, this application provides an image processing method that involves acquiring a standard dynamic range (SDR) image; inputting the SDR image into a color mapping model; and having the color mapping model perform color mapping on each pixel in the SDR image based on the color mapping relationship between the SDR image and a high dynamic range (HDR) image to obtain the color information of the HDR image. Based on the color information of the HDR image, an HDR image corresponding to the SDR image is generated. The color mapping model is trained using the SDR image as input and a real HDR image as the target.

[0007] In this application, by converting compatible SDR images into HDR images, the requirement for acquiring raw data according to HDR image acquisition requirements is eliminated, reducing the hardware requirements for electronic devices and effectively saving costs. Furthermore, the method in this application, which processes SDR images to obtain HDR images based on a color mapping model trained on a large amount of training data, reduces reliance on manual processing, improving efficiency while saving labor costs.

[0008] In one possible implementation, an image pyramid generated from an SDR image can be input into the color mapping model. The image pyramid from the SDR image can provide the color mapping model with more scaled input information as a reference, improving the quality of the generated HDR image.

[0009] In one possible implementation, the color mapping model includes a base model and a correction parameter prediction model. The SDR image is input into the color mapping model, which then performs color mapping on each pixel in the SDR image based on the color mapping relationship between the SDR image and the HDR image, obtaining the color information of the HDR image. This includes: inputting the SDR image into the base model, which then performs color mapping on each pixel in the SDR image based on the color mapping relationship, obtaining the initial color information of the HDR image; and adjusting the initial color information of the HDR image based on correction parameters to generate the color information of the HDR image corresponding to the SDR image. The correction parameters are obtained by inputting the SDR image into the correction parameter prediction model for correction parameter prediction.

[0010] In this application, after obtaining the color information of the initial HDR image, the color information of the initial HDR image can be adjusted based on the correction parameters to supplement the detail information of the SDR image and improve the image quality.

[0011] In one possible implementation, the correction parameters include a first parameter and a second parameter. The color information of the initial HDR image is adjusted based on the correction parameters to generate the color information of the HDR image corresponding to the SDR image. This includes multiplying the pixel value of each pixel in the color information of the initial HDR image by the first parameter and adding the product to the second parameter to obtain the pixel value of each pixel in the color information of the HDR image corresponding to the SDR image. In this embodiment, the difference between each pixel in the initial HDR image and the real HDR image can be adjusted based on the correction parameters to improve the image quality.

[0012] In one possible implementation, after generating an HDR image corresponding to the SDR image based on the color information of the HDR image, the method further includes: inputting the HDR image corresponding to the SDR image into a detail enhancement model, whereby the detail enhancement model performs detail enhancement processing on the HDR image corresponding to the SDR image to obtain the enhanced HDR image. The detail enhancement model is trained using the HDR image output by the color mapping model as input and a real HDR image as the target. In this application, by performing detail enhancement processing on the HDR image output by the color mapping model, details in the HDR image can be restored, banding and artifacts reduced, and the image display effect improved.

[0013] In one possible implementation, the HDR image corresponding to the SDR image is input into a detail enhancement model. The detail enhancement model then performs detail enhancement processing on the HDR image corresponding to the SDR image to obtain the enhanced HDR image. Afterward, the method further includes performing residual connection processing on the brightness information of the SDR image and the enhanced HDR image to obtain the processed HDR image. In this application, residual connection processing can supplement the detail information in the original SDR image into the enhanced HDR image, improving the image display effect.

[0014] In one possible implementation, the SDR image is an image from an SDR video, which can be acquired. The SDR video is then decomposed to obtain multiple frames of SDR images and audio data that make up the SDR video. For each frame of the multiple SDR images, the SDR image is input into a color mapping model. The color mapping model performs color mapping on each pixel of the SDR image according to the color mapping relationship between the SDR image and the high dynamic range imaging (HDR) image, thus obtaining the color information of the HDR image. This yields the color information of the multiple HDR images corresponding to the multiple SDR images. Based on the color information of the multiple HDR images, HDR images corresponding to each of the multiple SDR images are generated, thus obtaining the multiple HDR images. Finally, the multiple HDR images are fused with the audio data to obtain the HDR video corresponding to the SDR video.

[0015] In one possible implementation, acquiring a standard dynamic range (SDR) image includes: acquiring a standard dynamic range (SDR) image in response to a user's photo-taking action.

[0016] In one possible implementation, acquiring a standard dynamic range (SDR) image includes: acquiring the standard dynamic range (SDR) image in response to a user's selection operation of a standard dynamic range (SDR) image stored in an electronic device.

[0017] Secondly, this application provides an electronic device, the electronic device comprising: a processor and a memory; the memory is used to store program code and transmit the program code to the processor; the processor is used to execute the steps of an image processing method as described above according to the instructions in the program code.

[0018] Thirdly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of an image processing method as described above. Attached Figure Description

[0019] Figure 1A scene diagram for generating HDR images, provided in the embodiments of this application;

[0020] Figure 2 A scene diagram for generating a second type of HDR image provided in this application embodiment;

[0021] Figure 3 A scene diagram for generating a third type of HDR image provided in this application embodiment;

[0022] Figure 4 A schematic diagram illustrating a batch conversion of SDR images provided in an embodiment of this application;

[0023] Figure 5 A flowchart of an image processing method provided in this application embodiment;

[0024] Figure 6 A schematic diagram of the model structure of the first color mapping model provided in this application embodiment;

[0025] Figure 7 A schematic diagram of the model structure of the second color mapping model provided in this application embodiment;

[0026] Figure 8 This is a schematic diagram of SDR image processing provided in an embodiment of this application;

[0027] Figure 9 A schematic diagram of the model structure of an enhanced detail model provided in an embodiment of this application;

[0028] Figure 10 This is a schematic diagram illustrating detail enhancement of an HDR image, provided as an embodiment of this application.

[0029] Figure 11 This is a schematic diagram of SDR video processing provided in an embodiment of this application;

[0030] Figure 12 An example diagram illustrating the composition of an electronic device provided in an embodiment of this application;

[0031] Figure 13 This is an example diagram of the software structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0032] The terms "first," "second," and "third," etc., used in this application specification, claims, and drawings are used to distinguish different objects, not to limit a specific order.

[0033] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0034] Standard Dynamic Range (SDR) is a common and widely used color display method. Compared to SDR images, HDR images, with their higher dynamic range, contain more image details, and their brightness, contrast, and saturation are all improved. Dynamic range refers to the ratio of the maximum to the minimum brightness value.

[0035] In practical applications, to improve image display quality, electronic devices can acquire raw data according to the requirements for HDR image acquisition, such as ambient light brightness and the brightness of the captured image. After acquiring the raw data required to generate an HDR image, the raw data is processed through color correction, encoding, and other methods to generate the HDR image.

[0036] Generating HDR images requires high-quality raw data, necessitating advanced hardware configurations in the electronic devices. For example, a more sensitive photosensitive element can capture more data. Current methods for generating HDR images require electronic devices with highly sensitive photosensitive elements to acquire raw data that meets the requirements for HDR image capture. This reliance on high-performance hardware in electronic devices results in high costs.

[0037] Meanwhile, the current process of color correction and encoding of raw data relies on manual processing, which increases labor costs and has low processing efficiency.

[0038] Based on this, this application provides an image generation method. By acquiring a compatible SDR image, and based on the color mapping relationship between pixels in the SDR and HDR images, color mapping is performed on each pixel in the SDR image, mapping the colors and color gamut of the SDR image to the colors and color gamut of the HDR image, thus obtaining an HDR image. Compared to methods that directly generate HDR images, generating SDR images requires less raw data and has lower hardware requirements for electronic devices. In this application, by converting a compatible SDR image to an HDR image, it is not necessary to collect raw data according to the acquisition requirements of HDR images, thus reducing the hardware requirements of electronic devices and effectively saving costs while improving image display effects.

[0039] Furthermore, in this application, the method for processing SDR images to obtain HDR images based on a deep learning model trained from a large amount of training data reduces the reliance on manual processing, which can improve efficiency while saving labor costs.

[0040] The image generation method provided in this application can convert SDR images captured by a user into HDR images during image capture; it can also convert SDR images stored in an electronic device into HDR images. For example... Figures 1-2 As shown, these are scene diagrams for generating HDR images in the first and second ways provided in this application embodiment, respectively.

[0041] It should be noted that SDR images can be images captured or stored locally, or images from videos. To distinguish between images and images from videos, images are called still images, and images from videos are called moving images.

[0042] Users can enter the shooting interface 1200, also known as the camera preview interface, by clicking the camera icon 1101 in the initial user interface 1100 or by waking up via voice. Users can select photo mode or video mode in the shooting interface 1200.

[0043] In one possible implementation, the user can select a photo mode, such as... Figure 1 As shown. In photo mode, the user can click the shutter button 1201 to generate an HDR image. The electronic device responds to the user's click of the shutter button, collects data according to the SDR image acquisition requirements, and generates an SDR image. After generating the SDR image, based on the color mapping relationship between each pixel in the SDR and HDR images, a color mapping model is used to map the colors and color gamut of the SDR image to the colors and color gamut of the HDR image, generating the corresponding HDR image. After generating the HDR image, the electronic device can save it to a specified location, such as the device's photo album, so that the user can view the generated HDR image in the album.

[0044] For example, after generating an HDR image, the user can access the album interface 1300 through the album icon 1202 in the shooting interface 1200 to view the generated HDR image 1301.

[0045] In one possible implementation, the user can select the recording mode, such as... Figure 2As shown. In recording mode, the user can click the record button 1203 to generate an HDR video. The electronic device responds to the user's click of the record button, collects data according to the SDR video acquisition requirements, and generates an SDR video. After generating the SDR video, the electronic device can decompose the SDR video to obtain multiple frames of SDR motion images and audio data that make up the SDR video. For each frame of the multiple SDR motion images, the electronic device can perform color mapping on each pixel in the SDR motion image based on the color mapping relationship between the SDR motion image and the HDR motion image, mapping the color and color gamut of the SDR motion image to the color and color gamut of the HDR motion image, generating the corresponding HDR motion image, thus obtaining multiple frames of HDR motion images.

[0046] After acquiring multiple frames of HDR motion images, the electronic device can fuse these images with audio data to obtain an HDR video. The device can then save this HDR video to a specified location, such as its photo album, so that the user can view it there.

[0047] For example, after generating an HDR video, the user can access the album interface 1300 through the album icon 1202 in the shooting interface 1200 to view the generated HDR video 1302.

[0048] like Figure 3 As shown, this figure illustrates a scene of generating HDR images using a third method according to an embodiment of this application. Currently, a large number of SDR images exist, and in this embodiment, an algorithm for converting SDR images to HDR images can be used to achieve the display of high dynamic range images.

[0049] For example, a user can access a user interface containing SDR images, such as a photo album. The user can enter the photo album interface by clicking the album icon 1102 in the initial user interface 1100. If the album contains one or more SDR images, the user can click on the SDR image 1401 that needs to be converted to an HDR image. In response to the user's selection of the SDR image to be converted, the electronic device can perform color mapping on each pixel in the SDR image based on the color mapping relationship between the SDR and HDR images, mapping the colors and color gamut of the SDR image to the colors and color gamut of the HDR image, converting the SDR image to an HDR image, and storing it.

[0050] For example, a user can long-press the SDR image to be converted to select it and enter the selection interface 1500, such as... Figure 3As shown in the selection interface 1500, a selection box appears in the upper right corner of the SDR image. The SDR image selected by the user has a checkmark in the selection box, indicating that it has been selected. Below the selection interface 1500 are operation buttons, including a share button, a convert button 1501, a delete button, and more buttons. The user can click the convert button to convert the selected SDR image to an HDR image.

[0051] In another instance, such as Figure 4 As shown in the figure, this is a schematic diagram of a batch conversion of SDR images provided in an embodiment of this application. The user can long-press on an SDR image that needs to be converted to select the SDR image and enter the selection interface 1500. In the selection interface 1500, a selection box can appear in the upper right corner of the SDR image.

[0052] The SDR image can be a locally saved image or an image from a video, thus enabling simultaneous processing of both saved and video SDR images. For example... Figure 4 Taking an electronic device storing a first SDR video 2501, a first SDR image 2502, and a second SDR image 2503 as an example, the user can long-press the first SDR video 2501 to select it and enter the selection interface 1500. In the selection interface 1500, the selected first SDR video 2501 has a checkmark in its selection box, indicating that it has been selected. The user can then click on the first SDR image 2502 to select it. After selecting the first SDR video 2501 and the first SDR image 2502, both have checkmarks in their upper right corner selection boxes. Based on this, the user can use the conversion button 1501 at the bottom of the selection interface 1500 to convert the selected first SDR video 2501 and first SDR image 2502 into their corresponding HDR video and HDR image, achieving batch conversion of SDR images and improving the efficiency of obtaining HDR images.

[0053] The above describes the application scenarios for generating HDR images provided by the embodiments of this application. The following section will combine... Figure 5 The flowchart shown illustrates the image processing method provided in the embodiments of this application.

[0054] The SDR image in this application can be a captured or locally stored image, or it can be an image from a video. To distinguish between images and images from videos, images are referred to as still images, and images from videos are referred to as moving images. In the case where the SDR image is a captured or locally stored image, such as... Figure 5As shown, an HDR image can be obtained by executing S101-S102.

[0055] S101. Obtain the SDR image.

[0056] In one possible implementation, the electronic device can acquire an SDR image in response to a user's shooting action. For example, after a user takes a photo, the electronic device can generate an SDR image based on the acquired data, so that the SDR image can be subsequently converted to an HDR image based on the color mapping relationship between pixels in the SDR and HDR images.

[0057] Because SDR images are a standard image format, compared to directly generating HDR images, generating SDR images requires less raw data, resulting in lower hardware requirements for electronic devices, greater compatibility, and applicability to various electronic devices. Therefore, generating SDR images before taking photos effectively reduces the hardware requirements of electronic devices, improves applicability, and lowers costs.

[0058] In another possible implementation, the electronic device can acquire an SDR image in response to a user's selection operation of an SDR image in memory. Due to the compatibility of SDR images, a large number of images currently exist in SDR format. The method provided in this application embodiment can convert SDR images stored in the electronic device into HDR images. This can improve the applicability of the electronic device, reduce hardware costs, and improve the image display effect.

[0059] Taking an electronic device that stores a first SDR image, a second SDR image 2502, and a third SDR image as an example, in one example, when a user selects the first SDR image, the electronic device can respond to the user's selection operation of the first SDR image, acquire the first SDR image, and convert the first SDR image into an HDR format image.

[0060] In another instance, when a user selects a first SDR image and a second SDR image, the electronic device can, in response to the user's selection operation, acquire the first SDR image and the second SDR image in order to convert the first SDR image and the second SDR image into HDR format images.

[0061] It should be noted that the number of SDR images acquired at one time is not specifically limited in the embodiments of this application. The electronic device can acquire one SDR image at a time or acquire multiple SDR images in batches. When the electronic device acquires multiple SDR images at one time, it is convenient to convert multiple SDR images into HDR images in batches later, thereby improving the generation efficiency of HDR images.

[0062] S102. Based on the color mapping relationship between the pixels in the SDR image and the HDR image, perform color mapping on each pixel in the SDR image to obtain the color information of the HDR image.

[0063] There is a difference in color gamut between HDR and SDR images. For example, SDR typically uses the BT.709 standard, while HDR typically uses the wider BT.2020 color gamut. The BT.709 standard is the international standard for high-definition television, and it has a relatively small color gamut, the same as the color space of the universal color standard (standard Red Green Blue, sRGB) used for internet media. The BT.2020 color gamut is designed for ultra-high-definition television and wide color gamut applications, and its color gamut range far exceeds sRGB.

[0064] In this embodiment, based on the color mapping relationship between SDR and HDR images, each pixel in the SDR image can be color-mapped to obtain the color information of the corresponding HDR image. This color mapping can include both color mapping and color gamut mapping for each pixel in the SDR image, and the color information can include both the image's color information and color gamut information. Furthermore, the method provided in this application eliminates the need to collect data according to the acquisition requirements of HDR images, reducing the hardware requirements of electronic devices and thus effectively lowering costs.

[0065] In some embodiments, one way to obtain the color information of the HDR image by color mapping each pixel in the SDR image based on the color mapping relationship between pixels in the SDR image and the HDR image is as follows:

[0066] The SDR image is input into a color mapping model based on a deep learning model. The color mapping model can achieve one-to-one color mapping for each pixel through a convolutional neural network. That is, based on the color mapping relationship between the pixels in the SDR image and the HDR image, the color and color gamut of each pixel in the SDR image are mapped one by one to the color and color gamut of each pixel in the HDR image, and the corresponding HDR image is output.

[0067] In some embodiments, the color mapping model can be trained using a large number of SDR images and real HDR images, wherein the real HDR images can be generated by processing the original data corresponding to the HDR images, which are collected according to the acquisition requirements of HDR images.

[0068] Using a large number of SDR images as input to the model and real HDR images as training targets, the model parameters are continuously adjusted during training to increase the similarity between the generated HDR images and the real HDR images, resulting in a trained color mapping model. For example, the model parameters can be adjusted during the training of the color mapping model based on a parameter correction prediction model.

[0069] During the training process using a large amount of training data, the color mapping model can learn the color mapping relationship between pixels in SDR images and real HDR images. Based on this, the color mapping model can convert an acquired SDR image into an HDR image using the learned color mapping relationship between pixels in SDR and real HDR images, exhibiting high generalization ability. After training, the parameters learned by the model can characterize the color mapping relationship of each pixel in the corresponding HDR image of the SDR image.

[0070] In one possible implementation, the image pyramid generated based on the SDR image can be input into the color mapping model to obtain the HDR image corresponding to the SDR image.

[0071] Image pyramids are a type of multi-scale representation of images. An SDR image pyramid is a collection of images arranged in a pyramid shape with progressively decreasing resolution from bottom to top. All images in an SDR image pyramid originate from the same original image and can be considered as a set of image resolutions derived from that SDR image. The bottom of the image pyramid is a high-resolution representation of the image to be processed, while the top is a low-resolution approximation. As you move up the image pyramid, the image size and resolution decrease.

[0072] Using the original SDR image as the image to be processed, multi-resolution processing is performed on the SDR image to obtain its image pyramid. This image pyramid is then input into a color mapping model. Based on the learned color mapping relationships, this model can obtain the color information of the corresponding HDR image. Furthermore, the image pyramid of the SDR image provides more scale references for obtaining the color information of the HDR image, facilitating the subsequent conversion of the SDR image to an HDR image.

[0073] For example, a color mapping model can employ a Convolutional Neural Network (CNN) model. Its convolutional kernel size can be 1×1 with a stride of 1, thereby achieving one-to-one mapping of image pixels.

[0074] In another possible implementation, the color mapping model can include a base model and a modified parameter prediction model. For example... Figure 6 As shown, after obtaining the image pyramid of the SDR image, the image pyramid can be input into the base model, and the SDR image as the image to be processed can be input into the correction parameter prediction model. The base model is used to convert the SDR image into an initial HDR image; the correction parameter prediction model is used to adjust the color information of this initial HDR image.

[0075] For example, the base model uses the image pyramid of the SDR image as a reference, along with color information of a larger scale. Based on the color mapping relationship between pixels in the SDR and HDR images, it maps the color and color gamut of each pixel in the SDR image to the color and color gamut of each pixel in the HDR image, outputting an initial HDR image. The image pyramid of the SDR image can serve as additional prior information, providing a reference for the base model to output the initial HDR image. The pixel values ​​of this initial HDR image can be used to indicate the characteristics of the initial HDR image converted from the SDR image, that is, to indicate the color information of each pixel in the initial HDR image.

[0076] The modified parameter prediction model can use the SDR image as prior information to generate modified parameters. These modified parameters are used to fine-tune the color information of the initial HDR image output by the base model, thus obtaining the color information of the HDR image corresponding to the SDR image.

[0077] In this embodiment, after obtaining the color information and correction parameters of the initial HDR image, the color information of the initial HDR image output by the base model can be adjusted based on the correction parameters to complete color mapping and obtain the color information of the HDR image corresponding to the SDR image. The process of adjusting the color information of the initial HDR image output by the base model through the correction parameters can supplement some detailed information from the SDR image into the initial HDR image, thereby further improving the display effect of the generated HDR image.

[0078] For example, the base model in this application embodiment can be a Convolutional Neural Network (CNN) model; the parameter correction prediction model can be a Multilayer Perceptron (MLP) model, or a model with multiple convolutional layers. The base model's convolutional kernel size can be 1×1 with a stride of 1, thereby achieving a one-to-one mapping between image pixels.

[0079] The following combination Figure 7 This paper explains the process of processing SDR images to obtain HDR images using a color mapping model composed of a base model and a modified parameter prediction model.

[0080] Figure 7 In this paper, we will take the linear MLP model as an example of a modified parameter prediction model and the CNN model as an example of a base model. For instance, the image pyramid of an SDR image consists of a first SDR image, a first sub-SDR image, and a second sub-SDR image. The first SDR image is the original SDR image, i.e., the SDR image to be processed. The first and second sub-SDR images are images obtained by multi-resolution processing of the first SDR image, such as... Figure 7 In this diagram, the first SDR image is the original SDR image. The images are likened to a pyramid, with each layer representing a smaller image and lower resolution. For example, reducing the first SDR image by a factor of 2 yields the first sub-SDR image; reducing it by a factor of 4 yields the third sub-SDR image.

[0081] It should be noted that the scaling factor of the first SDR image is not specifically limited in the embodiments of this application. It can be understood that the above scaling factor of 2 and 4 of the first SDR image is only an example.

[0082] In this embodiment, the first SDR image can be used as prior information, and the first sub-SDR image and the second sub-SDR image can be used as additional prior information to generate more accurate correction parameters. Fine-tuning the initial HDR image based on these more accurate correction parameters can effectively improve the display effect of the corrected HDR image.

[0083] The image pyramid of the SDR image is input into a CNN model, which outputs the color information of the initial HDR image. This initial HDR image color information may include the pixel values ​​of each pixel in the uncorrected HDR image, i.e., indicating the pixel values ​​of each pixel in the initial HDR image. Inputting the first SDR image as prior information into a linear MLP model yields correction parameters; for example, these correction parameters may include a first parameter and a second parameter.

[0084] Based on the first and second parameters, the pixel values ​​of each pixel in the color information of the initial HDR image output by the CNN model can be corrected to obtain the pixel values ​​of each pixel in the color information of the HDR image corresponding to the SDR image, as shown in Equation (1):

[0085] M'=a×M+b(1)

[0086] Where M is the pixel value of each pixel in the color information of the initial HDR image output by the CNN model; M' is the pixel value of each pixel in the color information of the HDR image corresponding to the SDR image; a is the first parameter; b is the second parameter.

[0087] S103. Generate an HDR image based on the color information of the HDR image.

[0088] For example, an HDR image corresponding to an SDR image can be generated based on a color mapping model. The SDR image can have a color depth of 8 bits. It can be understood that the higher the color depth, the more colors are available. Inputting an 8-bit SDR image into a color mapping model can yield an HDR image with a color depth of 10-16 bits.

[0089] In this embodiment of the application, when training the color mapping model, a large number of SDR images can be used as input, and real HDR images can be used as the training target of the color mapping model. The model parameters can be adjusted by modifying the parameter prediction model. The loss function can be the mean squared error loss (L2 loss) function. When the model converges, the trained color mapping model can be obtained.

[0090] The modified parameter prediction model in this application embodiment can adjust and optimize the model parameters during the model training phase to obtain a trained color mapping model; after obtaining the initial HDR image corresponding to the SDR image, the color information of the initial HDR image can be adjusted to improve the display effect of the HDR image.

[0091] This application employs a method of acquiring SDR images and converting them into HDR images. This eliminates the need to collect raw data according to the acquisition requirements of HDR images, reducing the hardware requirements of electronic devices and thus effectively lowering costs. Furthermore, it utilizes a deep learning model trained on a large amount of training data to achieve the SDR-to-HDR conversion based on the color mapping relationship between pixels in the SDR and HDR images. This model exhibits high generalization ability, improving image display quality while saving costs.

[0092] Furthermore, the process of converting SDR images to HDR images based on a deep learning model in this application does not require manual image processing, effectively improving the efficiency of image generation.

[0093] Meanwhile, since pixels are non-linear, they correspond to the non-linear domain; while brightness is linear, it corresponds to the linear domain. Compared to image processing methods that require brightness processing in the linear domain, this application uses a deep learning model to directly obtain HDR images in the pixel domain. This is equivalent to directly converting between SDR and HDR images in the non-linear domain, allowing for the acquisition of HDR images in fewer steps and improving the efficiency of HDR image generation.

[0094] It should be noted that, through Figure 5 In step S103 of the image processing method shown, after generating an HDR image based on the color information of the HDR image, the HDR image may have problems such as loss of image details, tortuosity, and artifacts.

[0095] Based on this, in one possible implementation, after generating an HDR image using color information from the HDR image, the HDR image obtained through color mapping can be reconstructed to restore image details, reduce banding and artifacts, and improve the display effect of the HDR image. For example... Figure 8 As shown, the SDR image is input into the color mapping model to obtain the corresponding HDR image. The HDR image output by the color mapping model is then input into the detail enhancement model to obtain the enhanced HDR image.

[0096] In some embodiments, one way to reconstruct the HDR image obtained through color mapping to obtain an enhanced HDR image is as follows:

[0097] The HDR image obtained from the SDR image through a color mapping model is then input into a detail smoothing model to obtain the enhanced HDR image. The detail smoothing model is used to restore details in the HDR image and reduce banding and artifacts.

[0098] For example, during the training of the detail enhancement model, the output of the color mapping model can be used as input, that is, the image pyramid of the HDR image output by the color mapping model can be used as input, and the real HDR image can be used as the training target. During training, the mean squared error loss function can be used to optimize the model until it converges, thus obtaining the trained detail enhancement model.

[0099] The image pyramid for HDR images can be obtained by multi-resolution processing of the HDR image output from the color mapping model. The image pyramid of an HDR image provides more scale-specific reference information for the detail enhancement model, improving the quality of the output image.

[0100] This application's embodiments, based on obtaining an HDR image, can effectively improve the image display effect by enhancing its details. The detail enhancement model can adopt an encoder-decoder structure, such as... Figure 9 As shown in the figure, this is a schematic diagram of the model structure of a detail enhancement model provided in an embodiment of this application. The HDR image output by the color mapping model is input into the encoder, the encoder encodes the HDR image, and the encoded HDR image is input into the decoder.

[0101] In one example, after obtaining the HDR image output by the color mapping model, the HDR image can be processed in multiple resolutions to obtain the image pyramid of the HDR image. The image pyramid of the HDR image is then input into the encoder, which encodes the HDR image and inputs the encoded HDR image into the decoder.

[0102] The decoder takes the encoded HDR image as input and can recover details, reduce banding and artifacts, and obtain an enhanced HDR image. The brightness information of the SDR image can be used as supplementary information for the encoder-decoder model to perform residual connection processing on the enhanced HDR image, resulting in a processed HDR image.

[0103] In one possible implementation, such as Figure 10 As shown in the figure, this is a schematic diagram of detail enhancement of an HDR image provided by an embodiment of this application. The HDR image output by the color mapping model is processed through multi-resolution processing. The resulting image pyramid of the HDR image is used as input to the detail enhancement model. Multi-scale HDR images can be input; that is, after the image pyramid of the HDR image is fed into the detail enhancement model, an enhanced HDR image can be output.

[0104] Based on this, the brightness information of the original SDR image can be used, that is, the brightness map of the SDR image to be processed can be used to perform residual connection processing on the enhanced HDR image to obtain the processed HDR image.

[0105] For example, a residual connection method can be used to overlay the original SDR brightness image with the enhanced HDR image, which is equivalent to adding some detailed information of the SDR image to the enhanced HDR image to obtain a processed HDR image, thereby improving the display effect of the HDR image.

[0106] The principle of residual connections is to add the network's input and output together, thereby enabling the network to better learn residual information and improve network performance and training effectiveness. Its core idea is to directly add the network's input and output, allowing the network to directly learn residual information.

[0107] The luminance information of the SDR image is obtained in the form of a luminance map. The luminance map is calculated by converting the color space of the RGB image to the XYZ color space, obtaining the image of the Y channel, and using it as the luminance map of the SDR image. Here, XYZ is a color space encoding method, and "Y" can represent luminance. The formula for calculating Y in RGB images under different color gamuts is different. For example, the SDR image under the BT.709 color gamut can be obtained by equation (2):

[0108] Y=0.2126×R+0.7152×G+0.0722×B(2)

[0109] An SDR image in the DCI-P3 color gamut can be obtained using equation (3):

[0110] Y=0.2290×R+0.6917×G+0.0793×B(3)

[0111] In Equation (2), R, G, and B represent the RGB values ​​of a pixel in the BT.709 color gamut; in Equation (3), R, G, and B represent the RGB values ​​of a pixel in the DCI-P3 color gamut.

[0112] In this embodiment, by adding the luminance map of the original SDR image to the enhanced HDR image, the network can directly learn the residual information between the luminance map of the original SDR image and the enhanced HDR image. The output adjusted HDR image contains more detail information compared to the enhanced HDR image, thus improving the image display effect.

[0113] Unlike SDR images, which are captured or locally stored images, SDR images, which are images from video, require decomposition into multiple SDR frames and audio data before conversion to HDR. After converting these multiple SDR frames into corresponding HDR images, the HDR images and audio data are then merged to obtain the HDR video.

[0114] like Figure 11 As shown in the figure, this is a schematic diagram of SDR video processing provided in an embodiment of this application. The SDR video can be a video composed of multiple frames of SDR images and audio data. After acquiring the SDR video, it can be decomposed to obtain multiple frames of SDR images and audio data. The SDR video can be captured by camera or stored locally.

[0115] For each frame of a multi-frame SDR image, the processing methods described above for captured or locally stored images can be used to convert the multi-frame SDR image into a corresponding HDR image. After converting the multi-frame SDR image into the corresponding HDR image, a multi-frame HDR image is obtained. This multi-frame HDR image is then combined with audio data to obtain an HDR video.

[0116] In one possible implementation, to improve the display effect of HDR images, the method of detail enhancement processing of the HDR image corresponding to the SDR image described above can be used to obtain multi-frame enhanced HDR images or multi-frame processed HDR images. These multi-frame enhanced HDR images or multi-frame processed HDR images are then combined with audio data to obtain HDR video.

[0117] In one possible implementation, perceptual quantizer (PQ) coding or hybrid log-gamma (HLG) coding can be used to process multiple frames of HDR images to obtain processed multi-frame HDR images. These processed multi-frame HDR images are then fused with audio data to obtain HDR video, achieving a high dynamic range display effect.

[0118] Among them, PQ encoding allows image brightness levels to reach 10000 cd / m². 2 HLG encoding is a high dynamic range video encoding technology that can adjust image brightness and contrast to allow users to see richer and more realistic colors and details.

[0119] In summary, this application provides an image processing method that can convert a compatible SDR image into an HDR image using a deep learning model trained on a large amount of training data. This deep learning model can perform color mapping on each pixel in the SDR image according to the color mapping relationship between pixels in the SDR and HDR images, mapping the colors and color gamut of the SDR image to those of the HDR image, thus obtaining an HDR image with a higher dynamic range.

[0120] In this application, by converting compatible SDR images into HDR images, the original data does not need to be collected according to the acquisition requirements of HDR images, reducing the hardware requirements of electronic devices and improving the image display effect while saving costs. Simultaneously, by employing a deep learning model trained on a large amount of training data, the conversion from SDR to HDR images can be achieved based on the color mapping relationship between pixels in SDR and HDR images. This model exhibits high generalization ability, improving image display effect without requiring manual image processing, thus effectively increasing image generation efficiency.

[0121] This application can be applied to electronic devices. In some embodiments, the electronic device may be a mobile phone, tablet computer, desktop computer, laptop computer, ultra-mobile personal computer (UMPC), handheld computer, netbook, personal digital assistant (PDA), wearable electronic device, smartwatch, etc. This application does not impose any special limitations on the specific form of the above-mentioned electronic devices. In this embodiment, the structure of the electronic device may be as follows: Figure 12 As shown, Figure 12 This is a component example diagram of an electronic device provided in an embodiment of this application.

[0122] like Figure 12 As shown, the electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a sensor module 130, a camera 140, and a display screen 141, etc.

[0123] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0124] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0125] For example, in this embodiment, the processor 110 can acquire an SDR image and, based on the color mapping relationship between the SDR image and the HDR image, obtain the color information of each pixel in the HDR image corresponding to the SDR image. Based on the color information, color mapping is performed on each pixel in the SDR image to generate the HDR image corresponding to the SDR image.

[0126] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly, such as photo-taking instructions, video-recording instructions, etc. The electronic device can generate a static HDR image based on the photo-taking instruction and a dynamic HDR video based on the video-recording instruction.

[0127] In one example, the electronic device can store the generated HDR image in memory, and in response to a user's action of viewing the HDR image, the electronic device can retrieve the corresponding HDR image from memory and display it to the user in the user interface.

[0128] In another example, an SDR image may be stored in the memory, and the electronic device may convert the SDR image into a corresponding HDR image in response to a user's selection of the SDR image, and store the generated HDR image in the memory.

[0129] Electronic devices implement display functions through a GPU, a display screen 141, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 141 and the application processor. The GPU performs mathematical and geometric calculations and is used for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0130] Display screen 141 is used to display images, videos, etc. Display screen 141 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device may include one or N displays 141, where N is a positive integer greater than 1.

[0131] The display screen 141 of the electronic device can display a series of graphical user interfaces (GUIs). For example, in this embodiment, the image interfaces that can be displayed on the display screen 141 include an initial user interface, a camera interface, and a photo album interface. When a user presses the camera button or video recording button on the camera interface, they can take a photo or record a video to obtain an HDR image.

[0132] Electronic devices can achieve shooting functions through ISP, camera 140, video codec, GPU, display 141 and application processor.

[0133] Camera 140 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to an ISP for conversion into a digital image signal. In some embodiments, the electronic device may include one or N cameras 140, where N is a positive integer greater than 1.

[0134] Video codecs are used to compress or decompress digital video. Electronic devices can support one or more video codecs. This allows the electronic device to play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0135] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 110 through the external memory interface 120 to perform data storage functions. For example, in this embodiment, generated HDR images and other files are saved on the external memory card.

[0136] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in internal memory 121.

[0137] In addition, an operating system runs on top of these components. Examples include Apple's iOS, Google's Android, and Microsoft's Windows. Applications can be installed and run on this operating system.

[0138] The operating system of an electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application uses the layered architecture Android system as an example to illustrate the software structure of an electronic device.

[0139] Figure 13 This is an example diagram of the software structure of an electronic device provided in an embodiment of this application.

[0140] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0141] The application layer can include a series of application packages. For example... Figure 13 As shown, the application package can include applications such as camera and photo album.

[0142] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example... Figure 13 As shown, the application framework layer may include image conversion modules, etc.

[0143] The image conversion module may include a color mapping module and a detail enhancement module. The color mapping module is used to perform color mapping on each pixel in the SDR image based on the color mapping relationship between the pixels in the SDR image and the HDR image, mapping the colors and color gamut of the SDR image to the colors and color gamut of the HDR image to obtain the HDR image. The detail enhancement module is used to perform detail enhancement processing on the HDR image output by the color mapping module to restore the details in the HDR image and obtain the enhanced HDR image.

[0144] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0145] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0146] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0147] A system library can include multiple functional modules. For example, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0148] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0149] A 2D graphics engine is a graphics engine for 2D drawing.

[0150] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, and sensor driver.

[0151] It should be noted that although the embodiments of this application are illustrated using the Android system as an example, the basic principles are also applicable to electronic devices based on operating systems such as iOS and Windows.

[0152] This embodiment also provides a computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform the aforementioned method steps to implement the method described in the above embodiment.

[0153] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, include: Acquire standard dynamic range SDR images; The image pyramid generated based on the SDR image is input into the base model of the color mapping model. The base model performs color mapping on each pixel in the SDR image according to the color mapping relationship between the SDR image and the high dynamic range imaging (HDR) image to obtain the color information of the initial HDR image. The color information of the initial HDR image is adjusted based on the correction parameters to generate the color information of the HDR image corresponding to the SDR image. The correction parameters are obtained by inputting the SDR image into the correction parameter prediction model in the color mapping model to predict the correction parameters. The color mapping model is trained using SDR images as input and real HDR images as targets. Based on the color information of the HDR image, an HDR image corresponding to the SDR image is generated.

2. The method according to claim 1, characterized in that, The correction parameters include a first parameter and a second parameter. Adjusting the color information of the initial HDR image based on the correction parameters to generate the color information of the HDR image corresponding to the SDR image includes: The pixel values ​​of each pixel in the color information of the initial HDR image are multiplied by the first parameter and added to the second parameter to obtain the pixel values ​​of each pixel in the color information of the HDR image corresponding to the SDR image.

3. The method according to claim 1 or 2, characterized in that, After generating the HDR image corresponding to the SDR image based on the color information of the HDR image, the process further includes: The HDR image corresponding to the SDR image is input into the detail enhancement model, which performs detail enhancement processing on the HDR image corresponding to the SDR image to obtain the enhanced HDR image. The detail enhancement model is trained using the HDR image output by the color mapping model as input and the real HDR image as the target.

4. The method according to claim 3, characterized in that, After inputting the HDR image corresponding to the SDR image into the detail enhancement model, and having the detail enhancement model perform detail enhancement processing on the HDR image corresponding to the SDR image to obtain the enhanced HDR image, the process further includes: The brightness information of the SDR image and the enhanced HDR image are subjected to residual concatenation processing to obtain the processed HDR image.

5. The method according to claim 1, characterized in that, The SDR image is an image from an SDR video, and the acquisition of a standard dynamic range SDR image includes: Acquire SDR video; The SDR video is decomposed to obtain multiple frames of SDR images and audio data that make up the SDR video; The step involves inputting the image pyramid generated based on the SDR image into the base model of the color mapping model. The base model then performs color mapping on each pixel in the SDR image according to the color mapping relationship between the SDR image and the high dynamic range (HDR) image, obtaining the initial color information of the HDR image, including: For each SDR image in the multi-frame SDR image, the image pyramid generated based on the SDR image is input into the base model in the color mapping model. The base model performs color mapping on each pixel in the SDR image according to the color mapping relationship between the SDR image and the high dynamic range imaging HDR image to obtain the color information of the initial HDR image, and obtain the color information of the multiple initial HDR images corresponding to the multiple SDR images. The step of adjusting the color information of the initial HDR image based on correction parameters to generate the color information of the HDR image corresponding to the SDR image includes: For each initial HDR image in the multi-frame initial HDR image, the color information of the initial HDR image is adjusted based on the correction parameters to generate the color information of the HDR image, thereby obtaining the color information of the multi-frame HDR image corresponding to the multi-frame SDR image. The step of generating an HDR image corresponding to the SDR image based on the color information of the HDR image includes: Based on the color information of the multi-frame HDR images, generate HDR images corresponding to each of the multi-frame SDR images to obtain multi-frame HDR images; The process of generating HDR images corresponding to each of the SDR images based on the color information of the multiple HDR images, and obtaining the multiple HDR images further includes: The multi-frame HDR images are fused with the audio data to obtain the HDR video corresponding to the SDR video.

6. The method according to claim 1, characterized in that, The acquisition of standard dynamic range (SDR) images includes: In response to the user's photo-taking action, acquire a standard dynamic range (SDR) image.

7. The method according to claim 1, characterized in that, The acquisition of standard dynamic range (SDR) images includes: In response to the user's selection of a standard dynamic range (SDR) image stored in the electronic device, a standard dynamic range (SDR) image is acquired.

8. An electronic device, characterized in that, The electronic device includes: a processor and a memory; The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the steps of an image processing method as described in any one of claims 1-7 according to instructions in the program code.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of an image processing method as described in any one of claims 1-7.

Images