Image processing method and device and storage medium

Abstract

The invention provides an image processing method and device, and a storage medium. The method comprises the steps of obtaining a standard dynamic range (SDR) image; and inputting the SDR image into a color mapping model, performing color mapping and color gamut mapping on each pixel point in the SDR image by the color mapping model according to a color and color gamut mapping relation between the SDR image and a high dynamic range imaging HDR image to obtain color information and color gamut information of the HDR image, and generating the HDR image based on the color information and the color gamut information of the HDR image. The SDR image is converted into the HDR image, original data does not need to be collected according to the collection requirement of the HDR image, the requirement for hardware configuration of the electronic equipment is lowered, and the cost can be effectively saved. Meanwhile, the method for converting the SDR image into the HDR image based on the color mapping model obtained through training of a large amount of training data reduces the dependence on manual processing, and can save the labor cost while improving the efficiency.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device and storage medium. Background Art

[0002] With the development of image processing technology, people have higher and higher requirements for image display effects. At present, we can use the method of generating high dynamic range imaging (HDR) images to improve the image display effect. HDR is an image processing technology that can provide image brightness and contrast. HDR images can provide more dynamic range and retain more image details.

[0003] In actual application scenarios, electronic devices can collect raw data according to the acquisition requirements of HDR images, and perform color adjustment, encoding and other processing on the raw data to generate HDR images. Since the generation of HDR images has high requirements for the collected raw data, the electronic devices need to have high hardware configuration to collect raw data that meets the HDR image acquisition requirements, which leads to high costs. Summary of the invention

[0004] The present application provides an image processing method, device and storage medium, which aim to solve the problem of how to save costs while improving efficiency.

[0005] In order to achieve the above purpose, this application adopts the following technical solutions:

[0006] In the first aspect, the present application provides an image processing method, which obtains a standard dynamic range SDR image; inputs the SDR image into a color mapping model, and the color mapping model performs color mapping on each pixel in the SDR image according to the color mapping relationship between the SDR image and the high dynamic range imaging HDR image, obtains the color information of the HDR image, and generates an HDR image corresponding to the SDR image based on the color information of the HDR image. The color mapping model is trained with the SDR image as input and the real HDR image as the target.

[0007] In the present application, by converting a compatible SDR image into an HDR image, it is not necessary to collect original data according to the acquisition requirements of the HDR image, which reduces the requirements for the hardware configuration of the electronic device and can effectively save costs. At the same time, in the present application, based on the color mapping model trained by a large amount of training data, the method of processing the SDR image to obtain the HDR image reduces the reliance on manual processing, which can improve efficiency and save labor costs.

[0008] In a possible implementation, an image pyramid generated based on an SDR image may be input into a color mapping model, wherein the image pyramid of the SDR image may provide the color mapping model with input information of more scales as a reference, thereby improving the quality of the generated HDR image.

[0009] In a possible implementation, the color mapping model includes a basic model and a correction parameter prediction model, an SDR image is input into the color mapping model, and the color mapping model performs color mapping on each pixel in the SDR image according to the color mapping relationship between the SDR image and the high dynamic range imaging HDR image, so as to obtain the color information of the HDR image, including: inputting the SDR image into the basic model, and the basic model performs color mapping on each pixel in the SDR image according to the color mapping relationship between the SDR image and the high dynamic range imaging HDR image, so as to obtain the color information of the initial HDR image; adjusting the color information of the initial HDR image based on the correction parameter, so as to generate the color information of the HDR image corresponding to the SDR image. Among them, the correction parameter is obtained by inputting the SDR image into the correction parameter prediction model for correction parameter prediction.

[0010] In the present application, after obtaining the color information of the initial HDR image, the color information of the initial HDR image can be adjusted based on the correction parameters, thereby supplementing the detail information of the SDR image and improving the picture quality of the image.

[0011] In a possible implementation, the correction parameter includes a first parameter and a second parameter, and the color information of the initial HDR image is adjusted based on the correction parameter to generate the color information of the HDR image corresponding to the SDR image, including: multiplying the pixel value of each pixel point in the color information of the initial HDR image by the first parameter and adding the second parameter to obtain the pixel value of each pixel point in the color information of the HDR image corresponding to the SDR image. In the embodiment of the present application, the difference between each pixel point of the initial HDR image and the real HDR image can be adjusted based on the correction parameter to improve the picture quality of the image.

[0012] In one possible implementation, after generating an HDR image corresponding to an SDR image based on the color information of the HDR image, the method further includes: inputting the HDR image corresponding to the SDR image into a detail enhancement model, and performing detail enhancement processing on the HDR image corresponding to the SDR image by the detail enhancement model to obtain an enhanced HDR image. The detail enhancement model is trained by using the HDR image output by the color mapping model as input and the real HDR image as a target. In the present application, by performing detail enhancement processing on the HDR image output by the color mapping model, the details in the HDR image can be restored, faults and artifacts can be reduced, and the display effect of the image can be improved.

[0013] In one possible implementation, the HDR image corresponding to the SDR image is input into a detail enhancement model, and the detail enhancement model performs detail enhancement processing on the HDR image corresponding to the SDR image to obtain the enhanced HDR image, and further includes: performing residual connection processing on the brightness information of the SDR image and the enhanced HDR image to obtain the processed HDR image. In the present application, the detail information in the original SDR image can be supplemented into the enhanced HDR image through residual connection processing to improve the display effect of the image.

[0014] In a possible implementation, the SDR image is an image in an SDR video, and the SDR video can be obtained; the SDR video is decomposed to obtain multiple frames of SDR images and audio data that constitute the SDR video; for each SDR image in the multiple SDR images, the SDR image is input into a color mapping model, and the color mapping model performs color mapping on each pixel in the SDR image according to the color mapping relationship between the SDR image and the high dynamic range imaging HDR image to obtain color information of the HDR image, and obtain color information of multiple frames of HDR images corresponding to the multiple SDR images; based on the color information of the color information of the multiple HDR images, HDR images corresponding to each of the multiple SDR images are generated to obtain multiple HDR images; the multiple HDR images are fused with the audio data to obtain an HDR video corresponding to the SDR video.

[0015] In a possible implementation, acquiring a standard dynamic range SDR image includes: acquiring a standard dynamic range SDR image in response to a photo taking operation of a user.

[0016] In a possible implementation, acquiring a standard dynamic range SDR image includes: acquiring the standard dynamic range SDR image in response to a user's selection operation of the standard dynamic range SDR image stored in the electronic device.

[0017] Second aspect: The present application provides an electronic device, comprising: a processor and a memory; the memory is used to store program code and transmit the program code to the processor; the processor is used to execute the steps of an image processing method as described above according to the instructions in the program code.

[0018] A third aspect: The present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of an image processing method as described above are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1A first scene graph for generating an HDR image provided in an embodiment of the present application;

[0020] Figure 2 A second scene graph for generating an HDR image provided in an embodiment of the present application;

[0021] Figure 3 A third scene graph for generating an HDR image provided in an embodiment of the present application;

[0022] Figure 4 A schematic diagram of batch conversion of SDR images provided in an embodiment of the present application;

[0023] Figure 5 A flowchart of an image processing provided in an embodiment of the present application;

[0024] Figure 6 A schematic diagram of the model structure of the first color mapping model provided in an embodiment of the present application;

[0025] Figure 7 A schematic diagram of the model structure of the second color mapping model provided in an embodiment of the present application;

[0026] Figure 8 A schematic diagram of SDR image processing provided in an embodiment of the present application;

[0027] Fig. 9 A schematic diagram of a model structure of a detail enhancement model provided in an embodiment of the present application;

[0028] Fig.10 A schematic diagram of detail enhancement of an HDR image provided in an embodiment of the present application;

[0029] Fig.11 A schematic diagram of SDR video processing provided in an embodiment of the present application;

[0030] Fig.12 An exemplary diagram of the composition of an electronic device provided in an embodiment of the present application;

[0031] Fig.13 An example diagram of the software structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The terms "first", "second", "third", etc. in the specification, claims and drawings of this application are used to distinguish different objects rather than to limit a specific order.

[0033] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0034] Standard Dynamic Range (SDR) is a common color display method with high popularity. Compared with SDR images, HDR images with higher dynamic range contain more image details, and their brightness, contrast and saturation are improved in presentation effect. The dynamic range refers to the ratio of the maximum brightness to the minimum brightness.

[0035] In actual application scenarios, in order to improve the display effect of images, electronic devices can collect raw data according to the acquisition requirements of HDR images, such as the brightness of ambient light, the brightness of the acquired picture, etc. After the raw data required to generate the HDR image is collected, the HDR image is generated by coloring and encoding the raw data.

[0036] Since the generation of HDR images requires high requirements for the collected raw data, the electronic device needs to have a higher hardware configuration. For example, the higher the sensitivity of the photosensitive element, the more data information can be collected. The current method of generating HDR images requires the electronic device to be equipped with a high-sensitivity photosensitive element to collect raw data that meets the HDR image collection requirements. The method that relies on electronic devices with strong hardware configurations to obtain HDR images has the problem of high cost.

[0037] At the same time, the current process of color adjustment, encoding and other processing of raw data needs to rely on manual processing. This processing method not only increases labor costs, but also has the problem of low processing efficiency.

[0038] Based on this, the present application provides an image generation method, by obtaining a compatible SDR image, based on the color mapping relationship between the SDR image and the pixel points in the HDR image, color mapping is performed on each pixel point in the SDR image, and the color and color gamut of the SDR image are mapped to the color and color gamut of the HDR image to obtain an HDR image. Compared with the method of directly generating an HDR image, less raw data needs to be collected to generate an SDR image, and the hardware configuration requirements of the electronic device are lower. In the present application, by converting a compatible SDR image into an HDR image, it is not necessary to collect raw data according to the acquisition requirements of the HDR image, which reduces the requirements for the hardware configuration of the electronic device, and can effectively save costs while improving the image display effect.

[0039] Furthermore, in the present application, a method for processing SDR images to obtain HDR images based on a deep learning model trained with a large amount of training data reduces dependence on manual processing, which can save labor costs while improving efficiency.

[0040] The image generation method provided by the embodiment of the present application can convert the SDR image taken by the user into an HDR image when taking an image; or convert the SDR image stored in the electronic device into an HDR image. Figure 1-Figure 2 As shown, they are respectively the first scene graph for generating an HDR image provided in an embodiment of the present application and the second scene graph for generating an HDR image provided in an embodiment of the present application.

[0041] It should be noted that an SDR image can be a captured or locally stored image, or an image in a video. In order to distinguish an image from an image in a video, the image is called a static image, and the image in the video is called a dynamic image.

[0042] The user can enter the shooting interface 1200, also called the camera preview interface, by clicking the camera icon 1101 in the initial user interface 1100 or by voice wake-up, and the user can select the photo mode or the video recording mode in the shooting interface 1200.

[0043] In a possible implementation, the user can select a photo taking mode, such as Figure 1 As shown. In the photo mode, the user can click the photo button 1201 to generate an HDR image. In response to the user clicking the photo button, the electronic device collects data according to the acquisition requirements of the SDR image and generates an SDR image. After the SDR image is generated, based on the color mapping relationship between the SDR image and each pixel in the HDR image, each pixel in the SDR image is color mapped based on the color mapping model, and the color and color gamut of the SDR image are mapped to the color and color gamut of the HDR image to generate an HDR image corresponding to the SDR image. After generating the HDR image, the electronic device can save the HDR image in a specified location, for example, save the HDR image in an album of the electronic device so that the user can view the generated HDR image in the album.

[0044] Exemplarily, after the HDR image is generated, the user may enter the album interface 1300 through the album icon 1202 in the shooting interface 1200 to view the generated HDR image 1301 .

[0045] In a possible implementation, the user can select a video recording mode, such as Figure 2As shown. In the video recording mode, the user can click the video recording button 1203 to generate an HDR video. In response to the user clicking the video recording button, the electronic device collects data according to the collection requirements of the SDR video and generates an SDR video. After generating the SDR video, the electronic device can decompose the SDR video to obtain multiple frames of SDR dynamic images and audio data that constitute the SDR video. For each frame of the SDR dynamic image in the multiple frames of SDR dynamic images, the electronic device can perform color mapping on each pixel in the SDR dynamic image based on the color mapping relationship between the SDR dynamic image and each pixel in the HDR dynamic image, map the color and color gamut of the SDR dynamic image to the color and color gamut of the HDR dynamic image, generate an HDR dynamic image corresponding to the SDR dynamic image, and thus obtain multiple frames of HDR dynamic images.

[0046] After obtaining multiple frames of HDR dynamic images, the electronic device can fuse the multiple frames of HDR dynamic images with the audio data to obtain an HDR video. The electronic device can save the HDR video in a specified location, such as saving the HDR video in an album of the electronic device so that the user can view the generated HDR video in the album.

[0047] Exemplarily, after generating the HDR video, the user may enter the album interface 1300 through the album icon 1202 in the shooting interface 1200 to view the generated HDR video 1302 .

[0048] like Figure 3 As shown, this figure is a third scene diagram for generating HDR images provided in the embodiment of the present application. Currently, there are a large number of SDR images. In the embodiment of the present application, the display of high dynamic images can be achieved by converting SDR images to HDR images.

[0049] Exemplarily, the user can enter a user interface in which an SDR image is stored, such as an album interface. The user can click the album icon 1102 in the initial user interface 1100 to enter the album interface. In the case where one or more SDR images are stored in the album, the user can click on the SDR image 1401 that needs to be converted to an HDR image. In response to the SDR image that needs to be converted selected by the user, the electronic device can perform color mapping on each pixel in the SDR image based on the color mapping relationship between the SDR image and each pixel in the HDR image, map the color and color gamut of the SDR image to the color and color gamut of the HDR image, convert the SDR image into an HDR image and store it.

[0050] For example, the user can long press the SDR image to be converted to select the SDR image and enter the selection interface 1500, such as Figure 3As shown, in the selection interface 1500, a selection box may appear in the upper right corner of the SDR image, wherein a check mark is displayed in the selection box of the SDR image selected by the user, indicating that it has been selected. Operation buttons appear below the selection interface 1500, including a share button, a conversion button 1501, a delete button, and more buttons. The user can convert the selected SDR image into an HDR image by clicking the conversion button.

[0051] In another example, if Figure 4 As shown, this figure is a schematic diagram of a batch conversion of SDR images provided by an embodiment of the present application. The user can long press an SDR image to be converted to select the SDR image and enter the selection interface 1500. In the selection interface 1500, a selection box may appear in the upper right corner of the SDR image.

[0052] The SDR image may be a locally saved image or an image in a video, so that the saved SDR image and the SDR image in the video can be processed simultaneously. Figure 4 In the embodiment, taking the first SDR video 2501, the first SDR image 2502 and the second SDR image 2503 stored in the electronic device as an example, the user can long press the first SDR video 2501 to be converted to select the first SDR video 2501 and enter the selection interface 1500. In the selection interface 1500, there is a check mark in the selection box of the first SDR video 2501 selected by the user, indicating that it has been selected. Then the user can click the first SDR image 2502 to select the first SDR image 2502. After the user selects the first SDR video 2501 and the first SDR image 2502, there are check marks in the selection boxes in the upper right corners of the first SDR video 2501 and the first SDR image 2502. On this basis, the user can convert the selected first SDR video 2501 and the first SDR image 2502 into corresponding HDR videos and HDR images by clicking the conversion button 1501 at the bottom of the selection interface 1500, so as to realize batch conversion of SDR images and improve the efficiency of obtaining HDR images.

[0053] The above describes the application scenario of generating HDR images provided by the embodiments of the present application. Figure 5 The flowchart shown introduces the image processing method provided in the embodiment of the present application.

[0054] The SDR image in this application can be an image captured or stored locally, or an image in a video. In order to distinguish an image from an image in a video, an image is referred to as a static image, and an image in a video is referred to as a dynamic image. Figure 5As shown, an HDR image can be obtained by executing S101-S102.

[0055] S101. Obtain an SDR image.

[0056] In a possible implementation, the electronic device may acquire an SDR image in response to a user's shooting operation. Exemplarily, after the user performs a shooting operation, the electronic device may generate an SDR image based on the collected data in response to the user's shooting operation, so that the SDR image may be subsequently converted into an HDR image based on the color mapping relationship between the SDR image and the pixel points in the HDR image.

[0057] Since SDR images are in a standard format, less raw data is needed to generate SDR images than to directly generate HDR images, and the hardware configuration requirements for electronic devices are lower. It is compatible and applicable to various electronic devices. Therefore, generating SDR images before users take photos can effectively reduce the hardware requirements for electronic devices, improve the applicability of electronic devices, and reduce costs.

[0058] In another possible implementation, the electronic device can obtain an SDR image in response to a user's selection operation of an SDR image in a memory. Due to the compatibility of SDR images, a large number of images currently exist in SDR format. Based on the method provided in the embodiment of the present application, the SDR image stored in the electronic device can be converted into an HDR image. This can improve the applicability of the electronic device, reduce the hardware cost, and improve the display effect of the image.

[0059] Taking the example of storing a first SDR image, a second SDR image 2502 and a third SDR image in an electronic device, in one example, when a user selects the first SDR image, the electronic device can obtain the first SDR image in response to the user's selection operation on the first SDR image, so as to convert the first SDR image into an image in HDR format.

[0060] In another example, when the user selects the first SDR image and the second SDR image, the electronic device may acquire the first SDR image and the second SDR image in response to the user's selection operation on the first SDR image and the second SDR image so as to convert the first SDR image and the second SDR image into images in HDR format.

[0061] It should be noted that in the embodiments of the present application, there is no specific limitation on the number of SDR images acquired at one time. The electronic device can acquire one SDR image at a time, or acquire multiple SDR images in batches. When the electronic device acquires multiple SDR images at a time, it is convenient to subsequently convert the multiple SDR images into HDR images in batches, thereby improving the efficiency of generating HDR images.

[0062] S102. Based on the color mapping relationship between the SDR image and the pixel points in the HDR image, color mapping is performed on each pixel point in the SDR image to obtain color information of the HDR image.

[0063] There are differences in the color gamut between HDR images and SDR images. For example, SDR usually uses the BT.709 standard, while HDR usually uses the wider BT.2020 color gamut. The BT.709 standard is an international standard for high-definition television, which has a relatively small color gamut, the same as the color space of the universal color standard (standard Red Green Blue, sRGB) used for Internet media; the BT.2020 color gamut is designed for ultra-high-definition television and wide color gamut applications, and its color gamut range far exceeds sRGB.

[0064] In an embodiment of the present application, according to the color mapping relationship between the SDR image and the HDR image, each pixel in the SDR image can be color mapped to obtain the color information of the HDR image corresponding to the SDR image. Among them, color mapping each pixel in the SDR image may include color mapping and color gamut mapping for each pixel in the SDR image, and the color information may include color information and color gamut information of the image. At the same time, the method provided by the present application does not need to collect data according to the collection requirements of the HDR image, which reduces the hardware requirements for electronic equipment, thereby effectively reducing costs.

[0065] In some embodiments, based on the color mapping relationship between the pixels in the SDR image and the HDR image, color mapping is performed on each pixel in the SDR image, and one implementation method for obtaining the color information of the HDR image may be:

[0066] The SDR image is input into a color mapping model based on a deep learning model. The color mapping model can realize one-to-one color mapping for each pixel through a convolutional neural network. That is, based on the color mapping relationship between the pixels in the SDR image and the HDR image, the color and color gamut of each pixel in the SDR image are mapped one by one to the color and color gamut of each pixel in the HDR image, and the HDR image corresponding to the SDR image is output.

[0067] In some embodiments, the color mapping model can be trained using a large number of SDR images and real HDR images, wherein the real HDR image can be generated by collecting original data according to the acquisition requirements of the HDR image and processing the original data corresponding to the HDR image.

[0068] A large number of SDR images are used as the input of the model, and real HDR images are used as training targets. During the training process, the model parameters are continuously adjusted so that the similarity between the generated HDR images and the real HDR images becomes higher and higher, and the trained color mapping model is obtained. Exemplarily, the model parameters can be adjusted in the process of training the color mapping model based on the modified parameter prediction model.

[0069] In the process of training the model based on a large amount of training data, the color mapping model can learn the color mapping relationship between the pixels in the SDR image and the real HDR image. On this basis, the color mapping model can convert the acquired SDR image into an HDR image based on the learned color mapping relationship between the pixels in the SDR image and the real HDR image, and has high generalization. After the color mapping model is trained, the parameters learned by the model can characterize the color mapping relationship of each pixel in the HDR image corresponding to the SDR image.

[0070] In a possible implementation, an image pyramid generated based on the SDR image may be input into the color mapping model to obtain an HDR image corresponding to the SDR image.

[0071] Among them, the image pyramid is a kind of multi-scale expression of images. The image pyramid of an SDR image is a set of images arranged in a pyramid shape with resolutions gradually decreasing from bottom to top. The images in the image pyramid of an SDR image all come from the same original image and can be regarded as a set of image resolutions derived from the SDR image. The bottom of the image pyramid is a high-resolution representation of the image to be processed, while the top is a low-resolution approximation. When moving to the upper layer of the image pyramid, the size and resolution of the image decrease.

[0072] The original SDR image is used as the image to be processed, and the SDR image is subjected to multi-resolution processing to obtain an image pyramid of the SDR image. The image pyramid of the SDR image is input into the color mapping model, and the color mapping model can obtain the color information of the HDR image corresponding to the SDR image based on the learned color mapping relationship. At the same time, the image pyramid of the SDR image can provide more scale references for obtaining the color information of the HDR image, which is convenient for the subsequent conversion of the SDR image into an HDR image.

[0073] For example, the color mapping model may adopt a convolutional neural network (CNN) model, wherein the convolution kernel size may be 1×1 and the step length may be 1, thereby realizing one-by-one mapping of image pixels.

[0074] In another possible implementation, the color mapping model may include a basic model and a modified parameter prediction model. Figure 6 As shown, after obtaining the image pyramid of the SDR image, the image pyramid of the SDR image can be input into the basic model, and the SDR image to be processed can be input into the correction parameter prediction model. The basic model is used to convert the SDR image into an initial HDR image; the correction parameter prediction model is used to adjust the color information of the initial HDR image.

[0075] Exemplarily, the basic model is based on the image pyramid of the SDR image, takes color information of more sizes as a reference, and based on the color mapping relationship between the pixels in the SDR image and the HDR image, corresponds the color and color gamut of each pixel in the SDR image to the color and color gamut of each pixel in the HDR image, and outputs the initial HDR image. Among them, the image pyramid of the SDR image can be used as additional prior information to provide a reference for the basic model to output the initial HDR image. The pixel values ​​of the initial HDR image can be used to indicate the characteristics of the initial HDR image converted from the SDR image, that is, to indicate the color information of each pixel in the initial HDR image.

[0076] The correction parameter prediction model can use the SDR image to be processed as prior information to generate correction parameters. The correction parameters are used to fine-tune the color information of the initial HDR image output by the basic model to obtain the color information of the HDR image corresponding to the SDR image.

[0077] In the embodiment of the present application, after obtaining the color information of the initial HDR image and the correction parameters, the color information of the initial HDR image output by the basic model can be adjusted based on the correction parameters to complete the color mapping and obtain the color information of the HDR image corresponding to the SDR image. In the process of adjusting the color information of the initial HDR image output by the basic model through the correction parameters, some detail information in the SDR image can be supplemented to the initial HDR image, thereby further improving the display effect of the generated HDR image.

[0078] For example, the basic model in the embodiment of the present application can adopt a convolutional neural network (CNN) model; the modified parameter prediction model can adopt a multilayer perceptron neural network (MLP) model, or a model with multiple convolutional layer structures. The convolution kernel size of the basic model can be 1×1, with a step size of 1, so as to achieve a one-to-one mapping of image pixels.

[0079] The following combination Figure 7 , the implementation process of processing the SDR image to obtain the HDR image by constructing a color mapping model composed of a basic model and a modified parameter prediction model is explained.

[0080] Figure 7 In the example, the linear MLP model is used as the modified parameter prediction model, and the CNN model is used as the basic model. Exemplarily, the image pyramid of the SDR image is composed of a first SDR image, a first sub-SDR image, and a second sub-SDR image, wherein the first SDR image is the original SDR image, that is, the SDR image to be processed. The first sub-SDR image and the second sub-SDR image are images obtained by multi-resolution processing of the first SDR image, such as Figure 7 In the example, the first SDR image is the original SDR image. The layers of images are likened to a pyramid. The higher the level of the image pyramid, the smaller the image and the lower the resolution. For example, the first SDR image can be reduced by 2 times to obtain the first sub-SDR image; the first SDR image can be reduced by 4 times to obtain the third sub-SDR image.

[0081] It should be noted that in the embodiments of the present application, there is no specific limitation on the reduction factor of the first SDR image. It can be understood that the above-mentioned reduction of the first SDR image by 2 times and 4 times is only an example.

[0082] In the embodiment of the present application, the first SDR image can be used as prior information, and the first sub-SDR image and the second sub-SDR image can be used as additional prior information to generate a more accurate correction parameter. Fine-tuning the initial HDR image based on the more accurate correction parameter can effectively improve the display effect of the corrected HDR image.

[0083] The image pyramid of the SDR image is input into the CNN model, and the CNN model can output the color information of the initial HDR image. The color information of the initial HDR image may include the pixel value of each pixel in the uncorrected HDR image, that is, the pixel value of each pixel in the initial HDR image. The first SDR image is input into the linear MLP model as prior information to obtain the correction parameter, which may include a first parameter and a second parameter.

[0084] Based on the first parameter and the second parameter, the pixel value of each pixel in the color information of the initial HDR image output by the CNN model can be corrected to obtain the pixel value of each pixel in the color information of the HDR image corresponding to the SDR image, as shown in formula (1):

[0085] M'=a×M+b(1)

[0086] Among them, M is the pixel value of each pixel in the color information of the initial HDR image output by the CNN model; M' is the pixel value of each pixel in the color information of the HDR image corresponding to the SDR image; a is the first parameter; b is the second parameter.

[0087] S103: Generate an HDR image based on the color information of the HDR image.

[0088] Exemplarily, an HDR image corresponding to an SDR image can be generated based on a color mapping model. The color depth of the SDR image can be 8 bits. It is understandable that the higher the color depth, the more colors available. Inputting an 8-bit SDR image into the color mapping model can obtain an HDR image with a color depth of 10-16 bits.

[0089] In an embodiment of the present application, when training a color mapping model, a large number of SDR images can be used as input, and real HDR images can be used as training targets for the color mapping model. The model parameters can be adjusted by modifying the parameter prediction model. The loss function can use a mean square error loss (L2loss) function. When the model converges, the trained color mapping model can be obtained.

[0090] The modified parameter prediction model in the embodiment of the present application can adjust the model parameters and optimize the model parameters during the model training stage to obtain a trained color mapping model; after obtaining the initial HDR image corresponding to the SDR image, the color information of the initial HDR image can be adjusted to improve the display effect of the HDR image.

[0091] This application adopts a method of obtaining an SDR image and converting the SDR image into an HDR image. It is not necessary to collect original data according to the acquisition requirements of the HDR image, which reduces the hardware requirements for electronic equipment and can effectively reduce costs. At the same time, a deep learning model trained based on a large amount of training data is used to realize the conversion of SDR images to HDR images according to the color mapping relationship between the pixels of the SDR image and the HDR image. It has high generalization and can improve the display effect of the image while saving costs.

[0092] In addition, the process of converting SDR images into HDR images based on the deep learning model in this application does not require manual processing of the images, effectively improving the image generation efficiency.

[0093] At the same time, since pixels are nonlinear, they correspond to nonlinear domains; brightness is linear, it corresponds to linear domains. Compared with the method of processing brightness in the linear domain during image processing, the deep learning model used in this application can directly obtain HDR images in the pixel domain, which is equivalent to directly realizing the conversion between SDR images and HDR images in the nonlinear domain. HDR images can be obtained with fewer steps, which improves the generation efficiency of HDR images.

[0094] It should be noted that through Figure 5 In step S103 of the image processing method shown, after generating an HDR image based on the color information of the HDR image, the HDR image may have problems such as missing image details, faults and artifacts.

[0095] Based on this, in a possible implementation, after generating an HDR image based on the color information of the HDR image, the HDR image obtained by color mapping can also be reconstructed to restore image details, reduce faults and artifacts, and improve the display effect of the HDR image. Figure 8 As shown, the SDR image is input into the color mapping model to obtain the HDR image corresponding to the SDR image, and the HDR image output by the color mapping model is input into the detail enhancement model to obtain the enhanced HDR image.

[0096] In some embodiments, one implementation of reconstructing the HDR image obtained by color mapping to obtain an enhanced HDR image may be:

[0097] The HDR image obtained by the color mapping model of the SDR image is input into the detail enhancement (Detail Smooth) model to obtain an enhanced HDR image. The detail enhancement model is used to restore the details in the HDR image and reduce problems such as faults and artifacts.

[0098] For example, in the process of training the detail enhancement model, the output result of the color mapping model can be used as input, that is, the image pyramid of the HDR image output by the color mapping model is used as input, and the real HDR image is used as the training target. During training, the mean square error loss function can be used to optimize the model until the model converges to obtain the trained detail enhancement model.

[0099] The image pyramid of the HDR image can be obtained by performing multi-resolution processing on the HDR image output by the color mapping model. The image pyramid of the HDR image can provide more scale reference information for the detail enhancement model and improve the quality of the output image.

[0100] Based on the HDR image, the embodiment of the present application can effectively improve the display effect of the image by performing detail enhancement on it. The detail enhancement model can adopt a codec structure, such as Fig. 9 As shown, this figure is a schematic diagram of the model structure of a detail enhancement model provided in an embodiment of the present application. The HDR image output by the color mapping model is input to an encoder (Encoder), the encoder encodes the HDR image, and the encoded HDR image is input to a decoder (Decoder).

[0101] In one example, after obtaining the HDR image output by the color mapping model, multi-resolution processing can be performed on the HDR image to obtain an image pyramid of the HDR image, and the image pyramid of the HDR image is input into an encoder, the encoder encodes the HDR image, and the encoded HDR image is input into a decoder.

[0102] The decoder uses the encoded HDR image as input, which can restore the details in the HDR image, reduce problems such as faults and artifacts, and obtain an enhanced HDR image. The brightness information of the SDR image can be used as supplementary information for the encoder-decoder model to perform residual connection processing on the enhanced HDR image to obtain the processed HDR image.

[0103] In one possible implementation, Fig.10 As shown, this figure is a schematic diagram of detail enhancement of an HDR image provided by an embodiment of the present application. The HDR image output by the color mapping model is subjected to multi-resolution processing, and the image pyramid of the obtained HDR image is used as the input of the detail enhancement model. A multi-scale HDR image can be input, that is, after the image pyramid of the HDR image is input to the detail enhancement model, an enhanced HDR image can be output.

[0104] On this basis, the brightness information of the original SDR image, that is, the brightness map of the SDR image to be processed, can be used to perform residual connection processing on the enhanced HDR image to obtain a processed HDR image.

[0105] Exemplarily, the original SDR brightness image and the enhanced HDR image can be superimposed by using a residual connection, which is equivalent to adding some detail information of the SDR image to the enhanced HDR image to obtain a processed HDR image to improve the display effect of the HDR image.

[0106] The principle of residual connection is to add the input and output of the network so that the network can better learn the residual information and improve the performance and training effect of the network. The core idea is to add the input and output of the network directly so that the network can directly learn the residual information.

[0107] The brightness information of the SDR image is obtained in the form of a brightness map. The brightness map is calculated by converting the color space of the red, green and blue (RGB) image to the XYZ color space, obtaining the image of the Y channel, and using it as the brightness map of the SDR image. XYZ is a color space encoding method, and "Y" can represent brightness (Luminance). The formula for calculating Y for RGB images in different color gamuts is different. For example, the SDR image in the BT.709 color gamut can be obtained by formula (2):

[0108] Y=0.2126×R+0.7152×G+0.0722×B(2)

[0109] The SDR image under the DCI-P3 color gamut can be obtained by formula (3):

[0110] Y=0.2290×R+0.6917×G+0.0793×B(3)

[0111] Among them, R, G, and B in formula (2) represent the RGB values ​​of the pixel points in the BT.709 color gamut respectively; R, G, and B in formula (3) represent the RGB values ​​of the pixel points in the DCI-P3 color gamut respectively.

[0112] In an embodiment of the present application, by adding the brightness map of the original SDR image to the enhanced HDR image, the network can directly learn the residual information between the brightness map of the original SDR image and the enhanced HDR image. The output adjusted HDR image contains more detail information than the enhanced HDR image, thereby improving the image display effect.

[0113] Unlike SDR images that are captured or locally stored, when SDR images are images in a video, before converting the SDR images to HDR images, it is necessary to obtain multiple frames of SDR images and audio data by decomposing the SDR video. After converting the multiple frames of SDR images to corresponding HDR images, the multiple frames of HDR images are merged with the audio data to obtain an HDR video.

[0114] like Fig.11 As shown, this figure is a schematic diagram of SDR video processing provided by an embodiment of the present application. Among them, the SDR video can be a video composed of multiple frames of SDR images and audio data. After obtaining the SDR video, the SDR video can be decomposed and processed to obtain multiple frames of SDR images and audio data. Among them, the SDR video can be obtained by shooting or stored locally.

[0115] For each SDR image in the multiple SDR images, the multiple SDR images can be converted into corresponding HDR images based on the above-mentioned method for processing the captured or locally stored images. After converting the multiple SDR images into corresponding HDR images, multiple HDR images are obtained. The multiple HDR images are synthesized with the audio data to obtain an HDR video.

[0116] In a possible implementation, in order to improve the display effect of the HDR image, based on the above-mentioned method of performing detail enhancement processing on the HDR image corresponding to the SDR image, a multi-frame enhanced HDR image or a multi-frame processed HDR image can be obtained. The multi-frame enhanced HDR image or the multi-frame processed HDR image is synthesized with the audio data to obtain an HDR video.

[0117] In a possible implementation, perceptual quantizer (PQ) encoding or hybrid log-gamma (HLG) encoding can be used to process multiple HDR images to obtain processed multiple HDR images. The processed multiple HDR images are fused with audio data to obtain HDR video, achieving a high dynamic display effect.

[0118] Among them, PQ coding can allow the brightness level of the image to reach 10000cd / m 2 ;HLG encoding is a high dynamic range video encoding technology that enables users to see richer and more realistic colors and details by adjusting image brightness and contrast.

[0119] In summary, the present application provides an image processing method, which can convert an SDR image into an HDR image by acquiring a compatible SDR image and using a deep learning model trained based on a large amount of training data. The deep learning model can perform color mapping on each pixel in the SDR image according to the color mapping relationship between the pixels in the SDR image and the HDR image, map the color and color gamut of the SDR image to the color and color gamut of the HDR image, and obtain an HDR image with a higher dynamic range.

[0120] In this application, by converting compatible SDR images into HDR images, there is no need to collect original data according to the acquisition requirements of HDR images, which reduces the hardware requirements for electronic devices, can save costs while improving the image display effect. At the same time, using a deep learning model trained based on a large amount of training data, the conversion of SDR images to HDR images can be realized according to the color mapping relationship between the pixels of the SDR image and the HDR image, which has high generalization, can improve the image display effect, and does not require manual image processing, effectively improving the image generation efficiency.

[0121] The embodiments of the present application can be applied to electronic devices. In some embodiments, the electronic devices can be mobile phones, tablet computers, desktops, laptops, notebook computers, ultra-mobile personal computers (UMPC), handheld computers, netbooks, personal digital assistants (PDA), wearable electronic devices, smart watches and other devices. The present application does not impose any special restrictions on the specific forms of the above electronic devices. In this embodiment, the structure of the electronic device can be as follows: Fig.12 As shown, Fig.12 This is an example diagram of the composition of an electronic device provided in an embodiment of the present application.

[0122] like Fig.12 As shown, the electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a sensor module 130, a camera 140, a display screen 141, and the like.

[0123] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0124] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0125] For example, in an embodiment of the present application, the processor 110 may obtain an SDR image, and obtain color information of each pixel in the HDR image corresponding to the SDR image according to the color mapping relationship between the SDR image and the HDR image. Color mapping is performed on each pixel in the SDR image according to the color information to generate an HDR image corresponding to the SDR image.

[0126] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used, such as a photo taking instruction, a video recording instruction, etc. The electronic device may generate a static HDR picture based on the photo taking instruction, and may generate a dynamic HDR video based on the video recording instruction.

[0127] In one example, the electronic device may store the generated HDR image in a memory, and in response to a user's operation of viewing the HDR image, the electronic device may call the corresponding HDR image from the memory and display it to the user on a user interface.

[0128] In another example, an SDR image may be stored in the memory, and the electronic device may convert the SDR image into a corresponding HDR image in response to a user's selection operation on the SDR image, and store the generated HDR image in the memory.

[0129] The electronic device implements the display function through a GPU, a display screen 141, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 141 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0130] The display screen 141 is used to display images, videos, etc. The display screen 141 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oled, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device may include 1 or N display screens 141, where N is a positive integer greater than 1.

[0131] A series of graphical user interfaces (GUI) can be displayed on the display screen 141 of the electronic device. For example, in the embodiment of the present application, the image interfaces that can be displayed on the display screen 141 include an initial user interface, a photo taking interface, and an album interface, etc. The user presses a photo taking button or a video recording button on the photo taking interface to take a photo or record a video to obtain an HDR image.

[0132] The electronic device can realize the shooting function through the ISP, the camera 140, the video codec, the GPU, the display screen 141 and the application processor.

[0133] The camera 140 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. In some embodiments, the electronic device may include 1 or N cameras 140, where N is a positive integer greater than 1.

[0134] Video codecs are used to compress or decompress digital videos. Electronic devices can support one or more video codecs. In this way, electronic devices can play or record videos in multiple coding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0135] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, in an embodiment of the present application, the generated HDR image and other files are saved in the external memory card.

[0136] The internal memory 121 may be used to store computer executable program codes, which include instructions. The processor 110 executes the instructions stored in the internal memory 121 to perform various functional applications and data processing of the electronic device.

[0137] In addition, an operating system is running on the above components, such as the iOS operating system developed by Apple, the Android open source operating system developed by Google, and the Windows operating system developed by Microsoft. Application programs can be installed and run on the operating system.

[0138] The operating system of the electronic device may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device.

[0139] Fig.13 This is an example diagram of the software structure of an electronic device provided in an embodiment of the present application.

[0140] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0141] The application layer can include a series of application packages. Fig.13 As shown, the application package may include applications such as camera and photo album.

[0142] The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Fig.13 As shown, the application framework layer may include an image conversion module, etc.

[0143] The image conversion module may include a color mapping module and a detail enhancement module, wherein the color mapping module is used to perform color mapping on each pixel in the SDR image based on the color mapping relationship between the pixels in the SDR image and the HDR image, and map the color and color gamut of the SDR image to the color and color gamut of the HDR image to obtain the HDR image; the detail enhancement module is used to perform detail enhancement processing on the HDR image output by the color mapping module to restore the details in the HDR image and obtain the enhanced HDR image.

[0144] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0145] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0146] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0147] The system library may include multiple functional modules, such as media libraries, 3D graphics processing libraries (such as OpenGL ES), 2D graphics engines (such as SGL), etc.

[0148] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0149] A 2D graphics engine is a drawing engine for 2D drawings.

[0150] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, and sensor driver.

[0151] It should be noted that although the embodiments of the present application are described using the Android system as an example, its basic principles are also applicable to electronic devices based on operating systems such as iOS and Windows.

[0152] This embodiment further provides a computer-readable storage medium, which includes instructions. When the instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.

[0153] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: include: Get standard dynamic range SDR images; The SDR image is input into a color mapping model, and the color mapping model performs color mapping on each pixel in the SDR image according to a color mapping relationship between the SDR image and a high dynamic range imaging HDR image to obtain color information of the HDR image; the color mapping model is trained with the SDR image as input and the real HDR image as a target; Based on the color information of the HDR image, an HDR image corresponding to the SDR image is generated.

2. The method according to claim 1, characterized in that: The step of inputting the SDR image into a color mapping model comprises: An image pyramid generated based on the SDR image is input into the color mapping model.

3. The method according to claim 1, characterized in that The color mapping model includes a basic model and a modified parameter prediction model. The SDR image is input into the color mapping model, and the color mapping model performs color mapping on each pixel in the SDR image according to the color mapping relationship between the SDR image and the high dynamic range imaging HDR image to obtain color information of the HDR image, including: The SDR image is input into the basic model, and the basic model performs color mapping on each pixel in the SDR image according to the color mapping relationship between the SDR image and the high dynamic range imaging HDR image to obtain color information of the initial HDR image; The color information of the initial HDR image is adjusted based on correction parameters to generate color information of the HDR image corresponding to the SDR image, and the correction parameters are obtained by inputting the SDR image into the correction parameter prediction model to perform correction parameter prediction.

4. The method according to claim 3, characterized in that: The correction parameter includes a first parameter and a second parameter, and adjusting the color information of the initial HDR image based on the correction parameter to generate the color information of the HDR image corresponding to the SDR image includes: The pixel value of each pixel point in the color information of the initial HDR image is obtained by multiplying the product of the pixel value of each pixel point in the color information of the HDR image corresponding to the SDR image by the first parameter and adding the second parameter.

5. The method according to any one of claims 1 to 4, characterized in that: After generating the HDR image corresponding to the SDR image based on the color information of the HDR image, the method further includes: The HDR image corresponding to the SDR image is input into a detail enhancement model, and the detail enhancement model performs detail enhancement processing on the HDR image corresponding to the SDR image to obtain an enhanced HDR image. The detail enhancement model is trained using the HDR image output by the color mapping model as input and a real HDR image as a target.

6. The method according to claim 5, characterized in that After inputting the HDR image corresponding to the SDR image into a detail enhancement model, and performing detail enhancement processing on the HDR image corresponding to the SDR image by the detail enhancement model to obtain an enhanced HDR image, the method further includes: Residual connection processing is performed on the brightness information of the SDR image and the enhanced HDR image to obtain a processed HDR image.

7. The method according to claim 1, characterized in that The SDR image is an image in an SDR video, and obtaining a standard dynamic range SDR image includes: Get SDR video; Decomposing the SDR video to obtain multiple frames of SDR images and audio data constituting the SDR video; The step of inputting the SDR image into a color mapping model, and performing color mapping on each pixel in the SDR image according to a color mapping relationship between the SDR image and a high dynamic range imaging HDR image by the color mapping model to obtain color information of the HDR image includes: For each SDR image in the multiple SDR images, the SDR image is input into a color mapping model, and the color mapping model performs color mapping on each pixel in the SDR image according to a color mapping relationship between the SDR image and a high dynamic range imaging HDR image to obtain color information of the HDR image, thereby obtaining color information of the multiple HDR images corresponding to the multiple SDR images; The step of generating an HDR image corresponding to the SDR image based on the color information of the HDR image includes: Based on the color information of the multiple frames of HDR images, generate HDR images corresponding to the multiple frames of SDR images to obtain the multiple frames of HDR images; After generating HDR images corresponding to the multiple SDR images based on the color information of the multiple HDR images, and obtaining the multiple HDR images, the method further comprises: The multiple frames of HDR images are fused with the audio data to obtain an HDR video corresponding to the SDR video.

8. The method according to claim 1, characterized in that The step of acquiring a standard dynamic range SDR image comprises: In response to a user's photographing operation, a standard dynamic range (SDR) image is acquired.

9. The method according to claim 1, characterized in that: The step of acquiring a standard dynamic range SDR image comprises: In response to a user's selection operation of a standard dynamic range (SDR) image stored in the electronic device, a standard dynamic range (SDR) image is acquired.

10. An electronic device, characterized in that: The electronic device comprises: a processor and a memory; The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the steps of an image processing method as described in any one of claims 1 to 9 according to the instructions in the program code.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of an image processing method according to any one of claims 1 to 9 are implemented.

Images