Image processing model training method and high dynamic range image generation method

Abstract

A kind of image processing model training method, comprising: input low dynamic range image first initial image processing model, first high dynamic range image is generated by carrying out high dynamic range reconstruction processing to the low dynamic range image;The low dynamic range image is input second initial image processing model, generate first coefficient;According to the first high dynamic range image with the first coefficient, generate second high dynamic range image;According to the second high dynamic range image and the data pair of real high dynamic range image, generate loss function;Wherein, the real high dynamic range image is the real high dynamic range image corresponding to the low dynamic range image;Using the loss function, the first initial image processing model and the second initial image processing model are trained.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to an image processing model training method, a high dynamic range image generation method, an electronic device and a computer readable storage medium. BACKGROUND

[0002] High Dynamic Range Imaging (HDRI) technology is an image representation method used to realize a larger exposure range than ordinary digital images. High Dynamic Range (HDR) images can provide a larger luminance range and more light and dark details than ordinary digital images, which makes high dynamic range images be able to present luminance change information closer to real scenes. At present, there is a technology of converting a Low Dynamic Range (LDR) image into a high dynamic range image to restore luminance information close to real scenes. SUMMARY

[0003] In one aspect, an image processing model training method is provided, including: inputting a low dynamic range image into a first initial image processing model, performing high dynamic range reconstruction processing on the low dynamic range image to generate a first high dynamic range image. Inputting the low dynamic range image into a second initial image processing model to generate a first coefficient. Generating a second high dynamic range image according to the first high dynamic range image and the first coefficient. Generating a loss function according to a data pair of the second high dynamic range image and a real high dynamic range image, wherein the real high dynamic range image is a real high dynamic range image corresponding to the low dynamic range image. Training the first initial image processing model and the second initial image processing model using the loss function.

[0004] In some embodiments, the first coefficient is a weight coefficient generated by the second initial image processing model performing structural feature analysis on the low dynamic range image.

[0005] In some embodiments, the weight coefficient is a 1x1x3 matrix.

[0006] In some embodiments, the low dynamic range image, the first high dynamic range image and the second high dynamic range image further include a fourth channel in addition to the RGB three channels of the image; and the loss function is further related to a value of the fourth channel.

[0007] In some embodiments, the fourth channel is a pixel maximum value of the image. The loss function comprises an L1 loss and a tone mapping loss of a pair of RGB data of the second high dynamic range image and the real high dynamic range image, and an L1 loss of a pair of pixel maximum value data of the second high dynamic range image and the real high dynamic range image.

[0008] In some embodiments, the generating a second high dynamic range image according to the first high dynamic range image and the first coefficient comprises:

[0009] multiplying the first high dynamic range image and the first coefficient to generate the second high dynamic range image.

[0010] In some embodiments, before inputting the low dynamic range image into the second initial image processing model, further comprising: performing down-sampling processing on the low dynamic range image to generate a low dynamic range image subjected to down-sampling processing.

[0011] In some embodiments, the number of layers of the second initial image processing model is less than the number of layers of the first initial image processing model.

[0012] In some embodiments, in the process of training the first initial image processing model and the second initial image processing model using the loss function, in the case that the loss function does not converge, updating the parameters in the first initial image processing model and the second initial image processing model, the network optimizer adopts an Adam optimizer, and the learning rate is 1e-4.

[0013] In another aspect, a high dynamic range image generation method is provided, comprising: inputting a to-be-processed image into a first target image processing model to perform high dynamic range reconstruction processing on the to-be-processed image to generate a first processed image; the to-be-processed image is a low dynamic range image. Inputting the to-be-processed image into a second target image processing model to generate a second coefficient; wherein the first target image processing model and the second target image processing model are trained by the image processing model training method according to any one of the above. Generating a second processed image according to the first processed image and the second coefficient.

[0014] In some embodiments, the second coefficient is a weight coefficient generated by the second target image processing model performing structural feature analysis on the to-be-processed image.

[0015] In some embodiments, the generating a second processed image according to the first processed image and the second coefficient comprises: multiplying the first processed image and the second coefficient to generate the second processed image.

[0016] In some embodiments, before inputting the to-be-processed image into the second target image processing model, the method further includes: performing down-sampling processing on the to-be-processed image to generate a down-sampled to-be-processed image.

[0017] In some embodiments, the second target image processing model has a smaller number of layers than the first target image processing model.

[0018] In another aspect, an electronic device is provided, including a processor, a memory, and a computer program stored on the memory and executable on the processor. The processor is configured to perform inputting a low dynamic range image into a first initial image processing model, performing high dynamic range reconstruction processing on the low dynamic range image to generate a first high dynamic range image; the processor is further configured to perform inputting the low dynamic range image into a second initial image processing model to generate a first coefficient; the processor is further configured to perform generating a second high dynamic range image according to the first high dynamic range image and the first coefficient.

[0019] The processor is further configured to perform generating a loss function according to a pair of the second high dynamic range image and a real high dynamic range image; wherein the real high dynamic range image is a real high dynamic range image corresponding to the low dynamic range image; the processor is further configured to perform training the first initial image processing model and the second initial image processing model using the loss function; and the memory is configured to store data of the first high dynamic range image, the first coefficient, and data of the second high dynamic range image.

[0020] In some embodiments, the processor is further configured to perform multiplying the first high dynamic range image by the first coefficient to generate the second high dynamic range image. The processor is further configured to perform, before inputting the low dynamic range image into the second initial image processing model, performing down-sampling processing on the low dynamic range image to generate a down-sampled low dynamic range image.

[0021] In some embodiments, the processor is further configured to perform inputting the to-be-processed image into a first target image processing model, performing high dynamic range reconstruction processing on the to-be-processed image to generate a first processed image; the to-be-processed image is a low dynamic range image; the processor is further configured to perform inputting the to-be-processed image into a second target image processing model to generate a second coefficient; wherein the first target image processing model and the second target image processing model are trained by the image processing model training method according to any one of the above; the processor is further configured to perform generating a second processed image according to the first processed image and the second coefficient. The memory is configured to store the first processed image, the second coefficient and the second processed image.

[0022] In some embodiments, the processor is further configured to perform multiplying the first processed image and the second coefficient to generate the second processed image. The processor is further configured to perform, before inputting the to-be-processed image into the second target image processing model, performing down-sampling processing on the to-be-processed image to generate a to-be-processed image after down-sampling processing.

[0023] In another aspect, a computer readable storage medium is provided, the computer readable storage medium storing computer program instructions, the computer program instructions, when executed on a processor, causing the processor to perform one or more steps of the image processing model training method according to some embodiments described above, and / or one or more steps of the image processing method according to some embodiments described above.

[0024] In another aspect, a computer program product is provided. The computer program product comprises computer program instructions, the computer program instructions, when executed on a computer, causing the computer to perform one or more steps of the image processing model training method according to some embodiments described above, and / or one or more steps of the image processing method according to some embodiments described above.

[0025] In another aspect, a computer program is provided. The computer program, when executed on a computer, causes the computer to perform one or more steps of the image processing model training method according to some embodiments described above, and / or one or more steps of the image processing method according to some embodiments described above. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the present disclosure, the drawings needed to be used in some embodiments of the present disclosure will be briefly introduced as follows. Obviously, the drawings in the following description are only some drawings of the embodiments of the present disclosure, and other drawings can also be obtained by those skilled in the art according to these drawings. In addition, the drawings in the following description can be regarded as schematic diagrams, and are not limited to the actual size, actual process, actual time sequence, etc. of the products, methods, signals, etc. involved in the embodiments of the present disclosure.

[0027] Figure 1 A flowchart of a high dynamic range image training method according to some embodiments;

[0028] Figure 2 Another flowchart of a high dynamic range image training method according to some embodiments;

[0029] Figure 3 A step diagram of a high dynamic range image training method according to some embodiments;

[0030] Figure 4 A flowchart of a high dynamic range image generation method according to some embodiments;

[0031] Figure 5 Another flowchart of a high dynamic range image generation method according to some embodiments;

[0032] Figure 6 A step diagram of a high dynamic range image generation method according to some embodiments;

[0033] Figure 7 A structural diagram of an electronic device according to some embodiments. DETAILED DESCRIPTION

[0034] The technical solutions in some embodiments of the present disclosure will be described clearly and completely in combination with the drawings. Obviously, the described embodiments are only some of the embodiments of the present disclosure, but not all the embodiments. Based on the embodiments provided in the present disclosure, all other embodiments obtained by those skilled in the art belong to the scope of protection of the present disclosure.

[0035] Unless the context clearly requires otherwise, throughout the description and the claims, the words "comprise," "comprising," and the like are to be construed in an open, inclusive sense, as opposed to a closed or exclusive sense, so that, for example, the term "comprising" will be understood to mean "including but not limited to." As used in the description of the disclosure and the appended claims, the terms "a," "an" and "the" are used generically and not in their plural sense unless the context clearly indicates otherwise, and should not be construed as the exclusive singular, that is, not including plural instances, but rather the singular instances as well. The terms "another," "an," and "a" as used in the description of the disclosure and the appended claims are each to be construed as "at least one" and "one or more." As used in the description of the disclosure and the appended claims, the term "or" is used in the inclusive sense, that is, the term "or" is used to mean "and / or" unless the context clearly indicates otherwise.

[0036] Hereinafter, the terms "first" and "second" are used only for the purpose of description, and should not be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Thus, the features defined with "first" and "second" can explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, the meaning of "a plurality of" is two or more, unless otherwise specified.

[0037] In describing some embodiments, "coupled" and "connected," and variations thereof, can be used. For example, the term "connected" can be used to mean that two or more elements are in direct physical or electrical contact with each other. As another example, the term "coupled" can be used to mean that two or more elements are in direct physical or electrical contact with each other. However, the terms "coupled" or "communicatively coupled" can also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other. The embodiments disclosed herein are not necessarily limited to the context.

[0038] "A and / or B" includes the following three combinations: only A, only B, and a combination of A and B.

[0039] As used herein, the term "if' is, optionally, interpreted as meaning "when" or "while" or "in response to a determination" or "in response to a detection" depending on the context. Similarly, the phrase "if it is determined" or "if [stated condition or event] is detected" is, optionally, interpreted as meaning "upon a determination" or "in response to a determination" or "upon a detection" or "in response to a detection" of [stated condition or event], depending on the context.

[0040] The use of "adapted to" or "configured to" herein means open and inclusive language that does not exclude devices adapted to or configured to perform additional tasks or steps.

[0041] Additionally, the use of "based on" means open and inclusive, as a process, step, calculation, or other action "based on" one or more stated conditions or values can in fact be based on additional conditions or values beyond those stated.

[0042] A high dynamic range image can be used to describe a real scene with a large range of luminance changes, and can better exhibit the optical characteristics of high-brightness areas and low-dark areas in the scene. In general, an image taken by a normal camera with limited dynamic range is called a low dynamic range image. In order to accurately reflect the image in the real scene, it is necessary to convert the low dynamic range image into a high dynamic range image through some technology. In related technologies, for a single low dynamic range image, a high dynamic range reconstruction method based on deep learning is adopted. The low dynamic range image is converted into a high dynamic range image through a set image generation network, which can complete the conversion of the low dynamic range image into the high dynamic range image. However, the high dynamic range image obtained by the method in the related technology has low authenticity, and cannot accurately restore the luminance information of the real scene.

[0043] Based on this, some embodiments of the present disclosure provide an image processing model training method, as shown in Figure 1 、 Figure 2 and Figure 3 , the training method comprises S1-S7.

[0044] S1, input a low dynamic range image into a first initial image processing model, perform high dynamic range reconstruction processing on the low dynamic range image, and generate a first high dynamic range image.

[0045] Wherein, the low dynamic range image is a randomly selected low dynamic range image, and the low dynamic range image has its corresponding real high dynamic range image, which is called a real high dynamic range image. Illustratively, the image data pair of the low dynamic range image and its corresponding real high dynamic range image is selected from an existing public data set. The low dynamic range image is used as a sample to train the first initial image processing model and the second initial image processing model.

[0046] S2, input the low dynamic range image into the second initial image processing model to generate a first coefficient.

[0047] The first initial image processing model and the second initial image processing model are image processing networks, for example, the image processing network is a neural network, the first initial image processing model and the second initial image processing model have initial setting parameters, the first initial image processing model has an initial mapping relationship from the low dynamic range image to the high dynamic range image, and can convert the low dynamic range image into the high dynamic range image.

[0048] In some embodiments, in the above step, the second initial image processing model performs structural feature analysis on the low dynamic range image to generate a weight coefficient of the first high dynamic range image. The first coefficient is the weight coefficient generated by the second initial image processing model performing structural feature analysis on the low dynamic range image. The first coefficient can supplement and repair details of the first high dynamic range image.

[0049] In some embodiments, the weight coefficient is a 1x1x3 matrix.

[0050] In some embodiments, as shown in Figure 2 Before inputting the low dynamic range image into the second initial image processing model, the method further comprises:

[0051] S2', performing down-sampling processing on the low dynamic range image to generate a low dynamic range image processed by down-sampling.

[0052] Exemplarily, for a low dynamic range image with a size of MxN, s-fold down-sampling processing is performed on the low dynamic range image to obtain a resolution image with a size of (M / s)x(N / s), where s is a common divisor of M and N, that is, the image in an sxs window of the original image is changed into a pixel, and the value of the pixel point is the average value of all pixels in the window. In this case, the number of pixels of the image is reduced to s square times of the number of pixels in the sxs window.

[0053] For example, 2-fold bicubic down-sampling processing is performed on the low dynamic range image to obtain a resolution image with a size of (M / 2)x(N / 2), and the image in a 2x2 window of the original image is changed into a pixel. The value of the pixel point is obtained by the formula The size of the target image B after being reduced by K (2 in this case) times is MxN, i.e. K = m / M. The value of each pixel point of the original image A is known, and the value of each pixel point of the target image B is unknown. If the value of each pixel point (X, Y) of the target image B is required, the corresponding pixel (x, y) of the original image A must be found first, and then the 16 nearest pixel points of the pixel (x, y) of the original image A are used as parameters for calculating the pixel value of the pixel (X, Y) of the target image B, and the weights a ij of the 16 pixel points are calculated using the Bicubic base function, and the value of the pixel (X, Y) of the target image B is equal to the weighted superposition of the 16 pixel points.

[0054] In S2, the low dynamic range image after the downsampling processing is input into the second initial image processing model for processing.

[0055] The low dynamic range image is subjected to downsampling processing, and the low dynamic range image is reduced, so that the image size is reduced, which can improve the processing speed of the subsequent steps, for example, the efficiency of the structural feature analysis of the low dynamic range image in S2 is improved, and the training speed of the image processing model is improved.

[0056] In some embodiments, the number of layers of the second initial image processing model is less than the number of layers of the first initial image processing model. In S2', since the low dynamic range image is subjected to downsampling processing and the image size is reduced, the low dynamic range image after the downsampling processing is input into a smaller network model, which can make the image size more matched with the size of the network model, and also improve the training speed of the second initial image processing model.

[0057] For example, the first initial image processing model is a ResNet network model or a DenseNet network model, for example, the first initial image processing model is a ResNet network model with 34 layers, 50 layers or even 101 layers, and the second initial image processing model is a VGG network model, a GoogleNet Inception V1 network model or a MobileNets network model, for example, the second initial image processing model is a VGG network model with 16 layers or 19 layers.

[0058] The ResNet network model is a residual network, which is easy to optimize and can improve the accuracy by increasing the depth. The residual block inside uses a skip connection to alleviate the gradient vanishing problem caused by increasing the depth in the deep neural network.

[0059] The VGG network uses a plurality of convolution layers with small convolution kernels (3x3) instead of a convolution layer with a large convolution kernel (7x7), which can reduce the parameters on the one hand and perform more non-linear mapping on the other hand, thereby increasing the fitting / expression ability of the network.

[0060] In some embodiments, the second initial image processing model and the first initial image processing model can also use the same network model.

[0061] It should be noted that the execution order of S1 and S2 is not limited, and S1 and S2 can be executed simultaneously; or S1 can be executed first, and then S2 can be executed; or S2 can be executed first, and then S1 can be executed.

[0062] The first high dynamic range image and the first coefficient are obtained by S1 and S2 respectively, and S3 can be executed.

[0063] S3, generating a second high dynamic range image according to the first high dynamic range image and the first coefficient.

[0064] In some embodiments, S3 includes multiplying the first high dynamic range image by the first coefficient to generate the second high dynamic range image.

[0065] In some embodiments, the image adopted by the present disclosure is a color image, and the image data is a three-dimensional matrix, which has three channels, i.e., a first channel R channel, a second channel G channel, and a third channel B channel. For example, the low dynamic range image is an MxNx3 picture, where M and N are the number of rows and columns of pixels of the image, and 3 is the RGB three channels of each pixel.

[0066] In some embodiments, the picture mentioned in the present disclosure, i.e., the low dynamic range image, the first high dynamic range image, and the second high dynamic range image, in addition to the RGB three channels of the image, also includes a fourth channel; the fourth channel can reflect the brightness information of the picture.

[0067] In some examples, the fourth channel is the maximum value of the pixels of the image.

[0068] In some embodiments, before inputting the low dynamic range image into the first initial image processing model, the method further includes extracting the maximum value of the pixels of the low dynamic range image as a fourth channel, performing channel connection, and inputting the low dynamic range image with four channels into the first initial image processing model in S1.

[0069] For example, the maximum value of the pixels is a single value, which is expanded into an MxNx1 matrix, and the MxNx1 matrix is channel-connected with the three channels of the low dynamic range image (the image size is MxNx3), to obtain a low dynamic range image with an image size of MxNx4.

[0070] In S1, the generated first high dynamic range image is a three-dimensional matrix after processing by the first initial image processing model. The first high dynamic range image also includes four channels.

[0071] In S2, the weight coefficient of the first high dynamic range image is a 1x1x3 matrix after processing by the second initial image processing model, and the number 3 corresponds to the RGB three channels of the image itself.

[0072] In S3, the first high dynamic range image is multiplied by the first coefficient to obtain the final result, that is, the second high dynamic range image can be output, and the second high dynamic range image includes four channels. The first coefficient is the weight coefficient generated by the second initial image processing model for structural feature analysis of the low dynamic range image. The second high dynamic range image, on the basis of the first high dynamic range image, adjusts the weight coefficient to repair and supplement the details of the image, so that the second high dynamic range image is closer to the real high dynamic range image corresponding to the low dynamic range image than the first high dynamic range image, and can reflect more real scene illumination information.

[0073] S4, generating a loss function according to the data pair of the second high dynamic range image and the real high dynamic range image; wherein the real high dynamic range image is the real high dynamic range image corresponding to the low dynamic range image.

[0074] In the above steps, the value of the loss function is calculated to obtain the loss value of the low dynamic range image in the first initial image processing model and the second initial image processing model. The loss function is obtained according to the data pair of the second high dynamic range image and the real high dynamic range image.

[0075] In the present disclosure, the low dynamic range image, the first high dynamic range image and the second high dynamic range image include a fourth channel in addition to the RGB three channels of the image, so the loss function is related to the value of the fourth channel in addition to the value of the RGB three channels of the image, that is, related to the maximum value of the image.

[0076] In some embodiments, the loss function includes the L1 loss and the tone mapping loss of the RGB data pair of the second high dynamic range image and the real high dynamic range image, and the L1 loss of the maximum pixel data pair of the second high dynamic range image and the real high dynamic range image. The calculation formula of the loss function Loss is as follows:

[0077]

[0078] wherein, is the RGB data pair value of the second high dynamic range image, I gtlogarithmic value of RGB data of the real high dynamic range image, pixel maximum value data pair of the second high dynamic range image, pixel maximum value data pair of the real high dynamic range image. The coefficient in the formula is preset.

[0079] Because the extreme value of the image under different exposure conditions is different, the pixel maximum value of the image is extracted in the training process as the fourth channel of the image, and a constraint condition of pixel maximum value of the pixel maximum value of the second high dynamic range image and the pixel maximum value of the real high dynamic range image is added to the loss function, so as to further constrain the training process and more targetedly improve the training accuracy of the first initial image processing model and the second initial image processing model, optimize the training process of the network model, and improve the high dynamic range reconstruction performance of the first target image processing model and the second target image processing model finally obtained.

[0080] S5, training the first initial image processing model and the second initial image processing model using the loss function. It is judged whether the value of the loss function is no longer reduced; if the value of the loss function continues to decrease, S6 is executed, and if the value of the loss function is no longer reduced, S7 is executed.

[0081] S6, updating the parameters in the first initial image processing model and the second initial image processing model.

[0082] In some embodiments, in the process of training the first initial image processing model and the second initial image processing model using the loss function, updating the parameters in the first initial image processing model and the second initial image processing model, the network optimizer adopts the Adam optimizer, and the learning rate is 1e-4.

[0083] Re-read the above steps S1-S5 until the value of the loss function is no longer reduced, and then execute S7.

[0084] S7, taking the first initial image processing model and the second initial image processing model obtained by the last training as the trained first target image processing model and the second target image processing model.

[0085] In S5-S7, according to the training strategy, the first initial image processing model and the second initial image processing model are trained multiple times. In the training process, the parameters in the first initial image processing model and the second initial image processing model are updated constantly, so that the second high dynamic range image output each time is increasingly close to the dynamic range of the real scene, and the value of the loss function gradually decreases until the value of the loss function no longer decreases, that is, the loss function converges. Thus, the first initial image processing model and the second initial image processing model establish a mapping relationship from a single low dynamic range image to a high dynamic range image. The first initial image processing model and the second initial image processing model obtained after the last training are taken as the trained first target image processing model and the second target image processing model. The trained first target image processing model and the second target image processing model can be used to directly complete the reconstruction of a low dynamic range image to a high dynamic range image in the subsequent image generation process.

[0086] The image processing model training method provided by some embodiments of the present disclosure combines the first initial image processing model and the second initial image processing model, trains the two network models, generates a first high dynamic range image by training the first initial image processing model, and generates a first coefficient by the second initial image processing model. The first coefficient is a weight coefficient for analyzing the structural features of the low dynamic range image and the first high dynamic range image. The first high dynamic range image can be repaired in detail by the first coefficient. The final high dynamic range image is obtained according to the values output by the two, so that the final high dynamic range image is closer to the real high dynamic range image. Thus, the accuracy of the first target image processing model and the second target image processing model obtained after the final training can be ensured, and the high dynamic range reconstruction quality of the image can be ensured. Moreover, the second initial image processing model is subjected to down-sampling processing before being input into the second initial image processing model, and the second initial image processing model adopts a small network. Thus, the training efficiency is improved, and the working time is reduced.

[0087] As shown in Figures 4-6 Some embodiments of the present disclosure also provide a high dynamic range image generation method, which comprises:

[0088] S10, inputting a to-be-processed image into the first target image processing model to perform high dynamic range reconstruction processing on the to-be-processed image and generate a first processed image. The to-be-processed image is a low dynamic range image, and the first processed image is a high dynamic range image.

[0089] The first target image processing model is trained by the image processing model training method described above. The to-be-processed image is any low dynamic range image that needs to be subjected to high dynamic range reconstruction.

[0090] S20, inputting the to-be-processed image into the second target image processing model to generate a second coefficient.

[0091] The second coefficient is a weight coefficient generated by the second target image processing model performing structural feature analysis on the to-be-processed image. The second coefficient can supplement and repair details of the first processed image.

[0092] The second target image processing model is trained by the image processing model training method as described above.

[0093] In some embodiments, as Figure 5 As shown, before the to-be-processed image is input into the second target image processing model to generate a weight coefficient of the high dynamic range image, the method further includes: S20', performing down-sampling processing on the to-be-processed image to generate a to-be-processed image subjected to down-sampling processing.

[0094] In S20, the to-be-processed image subjected to down-sampling processing is input into the second target image processing model for processing.

[0095] The step of performing down-sampling processing on the to-be-processed image can refer to the step of performing down-sampling processing on the low dynamic range image as described above, which will not be described herein again.

[0096] The number of layers of the second initial image processing model is less than that of the first initial image processing model. In some embodiments, the number of layers of the second target image processing model is less than that of the first target image processing model. For example, the first target image processing model is a ResNet network model or a DenseNet network model, and the second target image processing model is a VGG network model, a GoogleNet Inception V1 network model, or a MobileNets network model.

[0097] In this way, the to-be-processed image is subjected to down-sampling processing to reduce the picture size, and the second target image processing model adopts a smaller network model, which can improve the processing rate of the picture.

[0098] S30, generating a second processed image according to the first processed image and the second coefficient.

[0099] The second processed image is a high dynamic range image, and the image quality of the second processed image is higher than that of the first processed image, and is closer to the real scene.

[0100] In some embodiments, S30 includes: multiplying data of the first processed image with the second coefficient to generate the second processed image.

[0101] The high dynamic range image generation method provided by some embodiments of the present disclosure adopts a first target image processing model and a second target image processing model to process a low dynamic range image, and the first target image processing model and the second target image processing model are trained by the image processing model training method as described above, so that the low dynamic range image is subjected to high dynamic range reconstruction by the first target image processing model to output a first processed image, and at the same time, the low dynamic range image is subjected to structure feature analysis by the second target image processing model to generate a weight coefficient (second coefficient), which can supplement and repair the details of the first processed image, so that a second processed image obtained according to the first processed image and the weight coefficient is closer to the effect of a real high dynamic image, so that the trained first target image processing model and the second target image processing model can be used to complete the reconstruction of a real high dynamic range image from a low dynamic range image as much as possible, and the method improves the reality of the generated high dynamic range image and can restore the luminance information of a real scene more accurately.

[0102] As shown in Figure 7 Some embodiments of the present disclosure also provide an electronic device 10, which comprises a processor 1, a memory 2, and a computer program stored on the memory 2 and executable on the processor 1.

[0103] The processor 1 is configured to perform inputting a low dynamic range image into a first initial image processing model, performing high dynamic range reconstruction processing on the low dynamic range image, and generating a first high dynamic range image.

[0104] The processor 1 is further configured to perform inputting the low dynamic range image into a second initial image processing model and generating a first coefficient.

[0105] The processor 1 is further configured to perform generating a second high dynamic range image according to data of the first high dynamic range image and the first coefficient.

[0106] The processor 1 is further configured to perform generating a loss function according to a pair of the second high dynamic range image and a real high dynamic range image; wherein the real high dynamic range image is a real high dynamic range image corresponding to the low dynamic range image.

[0107] The processor 1 is further configured to perform training the first initial image processing model and the second initial image processing model using the loss function.

[0108] The memory 2 is configured to store data of the first high dynamic range image, the first coefficient, and data of the second high dynamic range image.

[0109] In some embodiments, the processor 1 is further configured to perform multiplying data of the first high dynamic range image with the first coefficient to generate a second high dynamic range image.

[0110] The processor 1 is further configured to perform, before inputting the low dynamic range image into the second initial image processing model, downsampling processing on the low dynamic range image to generate a downsampled low dynamic range image.

[0111] In some embodiments, the processor 1 is further configured to perform inputting the to-be-processed image into the first target image processing model to perform high dynamic range reconstruction processing on the to-be-processed image to generate a first processed image; the to-be-processed image is a low dynamic range image.

[0112] The processor 1 is further configured to perform inputting the to-be-processed image into the second target image processing model to generate a second coefficient. The first target image processing model and the second target image processing model are trained by the image processing model training method provided in some embodiments of the present disclosure.

[0113] The processor 1 is further configured to perform generating a second processed image according to the first processed image and the second coefficient.

[0114] The memory 2 is configured to store the first processed image, the second coefficient and the second processed image.

[0115] In some embodiments, the processor 1 is further configured to perform multiplying the first processed image with the second coefficient to generate the second processed image.

[0116] The processor 1 is further configured to perform, before inputting the to-be-processed image into the second target image processing model, downsampling processing on the to-be-processed image to generate a downsampled to-be-processed image.

[0117] Exemplarily, the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0118] The memory can be a Read-Only Memory (ROM) or other type of static storage device that can store static information and instructions, a Random Access Memory (RAM), or other type of dynamic storage device that can store information and instructions, an Electrically Erasable Programmable Read-Only Memory (EEPROM), a Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, a magnetic disk storage or other magnetic storage devices or any other medium capable of storing and carrying desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to. The memory can exist independently of the processor and be connected to the processor via a communication bus. The memory can also be integrated with the processor.

[0119] The above-mentioned electronic device has the same beneficial effects as the image processing model training method and the image processing method described in some embodiments, and the beneficial effects of the above-mentioned image processing model training method and the image processing method will not be repeated here.

[0120] Some embodiments of the present disclosure provide a computer-readable storage medium (for example, a non-transitory computer-readable storage medium) having computer program instructions stored therein, which, when executed on a processor, cause the processor to perform the image processing model training method provided by any of the above-mentioned embodiments, and / or the high dynamic range image generation method provided by any of the above-mentioned embodiments.

[0121] Exemplarily, the above-mentioned computer-readable storage medium can include, but is not limited to, a magnetic storage device (for example, a hard disk, a floppy disk, or a magnetic tape, etc.), an optical disk (for example, a CD (Compact Disk), a DVD (Digital Versatile Disk), etc.), a smart card and a flash memory device (for example, an EPROM (Erasable Programmable Read-Only Memory), a card, a stick or a key drive, etc.).

[0122] The various computer-readable storage media described in the present disclosure can represent one or more devices and / or other machine-readable storage media for storing information. The term "machine-readable storage medium" can include, but is not limited to, a wireless channel and various other media capable of storing, containing, and / or carrying instructions and / or data.

[0123] Some embodiments of the present disclosure further provide a computer program product. The computer program product comprises computer program instructions, which, when executed on a computer, cause the computer to perform one or more steps of the image processing model training method according to some embodiments of the above-described embodiments, and / or one or more steps of the image processing method according to some embodiments of the above-described embodiments.

[0124] Some embodiments of the present disclosure further provide a computer program. When the computer program is executed on a computer, the computer program causes the computer to perform one or more steps of the image processing model training method according to some embodiments of the above-described embodiments, and / or one or more steps of the image processing method according to some embodiments of the above-described embodiments.

[0125] The computer-readable storage medium, the computer program product and the computer program described above have the same beneficial effects as the image processing model training method and the image processing method according to some embodiments of the above-described embodiments, and the beneficial effects will not be repeated here.

[0126] The above describes only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto, and any person skilled in the art can think of changes or replacements within the technical scope disclosed by the present disclosure, which should be covered by the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. An image processing model training method, comprising: inputting a low dynamic range image into a first initial image processing model, performing high dynamic range reconstruction processing on the low dynamic range image to generate a first high dynamic range image; inputting the low dynamic range image into a second initial image processing model to generate a first coefficient; generating a second high dynamic range image according to the first high dynamic range image and the first coefficient, wherein the generating of the second high dynamic range image according to the first high dynamic range image and the first coefficient comprises multiplying the first high dynamic range image and the first coefficient to generate the second high dynamic range image; generating a loss function according to a data pair of the second high dynamic range image and a real high dynamic range image, wherein the real high dynamic range image is a real high dynamic range image corresponding to the low dynamic range image; training the first initial image processing model and the second initial image processing model using the loss function.

2. The image processing model training method of claim 1, wherein, The first coefficient is a weight coefficient generated by the second initial image processing model performing structural feature analysis on the low dynamic range image.

3. The image processing model training method of claim 2, wherein, The weight coefficient is a 1x1x3 matrix.

4. The image processing model training method of claim 1, wherein, The low dynamic range image, the first high dynamic range image and the second high dynamic range image further comprise a fourth channel in addition to the RGB three channels of the image. The loss function is also related to the value of the fourth channel.

5. The image processing model training method of claim 4, wherein, The fourth channel is the maximum pixel value of the image. The loss function comprises L1 loss and tone mapping loss of the RGB data pair of the second high dynamic range image and the real high dynamic range image, and L1 loss of the maximum pixel value data pair of the second high dynamic range image and the real high dynamic range image.

6. The image processing model training method of claim 1, wherein, Before inputting the low dynamic range image into the second initial image processing model, further comprising: performing down-sampling processing on the low dynamic range image to generate a down-sampled low dynamic range image.

7. The image processing model training method of claim 6, wherein The number of layers of the second initial image processing model is less than the number of layers of the first initial image processing model.

8. The image processing model training method according to any one of claims 1 to 5, wherein In the process of training the first initial image processing model and the second initial image processing model using the loss function, in the case that the loss function does not converge, updating the parameters in the first initial image processing model and the second initial image processing model, the network optimizer adopts an Adam optimizer, and the learning rate is 1e-4.

9. A high dynamic range image generation method, comprising: inputting a to-be-processed image into a first target image processing model to perform high dynamic range reconstruction processing on the to-be-processed image to generate a first processed image; The to-be-processed image is a low dynamic range image; inputting the to-be-processed image into a second target image processing model to generate a second coefficient, wherein the first target image processing model and the second target image processing model are trained by the image processing model training method in any one of claims 1-8. generating a second processed image according to the first processed image and the second coefficient; wherein the generating the second processed image according to the first processed image and the second coefficient comprises: multiplying the first processed image and the second coefficient to generate the second processed image.

10. The high dynamic range image generation method of claim 9, wherein, The second coefficient is a weight coefficient generated by the second target image processing model performing structural feature analysis on the to-be-processed image.

11. The high dynamic range image generation method of claim 9, wherein, Before inputting the to-be-processed image into the second target image processing model, the method further comprises: performing down-sampling processing on the to-be-processed image to generate a down-sampled to-be-processed image.

12. The high dynamic range image generation method of claim 11, wherein, The number of layers of the second target image processing model is less than the number of layers of the first target image processing model.

13. An electronic device comprising: a processor, a memory, and a computer program stored on the memory and executable on the processor; The processor is configured to perform inputting a low dynamic range image into a first initial image processing model, performing high dynamic range reconstruction processing on the low dynamic range image to generate a first high dynamic range image; The processor is further configured to perform inputting the low dynamic range image into a second initial image processing model to generate a first coefficient; The processor is further configured to perform generating a second high dynamic range image according to the first high dynamic range image and the first coefficient; The processor is further configured to perform generating a loss function according to the second high dynamic range image and data of a real high dynamic range image; wherein the real high dynamic range image is a real high dynamic range image corresponding to the low dynamic range image; The processor is further configured to perform training the first initial image processing model and the second initial image processing model using the loss function; The memory is configured to store data of the first high dynamic range image, the first coefficient, and data of the second high dynamic range image; The processor is further configured to perform multiplying the data of the first high dynamic range image and the first coefficient to obtain data of the second high dynamic range image, and generating the second high dynamic range image.

14. The electronic device of claim 13, wherein, The processor is further configured to perform, before inputting the low dynamic range image into the second initial image processing model, performing down-sampling processing on the low dynamic range image to generate a down-sampled low dynamic range image.

15. The electronic device of claim 13, wherein, The processor is further configured to perform inputting a to-be-processed image into a first target image processing model, performing high dynamic range reconstruction processing on the to-be-processed image to generate a first processed image; the to-be-processed image is a low dynamic range image; The processor is further configured to perform inputting the to-be-processed image into a second target image processing model to generate a second coefficient; wherein the first target image processing model and the second target image processing model are trained by the image processing model training method in any one of claims 1-8; The processor is further configured to perform generating a second processed image according to the first processed image and the second coefficient; The memory is configured to store the first processed image, the second coefficient, and the second processed image.

16. The electronic device of claim 15, wherein, The processor is further configured to perform multiplication of the first processed image and the second coefficient to generate the second processed image. The processor is further configured to perform, before inputting the to-be-processed image into the second target image processing model, downsampling processing on the to-be-processed image to generate a to-be-processed image subjected to the downsampling processing. 17.A computer readable storage medium, having stored therein computer program instructions, which, when executed on a processor, cause the processor to perform the image processing model training method according to any one of claims 1 to 8, and / or the high dynamic range image generation method according to any one of claims 9 to 12.

Images