Image processing method and apparatus

Abstract

The application discloses an image processing method and device. The method comprises the following steps: acquiring a first image, wherein the first image is an image processed by an image signal processor (ISP); performing inverse processing on the first image by using an image inverse processing model, so as to obtain a second image, wherein the inverse processing corresponds to a target image processing process, and the target image processing process comprises other image processing processes except a high dynamic range (HDR) processing process in the ISP processing process; and performing blur processing on a local region in the second image by using an HDR inverse processing model, so as to obtain a third image. According to the image processing method, the simulation process of the ISP can be simplified, and the simulation effect is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to an image processing method and device. BACKGROUND

[0002] An image sensor is an important component of a camera. The image sensor can collect an optical image and convert the optical image into an image signal available to a back-end image signal processor (ISP). The ISP can perform high dynamic range imaging (HDR), auto white balance (AWB), color correction matrix (CCM), Gamma correction, and the like on an image signal output by the front-end image sensor (i.e., an input image signal) to better restore the details of the scene being photographed under different optical conditions and obtain a target image.

[0003] In the prior art, in order to simulate the input image of the ISP, the target image is usually subjected to inverse processing corresponding to the ISP processing, i.e., the target image is subjected to inverse Gamma transformation, inverse CCM color correction, inverse white balance, and the like to obtain the input image before the input of the ISP. In addition, the inverse processing corresponding to the HDR cannot be implemented at present.

[0004] However, the inverse processing process corresponding to the ISP processing is complex at present, and the accuracy of the obtained input image is low. That is, the inverse processing simulation process of the ISP in the prior art is complex, and the inverse processing simulation effect is poor. SUMMARY

[0005] Embodiments of the present application provide an image processing method, device, equipment, computer readable storage medium, and computer program product, which can simplify the inverse processing simulation process of the ISP and improve the simulation effect.

[0006] In a first aspect, embodiments of the present application provide an image processing method, which comprises:

[0007] obtaining a first image, the first image being an image processed by an image signal processor (ISP),

[0008] performing inverse processing on the first image by using an image inverse processing model, the inverse processing corresponding to a target image processing process, the target image processing process comprising other image processing processes in the ISP processing process except a high dynamic range imaging (HDR) processing process,

[0009] performing blur processing on a local region in the second image by using an HDR inverse processing model to obtain a third image.

[0010] In a possible implementation, before the image inverse processing model is used to perform inverse processing on the first image corresponding to a target image processing process to obtain a second image, the method further includes:

[0011] obtaining a plurality of first image samples processed by an ISP and a plurality of second image samples corresponding to the first image samples respectively,

[0012] performing feature extraction on the first image samples and the second image samples to obtain first image features corresponding to each of the first image samples respectively, and second image features corresponding to each of the second image samples respectively,

[0013] performing inverse processing on the first image features by using an initial image inverse processing model corresponding to the target image processing process to obtain first predicted image features corresponding to each of the first image features respectively,

[0014] determining a first loss function value according to similarities between the plurality of second image features and first predicted image features corresponding to the second image features respectively,

[0015] adjusting model parameters of the initial image inverse processing model according to the first loss function value to obtain the image inverse processing model through training.

[0016] In a possible implementation, the first loss function value is determined according to similarities between the plurality of second image features and first predicted image features corresponding to the second image features respectively, including:

[0017] calculating mean square errors between the plurality of first predicted image features and the plurality of second image features,

[0018] determining the mean square errors as the first loss function value.

[0019] In a possible implementation, before the HDR inverse processing model is used to perform blur processing on a local region in the second image to obtain a third image, the method further includes:

[0020] obtaining a plurality of third image samples processed by an HDR and a plurality of fourth image samples corresponding to the third image samples respectively,

[0021] performing blur processing on the local region of each of the third image samples by using an initial HDR inverse processing model to obtain second predicted images corresponding to each of the third image samples respectively,

[0022] determining a second loss function value according to similarities between the plurality of fourth image samples and second predicted images corresponding to the fourth image samples respectively,

[0023] Adjusting model parameters of the initial HDR inverse processing model according to the second loss function value, to obtain the HDR inverse processing model.

[0024] In a possible implementation, the obtaining of the plurality of HDR-processed third image samples comprises:

[0025] Obtaining a plurality of fourth images that are continuous in time,

[0026] Performing inverse processing corresponding to the target image processing process on the plurality of fourth images respectively by using the image inverse processing model to obtain a plurality of fifth images,

[0027] Obtaining three continuous third image samples from the plurality of fifth images.

[0028] In a possible implementation, the performing of the blurring processing on the local region of each third image sample by using the initial HDR inverse processing model to obtain a second prediction image corresponding to each third image sample respectively comprises:

[0029] Extracting, by using the initial HDR inverse processing model, a correlation feature between the three continuous third image samples,

[0030] Performing blurring processing on a local region of a target image sample according to the correlation feature to obtain a second prediction image corresponding to the target image sample, the target image sample being an image sample at a middle position in the three continuous third image samples.

[0031] In a possible implementation, the performing of the blurring processing on the local region of the target image sample according to the correlation feature to obtain the second prediction image corresponding to the target image sample comprises:

[0032] Determining the correlation feature as a target image feature corresponding to the target image sample,

[0033] Processing the target image feature by using a random algorithm to obtain a second prediction image feature,

[0034] Determining an image corresponding to the second prediction image feature as the second prediction image.

[0035] In a possible implementation, the image inverse processing model is a four-layer convolutional neural network model, and the HDR inverse processing model is a bidirectional long short-term memory recurrent neural network model.

[0036] In a possible implementation, the blurring processing comprises any one of underexposure processing and overexposure processing.

[0037] In a second aspect, an image processing apparatus is provided, and the apparatus includes:

[0038] A first obtaining module is configured to obtain a first image, the first image being an image processed by an image signal processor (ISP),

[0039] A first processing module is configured to perform inverse processing on the first image corresponding to a target image processing process by using an image inverse processing model, to obtain a second image, wherein the target image processing process includes other image processing processes in the ISP processing process except for a high dynamic range (HDR) processing process,

[0040] A second processing module is configured to perform blur processing on a local region in the second image by using an HDR inverse processing model, to obtain a third image.

[0041] In a third aspect, an electronic device is provided, and the device includes a processor and a memory storing computer program instructions,

[0042] The processor implements the method in any possible implementation method of the first aspect when executing the computer program instructions.

[0043] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores computer program instructions, and the computer program instructions are executed by a processor to implement the method in any possible implementation method of the first aspect.

[0044] In a fifth aspect, a computer program product is provided, and instructions in the computer program product are executed by a processor of an electronic device to cause the electronic device to perform the method in any possible implementation method of the first aspect.

[0045] The image processing method, apparatus, device, computer readable storage medium and computer program product provided in the embodiments of the present application can obtain a second image which has not been processed by an HDR inverse processing model, by performing inverse processing on a first image corresponding to a target image processing process by using an image inverse processing model. The inverse processing corresponding to an HDR processing process can be performed on a local region in the second image by using an HDR inverse processing model, to obtain a third image. Since the first image is an image processed by an ISP, the image processed by the ISP can be processed by using the image inverse processing model and the HDR inverse processing model, to obtain an image before ISP processing, which can simplify the simulation process of the ISP and improve the simulation effect. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly introduced. Those skilled in the art can further obtain other drawings according to these drawings without any creative effort.

[0047] Figure 1 is a schematic diagram of an ISP inverse processing principle provided by the embodiments of the present application,

[0048] Figure 2 is a flow schematic diagram of an image processing method provided by the embodiments of the present application,

[0049] Figure 3 is a schematic diagram of an ISP inverse processing neural network model provided by the embodiments of the present application,

[0050] Figure 4 is a structural schematic diagram of an image processing apparatus provided by the embodiments of the present application,

[0051] Figure 5 is a structural schematic diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0052] The features and exemplary embodiments of various aspects of the present application will be described in detail below, in order to make the purposes, technical solutions and advantages of the present application more clear and apparent, the present application will be further described in detail below in combination with the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, but not to limit the present application. The present application can be implemented without some of these specific details by those skilled in the art. The following description of the embodiments is only to provide a better understanding of the present application by showing examples of the present application.

[0053] It should be noted that, in this paper, relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the elements defined by the statement "include" do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0054] As described in the background section, the inverse processing process corresponding to ISP processing is complex, and the accuracy of the obtained input image is low.

[0055] Specifically, with the increasing demand for image quality, the progress of HDR technology has brought the display field to a new height. Most of the current ISPs have HDR functions. If the inverse transformation process is still performed sequentially according to the ISP processing (ISP Pipeline) process, there are many defects. First, if the ISP contains an HDR function, the HDR adopts a multi-frame fusion technology, which fuses multiple images with different exposure levels to obtain a high dynamic and more expressive image. If direct inverse transformation is used, it is not easy to restore the image before fusion. Second, under different color temperature conditions, the ISP dynamically adjusts the correction strength and white balance strength of the image, which means that the correction strength of the white balance is not fixed, increasing the difficulty of inverse transformation. Finally, the ISP processing process involves multiple image processing modules, resulting in a series of complex linear and nonlinear superposition relationships between the input image and the output image. The complex relationship is difficult to express through mathematics or fixed models.

[0056] Therefore, for the inverse processing simulation of the ISP, there is currently a lack of effective simulation method that can consider its dynamic changes. At the same time, for the ISP image processing unit containing the HDR function, there is also a lack of inverse processing simulation method that can consider the multi-frame fusion technology. Thus, the simulation process of the ISP is complex, and the simulation effect is poor.

[0057] Therefore, in order to solve the problems in the prior art, the embodiments of the present application provide an image processing method, device, equipment, computer readable storage medium and computer program product.

[0058] Here, the image processing method can be a simulation method for implementing ISP inverse processing using a neural network, that is, the inverse processing process of each module in the ISP is replaced by a neural network. The simulation method can input an image acceptable to the human eye, without noise and defects, into an ISP inverse processing neural network model. The ISP inverse processing neural network model can perform target image inverse processing (other image inverse processing in ISP inverse processing except for HDR inverse processing) and HDR inverse processing on the input image data. That is, the ISP inverse processing neural network model can include a neural network structure for performing target image inverse processing and a neural network structure for performing HDR inverse processing. The ISP inverse processing neural network model can output the image before inputting the ISP, thereby achieving the purpose of simulating the ISP input image. Based on this, a schematic diagram of an ISP inverse processing principle provided by the embodiments of the present application can be as shown in Figure 1

[0059] ​Firstly, the image processing method provided by the embodiments of the present application is introduced below.

[0060] Figure 2 A flowchart of an image processing method provided by the embodiments of the present application is shown. As shown in the figure, Figure 2 The image processing method provided by the embodiments of the present application includes steps S210 to S230.

[0061] S210, a first image is acquired, and the first image is an image processed by an image signal processor (ISP).

[0062] Here, the first image can be an image without defects such as noise, underexposure, and overexposure. Specifically, the first image can be an image processed by the ISP. The ISP processing can include HDR, AWB, CCM color correction, and Gamma correction, etc. Of course, the ISP processing can also include black level correction (BLC), lens shading correction (LSC), etc., which are not limited here.

[0063] S220, the first image is processed by an image inverse processing model to obtain a second image, wherein the target image processing process includes other image processing processes in the ISP processing process except the high dynamic range imaging (HDR) processing process.

[0064] Here, the image inverse processing model can have the function of performing target image inverse processing. The target image inverse processing is the inverse processing corresponding to the target image processing process. The target image processing process can include other image processing processes in the ISP processing process except the HDR processing process. That is, the target image processing process can include AWB, CCM color correction, BLC, LSC, and Gamma correction, etc. In this way, the second image can be an image with defects such as noise, underexposure, and overexposure. In addition, the image inverse processing model can be a separate neural network model, or it can be the first layer neural network structure in the ISP inverse processing neural network model.

[0065] As an example, the image inverse processing model can be a four-layer convolutional neural network (CNN) model. On this basis, the first layer neural network structure in the ISP inverse processing neural network model can be a four-layer convolutional neural network.

[0066] Based on this, in order to make the image inverse processing model have the function of performing target image inverse processing, in some embodiments, before S220, it can also include:

[0067] obtain a plurality of first image samples processed by ISP and a plurality of second image samples corresponding to the first image samples respectively,

[0068] perform feature extraction on the first image samples and the second image samples to obtain first image features corresponding to each first image sample respectively, and second image features corresponding to each second image sample respectively,

[0069] perform inverse processing corresponding to a target image processing process on the first image features by using an initial image inverse processing model to obtain first predicted image features corresponding to each first image feature respectively,

[0070] determine a first loss function value according to similarities between the plurality of second image features and the first predicted image features corresponding to the second image features respectively,

[0071] adjust model parameters of the initial image inverse processing model according to the first loss function value to obtain an image inverse processing model.

[0072] Here, the first image samples processed by ISP can be image samples without noise, underexposure, overexposure and other defects. The second image samples before ISP processing can be image samples with noise, underexposure, overexposure and other defects. For a whole image, noise, underexposure, overexposure and other defects can belong to a kind of local image features. In addition, the plurality of first image samples and the second image samples corresponding to the first image samples respectively can be divided into a training group and a test group in a ratio of 7:3 to train the initial image inverse processing model and obtain the image inverse processing model. The second image samples can be training sample labels.

[0073] As an example, in the case of a four-layer convolutional neural network model for the image inverse processing model, the convolutional neural network can be used to learn the inverse processing features of other modules in ISP except the HDR module. The convolutional neural network can use a 3*3 convolution kernel to convolve the first image samples and the second image samples to extract image feature information corresponding to the first image samples and the second image samples respectively to obtain the first image features and the second image features. In addition, a Softmax activation function can be set between each layer of the neural network. The activation function can map the input of the previous layer of network to the output end as data of the next layer of network.

[0074] As an example, the model parameters can include weight values, biases, and learning rates. In the initial case of model training, the model parameters can be randomly set. In the process of performing convolution iterations, the model parameters such as weight values, biases, and learning rates in the initial image inverse processing model are dynamically adjusted according to the first loss function value between the predicted value (the first predicted image feature) and the true value (the second image feature), until the model converges, that is, the training process of the initial image inverse processing model is completed, and the trained image inverse processing model is obtained.

[0075] Based on this, in some embodiments, the determination of the first loss function value according to the similarity between the plurality of second image features and their respective corresponding first predicted image features can include:

[0076] calculating the mean square error between the plurality of first predicted image features and the plurality of second image features,

[0077] determining the mean square error as the first loss function value.

[0078] That is, the model parameters such as weight values, biases, and learning rates in the initial image inverse processing model can be dynamically adjusted according to the mean square error between the predicted value (the first predicted image feature) and the true value (the second image feature). In this way, after multiple convolution iterations, the model converges when the mean square error is minimized, and the trained image inverse processing model is obtained. At the same time, the model parameters such as weight, bias, learning rate, etc. at the time of minimizing the mean square error can also be obtained.

[0079] In this way, by training the image inverse processing model based on the plurality of ISP-processed first image samples and their respective corresponding ISP-processed second image samples, the image inverse processing model can have the function of performing target image inverse processing.

[0080] S230, performing blur processing on the local region in the second image using the HDR inverse processing model to obtain a third image.

[0081] Here, the HDR inverse processing model can have a function of performing HDR inverse processing. The HDR inverse processing is inverse processing corresponding to the HDR processing process. Since the multi-frame fusion technology of the HDR is to fuse pictures under different exposure levels, to brighten the underexposed local part and to lower the brightness of the overexposed local part, the blurring processing on the local area in the second image can be underexposure processing or overexposure processing. The underexposure processing on the local area in the second image can be to lower the brightness of the local area in the second image, and the overexposure processing on the local area in the second image can be to increase the brightness of the local area in the second image. In this way, the overexposure and underexposure of the image before the HDR processing can be simulated. That is, the third image can be an image with noise, underexposure, overexposure and other defects. In addition, the HDR inverse processing model can be a separate neural network model, or can be a second layer neural network structure in the ISP inverse processing neural network model.

[0082] As an example, the HDR inverse processing model can be a bidirectional long short term memory (Bi-LSTM) model. On this basis, the second layer neural network structure in the ISP inverse processing neural network model can be a bidirectional long short term memory (Bi-LSTM) model.

[0083] Based on this, in order to enable the HDR inverse processing model to have the function of performing HDR inverse processing, in some embodiments, before S230, the method can further include:

[0084] obtaining a plurality of third image samples after HDR processing and a plurality of fourth image samples before HDR processing corresponding to the third image samples respectively,

[0085] blurring processing the local area of each third image sample using an initial HDR inverse processing model to obtain a second predicted image corresponding to each third image sample respectively,

[0086] determining a second loss function value according to the similarity between the plurality of fourth image samples and the second predicted images corresponding to the fourth image samples respectively,

[0087] adjusting the model parameters of the initial HDR inverse processing model according to the second loss function value to obtain the HDR inverse processing model.

[0088] Here, the third image sample after HDR processing and the fourth image sample before HDR processing can each be an image sample with noise, underexposure, overexposure, and the like. The third image sample is an image sample obtained after HDR processing of the corresponding fourth image sample. The plurality of third image samples and their respective corresponding fourth image samples can be divided into a training group and a test group in a ratio of 7:3, to be used to train the initial HDR inverse processing model, to obtain the HDR inverse processing model. The fourth image sample can be a training sample label. In addition, since the HDR module can be the first processing module in the ISP forward processing process, the fourth image sample before HDR processing and the second image sample before ISP processing can be the same image sample. Of course, the fourth image sample and the second image sample can also be different image samples, which are not limited here. In addition, the third image sample can be an image sample obtained after HDR processing of an image with noise, underexposure, overexposure, and the like, or an image sample obtained after processing of an image without noise, underexposure, overexposure, and the like by the image inverse processing model, which is not limited here.

[0089] Based on this, in some embodiments, the above-mentioned obtaining a plurality of third image samples after HDR processing can include:

[0090] obtaining a plurality of fourth images that are continuous in time,

[0091] using an image inverse processing model to respectively perform inverse processing corresponding to the target image processing process on the plurality of fourth images, to obtain a plurality of fifth images,

[0092] obtaining three continuous third image samples from the plurality of fifth images.

[0093] Here, the plurality of fourth images that are continuous in time can be a plurality of images in a video stream, or a plurality of images continuously captured of the same target, which are not limited here. In addition, the fourth image can be an image without noise, underexposure, overexposure, and the like. The image inverse processing model can be an image inverse processing model that has been trained. After using the image inverse processing model to perform inverse processing corresponding to the target image processing process on the fourth image, a fifth image with noise, underexposure, overexposure, and the like can be obtained. After obtaining the fifth image, each fifth image can be determined as a third image sample, or three continuous third image samples can be obtained from the plurality of fifth images.

[0094] In addition, as an example, the HDR inverse processing model can be used to simulate the HDR function in the ISP. In the initial case of model training, the model parameters can be randomly set. In the model training process, by dynamically adjusting the model parameters in the initial HDR inverse processing model according to the second loss function value between the predicted value (the second prediction image) and the true value (the fourth image sample), until the model converges, that is, the training process of the initial HDR inverse processing model is completed, and the trained HDR inverse processing model is obtained. Wherein, the predicted value can also be the second prediction image feature corresponding to the second prediction image, and the true value can also be the fourth image feature corresponding to the fourth image sample.

[0095] In this way, by training the HDR inverse processing model based on a plurality of third image samples after HDR processing and their respectively corresponding fourth image samples before HDR processing, the HDR inverse processing model can have the function of performing HDR inverse processing.

[0096] Based on this, in order to simulate the HDR function in the ISP, in some embodiments, the above-mentioned blurring processing of the local area of each third image sample using the initial HDR inverse processing model to obtain the second prediction image corresponding to each third image sample can include:

[0097] extracting the associated features between the three consecutive third image samples using the initial HDR inverse processing model,

[0098] blurring processing the local area of the target image sample according to the associated features to obtain the second prediction image corresponding to the target image sample, the target image sample being the image sample at the middle position in the three consecutive third image samples.

[0099] Here, the initial HDR inverse processing model can be a bidirectional long short-term memory recurrent neural network, which can have the function of extracting associated features between multiple images. If the associated features between the three consecutive third image samples are extracted, the associated features can include feature information corresponding to the current frame image, feature information corresponding to the previous frame image of the current frame image, and feature information corresponding to the next frame image of the current frame image. Here, the current frame image can be the target image sample at the middle position in the three consecutive third image samples. Since the target image feature includes feature information corresponding to the previous frame image sample of the target image sample and feature information corresponding to the next frame image sample of the target image sample, by blurring processing the local area of the target image sample, the HDR function in the ISP can be simulated.

[0100] Based on this, in order to simulate the overexposure and underexposure of the image before the HDR processing, in some embodiments, the above-mentioned blurring processing of the local region of the target image sample according to the associated features to obtain the second prediction image corresponding to the target image sample can include:

[0101] determining the associated features as target image features corresponding to the target image sample,

[0102] processing the target image features by using a random algorithm to obtain second prediction image features,

[0103] determining the image corresponding to the second prediction image features as the second prediction image.

[0104] Here, processing the target image features by using a random algorithm can be multiplying the weight matrix with a random seed set in the Bi-LSTM and the feature matrix corresponding to the target image features. After processing the target image features by using a random algorithm, the RGB values of the local range of the target image features can be reduced as a whole, or the RGB values of the local range of the target image features can be improved as a whole to obtain the second prediction image features. Wherein, reducing the RGB values of the local range of the target image features as a whole can be underexposure processing of the local region of the target image sample, and improving the RGB values of the local range of the target image features as a whole can be overexposure processing of the local region of the target image sample.

[0105] In this way, by processing the target image features by using a random algorithm, the overexposure and underexposure of the image before the HDR processing can be simulated.

[0106] The image processing method of the embodiments of the present application can obtain the second image which has not undergone the HDR inverse processing by using the image inverse processing model to perform inverse processing corresponding to the target image processing process on the first image. By blurring processing the local region in the second image by using the HDR inverse processing model, inverse processing corresponding to the HDR processing process can be performed to obtain the third image. Since the first image is the image processed by the ISP, by processing the image processed by the ISP by using the image inverse processing model and the HDR inverse processing model, the image before the ISP processing can be obtained, which can simplify the simulation process of the ISP and improve the simulation effect.

[0107] Based on this, in order to better describe the whole scheme, based on the above-mentioned embodiments, a specific example is given.

[0108] For example, as shown in Figure 3 , the ISP inverse processing neural network model can have two-layer neural network structure. Among them, the first layer neural network structure can be CNN, and the second layer neural network structure can be Bi-LSTM.

[0109] In this way, by inputting the image without noise, underexposure, overexposure and other defects into the ISP inverse processing neural network model, an image with noise, underexposure, overexposure and other defects can be output, thereby achieving the purpose of simulating the input image of the ISP. Since the ISP inverse processing neural network model can automatically perform the ISP inverse processing simulation process, compared with sequentially performing inverse transformation according to the ISP Piepline, the complexity can be reduced and the simulation accuracy can be improved. At the same time, the combination of CNN and Bi-LSTM network can make the simulation process compatible with the HDR inverse processing process. In this way, the simulation process of the ISP can be simplified and the simulation effect can be improved.

[0110] Based on the image processing method provided in the above embodiments, the application also provides a specific implementation of an image processing device. Please refer to the following embodiments.

[0111] As shown in the following embodiments, the image processing device 400 provided by the embodiments of the application includes the following modules: Figure 4

[0112] The first acquisition module 410 is configured to acquire a first image, the first image being an image processed by an image signal processor (ISP),

[0113] The first processing module 420 is configured to perform inverse processing on the first image corresponding to a target image processing process by using an image inverse processing model to obtain a second image, wherein the target image processing process includes other image processing processes in the ISP processing process except for a high dynamic range imaging (HDR) processing process,

[0114] The second processing module 430 is configured to perform blur processing on a local region in the second image by using an HDR inverse processing model to obtain a third image.

[0115] The above image processing device 400 will be described in detail as follows:

[0116] In some embodiments, the image processing device 400 can further include:

[0117] The second acquisition module is configured to acquire a plurality of first image samples processed by the ISP and a plurality of second image samples corresponding to the first image samples respectively before performing inverse processing on the first image corresponding to the target image processing process by using the image inverse processing model to obtain the second image,

[0118] The feature extraction module is configured to perform feature extraction on the first image samples and the second image samples to obtain first image features corresponding to each first image sample respectively and second image features corresponding to each second image sample respectively,

[0119] ​The third processing module is configured to perform inverse processing corresponding to the target image processing process on the first image features by using the initial image inverse processing model, to obtain first predicted image features corresponding to each first image feature respectively,

[0120] The first determining module is configured to determine a first loss function value according to similarities between the second image features and the first predicted image features corresponding to the second image features respectively,

[0121] The first training module is configured to adjust model parameters of the initial image inverse processing model according to the first loss function value, to train the image inverse processing model.

[0122] In some embodiments, the first determining module can include:

[0123] The calculating sub-module is configured to calculate mean square errors between the first predicted image features and the second image features,

[0124] The determining sub-module is configured to determine the mean square errors as the first loss function value.

[0125] In some embodiments, the image processing apparatus 400 can further include:

[0126] The third acquiring module is configured to acquire a plurality of third image samples processed by HDR processing and fourth image samples corresponding to the third image samples respectively before performing blur processing on local regions in the second image by using the HDR inverse processing model to obtain a third image,

[0127] The fourth processing module is configured to perform blur processing on the local regions of each third image sample respectively by using an initial HDR inverse processing model, to obtain second predicted images corresponding to each third image sample respectively,

[0128] The second determining module is configured to determine a second loss function value according to similarities between the fourth image samples and the second predicted images corresponding to the fourth image samples respectively,

[0129] The second training module is configured to adjust model parameters of the initial HDR inverse processing model according to the second loss function value, to train the HDR inverse processing model.

[0130] In some embodiments, the third acquiring module can include:

[0131] The first acquiring sub-module is configured to acquire a plurality of fourth images continuous in time,

[0132] The first processing sub-module is configured to perform inverse processing corresponding to the target image processing process on the fourth images respectively by using the image inverse processing model, to obtain fifth images,

[0133] The second acquisition sub-module is configured to acquire three continuous third image samples from the plurality of fifth images.

[0134] In some embodiments, the fourth processing module can include:

[0135] The extraction sub-module is configured to extract, by using the initial HDR inverse processing model, a correlation feature between the three continuous third image samples,

[0136] The second processing sub-module is configured to perform blur processing on a local region of a target image sample according to the correlation feature, to obtain a second prediction image corresponding to the target image sample, the target image sample being an image sample at a middle position in the three continuous third image samples.

[0137] In some embodiments, the second processing sub-module can include:

[0138] The first determination unit is configured to determine the correlation feature as a target image feature corresponding to the target image sample,

[0139] The processing unit is configured to process the target image feature by using a random algorithm to obtain a second prediction image feature,

[0140] The second determination unit is configured to determine an image corresponding to the second prediction image feature as the second prediction image.

[0141] In some embodiments, the image inverse processing model is a four-layer convolutional neural network model, and the HDR inverse processing model is a bidirectional long short-term memory recurrent neural network model.

[0142] In some embodiments, the blur processing includes any one of underexposure processing and overexposure processing.

[0143] The image processing apparatus provided in the embodiments of the present application can obtain a second image that has not undergone HDR inverse processing, by performing inverse processing corresponding to the target image processing process on the first image by using the image inverse processing model. The third image can be obtained by performing blur processing on a local region in the second image by using the HDR inverse processing model, to perform inverse processing corresponding to the HDR processing process. Since the first image is an image processed by the ISP, the image processed by the ISP can be obtained by processing the image processed by the ISP by using the image inverse processing model and the HDR inverse processing model, so that the simulation process of the ISP can be simplified and the simulation effect can be improved.

[0144] Based on the image processing method provided in the above embodiments, the embodiments of the present application further provide a specific implementation of an electronic device. Figure 5 A schematic diagram of an electronic device 500 provided in the embodiments of the present application is shown.

[0145] The electronic device 500 can include a processor 510 and a memory 520 storing computer program instructions.

[0146] In particular, the processor 510 described above can include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or can be configured to implement one or more integrated circuits that embody the embodiments of the present application.

[0147] The memory 520 can include a mass storage for data or instructions. By way of example and not limitation, the memory 520 can include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disc, a magneto-optical disc, magnetic tape, or a Universal Serial Bus (USB) drive or a combination of two or more of these. The memory 520 can include removable or non-removable (or fixed) media, where appropriate. The memory 520 can be internal or external to the integrated gateway disaster recovery device, as appropriate. In particular embodiments, the memory 520 is non-volatile, solid-state memory.

[0148] The memory can include read-only memory (ROM), random-access memory (RAM), magnetic disk storage mediums, optical storage mediums, flash memory devices, electrical, optical, or other physical / tangible memory storage devices. Thus, in general, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software that, when executed (by one or more processors), is operable to perform operations described with reference to the methods according to the first aspect of the present application.

[0149] The processor 510 implements any of the image processing methods described above in the embodiments by reading and executing the computer program instructions stored in the memory 520.

[0150] In one example, the electronic device 500 can further include a communication interface 530 and a bus 540. As shown, the processor 510, the memory 520, and the communication interface 530 are connected through the bus 540 and complete communication therebetween. Figure 5

[0151] The communication interface 530 is mainly used to realize the communication between the modules, devices, units and / or equipment in the embodiments of the present application.

[0152] ​Bus 540 includes hardware, software, or both, to couple electronic devices to each other in a manner that allows information to be passed between or among them. The bus can include, for example, an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand (IB) interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association local (VLB) bus, or another suitable bus or a combination of two or more of these. Where appropriate, bus 540 can include one or more buses. Although the example embodiments described and illustrated herein relate to a particular bus, the application contemplates any suitable bus or interconnect.

[0153] By way of example, electronic device 500 can be a cell phone, a tablet computer, a notebook computer, a handheld computer, a vehicle-mounted electronic device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc.

[0154] The electronic device can perform the image processing method in the embodiments of the application, thereby realizing the image processing method and device described in combination Figures 1 to 4 with the image processing method in the embodiments of the application.

[0155] In addition, in combination with the image processing method in the above embodiments, the embodiments of the application can provide a computer-readable storage medium to implement. The computer-readable storage medium has computer program instructions stored thereon, and the computer program instructions are executed by a processor to implement any one of the image processing methods in the above embodiments.

[0156] It needs to be made clear that the application is not limited to the specific configurations and processes described above and shown in the drawings. For the sake of brevity, detailed descriptions of well-known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the application are not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps, after understanding the spirit of the application.

[0157] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0158] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0159] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0160] The above merely describes a specific implementation of the present application. Those skilled in the art can clearly understand the specific working processes of the system, modules and units described above for the convenience and brevity of description, and can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein again. It should be understood that the protection scope of the present application is not limited to this, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application.

Claims

1. An image processing method, characterized in that, include: Acquire a first image, which is an image processed by an image signal processor (ISP). The first image is processed inversely using an image inverse processing model, corresponding to the target image processing process, to obtain the second image. The target image processing process includes all image processing steps in the ISP processing except for the high dynamic range (HDR) imaging process. The third image is obtained by blurring local regions in the second image using an HDR inverse processing model. Before using the HDR inverse processing model to blur local regions in the second image to obtain the third image, the method further includes: Obtain multiple HDR-processed third image samples and their corresponding unprocessed fourth image samples. The initial HDR inverse processing model is used to blur the local regions of each of the third image samples to obtain a second predicted image corresponding to each of the third image samples. The second loss function value is determined based on the similarity between the multiple fourth image samples and their corresponding second predicted images. The model parameters of the initial HDR inverse processing model are adjusted according to the second loss function value, and the HDR inverse processing model is trained to obtain the model.

2. The method according to claim 1, characterized in that, Before performing the inverse processing corresponding to the target image processing process on the first image using the image inverse processing model to obtain the second image, the method further includes: Obtain the first image samples after multiple ISP processing and their corresponding second image samples before ISP processing. Feature extraction is performed on the first image sample and the second image sample to obtain a first image feature corresponding to each of the first image samples and a second image feature corresponding to each of the second image samples. The first image features are processed using the inverse processing model corresponding to the target image processing, resulting in a first predicted image feature corresponding to each of the first image features. The first loss function value is determined based on the similarity between multiple second image features and their corresponding first predicted image features. The model parameters of the initial image inverse processing model are adjusted based on the first loss function value, and the image inverse processing model is trained to obtain the model.

3. The method according to claim 2, characterized in that, The step of determining the first loss function value based on the similarity between multiple second image features and their corresponding first predicted image features includes: Calculate the mean square error between multiple first predicted image features and multiple second image features. The mean squared error is determined as the value of the first loss function.

4. The method according to claim 1, characterized in that, The acquisition of multiple HDR-processed third image samples includes: Obtain multiple temporally consecutive fourth images. The image inverse processing model is used to perform inverse processing on multiple fourth images, corresponding to the target image processing process, to obtain multiple fifth images. Three consecutive frames of third image samples are obtained from the plurality of fifth images.

5. The method according to claim 4, characterized in that, The step of using an initial HDR inverse processing model to blur local regions of each of the third image samples to obtain a second predicted image corresponding to each of the third image samples includes: The initial HDR inverse processing model is used to extract the correlation features between the three consecutive third image samples. The local region of the target image sample is blurred according to the associated features to obtain a second predicted image corresponding to the target image sample, wherein the target image sample is the image sample in the middle position among the three consecutive third image samples.

6. The method according to claim 5, characterized in that, The step of blurring a local region of the target image sample based on the associated features to obtain a second predicted image corresponding to the target image sample includes: The associated features are determined as the target image features corresponding to the target image sample. The target image features are processed using a random algorithm to obtain the second predicted image features. The image corresponding to the features of the second predicted image is determined as the second predicted image.

7. The method according to claim 1, characterized in that, The image inverse processing model is a four-layer convolutional neural network model, and the HDR inverse processing model is a bidirectional long short-term memory recurrent neural network model.

8. The method according to claim 1, characterized in that, The blurring process includes either underexposure processing or overexposure processing.

9. An image processing apparatus, characterized in that, The device includes: The first acquisition module is used to acquire a first image, which is an image processed by an image signal processor (ISP). The first processing module is used to perform inverse processing on the first image corresponding to the target image processing process using an image inverse processing model to obtain a second image. The target image processing process includes all image processing processes in the ISP processing except for the high dynamic range (HDR) imaging process. The second processing module is used to blur local areas in the second image using an HDR inverse processing model to obtain a third image. The third acquisition module is used to acquire multiple HDR-processed third image samples and their corresponding HDR-unprocessed fourth image samples before blurring local regions in the second image using the HDR inverse processing model to obtain the third image. The fourth processing module is used to blur the local regions of each of the third image samples using the initial HDR inverse processing model, thereby obtaining a second predicted image corresponding to each of the third image samples. The second determining module is used to determine the second loss function value based on the similarity between the multiple fourth image samples and their corresponding second predicted images. The second training module is used to adjust the model parameters of the initial HDR inverse processing model according to the second loss function value, and train the HDR inverse processing model.

Images