Image processing method and model training method

Abstract

The invention provides an image processing method and a model training method, and relates to the technical field of artificial intelligence. The method is applied to an image processing model and comprises the steps of calling a feature extraction module, performing denoising and demosaicing processing on a target image to obtain a first feature vector, calling a feature mapping module, mapping the first feature vector to an implicit space to obtain an implicit vector Z; and calling a generation module, sampling the implicit vector Z, and performing image reconstruction based on a sampling result to obtain an RGB format image corresponding to the target image. According to the scheme, the first feature vector is mapped to the implicit space, the implicit vector Z representing probability distribution is sampled, the sampling result contains all possible data information, and all the possible data information contains data information except input data (the first feature vector). Therefore, image reconstruction is carried out based on the sampling result, and the obtained RGB format graph contains richer image information with higher resolution.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to an image processing method and a model training method. BACKGROUND

[0002] With the continuous development of electronic technology, electronic devices such as mobile phones and tablet computers gradually become common tools in people's daily life and work. At present, some electronic devices are provided with cameras, and the cameras provide users with photographing or video recording functions.

[0003] Among them, the camera is provided with an image sensor, and the image sensor records color information of pixels by using a color filter array (CFA), wherein each pixel has only one color component of RGB three primary colors, which causes a strong mosaic effect when viewed by the human eye. Therefore, the image output by the image sensor needs to be de-mosaiced to convert the image output by the image sensor into an RGB format image that meets the human visual characteristics.

[0004] At present, after the image output by the image sensor is de-mosaiced by using the existing method, the RGB format image obtained has the problems of low resolution and poor definition. SUMMARY

[0005] The present application provides an image processing method and a model training method, which aims to solve the problem that the RGB format image obtained after de-noising and de-mosaicing has low resolution and poor definition.

[0006] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0007] In a first aspect, an image processing method is provided, applied to an image processing model, the image processing model comprising a cascaded feature extraction module, a feature mapping module and a generation module. The method comprises: calling the feature extraction module to perform de-noising and de-mosaicing processing on a target image to obtain a first feature vector corresponding to a de-noising and de-mosaicing processing result, each pixel point in the target image containing one color component; calling the feature mapping module to map the first feature vector to a hidden space to obtain a hidden vector Z, the hidden vector Z representing a probability distribution of the first feature vector; calling the generation module to sample the hidden vector Z and perform image reconstruction based on the sampling result to obtain an RGB format image corresponding to the target image.

[0008] In the scheme, the first feature vector is mapped to a latent space, and the latent vector Z representing a probability distribution is sampled, and the sampling result contains all possible data information, which contains data information other than the input data (the first feature vector). Therefore, based on the sampling result, the image reconstruction is performed, and the obtained RGB format image contains more rich and high-resolution image information.

[0009] In a possible implementation of the first aspect, the image processing model further includes a scene classification submodule, an output end of the scene classification submodule is connected with an input end of the generation module, and the method further includes: calling the scene classification submodule to perform scene classification on the target image and output a second feature vector containing information of a scene to which the target image belongs; and calling the generation module to perform image reconstruction based on the sampling result and the second feature vector to obtain an RGB format image corresponding to the target image.

[0010] To improve the accuracy of the image generated by the image processing model, the embodiments of the present application propose a scheme of guiding the image generation process by using the scene to which the target image belongs. The scheme classifies the target image by scene to obtain a second feature vector containing information of the scene to which the target image belongs, and guides the reconstruction result by using the second feature vector in the image reconstruction process, which is beneficial to improve the accuracy of the image generated by the image processing model.

[0011] In a possible implementation of the first aspect, the scene classification submodule is preconfigured with scene vectors corresponding to different scenes respectively, and the method further includes: calling the scene classification submodule to determine and output the second feature vector according to the scene vector corresponding to the scene to which the target image belongs after performing scene classification on the target image.

[0012] In the embodiments of the present application, the scene vectors corresponding to different scenes are preconfigured, and the scene vectors corresponding to different scenes are one-dimensional vectors with the same length. By this preconfigured manner, the second feature vector can be quickly output, which is helpful to improve the image processing efficiency.

[0013] The second feature vector can be a matrix with the number of scenes as rows (or columns) and the length of the scene vector of each scene as columns (or rows). In the matrix, the remaining matrix elements except the scene vector of the scene to which the target image belongs are zero. Optionally, the scene vector of the scene to which the target image belongs can be located in the first row (or the first column) of the second feature vector.

[0014] In a possible implementation of the first aspect, the image processing model further includes a scene guidance sub-module, and the generation module further includes M network layers of different resolutions, and the image reconstruction is performed based on the sampling result and the second feature vector to obtain the RGB format image corresponding to the target image, including: calling the scene guidance sub-module to sequentially fuse the second feature vector into the feature vector corresponding to the mth network layer, when m traverses 1, the feature vector corresponding to the mth network layer is determined based on the sampling result, when m traverses from 2 to M, the feature vector corresponding to the mth network layer is determined based on the output of the (m-1)th network layer, and when m traverses M, the output of the mth network layer is used to determine the RGB format image corresponding to the target image.

[0015] In the embodiment of the application, the second feature vector is fused into the network layers of different resolutions respectively, so that the scene information is introduced into the process of processing the feature vector in each network layer of the generation module, and the accuracy of the image generated by the image processing model is improved.

[0016] In a possible implementation of the first aspect, the method further includes: calling the feature mapping module to map the first feature vector to the latent space to obtain a variance group and a mean group corresponding to the latent space, and obtaining the latent vector Z according to the variance group and the mean group corresponding to the latent space.

[0017] In the embodiment of the application, the first feature vector is mapped to the latent space, and more specifically, a variance group and a mean group that are exclusive to the first feature vector are generated, wherein the variance group and the mean group can determine a series of normal distributions, and the normal distributions are the basic units of the latent vector Z. It can be understood that the latent vector Z is a multi-dimensional distribution corresponding to the first feature vector.

[0018] In a possible implementation of the first aspect, the method further includes: calling the feature mapping module to map the first feature vector in the depth direction to obtain a variance group and a mean group corresponding to the latent space.

[0019] The determined elements and the sequence formed by the elements are different when the first feature vector is mapped from different dimensions, and then the latent vector Z is affected, and different latent vectors Z directly affect the image reconstruction result. In the embodiment of the application, the mapping is performed in the depth direction, so that the latent vector Z can be determined along the order of CWHN, and the latent vector Z can contain more abundant information, so that the resolution and the image quality of the RGB format image are improved when the image is reconstructed.

[0020] In a possible implementation manner of the first aspect, the implicit vector Z is subject to a normal distribution, and the method further includes: calling a feature mapping module, sampling a standard normal distribution to obtain a first parameter group; and determining the implicit vector Z according to the first parameter group, a variance group and a mean value group corresponding to the implicit space.

[0021] In a possible implementation manner of the first aspect, the variance group includes H variances, the mean value group includes H mean values, and the first parameter group includes H first parameters, where H is an integer greater than or equal to 1.

[0022] In the embodiment of the application, in order to facilitate model training, the first parameter ε is introduced, where the number of the first parameter ε is the same as the number of the mean values and the number of the variances.

[0023] In a possible implementation manner of the first aspect, the implicit vector Z is composed of a plurality of basic units, and a mathematical expression of the basic unit is: Z i = μ i + σ i ⊙ ∈ i wherein Z i is the i th basic unit, μ i is the i th mean value, σ i is the i th variance, ∈ i is the i th first parameter, and ⊙ represents an exclusive or operation.

[0024] Any d-dimensional distribution can be generated by mapping a group of d variables subject to a normal distribution through a sufficiently complex function. Based on this, in the embodiment of the application, the first feature vector is understood as an M-dimensional distribution, and it is determined that there is a group of M basic units subject to a normal distribution, and the M basic units subject to a normal distribution can obtain the M-dimensional distribution corresponding to the first feature vector through complex function mapping.

[0025] Since it is very difficult to directly determine the M-dimensional distribution corresponding to the first feature vector, it is relatively easy to extract a simple distribution, therefore, in the scheme, the first feature vector is mapped to a plurality of basic units subject to a normal distribution, and the basic units can obtain the M-dimensional distribution corresponding to the first feature vector through complex function mapping.

[0026] In a possible implementation manner of the first aspect, the size of the same-color pixels arranged together in the target image is N×N, and N is an integer greater than or equal to 2.

[0027] The greater the value of N is, the more data needs to be filled in when processing the target image into a three-channel image. In this case, the effect of denoising and demosaicking to obtain an RGB image is very fuzzy and has low resolution. In the present solution, instead of filling in the missing data based on the existing data, the probability distribution of the first feature vector is learned, and information not contained in the existing data is obtained based on the probability distribution, and an RGB image is output based on this information.

[0028] In a possible implementation of the first aspect, the target image includes multiple images.

[0029] In the present solution, when the target image includes multiple images, more context information can be obtained from the multiple images, which is conducive to improving the image quality of the generated image.

[0030] In a possible implementation of the first aspect, the image format of the target image is a raw format.

[0031] In a second aspect, an image processing model is provided, including a cascaded feature extraction module, a feature mapping module, and a generation module. The feature extraction module is configured to perform denoising and demosaicking on a target image to obtain a first feature vector corresponding to a denoising and demosaicking result. Each pixel point in the target image contains a color component. The feature mapping module is configured to map the first feature vector to a latent space to obtain a latent vector Z. The latent vector represents the probability distribution of the first feature vector. The generation module is configured to sample the latent vector Z and perform image reconstruction based on the sampling result to obtain an RGB image corresponding to the target image.

[0032] In a possible implementation of the second aspect, the image processing model further includes a scene classification submodule. An output end of the scene classification submodule is connected to an input end of the generation module. The scene classification submodule is configured to perform scene classification on the target image and output a second feature vector. The second feature vector contains information about the scene to which the target image belongs. The generation module is further configured to perform image reconstruction based on the sampling result and the second feature vector to obtain an RGB image corresponding to the target image.

[0033] In a possible implementation of the second aspect, the scene classification submodule is preconfigured with scene vectors corresponding to different scenes. After performing scene classification on the target image, the scene classification submodule determines and outputs the second feature vector according to the scene vector corresponding to the scene to which the target image belongs.

[0034] In a possible implementation of the second aspect, the image processing model further includes a scene guidance sub-module, and the generation module further includes M network layers of different resolutions, where the scene guidance sub-module is configured to sequentially fuse the second feature vector into a feature vector corresponding to an mth network layer, the feature vector corresponding to the mth network layer being determined based on the sampling result when m traverses to 1, the feature vector corresponding to the mth network layer being determined based on an output of an (m-1)th network layer when m traverses from 2 to M, and the output of the mth network layer being used to determine the RGB format image corresponding to the target image when m traverses to M.

[0035] In a possible implementation of the second aspect, the feature mapping module is specifically configured to map the first feature vector to an implicit space to obtain a variance set and a mean set corresponding to the implicit space, and obtain the implicit vector Z according to the variance set and the mean set corresponding to the implicit space.

[0036] In a possible implementation of the second aspect, the feature mapping module is specifically configured to map the first feature vector in a depth direction to obtain the variance set and the mean set corresponding to the implicit space.

[0037] In a possible implementation of the second aspect, the implicit vector Z is subject to a normal distribution, where the feature mapping module is further configured to sample a standard normal distribution to obtain a first parameter set, and determine the implicit vector Z according to the first parameter set, the variance set, and the mean set corresponding to the implicit space.

[0038] In a possible implementation of the second aspect, the variance set includes H variances, the mean set includes H means, the first parameter set includes H first parameters, and H is an integer greater than or equal to 1.

[0039] In a possible implementation of the second aspect, the implicit vector Z is composed of a plurality of basic units, and a mathematical expression of the basic unit is: Z i = i +σ i ⊙∈ i , where Z i is the i th basic unit, μ i is the i th mean, σ i is the i th variance, ∈ i is the i th first parameter, and ⊙ represents an exclusive or operation.

[0040] In a possible implementation of the second aspect, a size of the same-color pixels arranged together in the target image is N*N, and N is an integer greater than or equal to 2.

[0041] In a possible implementation of the second aspect, the target image includes a plurality of images.

[0042] In a possible implementation of the second aspect, the image format of the target image is a raw format.

[0043] In a third aspect, a model training method is provided. The method comprises: obtaining a first image sample and a second image sample, the first image sample being obtained by processing the second image sample, each pixel point in the first image sample containing a color component, and the second image sample being in an RGB format; constructing an initial image processing model, the initial image processing model comprising an initial feature extraction module, an initial feature mapping module, and an initial generation module connected in cascade; inputting the first image sample into the initial image processing model, performing denoising and demosaicking processing on the first image sample by the initial feature extraction module to obtain a first feature vector corresponding to a denoising and demosaicking processing result; mapping the first feature vector to a latent space by the initial feature mapping module to obtain a latent vector Z; sampling the latent vector Z by the initial generation module, and performing image reconstruction based on a sampling result to obtain a generated sample; calculating a KL divergence based on the latent vector Z and a standard normal distribution, and calculating a first loss based on the generated sample and the second image sample; adjusting the initial feature extraction module, the initial feature mapping module, and the initial generation module based on the KL divergence and the first loss, until the initial image processing model converges, wherein the KL divergence is used to measure the closeness between two probability distributions, and after the initial image processing model converges, the generated sample is used to determine an RGB format image corresponding to the first image sample.

[0044] In a possible implementation of the third aspect, the initial image processing model further comprises an initial discrimination module. After the generated sample is obtained, the method further comprises: discriminating the generated sample and the second image sample by the initial discrimination module; calculating a loss of the generated sample and a loss of the second image sample to obtain a second loss; and adjusting the initial feature extraction module, the initial feature mapping module, the initial generation module, and the initial discrimination module based on the KL divergence and the second loss.

[0045] In a possible implementation of the third aspect, the initial image processing model further comprises an initial scene classification submodule, an output end of the initial scene classification submodule being connected to an input end of the initial generation module. The method further comprises: performing scene classification on the first image sample by the initial scene classification submodule, and outputting a second feature vector containing information of a scene to which the first image sample belongs; and performing image reconstruction based on the sampling result and the second feature vector by the initial generation module to obtain the generated sample.

[0046] In a possible implementation manner of the third aspect, the initial scene classification sub-module is pre-configured with scene vectors corresponding to different scenes respectively, and the method further includes: determining and outputting, by the initial scene classification sub-module, the second feature vector according to the scene vector corresponding to the scene to which the first image sample belongs after performing scene classification on the first image sample.

[0047] In a possible implementation manner of the third aspect, the initial image processing model further includes an initial scene guidance sub-module, and the initial generation module further includes M initial network layers of different resolutions, and the method further includes: fusing, by the initial scene guidance sub-module, the second feature vector into the feature vector corresponding to the mth initial network layer, wherein when m traverses 1, the feature vector corresponding to the mth initial network layer is determined based on the sampling result, when m traverses from 2 to M, the feature vector corresponding to the mth initial network layer is determined based on the output of the (m-1)th initial network layer, and when m traverses to M, the output of the mth initial network layer is used to determine the generated sample.

[0048] In a possible implementation manner of the third aspect, the method further includes: mapping, by the initial feature mapping module, the first feature vector to the latent space to obtain a variance group and a mean group corresponding to the latent space, and obtaining the latent vector Z according to the variance group and the mean group corresponding to the latent space.

[0049] In a possible implementation manner of the third aspect, the method further includes: mapping, by the initial feature mapping module, the first feature vector in the depth direction to obtain a variance group and a mean group corresponding to the latent space.

[0050] In a possible implementation manner of the third aspect, the latent vector Z obeys a normal distribution, and the method further includes: sampling, by the initial feature mapping module, a standard normal distribution to obtain a first parameter group; and determining the latent vector Z according to the first parameter group, the variance group and the mean group corresponding to the latent space.

[0051] In a possible implementation manner of the third aspect, the variance group includes H variances, the mean group includes H means, the first parameter group includes H first parameters, and H is an integer greater than or equal to 1.

[0052] In a possible implementation manner of the third aspect, the latent vector Z is composed of a plurality of basic units, and a mathematical expression of the basic unit is: Z i = μ i + σ i ⊙ ∈ i , wherein Z i is the i th basic unit, μ i is the i th mean, σ i is the i th variance, and ∈ iFor the i-th first parameter, represents the exclusive-OR operation.

[0053] In a fourth aspect, an image processing apparatus is provided, including: an obtaining unit configured to obtain a target image; a processing unit configured to perform denoising and demosaicking processing on the target image to obtain a first feature vector corresponding to a denoising and demosaicking processing result, each pixel point in the target image containing one color component; map the first feature vector to a latent space to obtain a latent vector Z, the latent vector representing a probability distribution of the first feature vector; sample the latent vector Z, and perform image reconstruction based on a sampling result to obtain an RGB format image corresponding to the target image.

[0054] In a fifth aspect, a model training apparatus is provided, including: an obtaining unit configured to obtain a first image sample and a second image sample, the first image sample being obtained by processing the second image sample, each pixel point in the first image sample containing one color component, and the second image sample being in an RGB format; a processing unit configured to construct an initial image processing model, the initial image processing model including an initial feature extraction module, an initial feature mapping module, and an initial generation module connected in cascade; input the first image sample into the initial image processing model, perform denoising and demosaicking processing on the first image sample by the initial feature extraction module to obtain a first feature vector corresponding to a denoising and demosaicking processing result; map the first feature vector to a latent space by the initial feature mapping module to obtain a latent vector Z; sample the latent vector Z by the initial generation module, and perform image reconstruction based on a sampling result to obtain a generated sample; calculate a KL divergence based on the latent vector Z and a standard normal distribution, and calculate a first loss based on the generated sample and the second image sample, adjust the initial feature extraction module, the initial feature mapping module, and the initial generation module based on the KL divergence and the first loss, until the initial image processing model converges, wherein the KL divergence is used to measure the closeness between two probability distributions, and the generated sample is used to determine an RGB format image corresponding to the first image sample after the initial image processing model converges.

[0055] In a sixth aspect, an electronic device is provided, including a processor and a memory, the memory being configured to store program instructions, and the processor being configured to invoke the program instructions to execute the image processing method in the first aspect or any possible implementation manner of the first aspect, or the model training method in the third aspect or any possible implementation manner of the third aspect.

[0056] In a seventh aspect, a computer readable storage medium is provided, the computer readable storage medium storing program code for execution by a device, the program code including instructions for performing the image processing method in the first aspect or any possible implementation manner of the first aspect, or the model training method in the third aspect or any possible implementation manner of the third aspect.

[0057] On the basis of the implementation manners of the above aspects, the application can be further combined to provide more implementation manners. The technical effects obtained by the second aspect, the third aspect, the fourth aspect, the fifth aspect, the sixth aspect and the seventh aspect are similar to the technical effects obtained by the corresponding technical means in the first aspect, and thus will not be described herein. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 A structural schematic diagram of a camera is shown.

[0059] Figure 2 A schematic diagram of a CFA array on an image sensor is shown.

[0060] Figure 3 A demosaicing schematic diagram is shown.

[0061] Figure 4 A schematic diagram of a monochrome pixel array in the 4x4 pattern shown in (c) of FIG. 1 is shown. Figure 2

[0062] A structural schematic diagram of an electronic device is shown. Figure 5

[0063] A software structural block diagram of an electronic device according to an embodiment of the application is shown. Figure 6

[0064] An interface schematic diagram of an application scenario provided by an embodiment of the application is shown. Figure 7

[0065] A flowchart of an image processing method provided by an embodiment of the application is shown. Figure 8

[0066] A schematic diagram of an image processing model provided by an embodiment of the application is shown. Figure 9

[0067] A schematic diagram of another image processing model provided by an embodiment of the application is shown. Figure 10

[0068] A schematic diagram of an arrangement format of a first feature vector is shown. Figure 11

[0069] A flowchart of another image processing method provided by an embodiment of the application is shown. Figure 12

[0070] A schematic diagram of another image processing model provided by an embodiment of the application is shown. Figure 13

[0071] Figure 14 ​A flowchart of a model training method is shown.

[0072] Figure 15 An optional structure diagram of an image processing apparatus is shown.

[0073] Figure 16 An optional structure diagram of a model training apparatus is shown.

[0074] Figure 17 A hardware structure diagram of an electronic device is shown. DETAILED DESCRIPTION

[0075] The technical solutions in the present application will be described below with reference to the drawings.

[0076] In order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, the same items or similar items with basically the same function and role are distinguished by using "first", "second", etc. For example, the first chip and the second chip are only used to distinguish different chips, and do not limit the sequence. Those skilled in the art can understand that "first", "second", etc. do not limit the number and execution order, and "first", "second", etc. also do not necessarily mean different.

[0077] It should be noted that in the embodiments of the present application, "exemplary" or "for example" is used to represent an example, illustration or description. Any embodiment or design scheme described as "exemplary" or "for example" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the use of "exemplary" or "for example" is intended to present the relevant concept in a specific manner.

[0078] In the embodiments of the present application, "at least one" means one or more, and "multiple" means two or more. The "and / or" describes the association relationship between the associated objects, which means that there can be three kinds of relationships, for example, A and / or B, which can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single item or multiple items. For example, at least one of a, b, or c, can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0079] At present, some electronic devices are provided with cameras, and in the process of using the electronic devices, the cameras in the electronic devices can be used for photographing or video recording.

[0080] As shown in Figure 1 , Figure 1 a structural schematic diagram of a camera is shown. The camera can include a lens 10 and an image sensor 20 arranged along an optical path direction, the lens 10 can include a plurality of optical lenses stacked along an optical axis direction, and the image sensor 20 can also be referred to as a camera sensor.

[0081] The image sensor 20 includes a filter 21 and a photosensitive element 22, and the filter 21 is located between the lens 10 and the photosensitive element 22. The light reflected by the object is sequentially projected onto the photosensitive element 22 after passing through the lens 10 and the filter 21, and the photosensitive element 22 converts the optical signal into an electrical signal for imaging.

[0082] A color filter array (CFA) is arranged on the pixel array in the conventional image sensor, and the CFA generally adopts a red-green-blue three-color pixel cross-distribution, as shown in Figure 2 (a), Figure 2 The CFA on the pixel array shown in (a) is different in color between two adjacent pixels, and the size of the same-color pixels arranged together is 1x1, so Figure 2 the pixel array shown in (a) is also referred to as a 1x1 pattern.

[0083] With the development of imaging technology, along with the emergence of more and more high-resolution cameras, a new pixel arrangement technology has emerged, which arranges N adjacent same-color pixels together to form a pixel with an area N times larger than the original pixel, N being an integer greater than 1. For example, N can be equal to 4, which arranges four adjacent same-color pixels together to form a pixel with an area four times larger than the original pixel. As shown in Figure 2 (b), Figure 2 the pixel array shown in (b) is also referred to as a quadra bayer pattern, in which the row and column of the same-color pixels are both 2, so it is also referred to as a 2x2 pattern. In the quadra bayer pattern, the positions of the RGB three filters can be exchanged.

[0084] The pixel array of the image sensor includes a plurality of pixel sets 23, and each pixel set 23 in the plurality of pixel sets 23 includes four pixel units, wherein the arrangement of the pixel array in each pixel set 23 can be as shown in Figure 2 (c), Figure 2(c) shows four pixel units, namely first pixel unit 231, second pixel unit 232, third pixel unit 233, and fourth pixel unit 234. At least some of these four pixel units are pixel units of different colors, and each pixel unit contains 4 rows and 4 columns of pixels of the same color. Figure 2 The pixel array shown in (c) is also called a 4×4 pattern. For example, the first pixel unit 231 includes 16 red (R) pixels, the second pixel unit 232 includes 16 green (G) pixels, the third pixel unit 233 includes 16 green (G) pixels, and the fourth pixel unit 234 includes 16 blue (B) pixels.

[0085] It can be seen that, Figure 2 In the pixel arrays shown in (a), (b), and (c), each pixel contains only one of the three primary colors of RGB. This means that in the color image output by the image sensor, each pixel has only one color component. This results in a strong mosaic effect when the human eye directly views the color image output by the image sensor. Therefore, it is necessary to convert the raw format color image output by the image sensor (with the .raw extension) into an RGB format color image that conforms to the visual characteristics of the human eye. This process can be understood as filling in the missing two color components for each pixel, which is called demosaicing (or de-mosaicing, demosaicking, or debayering).

[0086] like Figure 3 As shown, Figure 3 A schematic diagram of de-mosaic is shown. De-mosaic refers to the process of recovering / reconstructing a full-color image from the incompletely sampled color signal output by the color filter array photosensitive device. In other words, it converts a single-channel color image into a three-channel RGB color image. Furthermore, the raw image inside the camera is unprocessed image data, typically containing unresolved sensor data and additional metadata. Due to the effects of electromagnetic and thermal effects on the image sensor, the camera usually introduces noise during image capture; this noise is also removed during the de-mosaic process.

[0087] by Figure 2 Taking the pixel array shown in (a) as an example, the missing two color components can be interpolated simply by averaging the surrounding pixels, and then converted to a 3-channel display to easily achieve raw to RGB conversion. However, for... Figure 2In terms of the 2x2 pattern shown in (b) in FIG. 1, it can be seen that the span of R, G, B in space is 4, and the missing value in the middle is 4 times that of the 1x1 pattern. In terms of the interpolation algorithm, the difficulty will also increase sharply.

[0088] As shown in Figure 4 , Figure 4 It is shown Figure 2 that the schematic diagram of the monochrome pixel array in the 4x4 pattern shown in (c) in FIG. 1. It can be seen that the spatial span in each monochrome pixel array becomes 8, and the number of missing values in the middle is 16 times that of the 1x1 pattern. This results in a sharp increase in difficulty in using traditional interpolation algorithms for denoising and demosaicking, and because the number of consecutive missing parts is too large, the resolution of the RGB image after demosaicking is low, and the image is blurred.

[0089] With the development of image sensor technology, the resolution of the camera will be higher and higher, and subsequent 5x5 patterns, 6x6 patterns and pixel arrays with larger sizes of the same color pixels may also appear, such as NxN pattern. When the value of N becomes larger and larger, the number of missing values will become larger and larger, and the span between the missing values will become higher and higher. Using traditional demosaicking schemes to convert the raw format color image of the N x N pattern into an RGB format color image will become more and more difficult.

[0090] To solve this problem, the present application proposes an image processing method, which comprises denoising and demosaicking a target image to obtain a first feature vector; then mapping the first feature vector to an implicit space, thereby determining a series of normal distributions, and the implicit vector Z formed by the normal distributions represents the probability distribution of the input data (the first feature vector). On this basis, the implicit vector Z is sampled, and the sampling result contains all possible data information, which contains data information other than the input data (the first feature vector). Therefore, the RGB format image reconstructed based on the sampling result contains more abundant and higher resolution image information.

[0091] The image processing method provided by the embodiments of the present application can be applied in an electronic device with a camera. The electronic device includes a terminal device, which can also be referred to as a terminal, a user equipment (UE), a mobile station (MS), a mobile terminal (MT), etc. The electronic device can be a mobile phone, a smart television, a wearable device, a tablet computer (Pad), a computer with wireless transceiver function, a virtual reality (VR) electronic device, an augmented reality (AR) electronic device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc. The embodiments of the present application do not limit the specific technology and specific device form of the electronic device.

[0092] In order to better understand the embodiments of the present application, the structure of the electronic device of the embodiments of the present application is introduced as follows.

[0093] Figure 5 The structure of the electronic device is shown. The electronic device 500 can include a processor 510, an external memory interface 520, an internal memory 521, a universal serial bus (USB) interface 530, a charge management module 540, a power management module 541, a battery 542, an antenna 1, an antenna 2, a mobile communication module 550, a wireless communication module 560, an audio module 570, a loudspeaker 570A, a receiver 570B, a microphone 570C, a headset interface 570D, a sensor module 580, a key 590, a motor 591, an indicator 592, a camera 593, a display screen 594, and a subscriber identification module (SIM) card interface 595, etc. The sensor module 580 can include a motion sensor 580A.

[0094] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 500. In other embodiments of the present application, the electronic device 500 can include more or fewer components than shown, or combine certain components, or split certain components, or different arrangement of components. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0095] The processor 510 can include one or more processing units, for example: the processor 510 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices, or can be integrated in one or more processors.

[0096] The controller can generate operation control signals according to instruction operation codes and timing signals, and complete the control of fetching instructions and executing instructions.

[0097] The memory in the processor 510 can also be provided for storing instructions and data. In some embodiments, the memory in the processor 510 is a cache memory. The memory can save instructions or data that the processor 510 has just used or repeatedly uses. If the processor 510 needs to use the instructions or data again, it can be called from the memory. Avoiding repeated access reduces the waiting time of the processor 510, thus improving the efficiency of the system.

[0098] The charging management module 540 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 540 can receive the charging input of the wired charger through the USB interface 530. In some wireless charging embodiments, the charging management module 540 can receive the wireless charging input through the wireless charging coil of the electronic device 500. The charging management module 540 charges the battery 542, and at the same time, it can also supply power to the electronic device through the power management module 541.

[0099] The power management module 541 is configured to connect the battery 542 and the charging management module 540 to the processor 510. The power management module 541 receives input from the battery 542 and / or the charging management module 540 to supply power to the processor 510, the internal memory 521, the display screen 594, the camera 593, the wireless communication module 560, and the like. The power management module 541 can also be configured to monitor parameters such as battery capacity, battery cycle count, battery health status (leakage, impedance), and the like. In some embodiments, the power management module 541 can also be disposed in the processor 510. In some other embodiments, the power management module 541 and the charging management module 540 can also be disposed in the same device.

[0100] The wireless communication function of the electronic device 500 can be implemented by the antenna 1, the antenna 2, the mobile communication module 550, the wireless communication module 560, the modem processor, and the baseband processor, and the like.

[0101] The antenna 1 and the antenna 2 are configured to transmit and receive electromagnetic wave signals. The mobile communication module 550 can provide a solution for wireless communication including 2G / 3G / 4G / 5G and the like applied to the electronic device 500. The mobile communication module 550 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), and the like. In some embodiments, at least part of the functions of the mobile communication module 550 can be disposed in the same device as at least part of the modules of the processor 510.

[0102] The wireless communication module 560 can provide a solution for wireless communication including wireless local area networks (WLAN) (e.g., wireless fidelity (Wi-Fi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, and the like, applied to the electronic device 500. The wireless communication module 560 can be one or more devices that integrate at least one communication processing module. The wireless communication module 560 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering on the electromagnetic wave signals, and transmits the processed signals to the processor 510. The wireless communication module 560 can also receive signals to be transmitted from the processor 510, perform frequency modulation and amplification thereon, and radiate the signals as electromagnetic waves via the antenna 2.

[0103] In some embodiments, antenna 1 and mobile communication module 550 of electronic device 500 are coupled, and antenna 2 and wireless communication module 560 are coupled, so that electronic device 500 can communicate with a network and other devices through wireless communication technology.

[0104] Electronic device 500 implements a display function through a GPU, display screen 594, and an application processor, etc. The GPU is a microprocessor for image processing, connected to display screen 594 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 510 can include one or more GPUs that execute program instructions to generate or change display information.

[0105] Display screen 594 is used to display images, display videos, and receive sliding operations, etc. Display screen 594 includes a display panel. The display panel can adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light emitting diode (QLED), etc. In some embodiments, electronic device 500 can include 1 or N display screens 594, N being a positive integer greater than 1.

[0106] Electronic device 500 can implement a shooting function through an ISP, camera 593, a video codec, a GPU, display screen 594, and an application processor, etc.

[0107] The ISP is used to process data fed back by camera 593. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing to convert it into an image visible to the naked eye. The ISP can also optimize algorithms for image noise, brightness, and skin color. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be disposed in camera 593.

[0108] The camera 593 is used to capture still images or videos. An object projects an optical image through a lens to a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into a standard RGB, YUV, or the like format image signal. In some embodiments, the electronic device 500 can include one or N cameras 593, where N is a positive integer greater than one.

[0109] The digital signal processor is used to process digital signals, in addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 500 is in frequency selection, the digital signal processor is used to perform Fourier transform on frequency energy, etc.

[0110] The video codec is used to compress or decompress digital video. The electronic device 500 can support one or more video codecs. In this way, the electronic device 500 can play or record videos in multiple encoding formats, such as moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0111] The NPU is a neural-network (NN) computing processor, which is inspired by the structure of biological neural networks, such as the transmission mode between human brain neurons, and can quickly process input information and continuously self-learn. Through the NPU, the electronic device 500 can realize intelligent cognition applications, such as image recognition, face recognition, speech recognition, text understanding, etc.

[0112] The external memory interface 520 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 500. The external memory card communicates with the processor 510 through the external memory interface 520 to realize data storage functions. For example, music, video, and other files are saved in the external memory card.

[0113] The internal memory 521 can be used to store computer-executable program codes, which include instructions. The internal memory 521 can include a program storage area and a data storage area. The program storage area can store an operating system, application programs (such as a sound playing function, an image playing function, etc.) required by at least one function, etc. The data storage area can store data (such as audio data, a phone book, etc.) created during use of the electronic device 500, etc. In addition, the internal memory 521 can include a high-speed random access memory, and can further include a non-volatile memory such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 510 executes various function applications and data processing of the electronic device 500 by running instructions stored in the internal memory 521 and / or instructions stored in a memory disposed in the processor.

[0114] The electronic device 500 can implement an audio function through an audio module 570, a speaker 570A, a receiver 570B, a microphone 570C, an earphone interface 570D, an application processor, etc. For example, music playing, recording, etc.

[0115] The motion sensor 580A can include at least one of an acceleration sensor, a gyroscope sensor, and a speed sensor.

[0116] The acceleration sensor can detect the magnitude of acceleration of the electronic device 500 in each direction (generally, three axes), and thus can monitor the motion state of the electronic device 500. When the electronic device 500 is stationary, the magnitude and direction of gravity can be detected. The acceleration sensor can also be used to identify the posture of the electronic device, and can be applied to a landscape / portrait screen switching application, a pedometer, etc.

[0117] The gyroscope sensor can be used to determine the motion posture of the electronic device 500. In some embodiments, the angular velocity of the electronic device 500 around three axes (i.e., x, y, and z axes) can be determined through the gyroscope sensor, and thus the motion state of the electronic device 500 can be monitored. The gyroscope sensor can be used for shooting anti-shake. For example, when the shutter is pressed, the gyroscope sensor detects the angle of shaking of the electronic device 500, and calculates the distance that the lens module needs to compensate according to the angle, so that the lens can offset the shaking of the electronic device 500 through reverse movement, thereby achieving anti-shake. The gyroscope sensor can also be used for navigation and motion sensing game scenarios.

[0118] The speed sensor can be used to detect the magnitude of speed of the electronic device 500 in each direction (generally, three axes), and thus can monitor the motion state of the electronic device 500.

[0119] The keys 590 include a power key, a volume key, and the like. The keys 590 can be mechanical keys. Alternatively, the keys 590 can be touch keys. The electronic device 500 can receive a key input, and generate a key signal input related to user settings and function control of the electronic device 500.

[0120] The motor 591 can generate a vibration prompt. The motor 591 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations on different application programs (e.g., taking pictures, playing audio, and the like) can correspond to different vibration feedback effects. Touch operations on different regions of the display screen 594 can also correspond to different vibration feedback effects of the motor 591. Different application scenarios (e.g., time reminders, received messages, alarms, games, and the like) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0121] The indicator 592 can be an indicator light, which can be used to indicate a charging state, a power change, and can also be used to indicate a message, a missed call, a notification, and the like.

[0122] The SIM card interface 595 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 595 to achieve contact and separation with the electronic device 500. The electronic device 500 can support one or N SIM card interfaces, and N is a positive integer greater than 1. The SIM card interface 595 can support a Nano SIM card, a Micro SIM card, a SIM card, and the like. The same SIM card interface 595 can simultaneously insert multiple cards. The types of the multiple cards can be the same or different. The SIM card interface 595 can also be compatible with different types of SIM cards. The SIM card interface 595 can also be compatible with external storage cards. The electronic device 500 interacts with a network through the SIM card to implement functions such as calling and data communication. In some embodiments, the electronic device 500 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 500 and cannot be separated from the electronic device 500.

[0123] The software system of the electronic device 500 can use a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. The embodiments of the present application take an Android system with a layered architecture as an example to exemplarily illustrate the software structure of the electronic device 500.

[0124] Figure 6A software structure block diagram of the electronic device 500 of embodiments of the present application is shown. The layered architecture divides the software into several layers, each of which has a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom, the application layer, the application framework layer, the Android runtime and system library, the hardware abstraction layer, and the kernel layer.

[0125] The application layer can include a series of application packages. As shown, the application packages can include applications such as a camera, settings, and a calendar. Figure 6

[0126] The camera application is an application with the functions of taking pictures and recording videos, and the electronic device can respond to a user's operation of opening the camera application to take pictures or record videos. It can be understood that the picture taking and video recording functions of the camera application can also be called by other applications.

[0127] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications of the application layer. The application framework layer includes some pre-defined functions.

[0128] As shown, the application framework layer can also include a camera service, which can be called by the camera application to implement functions such as picture taking or video recording. Figure 6

[0129] In some embodiments, the application framework layer can include a camera access interface and a Java native interface (JNI). The camera access interface can be used to provide an interface for accessing the camera, and the JNI interface provides a number of APIs that enable communication between Java and other languages.

[0130] The entire camera service is running at runtime, which can be roughly divided into a client process and a server process, and the inter-process communication between the two is implemented using the binder mechanism. The binder mechanism is an inter-process communication (IPC) mechanism, the client process refers to the party that initiates the process request, and the server process refers to the process that is requested to execute the camera service.

[0131] In addition, as shown, the application framework layer can also include a window manager, a content provider, a resource manager, and a view system, etc. Figure 6

[0132] ​​​The window manager is used to manage windows programs. The window manager can acquire the display screen size, determine whether there is a status bar, lock the screen, and capture the screen, etc.

[0133] The content provider is used to store and acquire data, and make the data accessible to application programs. The data can include videos, images, audios, dialed and received calls, browsing history and bookmarks, phone books, etc.

[0134] The resource manager provides various resources for application programs, such as localized strings, icons, pictures, layout files, video files, etc.

[0135] The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build application programs. A display interface can be composed of one or more views. For example, a display interface including a short message notification icon can include a view for displaying text and a view for displaying pictures.

[0136] The Android runtime includes a core library and a virtual machine. The Android runtime is responsible for scheduling and managing the Android system.

[0137] The core library contains two parts: one part is the function function that the java language needs to call, and the other part is the core library of Android.

[0138] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the java files of the application layer and the application framework layer into binary files. The virtual machine is used to perform object lifecycle management, stack management, thread management, security and exception management, and garbage collection, etc.

[0139] The system library can include multiple functional modules. For example: surface manager, media library, three-dimensional graphics processing library (such as: OpenGL ES), two-dimensional graphics engine (such as: SGL), etc.

[0140] The surface manager is used to manage the display subsystem, and provides 2D and 3D layer fusion for multiple application programs.

[0141] The media library supports multiple commonly used audio, video format playback and recording, and static image files, etc. The media library can support multiple audio and video encoding formats, such as: MPEG2, H.262, MP3, AAC, AMR, JPG, PNG, etc.

[0142] The three-dimensional graphics processing library is used to realize three-dimensional graphics drawing, image rendering, synthesis and layer processing, etc. The two-dimensional graphics engine is a drawing engine for 2D drawing.

[0143] The hardware abstraction layer is an abstract layer between the kernel layer and the Android runtime. The hardware abstraction layer can be a package of hardware drivers of the kernel layer, and provides a calling interface for the application framework layer.

[0144] In the embodiments of the present application, the hardware abstraction layer can include a camera hardware abstraction module (camera HAL). In some embodiments, the camera hardware abstraction module is configured to determine a light sensitivity gain ISOmap according to a light sensitivity of the camera.

[0145] The kernel layer is a layer between hardware and software. The kernel layer at least includes a camera driver, a sensor driver, and a display driver. In some embodiments, the camera driver is configured to control the camera to run, the sensor driver is configured to control the motion sensor to run, and the display driver is configured to control the display screen to display an image.

[0146] The hardware can be a camera, a motion sensor, and a display screen. In the embodiments of the present application, the camera can be a front camera or a rear camera.

[0147] It should be noted that, although the embodiments of the present application are described with reference to the Android system, the principle of the image processing method is also applicable to electronic devices running iOS or windows operating systems.

[0148] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with reference to specific embodiments. The following specific embodiments can be implemented independently, or can be combined with each other. For the same or similar concepts or processes, some embodiments can not be described again.

[0149] In order to facilitate understanding, the specific application scenarios of the embodiments of the present application are described below.

[0150] Application scenario one: field of photographing of intelligent terminal

[0151] In an embodiment, the CFA of the camera of the intelligent terminal is an NxN pattern. When the object is photographed by the camera of the intelligent terminal, the raw format image output by the image sensor of the camera is jointly denoised and demosaicked by the image processing method provided in the embodiments of the present application, and then the first feature vector after denoising and demosaicking is mapped to the implicit space to obtain an implicit vector Z. The implicit vector Z is sampled, and the image is reconstructed based on the sampling result, so as to convert the raw format image into an RGB format image (or an RGB format video) conforming to the human visual characteristics.

[0152] Application scenario two: autonomous driving field

[0153] In an embodiment, the image processing method provided in the embodiments of the present application can be applied to the field of autonomous driving. For example, it can be applied to the navigation system of an autonomous vehicle. Through the image processing method in the present application, the surrounding environment can be photographed by the camera on the autonomous vehicle during the navigation process of the autonomous vehicle driving on the road. Then, the raw format image output by the image sensor of the camera is jointly denoised and demosaicked through the image processing method provided in the embodiments of the present application. Then, the first feature vector after denoising and demosaicking is mapped to the implicit space to obtain the implicit vector Z. The implicit vector Z is sampled, and the image is reconstructed based on the sampling result, so as to convert the raw format image into the RGB format image (or RGB format video) conforming to the human visual characteristics, thereby realizing the safety of the autonomous vehicle.

[0154] Application scenario three: field of safe city

[0155] In an embodiment, the image processing method provided in the embodiments of the present application can be applied to the field of safe city, such as the field of security. For example, the image processing method provided in the embodiments of the present application can be applied to the monitoring image processing of the safe city, such as the raw format image collected by the monitoring device in the public place. Through the image processing method in the present application, the collected raw format image can be jointly denoised and demosaicked. Then, the first feature vector after denoising and demosaicking is mapped to the implicit space to obtain the implicit vector Z. The implicit vector Z is sampled, and the image is reconstructed based on the sampling result, so as to convert the raw format image into the RGB format image (or RGB format video) conforming to the human visual characteristics, thereby obtaining the high-resolution street view image or video.

[0156] It should be noted that the image processing method provided in the embodiments of the present application is also applicable to the expansion, limitation, explanation and description of the image processing method in the related embodiments in the following Figures 8 to 13 embodiments, which will not be repeated here. It should be understood that the above is an example of the application scenario, and does not limit the application scenario of the present application.

[0157] In order to facilitate understanding, the embodiments of the present application take a mobile phone as an electronic device, and first illustrate the application scenario of the image processing method in combination with some user interfaces shown in the embodiments of the present application.

[0158] When the user lights up the screen of the electronic device and controls the electronic device to be in an unlocked state, the electronic device can display a user interface as shown in FIG. 1A. Figure 7The first interface 701 shown in (a) is an example. The first interface 701 can be the desktop of an electronic device, on which icons of multiple installed applications are displayed, such as file management application icon, email application icon, weather application icon, calculator application icon, clock application icon, voice recorder application icon, music application icon, settings application icon, address book application icon, phone application icon, messaging application icon, and camera application icon 7011, etc.

[0159] Users can perform touch operations on the camera application icon 7011, such as tapping or long-pressing, so that the electronic device receives the user's touch operation on the camera application icon 7011 and launches the camera application in response to the touch operation.

[0160] After the camera application is launched, the camera can capture images of the target. The camera is equipped with a CFA array, where pixels of the same color are N×N. The electronic device is configured with the image processing model provided in this embodiment. The camera outputs an N×N raw image, which is then input into the image processing model. After processing by the image processing model, an RGB format image is output, which the electronic device can display as shown in the image processing module. Figure 7 The second interface 702 shown in (b) is a preview interface provided by a camera application to implement shooting functions, including a preview frame 7021, shooting controls 7022, and function controls corresponding to various shooting modes.

[0161] Understandable Figure 7 (a) and Figure 7 The interface shown in (b) is merely an example of the user interface in which an electronic device launches a camera application by touching the camera application icon, and does not constitute a limitation on the embodiments of this application.

[0162] In another scenario, users can also access the camera application of an electronic device by calling the corresponding interface through a third-party application installed on the device, thereby launching the camera application. After the camera application is launched, the electronic device can execute the process corresponding to the image processing method provided in the embodiments of this application, and display the processed RGB format image in real time on the preview interface of the camera application.

[0163] In another scenario, the camera application can be launched when it switches from running in the background to running in the foreground. After the camera application is launched, the electronic device can execute the process corresponding to the image processing method provided in the embodiments of this application, and display the processed RGB format image in real time on the preview interface of the camera application.

[0164] The camera application running in the foreground refers to that the camera application is currently displayed on the screen of the electronic device and is running; and the camera application running in the background refers to that the camera application is running in the background of the system, and at this time, the electronic device does not display the interface corresponding to the camera application.

[0165] It should be noted that when the camera application is switched from running in the foreground to running in the background, the process corresponding to the image processing method provided in the embodiments of the present application can not be executed.

[0166] The image processing method and the image processing model provided in the embodiments of the present application will be described below.

[0167] As shown in Figure 8 and Figure 9 , the present application provides an image processing method, Figure 8 a flowchart of an image processing method provided in the embodiments of the present application is shown, Figure 9 a schematic diagram of an image processing model provided in the embodiments of the present application is shown. The method can be applied in an electronic device, and the electronic device can include a camera application and a camera. The size of the same color pixels in the set CFA array on the camera is N x N.

[0168] The image processing model includes a cascaded feature extraction module, a feature mapping module and a generation module. The feature extraction module has the functions of joint denoising and demosaicking the target image, and transmits the first feature vector corresponding to the denoising and demosaicking result to the feature mapping module. The feature mapping module maps the first feature vector to the implicit space and obtains the implicit vector Z. The process of mapping the first feature vector to the implicit space can be understood as extracting a simple distribution from the first feature vector. The extracted simple distribution can be combined to form a multi-dimensional distribution corresponding to the first feature vector, that is, the implicit vector Z. Then the generation module samples the implicit vector Z, and reconstructs the image based on the sampling result. The sampling result contains all possible data information, and the all possible data information contains data information other than the input number (the first feature vector). Therefore, the image reconstructed based on the sampling result can represent more rich and detailed image information, and improve the resolution of the generated RGB format image.

[0169] The image processing method provided in the embodiments of the present application includes steps 801 to 804.

[0170] Step 801, obtaining a target image.

[0171] In the embodiments of the present application, each pixel point in the target image contains a color component, such as Figure 2The target image can be a multi-frame image. It can be understood that, compared with a single-frame image, more and richer image information can be obtained by processing a multi-frame image in an image processing process.

[0172] Optionally, in the embodiments of the present application, the size of the same-color pixels arranged together in the target image is N*L, where L is an integer greater than 1. That is, the same-color pixels arranged together form a non-square color block.

[0173] Optionally, in the embodiments of the present application, the size of the same-color pixels arranged together in the target image is N*N, where N is an integer greater than or equal to 2. For example, when N=4, the pixels in the target image can be arranged as shown in (c) of FIG. 1. Figure 2

[0174] Optionally, in the embodiments of the present application, the target image can be a multi-frame image. It can be understood that, compared with a single-frame image, more and richer image information can be obtained by processing a multi-frame image in an image processing process.

[0175] In the embodiments of the present application, when the target image is a multi-frame image, the multi-frame image can refer to continuous multi-frame images, or the multi-frame image can refer to multi-frame images corresponding to the same scene or the same target, or the multi-frame image can be discontinuous multi-frame images arranged in a time sequence according to a chronological relationship.

[0176] Optionally, in the embodiments of the present application, the electronic device can obtain the target image in real time based on the camera of the electronic device after the user starts the camera application. The target image is, for example, a multi-frame N*N raw format image. Alternatively, the electronic device can store the image in a raw format (i.e., a raw format image) and allow the camera application or other third-party application in the electronic device to extract the raw image, so as to obtain the target image. Alternatively, the electronic device can receive the target image sent by another electronic device.

[0177] In the embodiments of the present application, after obtaining the target image, the electronic device can perform preprocessing on the target image. The preprocessing process includes, for example, black level correction, white balance correction, and the like.

[0178] When the target image is a multi-frame image, after the black level correction and the white balance correction, the electronic device can further determine a reference frame from the multi-frame image, perform registration processing on the multi-frame image based on the reference frame, and obtain an ISOmap of the reference frame. The ISOmap of the reference frame can be understood as the light gain of the reference frame.

[0179] ​The reference frame can be a clearest frame of the multiple frames of images, or a first frame of the multiple frames of images. The reference frame can be understood as a frame of images that needs to be processed. The frames of images other than the reference frame in the multiple frames of images are used to enrich image information in the process of image processing.

[0180] The registration processing of the multiple frames of images based on the reference frame can refer to using a KLT feature point tracking algorithm to register or align the multiple frames of images of the same size. The purpose of this process is to find corresponding feature points in the multiple frames of images, and then use the motion information of the feature points to align or register the images in the same coordinate system. In the KLT registration process, if there are unregistered image frames, the unregistered image frames in the target image are set to 0, and the image frames set to 0 are still part of the target image and enter the subsequent image processing step.

[0181] ISO is a unit of sensitivity. Sensitivity indicates the sensitivity of the camera in recording light in shooting. The higher the sensitivity, the higher the ability of the camera to record light, and it is easier to record low brightness scene information, but it will also increase the noise. The lower the sensitivity, the weaker the ability of the camera to record light, and the dark details in the scene are also easy to lose. In the embodiment of the application, the electronic device can obtain the ISO value of the reference frame from the front-end application, and then obtain the ISOmap of the reference frame according to the ISO value of the reference frame. Alternatively, the ISOmap of the reference frame is determined according to the ratio of the ISO value of the reference frame to the predefined highest ISO value. The predefined highest ISO value is, for example, 12800.

[0182] In the image processing process, the ISOmap of the reference frame can adjust the exposure rate of the image, thereby helping to improve the picture quality of the final processed RGB format image.

[0183] In the embodiment of the application, the ISOmap of the reference frame is the same size as the target image, and the electronic device can obtain the ISOmap of the target image by adjusting the ISOmap of the reference frame. Figure 1 The input image processing model is used to perform image processing.

[0184] In step 802, the feature extraction module is called to perform denoising and demosaicing processing on the target image to obtain a first feature vector.

[0185] In the embodiment of the application, the electronic device can call the feature extraction module to perform denoising and demosaicing on the target image based on the algorithm preset in the feature extraction module to obtain a denoising and demosaicing result.

[0186] In the embodiment of the present application, after the target image is denoised and demosaicked, the denoising and demosaicking result is not output as an image, but is represented in the form of a first feature vector and transmitted to a feature mapping module of the image processing model for subsequent processing operations.

[0187] The first feature vector obtained after the target image is denoised and demosaicked contains three-channel color information.

[0188] The preset algorithm in the feature extraction module may include, for example, an interpolation algorithm, an image inpainting algorithm, a deep learning algorithm, etc.

[0189] At step 803, the feature mapping module is invoked to map the first feature vector to a latent space to obtain a latent vector Z.

[0190] The latent vector Z represents the probability distribution of the first feature vector.

[0191] Optionally, in the embodiment of the present application, the latent vector Z follows a normal distribution.

[0192] At step 804, the generation module is invoked to sample the latent vector Z, and an RGB format image corresponding to the target image is reconstructed based on the sampling result.

[0193] Before describing the image processing model of the present application, a brief description of a conventional model is first given. In the conventional model, an encoder is generally used to encode the input data to extract key information and obtain an intermediate variable h, and then a decoder is used to reconstruct the original data based on the intermediate variable h. The intermediate variable h is a fixed value, or it can be understood that the encoder can only learn the existing information contained in the input data, but cannot learn the information not contained in the input data.

[0194] The image processing model provided in the embodiment of the present application is different from the conventional model. The image processing model provided in the embodiment of the present application not only learns the existing information contained in the input data, but also learns the probability distribution of the input data. The probability distribution of the input data represents an uncertain variable, which can contain more and richer information than the fixed intermediate variable in the conventional model. Based on the probability distribution, data information not contained in the input data can also be obtained.

[0195] It can be understood that the image processing model provided in the embodiment of the present application is to determine the probability distribution of the input data (the first feature vector), which contains information not contained in the input data. Based on the sampling of the probability distribution and the image reconstruction based on the sampling result, the RGB format image obtained contains richer and higher-resolution image information. Compared with the data reconstructed based on the fixed value, the resolution is higher and the image clarity is better.

[0196] The implementation process of the technical solution of the present application is described below.

[0197] As is known to all, any d-dimensional distribution can be generated by a set of d variables subject to normal distribution through a complex function. Based on this, in the embodiments of the present application, the first feature vector is understood as an M-dimensional distribution, and it is determined that there is a set of M basic units subject to normal distribution, and the M basic units subject to normal distribution can be mapped to the M-dimensional distribution corresponding to the first feature vector through a complex function.

[0198] Since it is very difficult to directly determine the M-dimensional distribution corresponding to the first feature vector, and it is relatively easy to extract a simple distribution, the embodiments of the present application propose to map the first feature vector to an implicit space, and determine a plurality of basic units subject to normal distribution, which can be mapped to the M-dimensional distribution corresponding to the first feature vector through a complex function. This process is realized based on the model parameters in the trained feature mapping module.

[0199] In the embodiments of the present application, the first feature vector is mapped to a plurality of basic units subject to normal distribution by calling the trained feature mapping module, and the basic units can be combined to form an implicit vector Z. It can be understood that the implicit vector Z can be used to fit the M-dimensional distribution corresponding to the first feature vector.

[0200] The shape and characteristics of the normal distribution are generally described by the mean μ and variance σ 2 of the normal distribution, where:

[0201] Mean μ: The mean of the normal distribution determines the center position of the distribution. On a normal distribution curve, the mean is the center of symmetry of the curve, and is usually located at the highest point of the distribution.

[0202] Variance σ 2 : Variance represents the dispersion or fluctuation degree of the data distribution. The variance of the normal distribution describes the dispersion degree of the data points relative to the mean of the distribution. The larger the variance of the normal distribution, the greater the fluctuation of the data points relative to the mean, and the wider the distribution curve. Conversely, the smaller the variance, the smaller the fluctuation of the data points relative to the mean, and the narrower the distribution curve.

[0203] Based on this, in the embodiments of the present application, the first feature vector is mapped to an implicit space by calling the feature mapping module, the variance set and the mean set corresponding to the implicit space are obtained, and the implicit vector Z is obtained according to the variance set and the mean set corresponding to the implicit space.

[0204] Please refer to Figure 10 , Figure 10A schematic diagram of another image processing model provided by an embodiment of the present application is shown. The feature mapping module includes a mean-variance calculation module, which is configured to calculate the first feature vector to obtain a variance group and a mean group corresponding to the implicit space.

[0205] Optionally, the mean-variance calculation module calculates the first feature vector to obtain a plurality of parameter groups, each parameter group including a mean and a variance. The means in the plurality of parameter groups are extracted to obtain the mean group corresponding to the implicit space, and the variances in the plurality of parameter groups are extracted to obtain the variance group corresponding to the implicit space.

[0206] Optionally, the feature mapping module includes at least two neural network units, which are respectively configured to output the variance σ 2 and the mean μ.

[0207] For example, the mathematical expression of the neural network unit 1 is μ=f1(X). The mathematical expression of the neural network unit 2 is logσ 2 =f2(X), where X represents the input data (the first feature vector).

[0208] It should be noted that in the present application, logσ 2 is fitted by the neural network unit instead of directly fitting σ 2 , because σ 2 is always non-negative and needs to be processed by an activation function, while fitting logσ 2 does not need to be processed by an activation function.

[0209] Optionally, in the present application, the electronic device can call the feature mapping module to map the first feature vector in the depth direction to obtain the variance group and the mean group corresponding to the implicit space.

[0210] Please refer to Figure 11 , Figure 11 A schematic diagram of the arrangement format of the first feature vector is shown, where N represents the number of image frames included in the target image, H represents the number of pixels in the vertical direction of the image, W represents the number of pixels in the horizontal direction of the image, and C represents the number of channels.

[0211] Here, mapping in the depth direction can be understood as mapping along the C direction, and the mapping process can be represented as CWHN, for example Figure 11 In the above example, the first element is 000, the second element is 020 along the C direction, 040, 060, and so on until 300, then along the W direction, 001 021 041 061... 301... to 303, and then along the H direction, 004 024... 304... and finally to 319, which becomes the N direction, 320 340....

[0212] It should be noted that in the embodiments of the present application, the electronic device can also map based on the manner of WHCN, for example, the first element is 000, the second element is along the w direction, that is, 001, and so on 002 003, and then along the H direction, that is, 004 005 006 007... and so on to 019, along the C direction, and it is the turn of 020, and then 021 022... up to 319, and then along the N direction. The mapping process is not limited in the embodiments of the present application.

[0213] In the embodiments of the present application, the electronic device can call the feature mapping module to calculate Figure 11 The variance and mean corresponding to each element shown in the above formula are obtained, thereby forming a variance group and a mean group.

[0214] In another implementation manner, when mapping the first feature vector, the electronic device can map based on a plurality of different mapping directions, thereby obtaining elements corresponding to different mapping manners. The order of the elements determined by different mapping manners is different, which will affect the information extracted by the image processing model. In the embodiments of the present application, the mean and variance of the elements corresponding to a plurality of different mapping manners are used to determine the implicit vector Z, which is beneficial to improve the richness of the information contained in the implicit vector Z.

[0215] In the embodiments of the present application, after the variance group and the mean group special for the first feature vector are determined, a series of normal distributions corresponding to the first feature vector can be determined. As shown in the above formula, in the embodiments of the present application, the electronic device can call the feature mapping module to extract a plurality of variances and means corresponding to the variance group and the mean group, and fit a normal distribution based on each variance and mean. Figure 10

[0216] As shown in the above formula, the variance group includes H variances, and the mean group includes H means, and then H normal distributions can be fitted. Figure 10

[0217] The mathematical expression of the normal distribution can be represented as: Z i =μ i +σ i

[0218] Wherein, Z i is the i-th normal distribution, μ i is the i-th mean, and σ i is the i-th variance, and i is from 0 to H.

[0219] Wherein, the H normal distributions constitute the implicit vector Z.

[0220] ​​In an implementation manner, in the embodiment of the present application, the implicit vector Z is subject to a normal distribution, which can be expressed as Z ~ N(μ, σ 2 ), and the expression can be transformed as:

[0221]

[0222] That is, (Z-μ) / σ is subject to a standard normal distribution with a mean of 0 and a variance of 1. Therefore, the embodiment of the present application introduces a first parameter ε, which is a noise term, and sets ε=(Z-μ) / σ, so that the operation of sampling Z from N(μ, σ 2 ) is converted into sampling the first parameter ε from N(0, I), and then setting Z=μ+σ⊙ε. In this way, the gradient calculation can be directly passed through μ and σ, without involving the gradient of random sampling.

[0223] Based on this, the embodiment of the present application proposes sampling the standard normal distribution to obtain a first parameter group, and then determining the implicit vector Z according to the first parameter group, a variance group and a mean group corresponding to the implicit space.

[0224] For example, the variance group includes H variances, the mean group includes H means, and the first parameter group includes H first parameters, where H is an integer greater than or equal to 1.

[0225] Then H pairs of parameter combinations can be formed, each pair of parameter combinations including a variance, a mean and a first parameter, and each pair of parameter combinations can calculate a basic unit of the implicit vector Z.

[0226] The mathematical expression of the basic unit of the implicit vector Z is:

[0227] Z i =μ i +σ i ⊙∈ i

[0228] Where Z i is the i-th basic unit of the implicit vector Z, μ i is the i-th mean, σ i is the i-th variance, and ∈ i is the i-th first parameter.

[0229] In the embodiment of the present application, the electronic device can call the generation module to sample the implicit vector Z. In an implementation manner, the electronic device can sample the H normal distributions contained in the implicit vector Z respectively to obtain a sampling result. In another implementation manner, the electronic device can fit the H normal distributions contained in the implicit vector Z to obtain an M-dimensional distribution corresponding to the first feature vector, and then sample the M-dimensional distribution to obtain a sampling result.

[0230] In the embodiments of the present application, the electronic device can call the generation module to perform image reconstruction based on the sampling result.

[0231] First, the generation module is described. In the embodiments of the present application, the generation module includes a Unet network, which can include multiple network layers of different resolutions. The multiple network layers of different resolutions can use a step-down sampling manner, so that the image size is reduced by a factor of 2, until the output of the last network layer is the smallest network layer in the processing process of the generation module. In addition, the multiple network layers of different resolutions can also use a step-up sampling manner, so that the image size is increased by a factor of 2, until the output of the last network layer is the output result required by the generation module. In an implementation manner, the network layer corresponding to the above-mentioned up-sampling process and the network layer corresponding to the down-sampling process are a completely symmetrical structure.

[0232] It should be noted that the down-sampling in the above-mentioned sampling process is to reduce the sampling rate of the signal or image, that is, to reduce the number of data points. Through down-sampling (pooling operation), the size of the feature map can be reduced, the main features can be extracted, the calculation amount can be reduced, and the calculation efficiency can be improved. In image processing, common down-sampling methods include: average pooling (Average Pooling): taking the average value in the local region of the image as the new pixel value. Max pooling (MaxPooling): taking the maximum value in the local region of the image as the new pixel value.

[0233] The up-sampling in the above-mentioned sampling process is to increase the sampling rate of the signal or image, that is, to increase the number of data points, which is usually filled with new data points by interpolation and other methods.

[0234] In the embodiments of the present application, the sampling result contains multiple discrete information, and the electronic device can call the generation module to perform image reconstruction based on the sampling result, or can be understood as combining the sampling result to obtain the target image corresponding to the RGB format image.

[0235] Optionally, in the embodiments of the present application, after outputting the RGB format image, the electronic device can further perform a series of color processing on the RGB format image, such as global tone mapping (GTM), gamma correction (GAMM), sharpening, color adjustment, 3D lut (Look-Up Table), etc., to obtain a final RGB format image. Among them, GTM is used to adjust the global contrast and brightness of the image, so that the image looks more natural and balanced. GAMM is used to adjust the brightness level of the image to better match the brightness response characteristics of the display device, and is usually used to eliminate the problem of too dark or too bright image. Sharpening is used to enhance the edges and details of the image, making the image look clearer and sharper. This is usually achieved by enhancing the high frequency part (i.e. the edges in the image). Color Adjustment adjusts the color and tone of the image to change the overall color appearance or saturation, brightness, etc. of a specific color. LUT (Look-Up Table): a lookup table is a technique used to grade and adjust the color of an image. By applying a pre-defined color mapping table, the color appearance of the image can be quickly changed and adjusted, such as increasing contrast, adjusting tone, etc. These techniques and tools are used in image processing to adjust the visual effects of the image, including brightness, contrast, sharpening effect and color appearance, to achieve better visual effects or specific artistic purposes.

[0236] It should be noted that the image processing method provided in the embodiments of the present application, in the CFA array on the camera, the size N of the same color pixel is getting larger and larger, and the disadvantages of the traditional image processing scheme will become more and more obvious. In the present scheme, a completely different processing idea from the traditional scheme is adopted, so the image quality is not affected by the size of the same color pixel, and the image processing effect is better than that of the traditional scheme.

[0237] The image processing method provided in the embodiments of the present application maps the first feature vector obtained after denoising and demosaicking to the implicit space, determines a plurality of basic units subject to normal distribution corresponding to the first feature vector, the basic units can obtain the M-dimensional distribution corresponding to the first feature vector through complex function mapping, and then determines the implicit vector Z according to the plurality of basic units subject to normal distribution. The implicit vector Z represents the probability distribution of the first feature vector. The probability distribution contains information that the input data does not have, so that the RGB format image obtained by sampling the probability distribution and reconstructing the image based on the sampling result contains more rich and high-resolution image information.

[0238] On the basis of the above embodiments, asFigure 12 and Figure 13 as shown in FIG. 1, Figure 13 a flowchart of another image processing method provided by an embodiment of the present application is shown, Figure 13 a schematic diagram of another image processing model provided by an embodiment of the present application is shown. The method can be applied in an electronic device, which can include a camera application and a camera, and the pixel array on the camera is an N x N array.

[0239] The image processing model includes cascaded feature extraction modules, feature mapping modules and a generation module. The functions of the feature extraction modules, the feature mapping modules and the generation module can refer to the content disclosed above, and will not be repeated here.

[0240] In an embodiment of the present application, as shown in FIG. 1, Figure 14 The image processing model further includes a scene classification submodule, the output end of the scene classification submodule is connected with the input end of the generation module, the scene classification submodule is used for classifying the scene of the target image and outputting a second feature vector, and the second feature vector contains the information of the scene to which the target image belongs. The second feature vector can guide the reconstruction result in the image reconstruction process.

[0241] In an implementation mode, the scene classification submodule is a submodule included in the feature extraction module, wherein the feature extraction module sends the target image to the scene classification submodule for classification after obtaining the target image.

[0242] In another implementation mode, in the image processing model, the scene classification submodule and the feature extraction module are parallel, and the data receiving end of the image processing model transmits the target image to the scene classification submodule and the feature extraction module respectively after obtaining the target image.

[0243] Optionally, the image processing model further includes a scene guidance submodule. The generation module includes M network layers with different resolutions.

[0244] In an implementation mode, the scene guidance submodule is a submodule included in the generation module, wherein the scene guidance submodule is used for sequentially fusing the second feature vector into the feature vector corresponding to each network layer respectively. The feature vector corresponding to the network layer refers to the feature vector to be processed by the network layer.

[0245] In another implementation mode, in the image processing model, the scene guidance submodule and the generation module are parallel. The scene guidance submodule obtains the second feature vector, the generation module obtains the sampling result, and the sampling result is input into the network layer. The scene guidance submodule fuses the second feature vector into the feature vector corresponding to the network layer.

[0246] In another implementation, the scene guiding sub-modules are M, and the generation module includes M network layers of different resolutions, and each network layer is provided with a scene guiding sub-module.

[0247] For example, the nth network layer is provided with an nth scene guiding sub-module, and the nth scene guiding sub-module can fuse the second feature vector into the feature vector corresponding to the nth network layer.

[0248] The embodiments of the present application do not limit the specific setting mode of the scene classification sub-module and the scene guiding sub-module, Figure 14 only one possible structure of an image processing model is exemplarily shown.

[0249] The image processing method provided by the embodiments of the present application includes steps 1201 to 1204.

[0250] In step 1201, a plurality of N×N raw format images are acquired.

[0251] Wherein, N×N refers to the size of the same color pixels in the CFA array in the raw format image.

[0252] In the embodiments of the present application, after acquiring the plurality of N×N raw format images, the plurality of N×N raw format images can be preprocessed. The process of preprocessing can refer to the content disclosed in the above embodiments, which will not be repeated here.

[0253] In step 1202, a feature extraction module is called to perform denoising and demosaicing processing on the plurality of N×N raw format images to obtain a first feature vector. A scene classification sub-module is called to perform scene classification on the plurality of N×N raw format images and output a second feature vector.

[0254] The second feature vector contains information of the scene to which the target image belongs.

[0255] In step 1203, a feature mapping module is called to map the first feature vector to a hidden space to obtain a hidden vector Z.

[0256] The hidden vector Z represents the probability distribution of the first feature vector.

[0257] In step 1204, a generation module is called to sample the hidden vector Z, and based on the sampling result and the second feature vector, an image is reconstructed to obtain an RGB format image corresponding to the plurality of N×N raw format images.

[0258] In the process of generating an image by the generation module, in order to avoid errors, for example, a face pattern is generated on a tree, it is necessary to guide the image generation process. The embodiments of the present application propose a scheme of guiding the image generation process based on the scene to which the target image belongs.

[0259] Among them, the scene to which the image belongs can be various, depending on the content and background displayed in the image. The following are some common image scene categories: 1. Natural scenery: natural landscapes such as mountains, lakes, rivers, etc. Different types of natural environments such as forests, grasslands, deserts, etc. Natural phenomena at specific times such as sunrise, sunset, starry sky, etc. 2. Urban landscape: urban streets, building complexes, iconic buildings. Internal landscape of the city such as city center, business district, residential area, etc. City night scene, traffic scene, urban life activities, etc. 3. Indoor scene: different types of indoor space such as living room, bedroom, kitchen, office, etc. Indoor decoration and equipment such as furniture, appliances, decorations, etc. 4. Human and social scenes: daily life scenes of people such as work, study, leisure, etc. Interpersonal scenes such as social activities, parties, holiday celebrations, etc. 5. Specific events and activities: specific activity sites such as competitions, performances, performances, etc. Public or private activities such as celebrations, ceremonies, rallies, etc. 6. Abstract and artistic scenes: abstract art works, sculptures, architectural designs, etc. Fantasy, science fiction, surrealism, etc. Artistic creation scenes. 7. Historical and cultural scenes: historical and cultural scenes such as cultural relics, historical sites, historical buildings, etc. Traditional festivals, folk activities, religious ceremonies, etc. Cultural activities.

[0260] Each scene type has its own unique visual features and background information, and by analyzing these features and information, the specific scene to which the image belongs can be accurately understood.

[0261] In the embodiments of the present application, the scene classification sub-module uses a deep learning model such as a convolutional neural network to perform image classification and object recognition on the multi-frame N x N raw format image. Alternatively, the scene classification sub-module can analyze the visual features and background information in the multi-frame N x N raw format image to determine the specific scene to which the multi-frame N x N raw format image belongs.

[0262] In the embodiments of the present application, after determining the specific scene to which the multi-frame N x N raw format image belongs, the scene classification sub-module can output a second feature vector according to the scene to which the multi-frame N x N raw format image belongs.

[0263] In one implementation, the scene classification sub-module can generate a second feature vector based on its own algorithm based on the scene to which the multi-frame N x N raw format image belongs.

[0264] In another implementation, the scene classification sub-module has pre-stored scene vectors corresponding to different scenes. After classifying the scene of the multi-frame N x N raw format image, the scene classification sub-module determines and outputs a second feature vector according to the scene vector corresponding to the scene to which the multi-frame N x N raw format image belongs.

[0265] For example, 1000 scenes are preset, and a scene vector corresponding to each scene is determined, that is, there are 1000 scene vectors. It should be noted that the 1000 scene vectors are one-dimensional vectors of the same size.

[0266] In the embodiment of the application, when the scene classification submodule determines that the multiple frames of N*N raw format images belong to the A scene, the second feature vector can be determined based on the scene vector of the A scene stored in advance.

[0267] In an implementation manner, the second feature vector is a one-dimensional vector. The electronic device can take the scene vectors of the A scene as the second feature vector.

[0268] In another implementation manner, the second feature vector is a fixed-size data matrix with the total number of scenes as the number of rows and the length of the scene vector as the number of columns. Alternatively, the second feature vector is a fixed-size data matrix with the total number of scenes as the number of columns and the length of the scene vector as the number of rows. The second feature vector contains the scene vector of the A scene. The part of the second feature vector other than the scene vector of the A scene is all 0.

[0269] In another implementation manner, different scenes are encoded in advance, and the positions of the scenes in the second feature vector are determined. When it is determined that the multiple frames of N*N raw format images belong to the A scene, the position corresponding to the A scene in the second feature vector is filled with the scene vector of the A scene, and the remaining unfilled part is all 0.

[0270] In the embodiment of the application, the electronic device can call the feature mapping module to map the first feature vector to the implicit space based on the content disclosed in the above embodiments to obtain an implicit vector Z. Then the electronic device can call the generation module to sample the implicit vector Z, reconstruct the image based on the sampling result and the second feature vector, and obtain the RGB format image corresponding to the multiple frames of N*N raw format images.

[0271] In the embodiment of the application, in the image generation stage, the second feature vector is used for guidance, so that the accuracy of the generated RGB format image is higher.

[0272] In an implementation manner, the electronic device can fuse the second feature vector into the feature vector corresponding to the generation module, wherein the feature vector corresponding to the generation module refers to the feature vector that needs to be processed by the generation module. The feature vector that needs to be processed by the generation module can be a feature vector determined based on the sampling result, or a feature vector obtained by processing the sampling result.

[0273] In another implementation manner, the generation module further includes M network layers of different resolutions, and the electronic device can call the scene guidance submodule to fuse the second feature vector into the feature vector corresponding to each network layer of different resolutions in sequence.

[0274] For example, the calling scene guiding submodule fuses the second feature vector into the feature vector corresponding to the mth network layer in turn. When m traverses to 1, the feature vector corresponding to the mth network layer is determined based on the sampling result. In the process in which m traverses from 2 to M, the feature vector corresponding to the mth network layer is determined based on the output of the (m-1)th network layer. When m traverses to M, the output of the mth network layer is used to determine the RGB format image corresponding to the multi-frame N*N raw format image.

[0275] The feature vector corresponding to the mth network layer can refer to the feature vector received by the mth network layer from the (m-1)th network layer, or can also be an intermediate vector obtained by processing the received feature vector by the mth network layer, or can also be the feature vector to be output to the next network layer by the mth network layer.

[0276] In the embodiment of the application, the second feature vector is obtained by classifying and identifying the scene to which the multi-frame N*N raw format image belongs, and then the second feature vector is fused into the network layer of different resolutions included in the generation module, so that the image generation process is guided based on the scene to which the multi-frame N*N raw format image belongs, so that the generated RGB format image is more accurate and avoids generating non-adaptive information.

[0277] On the basis of the above-mentioned embodiment, a model training method provided by an embodiment of the application is described below. The model training method is used to train an image processing model. As shown in Figures 1 to 14 Figure 15 A flowchart of a model training method provided by an embodiment of the application is shown. The method is applied to an electronic device. The method comprises the following steps:

[0278] In step 1401, a first image sample and a second image sample are obtained.

[0279] The first image sample is obtained by processing the second image sample, and each pixel point in the first image sample contains a color component. The second image sample is in RGB format.

[0280] In an implementation manner, the first image sample is obtained by performing degradation processing on the second image sample. For example, a single-channel image is extracted from the three-channel second image sample to obtain the first image sample, wherein each pixel point in the single-channel image contains a color component. Optionally, after obtaining the single-channel image, noise can be added to the single-channel image, and the single-channel image after adding the noise is determined as the first image sample. It should be noted that the specific implementation manner of the degradation processing is not limited in the application.

[0281] ​Step 1402, an initial image processing model is constructed.

[0282] The initial image processing model includes a cascaded initial feature extraction module, an initial feature mapping module and an initial generation module.

[0283] In the embodiment of the application, the initial feature extraction module includes a self-encoder and a denoising and demosaicking unit, and the denoising and demosaicking unit can be an algorithm module that has been trained.

[0284] The initial feature mapping module and the initial generation module each include an initial weight, and the initial weight is adjusted after model training until the adjusted weight can enable each module to complete the corresponding function.

[0285] Step 1403, a first image sample is input into the initial image processing model, the first image sample is processed by the initial feature extraction module to obtain a first feature vector corresponding to a denoising and demosaicking processing result, the first feature vector is mapped to a hidden space by the initial feature mapping module to obtain a hidden vector Z, and the hidden vector Z is sampled by the initial generation module, and an image is reconstructed based on the sampling result to obtain a generated sample.

[0286] It should be noted that after the initial image processing model converges, the generated sample is used to determine the RGB format image corresponding to the first image sample.

[0287] Step 1404, a KL divergence is calculated according to the hidden vector Z and a standard normal distribution, a first loss is calculated according to the generated sample and a second image sample, and the initial feature extraction module, the initial feature mapping module and the initial generation module are adjusted according to the KL divergence and the first loss.

[0288] The KL divergence is used to measure the closeness between two probability distributions.

[0289] The ideas and core formulas involved in the model training method provided in the embodiments of the application are described below.

[0290] In the embodiments of the application, it can be considered that the first feature vector is subject to a certain distribution, and the technical solution of the application expects to map multiple normal distributions close to the standard normal distribution from the first feature vector, and the fitting of the normal distributions can obtain a certain distribution (i.e., the hidden vector Z) that is exclusively subject to the first feature vector. Then, the hidden vector Z is sampled, and it is hoped that the picture can be restored based on the sampling result. Therefore, how to map the first feature vector to the hidden space and determine the hidden vector Z is the focus.

[0291] Assuming there is a vector z, find a mapping relationship to map the vector z into a series of normal distribution parameter vectors μ(z) and σ(z), and with the series of normal distribution parameters, the superimposed P(x) can be obtained. Where p(x|z) is obtained by updating the weight of the feature mapping module in the model training process.

[0292] For a single vector z, P(x) can be obtained by mapping through the mapping relationship p(x|z). Then for continuous variable z, according to the total probability formula, there is:

[0293] P(x) = ∫p(x|z)p(z)dz

[0294] However, the implicit space of the continuous variable z is usually high-dimensional and contains complex interactions, and it is almost impossible to directly integrate to calculate P(x). Therefore, the embodiments of the present application simplify the problem of calculating P(x) to the problem of variational lower bound (ELBO), the process is as follows:

[0295] The posterior distribution p(z|x) of z given x is approximately equal to the distribution of z that maximizes the generation probability of x. Based on this, the present application proposes to introduce an approximate posterior distribution q(z|x), in which | is read as given. q(z|x) represents the probability distribution of z given x.

[0296] Substitute the approximate posterior distribution q(z|x) for the real posterior distribution p(z|x). Then the posterior distribution p(z|x) is related to the approximate posterior distribution q(z|x) through the KL divergence. According to the formula of KL divergence, we have:

[0297] D[q(z|x)||p(z|x)] = logq(z|x)-logp(z|x)

[0298] Use the Bayes formula to transform p(z|x) on the right side, and the equation is transformed as follows:

[0299] D[q(z|x)||p(z|x)] = logq(z|x)-logp(x|z)-logp(z)+logp(x)

[0300] Further moving the term, we have:

[0301] logp(x)-D[q(z|x)||p(z|x)] = logp(x|z)-logq(z|x)-logp(z)

[0302] Rewrite using the KL divergence formula, we have:

[0303] logp(x) - D[q(z|x) || p(z)] = [logp(x|z)] - D[q(z|x) || p(z)]

[0304] Since the KL divergence is non-negative, it can be understood that:

[0305] logp(x) ≥ [logp(x|z)] - D[q(z|x) || p(z)]

[0306] So far, in this scheme, the difference between the target and the ELBO is the difference between the approximate posterior distribution q(z|x) and the true posterior distribution p(z|x), so this scheme converts the problem of "maximizing P(x)" into the problem of maximizing the variational lower bound. Maximizing the variational lower bound can be understood as maximizing logp(x|z) on the one hand, that is, making the probability distribution expectation value of x given z larger, and minimizing D[q(z|x) || p(z)] on the other hand. Minimizing D[q(z|x) || p(z)] can be understood as making the KL divergence as small as possible. That is, the approximate posterior distribution of the implicit vector Z in the implicit space is as close as possible to the prior distribution of the implicit vector Z.

[0307] On this basis, the embodiment of the present application constructs an initial image processing model, and the mathematical model in the initial feature mapping module in the initial image processing model is to minimize the KL divergence, and the mathematical model in the initial generation module is the mathematical model to maximize the above-mentioned variational lower bound.

[0308] Optionally, the initial feature mapping module can further include two neural networks, which respectively output the mean and the variance, and the mathematical model of the two neural networks can be represented as:

[0309] μ = f1(X)

[0310] logσ 2 = f2(X)

[0311] First, the first image sample enters the initial feature extraction module, the initial feature extraction module denoises and demosaics the first image sample, and obtains a first feature vector based on the denoising and demosaicing result; then the initial feature mapping module maps the first feature vector, learns and outputs the mean and variance of the implicit space, and obtains an implicit vector Z of the implicit space. Then sample from the implicit space to obtain a sampling result, and the initial generation module reconstructs according to the sampling result to obtain a generated sample.

[0312] In the model training process in the embodiments of this application, the closeness between the plurality of basic units constituting the implicit vector Z and the standard normal distribution is measured by the KL divergence. The KL divergence is used to calculate the similarity of two probability distributions according to the expression of the two probability distributions. The KL divergence constitutes part of the loss in the model training process, and is back propagated to adjust the weights of the initial feature extraction module, the initial feature mapping module and the initial generation module in the initial image processing model.

[0313] In addition, in the model training stage, the model loss can also be calculated according to the generated sample and the second image sample to obtain the first loss, and the weights of the various modules in the initial image processing model are adjusted based on the first loss and the KL divergence.

[0314] On the basis of the above embodiments, in the embodiments of this application, the initial image processing model can further include an initial discrimination module. The main responsibility of the initial discrimination module is to distinguish the generated sample generated by the initial generation module from the second image sample. We want the confidence output for the second image sample to be closer to 1, and the confidence output for the generated sample to be closer to 0.

[0315] Unlike the traditional network training method, in the model training process of the present scheme, the image processing part constituted by the initial feature extraction module, the initial feature mapping module and the initial generation module needs to be trained, and the discrimination module also needs to be trained, so the training difficulty is greater. After the model converges, the discrimination module is removed, and only the feature extraction module, the feature mapping module and the generation module are retained to constitute the final image processing model.

[0316] In the embodiments of this application, the second image sample and the generated sample are given to the initial discrimination module. The goal of the initial discrimination module is to accurately distinguish the second image sample from the generated sample, and the training goal of the initial generation module is to deceive the initial discrimination module as much as possible, so that the initial discrimination module cannot identify the generated sample. That is, the initial generation module and the initial discrimination module constitute an antagonistic competitive relationship.

[0317] After the initial discrimination module gives the discrimination result, the loss corresponding to the generated sample and the loss corresponding to the second image sample can be calculated respectively to obtain the second loss. The second loss and the aforementioned KL divergence are back propagated together to adjust the weights of the initial feature extraction module, the initial feature mapping module, the initial generation module and the initial discrimination module in the initial image processing model.

[0318] Further, in the embodiments of the present application, since the image reconstruction is based on the data sampled from the implicit vector Z (implicit space), the "sampling" operation cannot be back propagated in the model training process. In order to solve this problem, the embodiments of the present application introduce a noise term ε (i.e., the first parameter), i.e., ε = (Z-μ) / σ, wherein (Z-μ) / σ obeys the standard normal distribution with a mean of 0 and a variance of 1, that is, the first parameter ε obeys the standard normal distribution with a mean of 0 and a variance of 1, so that the gradient calculation can be directly transmitted through μ and σ, and no longer involves the gradient of random sampling.

[0319] On the basis of the above-mentioned embodiments, in the embodiments of the present application, the initial image processing model can also be provided with an initial scene classification submodule for obtaining scene information of the first image sample and outputting a second feature vector, which can be guided in the subsequent generation stage to improve the accuracy of the reconstruction result.

[0320] The initial scene classification submodule can be a module parallel to the initial feature extraction module, or a submodule contained in the initial feature extraction module.

[0321] Correspondingly, the initial image processing model also includes an initial scene guidance submodule, which can be a module parallel to the initial generation module, or a submodule contained in the initial generation module.

[0322] In addition, the initial generation module can include a plurality of initial network layers of different resolutions, and the initial scene guidance submodule can fuse the second feature vector into the feature vector corresponding to each initial network layer of different resolutions, so as to strengthen the scene information of the image in each initial network layer of the generation module and improve the accuracy of the reconstruction result.

[0323] Illustratively, the initial generation module further includes M initial network layers of different resolutions, and the second feature vector is fused into the feature vector corresponding to the mth initial network layer by the initial scene guidance submodule, wherein when m traverses 1, the feature vector corresponding to the mth initial network layer is determined based on the sampling result, in the process of m traversing from 2 to M, the feature vector corresponding to the mth initial network layer is determined based on the output of the (m-1)th initial network layer, and when m traverses M, the output of the mth initial network layer is used to determine the generated sample.

[0324] Optionally, the initial scene classification submodule is pre-provided with scene vectors corresponding to different scenes, and the second feature vector is determined and output according to the scene vector corresponding to the scene to which the first image sample belongs after the initial scene classification submodule classifies the first image sample.

[0325] Optionally, the first feature vector is mapped to the implicit space through the initial feature mapping module to obtain the variance group and mean group corresponding to the implicit space, and the implicit vector Z is obtained based on the variance group and mean group corresponding to the implicit space.

[0326] Optionally, the first feature vector is mapped along the depth direction through the initial feature mapping module to obtain the variance group and mean group corresponding to the implicit space.

[0327] Optionally, the standard normal distribution is sampled through the initial feature mapping module to obtain the first parameter set; the implicit vector Z is determined based on the first parameter set, the variance set and the mean set corresponding to the implicit space.

[0328] Optionally, the variance group includes H variances, the mean group includes H means, and the first parameter group includes H first parameters, where H is an integer greater than or equal to 1.

[0329] Optionally, the implicit vector Z consists of multiple basic units, and the mathematical expression of the basic unit is:

[0330] Z i =μ i +σ i ⊙∈ i

[0331] Among them, Z i For the i-th basic unit, μ i Let σ be the i-th mean. i Let ε be the variance of the i-th variable. i For the i-th first parameter, ⊙ represents the XOR operation.

[0332] It is understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this embodiment, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0333] The above combination Figure 15 The image processing method, image processing model, and image processing model training method provided in the embodiments of this application have been described. The apparatus for executing the above methods provided in the embodiments of this application is described below. Figure 16 As shown, Figure 16An optional structural schematic diagram of an image processing apparatus provided by an embodiment of the present application is shown. The image processing apparatus can be an execution device in the embodiment of the present application, or a chip or chip system in the execution device. The image processing apparatus 1500 comprises an acquisition unit 1501 and a processing unit 1502, wherein the acquisition unit 1501 is configured to acquire a target image; the processing unit 1502 is configured to perform denoising and demosaicking processing on the target image to obtain a first feature vector corresponding to a denoising and demosaicking processing result, each pixel point in the target image containing a color component; map the first feature vector to a latent space to obtain a latent vector Z, the latent vector representing a probability distribution of the first feature vector; sample the latent vector Z, and perform image reconstruction based on a sampling result to obtain an RGB format image corresponding to the target image.

[0334] As shown in Figure 17 Figure 17 An optional structural schematic diagram of a model training apparatus provided by an embodiment of the present application is shown. The model training apparatus can be an execution device in the embodiment of the present application, or a chip or chip system in the execution device. The model training apparatus 1600 comprises an acquisition unit 1601 and a processing unit 1602, the acquisition unit 1601 is configured to acquire a first image sample and a second image sample, the first image sample being obtained by performing degradation processing on the second image sample, each pixel point in the first image sample containing a color component, and the second image sample being in an RGB format; the processing unit 1602 is configured to construct an initial image processing model, the initial image processing model comprising an initial feature extraction module, an initial feature mapping module and an initial generation module connected in cascade; input the first image sample into the initial image processing model, perform denoising and demosaicking processing on the first image sample through the initial feature extraction module to obtain a first feature vector corresponding to a denoising and demosaicking processing result; map the first feature vector to a latent space through the initial feature mapping module to obtain a latent vector Z; sample the latent vector Z through the initial generation module, and perform image reconstruction based on a sampling result to obtain a generated sample; calculate a KL divergence according to the latent vector Z and a standard normal distribution, and calculate a first loss according to the generated sample and the second image sample, adjust the initial feature extraction module, the initial feature mapping module and the initial generation module according to the KL divergence and the first loss, until the initial image processing model converges, wherein the KL divergence is used to measure the closeness between two probability distributions, and after the initial image processing model converges, the generated sample is used to determine an RGB format image corresponding to the first image sample.

[0335] ​Optionally, the initial image processing model further comprises an initial discrimination module, after obtaining the generated sample, the processing unit 1602 discriminates the generated sample and the second image sample through the initial discrimination module; calculates the loss of the second image sample and the loss of the generated sample to obtain a second loss; adjusts the initial feature extraction module, the initial feature mapping module, the initial generation module and the initial discrimination module according to the KL divergence and the second loss.

[0336] It should be noted that the apparatuses 1500 and 1600 are embodied in the form of functional units. The term "unit" herein can be implemented in the form of software and / or hardware, and is not specifically limited. For example, the "unit" can be a software program, hardware circuit or a combination of both, which realizes the above functions. The hardware circuit can include an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a dedicated processor or a group processor) and a memory for executing one or more software or firmware programs, a combination logic circuit and / or other suitable components supporting the described functions.

[0337] Therefore, the units of each example described in the embodiments of the present application can be realized in the form of electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0338] Figure 1 A hardware structure schematic diagram of an electronic device provided by an embodiment of the present application is shown. Figure 1 The electronic device 1700 shown (which can be a server in particular) includes one or more (including two) processors 1710, a communication line 1740 and a communication interface 1730, and optionally, the electronic device 1700 further includes a memory 1720. The electronic device can be used to train an image processing model, and can also be configured to train an image processing model and apply it.

[0339] In some embodiments, the memory 1720 stores the following elements: executable modules or data structures, or a subset thereof, or an extended set thereof.

[0340] The method described in the embodiments of the present application can be applied to the processor 1710 or implemented by the processor 1710. The processor 1710 can be an integrated circuit chip having a signal processing capability. In the implementation process, each step of the above method can be completed by the integrated logic circuit or the instruction in the form of software in the processor 1710. The processor 1710 described above can be a general processor (for example, a microprocessor or a conventional processor), a digital signal processor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices or discrete hardware components, and the processor 1710 can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application.

[0341] The steps of the method disclosed in the embodiments of the present application can be directly embodied as hardware code processing executed by a code processor, or executed by a combination of hardware and software modules in the code processor. Among them, the software module can be located in a storage medium mature in the art such as random access memory, read-only memory, programmable read-only memory or electrically erasable programmable read-only memory (EEPROM). The storage medium is located in the memory 1720, and the processor 1710 reads the information in the memory 1720 and combines the hardware to complete the steps of the above method.

[0342] The processor 1710, the memory 1720 and the communication interface 1730 can communicate through the communication line 1740.

[0343] In the above embodiments, the instructions stored in the memory for the processor to execute can be implemented in the form of a computer program product. Among them, the computer program product can be written in the memory in advance, or downloaded and installed in the memory in the form of software.

[0344] The embodiments of the present application further provide a computer program product including one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. including one or more available media sets. For example, the available media can include magnetic media (for example, floppy disk, hard disk or magnetic tape), optical media (for example, digital versatile disc (DVD)), or semiconductor media (for example, solid state disk (SSD)) and the like.

[0345] The embodiments of the present application provide a chip. The chip includes a processor configured to invoke a computer program in a memory to execute the technical solutions in the above-described embodiments. The implementation principles and technical effects are similar to those of the above-described related embodiments, which will not be described here.

[0346] The embodiments of the present application further provide a computer-readable storage medium. The computer-readable storage medium stores computer programs or instructions. The computer programs or instructions are executed by the processor to implement the above-described method. The method described in the above-described embodiments can be wholly or partially implemented by software, hardware, firmware or any combination thereof. If implemented in software, the functions can be stored as one or more instructions or codes on a computer-readable medium or transmitted on a computer-readable medium. The computer-readable medium can include computer storage medium and communication medium, and can also include any medium that can transfer computer programs from one place to another. The storage medium can be any target medium that can be accessed by a computer.

[0347] As a possible design, the computer readable medium can include a compact disc read-only memory (CD-ROM), a RAM, a ROM, an EEPROM, or other optical disk storage; the computer readable medium can include a magnetic disk storage or other magnetic disk storage device. Moreover, any connection line can also be properly referred to as a computer readable medium. For example, if software is transmitted from a website, a server, or other remote source using a coaxial cable, an optical fiber cable, a twisted pair, a DSL, or wireless technology (such as infrared, radio, and microwave), the coaxial cable, the optical fiber cable, the twisted pair, the DSL, or the wireless technology (such as infrared, radio, and microwave) is included in the definition of the medium. As used herein, a disk and a disc include a compact disc (CD), a laser disc, an optical disc, a DVD, a floppy disk, and a Blu-ray disc, where a disk usually reproduces data magnetically, while a disc reproduces data optically with a laser. Combinations of the above should also be included in the scope of the computer readable medium.

[0348] Embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the computer or other programmable data processing apparatus generate a means for implementing the functions specified in the flowcharts and / or block diagrams. ​ one or more flows and / or blocks ​ an apparatus that carries out the functions specified in one or more flows and / or blocks.

[0349] The above detailed description of the application is further detailed to explain the purposes, technical solutions, and beneficial effects of the present application. It should be understood that the above is only a specific embodiment of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solutions of the present application should be included in the protection scope of the present application.

Claims

1. An image processing method, characterized by, Applied to an image processing model, the image processing model comprising a cascaded feature extraction module, a feature mapping module and a generation module, the method comprising: calling the feature extraction module, carrying out denoising and demosaicing processing on a target image to obtain a first feature vector, each pixel point in the target image containing a color component; calling the feature mapping module, mapping the first feature vector to a latent space to obtain a latent vector Z, the latent vector representing the probability distribution of the first feature vector; calling the generation module, sampling the latent vector Z, and reconstructing an image based on the sampling result to obtain an RGB format image corresponding to the target image.

2. The method of claim 1, wherein, The image processing model further comprises a scene classification submodule, the output end of the scene classification submodule being connected with the input end of the generation module, and the method further comprising: calling the scene classification submodule, classifying the target image according to a scene, and outputting a second feature vector, the second feature vector containing information of the scene to which the target image belongs; calling the generation module, reconstructing an image based on the sampling result and the second feature vector to obtain an RGB format image corresponding to the target image.

3. The method of claim 2, wherein, The scene classification submodule is preconfigured with scene vectors corresponding to different scenes respectively, and the method further comprises: calling the scene classification submodule, determining and outputting the second feature vector according to the scene vector corresponding to the scene to which the target image belongs after classifying the target image according to a scene.

4. The method according to claim 2 or 3, characterized in that, The image processing model further comprises a scene guide submodule, and the generation module further comprises M network layers with different resolutions, and the reconstructing an image based on the sampling result and the second feature vector to obtain an RGB format image corresponding to the target image comprises: calling the scene guide submodule to fuse the second feature vector into a feature vector corresponding to the mth network layer, when m traverses to 1, the feature vector corresponding to the mth network layer is determined based on the sampling result, in the process of m traversing from 2 to M, the feature vector corresponding to the mth network layer is determined based on the output of the (m-1)th network layer, and when m traverses to M, the output of the mth network layer is used to determine the RGB format image corresponding to the target image.

5. The method according to any one of claims 1-4, characterized in that, The size of the same color pixels arranged together in the target image is N×N, and N is an integer greater than or equal to 2.

6. The method according to any one of claims 1-4, characterized in that, The method further comprises: calling the feature mapping module, mapping the first feature vector to a latent space to obtain a variance group and a mean group corresponding to the latent space, and obtaining a latent vector Z according to the variance group and the mean group corresponding to the latent space.

7. The method of claim 6, wherein, The method further comprises: calling the feature mapping module, mapping the first feature vector in the depth direction to obtain a variance group and a mean group corresponding to the latent space.

8. The method according to claim 6 or 7, characterized in that, The latent vector Z obeys a normal distribution, and the method further comprises: calling the feature mapping module, sampling a standard normal distribution to obtain a first parameter group; determining the latent vector Z according to the first parameter group, the variance group and the mean group corresponding to the latent space.

9. The method of claim 8, wherein, The variance group includes H variances, the mean value group includes H mean values, the first parameter group includes H first parameters, and H is an integer greater than or equal to 1.

10. The method of claim 8, wherein, The implicit vector Z is composed of a plurality of basic units, and a mathematical expression of the basic unit is: Z i = μ i + σ i ⊙ ∈ i where Z i is the ith basic unit, μ i is the ith mean, σ i is the ith variance, ∈ i is the ith first parameter, and ⊙ denotes the exclusive OR operation.

11. The method according to any one of claims 1-10, characterized in that, The target image includes a plurality of image frames.

12. The method according to any one of claims 1-10, characterized in that, The image format of the target image is a raw format.

13. A model training method, comprising: The method comprises: obtaining a first image sample and a second image sample, the first image sample being obtained after processing the second image sample, each pixel point in the first image sample containing a color component, and the second image sample being in RGB format; constructing an initial image processing model, the initial image processing model including a cascaded initial feature extraction module, an initial feature mapping module, and an initial generation module; inputting the first image sample into the initial image processing model, denoising and demosaicking the first image sample through the initial feature extraction module to obtain a first feature vector corresponding to a denoising and demosaicking result, mapping the first feature vector to an implicit space through the initial feature mapping module to obtain an implicit vector Z, and sampling the implicit vector Z through the initial generation module and reconstructing an image based on the sampling result to obtain a generated sample; calculating a KL divergence based on the implicit vector Z and a standard normal distribution, calculating a first loss based on the generated sample and the second image sample, adjusting the initial feature extraction module, the initial feature mapping module, and the initial generation module based on the KL divergence and the first loss until the initial image processing model converges, wherein the KL divergence is used to measure the closeness between two probability distributions, and the generated sample is used to determine an RGB format image corresponding to the first image sample after the initial image processing model converges.

14. The method of claim 13, wherein, The initial image processing model further includes an initial discrimination module, and the method further comprises: discriminating the generated sample and the second image sample through the initial discrimination module; calculating a loss of the generated sample and a loss of the second image sample to obtain a second loss; adjusting the initial feature extraction module, the initial feature mapping module, the initial generation module, and the initial discrimination module based on the KL divergence and the second loss.

15. The method according to claim 13 or 14, characterized in that, The initial image processing model further includes an initial scene classification submodule, an output end of the initial scene classification submodule is connected to an input end of the initial generation module, and the method further comprises: classifying a scene of the first image sample through the initial scene classification submodule and outputting a second feature vector, the second feature vector containing information of a scene to which the first image sample belongs; reconstructing an image based on the sampling result and the second feature vector through the initial generation module to obtain the generated sample.

16. The method of claim 15, wherein, The initial scene classification submodule is pre-provided with scene vectors corresponding to different scenes, and the method further comprises: The initial scene classification submodule is configured to determine and output the second feature vector according to a scene vector corresponding to a scene to which the first image sample belongs after performing scene classification on the first image sample.

17. The method according to claim 15 or 16, characterized in that, The initial image processing model further includes an initial scene guidance submodule, and the initial generation module further includes M initial network layers with different resolutions. The initial scene guidance submodule is configured to fuse the second feature vector into a feature vector corresponding to an mth initial network layer, where m is an integer ranging from 1 to M.

18. The method according to any one of claims 13-17, characterized by, The method further includes: The initial feature mapping module is configured to map the first feature vector to an implicit space to obtain a variance group and a mean group corresponding to the implicit space, and obtain an implicit vector Z according to the variance group and the mean group corresponding to the implicit space.

19. The method of claim 18, wherein, The method further includes: The initial feature mapping module is configured to map the first feature vector in a depth direction to obtain the variance group and the mean group corresponding to the implicit space.

20. The method of claim 18 or 19, wherein, The implicit vector Z is subject to a normal distribution, and the method further includes: The initial feature mapping module is configured to sample a standard normal distribution to obtain a first parameter group, and determine the implicit vector Z according to the first parameter group, the variance group and the mean group corresponding to the implicit space.

21. The method of claim 20, wherein, The variance group includes H variances, the mean group includes H means, and the first parameter group includes H first parameters, where H is an integer greater than or equal to 1.

22. The method of claim 20, wherein, The implicit vector Z is composed of a plurality of basic units, and a mathematical expression of the basic unit is: Z i = μ i + σ i ⊙ ∈ i where Z i is the i-th basic unit, μ i is the i-th mean, σ i is the i-th variance, ∈ i is the i-th first parameter, and ⊙ denotes the exclusive OR operation.

23. An image processing apparatus characterized by comprising: The method further includes: An acquisition unit is configured to acquire a target image; A processing unit is configured to perform denoising and demosaicing processing on the target image to obtain a first feature vector corresponding to a denoising and demosaicing processing result, each pixel point in the target image including one color component; The first feature vector is mapped to an implicit space to obtain an implicit vector Z, the implicit vector representing a probability distribution of the first feature vector; The implicit vector Z is sampled, and an image is reconstructed based on a sampling result to obtain an RGB format image corresponding to the target image.

24. A model training apparatus, comprising: The method further includes: An acquisition unit is configured to acquire a first image sample and a second image sample, the first image sample being obtained by processing the second image sample, each pixel point in the first image sample including one color component, and the second image sample being in an RGB format; A processing unit is configured to construct an initial image processing model, the initial image processing model including an initial feature extraction module, an initial feature mapping module and an initial generation module connected in cascade; inputting the first image sample into the initial image processing model, performing denoising and demosaicking processing on the first image sample by the initial feature extraction module to obtain a first feature vector corresponding to a denoising and demosaicking processing result, mapping the first feature vector to a hidden space by the initial feature mapping module to obtain a hidden vector Z, sampling the hidden vector Z by the initial generation module, and performing image reconstruction based on a sampling result to obtain a generated sample; calculating a KL divergence according to the hidden vector Z and a standard normal distribution, and calculating a first loss according to the generated sample and the second image sample, adjusting the initial feature extraction module, the initial feature mapping module and the initial generation module according to the KL divergence and the first loss until the initial image processing model converges, wherein the KL divergence is used to measure the closeness between two probability distributions, and after the initial image processing model converges, the generated sample is used to determine an RGB format image corresponding to the first image sample.

25. An electronic device, comprising: The computer readable storage medium stores program code for execution by a device, the program code comprising instructions for performing any of the methods of claims 1-12 or 13-22.

26. A computer-readable storage medium, characterized in that, The computer readable storage medium stores program code for execution by a device, the program code comprising instructions for performing any of the methods of claims 1-12 or 13-22.

Images