Image processing method, electronic device, computer readable medium, and program product

Abstract

The present disclosure belongs to the field of image processing, and provides an image processing method, electronic device, computer-readable medium and program product, which can at least partially solve the problem that the ISP parameters corresponding to the image cannot be determined after the existing image inverse processing. The image processing method of the present disclosure includes: processing ISP parameters according to the target image signal, inputting the first image into N inverse processing modules in sequence; N is a positive integer; obtaining the first result image output by the Nth inverse processing module; wherein, at least some of the N inverse processing modules include: a first inverse processing submodule and a correction module, and the input end of the correction module is connected to the output end of the first inverse processing submodule; the first inverse processing submodule is used to perform irreversible ISP sub-processing on the input image according to at least part of the target ISP parameters; and the correction module is used to correct the processing result of the first inverse processing submodule.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, electronic device, computer-readable medium, and program product. Background Art

[0002] Different deep learning models are typically trained on different image datasets. These datasets can be collections of RGB (Red, Green, and Blue) images, RAW images, or intermediate images from the Image Signal Processing (ISP) process.

[0003] RAW images or intermediate images from the ISP processing process often require real-world photography to obtain, but this process is not only time-consuming and labor-intensive, but also difficult to fully cover the various scenarios required by deep learning models. Compared to RAW images and intermediate images from the ISP processing process, RGB images are easier to obtain. For example, high-quality RGB images of various styles can be obtained from the internet.

[0004] However, if you choose to synthesize RAW images or intermediate images from the ISP processing process using easily available RGB images to construct the image dataset required by the deep learning model, you may face problems such as large errors between the synthesized images and the expected images, and the inability to determine the image metadata corresponding to the synthesized images. Summary of the Invention

[0005] The present disclosure provides an image processing method, an electronic device, a computer-readable medium, and a program product.

[0006] In a first aspect, an embodiment of the present disclosure provides an image processing method, comprising: inputting a first image into N inverse processing modules in sequence according to target image signal processing ISP parameters; N is a positive integer; obtaining a first result image output by the Nth inverse processing module; wherein at least some of the N inverse processing modules include: a first inverse processing sub-module and a correction module, and the input end of the correction module is connected to the output end of the first inverse processing sub-module; the first inverse processing sub-module is used to perform irreversible ISP sub-processing on the input image according to at least some of the target ISP parameters; and the correction module is used to correct the processing result of the first inverse processing sub-module.

[0007] In some embodiments, the method further includes: inputting the first result image output by the Nth inverse processing module into an enhancement module; the input end of the enhancement module is connected to the output end of the Nth inverse processing module, and the enhancement module is used to perform image enhancement processing on the first result image; and obtaining the second result image output by the enhancement module.

[0008] In some embodiments, the enhancement module includes: n downsampling modules connected in sequence, where n is an integer greater than or equal to 1; a self-attention module located after the n downsampling modules; n upsampling modules located after the self-attention module, where the n upsampling modules are connected in sequence; wherein the upsampling modules correspond one-to-one to the downsampling modules, and the input information of each upsampling module includes: the output information of the corresponding downsampling module, and the output information of the previous upsampling module connected to it.

[0009] In some embodiments, at least some of the N inverse processing modules include at least one second inverse processing sub-module; the at least one second inverse processing sub-module is used to perform reversible ISP sub-processing on the input image according to at least some target ISP parameters.

[0010] In some embodiments, the reversible ISP sub-processing of the input image based on at least part of the target ISP parameters includes: extending at least part of the target ISP parameters to the same dimension as the spatial dimension corresponding to the input image; feature-connecting at least part of the target ISP parameters of the same spatial dimension with the input image to obtain intermediate features; and performing reversible ISP sub-processing on the intermediate features using the at least one second inverse processing sub-module.

[0011] In some embodiments, after obtaining the first result image output by the Nth inverse processing module, it also includes: performing partial ISP processing on the first result image based on the target ISP parameters to obtain a second result image; or performing full ISP processing on the first result image based on the target ISP parameters to obtain a third result image.

[0012] In some embodiments, the method further includes: acquiring a fifth result image output by at least part of the inverse processing module.

[0013] In some embodiments, the first result image is an originally captured RAW image, and the first image is a red, green, and blue (RGB) image corresponding to the target ISP parameters.

[0014] In a second aspect, an embodiment of the present disclosure provides an electronic device comprising a memory and a processor; the memory stores a computer program that can be executed by the processor, and when the computer program is executed by the processor, the image processing method described in the first aspect is implemented.

[0015] In a third aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon, and when the computer program is executed by a processor, the image processing method described in the first aspect is implemented.

[0016] In a fourth aspect, an embodiment of the present disclosure provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the image processing method described in the first aspect.

[0017] In the embodiment of the present disclosure, the first image is sequentially input into N inverse processing modules, and at least some of the N inverse processing modules include a first inverse processing sub-module and a correction module, wherein the first inverse processing sub-module is used to perform irreversible ISP sub-processing on the input image according to at least some of the target ISP parameters. Introducing the target ISP parameters during the inverse processing can overcome the problem that the ISP parameters corresponding to the image cannot be determined after the inverse processing. The input end of the correction module is connected to the output end of the first inverse processing sub-module for correcting the processing result of the first inverse processing sub-module, which can reduce the processing error of the first inverse processing sub-module. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In the accompanying drawings of the embodiments of the present disclosure:

[0019] Figure 1 A flowchart of an image processing method provided in an embodiment of the present disclosure.

[0020] Figure 2 A schematic diagram of an exemplary process for color restoration of an image provided in an embodiment of the present disclosure.

[0021] Figure 3 A schematic diagram of the network architecture of an exemplary enhancement module provided in an embodiment of the present disclosure.

[0022] Figure 4 A schematic diagram of the structure of an exemplary convolution module provided in an embodiment of the present disclosure.

[0023] Figure 5 A schematic diagram of the architecture of the UNet network corresponding to the exemplary enhancement module provided in the embodiments of the present disclosure.

[0024] Figure 6 A schematic diagram of the structure of an exemplary self-attention module provided in an embodiment of the present disclosure.

[0025] Figure 7A schematic diagram of an exemplary image processing flow of a second inverse processing submodule provided in an embodiment of the present disclosure.

[0026] Figure 8 A schematic structural diagram of an electronic device provided in an embodiment of the present disclosure.

[0027] Figure 9 A schematic structural diagram of N exemplary inverse processing modules provided in an embodiment of the present disclosure.

[0028] Figure 10 A schematic diagram of the structure of an exemplary N-1 inverse processing modules and enhancement modules provided in an embodiment of the present disclosure.

[0029] Figure 11 A schematic diagram of an exemplary image processing flow of a first image provided in an embodiment of the present disclosure.

[0030] Figure 12 A schematic diagram of an exemplary RGB image to be processed provided in an embodiment of the present disclosure.

[0031] Figure 13 A schematic diagram of an exemplary first RGB image provided for an embodiment of the present disclosure.

[0032] Figure 14 A schematic diagram of an exemplary second RGB image provided for an embodiment of the present disclosure.

[0033] Figure 15 A structural diagram of another exemplary N inverse processing modules provided in an embodiment of the present disclosure.

[0034] Figure 16 A schematic diagram of an exemplary image processing flow of a first image provided in an embodiment of the present disclosure.

[0035] Figure 17 A schematic structural diagram of two exemplary inverse processing modules provided in an embodiment of the present disclosure.

[0036] Figure 18 A schematic diagram of an exemplary RGB image to be processed provided in an embodiment of the present disclosure.

[0037] Figure 19 A schematic diagram of an exemplary fourth RGB image provided for an embodiment of the present disclosure.

[0038] Figure 20 A schematic diagram of an exemplary fifth RGB image provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0040] The present disclosure will be described more fully hereinafter with reference to the accompanying drawings, but the illustrated embodiments may be embodied in different forms, and the present disclosure should not be construed as limited to the embodiments set forth below. Rather, these embodiments are provided so that the present disclosure will be thorough and complete and will fully understand the scope of the present disclosure to those skilled in the art.

[0041] The accompanying drawings of the embodiments of the present disclosure are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the detailed embodiments, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing the detailed embodiments with reference to the accompanying drawings.

[0042] The present disclosure may be described with reference to plan views and / or cross-sectional views by way of ideal schematic views of the present disclosure. Therefore, the exemplary illustrations may be modified according to manufacturing techniques and / or tolerances.

[0043] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0044] The terms used in this disclosure are only used to describe specific embodiments and are not intended to limit the disclosure. As used in this disclosure, the term "and / or" includes any and all combinations of one or more related enumerated items. As used in this disclosure, the singular forms "a" and "the" are also intended to include plural forms, unless the context clearly indicates otherwise. As used in this disclosure, the terms "comprising" and "made of" specify the presence of the features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof.

[0045] Unless otherwise defined, all terms (including technical and scientific terms) used in this disclosure have the same meanings as those commonly understood by those skilled in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning unless expressly defined in this disclosure.

[0046] The present disclosure is not limited to the embodiments shown in the drawings, but includes modifications of the configurations formed based on the manufacturing process. Therefore, the regions illustrated in the drawings have schematic properties, and the shapes of the regions shown in the drawings illustrate the specific shapes of the regions of the elements, but are not intended to be limiting.

[0047] Among some related technologies, deep learning networks are widely used in the field of image processing. The corresponding construction process of the network used for image processing usually includes: first determining the input object and output object, then designing the deep learning network, training the deep learning network according to the training set (a set of image data sets), and verifying and testing the trained deep learning network according to the validation set (another set of image data sets).

[0048] During this process, both the network training process and the post-training verification and testing processes are inseparable from the image dataset. The number, generalization, and quality of the sample images in this dataset will affect the ultimate training results of the deep learning network. Furthermore, different deep learning networks rely on different image datasets for training and verification. Therefore, it is rare for a single, large, and comprehensive image dataset to be suitable for the training, verification, and testing processes of all deep learning networks.

[0049] In a related technology, a set of image data is acquired through real-world photography. Although real-world photography can simultaneously acquire the corresponding image metadata (i.e., ISP parameters), the real-world photography process is time-consuming and labor-intensive, and the acquired images also have problems such as incomplete scene coverage. Compared to RAW images and intermediate images from the ISP processing process, RGB images are easier to acquire. RGB images can be obtained not only through real-world photography, but also from the internet in different styles and with high quality. By synthesizing sample images (RAW images and / or intermediate images from the ISP processing process) that meet the target camera characteristics (ISP parameters) from RGB images, the difficulty in acquiring RAW images and intermediate images from the ISP processing process can be overcome to a certain extent. However, how to avoid a large error between the sample images obtained through RGB image processing and the expected sample images, and how to avoid the sample images obtained through RGB image processing from being unable to determine the ISP parameters that can be acquired simultaneously during real-world photography, are issues that need to be urgently addressed.

[0050] Among them, RGB image refers to a color image composed of three channels of red, green and blue; RAW image refers to an unprocessed data image captured directly from the original image sensor of the camera.

[0051] In some embodiments, ISP parameters are parameters required by the hardware, processing unit, or a series of image processing algorithms used to process the image sensor output signal. Common ISP parameters include shutter time, exposure gain, white balance parameters, color reproduction parameters, and vignetting correction parameters, but this disclosure is not limited to these parameters.

[0052] In some embodiments, ISP parameters can be used to process the output image of the deep learning network through sample image simulation ISP Pipeline (Image Signal Processing Pipeline) to generate RGB images that conform to human visual perception, thereby facilitating the evaluation of the image quality performance of the network model through visual observation.

[0053] First, refer to Figure 1 , an embodiment of the present disclosure provides an image processing method, which includes the following steps S1 to S2.

[0054] S1. Process the ISP parameters according to the target image signal and input the first image into N inverse processing modules in sequence; N is a positive integer.

[0055] S2. Obtain a first result image output by the Nth inverse processing module.

[0056] At least some of the N inverse processing modules include: a first inverse processing submodule and a correction module, and an input end of the correction module is connected to an output end of the first inverse processing submodule.

[0057] The first inverse processing submodule is used to perform irreversible ISP subprocessing on the input image according to at least part of the target ISP parameters; the correction module is used to correct the processing result of the first inverse processing submodule.

[0058] In an embodiment of the present disclosure, a first image is sequentially input into N inverse processing modules, and at least some of the N inverse processing modules include a first inverse processing sub-module and a correction module, wherein the first inverse processing sub-module is used to perform irreversible ISP sub-processing on the input image according to at least some of the target ISP parameters. Introducing the target ISP parameters in the inverse processing process can overcome the problem of being unable to determine the ISP parameters corresponding to the image. The input end of the correction module is connected to the output end of the first inverse processing sub-module. The correction module can correct the image (i.e., the processing result) after the irreversible ISP sub-processing by the first inverse processing sub-module, thereby reducing the error generated by the irreversible ISP sub-processing process.

[0059] The present disclosure does not impose any particular limitation on the number of first images, which may be a single-frame image or multiple-frame images. In some embodiments, for image processing of multiple frames of first images, each of the first frames is input into N inverse processing modules to obtain multiple first result images.

[0060] In some embodiments, when the sample images required by the model to be trained are synthetic videos or datasets required for HDR (High Dynamic Range) processing, the first image is multiple frames.

[0061] The present disclosure does not impose any particular restrictions on the determination process of ISP parameters or the parameter types of ISP parameters. In some embodiments, ISP parameters can be a color restoration matrix, a white balance gain, a lens shading correction gain grid, etc.

[0062] As an example, given known camera calibration parameters, color reproduction matrices at multiple color temperatures are obtained through calibration. The color reproduction matrix at color temperature CCT1 is CCM1, and the color reproduction matrix at color temperature CCT2 is CCM2. The CCM (Color Correction Matrix) is a matrix used for color correction, typically a 3×3 matrix. It can be used to convert device-dependent color spaces into device-independent color spaces to ensure image color accuracy.

[0063] The color restoration matrix corresponding to the target scene color temperature corresponding to the RGB image to be converted is taken as CCT (Correlated Color Temperature). Correlated color temperature is a standard for measuring the color of light sources, and the unit is Kelvin (K). Specifically, CCT refers to when the color of light emitted by a light source matches the color of light radiated by a black body at a certain temperature, the color temperature of the light source is expressed by the temperature of the black body. The color restoration matrix corresponding to the target scene color temperature is calculated by formula (1) as CCT.

[0064] (1).

[0065] In some embodiments, irreversible ISP sub-processing refers to a process in which the original image before forward processing cannot be completely restored by performing reverse processing on the image. For example, the original image is , the first sub-image after forward ISP processing is , for the first sub-image Perform irreversible ISP sub-processing to obtain the second sub-image , the second sub-image Unable to completely restore the original image before processing .

[0066] Accordingly, reversible ISP sub-processing refers to a process in which the image can be completely restored to the original image before forward processing by performing reverse processing on the image.

[0067] In some embodiments, the correction process of the correction module is independent of at least some target ISP parameters. The correction module is trained based on a deep learning network to reduce the error between the image after the irreversible ISP sub-processing and the original image (i.e., the expected image). This error is caused by the irreversibility of the forward ISP processing corresponding to the inverse processing module.

[0068] In some embodiments, the network architecture of the deep learning network corresponding to the correction module is composed of an encoder and a decoder.

[0069] The embodiment of the present disclosure does not impose any particular limitation on the type of irreversible ISP sub-processing. As an example, the irreversible ISP sub-processing may be a color restoration process.

[0070] In some embodiments, the training process of the inverse processing module (including the first inverse processing submodule and the correction module) is:

[0071] Take multiple high-quality RGB images of different scenes to form the first training data set.

[0072] When the camera calibration parameters are known, ISP parameters are randomly generated for each RGB image within a preset value range (for example, white balance gain values ​​and digital gain values ​​are randomly generated for each RGB image).

[0073] For each RGB image I rgb,i , according to the generated ISP parameters, the first RAW image is obtained after being processed by the first inverse processing submodule , then the first RAW image The image is processed as the input image of the correction module and the second RAW image I is output. raw,i .

[0074] Utilize the forward processing module corresponding to the inverse processing module to process each second RAW image I raw,i Perform forward ISP processing to obtain the corresponding target RGB image .

[0075] According to each RGB image I rgb,i and its corresponding target RGB image , based on the back propagation of the loss optimization function, the trainable parameters in the inverse processing module (including the first inverse processing submodule and the correction module) are updated to obtain the trained inverse processing module.

[0076] Through the above training process, if the second RAW image I raw,i If the image is close to the expected image, the forward processing module corresponding to the inverse processing module is used to process the second RAW image I raw,i The target RGB image obtained after forward ISP processing Should be compared with the RGB image I rgb,i Therefore, according to the RGB image I rgb,i and its corresponding target RGB image , update the trainable parameters of the inverse processing module. The training process is simple and can enable the correction module in the inverse processing module to effectively correct the error caused by the irreversible ISP sub-processing, so that the second RAW image I raw,i Approximate to or identical to the expected image. The ISP parameters are applied in the first inverse processing submodule in the inverse processing module, and the correction module does not need to use the ISP parameters for correction.

[0077] In some other embodiments, the training process of the inverse processing module (including the first inverse processing submodule and the correction module) is:

[0078] A plurality of RAW images captured under known camera calibration parameters and their ISP parameters are selected to obtain a second training data set, for example, the existing data set AdobeFiveK.

[0079] For each RAW image I in the second training dataset raw,i , use RAW image processing tools to process RAW images according to ISP parameters I raw,i , get RGB image I rgb,i .in, , In RAW image I raw,i If the corresponding ISP parameters are missing, the missing ISP parameters are determined based on the known camera calibration parameters.

[0080] For each RAW image I raw,i The corresponding RGB image I rgb,i , the RGB image I is processed in sequence by using the first inverse processing submodule of the inverse processing module and the correction module rgb,i Process and get the target RAW image .

[0081] According to each RAW image I raw,i and its corresponding target RAW image , based on the back propagation of the loss optimization function, the trainable parameters in the inverse processing module are updated to obtain the trained inverse processing module.

[0082] As an example of an embodiment of the present disclosure, refer to Figure 2 , when image I src When the corresponding ISP processing is color restoration, the image I src The corresponding forward ISP processing process is as follows:

[0083] Image I src Input color restoration module 201 and forward correction module 202, image I is converted into src Perform forward color restoration to obtain the first sub-image I dst :

[0084] (2);

[0085] in, It is a 3×3 color restoration matrix CCM (i.e. ISP parameter), and the matrix CCM is a reversible matrix. dst is the first sub-image.

[0086] is the color restoration formula, and its calculated value depends on the value of x (i.e. ) is corrected according to formula (3):

[0087] (3);

[0088] MaxValue is the preset maximum value of x.

[0089] From the forward ISP processing (i.e., color restoration processing clip) corresponding to the above formulas (2) and (3), it can be seen that the corresponding inversion operation is an irreversible processing process, that is, the inversion operation cannot directly convert the first sub-image I dst Completely restore the image before color restoration I src .

[0090] Therefore, the inverse processing module learns the inverse operation based on NN (Neural Network). The trained inverse processing module includes a first inverse processing submodule and a reverse correction module. When the first sub-image I dst When performing reverse ISP processing, the output of the reverse correction module is the second sub-image , the second sub-image Input the first inverse processing submodule, according to the color restoration matrix CCM, through the inverse operation , get image I src .

[0091] In some embodiments, the method further comprises:

[0092] Inputting the first result image output by the Nth inverse processing module into an enhancement module; the input end of the enhancement module is connected to the output end of the Nth inverse processing module, and the enhancement module is used to perform image enhancement processing on the first result image;

[0093] A second result image output by the enhancement module is obtained.

[0094] In the disclosed embodiment, the enhancement module is used to perform image enhancement processing on the first result image. When the second result image is further used to train a different model, the enhancement module can suppress the error between the synthesized sample image (the second result image) and the expected sample image.

[0095] In some embodiments, the enhancement module is trained based on a deep learning network. Figure 3 ,The network architecture of the deep learning network corresponding to the enhancement module is composed of an encoder and a decoder. N-1 is the first result image, I N is the second result image.

[0096] In some embodiments, the enhancement module includes:

[0097] n downsampling modules connected in sequence, where n is an integer greater than or equal to 1;

[0098] a self-attention module following the n downsampling modules;

[0099] n upsampling modules located after the self-attention module, wherein the n upsampling modules are connected in sequence;

[0100] The up-sampling modules correspond to the down-sampling modules on a one-to-one basis, and the input information of each up-sampling module includes: the output information of the corresponding down-sampling module and the output information of the previous up-sampling module connected thereto.

[0101] In an embodiment of the present disclosure, each downsampling module includes a downsampling submodule and a convolution module, and each upsampling module includes an upsampling submodule and a convolution module. In some embodiments, in response to an upsampling module receiving first output information of a downsampling module corresponding to the upsampling module and second output information of a previous upsampling module connected to the upsampling module, the first output information is concatenated with the second output information.

[0102] Each convolutional module is composed of one or more stacked convolutional layers and activation layers. The downsampling submodule is composed of a pooling layer. In some embodiments, the pooling layer is a max pooling layer. The upsampling submodule is composed of a transposed convolutional layer or a bilinear upsampling layer.

[0103] In some embodiments, the self-attention module can be replaced by a convolutional layer module.

[0104] In some embodiments, when the number of inverse processing modules is N-1, the training process of the enhancement module is:

[0105] The third training dataset is composed of multiple high-quality RGB images of different scenes.

[0106] Given known camera calibration parameters, the ISP parameters are randomly generated or determined for each RGB image within a preset range. For example, white balance gain and digital gain values ​​are randomly generated, and the color reproduction matrix (CCM) at multiple color temperatures is determined through calibration.

[0107] For each RGB image I rgb,i , according to the ISP parameters, after (N-1) inverse processing modules, the first result image is obtained , the image is a RAW image. The processing of the (N-1) inverse processing modules corresponds to formula (4).

[0108] (4).

[0109] in, It means that it is processed by (N-1) inverse processing modules in sequence, that is, Indicates the processing of the N-1th inverse processing module, Indicates the processing of the first inverse processing module.

[0110] RAW images It is processed as the input image of the enhancement module and outputs the second result image I raw,i , the image is a RAW image. The processing of the enhancement module corresponds to formula (5).

[0111] (5).

[0112] in, Indicates that it has been processed by the enhancement module.

[0113] Utilize the (N-1) forward processing modules corresponding to the (N-1) inverse processing modules to process each second result image I raw,i Perform forward ISP processing to obtain the corresponding target RGB image The forward ISP processing process of the (N-1) forward processing modules corresponds to formula (6).

[0114] (6).

[0115] in, It means that it is processed in sequence by (N-1) forward processing modules. Indicates the processing of the N-1th forward processing module, Indicates the processing of the first forward processing module.

[0116] According to each RGB image I rgb,i and its corresponding target RGB image , based on the back propagation of the loss optimization function, the trainable parameters in the enhancement module are updated. The calculation process of the loss optimization function corresponds to formula (7).

[0117] (7).

[0118] Where N is the RGB image I in the third training dataset rgb,i The number of

[0119] Through the above training process, if the second result image I raw,i If the image is close to the expected image, the (N-1) forward processing modules corresponding to the (N-1) inverse processing modules are used to process the second result image I raw,i The target RGB image obtained after forward ISP processing Should be the same as the original input RGB image I rgb,i Similar or identical, therefore, according to the RGB image I rgb,i and its corresponding target RGB image , updating the trainable parameters of the enhancement module. This training process is both simple and enables the enhancement module to effectively correct the errors corresponding to the processing results of the (N-1) inverse processing modules (i.e., the first result image), so that the second result image approaches or is identical to the expected image. The ISP parameters are used in the first inverse processing submodule of the inverse processing module, and the correction module does not need to use ISP parameters for correction.

[0120] In some other embodiments, when the number of inverse processing modules is N-1, the training process of the enhancement module is:

[0121] A fourth training data set is obtained by selecting multiple RAW images taken under known camera calibration parameters and their ISP parameters.

[0122] For each RAW image I in the fourth training dataset raw,i , use RAW image processing tools to process RAW images according to ISP parameters I raw,i , get RGB image I rgb,i .in, , In RAW image I raw,i If the corresponding ISP parameters are missing, the missing ISP parameters are determined based on the known camera calibration parameters.

[0123] For each RAW image I raw,i The corresponding RGB image I rgb,i , using (N-1) inverse processing modules and enhancement modules to sequentially process the RGB image I rgb,i Process and get the target RAW image .

[0124] According to each RAW image I raw,i and its corresponding target RAW image , based on the back propagation of the loss optimization function, the trainable parameters in the enhancement module are updated to obtain the trained enhancement module. The calculation process of the loss optimization function corresponds to formula (8).

[0125] (8).

[0126] Where N is the RAW image I in the fourth training dataset raw,i The number of

[0127] As an example, see Figure 4 The convolution modules in the downsampling module and the upsampling module are composed of stacked i convolution layers and activation layers. The output information of the convolution layer in each convolution module is used as the input information of the activation layer, and the output information of the activation layer in each convolution module is used as the input information of the convolution layer in the next convolution module in the stack.

[0128] As an example of an embodiment of the present disclosure, refer to Figure 5 The enhancement module is a residual structure, and the residual branch uses a UNet (U-shaped Network) with self-attention. The network includes an encoder and a decoder.

[0129] The encoder is composed of three modules for downsampling (i.e., module 501, module 502, module 503) connected in sequence. Figure 5 , wherein each module 501, 502, 503 includes a downsampling submodule and a convolution module, and the three modules 501, 502, 503 form a contraction path of the network. The first result image I N-1 With each downsampling block in the contraction path, the resolution decreases and the number of channels doubles.

[0130] The decoder is composed of three modules for upsampling (i.e., module 505, module 506, module 507) connected in sequence. Figure 5 ,Each module 505, 506, 507 includes an upsampling submodule and a convolution module. The three modules 505, 506, 507 constitute the expansion path of the network. When the image passes through each upsampling module in the contraction path, the resolution is reduced and the number of channels is doubled.

[0131] First result image I N-1 The processing process of inputting the UNet network corresponding to the enhancement module includes:

[0132] First result image I N-1Input modules 501 to 503 in sequence, the input information of module 501 is the first result image I N-1 , obtaining the first intermediate output information; the first intermediate output information is input into module 502, obtaining the second intermediate output information; the second intermediate output information is input into module 503, obtaining the third intermediate output information.

[0133] The third intermediate output information is used as the input information of the self-attention module 504, and after the self-attention calculation process, the fourth intermediate output information is obtained; the third intermediate output information and the fourth intermediate output information are connected and input into module 505 to obtain the fifth intermediate output information; the fifth intermediate output information and the second intermediate output information output by module 502 are connected and input into module 506 to obtain the sixth intermediate output information; the sixth intermediate output information and the first intermediate output information output by module 501 are connected and input into module 507 to obtain the seventh intermediate output information.

[0134] The seventh intermediate output information is combined with the first result image I N-1 Add and get the second result image I N .

[0135] Among them, reference Figure 6 , the calculation process of the self-attention module 504 includes:

[0136] The input information of the self-attention module 504 is x, , the output information is y, The preset trainable parameters include W Q 、W K 、W V and W0, where W Q 、W K 、W V and W0 are matrix parameters, , , , for the calculation of single-head attention, .

[0137] According to the preset trainable parameter W Q 、W K 、W V And the input information x of the self-attention module 504 is calculated by formula (9) to obtain the tensor: query Q, key K and value V.

[0138] (9).

[0139] in, , , , N is the number of input information x (batch size), h is the number of attention heads. When calculating single-head attention, h=1. Token count is the number of tokens, d K is the dimension of the key K, d V is the dimension of the value V. The linear transformation in formula (9) is implemented by batch matrix multiplication.

[0140] According to the query Q and key K, the weighted weight of the value (V) is determined by formula (10).

[0141] (10).

[0142] in, is the transposed value of key K, is the dimension of the key K, is the Softmax function (normalized exponential function), an activation function, , .

[0143] The output linear transformation W is performed by formula (11) O , and get the output information y.

[0144] (11).

[0145] The linear transformation in formula (11) is implemented through batch matrix multiplication.

[0146] In some embodiments, at least some of the N inverse processing modules include at least one second inverse processing submodule;

[0147] The at least one second inverse processing submodule is used to perform reversible ISP sub-processing on the input image according to at least part of the target ISP parameters.

[0148] In an embodiment of the present disclosure, the second inverse processing submodule performs reversible ISP sub-processing on the input image based on at least part of the target ISP parameters. The reversible ISP sub-processing process can completely restore the original image before forward processing by performing inverse processing on the image.

[0149] In some embodiments, the second inverse processing submodule is a reversible affine layer, and at least one second inverse processing submodule is a plurality of stacked reversible affine layers.

[0150] In some embodiments, for each of the stacked multiple reversible affine layers, a reversible 1x1 convolution is used as a learnable permutation function to reverse the channel order.

[0151] In some embodiments, performing reversible ISP sub-processing on the input image according to at least part of the target ISP parameters includes:

[0152] Expanding at least part of the target ISP parameters to a dimension that is the same as a spatial dimension corresponding to the input image;

[0153] Performing feature concatenation on at least part of the target ISP parameters of the same spatial dimension and the input image to obtain intermediate features;

[0154] The at least one second inverse processing submodule is used to perform reversible ISP sub-processing on the intermediate features.

[0155] In an embodiment of the present disclosure, at least part of the target ISP parameters are extended to the same dimension as the spatial dimension corresponding to the input image, and the target ISP parameters of the same dimension are feature-connected with the input image, so that the target ISP parameters can be used as input of the second inverse processing sub-module for reversible ISP sub-processing, and the second inverse processing sub-module can be applicable to reversible ISP sub-processing corresponding to different target ISP parameters.

[0156] In some embodiments, reference Figure 7 , expand the target ISP parameter 702 with the number of channels k to the input feature 701 with the number of channels d (image I in ) of the same spatial dimension, and then the target ISP parameters 702 of the same dimension and the input features 701 (image I in ) to perform feature connection to obtain an intermediate feature 703 (i.e., the connected feature). The number of channels of the intermediate feature 703 is d+k. The intermediate feature 703 is input into a reversible deep learning layer 704 (composed of at least one second inverse processing submodule) for reversible ISP sub-processing to obtain an output feature 705 (image I out ), the number of channels of the output feature 705 is Dd, where D is the image I out The total number of channels.

[0157] In some embodiments, the input features of the stacked multiple reversible affine layers are images I in , the output feature is I out .in, , For RGB images, the number of channels D is 3; for RAW images, the same color pixels are stored in each channel, and the number of channels D is 4.

[0158] For each reversible affine layer, the output feature image I is calculated by formula (12) out (Dd) dimensional data in the channel dimension.

[0159] (12).

[0160] For each reversible affine layer, the output feature image I is calculated by formula (13) out The first d dimensions of data in the channel dimension.

[0161] (13).

[0162] Among them, 0 <d<k, Represents image I out (Dd) dimensional data in the channel dimension; Represents image I in The first d-dimensional data in the channel dimension; Represents element-wise multiplication; Represents image I out The first d dimensions of data in the channel dimension, Represents image I in (Dd) dimensional data in the channel dimension; 、 and is a function used for different deep learning tasks, whose internal feature channel dimension is more than d and (Dd); p is a predetermined target ISP parameter.

[0163] As an example of an embodiment of the present disclosure, when the target ISP parameter is white balance gain, the image I of the input feature in When it is a RAW image, the spatial dimension of the RAW image is half of the image spatial resolution, that is, .

[0164] White balance gain includes: WBGain of RAW image on R channel R , RAW image gain WBGain on Gr channel Gr , RAW image gain on the Gb channel WBGain Gb And the gain WBGain of the RAW image on the B channel B For RAW images, the number of channels D is 4. After the white balance gain is expanded to the same dimension as the spatial dimension corresponding to the RAW image, the parameters of the white balance gain are . Among them, each element P of the first channel of P i,j,1 =WBGain R , each element P of the second channel i,j,2 =WBGain Gr , each element P of the third channel i,j,3 =WBGain Gb , each element P of the fourth channel i,j,4 =WBGainB .

[0165] As another example of the embodiment of the present disclosure, when the target ISP parameter is the lens shading correction gain grid LSCGainGrid, the image I of the input feature in When it is a RAW image, the spatial dimension of the RAW image is half of the image spatial resolution, that is, .in, .

[0166] In w lsc and h lsc In the case of 32×32, w lsc Much smaller than , h lsc Much smaller than , the bilinear interpolation algorithm LSCGain=BilinearInterpolate(LSCGainGrid) is used to upsample the lens shading correction gain grid LSCGainGrid. The spatial resolution of the upsampled lens shading correction gain grid parameter LSCGain is equal to ,Right now .

[0167] In some embodiments, the first result image is an originally captured RAW image, and the first image is an RGB image corresponding to the target ISP parameters.

[0168] In some embodiments, after obtaining the first result image output by the Nth inverse processing module, the method further includes:

[0169] Based on the target ISP parameters, perform partial ISP processing on the first result image to obtain a second result image; or

[0170] Based on the target ISP parameters, all ISP processing is performed on the first result image to obtain a third result image.

[0171] In an embodiment of the present disclosure, when the first image is an RGB image and the first result image is a RAW image, N inverse processing modules are sequentially input based on the first image to perform inverse ISP processing to obtain a first result image, which is close to or equivalent to the image captured by the original image sensor.

[0172] The first result image is used as the image captured by the original image sensor, and partial ISP processing is performed on it according to the target ISP parameters to obtain a second result image. This second result image is an intermediate image during the ISP processing. The first result image is used as the image captured by the original image sensor, and full ISP processing is performed on it according to the target ISP parameters to obtain a third result image. This third result image is an RGB image. The first, second, third, and fourth result images can all be used as sample images for models with different training requirements.

[0173] In some embodiments, the sample data set includes: ISP parameters, at least one of a first result image, a second result image, a third result image, and a fourth result image.

[0174] In some embodiments, the first image may also be an intermediate image in the ISP processing process corresponding to the target ISP parameters, or may be a RAW image.

[0175] The embodiment of the present disclosure does not impose any special restrictions on the image type used as the sample image, and it can be determined according to training requirements.

[0176] It is worth noting that in addition to performing partial or full ISP processing on the first result image or the second result image to obtain the intermediate image after ISP processing and the RGB image after ISP processing, respectively, the first result image or the second result image can also be processed by adding noise, adding lens shadow effects, etc. according to the training requirements of different models. The present disclosure is not limited to this.

[0177] In some embodiments, partial ISP processing may be performed on the second result image based on the target ISP parameters, or the second result image may be fully ISP processed.

[0178] In some embodiments, additional processing is performed on the first result image and / or the second result image so that the additionally processed images can be suitable for the training requirements of various models, for example, adding noise, adding lens shading effects, etc., which is not limited in this disclosure.

[0179] As an example, for each first image I in a set of first images to be processed RGB , according to the target ISP parameters, the first image I RGB Input N inverse processing modules in sequence to obtain the result image I output by the Nth inverse processing module RAW The first image set after processing is used as the sample image of the denoising model, so it is necessary to RAW Add noise.

[0180] The standard deviation of the added noise is determined by formula (14):

[0181] (14).

[0182] in, is the Poisson noise intensity, is the Gaussian noise intensity.

[0183] As an example, for each second image I in a set of second images to be processed RGB , according to the target ISP parameters, the second image I RGB Input N inverse processing modules in sequence to obtain the result image I output by the Nth inverse processing module RAW The processed second image set is used as the sample image of the lens shading correction model, so it is necessary to perform a RAW Performs positive lens shading correction.

[0184] In some embodiments, the method further comprises:

[0185] A fifth result image output by at least part of the inverse processing module is obtained.

[0186] In an embodiment of the present disclosure, a fifth result image output by at least a portion of the inverse processing module may serve as an intermediate image during ISP processing. The fifth result image refers to an image that has undergone inverse ISP processing by at least a portion of the inverse processing module and retains the ISP processing results corresponding to the remaining inverse processing modules.

[0187] In the above embodiment of the present disclosure, the first image is sequentially input into N inverse processing modules, and at least some of the N inverse processing modules include a first inverse processing sub-module and a correction module, wherein the first inverse processing sub-module is used to perform irreversible ISP sub-processing on the input image according to at least some of the target ISP parameters, and introduce the target ISP parameters in the inverse processing process, thereby overcoming the problem of being unable to determine the ISP parameters corresponding to the image in the inverse processing process, and the input end of the correction module is connected to the output end of the first inverse processing sub-module, and is used to correct the processing result of the first inverse processing sub-module, thereby suppressing the error between the acquired first result image and the expected image.

[0188] Among them, the target ISP parameters are introduced into the first inverse processing sub-module of the inverse processing module. The parameters can be calculated through the calibration data of the real camera or set by the virtual camera. The inverse processing module is trained using multiple sets of images and their corresponding ISP parameters, thereby improving the accuracy of the output image of the inverse processing module.

[0189] Furthermore, the number N of inverse processing modules, as well as the output and input images, is configurable, making it easier to obtain sample images based on models with different training requirements. For example, if the model to be trained requires an intermediate image from the ISP processing process, the RGB image can be sequentially input into N inverse processing modules, with the output images from the 1st to the N-1st inverse processing modules serving as the intermediate images (sample images) during the ISP processing. Alternatively, the RGB image can be input into N inverse processing modules, with the first resulting image output from the Nth inverse processing module serving as the RAW image. This RAW image can then be partially ISP processed to obtain the intermediate image (sample image) during the ISP processing.

[0190] In a second aspect, the present disclosure provides an electronic device, referring to Figure 8 , which includes a memory 802 and a processor 801; the memory 802 stores a computer program that can be executed by the processor 801, and when the computer program is executed by the processor 801, the image processing method described in the first aspect is implemented.

[0191] The processor 801 and the memory 802 are connected via one or more I / O interfaces 803 , and the interfaces 803 are configured to implement information exchange between the processor 801 and the memory 802 .

[0192] The processor 801 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 802 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read-write interface) 803 is connected between the processor 801 and the memory 802, and can realize information exchange between the processor 801 and the memory 802, including but not limited to a data bus (Bus), etc.

[0193] It should be noted that the electronic device is a device corresponding to the above-mentioned image processing method, and all implementation methods in the above-mentioned method embodiments are applicable to the embodiments of the device and can achieve the same technical effects.

[0194] In a third aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon, and when the computer program is executed by a processor, the image processing method described in the first aspect is implemented.

[0195] It should be noted that the computer-readable medium is a medium corresponding to the above-mentioned image processing method. All implementation methods in the above-mentioned method embodiments are applicable to the embodiments of the computer-readable medium and can achieve the same technical effects.

[0196] In a fourth aspect, an embodiment of the present disclosure provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the image processing method described in the first aspect.

[0197] It should be noted that the computer program product is a program product corresponding to the above-mentioned image processing method. All implementation methods in the above-mentioned method embodiments are applicable to the embodiments of the computer program product and can achieve the same technical effects.

[0198] In order to enable those skilled in the art to more clearly understand the technical solutions provided by the embodiments of the present disclosure, the technical solutions provided by the embodiments of the present disclosure are described in detail below through specific examples.

[0199] Example 1

[0200] For example, as a specific form of the embodiment of the present disclosure, refer to Figure 9 The model to be trained requires multiple target sample images for training. The N modules for inverse processing process the first image into the sample image required by the model to be trained. The first image is an RGB image 901, and its color space can be sRGB or other RGB color spaces. The process of obtaining the target sample image is as follows:

[0201] The RGB image 901 is sequentially input into the first module 902, the second module...the N-1 module and the N-1 module 903 for inverse processing, and the output image of the N-1 module 903 is determined as the sample image 904. Among them, the N-1 module 903 is trained based on a deep learning network, and is used to enhance the image output by the N-1 module and suppress the errors generated in the processing of the RGB image 901 by the first to N-1 modules. The first to N-1 modules can be modules trained based on a deep learning network, or they can be traditional ISP pipeline processing modules for direct inversion. The sample image 904 can be the target sample image required by the model to be trained, or it can be a quasi-sample image. In the case that the sample image 904 is a quasi-sample image, the sample image 904 is further processed to obtain the target sample image required by the model to be trained.

[0202] It is worth noting that, in other examples, when the Nth module is not a model trained based on a deep learning network, at least one module among the 1st to N-1th modules is a module trained based on a deep learning network.

[0203] Example 2

[0204] For example, as a specific form of the embodiment of the present disclosure, refer to Figure 10, the first image is processed into the target sample image required by the model to be trained, the first image is an RGB image, and the first image is processed in sequence using N-1 inverse processing modules and the deep learning enhancement module 1002. Figure 11 , the processing of the first image includes:

[0205] Step 1101 : Select an RGB image to be processed (a first image). The RGB image may be a single-frame image or a multi-frame image.

[0206] Step 1102: Input the RGB image into N-1 inverse processing modules to obtain a first RAW image; wherein each inverse processing module performs inverse ISP pipeline processing on the RGB image.

[0207] Specifically, the RGB image is , ,in, is the resolution of the RGB image, and 3 is the number of color channels of the RGB image.

[0208] When the first inverse processing module 1001 performs tone inverse mapping on the RGB image, the global tone mapping is an S-shaped curve. After being input into the first inverse processing module 1001, the first intermediate image I1 is obtained. .

[0209] When the second inverse processing module performs inverse Gamma transformation on the first intermediate image I1, , processing the first intermediate image I1 into the second intermediate image I2.

[0210] Accordingly, the first RAW image output by the N-1th inverse processing module is I N-1 ,in, .

[0211] Step 1103: input the first RAW image into the deep learning enhancement module 1002 for correction processing to obtain a second RAW image I N The deep learning enhancement module 1002 is obtained based on deep learning network training and is used to reduce the first RAW image I N-1 The error from the expected RAW image.

[0212] Step 1104: According to the training requirements of the model to be trained, the second RAW image I N Perform additional post-processing to obtain the target sample image. Additional post-processing can include adding noise, adding lens shading effects, etc.

[0213] Step 1105 , record the target ISP parameters involved in the 1st to N-1th inverse processing modules, and use the target ISP parameters as image metadata.

[0214] Step 1106 : Determine a sample image dataset for the model to be trained, wherein the sample image may be one or more of the second RAW image, the target sample image, the RGB image to be processed, and image metadata.

[0215] In the above example, refer to Figure 12 , Figure 12 An RGB image to be processed (ie, the first image) is shown.

[0216] The RGB image to be processed is processed through steps 1101 to 1102 to obtain a first RAW image I N-1 , and for the first RAW image I N-1 The first RGB image is obtained by performing the forward ISP processing corresponding to the 1st to N-1th inverse processing modules. Figure 13 , Figure 13 The first RGB image is shown, in which the overexposed area of ​​the white flower in the first RGB image has obvious purple pseudo-color defects. The pseudo-color defects are generated by the first RAW image I generated by the processing of the 1st to N-1th inverse processing modules. N-1 The error between the image and the expected RAW image.

[0217] The first RAW image I N-1 After processing in step 1103, the second RAW image I is obtained. N , and for the second RAW image I N The forward ISP processing corresponding to the 1st to N-1th inverse processing modules is used to obtain the second RGB image. Figure 14 , Figure 14 The second RGB image is shown, in which no purple pseudo-color artifacts appear. It can be seen that the deep learning enhancement module 1002 can effectively eliminate the accumulated errors in the processing of the first to N-1 inverse processing modules, thereby avoiding obvious artifacts caused by overexposed areas.

[0218] Example 3

[0219] For example, as a specific form of the embodiment of the present disclosure, refer to Figure 15 , processing the first image into the target sample image required by the model to be trained, the first image being an RGB image, and sequentially processing the first image using N inverse processing modules, at least some of which include a first inverse processing submodule and a correction module. Figure 16 , the processing of the first image includes:

[0220] Step 1601: Select an RGB image to be processed (first image). The RGB image may be a single-frame image or a multi-frame image.

[0221] Step 1602: Determine target ISP parameters corresponding to the RGB image, that is, image metadata.

[0222] Step 1603: Input the RGB image into N inverse processing modules to obtain a third RAW image; wherein, each inverse processing module performs inverse ISP pipeline processing on the RGB image, and at least some of the inverse processing modules are trained based on a deep learning network containing ISP parameters, and are used to perform irreversible ISP sub-processing and correction on the input image according to at least some of the target ISP parameters.

[0223] In step 1604, the third RAW image is subjected to additional post-processing according to the training requirements of the model to be trained to obtain a target sample image. The additional post-processing may include adding noise, adding lens shading effects, etc.

[0224] Step 1605: Obtain a sample image dataset of the model to be trained, wherein the sample image may be one or more of the third RAW image, the target sample image, the RGB image to be processed, and image metadata.

[0225] Example 4

[0226] For example, as a specific form of the embodiment of the present disclosure, Figure 15 Based on the structural diagram of the N inverse processing modules shown in FIG. Figure 17 , if the number of inverse processing modules is 2, the processing of the first image includes:

[0227] An RGB image 1701 (first image) to be processed is selected. The RGB image 1701 may be a single-frame image or a multi-frame image.

[0228] RGB image 1701 is input into a first inverse processing module 1702 to obtain a first intermediate processed image. First inverse processing module 1702 includes a first inverse processing submodule and a correction module. The RGB image 1701 is input into the first inverse processing submodule, which performs irreversible inverse ISP processing on the RGB image 1701 based on at least some target ISP parameters. The correction module has an input connected to an output of the first inverse processing submodule for correcting the image output by the first inverse processing submodule.

[0229] The first intermediate processed image is input into the mosaic module 1703 to obtain a second intermediate processed image; wherein the mosaic module 1703 is a module that does not require training.

[0230] The second intermediate processed image is input into a second inverse processing module 1704 to obtain a sample image 1705. Second inverse processing module 1704 includes at least one second inverse processing sub-module configured to perform reversible ISP sub-processing on the input image based on at least a portion of the target ISP parameters. Sample image 1705 is obtained by retaining the value of only one color channel at each pixel position in the image.

[0231] Among them, the process of determining at least part of the target ISP parameters required by the first inverse processing module 1702 and the second inverse processing module 1704 is: inputting the target ISP parameters (i.e., image metadata) into the first ISP parameter calculation module to obtain the target ISP parameters for the first inverse processing module; and, inputting the target ISP parameters (i.e., image metadata) into the second ISP parameter calculation module to obtain the target ISP parameters for the second inverse processing module.

[0232] In the above example, refer to Figure 18 , Figure 18 An RGB image to be processed (ie, the first image) is shown.

[0233] The RGB image to be processed is processed by the first inverse processing submodule of the first inverse processing module 1702 to obtain a third RAW image, and the third RAW image is processed by the forward ISP corresponding to the first inverse processing module 1702 to obtain a fourth RGB image. Figure 19 , Figure 19 The fourth RGB image shows overexposed areas of the sky and lake, with discontinuities, outlines, and purple pseudo-color artifacts. The pseudo-color artifacts are caused by the error between the third RAW image and the expected image.

[0234] The third RAW image is processed by the correction module to obtain a first intermediate processed image, and the first intermediate processed image is processed by the forward ISP corresponding to the first inverse processing submodule to obtain a fifth RGB image. Figure 20 , Figure 20 The fifth RGB image is shown, in which no faults, contours or purple pseudo-color defects appear. It can be seen that the error in the processing process corresponding to the first inverse processing submodule can be effectively eliminated by setting the correction module.

[0235] Among them, the processor is a device with data processing capabilities, including but not limited to the central processing unit (CPU); the memory is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically such as SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); the I / O interface (read-write interface) is connected between the processor and the memory, which can realize information exchange between the memory and the processor, including but not limited to the data bus (Bus), etc.

[0236] Those skilled in the art will appreciate that all or some of the steps, systems, and functional modules / units in the apparatus disclosed above may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0237] In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed by several physical components in cooperation.

[0238] Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit (CPU), a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on computer-readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM, more specifically, SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH) or other magnetic disk storage; compact disc read-only ROM (CD-ROM), digital versatile disk (DVD) or other optical disk storage; magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage; and any other medium that can be used to store the desired information and that can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0239] The present disclosure has disclosed example embodiments, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for purposes of limitation. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly indicated, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, it will be understood by those skilled in the art that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. An image processing method, characterized in that: include: According to the target image signal processing ISP parameters, the first image is sequentially input into N inverse processing modules; N is a positive integer; Obtaining a first result image output by the Nth inverse processing module; Wherein, at least part of the N inverse processing modules include: a first inverse processing submodule and a correction module, and the input end of the correction module is connected to the output end of the first inverse processing submodule; The first inverse processing submodule is used to perform irreversible ISP subprocessing on the input image according to at least part of the target ISP parameters; the correction module is used to correct the processing result of the first inverse processing submodule; At least some of the N inverse processing modules include at least one second inverse processing submodule; The at least one second inverse processing submodule is configured to perform reversible ISP sub-processing on the input image according to at least part of the target ISP parameters; The reversible ISP sub-processing process is a process of completely restoring the original image before forward processing by performing inverse processing on the image; the second inverse processing sub-module is a reversible affine layer, and at least one second inverse processing sub-module is a stack of multiple reversible affine layers.

2. The method according to claim 1, characterized in that The method further comprises: Inputting the first result image output by the Nth inverse processing module into an enhancement module; the input end of the enhancement module is connected to the output end of the Nth inverse processing module, and the enhancement module is used to perform image enhancement processing on the first result image; A second result image output by the enhancement module is obtained.

3. The method according to claim 2, characterized in that The enhancement module includes: n downsampling modules connected in sequence, where n is an integer greater than or equal to 1; a self-attention module following the n downsampling modules; n upsampling modules located after the self-attention module, wherein the n upsampling modules are connected in sequence; The up-sampling modules correspond to the down-sampling modules on a one-to-one basis, and the input information of each up-sampling module includes: the output information of the corresponding down-sampling module and the output information of the previous up-sampling module connected thereto.

4. The method according to claim 1, wherein The performing reversible ISP sub-processing on the input image according to at least part of the target ISP parameters comprises: Expanding at least part of the target ISP parameters to a dimension that is the same as a spatial dimension corresponding to the input image; Performing feature concatenation on at least part of the target ISP parameters of the same spatial dimension and the input image to obtain intermediate features; The at least one second inverse processing submodule is used to perform reversible ISP sub-processing on the intermediate features.

5. The method according to claim 1, wherein After obtaining the first result image output by the Nth inverse processing module, the method further includes: Based on the target ISP parameters, perform partial ISP processing on the first result image to obtain a second result image; or Based on the target ISP parameters, all ISP processing is performed on the first result image to obtain a third result image.

6. The method according to claim 1, characterized in that The method further comprises: A fifth result image output by at least part of the inverse processing module is obtained.

7. The method according to claim 1, characterized in that The first result image is the original captured RAW image, and the first image is the red, green, and blue (RGB) image corresponding to the target ISP parameters.

8. An electronic device, characterized in that: The image processing apparatus comprises a memory and a processor; the memory stores a computer program that can be executed by the processor, and the computer program, when executed by the processor, implements the image processing method according to any one of claims 1 to 7.

9. A computer-readable medium, characterized in that A computer program is stored, and when the computer program is executed by a processor, the image processing method according to any one of claims 1 to 7 is implemented.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the image processing method according to any one of claims 1 to 7 when executed by a processor.

Images