Training sample pair acquisition method and device, electronic equipment and storage medium

Abstract

The invention provides a training sample pair acquisition method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the inverse operation processing of an image signal processing pipeline of a to-be-processed image, and obtaining at least one frame of target RAW image; performing noise adding processing on each frame of target RAW image to obtain an initial noise image corresponding to each frame of target RAW image, and forming an initial image pair by each frame of target RAW image and the corresponding initial noise image; the initial image pair is input into a camera image processing model of the electronic equipment to be processed into an RGB image, a noise sample image and a clean sample image before noise reduction processing are obtained, and the noise sample image and the clean sample image form a training sample pair. According to the embodiment, the training sample pair is matched with the noise model of the image actually shot by the camera, so that the accuracy of the neural network trained by the training sample pair is improved, and the image noise reduction effect is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a method and device for acquiring training sample pairs, an electronic device, and a storage medium. Background Art

[0002] Image denoising uses prior knowledge of the image to remove noise from the image and retain the more important details in the image, making the output image look clear and clean. The input of the image denoising task can be a single frame image or multiple frames of continuously exposed images.

[0003] In the process of imaging or transmission, images in reality are often interfered by the imaging equipment and external environment noise. The images produced by this influence are called noisy images or noisy images. If they are not processed by image denoising algorithms, most of the effective information in the image will be destroyed by noise.

[0004] In the related technology, image denoising includes a learning-based method, that is, using noisy images and clean images to train a neural network. Since the neural network has strong fitting ability and image understanding ability, it can learn the mapping relationship between noisy images and clean images; using the trained neural network to denoise the image can achieve a better denoising effect.

[0005] However, the noise model corresponding to the dataset used to train the neural network may not match the noise model of the actual processed image, which makes it difficult for the neural network to map the noise model outside the dataset, thus reducing the denoising effect. Summary of the invention

[0006] The present disclosure provides a method and device for acquiring training sample pairs, an electronic device, and a storage medium to solve the above technical problems.

[0007] According to a first aspect of the present disclosure, a method for acquiring training sample pairs is provided, the method comprising:

[0008] Perform an inverse operation of the image signal processing pipeline on the image to be processed to obtain at least one frame of target RAW image; the target RAW image is used to represent the original image output by the image sensor of the electronic device;

[0009] Performing noise processing on each frame of the target RAW image to obtain an initial noise image corresponding to each frame of the target RAW image, wherein each frame of the target RAW image and the corresponding initial noise image constitute an initial image pair;

[0010] The initial image pair is input into a camera image processing model of an electronic device and processed into an RGB image to obtain a noise sample image and a clean sample image before noise reduction processing, wherein the noise sample image and the clean sample image constitute a training sample pair.

[0011] Optionally, performing an inverse operation of an image signal processing pipeline on the image to be processed to obtain at least one frame of target RAW image includes:

[0012] Performing a first inverse operation of an image signal processing pipeline on the image to be processed to obtain at least one frame of initial image;

[0013] Performing image enhancement processing on each frame of initial image to obtain at least one frame of intermediate image;

[0014] The second inverse operation of the image signal processing pipeline is performed on each frame of the intermediate image to obtain at least one frame of the target RAW image.

[0015] Optionally, performing a first inverse operation of an image signal processing pipeline on the image to be processed to obtain at least one frame of initial image includes:

[0016] Performing an inverse transformation of a gamma transformation on the image to be processed to obtain a first transformed image;

[0017] Performing an inverse transformation process of global tone mapping on the first transformed image to obtain a second transformed image.

[0018] Optionally, performing an inverse transformation of a gamma transformation on the image to be processed to obtain a first transformed image includes:

[0019] Acquire the power of pixel data of each pixel point of the image to be processed; the power value of the power and the power value of the gamma transform in the image signal processing pipeline are inverse numbers of each other;

[0020] The pixel data of each pixel point of the image to be processed is updated to a corresponding power to obtain the first transformed image.

[0021] Optionally, performing an inverse transformation process of global tone mapping on the first transformed image to obtain a second transformed image includes:

[0022] Obtaining the product of pixel data of each pixel point of the first transformed image and a first preset constant;

[0023] Obtaining a difference between a second preset constant and the product, and obtaining an arcsine value of the difference;

[0024] Obtaining a ratio of the arc sine value to a third preset constant, and obtaining a sine value of the ratio;

[0025] Obtaining a difference between a fourth preset constant and the sine value to obtain updated pixel data of each pixel point;

[0026] The pixel data of each pixel point of the first transformed image is updated to obtain the second transformed image.

[0027] Optionally, performing image enhancement processing on each frame of the initial image to obtain at least one frame of the intermediate image includes:

[0028] Affine transformation is performed on the second transformed image to obtain at least one frame of exposure image sequence.

[0029] Optionally, performing affine transformation processing on the second transformed image to obtain at least one frame of exposure image sequence includes:

[0030] The following operations are performed on the second transformed image:

[0031] Performing shearing transformation processing on the second transformed image using any random value within a preset shearing transformation range to obtain a shearing transformation image;

[0032] Performing rotation transformation processing on the shearing transformation image using any random value within a preset rotation transformation range to obtain a rotation transformation image;

[0033] Performing translation transformation processing on the rotation transformation image using any random value within a preset translation transformation range to obtain a frame of exposure image;

[0034] Repeat the above steps at least once to obtain at least one frame of exposure image sequence.

[0035] Optionally, performing a second inverse operation of the image signal processing pipeline on each frame of the intermediate image to obtain at least one frame of the target RAW image includes:

[0036] Performing a mosaic operation on the at least one frame of exposure image sequence to obtain at least one frame of initial RAW image;

[0037] Performing exposure enhancement processing on each frame of the initial RAW image to obtain a first intermediate RAW image;

[0038] Performing an inverse operation of lens distortion correction on each frame of the first intermediate RAW image to obtain a second intermediate RAW image;

[0039] Performing an inverse operation of the white balance operation on each frame of the second intermediate RAW image to obtain a bracketed exposure RAW image as a third intermediate RAW image;

[0040] The pixel data of each pixel point of each frame of the third intermediate RAW image is increased by a preset black level value to obtain a target RAW image without removing the black level.

[0041] Optionally, performing exposure enhancement processing on each frame of the initial RAW image to obtain a first intermediate RAW image includes:

[0042] Get the preset Bayer array format;

[0043] Acquire the target color channel value required by the Bayer array format from the pixel data of each pixel unit of each exposure image frame in at least one exposure image sequence;

[0044] A Bayer pixel value corresponding to each pixel unit is generated according to the target color channel value to obtain a first intermediate RAW image.

[0045] Optionally, performing an inverse operation of lens distortion correction on each frame of the first intermediate RAW image to obtain a second intermediate RAW image includes:

[0046] Performing linear interpolation processing on a preset initial attenuation coefficient lookup table to obtain a target attenuation coefficient lookup table having the same size as the image to be processed;

[0047] The product of each pixel point of the first intermediate RAW image of each frame and the attenuation coefficient at the same position in the target attenuation coefficient lookup table is obtained to obtain a second intermediate RAW image corresponding to the first intermediate RAW image of each frame; the target RAW image has a lens distortion feature compared with the initial RAW image.

[0048] Optionally, performing an inverse operation of a white balance operation on each frame of the second intermediate RAW image to obtain a bracketed exposure RAW image as a third intermediate RAW image includes:

[0049] Generate a white balance gain coefficient according to the white balance coefficient data distribution, and obtain a red pixel value gain coefficient and a blue pixel value gain coefficient;

[0050] Dividing the red sub-pixel data of each pixel point of the second intermediate RAW image of each frame by the red pixel value gain coefficient, and dividing the blue sub-pixel data by the blue pixel value gain coefficient, to obtain updated pixel data of each pixel point;

[0051] The pixel data of each pixel point of each frame of the second intermediate RAW image is updated to obtain a bracketed exposure image as the third intermediate RAW image.

[0052] Optionally, performing noise processing on each frame of the target RAW image to obtain an initial noise image corresponding to each frame of the target RAW image includes:

[0053] Get the exposure gain of each frame of the target RAW image;

[0054] Obtaining a noise curve corresponding to the exposure gain;

[0055] Obtaining the slope and intercept of the noise curve;

[0056] Generate pixel data with added noise according to the slope, the intercept and the pixel data of each pixel point in the target RAW image of each frame;

[0057] The pixel data of each frame of the target RAW image is updated to obtain an initial noise image.

[0058] According to a second aspect of the present disclosure, a training sample pair acquisition device is provided, the device comprising:

[0059] A RAW image acquisition module, used to perform an inverse operation of the image signal processing pipeline on the image to be processed to obtain at least one frame of target RAW image; the target RAW image is used to represent the original image output by the image sensor of the electronic device;

[0060] An image pair acquisition module is used to perform noise processing on each frame of the target RAW image to obtain an initial noise image corresponding to each frame of the target RAW image, wherein each frame of the target RAW image and the corresponding initial noise image constitute an initial image pair;

[0061] The sample pair acquisition module is used to input the initial image pair into the camera image processing model of the electronic device to process it into an RGB image, obtain a noisy sample image and a clean sample image before noise reduction processing, and the noisy sample image and the clean sample image constitute a training sample pair.

[0062] Optionally, the RAW image acquisition module includes:

[0063] An initial image acquisition submodule, used for performing a first inverse operation of an image signal processing pipeline on the image to be processed to obtain at least one frame of initial image;

[0064] The intermediate image acquisition submodule is used to perform image enhancement processing on each frame of the initial image to obtain at least one frame of the intermediate image;

[0065] The RAW image acquisition submodule is used to perform the second inverse operation of the image signal processing pipeline on each frame of the intermediate image to obtain at least one frame of target RAW image.

[0066] Optionally, the initial image acquisition submodule includes:

[0067] A first image acquisition unit, configured to perform an inverse transformation of a gamma transformation on the image to be processed to obtain a first transformed image;

[0068] The second image acquisition unit is used to perform inverse transformation processing of global tone mapping on the first transformed image to obtain a second transformed image.

[0069] Optionally, the first image acquisition unit includes:

[0070] A power acquisition subunit, used for acquiring the power of pixel data of each pixel point of the image to be processed; the power value of the power is the inverse of the power value of the gamma transform in the image signal processing pipeline;

[0071] The first image acquisition unit is used to update the pixel data of each pixel point of the image to be processed to a corresponding power to obtain the first transformed image.

[0072] Optionally, the second image acquisition unit includes:

[0073] A product acquisition subunit, used for acquiring the product of the pixel data of each pixel point of the first transformed image and a first preset constant;

[0074] An arcsine value acquisition subunit, used to acquire a difference between a second preset constant and the product, and acquire an arcsine value of the difference;

[0075] a sine value acquisition subunit, used to acquire a ratio of the arc sine value to a third preset constant, and acquire a sine value of the ratio;

[0076] A pixel data acquisition subunit, used to acquire a difference between a fourth preset constant and the sine value, and obtain updated pixel data of each pixel point;

[0077] The second image acquisition subunit is used to update the pixel data of each pixel point of the first transformed image to obtain the second transformed image.

[0078] Optionally, the intermediate image acquisition submodule includes:

[0079] The intermediate image acquisition unit is used to perform affine transformation processing on the second transformed image to obtain at least one frame of exposure image sequence as an intermediate image.

[0080] Optionally, the intermediate image acquisition unit includes the following subunits, configured to perform the following operations on the second transformed image:

[0081] a shearing transformation image acquisition subunit, configured to perform shearing transformation processing on the second transformation image using any random value within a preset shearing transformation range to obtain a shearing transformation image;

[0082] A rotation transformation image acquisition subunit, used for performing a rotation transformation process on the shear transformation image using any random value within a preset rotation transformation range to obtain a rotation transformation image;

[0083] An exposure image acquisition subunit is used to perform translation transformation processing on the rotation transformation image using any random value within a preset translation transformation range to obtain a frame of exposure image;

[0084] The repeat determination subunit is used to repeatedly execute the above steps at least once to obtain at least one frame of exposure image sequence.

[0085] Optionally, the RAW image acquisition submodule includes:

[0086] An initial RAW image acquisition unit, configured to perform a mosaic operation on the at least one frame of the exposure image sequence to obtain at least one frame of the initial RAW image;

[0087] A first RAW image acquisition unit is used to perform exposure enhancement processing on each frame of the initial RAW image to obtain a first intermediate RAW image;

[0088] A second RAW image acquisition unit, configured to perform an inverse operation of lens distortion correction on each frame of the first intermediate RAW image to obtain a second intermediate RAW image;

[0089] a third RAW image acquisition unit, configured to perform an inverse operation of the white balance operation on each frame of the second intermediate RAW image to obtain a bracketed exposure RAW image as a third intermediate RAW image;

[0090] The target RAW image acquisition unit is used to increase the pixel data of each pixel point of each frame of the third intermediate RAW image by a preset black level value to obtain a target RAW image without removing the black level.

[0091] Optionally, the first RAW image acquisition unit includes:

[0092] A Bayer format acquisition subunit, used for acquiring a preset Bayer array format;

[0093] A target color acquisition subunit, used to acquire the target color channel value required by the Bayer array format from the pixel data of each pixel unit of each exposure image frame in at least one exposure image sequence;

[0094] The first RAW image acquisition subunit is used to generate a Bayer pixel value corresponding to each pixel unit according to the target color channel value to obtain a first intermediate RAW image.

[0095] Optionally, the second RAW image acquisition unit includes:

[0096] A lookup table acquisition subunit is used to perform linear interpolation processing on a preset initial attenuation coefficient lookup table to obtain a target attenuation coefficient lookup table having the same size as the image to be processed;

[0097] The second RAW image acquisition unit is used to obtain the product of each pixel point of the first intermediate RAW image of each frame and the attenuation coefficient at the same position in the target attenuation coefficient lookup table to obtain the second intermediate RAW image corresponding to the first intermediate RAW image of each frame; the target RAW image has lens distortion characteristics compared with the initial RAW image.

[0098] Optionally, the third RAW image acquisition unit includes:

[0099] The gain coefficient acquisition subunit is used to generate a white balance gain coefficient according to the white balance coefficient data distribution, and obtain a red pixel value gain coefficient and a blue pixel value gain coefficient;

[0100] a pixel data acquisition subunit, configured to divide the red sub-pixel data of each pixel point of the second intermediate RAW image of each frame by the red pixel value gain coefficient, and divide the blue sub-pixel data by the blue pixel value gain coefficient, to obtain updated pixel data of each pixel point;

[0101] The third RAW image acquisition subunit is used to update the pixel data of each pixel point of each frame of the second intermediate RAW image to obtain a bracketed exposure image as the third intermediate RAW image.

[0102] Optionally, the image pair acquisition module includes:

[0103] The exposure gain acquisition submodule is used to obtain the exposure gain of each frame of the target RAW image;

[0104] A noise curve acquisition submodule, used to acquire a noise curve corresponding to the exposure gain;

[0105] A slope-intercept acquisition submodule, used to acquire the slope and intercept of the noise curve;

[0106] A pixel data acquisition submodule, used for generating pixel data with added noise according to the slope, the intercept and the pixel data of each pixel point in the target RAW image of each frame;

[0107] The noise image acquisition submodule is used to update the pixel data of each frame of the target RAW image to obtain an initial noise image.

[0108] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0109] Processor and memory;

[0110] The memory is used to store a computer program executable by the processor;

[0111] The processor is configured to execute the computer program in the memory to implement the method as described in any one of the first aspects.

[0112] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, and when an executable computer program in the storage medium is executed by a processor, the method described in any one of the first aspects can be implemented.

[0113] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0114] The scheme of this embodiment can perform an inverse operation of the image signal processing pipeline on the image to be processed to obtain at least one frame of target RAW image; the target RAW image is used to represent the original image output by the image sensor of the electronic device; then, each frame of the target RAW image is subjected to noise processing to obtain an initial noise image corresponding to each frame of the target RAW image, and each frame of the target RAW image and the corresponding initial noise image constitute an initial image pair; then, the initial image pair is input into the camera image processing model of the electronic device to be processed into an RGB image, and a noise sample image and a clean sample image before noise reduction processing are obtained, and the noise sample image and the clean sample image constitute a training sample pair. In this way, in this embodiment, through pipeline inverse operation processing and camera image processing model processing, the training sample pair can be matched with the noise model of the image actually captured by the camera, which is conducive to improving the accuracy of the neural network trained by the training sample pair, thereby improving the image noise reduction effect.

[0115] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0116] Figure 1 The present invention is a flowchart of a method for obtaining training sample pairs according to an embodiment of the present invention.

[0117] Figure 2 The present invention is a flowchart of obtaining a target RAW image according to an embodiment of the present invention.

[0118] Figure 3 The present invention is a flowchart of obtaining an exposure image sequence according to an embodiment of the present invention.

[0119] Figure 4 The present invention is a flowchart of obtaining an initial RAW image according to an embodiment of the present invention.

[0120] Figure 5 The present invention is a flowchart of obtaining a target RAW image according to an embodiment of the present invention.

[0121] Figure 6The present invention is a flowchart of obtaining an initial image pair according to an embodiment of the present invention.

[0122] Figure 7 The present invention is a block diagram of a training sample pair acquisition device according to an embodiment of the present invention.

[0123] Figure 8 The present invention is a block diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0124] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices consistent with some aspects of the present disclosure as detailed in the appended claims.

[0125] The embodiments of the present disclosure provide a method and device for acquiring training sample pairs, an electronic device, and a storage medium. The above training sample pair acquisition method is applicable to electronic devices. The electronic devices include but are not limited to servers, server clusters, smart phones, tablet computers, and other devices, which can be set according to specific scenarios and are not limited here.

[0126] The present disclosure also provides a method for obtaining training sample pairs. Figure 1 The method includes steps 11 to 13.

[0127] In step 11, the image to be processed is subjected to an inverse operation of the image signal processing pipeline to obtain at least one frame of target RAW image; the target RAW image is used to represent the original image output by the image sensor of the electronic device.

[0128] Considering that the sources of the images to be processed are different, such as different public data sets, to ensure that each image has the same size and sufficient texture data, the electronic device can resize the original image in this step, such as downsampling, downsampling each frame of the original image to obtain an image to be processed of a preset size. The preset size can be selected according to the specific scenario, such as 500 pixels * 500 pixels, which is not limited here.

[0129] In this step, the electronic device can perform the inverse operation of the image signal processing pipeline on the image to be processed to obtain at least one frame of target RAW image. Figure 2 , including steps 21 to 23.

[0130] In step 21, the electronic device may perform a first inverse operation of the image signal processing pipeline on the image to be processed to obtain at least one frame of initial image.

[0131] In this step, the Image Signal Process Pipeline (ISP Pipeline) may include but is not limited to at least one or more functions of black level compensation, lens shading correction, bad pixel correction, color interpolation (demosaic), white balance (AWB) correction, color correction, gamma correction, color space conversion (such as RGB to YUV conversion), color noise removal and edge enhancement, color and contrast enhancement, automatic exposure control, etc. The corresponding function can be selected according to the specific scenario, and the execution order after selecting at least one function can also be adjusted, and the corresponding scheme falls within the protection scope of the present disclosure.

[0132] In this step, different types of electronic devices may use different image signal processing pipelines (ISP Pipelines). Therefore, when determining the type of electronic device to which the training sample is applicable, the type of the IS Pipeline of the electronic device can be determined, thereby obtaining the setting parameters and the value range of the setting parameters in the ISP Pipeline. In other words, before training the neural network model, the setting parameters and the value range of the setting parameters in the ISP Pipeline are known.

[0133] In this step, after determining the processing operation of the image signal processing pipeline, the electronic device performs the first inverse operation of the image signal processing pipeline on the image to be processed to obtain at least one frame of initial image. Figure 3 , including steps 31 to 32.

[0134] In step 31, the electronic device may perform an inverse transformation of a gamma transformation on the image to be processed to obtain a first transformed image.

[0135] In this step, the electronic device can obtain the power of the pixel data of each pixel point of the image to be processed; the power value of the above power and the power value of the gamma transform in the image signal processing pipeline are reciprocal numbers; then, the electronic device can update the pixel data of each pixel point of the image to be processed to the corresponding power to obtain the first transformed image.

[0136] Assume that the gamma transform formula of the image signal processing pipeline is as shown in formula (1):

[0137] y1=x1 a ; (1)

[0138] In formula (1), y1 represents pixel data after gamma transformation; x1 represents pixel data before gamma transformation; a represents power, which can be selected according to the specific scene. In one example, a takes a value of -2.2.

[0139] In this step, the electronic device can perform inverse transformation processing on the above gamma transformation as shown in formula (2):

[0140] y2=x2 -a ; (2)

[0141] In formula (2), y2 represents the pixel data after the inverse transformation of the gamma transformation, that is, the pixel data x1 before the gamma transformation in formula (1); x2 represents the pixel data before the inverse transformation of the gamma transformation, that is, the pixel data y1 after the gamma transformation in formula (1); -a represents power. When a = -2.2, formula (2) can be converted into formula (3):

[0142] y2=x2 2.2 (3)

[0143] In this way, the electronic device can obtain the first transformed image by replacing the pixel data of each pixel point of the image to be processed with the pixel data after the inverse transformation of the gamma transformation.

[0144] In step 32, the electronic device may perform an inverse transformation process of global tone mapping on the first transformed image to obtain a second transformed image.

[0145] In this step, the electronic device can obtain the product of the pixel data of each pixel point of the first transformed image and a first preset constant; then, obtain the difference between the second preset constant and the product, and obtain the arc sine value of the difference; then, obtain the ratio of the arc sine value to a third preset constant, and obtain the sine value of the ratio; further, obtain the difference between the fourth preset constant and the sine value to obtain the updated pixel data of each pixel point; finally, update the pixel data of each pixel point of the first transformed image to obtain the second transformed image.

[0146] Based on the above, it is assumed that the inverse transform processing formula of global tone mapping is as shown in formula (4):

[0147]

[0148] In formula (4), x3 represents the pixel data before the inverse transformation of the first global tone mapping, that is, the pixel data of the first transformation image; y3 represents the pixel data after the inverse transformation of the first global tone mapping, that is, the pixel data of the second transformation image.

[0149] It should be noted that formula (4) describes a possible example, which can be derived according to the formula of global tone mapping in the ISP Pipeline.

[0150] In step 22, the electronic device may perform image enhancement processing on each frame of initial image to obtain at least one frame of intermediate image.

[0151] In this step, the electronic device may perform affine transformation processing on the second transformed image to obtain at least one frame of exposure image sequence. Affine transformation processing may include but is not limited to shear transformation, rotation transformation and translation transformation, etc., and may be selected according to the specific scene. Taking the affine transformation processing including shear transformation, rotation transformation and translation transformation as an example, the electronic device performs affine transformation processing on the second transformed image to obtain at least one frame of exposure image sequence, including:

[0152] The electronic device may perform the following operations on the second transformed image:

[0153] Performing shearing transformation processing on the second transformed image using any random value within a preset shearing transformation range to obtain a shearing transformation image;

[0154] Performing rotation transformation processing on the shearing transformation image using any random value within a preset rotation transformation range to obtain a rotation transformation image;

[0155] Performing translation transformation processing on the rotation transformation image using any random value within a preset translation transformation range to obtain a frame of exposure image;

[0156] Repeat the above steps at least once to obtain at least one frame of exposure image sequence.

[0157] It should be noted that the preset shearing transformation range is the distribution range formed by the combination of shearing values ​​used in different scenes (such as strong light scenes, dark light scenes, sun scenes, shade scenes, night scenes, etc.) when the camera takes images; alternatively, a part of the scenes is selected to determine the shearing values, and then these shearing values ​​are interpolated and fitted to obtain the above-mentioned preset shearing transformation range.

[0158] It should be noted that the preset rotation transformation range is the distribution range formed by the combination of rotation values ​​used in different scenes (such as strong light scenes, dark light scenes, sun scenes, shade scenes, night scenes, etc.) when the camera takes images; alternatively, a part of the scenes are selected to determine the rotation values, and then these rotation values ​​are interpolated and fitted to obtain the above-mentioned preset rotation transformation range.

[0159] It should be noted that the preset translation transformation range is the distribution range formed by the combination of the translation amounts used in different scenes (such as strong light scenes, dark light scenes, sun scenes, shade scenes, night scenes, etc.) when the camera takes images; alternatively, a part of the scenes are selected to determine the translation amount, and then these translation amounts are interpolated and fitted to obtain the above-mentioned preset translation transformation range.

[0160] In step 23, the electronic device may perform the second inverse operation of the image signal processing pipeline on each frame of the intermediate image to obtain at least one frame of the target RAW image.

[0161] In this step, the electronic device can perform the second inverse operation of the image signal processing pipeline on each frame of the intermediate image to obtain at least one frame of the target RAW image. Figure 4 , including steps 41 to 45.

[0162] In step 41, the electronic device may perform a mosaic operation on the at least one frame of the exposure image sequence to obtain at least one frame of initial RAW image.

[0163] In this step, the electronic device can obtain a preset Bayer array format. In one example, the Bayer array format can include 2 green sub-pixels G, 1 green sub-pixel R, and 1 blue sub-pixel B. These 4 sub-pixels can be arbitrarily combined to obtain different Bayer array formats, such as RGGB, GBGR, etc. The technician can set the Bayer array format according to the specific scene. In this step, the electronic device can obtain the Bayer array format from the camera.

[0164] In this step, the electronic device can obtain the target color channel value required by the above-mentioned Bayer array format from the pixels of each pixel unit of each exposure image frame in at least one exposure image sequence, thereby discarding unnecessary pixels; then, generate the Bayer pixel value corresponding to each pixel unit according to the above-mentioned target color channel value to obtain a RAW image, which is subsequently referred to as the first intermediate RAW image for distinction.

[0165] In step 42, the electronic device may perform exposure enhancement processing on each frame of the initial RAW image to obtain a first intermediate RAW image.

[0166] In this step, the electronic device can perform image exposure enhancement processing on each frame of the initial RAW image to obtain a first intermediate RAW image. Figure 5 , including steps 51 to 52.

[0167] In step 51, the electronic device may suppress the dark area and enhance the bright area of ​​each frame of the initial RAW image to obtain a regional enhanced image. The suppression process refers to reducing the level value of the pixel data, and the enhancement process refers to increasing the level value of the pixel data. For example, the electronic device may count the pixel data of all pixels of each frame of the initial RAW image to obtain a dark area less than or equal to a first threshold and a bright area greater than or equal to a second threshold. The value of the first threshold may be a brightness variation range (0, 20%], and the value of the first threshold may be selected according to the specific scene. The value of the second threshold may be a brightness variation range [80%, 100%), and the value of the second threshold may be selected according to the specific scene. The brightness variation range is determined according to the number of bits of the pixel data. Taking 8-bit data as an example, the brightness variation range is 0 to 255.

[0168] Then, for each pixel point in the dark area and the bright area, the square value and the cubic value of the pixel data of each pixel point of each frame of the initial RAW image are obtained; then, the first product of the square value and the first preset value is obtained, and the second product of the cubic value and the second preset value is obtained; then, the difference between the first product and the second product is obtained as the updated pixel data; finally, the pixel data of each pixel point of the initial RAW image of each frame is updated to obtain the regional enhanced image. In one example, the electronic device can select an s-type mapping function to implement the above solution as shown in formula (5):

[0169] y4=3x4 2 -2x4 3 ; (5)

[0170] In formula (5), x4 represents the pixel data before processing, that is, the pixel data of the initial RAW image; y4 represents the pixel data after processing, that is, the pixel data of the region enhanced image.

[0171] In step 52, the electronic device may perform pixel enhancement processing on each pixel point of the regional enhanced image of each frame using any random value within a preset exposure gain range to obtain a first intermediate RAW image.

[0172] In this step, the electronic device can obtain a preset exposure gain range, which in one example is (0, 1.3); obtain any random value from the preset exposure gain range, and use the exposure gain random value to simulate the effect of underexposure or overexposure of the image, that is, the above exposure gain random value is randomly generated according to the corresponding data distribution of the underexposure image actually taken by the camera.

[0173] It should be noted that the exposure gain coefficient is smaller when under-exposed, and is larger when over-exposed. The corresponding exposure gain coefficient can be selected according to the exposure degree of the image.

[0174] In this step, the electronic device can multiply each pixel of each frame region enhanced image by the above-mentioned exposure gain random value to obtain updated pixel data; after the pixel data of each pixel is updated, the target RAW image can be obtained.

[0175] In step 43, the electronic device may perform an inverse operation of lens distortion correction on each frame of the first intermediate RAW image to obtain a second intermediate RAW image.

[0176] In this step, the electronic device may perform an inverse operation of lens distortion correction on each frame of the first intermediate RAW image. Figure 6 , including steps 61 to 62.

[0177] In step 61, the electronic device may perform linear interpolation processing on a preset initial attenuation coefficient lookup table to obtain a target attenuation coefficient lookup table having the same size as the image to be processed.

[0178] In this step, a preset initial attenuation coefficient lookup table is stored in the electronic device. The electronic device can perform linear interpolation processing on the initial attenuation coefficient lookup table to obtain a target attenuation coefficient lookup table. The size of the target attenuation coefficient lookup table is the same as the size of the image to be processed.

[0179] It should be noted that since the image to be processed is resized to increase the number of textures, the initial attenuation coefficient lookup table needs to be resized in this step to ensure that the lookup table has the same size as the image to be processed, so as to achieve the effect of adjusting each pixel.

[0180] In step 62, the electronic device can obtain the product of each pixel point of the initial RAW image of each frame and the attenuation coefficient at the same position in the target attenuation coefficient lookup table to obtain the RAW image corresponding to the initial RAW image of each frame, which is subsequently referred to as the second intermediate RAW image for distinction; the second intermediate RAW image has lens distortion characteristics compared with the first intermediate RAW image.

[0181] In this example, lens distortion processing is performed on the first intermediate RAW image so that the second intermediate RAW image has lens distortion characteristics, thereby conforming to the camera shooting scene.

[0182] In step 44 , the electronic device may perform an inverse operation of the white balance operation on each frame of the second intermediate RAW image to obtain a bracketed exposure RAW image as the third intermediate RAW image.

[0183] In this step, the electronic device may perform an inverse operation of the white balance operation on each frame of the second intermediate RAW image to obtain at least one third intermediate RAW image, including: the electronic device may generate a white balance gain coefficient according to the white balance coefficient data distribution to obtain a red pixel value gain coefficient and a blue pixel value gain coefficient. Then, the electronic device may divide the red sub-pixel data of each pixel point of each frame of the second intermediate RAW image by the red pixel value gain coefficient, and divide the blue sub-pixel data by the blue pixel value gain coefficient to obtain updated pixel data of each pixel point. After that, after updating the pixel data of each pixel point of the second intermediate RAW image, the third intermediate RAW image is obtained.

[0184] In step 45 , the electronic device may increase the pixel data of each pixel point of each frame of the third intermediate RAW image by a preset black level value to obtain a target RAW image without removing the black level.

[0185] In this step, the electronic device may increase the pixel data of each pixel point of each frame of the third intermediate RAW image by a preset black level value to obtain a target RAW image without removing the black level.

[0186] It should be noted that in the scheme of this embodiment, at least one of the exposure enhancement processing, the inverse operation processing of the lens distortion correction, the inverse operation of the white balance operation and the increase of the preset black level value can be selected according to the specific scenario. The above scheme is only an example and does not constitute a limitation of this scheme.

[0187] In step 12, noise processing is performed on each frame of the target RAW image to obtain an initial noise image corresponding to each frame of the target RAW image. The target RAW image of each frame and the corresponding initial noise image constitute an initial image pair.

[0188] In this step, the electronic device can perform noise processing on each frame of the target RAW image to obtain an initial noise image corresponding to each frame of the target RAW image. Figure 7 , including steps 71 to 75.

[0189] In step 71, the electronic device may obtain the exposure gain of each frame of the target RAW image. Figure 5 The content of the example solution will not be repeated here.

[0190] In step 72, the electronic device may obtain a noise curve corresponding to the exposure gain. The electronic device stores a preset noise curve, and each exposure gain corresponds to a different noise curve. After obtaining the exposure gain of the target RAW image, the corresponding noise curve may be searched from the storage location according to the exposure gain.

[0191] In step 73, the electronic device may obtain the slope and intercept of the noise curve. In one example, the slope of the noise curve is the mean of Poisson noise, and the intercept of the noise curve is the variance of Gaussian noise.

[0192] In step 74, the electronic device may generate pixel data with added noise according to the slope, the intercept and the pixel data of each pixel point in each frame of the target RAW image, as shown in formula (6):

[0193]

[0194] In formula (6), x5 represents the pixel data before adding noise; y5 represents the pixel data after adding noise; poisson represents the function of generating Poisson distribution random numbers, and the coefficient in the brackets is the mean of the Poisson distribution; gaussian represents the function of generating Gaussian distribution random numbers, and the coefficient in the brackets is the variance of the Gaussian distribution.

[0195] In step 75, the electronic device may update the pixel data of each frame of the target RAW image to obtain an initial noise image.

[0196] The electronic device may determine that each frame of the target RAW image and the corresponding initial noise image constitute the initial image pair.

[0197] In step 13, the initial image pair is input into a camera image processing model of an electronic device to be processed into an RGB image, and a noise sample image and a clean sample image before noise reduction processing are obtained, wherein the noise sample image and the clean sample image constitute a training sample pair.

[0198] In this step, the electronic device can input the initial image pair into the camera image processing model of the electronic device, and export data before the noise reduction module of the neural network of the camera image processing model, respectively obtaining a corrected image pair including a noise sample image and a clean sample image, and using the sample image pair as a training sample pair. Among them, the initial noise image in the initial image pair is processed by the camera image processing model and exported before the noise reduction module as the noise sample image, and the target RAW image is processed by the camera image processing model and exported before the noise reduction module as the clean sample image.

[0199] It can be understood that in this step, the initial image pair is input into the camera image processing model and the data is exported before the noise reduction module, so that the noisy image used for training undergoes a series of nonlinear operations at the front end of the noise reduction module, ensuring that the noise model of the training neural network matches the noise model of the actual captured image.

[0200] In one embodiment, the electronic device may repeatedly perform steps 11 to 13 for several times, thereby obtaining several training sample pairs, that is, obtaining a set of training sample pairs, to facilitate neural network training.

[0201] So far, the training samples in this embodiment are obtained through the inverse operation processing of the image signal processing pipeline, noise addition processing and input into the camera image processing model. Through the pipeline inverse operation processing and the camera image processing model processing, the training sample pairs can be matched with the noise model of the actual image taken by the camera, which is beneficial to improve the accuracy of the training sample pairs for the trained neural network, thereby improving the image denoising effect.

[0202] In this embodiment, the image to be processed can be an image in a public data set. Through the scheme disclosed in this invention, a training sample pair can be constructed efficiently, conveniently and accurately to generate data that conforms to the noise model of the actual captured image. Moreover, in this embodiment, it can be ensured that the noise model of the data set used to train the denoising neural network matches the noise model of the actual camera shooting, so that it can be applied to the camera image processing model that requires neural network denoising to achieve a better denoising effect.

[0203] The present disclosure provides a method for acquiring a training sample pair. The present disclosure also provides a device for acquiring a training sample pair, the device comprising:

[0204] A RAW image acquisition module, used to perform an inverse operation of the image signal processing pipeline on the image to be processed to obtain at least one frame of target RAW image; the target RAW image is used to represent the original image output by the image sensor of the electronic device;

[0205] An image pair acquisition module is used to perform noise processing on each frame of the target RAW image to obtain an initial noise image corresponding to each frame of the target RAW image, wherein each frame of the target RAW image and the corresponding initial noise image constitute an initial image pair;

[0206] The sample pair acquisition module is used to input the initial image pair into the camera image processing model of the electronic device to process it into an RGB image, obtain a noisy sample image and a clean sample image before noise reduction processing, and the noisy sample image and the clean sample image constitute a training sample pair.

[0207] Optionally, the RAW image acquisition module includes:

[0208] An initial image acquisition submodule, used for performing a first inverse operation of an image signal processing pipeline on the image to be processed to obtain at least one frame of initial image;

[0209] The intermediate image acquisition submodule is used to perform image enhancement processing on each frame of the initial image to obtain at least one frame of the intermediate image;

[0210] The RAW image acquisition submodule is used to perform the second inverse operation of the image signal processing pipeline on each frame of the intermediate image to obtain at least one frame of target RAW image.

[0211] Optionally, the initial image acquisition submodule includes:

[0212] A first image acquisition unit, configured to perform an inverse transformation of a gamma transformation on the image to be processed to obtain a first transformed image;

[0213] The second image acquisition unit is used to perform inverse transformation processing of global tone mapping on the first transformed image to obtain a second transformed image.

[0214] Optionally, the first image acquisition unit includes:

[0215] A power acquisition subunit, used for acquiring the power of pixel data of each pixel point of the image to be processed; the power value of the power is the inverse of the power value of the gamma transform in the image signal processing pipeline;

[0216] The first image acquisition unit is used to update the pixel data of each pixel point of the image to be processed to a corresponding power to obtain the first transformed image.

[0217] Optionally, the second image acquisition unit includes:

[0218] A product acquisition subunit, used for acquiring the product of the pixel data of each pixel point of the first transformed image and a first preset constant;

[0219] An arcsine value acquisition subunit, used to acquire a difference between a second preset constant and the product, and acquire an arcsine value of the difference;

[0220] a sine value acquisition subunit, used to acquire a ratio of the arc sine value to a third preset constant, and acquire a sine value of the ratio;

[0221] A pixel data acquisition subunit, used to acquire a difference between a fourth preset constant and the sine value, and obtain updated pixel data of each pixel point;

[0222] The second image acquisition subunit is used to update the pixel data of each pixel point of the first transformed image to obtain the second transformed image.

[0223] Optionally, the intermediate image acquisition submodule includes:

[0224] The intermediate image acquisition unit is used to perform affine transformation processing on the second transformed image to obtain at least one frame of exposure image sequence as an intermediate image.

[0225] Optionally, the intermediate image acquisition unit includes the following subunits, configured to perform the following operations on the second transformed image:

[0226] a shearing transformation image acquisition subunit, configured to perform shearing transformation processing on the second transformation image using any random value within a preset shearing transformation range to obtain a shearing transformation image;

[0227] A rotation transformation image acquisition subunit, used for performing a rotation transformation process on the shear transformation image using any random value within a preset rotation transformation range to obtain a rotation transformation image;

[0228] An exposure image acquisition subunit is used to perform translation transformation processing on the rotation transformation image using any random value within a preset translation transformation range to obtain a frame of exposure image;

[0229] The repeat determination subunit is used to repeatedly execute the above steps at least once to obtain at least one frame of exposure image sequence.

[0230] Optionally, the RAW image acquisition submodule includes:

[0231] An initial RAW image acquisition unit, configured to perform a mosaic operation on the at least one frame of the exposure image sequence to obtain at least one frame of the initial RAW image;

[0232] A first RAW image acquisition unit is used to perform exposure enhancement processing on each frame of the initial RAW image to obtain a first intermediate RAW image;

[0233] A second RAW image acquisition unit, configured to perform an inverse operation of lens distortion correction on each frame of the first intermediate RAW image to obtain a second intermediate RAW image;

[0234] a third RAW image acquisition unit, configured to perform an inverse operation of the white balance operation on each frame of the second intermediate RAW image to obtain a bracketed exposure RAW image as a third intermediate RAW image;

[0235] The target RAW image acquisition unit is used to increase the pixel data of each pixel point of each frame of the third intermediate RAW image by a preset black level value to obtain a target RAW image without removing the black level.

[0236] Optionally, the first RAW image acquisition unit includes:

[0237] A Bayer format acquisition subunit, used for acquiring a preset Bayer array format;

[0238] A target color acquisition subunit, used to acquire the target color channel value required by the Bayer array format from the pixel data of each pixel unit of each exposure image frame in at least one exposure image sequence;

[0239] The first RAW image acquisition subunit is used to generate a Bayer pixel value corresponding to each pixel unit according to the target color channel value to obtain a first intermediate RAW image.

[0240] Optionally, the second RAW image acquisition unit includes:

[0241] A lookup table acquisition subunit is used to perform linear interpolation processing on a preset initial attenuation coefficient lookup table to obtain a target attenuation coefficient lookup table having the same size as the image to be processed;

[0242] The second RAW image acquisition unit is used to obtain the product of each pixel point of the first intermediate RAW image of each frame and the attenuation coefficient at the same position in the target attenuation coefficient lookup table to obtain the second intermediate RAW image corresponding to the first intermediate RAW image of each frame; the target RAW image has lens distortion characteristics compared with the initial RAW image.

[0243] Optionally, the third RAW image acquisition unit includes:

[0244] The gain coefficient acquisition subunit is used to generate a white balance gain coefficient according to the white balance coefficient data distribution, and obtain a red pixel value gain coefficient and a blue pixel value gain coefficient;

[0245] a pixel data acquisition subunit, configured to divide the red sub-pixel data of each pixel point of the second intermediate RAW image of each frame by the red pixel value gain coefficient, and divide the blue sub-pixel data by the blue pixel value gain coefficient, to obtain updated pixel data of each pixel point;

[0246] The third RAW image acquisition subunit is used to update the pixel data of each pixel point of each frame of the second intermediate RAW image to obtain a bracketed exposure image as the third intermediate RAW image.

[0247] Optionally, the image pair acquisition module includes:

[0248] The exposure gain acquisition submodule is used to obtain the exposure gain of each frame of the target RAW image;

[0249] A noise curve acquisition submodule, used to acquire a noise curve corresponding to the exposure gain;

[0250] A slope-intercept acquisition submodule, used to acquire the slope and intercept of the noise curve;

[0251] A pixel data acquisition submodule, used for generating pixel data with added noise according to the slope, the intercept and the pixel data of each pixel point in the target RAW image of each frame;

[0252] The noise image acquisition submodule is used to update the pixel data of each frame of the target RAW image to obtain an initial noise image.

[0253] It should be noted that the device embodiment shown in this embodiment matches the content of the above method embodiment, and the content of the above method embodiment can be referred to, which will not be repeated here.

[0254] Figure 8 8 is a block diagram of an electronic device according to an exemplary embodiment. For example, the electronic device 800 may be a smart phone, a computer, a digital broadcast terminal, a tablet device, a medical device, a fitness device, a personal digital assistant, a server or a server cluster, etc.

[0255] Reference Figure 8 The electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , a communication component 816 , and an image acquisition component 818 .

[0256] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute computer programs. In addition, the processing component 802 may include one or more modules to facilitate interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802. In an example, the processor 820 may implement the scheme of the above-mentioned training sample pair acquisition method.

[0257] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include computer programs for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0258] The power supply component 806 provides power to various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800. The power supply component 806 may include a power supply chip, and the controller may communicate with the power supply chip to control the power supply chip to turn on or off the first switch device, so that the battery supplies power to the mainboard circuit or does not supply power.

[0259] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the target object. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input information from the target object. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0260] The audio component 810 is configured to output and / or input audio file information. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive external audio file information. The received audio file information can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio file information.

[0261] The I / O interface 812 provides an interface between the processing component 802 and a peripheral interface module, which may be a keyboard, a click wheel, a button, etc.

[0262] The sensor component 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor component 814 can detect the open / closed state of the electronic device 800, the relative positioning of components, such as the display screen and keypad of the electronic device 800, and the sensor component 814 can also detect the position change of the electronic device 800 or a component, the presence or absence of contact between the target object and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. In this example, the sensor component 814 can include a magnetic sensor, a gyroscope, and a magnetic field sensor, and can also include an inertial sensor, an image sensor, etc., wherein the magnetic field sensor includes at least one of the following: a Hall sensor, a thin film magnetoresistive sensor, and a magnetic liquid acceleration sensor.

[0263] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives broadcast information or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0264] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components.

[0265] In an exemplary embodiment, the present disclosure also provides a non-transitory computer-readable storage medium, and when an executable computer program in the storage medium is executed by a processor, the above-mentioned method can be implemented.

[0266] In an exemplary embodiment, a chip is also provided, the chip including a processor and an interface for reading a computer program through the interface to implement the above method. The chip may be a conventional CPU (central processing unit) chip, a GPU (graphics processing unit) chip, etc., or an acceleration chip dedicated to artificial intelligence technology, such as an AI (Artificial Intelligence) accelerator, etc.

[0267] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0268] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for obtaining training sample pairs, characterized in that: The method comprises: Perform an inverse operation of the image signal processing pipeline on the image to be processed to obtain at least one frame of target RAW image; the target RAW image is used to represent the original image output by the image sensor of the electronic device; Performing noise processing on each frame of the target RAW image to obtain an initial noise image corresponding to each frame of the target RAW image, wherein each frame of the target RAW image and the corresponding initial noise image constitute an initial image pair; The initial image pair is input into a camera image processing model of an electronic device and processed into an RGB image to obtain a noise sample image and a clean sample image before noise reduction processing, wherein the noise sample image and the clean sample image constitute a training sample pair.

2. The method according to claim 1, characterized in that Performing an inverse operation of the image signal processing pipeline on the image to be processed to obtain at least one frame of target RAW image, including: Performing a first inverse operation of an image signal processing pipeline on the image to be processed to obtain at least one frame of initial image; Performing image enhancement processing on each frame of initial image to obtain at least one frame of intermediate image; The second inverse operation of the image signal processing pipeline is performed on each frame of the intermediate image to obtain at least one frame of the target RAW image.

3. The method according to claim 2, characterized in that Performing a first inverse operation of the image signal processing pipeline on the image to be processed to obtain at least one frame of initial image, including: Performing an inverse transformation of a gamma transformation on the image to be processed to obtain a first transformed image; Performing an inverse transformation process of global tone mapping on the first transformed image to obtain a second transformed image.

4. The method according to claim 3, characterized in that Performing an inverse transformation of a gamma transformation on the image to be processed to obtain a first transformed image, comprising: Acquire the power of pixel data of each pixel point of the image to be processed; the power value of the power and the power value of the gamma transform in the image signal processing pipeline are inverse numbers of each other; The pixel data of each pixel point of the image to be processed is updated to a corresponding power to obtain the first transformed image.

5. The method according to claim 3, characterized in that: Performing an inverse transformation process of global tone mapping on the first transformed image to obtain a second transformed image includes: Obtaining the product of pixel data of each pixel point of the first transformed image and a first preset constant; Obtaining a difference between a second preset constant and the product, and obtaining an arcsine value of the difference; Obtaining a ratio of the arc sine value to a third preset constant, and obtaining a sine value of the ratio; Obtaining a difference between a fourth preset constant and the sine value to obtain updated pixel data of each pixel point; The pixel data of each pixel point of the first transformed image is updated to obtain the second transformed image.

6. The method according to claim 2, characterized in that Performing image enhancement processing on each frame of initial image to obtain at least one frame of intermediate image, including: Affine transformation is performed on the second transformed image to obtain at least one frame of exposure image sequence.

7. The method according to claim 6, characterized in that Performing affine transformation processing on the second transformed image to obtain at least one frame of exposure image sequence, including: The following operations are performed on the second transformed image: Performing shearing transformation processing on the second transformed image using any random value within a preset shearing transformation range to obtain a shearing transformation image; Performing rotation transformation processing on the shearing transformation image using any random value within a preset rotation transformation range to obtain a rotation transformation image; Performing translation transformation processing on the rotation transformation image using any random value within a preset translation transformation range to obtain a frame of exposure image; Repeat the above steps at least once to obtain at least one frame of exposure image sequence.

8. The method according to claim 2, characterized in that: Performing the second inverse operation of the image signal processing pipeline on each intermediate image frame to obtain at least one target RAW image frame, including: Performing a mosaic operation on the at least one frame of exposure image sequence to obtain at least one frame of initial RAW image; Performing exposure enhancement processing on each frame of the initial RAW image to obtain a first intermediate RAW image; Performing an inverse operation of lens distortion correction on each frame of the first intermediate RAW image to obtain a second intermediate RAW image; Performing an inverse operation of the white balance operation on each frame of the second intermediate RAW image to obtain a bracketed exposure RAW image as a third intermediate RAW image; The pixel data of each pixel point of each frame of the third intermediate RAW image is increased by a preset black level value to obtain a target RAW image without removing the black level.

9. The method according to claim 8, characterized in that Performing exposure enhancement processing on each frame of the initial RAW image to obtain a first intermediate RAW image, including: Get the preset Bayer array format; Acquire the target color channel value required by the Bayer array format from the pixel data of each pixel unit of each exposure image frame in at least one exposure image sequence; A Bayer pixel value corresponding to each pixel unit is generated according to the target color channel value to obtain a first intermediate RAW image.

10. The method according to claim 8, characterized in that Performing an inverse operation of lens distortion correction on each frame of the first intermediate RAW image to obtain a second intermediate RAW image, comprising: Performing linear interpolation processing on a preset initial attenuation coefficient lookup table to obtain a target attenuation coefficient lookup table having the same size as the image to be processed; The product of each pixel point of the first intermediate RAW image of each frame and the attenuation coefficient at the same position in the target attenuation coefficient lookup table is obtained to obtain a second intermediate RAW image corresponding to the first intermediate RAW image of each frame; the target RAW image has a lens distortion feature compared with the initial RAW image.

11. The method according to claim 8, characterized in that Performing an inverse operation of the white balance operation on each frame of the second intermediate RAW image to obtain a bracketed exposure RAW image as a third intermediate RAW image, comprising: Generate a white balance gain coefficient according to the white balance coefficient data distribution, and obtain a red pixel value gain coefficient and a blue pixel value gain coefficient; Dividing the red sub-pixel data of each pixel point of the second intermediate RAW image of each frame by the red pixel value gain coefficient, and dividing the blue sub-pixel data by the blue pixel value gain coefficient, to obtain updated pixel data of each pixel point; The pixel data of each pixel point of each frame of the second intermediate RAW image is updated to obtain a bracketed exposure image as the third intermediate RAW image.

12. The method according to claim 1, characterized in that Perform noise processing on each frame of the target RAW image to obtain the initial noise image corresponding to each frame of the target RAW image, including: Get the exposure gain of each frame of the target RAW image; Obtaining a noise curve corresponding to the exposure gain; Obtaining the slope and intercept of the noise curve; Generate pixel data with added noise according to the slope, the intercept and the pixel data of each pixel point in the target RAW image of each frame; The pixel data of each frame of the target RAW image is updated to obtain an initial noise image.

13. A training sample pair acquisition device, characterized in that: The device comprises: A RAW image acquisition module, used to perform an inverse operation of the image signal processing pipeline on the image to be processed to obtain at least one frame of target RAW image; the target RAW image is used to represent the original image output by the image sensor of the electronic device; An image pair acquisition module is used to perform noise processing on each frame of the target RAW image to obtain an initial noise image corresponding to each frame of the target RAW image, wherein each frame of the target RAW image and the corresponding initial noise image constitute an initial image pair; The sample pair acquisition module is used to input the initial image pair into the camera image processing model of the electronic device to process it into an RGB image, obtain a noisy sample image and a clean sample image before noise reduction processing, and the noisy sample image and the clean sample image constitute a training sample pair.

14. An electronic device, characterized in that: include: Processor and memory; The memory is used to store a computer program executable by the processor; The processor is configured to execute the computer program in the memory to implement the method according to any one of claims 1 to 12.

15. A non-transitory computer-readable storage medium, characterized in that: When the executable computer program in the storage medium is executed by a processor, the method according to any one of claims 1 to 12 can be implemented.

Images