An image processing method, device and storage medium

Abstract

The present disclosure provides an image processing method, device and storage medium, the method comprising: acquiring a plurality of original images, processing the plurality of original images to obtain a fusion frame, and then performing image processing on the fusion frame based on the pixel value of the fusion frame. The image processing method provided by the present disclosure can accurately reflect the color temperature of the final image, restore the color to the greatest extent, ensure the accuracy and precision of image processing, and ensure that the image is not distorted.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image processing method, apparatus and storage medium. Background Technology

[0002] Photos taken under different lighting conditions will have different effects (for example, white objects will appear reddish under low color temperature lighting). Therefore, white balance processing is usually required to restore the colors.

[0003] In related technologies, white balance processing mainly involves: generating statistical information based on the preview frame, then using an algorithm to perform AWB (Automatic White Balance) calculation on the statistical information to obtain the AWB result, and finally performing white balance processing on the final image based on the AWB result.

[0004] However, when shooting in complex lighting conditions (such as high dynamic range scenes containing both shade and sunlight), multi-frame fusion techniques are typically required to address issues like image noise and dynamic range. Since the fused image differs significantly from the preview image, the final fused frame also differs considerably from the preview frame. In this case, using white balance processing (i.e., applying white balance to the fused frame based on statistical information generated from the preview frame) will result in color distortion. Furthermore, in low-light environments, using white balance processing will fail because the preview frame lacks sufficient information, making it impossible to generate statistical information based on the preview frame and thus hindering white balance processing in low-light conditions. Summary of the Invention

[0005] This disclosure provides an image processing method, apparatus, and storage medium to at least solve the technical problem that white balance processing in related technologies is not suitable for extreme shooting scenarios.

[0006] The first aspect of this disclosure provides an image processing method, including:

[0007] Acquire multiple original images and process them to obtain a fused frame;

[0008] Image processing is performed on the fused frame based on its pixel values.

[0009] A second aspect of this disclosure provides an image processing apparatus, the apparatus comprising:

[0010] The acquisition module is used to acquire multiple frames of original images and process the multiple frames of original images to obtain fused frames;

[0011] The processing module is used to perform image processing on the fused frame based on the pixel values ​​of the fused frame.

[0012] According to a third aspect of this disclosure, a computer storage medium is provided that, when the computer-executable instructions are executed by a processor, enables the image processing method as described in the first aspect.

[0013] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0014] In summary, the image processing method, device, and storage medium proposed in this disclosure first acquire multiple frames of original images, then process these frames to obtain a fused frame. Subsequently, image processing is performed on the fused frame based on its pixel values. Since the pixel values ​​of the fused frame accurately reflect its image information, image processing based on these pixel values ​​accurately reflects the final color temperature of the image, maximizing color reproduction and ensuring the precision and accuracy of image processing. This prevents image distortion and solves the technical problem in related technologies where "when image processing is performed on the fused frame based on the statistical information of the preview frame, the color reproduction of the processed fused frame is distorted due to the significant difference between the preview frame and the fused frame."

[0015] Furthermore, since the fused frame is obtained by processing multiple original images, the effective information of the fused frame can be greatly improved. Based on this, even in extreme shooting scenarios (such as low-light environments), the effective information of the fused frame can still be ensured, thereby ensuring the accurate acquisition of the pixel values ​​of the fused frame in the future. This solves the technical problem in related technologies that "due to the loss of effective information in the preview frame in low-light environments, it is impossible to generate statistical information based on the preview frame, which in turn makes it impossible to perform image processing in low-light environments."

[0016] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure. Attached Figure Description

[0017] The above and / or additional aspects and advantages of this disclosure will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, in which:

[0018] Figure 1 This is a schematic flowchart of an image processing method provided according to an embodiment of the present disclosure;

[0019] Figure 2 This is a schematic diagram of the structure of an image processing apparatus provided according to an embodiment of the present disclosure;

[0020] Figure 3 This is a schematic flowchart of an image processing method provided according to an embodiment of the present disclosure. Detailed Implementation

[0021] Embodiments of this disclosure are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting this disclosure.

[0022] The image processing method and apparatus of this disclosure are described below with reference to the accompanying drawings.

[0023] Example 1

[0024] Figure 1 This is a schematic flowchart of an image processing method provided according to an embodiment of the present disclosure, executed by an image processing device, such as... Figure 1 As shown, the image processing method includes the following steps:

[0025] Step 101: Obtain multiple frames of original images and process them to obtain fused frames.

[0026] It should be noted that, in the embodiments of this disclosure, multiple frames of original images can be multiple raw images. Here, a raw image is the original data from which the image sensor converts the captured light source signal into a digital signal. Furthermore, the aforementioned multiple frames of raw images can be multiple raw images with different brightness exposures. By acquiring multiple raw images with different brightness exposures, the algorithm's need for effective image information in different scenarios can be met.

[0027] In some embodiments of this disclosure, the multiple original images may be obtained by an image processing device by sending a request to an image sensor.

[0028] Furthermore, in embodiments of this disclosure, the method for processing the original image to obtain the fused frame may include the following steps:

[0029] Step a: Perform RAW algorithm calculations on multiple frames of original images.

[0030] Step b: Perform multi-frame fusion and noise reduction processing on the multi-frame original images calculated by the algorithm.

[0031] In the embodiments of this disclosure, an AI (Artificial Intelligence) denoising model can be used to perform multi-frame fusion denoising processing on the calculated multi-frame original images. Furthermore, in the embodiments of this disclosure, when using the AI ​​denoising model for multi-frame fusion denoising processing, details in the multi-frame original images can be preserved as much as possible while suppressing noise.

[0032] Step c: Perform a fusion process on the multiple original frames after noise reduction to obtain a fused single frame.

[0033] In one embodiment of this disclosure, the fusion process may include fusing dark areas of a bright image with a brightness greater than a first threshold and bright areas of a dark image with a brightness less than a second threshold from multiple original images. For example, dark areas in a brighter image and bright areas in a darker image can be extracted from multiple original images and fused.

[0034] Step d: Brighten the merged single frame to obtain the merged frame.

[0035] Furthermore, in the embodiments of this disclosure, a fused frame with good noise, brightness, dynamic range, and resolution can be obtained through the above-described step ad.

[0036] It should be noted that in the embodiments of this disclosure, different original images will be acquired when the shooting scene is different, and the processing method of "processing the original image to obtain the fused frame" will also be different.

[0037] Specifically, when the shooting scene is a low-light scene (e.g., a night with poor lighting), the acquired raw images can be multiple high-exposure raw images. Furthermore, when obtaining a fused frame based on multiple high-exposure raw images acquired in a low-light scene, noise reduction and brightening processing can be prioritized to ensure that the final fused frame can retain details to the greatest extent and have sufficient effective information.

[0038] When shooting in a high dynamic range (HVR) scene (e.g., a scene containing both shade and sunlight), the acquired multiple frames of raw images can be multiple normally exposed RAW images. Furthermore, when obtaining a fused frame from multiple normally exposed RAW images acquired in a HVR scene, the focus can be on performing fusion processing, while ignoring or omitting noise reduction and brightening processing. Moreover, when performing fusion processing on multiple normally exposed RAW images, exposure bracketing can be used to fuse multiple frames of normally exposed RAW images to reduce the dynamic range of the image.

[0039] In addition, it should be noted that when the exposure levels of the multiple raw images acquired are different due to different shooting scenarios, the calculation method of the above-mentioned raw algorithm will also be different.

[0040] Step 102: Perform image processing on the fused frame based on the pixel values ​​of the fused frame.

[0041] In some embodiments of this disclosure, image processing may include white balance processing.

[0042] Furthermore, in embodiments of this disclosure, the method for image processing of a fused frame based on the pixel values ​​of the fused frame may include the following steps:

[0043] Step 1021: Calculate the first statistical information of the fused frame.

[0044] In the embodiments of this disclosure, the first statistical information can be used to reflect the degree of color deviation of the fused frame.

[0045] Furthermore, in embodiments of this disclosure, the method for calculating the first statistical information of the fused frame may include the following steps:

[0046] Step 1: Divide the fused frame into N×M regions, where N and M are both positive integers.

[0047] In the embodiments of this disclosure, each region includes at least one pixel.

[0048] For example, in an embodiment of this disclosure, the fused frame can be divided into 17×13 regions.

[0049] Step 2: Calculate the average pixel value of each pixel channel in each region, and determine the first statistical information for each region based on the average pixel value of each region.

[0050] In the embodiments of this disclosure, the pixel channels included in the fused frame can be: R channel, Gr channel, Gb channel, and B channel. Furthermore, in the embodiments of this disclosure, each region may include multiple pixels, each pixel can correspond to one pixel channel, and each pixel can correspond to one pixel value. Based on this, the average value corresponding to each pixel channel in the region can be calculated, wherein: the average value corresponding to the R channel in the region = the sum of pixel values ​​of all pixels corresponding to the R channel in the region ÷ the number of all pixels corresponding to the R channel in the region; the average value corresponding to the Gr channel in the region = the sum of pixel values ​​of all pixels corresponding to the Gr channel in the region ÷ the number of all pixels corresponding to the Gr channel in the region; the average value corresponding to the Gb channel in the region = the sum of pixel values ​​of all pixels corresponding to the Gb channel in the region ÷ the number of all pixels corresponding to the Gb channel in the region; the average value corresponding to the B channel in the region = the sum of pixel values ​​of all pixels corresponding to the B channel in the region ÷ the number of all pixels corresponding to the B channel in the region.

[0051] Furthermore, in the embodiments of this disclosure, the first statistical information for each region may include a first value and a second value, wherein the first value = the average value of the R channel for each region ÷ the average value of the Gr channel for each region; and the second value = the average value of the B channel for each region ÷ the average value of the Gb channel for each region.

[0052] Step 1022: Calculate the lens shading compensation coefficient based on the first statistical information and the image information of the fused frame using the LSC (Lens Shading Correction) algorithm, and perform shading compensation processing on the fused frame based on the lens shading compensation coefficient.

[0053] It should be noted that the above-mentioned "specific process for calculating the lens shadow compensation coefficient and shadow compensation processing" are all existing technical means. For a detailed introduction to this part, please refer to the existing technical introduction. This disclosure will not elaborate further here.

[0054] Step 1023: Calculate the second statistical information of the processed fused frame and output the second statistical information to calculate the AWB (Automatic White Balance) result based on the second statistical information.

[0055] In embodiments of this disclosure, the method for calculating the second statistical information of the processed fused frame may include the following steps:

[0056] Step 1: Divide the processed fused frame into N×M regions, where N and M are both positive integers.

[0057] In the embodiments of this disclosure, each region includes at least one pixel.

[0058] For example, in an embodiment of this disclosure, the processed fused frame can be divided into 17×13 regions.

[0059] Step 2: Calculate the average pixel value for each pixel channel in each region, and determine the second statistical information for each region based on the average pixel value for each region.

[0060] In the embodiments of this disclosure, the processed fused frame includes the following pixel channels: R channel, Gr channel, Gb channel, and B channel. For a detailed explanation of how to calculate the average pixel value corresponding to each pixel channel in each region, please refer to the calculation method in step 1021; this disclosure will not elaborate further here.

[0061] Furthermore, in the embodiments of this disclosure, the second statistical information corresponding to each region may include a first value and a second value, wherein the first value = the average value of the R channel of each region ÷ the average value of the Gr channel of each region; and the second value = the average value of the B channel of each region ÷ the average value of the Gb channel of each region.

[0062] Furthermore, in the embodiments of this disclosure, after obtaining the second statistical information of the fused frame after calculation and processing, the obtained second statistical information can be input into the AWB algorithm for calculation to obtain the AWB result that conforms to the fused frame. Then, the obtained AWB result can be used to perform white balance processing on the fused frame, thereby restoring the color of the fused frame.

[0063] In summary, the image processing method proposed in this disclosure first acquires multiple frames of original images and processes them to obtain a fused frame. Then, image processing is performed on the fused frame based on its pixel values. Since the pixel values ​​of the fused frame accurately reflect its image information, image processing based on these values ​​accurately reflects the color temperature of the final image, maximizing color reproduction and ensuring the precision and accuracy of image processing. This prevents image distortion and solves the technical problem in related technologies where "when image processing is performed on the fused frame based on the statistical information of the preview frame, the color reproduction of the processed fused frame is distorted due to the large difference between the preview frame and the fused frame."

[0064] Furthermore, since the fused frame is obtained by processing multiple original images, the effective information of the fused frame can be greatly improved. Based on this, even in extreme shooting scenarios (such as low-light environments), the effective information of the fused frame can still be ensured, thereby ensuring the accurate acquisition of the pixel values ​​of the fused frame in the future. This solves the technical problem in related technologies that "due to the loss of effective information in the preview frame in low-light environments, it is impossible to generate statistical information based on the preview frame, which in turn makes it impossible to perform image processing in low-light environments."

[0065] Example 2

[0066] Figure 2 This is a schematic flowchart of an image processing apparatus provided according to an embodiment of the present disclosure, such as... Figure 2 As shown, it includes:

[0067] The acquisition module is used to acquire multiple frames of original images and process them to obtain a fused frame.

[0068] The processing module is used to perform image processing on the fused frame based on the pixel values ​​of the fused frame.

[0069] In summary, the image processing device proposed in this disclosure first acquires multiple frames of original images and processes them to obtain a fused frame. Then, it performs image processing on the fused frame based on the pixel values. Since the pixel values ​​of the fused frame accurately reflect its image information, image processing based on these pixel values ​​accurately reflects the color temperature of the final image, maximizing color reproduction and ensuring the precision and accuracy of image processing. This prevents image distortion and solves the technical problem in related technologies where "when image processing is performed on the fused frame based on the statistical information of the preview frame, the color reproduction of the processed fused frame is distorted due to the large difference between the preview frame and the fused frame."

[0070] Furthermore, since this fused frame is obtained by processing multiple original images, its effective information is greatly enhanced. Therefore, even in extreme shooting scenarios (such as low-light environments), the effective information of the fused frame can still be ensured, thereby guaranteeing the accurate acquisition of pixel values ​​for the subsequent fused frame. This solves the technical problem in related technologies where "effective information in the preview frame is lost in low-light environments, making it impossible to generate statistical information based on the preview frame, thus hindering image processing in low-light environments."

[0071] Optionally, multiple original images can be multiple raw images, and multiple raw images can be multiple raw images with different brightness exposures.

[0072] Optionally, the multiple original images are multiple high-exposure raw images, and the above acquisition module is also used for:

[0073] Perform RAW algorithm calculations on multiple frames of original images;

[0074] Perform multi-frame fusion and noise reduction processing on the multi-frame raw images calculated by the algorithm;

[0075] The denoised multi-frame raw images are fused to obtain a fused single frame; the fusion process includes: fusing the dark areas of the bright images with brightness greater than a first threshold and the bright areas of the dark images with brightness less than a second threshold in the multi-frame original images;

[0076] The merged single frame is brightened to obtain the merged frame.

[0077] Optionally, the above processing module is also used for:

[0078] Calculate the first statistical information of the fused frame based on the pixel values ​​of the fused frame;

[0079] The lens shadow compensation coefficient is calculated based on the first statistical information and the image data of the fused frame using the LSC algorithm, and shadow compensation processing is performed on the fused frame based on the lens shadow compensation coefficient.

[0080] The second statistical information of the fused frame is calculated based on the pixel values ​​of the fused frame, and the second statistical information is output to perform image processing on the fused frame based on the second statistical information.

[0081] Optionally, the above processing module is also used for:

[0082] The fused frame is divided into N×M regions, where N and M are both positive integers; each region includes at least one pixel.

[0083] Calculate the average pixel value for each pixel channel in each region, and determine the first statistical information for each region based on the average pixel value for each region.

[0084] Optionally, the pixel channels included in the fused frame are: R channel, Gr channel, Gb channel, and B channel;

[0085] The primary statistical information for each region includes:

[0086] First value = Average value of R channel for each region ÷ Average value of Gr channel for each region;

[0087] The second value = the average value of the B channel in each region ÷ the average value of the Gb channel in each region.

[0088] Optionally, the above processing module is also used for:

[0089] The processed fused frame is divided into N×M regions, where N and M are both positive integers; each region includes at least one pixel.

[0090] Calculate the average pixel value for each pixel channel in each region, and determine the second statistical information for each region based on the average pixel value for each region.

[0091] Optionally, the pixel channels included in the processed fused frame are: R channel, Gr channel, Gb channel, and B channel;

[0092] The second statistical information for each region includes:

[0093] First value = Average value of R channel for each region ÷ Average value of Gr channel for each region;

[0094] The second value = the average value of the B channel in each region ÷ the average value of the Gb channel in each region.

[0095] The following is a detailed flowchart illustrating the specific process of image processing by the image processing device. Figure 3This is a schematic flowchart of an image processing method provided according to an embodiment of the present disclosure, with reference to... Figure 3 It is known that the image processing method specifically includes a Raw algorithm processing flow and a Post-AWB processing flow. The Raw algorithm processing flow may include: acquiring N original raw images from the image sensor, then performing a multi-frame fusion algorithm (e.g., raw algorithm) on the acquired N raw images, and then performing noise reduction and brightening processing on the calculated raw images to obtain a fused frame of the raw images. After that, the fused frame is input into ISP1 (Image Signal Processing) to perform the Post-AWB processing flow on the fused frame.

[0096] The Post-AWB processing flow may include: calculating the first statistical information of the fused frame using ISP1 (e.g., the Tintless statistical information required by the LSC algorithm); inputting the first statistical information and the fused frame into ISP2, so that ISP2 uses the LSC algorithm to calculate the lens shading compensation coefficient based on the first statistical information and the image information of the fused frame, and applies the lens shading compensation coefficient to the fused frame to perform shading compensation processing; then calculating the second statistical information of the processed fused frame (e.g., the AWB statistical information required by the AWB algorithm), and inputting the obtained second statistical information into the AWB algorithm for calculation to obtain the AWB result that conforms to the fused frame; and finally, using the obtained AWB result to perform image processing (e.g., white balance processing) on ​​the fused frame to restore the color of the fused frame.

[0097] To implement the above embodiments, this disclosure also proposes a computer storage medium.

[0098] The computer storage medium provided in this embodiment stores an executable program; after the executable program is executed by a processor, it can achieve the following: Figure 1 The method shown.

[0099] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0100] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain.

[0101] Although embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present disclosure.

Claims

1. An image processing method, characterized in that, The method includes: Acquire multiple frames of original images, process the multiple frames of original images to obtain a fused frame, wherein the multiple frames of original images are multiple raw images, and the multiple frames of raw images are multiple raw images with different brightness exposures; Image processing is performed on the fused frame based on its pixel values. The fused frame is divided into N×M regions, where N and M are both positive integers. The average pixel value of each pixel channel (R, Gr, Gb, B) in each region is calculated. First statistical information for each region is obtained based on the average values ​​of each pixel channel. This first statistical information reflects the degree of color deviation in the fused frame. A lens shadow correction algorithm is used to calculate a lens shadow compensation coefficient based on the first statistical information and the image data of the fused frame. Shadow compensation processing is performed on the fused frame based on this compensation coefficient. Second statistical information for each region is recalculated on the shadow-compensated fused frame. The calculation method for the second statistical information is the same as that for the first statistical information. Image processing is then performed on the shadow-compensated fused frame based on the second statistical information.

2. The method as described in claim 1, characterized in that, The process of processing the original image to obtain the fused frame includes: Perform RAW algorithm calculations on the multiple frames of original images; Perform multi-frame fusion and noise reduction processing on the original images after algorithm calculation; The original images of multiple frames after noise reduction are fused to obtain a fused single frame; the fusion process includes: fusing the dark areas of bright images with brightness greater than a first threshold and the bright areas of dark images with brightness less than a second threshold in the original images of multiple frames; The merged single frame is then brightened to obtain the merged frame.

3. The method as described in claim 1, characterized in that, The fused frame includes the following pixel channels: R channel, Gr channel, Gb channel, and B channel; The primary statistical information for each region includes: First value = Average value of R channel for each region ÷ Average value of Gr channel for each region; The second value = the average value of the B channel in each region ÷ the average value of the Gb channel in each region.

4. The method as described in claim 1, characterized in that, The image processing includes white balance processing.

5. An image processing device, characterized in that, The device includes: The acquisition module is used to acquire multiple frames of raw images, process the multiple frames of raw images to obtain a fused frame, wherein the multiple frames of raw images are multiple raw images with different brightness exposures; The processing module is used to perform image processing on the fused frame based on the pixel values ​​of the fused frame. The processing module is used to divide the fused frame into N×M regions, where N and M are both positive integers; calculate the average pixel value of each pixel channel (R, Gr, Gb, B) in each region; obtain first statistical information for each region based on the average value of each pixel channel in each region; the first statistical information is used to reflect the degree of color deviation of the fused frame; calculate the lens shadow compensation coefficient based on the first statistical information and the image data of the fused frame using a lens shadow correction algorithm; perform shadow compensation processing on the fused frame based on the compensation coefficient; recalculate the second statistical information of each region for the shadow-compensated fused frame, the calculation method of the second statistical information is the same as that of the first statistical information; and perform image processing on the shadow-compensated fused frame based on the second statistical information.

6. A computer storage medium, wherein, The computer storage medium stores computer-executable instructions; when the computer-executable instructions are executed by the processor, they can implement the method described in claims 1-4.

Images