Image processing method and device, electronic equipment and computer readable storage medium

Abstract

This application relates to an image processing method, apparatus, computer device, storage medium, and computer program product. The method includes: acquiring multiple images of different brightness; determining a set of images to be processed corresponding to each image processing module from the multiple images of different brightness; the set of images to be processed includes at least one frame of the image; and inputting the images to be processed into their respective corresponding image processing modules for processing to obtain a target image. This method can improve image quality.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image technology, and in particular to an image processing method and device, electronic equipment and a computer readable storage medium. BACKGROUND

[0002] With the development of image technology, electronic equipment can take clearer and clearer images. However, electronic equipment is limited by the size, which leads to the problem of low image quality. The traditional image processing method usually uses a multi-frame algorithm to improve the image quality.

[0003] However, the traditional image processing method has the problem of low image quality after processing. SUMMARY

[0004] The embodiments of the present application provide an image processing method, device, electronic equipment, computer readable storage medium and computer program product, which can improve the image quality.

[0005] In a first aspect, the present application provides an image processing method. The method comprises:

[0006] obtaining a plurality of images with different brightness;

[0007] determining, from the plurality of images with different brightness, a set of images to be processed corresponding to each image processing module; the set of images to be processed includes at least one frame of the images;

[0008] inputting the set of images to be processed into the corresponding image processing module respectively for processing to obtain a target image.

[0009] In a second aspect, the present application further provides an image processing device. The device comprises:

[0010] an acquisition unit configured to obtain a plurality of images with different brightness;

[0011] a determination unit configured to determine, from the plurality of images with different brightness, a set of images to be processed corresponding to each image processing module; the set of images to be processed includes at least one frame of the images;

[0012] a processing unit configured to input the set of images to be processed into the corresponding image processing module respectively for processing to obtain a target image.

[0013] In a third aspect, the present application further provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0014] obtaining a plurality of images with different brightness;

[0015] From multiple images with different brightness levels, determine the image set to be processed corresponding to each image processing module; the image set to be processed includes at least one frame of the image.

[0016] The set of images to be processed is input into their respective image processing modules for processing to obtain the target image.

[0017] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0018] Acquire multiple images with different brightness levels;

[0019] From multiple images with different brightness levels, determine the image set to be processed corresponding to each image processing module; the image set to be processed includes at least one frame of the image.

[0020] The set of images to be processed is input into their respective image processing modules for processing to obtain the target image.

[0021] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0022] Acquire multiple images with different brightness levels;

[0023] From multiple images with different brightness levels, determine the image set to be processed corresponding to each image processing module; the image set to be processed includes at least one frame of the image.

[0024] The set of images to be processed is input into their respective image processing modules for processing to obtain the target image.

[0025] The aforementioned image processing methods, apparatuses, electronic devices, computer-readable storage media, and computer program products acquire multiple images with different brightness levels. For each image processing module, a corresponding set of images to be processed can be determined from these multiple images with different brightness levels. This set of images to be processed includes at least one frame, avoiding the problems of low image processing efficiency and inaccurate image processing caused by each image processing module using a fixed frame-taking method to process the same set of images to be processed. Therefore, by inputting the sets of images to be processed into their respective corresponding image processing modules for processing, a target image with higher image processing efficiency, higher clarity, and better image quality can be obtained. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart of an image processing method in one embodiment;

[0028] Figure 2 A flowchart of an image processing method in another embodiment;

[0029] Figure 3 This is a schematic diagram of two parallel ADC readout circuits in one embodiment;

[0030] Figure 4 This is a schematic diagram of eight parallel ADC readout circuits in one embodiment;

[0031] Figure 5 This is a circuit diagram of an ADC readout circuit that simultaneously reads out pixels in one embodiment;

[0032] Figure 6 Here is a circuit diagram of an 8-share pixel structure in one embodiment;

[0033] Figure 7 This is a structural block diagram of an image processing device in one embodiment;

[0034] Figure 8 This is a diagram of the internal structure of an electronic device in one embodiment. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0036] In one embodiment, such as Figure 1As shown, an image processing method is provided. This embodiment illustrates the application of this method to an electronic device, which can be a terminal or a server; it can also be applied to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart vehicle devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. The server can be a standalone server or a server cluster consisting of multiple servers.

[0037] In this embodiment, the image processing method includes the following steps:

[0038] Step 102: Acquire multiple images with different brightness levels.

[0039] Optionally, the electronic device can expose the image with different exposure parameters to acquire multiple images of varying brightness. Exposure parameters include exposure amount and exposure time.

[0040] Exposure refers to the integral of the illuminance (EV) received by a specific surface element of an object over time t. Exposure = Illuminance × Exposure Time. Illuminance is determined by the aperture, while exposure time is controlled by the shutter speed. The aperture size and shutter speed determine the amount of exposure. Therefore, exposure is controlled by both the aperture and shutter speed. Exposure time refers to the duration the shutter must be open to project light onto the photosensitive surface of the photographic material.

[0041] Multiple images of different brightness can include an image of first brightness, an image of second brightness, and an image of third brightness; the first brightness, the second brightness, and the third brightness increase sequentially.

[0042] It is understandable that the exposure time of the image with the first brightness, the image with the second brightness, and the image with the third brightness increases sequentially, as does the exposure amount of the image with the first brightness, the image with the second brightness, and the image with the third brightness.

[0043] For example, the image with the first brightness can be EV-, the image with the second brightness can be EV0, and the image with the third brightness can be EV+. Among them, EV- can be EV-1 or EV-2, etc., and EV+ can be EV+1 or EV+2, etc., and is not limited thereto.

[0044] Optionally, the electronic device triggers a high-quality image capture mode to obtain the current scene situation and judge it with the scene conditions to obtain the target scene where it is currently located; based on the target scene where it is currently located, it determines the required image and sends the frame capture control command to the image sensor to acquire multiple images with different brightness; the frame capture control command includes information about the required image.

[0045] Optionally, the electronic device acquires at least one of the following: current ambient light intensity, ambient dynamic range, and motion state of the captured content; and determines the current target scene based on at least one of the following: current ambient light intensity, ambient dynamic range, and motion state of the captured content. If the electronic device is in an indoor environment or during the day, the current ambient light intensity is less than a preset brightness threshold; if the electronic device is in an outdoor environment or at night, the current ambient light intensity is less than a preset brightness threshold. The ambient dynamic range includes high dynamic range and low dynamic range, and the motion state of the captured content includes stable and unstable states.

[0046] For example, the current target scenario can be an outdoor environment, a high dynamic range, and a jittery state. For example, the current target scenario can also be an indoor environment, a high dynamic range, and a stable state.

[0047] Furthermore, the electronic device can dynamically determine the target requirements based on the current target scene, acquire multiple images with different brightness levels corresponding to the target requirements, and thus obtain a target image that achieves the target requirements. These target requirements include at least one of the following: noise reduction, high definition, high dynamic range, and deblurring.

[0048] For example, if the target requirement is to take a picture, and both the electronic device and the subject being photographed are in a stable state, then in response to the shooting operation, multiple images with different brightness are acquired from the image exposed before the shooting operation; if the target requirement is to take a picture, and the electronic device is in a stable state, but the subject being photographed is in an unstable state, then in response to the shooting operation, multiple images with different brightness are acquired from the image exposed after the shooting operation, and a first image to be processed EV- is acquired as a supplementary frame; if the target requirement is to take a picture, and the electronic device is in an unstable state, then in response to the shooting operation, multiple images with different brightness are acquired from the image exposed after the shooting operation, and a first image to be processed EV- is acquired as a supplementary frame.

[0049] For example, if the target demand is a first dynamic range, then in response to the shooting operation, multiple images with different brightness are acquired from the image exposed before the shooting operation; if the target demand is a second dynamic range, then in response to the shooting operation, multiple images with different brightness are acquired from the image exposed after the shooting operation, and a first image to be processed (EV-) is acquired as a supplementary frame; if the target demand is a third dynamic range, then in response to the shooting operation, multiple images with different brightness are acquired from the image exposed after the shooting operation, and the first image to be processed (EV- or EV-2 image) or the third image to be processed (EV+) is used as a supplementary frame based on environmental requirements; wherein the first dynamic range, the second dynamic range, and the third dynamic range do not overlap and increase sequentially. For example, the first dynamic range is less than 70dB, the second dynamic range is greater than or equal to 70dB and less than 88dB, and the third dynamic range is greater than 88dB.

[0050] For example, if the target requirement is noise reduction and the ISO sensitivity is less than a preset ISO threshold, then in response to the shooting operation, multiple images with different brightness levels are acquired from the image exposed before the shooting operation; if the target requirement is noise reduction and the ISO sensitivity is greater than or equal to the preset ISO threshold, then in response to the shooting operation, multiple images with different brightness levels are acquired from the image exposed after the shooting operation, and a first image to be processed (EV-) is acquired as a supplementary frame. The preset ISO threshold can be 1600.

[0051] For example, if the target requirement is ultra-high definition, then based on the current target scene, it is determined whether the ultra-high definition function needs to be activated; if the ultra-high definition function is activated, then in response to the shooting operation, multiple images with different brightness are acquired from the images exposed before the shooting operation.

[0052] Step 104: Determine the image set to be processed for each image processing module from multiple images with different brightness; the image set to be processed includes at least one frame of image.

[0053] The image processing module includes at least one of a sharpening module, a noise reduction module, and a high dynamic range (HDR) imaging module. The sharpening module includes at least one of a pixel shift module and a deblur module. The noise reduction module can be a multi-frame noise reduction (MFNR) module or a single-frame noise reduction module.

[0054] In other optional embodiments, the image processing module may also include an image compression module, an image encoding module, an enhancement and restoration module, etc., and is not limited thereto.

[0055] Optionally, the electronic device acquires the image information to be processed required by each image processing module, and based on the image information to be processed required by each image processing module, determines the image set to be processed corresponding to each image processing module from multiple images with different brightness.

[0056] Step 106: Input the image set to be processed into their respective image processing modules for processing to obtain the target image.

[0057] The various image processing modules can be connected in series or in parallel.

[0058] In one optional implementation, the electronic device inputs a set of images to be processed into an integrated module group consisting of multiple image processing modules. Within this integrated module group, the set of images to be processed is input into its respective corresponding image processing module for processing to obtain the target image; the various image processing modules are connected in parallel. The multiple image processing modules are integrated into the integrated module group using AI (Artificial Intelligence) technology.

[0059] In one optional implementation, the electronic device sequentially determines the set of images to be processed for the series-connected image processing modules, and sequentially inputs the set of images to be processed into the series-connected image processing modules to obtain the target image. The order in which the image processing modules are connected in series is not limited and can be set as needed.

[0060] It should be noted that in the cascaded image processing modules, the input of the image processing module includes the set of images to be processed, as well as the image output after processing by the previous image processing module.

[0061] For example, the image processing modules connected in series sequentially include a pixel displacement module, a noise reduction module, a deblurring module, and an HDR module. If the electronic device determines that the image set to be processed corresponding to the pixel displacement module includes N EV0s, it inputs the N EV0s into the pixel displacement module and outputs a frame of pixel-displaced processed image. If it determines that the image set to be processed corresponding to the noise reduction module includes N EV0s, it inputs the N EV0s and a frame of pixel-displaced processed image into the noise reduction module and outputs a frame of noise-reduced image. If it determines that the image set to be processed corresponding to the deblurring module includes N EV- and M EV0s, it inputs the N EV-, M EV0s, and a frame of noise-reduced image into the deblurring module and outputs a frame of deblurred image. If it determines that the image set to be processed corresponding to the HDR module includes N EV-, M EV0s, and K EV+s, it inputs the N EV-, M EV0s, K EV+s, and a frame of deblurred image into the HDR module and outputs a frame of HDR processed image, thus obtaining the target image. Here, N, M, and K are all positive integers.

[0062] The image processing method described above acquires multiple images with different brightness levels. For each image processing module, a corresponding set of images to be processed can be determined from the multiple images with different brightness levels. This set of images to be processed includes at least one frame. This avoids the problem of low image processing efficiency and inaccurate image processing caused by each image processing module using a fixed frame-taking method to process the same set of images to be processed. Therefore, by inputting the set of images to be processed into their respective image processing modules for processing, a target image with higher image processing efficiency, higher clarity, and better image quality can be obtained.

[0063] In one embodiment, determining the image set to be processed corresponding to each image processing module from multiple images of different brightness includes: determining a first image subset and a second image subset to be processed corresponding to the deblurring module from multiple images of different brightness; determining a second image subset to be processed corresponding to both the pixel shifting module and the noise reduction module; and determining a first image subset, a second image subset to be processed, and a third image subset to be processed corresponding to the HDR module; wherein the image brightness of the first image subset, the second image subset to be processed, and the third image subset to be processed increases sequentially.

[0064] The image brightness of the first, second, and third image subsets increases sequentially. That is, the images in the first image subset are underexposed and have low brightness, the images in the second image subset are normally exposed and have medium brightness, and the images in the third image subset are overexposed and have high brightness.

[0065] For example, the images in the first subset of images to be processed are EV-, which have a short exposure time, so that the bright areas in the image are not overexposed and are less prone to blurring due to jitter; the images in the third subset of images to be processed are EV+, which can improve the low-light brightness and SNR (signal-noise ratio) of the image.

[0066] Understandably, the deblurring module can optimize the movement of blurred areas, and the images to be processed are the images of the first subset of images to be processed and the images of the second subset of images to be processed; the pixel shifting module can improve the image sharpness, and the images to be processed are the images of the second subset of images to be processed; the noise reduction module can improve the signal-to-noise ratio of the image, and the images to be processed are the images of the second subset of images to be processed; the HDR module can improve the dynamic range of the image, and the images to be processed are the images of the first subset of images to be processed, the images of the second subset of images to be processed, and the images of the third subset of images to be processed.

[0067] For example, the electronic device can determine that the deblurring module needs to process N EV- and M EV0 images, the pixel shifting module and the noise reduction module need to process N EV0 images, and the HDR module needs to process N EV-, M EV0, and K EV+ images. Here, N, M, and K can be set as needed and are not limited here; EV- represents images in the first subset of images to be processed, EV0 represents images in the second subset of images to be processed, and EV+ represents images in the third subset of images to be processed.

[0068] Understandably, electronic devices use multi-frame image processing to achieve functions such as noise reduction, high definition, high dynamic range, and deblurring. Each function's corresponding image processing module has different requirements for the subset of images to be processed. For example, the noise reduction module requires a larger number of frames in the second subset of images to be processed (EV0), but the high dynamic range function implemented by the HDR module requires even more frames in the first subset of images to be processed (EV-) and the third subset (EV+1). The deblurring function implemented by the deblurring module even requires the first subset of images to be processed to be EV-2.

[0069] Therefore, electronic devices determine the required images and the number of each image based on the scene and the content being captured, and then acquire multiple images with different brightness levels.

[0070] For example, images taken in outdoor scenes have relatively low noise. If the electronic device shakes a lot (such as when walking while holding the camera), then the overall logic is that the frame rate requirement for EV0 is not high, while the number of EV- will be more.

[0071] Furthermore, the electronic device determines the required images and the number of each image based on the scene, the content being captured, and the power consumption, and then acquires multiple images with different brightness levels.

[0072] Understandably, there's a conflict between the number of images and computing power / power consumption; more images result in more power consumption and more computing power. Therefore, when shooting photos, the required computing power and power consumption are relatively low, allowing for the acquisition of more images. Conversely, when shooting videos, the required computing power and power consumption are higher, necessitating controlling the output frame rate within a preset range to achieve a relative balance between frame rate, power consumption, and computing power.

[0073] In this embodiment, the electronic device determines the first and second subsets of images to be processed corresponding to the deblurring module from multiple images with different brightness, determines the second subset of images to be processed corresponding to both the pixel shifting module and the noise reduction module, and determines the first, second, and third subsets of images to be processed corresponding to the HDR module. This allows for the determination of images to be processed according to different requirements for different image processing modules, thereby enabling more accurate image processing for each image.

[0074] Furthermore, the current algorithm framework design is based on a comprehensive consideration of pixel shifting, noise reduction, deblurring, and HDR modules, which can improve the overall efficiency of image processing and image quality.

[0075] In one embodiment, determining the image set to be processed corresponding to each image processing module from multiple images with different brightness includes: determining the current target scene; determining the weight corresponding to each image processing module based on the target scene; and determining the image set to be processed corresponding to each image processing module from multiple images with different brightness based on the weight corresponding to each image processing module.

[0076] Optionally, determining the current target scene includes: determining the current target scene based on at least one of the current ambient light intensity, environmental dynamic range, and motion state of the captured content.

[0077] For example, the target scene could be characterized by low ambient light, low dynamic range, and a stable motion state of the subject being filmed. Alternatively, the target scene could be characterized by high ambient light, high dynamic range, and an unstable motion state of the subject being filmed.

[0078] Optionally, the electronic device determines the weight of each image processing module based on the target scene and the preset correspondence between the scene and the weights of each image processing module. The preset correspondence between the scene and the image processing modules can be set as needed. For example, if the scene is an outdoor environment with shaking, the deblurring module and the HDR module will have higher weights, while the pixel shifting module and the noise reduction module will have lower weights.

[0079] Optionally, the electronic device determines the target number of images to be processed for each image processing module from multiple images with different brightness levels, based on the weights corresponding to each image processing module. The weights of the image processing modules are positively correlated with the number of images in the corresponding image sets to be processed.

[0080] In this embodiment, the electronic device determines the current target scene; based on the target scene, it determines the weight corresponding to each image processing module; based on the weight corresponding to each image processing module, it can determine the image set to be processed corresponding to each image processing module from multiple images with different brightness; then each image processing module processes its corresponding image set to be processed to obtain a target image that is more consistent with the target scene.

[0081] In one embodiment, based on the weight corresponding to each image processing module, the set of images to be processed corresponding to each image processing module is determined from multiple images with different brightness. This includes: for each image processing module, determining the number of images of each type of image to be processed corresponding to the image processing module based on the target scene and the weight of the image processing module; and obtaining the corresponding set of images of each type of image to be processed from multiple images with different brightness according to the number of images of each type of image to be processed, wherein the images to be processed are divided into different types according to brightness.

[0082] Optionally, the electronic device obtains the correspondence between scene and module weights and the number of images in each type of image to be processed. For each image processing module, based on the target scene and the weights of the image processing module, the number of images in each type of image to be processed corresponding to the image processing module can be determined from the correspondence, and the number of images to be processed in each type of image can be obtained from multiple images with different brightness.

[0083] It is understandable that the weight of the image processing module is positively correlated with the number of images in the corresponding image set to be processed. That is, the greater the weight of the image processing module, the more images in the image set to be processed will be input into the image processing module. At the same time, the number of images of each type of image to be processed corresponding to the image processing module can be adjusted based on the target scene.

[0084] For example, if the weight of image processing module A is 0.8, then the number of images of each type of image to be processed, EV- and EV+, is relatively large. Since the target scene is a low-light environment, the number of images of EV+ to be processed can be increased and the number of images of EV- to be processed can be decreased.

[0085] In this embodiment, for each image processing module, based on the target scene and the weight of the image processing module, the number of images of each type of image to be processed corresponding to the image processing module can be determined more accurately, thereby obtaining more accurate images of each type of image to be processed by the image processing module.

[0086] In one embodiment, the method further includes: determining the current target scene; determining the current target requirement based on the target scene; and determining the base frame and supplementary frame from each image based on the target requirement.

[0087] In one implementation, the target scene includes the current ambient light intensity. If the current ambient light intensity is lower than a preset brightness threshold, the current target requirement is obtained.

[0088] The preset brightness threshold can be set as needed. Target requirements may include at least one of the following: capture requirements, HDR requirements, NR (Noise Reduction) requirements, and SR (Super-Resolution) requirements.

[0089] In another implementation, the target scene includes the current ambient light brightness. If the current ambient light brightness is higher than or equal to a preset brightness threshold, it means that the electronic device is in a high-light environment with sufficient exposure time. There is no conflict between the images to be processed by each image processing module, and multiple images with different brightness can be acquired as needed.

[0090] The base frame is the primary image to be processed. Supplementary frames are images used to add image information to the base frame.

[0091] Optionally, the electronic device acquires the correspondence between the base frame and the supplementary frame and the requirement, determines the base frame and supplementary frame corresponding to the target requirement from the correspondence, and acquires the base frame and supplementary frame from each image.

[0092] In this embodiment, by detecting the current ambient light intensity, multiple images with different brightness levels corresponding to the current ambient light intensity can be acquired. Furthermore, if the current ambient light intensity is lower than a preset brightness threshold, a target requirement is obtained, and multiple images with different brightness levels are acquired based on the target requirement. This allows for the processing of these multiple images with different brightness levels that meet the target requirement, resulting in a target image that better meets the target requirement.

[0093] In one embodiment, determining a base frame and a supplementary frame from various images based on a target requirement includes: if the target requirement is high dynamic range imaging, acquiring at least one second image to be processed as a base frame and acquiring a first image to be processed as a supplementary frame; wherein the brightness of the first image to be processed is less than the brightness of the second image to be processed; if the target requirement is noise reduction, acquiring at least one first image to be processed as a base frame and acquiring a second image to be processed as a supplementary frame; if the target requirement is super-resolution imaging, acquiring at least one first image to be processed as a base frame and acquiring a second image to be processed as a supplementary frame; and inputting the set of images to be processed into their respective corresponding image processing modules for processing to obtain a target image, including: inputting the set of images to be processed into their respective corresponding image processing modules, and supplementing the base frame with the information of the supplementary frame to obtain the target image.

[0094] For example, if the target requirement is high dynamic range imaging, then in response to the shooting operation, four second images to be processed EV0 are acquired from the image before the shooting operation as base frames, and a first image to be processed EV- with a short exposure time (shutter time-) and a first image to be processed EV- with negative gain (gain-) are acquired as supplementary frames; if the target requirement is noise reduction, then four or more first images to be processed EV- with short exposure times are acquired as base frames, and a second image to be processed EV0 with normal exposure time and a second image to be processed EV0 with long exposure time are acquired as supplementary frames; if the target requirement is super-resolution imaging, then four or more first images to be processed EV- with short exposure times are acquired as base frames, and four second images to be processed are acquired as supplementary frames for high-definition requirements.

[0095] Optionally, during the processing, the image processing module in the electronic device uses the base frame as the main image to be processed, and adds the information of the supplementary frame to the base frame for processing to obtain the target image.

[0096] In this embodiment, the electronic device acquires different images to be processed as base frames and supplementary frames based on different target requirements, thereby enabling more accurate processing of the images to be processed and obtaining target images that better meet the target requirements.

[0097] In one embodiment, such as Figure 2 As shown, the electronic device triggers a high-quality image capture mode to acquire the current scene conditions and determine the target scene. Based on the target scene, it identifies the required images and sends a frame acquisition control command to the image sensor to acquire multiple images with different brightness levels. The frame acquisition control command includes information about the required images. Simultaneously, the electronic device performs weight allocation based on the target scene, determining the weight corresponding to each image processing module. Based on the weight corresponding to each image processing module, it determines the image set to be processed for each image processing module from the multiple images with different brightness levels. The image sets to be processed are then input into their respective image processing modules for processing to obtain the target image.

[0098] In one embodiment, the electronic device uses DOL (Digital Overlap) technology for taking pictures. In outdoor, indoor, or relatively bright night scenes, the preview scale is dynamically adjusted according to the dynamic range of the environment to ensure the dynamic range of the preview.

[0099] In outdoor, indoor, or relatively bright night scenes, if the electronic device and the subject are stable, and the image brightness requirement is 0EV-4.58EV, then N EV0 images are acquired using ZSL (Zero Shutter Lag) mode. EV0 images include long exposure zero-latency images (LZSL, Long Zero Shutter Lag) and single exposure zero-latency images (SZSL, Short Zero Shutter Lag). If the electronic device and the subject are stable, and the image brightness requirement is 4.58EV-6EV, then EV-(-6EV PSL) and N EV0 images are acquired using ZSL (Zero Shutter Lag) mode and PSL (Post Frame Capture, i.e., capturing frames after the shooting operation). EV0 images include long exposure zero-latency images and single exposure zero-latency images.

[0100] In outdoor, indoor, or relatively bright night scenes, when taking photos, if the electronic device is stable, the subject is unstable, and the image brightness requirement is 0EV-4.58EV, then N EV0 / 4 values ​​are obtained using ZSL method, along with EV+ for long exposure zero-delay photography, or N EV0 values ​​and EV0 / 4 values ​​are obtained.

[0101] In outdoor, indoor, or relatively bright night scenes, when taking photos, if the electronic device is stable, the subject is unstable, and the image brightness requirement is 4.58EV-6EV, then N EV0 / 4 and EV--(-6EVPSL) are obtained using ZSL and PSL methods.

[0102] In outdoor, indoor, or relatively bright night scenes, when taking photos, if the electronic device is unstable, the subject being photographed is unstable, and the image brightness requirement is 0EV-4.58EV, then N EV0 / 4 values ​​are obtained using the ZSL method.

[0103] In outdoor, indoor, or relatively bright night scenes, if the electronic device and the subject being photographed are unstable, and the image brightness requirement is 4.58EV-6EV, then N EV0 / 4 and EV--(-6EVPSL) are obtained using ZSL and PSL methods.

[0104] In low-light scenes, if the electronic device and the subject are stable, and the image brightness requirement is 0EV-6EV, then switch to binning mode and capture frames using PSL.

[0105] In low-light scenes, when taking photos, if the electronic device is stable, the subject is unstable, and the image brightness requirement is 0EV-6EV, then switch to binning mode and capture frames using PSL method to obtain N EV0 / 4 and EV-, or obtain EV0 / 4, EV- and N EV0.

[0106] In low-light scenes, if the electronic device is unstable and the image brightness requirement is 0EV-6EV, switch to binning mode and capture frames using PSL to obtain EV-(-6EV,PSL) and N EV0 / 4.

[0107] In one embodiment, the method further includes: using multiple sets of digital-analog conversion (ADC) simultaneous readout circuits to read out pixels, and obtaining an image based on each readout pixel; each set of ADC readout circuits includes multiple ADC readout circuits, each ADC readout circuit reads out one pixel, and each set of ADC readout circuits simultaneously reads out one row of pixels.

[0108] It is understandable that if each ADC readout circuit can read one row of pixels, then using multiple ADC readout circuits simultaneously to read multiple rows of pixels can improve image readout efficiency and increase the output frame rate of the CIS sensor (CMOS Image Sensor).

[0109] like Figure 3 As shown, the image sensor has two sets of ADC readout circuits, which can read out two rows of pixels simultaneously. Figure 4 As shown, the image sensor has 8 sets of ADC readout circuits, which can read out 8 rows of pixels at the same time, and the frame rate is 4 times that of 2 sets of ADC readout circuits (2-core ADC).

[0110] like Figure 5 As shown, the multi-row ADC readout circuit can simultaneously read out multiple rows of pixels, therefore a 4-in-1 sensor (4-share pixel structure) is used in the circuit design. Here, 4-share means that 4 photodiodes share one source follower, which can save the number of transistors.

[0111] exist Figure 5 In the first column, the pixels in rows 8N+1 share a single ADC readout circuit. Similarly, the pixels in rows 8N+2 of the first column share a single ADC readout circuit, and the pixels in rows 8N+8 of the first column share a single ADC readout circuit.

[0112] During the operating cycle of the ADC readout circuit, the first column of readout pixels is: C1R1, C1R2, C1R3, C1R4, C1R5, C1R6, C1R7, C1R8. For an 8-share pixel structure, the basic principle is the same, with the first column of readout pixels being: C1R1, C1R3, C1R5, C1R7, C1R9, C1R11, C1R13, C1R15. Where C: Col, representing column; R: Row, representing row. The 8-share pixel structure is as follows... Figure 6 As shown. Here, 8share means that 8 photodiodes share one source follower, which can save the number of transistors.

[0113] In this embodiment, the electronic device uses multiple sets of digital-to-analog converter (ADC) readout circuits. That is, through hardware high frame rate design, multiple rows of pixels can be read out simultaneously. By combining hardware and software design, more image information can be acquired per unit time, resulting in a target image with higher output efficiency and enhanced image quality.

[0114] With the current CIS using the rolling shutter exposure method, high frame rate images can provide image input with less rolling shutter effect and smaller differences between frames, which helps to simplify the algorithm and thus optimize overall power consumption.

[0115] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0116] Based on the same inventive concept, this application also provides an image processing apparatus for implementing the image processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more image processing apparatus embodiments provided below can be found in the limitations of the image processing method described above, and will not be repeated here.

[0117] In one embodiment, such as Figure 7As shown, an image processing apparatus is provided, including: an acquisition unit 702, a determination unit 704, and a processing unit 706, wherein:

[0118] The acquisition unit 702 is used to acquire multiple images with different brightness levels.

[0119] The determining unit 704 is used to determine the image set to be processed corresponding to each image processing module from multiple images with different brightness; the image set to be processed includes at least one frame of image.

[0120] The processing unit 706 is used to input the images to be processed into their respective corresponding image processing modules for processing to obtain the target image.

[0121] The aforementioned image processing device acquires multiple images with different brightness levels. Therefore, for each image processing module, the corresponding image to be processed can be determined from the multiple images with different brightness levels. This avoids the problem of low image processing efficiency and inaccurate image processing caused by each image processing module using a fixed frame-taking method to process the same image to be processed. Therefore, by inputting the images to be processed into their respective corresponding image processing modules for processing, a target image with higher image processing efficiency, higher clarity, and better image quality can be obtained.

[0122] In one embodiment, the image processing module includes at least one of a sharpening module, a noise reduction module, and a high dynamic range imaging (HDR) module.

[0123] In one embodiment, the sharpening module includes at least one of a pixel shifting module and a deblurring module.

[0124] In one embodiment, the determining unit 704 is further configured to determine, from multiple images with different brightness, a first subset of images to be processed and a second subset of images to be processed corresponding to the deblurring module, a second subset of images to be processed corresponding to both the pixel shifting module and the noise reduction module, and a first subset of images to be processed, a second subset of images to be processed, and a third subset of images to be processed corresponding to the HDR module; wherein the image brightness of the first subset of images to be processed, the second subset of images to be processed, and the third subset of images to be processed increases sequentially.

[0125] In one embodiment, the determining unit 704 is further configured to determine the current target scene; determine the weight corresponding to each image processing module based on the target scene; and determine the set of images to be processed corresponding to each image processing module from multiple images with different brightness based on the weight corresponding to each image processing module.

[0126] In one embodiment, the determining unit 704 is further configured to determine the current target scene based on at least one of the current ambient light intensity, the dynamic range of the environment, and the motion state of the shooting content.

[0127] In one embodiment, the weight of the image processing module is positively correlated with the number of images in the corresponding image set to be processed.

[0128] In one embodiment, the determining unit 704 is further configured to, for each image processing module, determine the number of images of each type of image to be processed corresponding to the image processing module based on the target scene and the weight of the image processing module, and obtain the corresponding set of images of each type of image to be processed from multiple images of different brightness according to the number of images of each type of image to be processed, wherein the images to be processed are divided into different categories according to brightness.

[0129] In one embodiment, the determining unit 704 is further configured to determine the current target scene; determine the current target requirement based on the target scene; and determine the base frame and supplementary frame from each image based on the target requirement.

[0130] In one embodiment, the determining unit 704 is further configured to, if the target requirement is high dynamic range imaging, acquire at least one second image to be processed as a base frame and acquire a first image to be processed as a supplementary frame; wherein the brightness of the first image to be processed is less than the brightness of the second image to be processed; if the target requirement is noise reduction, acquire at least one first image to be processed as a base frame and acquire a second image to be processed as a supplementary frame; if the target requirement is super-resolution imaging, acquire at least one first image to be processed as a base frame and acquire a second image to be processed as a supplementary frame; the processing unit 706 is further configured to input the set of images to be processed into their respective corresponding image processing modules, supplement the information of the supplementary frames into the base frames, and obtain the target image.

[0131] In one embodiment, the above-described apparatus further includes a readout module; the readout module is used to simultaneously read out pixels using multiple sets of digital-to-analog converter (ADC) readout circuits, and obtain an image based on each readout pixel; each set of ADC readout circuits includes multiple ADC readout circuits, each ADC readout circuit reads out one pixel, and each set of ADC readout circuits simultaneously reads out one row of pixels.

[0132] Each module in the aforementioned image processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0133] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements an image processing method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0134] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0135] This application also provides a computer-readable storage medium. One or more non-volatile computer-readable storage media containing computer-executable instructions, which, when executed by one or more processors, cause the processors to perform the steps of an image processing method.

[0136] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform an image processing method.

[0137] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0138] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0139] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0140] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image processing method, characterized in that, include: Acquire multiple images with different brightness levels; Determining the image set to be processed for each image processing module from multiple images of different brightness levels includes: determining the current target scene; determining the weight of each image processing module based on the target scene; determining the image set to be processed for each image processing module from multiple images of different brightness levels based on the weight of each image processing module, wherein the weight of the image processing module is positively correlated with the number of images in the corresponding image set to be processed; the image set to be processed includes at least one frame of the image; the image processing modules are connected in series or in parallel; in the series-connected image processing modules, the input of the image processing module includes the required image set to be processed, and also includes the image output after processing by the previous image processing module; The set of images to be processed is input into their respective image processing modules for processing to obtain the target image.

2. The method according to claim 1, characterized in that, The image processing module includes at least one of a sharpening module, a noise reduction module, and a high dynamic range imaging (HDR) module.

3. The method according to claim 2, characterized in that, The sharpening module includes at least one of a pixel shifting module and a deblurring module.

4. The method according to claim 3, characterized in that, The step of determining the set of images to be processed for each image processing module from multiple images of different brightness includes: From multiple images with different brightness levels, a first subset of images to be processed and a second subset of images to be processed corresponding to the deblurring module are determined; a second subset of images to be processed corresponding to both the pixel shifting module and the noise reduction module are determined; and a first subset of images to be processed, a second subset of images to be processed, and a third subset of images to be processed corresponding to the HDR module are determined; wherein, the image brightness of the first subset of images to be processed, the second subset of images to be processed, and the third subset of images to be processed increases sequentially.

5. The method according to claim 1, characterized in that, Determining the current target scenario includes: The target scene is determined based on at least one of the following: ambient light intensity, dynamic range of the environment, and motion state of the subject being photographed.

6. The method according to claim 1, characterized in that, The step of determining the set of images to be processed for each image processing module from multiple images of different brightness levels based on the weights corresponding to each image processing module includes: For each image processing module, based on the target scene and the weight of the image processing module, the number of images of each type of image to be processed corresponding to the image processing module is determined. Based on the number of images of each type of image to be processed, the corresponding set of images to be processed for each type is obtained from multiple images with different brightness, wherein the images to be processed are divided into different types according to brightness.

7. The method according to claim 1, characterized in that, The method further includes: Determine the current target scenario; Based on the target scenario, determine the current target requirements; Based on the target requirements, base frames and supplementary frames are determined from each image.

8. The method according to claim 7, characterized in that, The process of determining the base frame and supplementary frame from each image based on the target requirement includes: If the target requirement is high dynamic range imaging, then at least one second image to be processed is acquired as a base frame, and a first image to be processed is acquired as a supplementary frame; wherein, the brightness of the first image to be processed is less than the brightness of the second image to be processed. If the target requirement is noise reduction, then at least one first image to be processed is obtained as a base frame, and a second image to be processed is obtained as a supplementary frame. If the target requirement is super-resolution imaging, then at least one first image to be processed is acquired as a base frame, and a second image to be processed is acquired as a supplementary frame. The step of inputting the set of images to be processed into their respective corresponding image processing modules for processing to obtain the target image includes: The set of images to be processed is input into their respective image processing modules, and the information of the supplementary frames is added to the base frames to obtain the target image.

9. The method according to any one of claims 1 to 8, characterized in that, The method further includes: Multiple sets of digital-to-analog converter (ADC) readout circuits are used to read out pixels simultaneously, and an image is obtained based on each readout pixel. Each ADC readout circuit includes multiple ADC readout circuits, each ADC readout circuit reads out one pixel, and each set of ADC readout circuits reads out one row of pixels simultaneously.

10. An image processing apparatus, characterized in that, include: The acquisition unit is used to acquire multiple images with different brightness levels; The determining unit is configured to determine the image set to be processed corresponding to each image processing module from multiple images of different brightness levels, including: determining the current target scene; determining the weight corresponding to each image processing module based on the target scene; determining the image set to be processed corresponding to each image processing module from multiple images of different brightness levels based on the weight corresponding to each image processing module, wherein the weight corresponding to the image processing module is positively correlated with the number of images in the corresponding image set to be processed; the image set to be processed includes at least one frame of the image; the image processing modules are connected in series or in parallel; in the series-connected image processing modules, the input of the image processing module includes the required image set to be processed, and also includes the image output after processing by the previous image processing module; The processing unit is used to input the set of images to be processed into their respective corresponding image processing modules for processing to obtain the target image.

11. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the processor, the processor performs the steps of the image processing method as described in any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images