Image processing method, device and medium
By acquiring the image when the image sensor is in the camera, filtering and stabilizing the black level, and combining the black level calibration value, the black level in the visible light area is determined, which solves the problem of black level drift and improves the image quality.
Patent Information
- Application Number
- CN202510354298.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-24
- Publication Date
- 2025-06-24
AI Technical Summary
In environments with weak light, the camera's image sensor is prone to black level drifting problems, affecting image quality.
By acquiring the image when the image sensor is photosensitive, the black level of the optical black area is determined, and filtering and stably processing it based on the gain of the automatic exposure algorithm to obtain a more stable black level. Then, based on the stabilized black level and the pre-calibrated black level calibration value, the black level of the visible light area is determined to generate a more accurate image.
By suppressing black level drift, the accuracy of black level in visible light areas is improved, thereby improving image quality.
Smart Images
Figure CN120201323A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the technical field of electronic devices, and particularly relates to an image processing method, device, and medium. Background Art
[0002] When a camera takes pictures in an environment with weak light, the image sensor of the camera will have a black level drift problem due to the weak light, which affects the image quality.
[0003] In order to improve the image quality, it is necessary to subtract the black level during the image processing.
[0004] In related technologies, usually the black level and the sensitivity (ISO) are linked to determine the black level to be subtracted. When using the method of linking the black level and ISO to determine the black level to be subtracted, the black level will have a drift fluctuation, which results in inaccurate determination of the black level to be subtracted, and further affects the image quality. Summary of the Invention
[0005] The purpose of the embodiments of this application is to provide an image processing method, device, and medium, which can at least solve the problem of black level drift.
[0006] In a first aspect, the embodiments of this application provide an image processing method, including:
[0007] Obtain a first image collected by an image sensor when the image sensor is sensitive, where the first image includes a visible light region and an optical black region;
[0008] Determine the first black level of the optical black region;
[0009] Perform filtering and stabilization processing on the first black level based on the gain of the automatic exposure algorithm to obtain a second black level;
[0010] Determine the third black level of the visible light region according to the second black level and the black level calibration value; where the black level calibration value is the difference between the pre-calibrated black level of the visible light region and the black level of the optical black region;
[0011] Generate a second image based on the first image and the third black level.
[0012] In a second aspect, the embodiments of this application provide an electronic device, and the electronic device includes an image sensor and an image signal processor;
[0013] The image sensor includes a collection module, and the collection module is used to collect a first image, where the first image is the image collected by the image sensor when the image sensor is sensitive, and the first image includes a visible light region and an optical black region;
[0014] The image signal processor includes:
[0015] A first receiving module, configured to receive a first image sent by an image sensor;
[0016] A first determining module, configured to determine a first black level of an optical black region;
[0017] A filtering module, configured to perform filtering and stabilization processing on the first black level based on an automatic exposure algorithm gain to obtain a second black level;
[0018] A second determining module, configured to determine a third black level of a visible light region according to the second black level and a black level calibration value; wherein, the black level calibration value is a difference between a black level of a pre-calibrated visible light region and a black level of an optical black region;
[0019] A generating module, configured to generate a second image based on the first image and the second black level.
[0020] In a third aspect, an embodiment of the present application provides an electronic device, where the electronic device includes a processor and a memory, the memory stores a program or instruction that can run on the processor, and when the program or instruction is executed by the processor, the steps of the image processing method provided by the embodiment of the present application are implemented.
[0021] In a fourth aspect, an embodiment of the present application provides a readable storage medium, where a program or instruction is stored on the readable storage medium, and when the program or instruction is executed by a processor, the steps of the image processing method provided by the embodiment of the present application are implemented.
[0022] In a fifth aspect, an embodiment of the present application provides a chip, where the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is configured to run a program or instruction to implement the steps of the image processing method provided by the embodiment of the present application.
[0023] In a sixth aspect, an embodiment of the present application provides a computer program product, where the computer program product is stored in a storage medium, and the computer program product is executed by at least one processor to implement the steps of the image processing method provided by the embodiment of the present application.
[0024] In an embodiment of the present application, by acquiring a first image collected by an image sensor when the image sensor senses light, where the first image includes a visible light region and an optical black region; determining a first black level of the optical black region; performing filtering and stabilization processing on the first black level based on the gain of the automatic exposure algorithm to obtain a second black level; determining a third black level of the visible light region according to the second black level and a black level calibration value, where the black level calibration value is the difference between the black level of the pre-calibrated visible light region and the black level of the optical black region; generating a second image based on the first image and the third black level. When determining the black level of the visible light region, performing filtering and stabilization processing on the black level of the optical black region based on the gain of the automatic exposure algorithm to suppress black level drift. Therefore, the accuracy of the black level of the visible light region determined by the black level of the optical black region is relatively high, and thus the image quality can be improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 is a schematic flowchart of an image processing method provided by an embodiment of the present application;
[0026] Figure 2 is a schematic diagram of the positions of the OB region and the visible light region provided by an embodiment of the present application;
[0027] Figure 3 is a timing diagram of configuring the black level of the visible light region provided by an embodiment of the present application;
[0028] Figure 4 is a schematic diagram of the overall process of an image processing method provided by an embodiment of the present application;
[0029] Figure 5 is a first schematic structural diagram of an electronic device provided by an embodiment of the present application;
[0030] Figure 6 is a second schematic structural diagram of an electronic device provided by an embodiment of the present application;
[0031] Figure 7 is a third schematic structural diagram of an electronic device provided by an embodiment of the present application;
[0032] Figure 8 is a schematic hardware structure diagram of an electronic device implementing an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0033] Next, the technical solutions in the embodiments of the present application will be clearly described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application belong to the scope of protection of the present application.
[0034] The terms "first", "second", etc. in the description and claims of this application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such terms can be interchanged under appropriate circumstances, so that the embodiments of this application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and do not limit the number of objects. For example, the first object can be one or more. In addition, "and / or" in the description and claims means at least one of the connected objects, and the character " / ", generally represents an "or" relationship between the associated objects before and after.
[0035] The following will combine the accompanying drawings and, through specific embodiments and their application scenarios, detail the image processing method, device, and medium provided by the embodiments of this application.
[0036] Figure 1 It is a schematic flowchart of the image processing method provided by the embodiments of this application. The image processing method may include:
[0037] Step 101: Obtain a first image collected by an image sensor when the image sensor is photosensitive, where the first image includes a visible light region and an optical black region;
[0038] In some possible implementations of the embodiments of this application, the first image in the embodiments of this application is an unprocessed original image collected by the image sensor.
[0039] In some possible implementations of the embodiments of this application, the format of the first image in the embodiments of this application includes, but is not limited to: RGGB (Red Green Green Blue) format, BGGR (Blue Green Green Red) format, GBRG (Green Blue Red Green) format, GRBG (Green Red Blue Green) format, RGBW (Red Green Blue White) format, etc.
[0040] In some possible implementations of the embodiments of this application, the optical black (OB) region in the embodiments of this application is a black light region where the image sensor is not photosensitive, where the OB region will not be photosensitive.
[0041] In some possible implementations of the embodiments of this application, the OB region in the embodiments of this application can be located on at least one side of the visible light region; for example, the OB region is located above the visible light region, and for another example, the OB region is located on the left and right sides of the visible light region.
[0042] Exemplarily, as Figure 2 shown, Figure 2 is a schematic diagram of the positions of the OB region and the visible light region provided by the embodiments of this application. In Figure 2 there are OB regions around the visible light region.
[0043] Step 102: Determine the first black level of the optical black region;
[0044] In some possible implementations of the embodiments of the present application, software for statistically calculating the black levels of the optical black region and the visible light region can be pre-written, and the black levels of the optical black region and the visible light region are statistically calculated through this software to obtain the black level of the optical black region.
[0045] In some possible implementations of the embodiments of the present application, the software for statistically calculating the black levels of the optical black region and the visible light region can be configured in a module, which can be called a black level statistical module.
[0046] It should be noted that when the image sensor is photosensitive, only the black level of the optical black region can be statistically calculated; only when the image sensor is not photosensitive can the black level of the visible light region be statistically calculated.
[0047] In some possible implementations of the embodiments of the present application, the hardware for determining the black level of the optical black region includes but is not limited to: Image Signal Processor (ISP), Digital Signal Processing (DSP), Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), and Central Processing Unit (CPU), etc.
[0048] When the hardware for determining the black level of the optical black region is an ISP, the width and height of the video input channel of the ISP need to be the same as the width and height of the image transmitted by the image sensor through the Mobile Industry Processor Interface (MIPI).
[0049] When the hardware for determining the black level of the optical black region is a DSP, the width and height of the video input channel of the ISP need to be the same as the width and height of the image after the OB region is cropped after the DSP determines the black level of the optical black region. When using a DSP to determine the black level of the optical black region, the DSP chip manufacturer needs to adapt the corresponding Software Development Kit (SDK) to ensure that the black level statistically calculated by the DSP can be transmitted to the ISP chip.
[0050] In some possible implementations of the embodiments of the present application, when determining the black level of the optical black region, the average black level of the whole can be statistically calculated, or the average black level can be statistically calculated in blocks.
[0051] In some possible implementations of the embodiments of the present application, when determining the black level of the optical black area, the black levels within a preset statistical range may be statistically counted.
[0052] Step 103: Perform filtering and stabilization processing on the first black level based on the gain of the automatic exposure algorithm to obtain a second black level;
[0053] The embodiments of the present application do not limit the stabilization algorithm used for filtering and stabilizing the first black level, and any available method can be applied to the embodiments of the present application. For example, an infinite impulse response (IIR) filter algorithm, a mean filtering algorithm, etc.
[0054] In the embodiments of the present application, by performing filtering and stabilization processing on the first black level, it is possible to avoid excessive differences in black levels between different video frames of a video and ensure the brightness stability of each video frame of the video.
[0055] In some possible implementations of the embodiments of the present application, the video mode involves many timing issues of image quality. If the black level fluctuates too much, it will cause sudden changes in video brightness. Therefore, in the video mode, it is necessary to stabilize the black level of the OB area. Usually, in the video mode, the change value of the gain value of the automatic exposure (AE) algorithm is relatively small. Based on this, step 103 may include: when the change value of the gain value of the automatic exposure algorithm is less than or equal to a first threshold, taking the average value of the first black level and the fourth black level as the second black level, where the fourth black level is the black level of the optical black area in the previous frame image of the first image.
[0056] In some possible implementations of the embodiments of the present application, the first threshold in the embodiments of the present application can be set according to actual requirements.
[0057] Exemplarily, assume that the first threshold is 10 decibels (dB), the change value of the gain value of the automatic exposure algorithm is 5 dB, the first black level is 10 IRE, and the fourth black level is 5 IRE. Then 7.5 IRE is taken as the second black level. Here, IRE is the unit of the black level.
[0058] Another example, assume that the first threshold is 10 dB, the change value of the gain value of the automatic exposure algorithm is 10 dB, the first black level is 10 IRE, and the fourth black level is 5 IRE. Then 7.5 IRE is taken as the second black level.
[0059] In the embodiments of the present application, when the change value of the gain value of the automatic exposure algorithm is less than or equal to the first threshold, by using the average value of the black level of the optical black area of the first image and the black level of the optical black area in the previous frame image of the first image as the second black level, it is possible to avoid excessive differences in the black levels between the video frames of the video, and ensure the brightness stability of each video frame of the video.
[0060] In some possible implementations of the embodiments of the present application, step 103 may include: when the change value of the gain value of the automatic exposure algorithm is greater than the first threshold, using the first black level as the second black level.
[0061] Exemplarily, assume that the first threshold is 10 db, the change value of the gain value of the automatic exposure algorithm is 15 db, the first black level is 10 IRE, and the fourth black level is 5 IRE. Then, 10 IRE is used as the second black level.
[0062] Another example, assume that the first threshold is 10 db, the change value of the gain value of the automatic exposure algorithm is 15 db, the first black level is 10 IRE, and the fourth black level is 15 IRE. Then, 10 IRE is used as the second black level.
[0063] In the embodiments of the present application, when the change value of the gain value of the automatic exposure algorithm is greater than the first threshold, by using the black level of the optical black area of the first image as the second black level, it is possible to avoid excessive differences in the black levels between the video frames of the video, and ensure the brightness stability of each video frame of the video.
[0064] In some possible implementations of the embodiments of the present application, before step 103, the image processing method provided by the embodiments of the present application may further include: performing denoising processing on the first black level to obtain a fifth black level; correspondingly, step 103 may include: performing filtering and stabilization processing on the fifth black level based on the gain of the automatic exposure algorithm to obtain the second black level.
[0065] In some possible implementations of the embodiments of the present application, there may be black level noise in the OB area, and the existence of the black level noise will affect the accuracy of the black level in the visible light area. When performing denoising processing on the black level, the black level noise can be screened out by using the black level noise threshold, and then, filtering and stabilization processing is performed on the black level of the optical black area after screening out the black level noise. It can be understood that the black level of the optical black area after screening out the black level noise is the fifth black level in the embodiments of the present application.
[0066] In the embodiments of the present application, by performing denoising processing on the black level of the optical black area, the clarity of the image can be improved.
[0067] Step 104: Determine the third black level of the visible light region according to the second black level and the black level calibration value, where the black level calibration value is the difference between the black level of the pre-calibrated visible light region and the black level of the optical black region.
[0068] In some implementations of the embodiments of the present application, during imaging, since the image sensor is sensitive to light, the black level of the visible light region cannot be directly counted at this time. At this time, the black level of the optical black region can be counted, and then, according to the black level of the optical black region and the black level calibration value, the black level of the visible light region is determined.
[0069] In some possible implementations of the embodiments of the present application, step 104 may include: using the sum of the second black level and the black level calibration value as the third black level.
[0070] In some implementations of the embodiments of the present application, during imaging, since the image sensor is sensitive to light, the black level of the visible light region cannot be directly counted at this time. At this time, the black level of the optical black region can be counted. After the black level of the optical black region is counted, the black level of the optical black region is filtered and stabilized, and then, the sum of the black level obtained after the filtering and stabilization process and the black level calibration value is calculated to obtain the black level of the visible light region.
[0071] Exemplarily, assuming that the second black level is 10 IRE and the black level calibration value is 5 IRE, the black level of the visible light region included in the first image collected by the image sensor when the image sensor is sensitive to light is 15 IRE.
[0072] In some possible implementations of the embodiments of the present application, the visible light region of the image sensor can be pre-shaded, and then an image collected by the image sensor when it is not sensitive to light is obtained. The black level of the visible light region included in the image and the black level of the optical black region included in the image are counted, and the difference between the black level of the visible light region included in the image and the black level of the optical black region included in the image is used as the black level calibration value. Based on this, before step 104, the image processing method provided by the embodiments of the present application may further include: obtaining a third image collected by the image sensor when it is not sensitive to light; determining the sixth black level of the visible light region and the seventh black level of the optical black region included in the third image; using the difference between the sixth black level and the seventh black level as the black level calibration value.
[0073] In some possible implementations of the embodiments of the present application, the software configured in the black level statistics module for counting the black levels of the optical black region and the visible light region can be used to count the black levels of the visible light region and the optical black region included in the third image. When the black levels of the visible light region and the optical black region included in the third image are counted, the difference between the black level of the visible light region and the black level of the optical black region included in the third image is calculated, and this difference is used as the black level calibration value.
[0074] Exemplarily, assume that the black level of the visible light region included in the third image is 13 IRE, and the black level of the optical black region is 6 IRE. Then, the black level calibration value is 7 IRE.
[0075] In some possible implementations of the embodiments of the present application, there may be black level noise in the OB region and the visible light region. The existence of the black level noise will affect the accuracy of the black level calibration value. When calibrating the black level calibration value, the black level noise can be filtered out by using the black level noise threshold. Then, the difference between the black level of the visible light region with the black level noise filtered out and the black level of the optical black region with the black level noise filtered out is used as the black level calibration value.
[0076] In some possible implementations of the embodiments of the present application, the black level noise follows a Gaussian distribution, and the black level noise threshold can be determined through the noise distribution.
[0077] In some possible implementations of the embodiments of the present application, when statistically calculating the black level of the optical black region, the black levels of the optical black region in each channel can be statistically calculated. Herein, the channels in the embodiments of the present application include but are not limited to the red channel, the green channel, and the blue channel. Correspondingly, the black level calibration value in the embodiments of the present application can include the black level calibration values of each channel. After statistically calculating the black level of the optical black region in a certain channel, the black level of the visible light region in this channel can be determined according to the black level and the black level calibration value corresponding to this channel.
[0078] Step 105: Generate a third image based on the first image and the third black level.
[0079] In some possible implementations of the embodiments of the present application, when the black level of the visible light region is obtained, the black level of the visible light region can be configured in each image processing module in the image signal processor. Each module of the image signal processor processes the first image based on the black level of the visible light region to obtain a second image.
[0080] The image processing modules in the image signal processor in the embodiments of the present application include but are not limited to: a bad pixel correction module, an automatic white balance module, a demosaicing module, a color correction module, a color space conversion module, etc.
[0081] In some possible implementations of the embodiments of the present application, when the image processing module processes the first image based on the black level of the visible light region, it can first subtract the black level of the visible light region and then perform corresponding image processing.
[0082] In an embodiment of the present application, a first image collected by an image sensor during photosensing is obtained, where the first image includes a visible light region and an optical black region; a first black level of the optical black region is determined; the first black level is filtered and stabilized based on the gain of an automatic exposure algorithm to obtain a second black level; a third black level of the visible light region is determined according to the second black level and a black level calibration value, where the black level calibration value is the difference between the black level of the visible light region and the black level of the optical black region calibrated in advance; a second image is generated based on the first image and the third black level. The black level of the optical black region changes with temperature and brightness. When determining the black level of the visible light region, both brightness and temperature are considered. Therefore, the accuracy of the black level of the visible light region determined through the black level of the optical black region is relatively high, and thus the image quality can be improved.
[0083] In some possible implementations of the embodiment of the present application, since there is a delay in the gain value of the automatic exposure algorithm taking effect after being configured into the image sensor, at this time, the value of the black level of the optical black region can be stored in a linked list and transmitted to the image signal processor with a delay to ensure that the black level and the gain value of the automatic exposure algorithm take effect simultaneously.
[0084] Figure 3 is a timing diagram for configuring the black level of the visible light region provided by the embodiment of the present application. In Figure 3 it, the image signal processor statistics the black level of the optical black region and transmits the black level of the optical black region to the kernel state driven by the image signal processor; the user state driven by the image signal processor detects the gain value of the automatic exposure algorithm and transmits the detected gain value to the kernel state driven by the image signal processor; the kernel state driven by the image signal processor determines whether the gain value has changed significantly. If the gain value has not changed significantly, the black level of the optical black region is stabilized, and the black level of the visible light region is determined using the stabilized black level of the optical black region and the black level calibration value; the kernel state driven by the image signal processor transmits the black level of the visible light region to the image signal processor; the image signal processor configures the black level of the visible light region to each image processing module.
[0085] In the embodiment of the present application, by filtering and stabilizing the black level, the brightness of each frame of the video can be ensured to be stable, and the brightness jump of the video image can be prevented.
[0086] In some possible implementations of the embodiment of the present application, some image sensors have a black level correction and stabilization function. When this function is enabled, it affects the image sensor to output an image with an OB region, thereby affecting the determination of the black level of the visible light region. Based on this, before step 101, the image processing method provided by the embodiment of the present application further includes: turning off the black level correction and stabilization function when the image sensor has the black level correction and stabilization function.
[0087] In the embodiments of the present application, by turning off the black level correction and stabilization function of the image sensor, it is possible to ensure that the image sensor outputs an image with an OB area, and then the black level of the OB area can be statistically calculated.
[0088] In some possible implementations of the embodiments of the present application, when using the ISP to determine the black level of the OB area, the black level statistics module, the black level statistics range, the black level noise threshold, and the black level stabilization algorithm can be configured into the ISP chip. When using the DSP to determine the black level of the OB area, the black level statistics module, the black level statistics range, the black level noise threshold, and the black level stabilization algorithm can be configured into the DSP chip.
[0089] The overall process of the image processing method provided in the embodiments of the present application will be described below. As Figure 4 shown, Figure 4 is a schematic diagram of the overall process of the image processing method provided in the embodiments of the present application. The overall process of the image processing method includes the following steps:
[0090] Step 401: Calibrate the black level calibration value using a calibration tool;
[0091] When calibrating the black level calibration value using a calibration tool, the visible light area of the image sensor can be covered with black; obtain the third image collected by the image sensor when the image sensor is not sensitive to light; statistically calculate the black level of the visible light area included in the third image and the black level of the optical black area included in the third image; perform denoising processing on the black level of the visible light area included in the third image and the black level of the optical black area included in the third image, and use the difference between the black level of the visible light area obtained after denoising processing and the black level of the optical black area obtained after denoising processing as the black level calibration value.
[0092] Step 402: Generate a configuration file according to the black level calibration value;
[0093] The configuration file may include the black level calibration value, the black level statistics range parameter, the black level noise threshold parameter, the black level filtering and stabilization processing algorithm, etc.
[0094] Step 403: After the electronic device is powered on, import the configuration file and turn off the black level correction and stabilization function;
[0095] If the image sensor has the black level correction and stabilization function, turn off the black level correction and stabilization function, and then obtain the first image collected by the image sensor; if the image sensor does not have the black level correction and stabilization function, directly obtain the first image collected by the image sensor.
[0096] Step 404: After starting the camera, select the real-time black level statistics mode or default to this mode when the electronic device is powered on;
[0097] The camera may include multiple shooting modes, for example, portrait mode, night scene mode, real-time statistical black level mode, etc. Among them, in the real-time statistical black level mode, the electronic device will statistically calculate the black level of the optical black area of the image collected by the image sensor in real time, and then determine the black level of the visible light area based on the black level of the optical black area, and then perform image processing based on the black level of the visible light area.
[0098] Step 405: The image signal processor configures the width and height of the Mobile Industry Processor Interface (MIPI) and the video input channel to allow the optical black area of the first image collected by the image sensor to enter the image signal processor;
[0099] The width and height of the MIPI should be the same as the width and height of the first image output by the image sensor, otherwise the data stream of the first image cannot pass through the MIPI. When the image signal processor statistically calculates the black level of the optical black area, the width and height of the video input channel of the image signal processor need to be the same as the width and height of the MIPI. After statistically calculating the black level of the optical black area, the optical black area in the first image needs to be cropped, otherwise the optical black area will appear in the picture; when the digital signal processor statistically calculates the black level of the optical black area, after statistically calculating the black level of the optical black area, the optical black area needs to be cropped, and the width and height of the video input channel of the image signal processor are the same as the width and height of the image after cropping the optical black area after the digital signal processor statistically calculates the black level of the optical black area;
[0100] Step 406: According to the black level statistical range parameter, statistically calculate the black level of the optical black area of the image collected by the image sensor;
[0101] Step 407: Denoise the black level of the optical black area using the black level noise threshold parameter, and perform filtering and stabilization processing on the denoised black level using the black level filtering and stabilization processing algorithm;
[0102] Step 408: Add the black level of the optical black area obtained after filtering and stabilization processing to the black level calibration value to obtain the black level of the visible light area;
[0103] Step 409: Configure the black level of the visible light area in each image processing module in the image signal processor, and each module of the image signal processor processes the first image based on the black level of the visible light area to obtain a second image.
[0104] Figure 5 It is the first structural schematic diagram of the electronic device provided by the embodiment of the present application. The electronic device 500 may include: an image sensor 501 and an image signal processor 502; among them, the image signal processor 502 receives the image transmitted by the image sensor 501 through the MIPI interface through the video input channel;
[0105] The image sensor 501 includes an acquisition module 5011, and the acquisition module 5011 is used to acquire a first image. Herein, the first image is the image acquired when the image sensor is photosensitive, and the first image includes a visible light region and an optical black region;
[0106] The image signal processor 502 includes:
[0107] A first receiving module 5021, which is used to receive the first image sent by the image sensor;
[0108] A first determination module 5022, which is used to determine the first black level of the optical black region;
[0109] A filtering module 5023, which is used to perform filtering and stabilization processing on the first black level based on the gain of the automatic exposure algorithm to obtain a second black level;
[0110] A second determination module 5024, which is used to determine the third black level of the visible light region according to the second black level and the black level calibration value; wherein, the black level calibration value is the difference between the black level of the visible light region and the black level of the optical black region calibrated in advance;
[0111] A generation module 5025, which is used to generate a second image based on the first image and the third black level.
[0112] In some possible implementations of the embodiments of the present application, the filtering module 5023 includes:
[0113] A first filtering sub-module, which is used to use the average value of the first black level and the fourth black level as the second black level when the change value of the gain value of the automatic exposure algorithm is less than or equal to the first threshold, wherein the fourth black level is the black level of the optical black region in the previous frame image of the first image.
[0114] In some possible implementations of the embodiments of the present application, the filtering module 5023 includes:
[0115] A second filtering sub-module, which is used to use the first black level as the second black level when the change value of the gain value of the automatic exposure algorithm is greater than the first threshold.
[0116] In some possible implementations of the embodiments of the present application, the image signal processor 502 further includes:
[0117] A denoising module, which is used to perform denoising processing on the first black level to obtain a fifth black level;
[0118] Correspondingly, the filtering module 5023 is specifically used for:
[0119] Performing filtering and stabilization processing on the fifth black level based on the gain of the automatic exposure algorithm to obtain a second black level.
[0120] In some possible implementations of the embodiments of the present application, the second determination module 5024 is specifically configured to:
[0121] Use the sum of the second black level and the black level calibration value as the third black level.
[0122] In some possible implementations of the embodiments of the present application, the first receiving module 5021 is further configured to receive a third image sent by the image sensor; wherein, the third image is an image collected when the image sensor is not sensitive to light;
[0123] Correspondingly, the image signal processor 502 further includes:
[0124] A third determination module, configured to determine a sixth black level in the visible light region included in the third image and a seventh black level in the optical black region;
[0125] A fourth determination module, configured to use the difference between the sixth black level and the seventh black level as the black level calibration value.
[0126] In some possible implementations of the embodiments of the present application, the electronic device 500 further includes: a digital signal processor 503. As Figure 6 shown, Figure 6 is the second structural schematic diagram of the electronic device provided by the embodiments of the present application. In Figure 6 it, the digital signal processor 503 receives the image transmitted by the image sensor 501 through the MIPI interface.
[0127] The digital signal processor 503 includes:
[0128] A second receiving module 5031, configured to receive a first image sent by the image sensor 501;
[0129] A fifth determination module 5032, configured to determine the first black level in the optical black region;
[0130] A removal module 5033, configured to remove the optical black region in the first image to obtain a fourth image;
[0131] A sending module 5034, configured to send the fourth image and the first black level to the image signal processor;
[0132] Correspondingly, the first receiving module 5021 is further configured to receive the fourth image and the first black level sent by the digital signal processor 503;
[0133] A generation module 5025, further configured to generate a second image based on the fourth image and the third black level.
[0134] The electronic device in the embodiments of the present application can be a terminal or other devices other than terminals. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a handheld computer, an in-vehicle electronic device, a Mobile Internet Device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. It can also be a server, a Network Attached Storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, etc. The embodiments of the present application do not make specific limitations.
[0135] The electronic device in the embodiments of the present application can be an electronic device with an operating system. The operating system can be the Android operating system, the iOS operating system, or other possible operating systems. The embodiments of the present application do not make specific limitations.
[0136] The electronic device provided by the embodiments of the present application can implement Figures 1 to 4 each process implemented by the embodiment of the image processing method. To avoid repetition, it will not be elaborated here.
[0137] Optionally, as Figure 7 shown, the embodiments of the present application further provide an electronic device 700, including a processor 701 and a memory 702. A program or instruction that can run on the processor 701 is stored on the memory 702. When the program or instruction is executed by the processor 701, it implements each step of the image processing method provided by the embodiments of the present application and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.
[0138] Figure 8 is a schematic diagram of the hardware structure of the electronic device implementing the embodiments of the present application.
[0139] The electronic device 800 includes but is not limited to: a radio frequency unit 801, a network module 802, an audio output unit 803, an input unit 804, a sensor 805, a display unit 806, a user input unit 807, an interface unit 808, a memory 809, and a processor 810, etc.
[0140] Those skilled in the art can understand that the electronic device 800 may further include a power source (such as a battery) for powering each component. The power source can be logically connected to the processor 810 through a power management system, so as to manage functions such as charging, discharging, and power consumption management through the power management system. Figure 8 The structure of the electronic device shown in Figure 8 does not limit the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements, which will not be elaborated here.
[0141] Among them, the processor 810 is used to: obtain a first image collected by the image sensor when the image sensor senses light, where the first image includes a visible light region and an optical black region; determine a first black level of the optical black region; perform a filtering and stabilizing process on the first black level based on the gain of the automatic exposure algorithm to obtain a second black level; determine a third black level of the visible light region according to the second black level and the black level calibration value, where the black level calibration value is the difference between the black level of the visible light region and the black level of the optical black region pre-calibrated; generate a second image based on the first image and the third black level.
[0142] In some possible implementations of the embodiments of the present application, the processor 810 is specifically used to:
[0143] When the change value of the gain value of the automatic exposure algorithm is less than or equal to a first threshold, use the average value of the first black level and the fourth black level as the second black level, where the fourth black level is the black level of the optical black region in the previous frame image of the first image.
[0144] In some possible implementations of the embodiments of the present application, the processor 810 is specifically used to:
[0145] When the change value of the gain value of the automatic exposure algorithm is greater than the first threshold, use the first black level as the second black level.
[0146] In some possible implementations of the embodiments of the present application, the processor 810 is further used to:
[0147] Perform a denoising process on the first black level to obtain a fifth black level;
[0148] Perform a filtering and stabilizing process on the fifth black level based on the gain of the automatic exposure algorithm to obtain a second black level.
[0149] In some possible implementations of the embodiments of the present application, the processor 810 is specifically used to:
[0150] Use the sum of the second black level and the black level calibration value as the third black level.
[0151] In some possible implementations of the embodiments of the present application, the processor 810 is further used to:
[0152] When the image sensor has the black level correction stabilization function, turn off the black level correction stabilization function.
[0153] In some possible implementations of the embodiments of the present application, the processor 810 is further configured to:
[0154] Obtain a third image captured by the image sensor when the image sensor is not sensitive to light;
[0155] Determine a sixth black level in the visible light region included in the third image and a seventh black level in the optical black region;
[0156] Use the difference between the sixth black level and the seventh black level as the black level calibration value.
[0157] It should be understood that in the embodiments of the present application, the input unit 804 may include a Graphics Processing Unit (GPU) 8041 and a microphone 8042. The graphics processor 8041 processes the image data of static pictures or videos obtained by an image capturing device (such as a camera) in a video capture mode or an image capture mode. The display unit 806 may include a display panel 8061, and the display panel 8061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 807 includes at least one of a touch panel 8071 and other input devices 8072. The touch panel 8071 is also referred to as a touch screen. The touch panel 8071 may include two parts: a touch detection device and a touch controller. The other input devices 8072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be elaborated here.
[0158] The memory 809 can be used to store software programs and various data. The memory 809 mainly includes a first storage area for storing programs or instructions and a second storage area for storing data. Among them, the first storage area can store an operating system, application programs or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 809 can include a volatile memory or a non-volatile memory, or the memory 809 can include both a volatile memory and a non-volatile memory. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDR SDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synch link dynamic random access memory (SLDRAM), and a direct rambus random access memory (DRRAM). The memory 809 in the embodiments of the present application includes, but is not limited to, these and any other suitable types of memories.
[0159] The processor 810 can include one or more processing units; optionally, the processor 810 integrates an application processor and a modem processor. Among them, the application processor mainly processes operations related to the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above modem processor may not be integrated into the processor 810 either.
[0160] The embodiments of the present application also provide a readable storage medium. A program or instructions are stored on the readable storage medium. When the program or instructions are executed by a processor, each process of the image processing method embodiment provided by the embodiments of the present application is implemented, and the same technical effects can be achieved. To avoid repetition, it will not be elaborated here.
[0161] Among them, the processor is the processor in the electronic device in the above-mentioned embodiment. The readable storage medium includes a computer-readable storage medium. Examples of the computer-readable storage medium include non-transitory computer-readable media, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc, etc.
[0162] An embodiment of the present application further provides a chip. The chip includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is configured to run programs or instructions to implement each process of the image processing method embodiment provided by the embodiment of the present application, and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.
[0163] It should be understood that the chip mentioned in the embodiment of the present application may also be referred to as a system-on-chip, a system chip, a chip system, or a system-on-chip, etc.
[0164] An embodiment of the present application further provides a computer program product. The computer program product is stored in a storage medium. The computer program product is executed by at least one processor to implement each process of the image processing method embodiment provided by the embodiment of the present application, and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.
[0165] It should be noted that in this article, the term "comprising", "including" or any other variant thereof is intended to cover a non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising a..." does not exclude the existence of additional identical elements in the process, method, article or device comprising such element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in a reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
[0166] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-described embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, in essence or the part that contributes to the prior art, can be embodied in the form of a computer software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions for causing a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of the present application.
[0167] The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present application, those of ordinary skill in the art can also make many forms without departing from the purpose of the present application and the scope protected by the claims, and all of them belong to the protection scope of the present application.
Claims
1. An image processing method, characterized in that: The method comprises: Acquire a first image captured by the image sensor when the image sensor is sensitive to light, wherein the first image includes a visible light area and an optical black area; determining a first black level of the optical black area; Performing filtering and stabilization processing on the first black level based on the automatic exposure algorithm gain to obtain a second black level; Determine a third black level in the visible light region according to the second black level and the black level calibration value; wherein the black level calibration value is the difference between the pre-calibrated black level in the visible light region and the black level in the optical black region; Based on the first image and the third black level, a second image is generated.
2. The method according to claim 1, characterized in that The filtering and stabilizing the first black level based on the automatic exposure algorithm gain to obtain the second black level includes: When the change value of the gain value of the automatic exposure algorithm is less than or equal to the first threshold, the average value of the first black level and the fourth black level is used as the second black level, wherein the fourth black level is the black level of the optical black area in the previous frame image of the first image.
3. The method according to claim 1, characterized in that The filtering and stabilizing the first black level based on the automatic exposure algorithm gain to obtain the second black level includes: When the change value of the gain value of the automatic exposure algorithm is greater than a first threshold, the first black level is used as the second black level.
4. The method according to claim 1, characterized in that Before performing filtering and stabilizing processing on the first black level based on the automatic exposure algorithm gain to obtain the second black level, the method further includes: performing denoising processing on the first black level to obtain a fifth black level; The filtering and stabilizing the first black level based on the automatic exposure algorithm gain to obtain the second black level includes: The fifth black level is subjected to filtering and stabilization processing based on the automatic exposure algorithm gain to obtain the second black level.
5. The method according to claim 1, characterized in that Determining the third black level in the visible light region according to the second black level and the black level calibration value includes: The sum of the second black level and the black level calibration value is used as the third black level.
6. The method according to any one of claims 1 to 5, characterized in that: Before acquiring the first image captured by the image sensor when the image sensor is sensitive to light, the method further includes: In the case where the image sensor has a black level correction and stabilization function, the black level correction and stabilization function is turned off.
7. The method according to any one of claims 1 to 5, characterized in that: Before acquiring the first image captured by the image sensor when the image sensor is sensitive to light, the method further includes: Acquire a third image captured by the image sensor when the image sensor is not sensitive to light; determining a sixth black level of a visible light region and a seventh black level of an optical black region included in the third image; The difference between the sixth black level and the seventh black level is used as the black level calibration value.
8. An electronic device, characterized in that: The electronic device includes an image sensor and an image signal processor; The image sensor includes a collection module, and the collection module is used to collect a first image, wherein the first image is an image collected when the image sensor is sensitive to light, and the first image includes a visible light area and an optical black area; The image signal processor comprises: A first receiving module, used for receiving a first image sent by the image sensor; A first determining module, used to determine a first black level of the optical black area; A filtering module, configured to perform filtering and stabilization processing on the first black level based on the gain of an automatic exposure algorithm to obtain a second black level; A second determination module, configured to determine a third black level of the visible light region according to the second black level and a black level calibration value; wherein the black level calibration value is a difference between a pre-calibrated black level of the visible light region and a black level of the optical black region; A generating module is used to generate a second image based on the first image and the third black level.
9. The electronic device according to claim 8, characterized in that: The filtering module comprises: The first filtering submodule is used to use the average value of the first black level and the fourth black level as the second black level when the change value of the gain value of the automatic exposure algorithm is less than or equal to the first threshold, wherein the fourth black level is the black level of the optical black area in the previous frame image of the first image.
10. The electronic device according to claim 8, characterized in that: The filtering module comprises: The second filtering submodule is configured to use the first black level as the second black level when a change value of a gain value of an automatic exposure algorithm is greater than a first threshold.
11. The electronic device according to claim 8, characterized in that: The image signal processor further includes: a denoising module, configured to perform denoising on the first black level to obtain a fifth black level; The filtering module is specifically used for: The fifth black level is subjected to filtering and stabilization processing based on the automatic exposure algorithm gain to obtain the second black level.
12. The electronic device according to claim 8, characterized in that: The second determining module is specifically used for: The sum of the second black level and the black level calibration value is determined as the third black level.
13. The electronic device according to any one of claims 8 to 12, characterized in that: The first receiving module is further used to receive a third image sent by the image sensor; wherein the third image is an image collected when the image sensor is not sensitive to light; The image signal processor further comprises: A third determining module, configured to determine a sixth black level of a visible light region and a seventh black level of an optical black region included in the third image; The fourth determining module is used to determine the difference between the sixth black level and the seventh black level as the black level calibration value.
14. The electronic device according to claim 8, characterized in that: The electronic device further comprises: a digital signal processor; The digital signal processor comprises: A second receiving module, configured to receive the first image sent by the image sensor; A fifth determining module, configured to determine a first black level of the optical black area; A removal module, used for removing the optical black area in the first image to obtain a fourth image; A sending module, configured to send the fourth image and the first black level to the image signal processor; The first receiving module is further used to receive the fourth image and the first black level sent by the digital signal processor; The generating module is further configured to generate the second image based on the fourth image and the third black level.
15. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image processing method according to any one of claims 1 to 7 are implemented.
16. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the image processing method according to any one of claims 1 to 7 are implemented.