Image processing method, imaging apparatus, and device

Abstract

The application provides an image processing method, an imaging device and equipment. The method comprises the following steps: acquiring a first image signal and a second image signal based on an exposure control parameter; outputting the first image signal when high conversion gain is used, and outputting the second image signal when low conversion gain is used; synthesizing the first image signal and the second image signal based on a processing control parameter to obtain a synthesized image, performing gain processing and data compression processing on the synthesized image to obtain a to-be-output image, and outputting the to-be-output image; obtaining a target exposure control parameter and a target processing control parameter based on target statistical information; wherein the target exposure control parameter is used for controlling the acquisition of a preset number of frames of images after a current frame, and the target processing control parameter is used for controlling the processing of the preset number of frames of images after the current frame. Through the technical scheme, the dynamic range of the image is improved, the image with a higher signal-to-noise ratio is obtained, the better noise reduction effect is realized, and the image information quantity is improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an image processing method, imaging device and equipment. Background Technology

[0002] After an image is captured by an image acquisition device (such as a camera), it can usually be processed to improve its quality. For example, image processing can include gain adjustment, noise reduction, black level adjustment, white balance adjustment, deblurring, and sharpening. Gain adjustment increases image brightness. Noise reduction reduces image noise. Black level adjustment preserves details in dark areas. White balance adjustment controls image tone. Deblurring corrects blurred areas. Sharpening compensates for image contours, enhances edges and grayscale transitions, and improves image clarity.

[0003] Although image processing can be performed on images, such as gain adjustment, noise reduction, black level adjustment, white balance adjustment, deblurring, and sharpening, problems such as poor image quality still exist. Summary of the Invention

[0004] This application provides an image processing method, the method comprising:

[0005] A first image signal and a second image signal are acquired based on exposure control parameters; wherein the exposure control parameters are determined based on target statistical information corresponding to a preset number of frames of images acquired before the current frame; wherein the image sensor supports high conversion gain and low conversion gain, outputting a first image signal when using the high conversion gain and outputting a second image signal when using the low conversion gain;

[0006] Based on processing control parameters, the first image signal and the second image signal are synthesized to obtain a synthesized image. The synthesized image is then subjected to gain processing and data compression processing to obtain an output image, which is then output. During gain processing, data truncation is not performed, or the data is truncated to a value range not less than the input image's value range. The compression parameters used during data compression processing are related to the gain parameters used during gain processing.

[0007] Target exposure control parameters and target processing control parameters are obtained based on target statistical information; wherein, the target exposure control parameters are used to control the acquisition of a preset number of frames after the current frame, and the target processing control parameters are used to control the processing of a preset number of frames after the current frame.

[0008] This application provides an imaging device, the imaging device comprising:

[0009] An image acquisition unit is used to acquire a first image signal and a second image signal based on exposure control parameters; wherein, the exposure control parameters are determined based on target statistical information corresponding to a preset number of frames of images acquired before the current frame; the image acquisition unit supports high conversion gain and low conversion gain, outputting the first image signal when using high conversion gain and outputting the second image signal when using low conversion gain;

[0010] The first image signal and the second image signal are output to the image processing unit;

[0011] An image processing unit is configured to synthesize a composite image from the first image signal and the second image signal based on processing control parameters, perform gain processing and data compression processing on the composite image to obtain an output image, and output the output image; wherein, during gain processing, data truncation is not performed, or the data is truncated to a value range not less than the range of values ​​of the input image; wherein, the compression parameters used during data compression processing are related to the gain parameters used during gain processing;

[0012] Statistical unit, used to obtain target statistical information;

[0013] An exposure control unit is used to acquire target exposure control parameters and target processing control parameters based on the target statistical information; wherein, the target exposure control parameters are input to the image acquisition unit to control the acquisition of a preset number of frames after the current frame; and the target processing control parameters are input to the image processing unit to control the processing of a preset number of frames after the current frame.

[0014] As can be seen from the above technical solutions, in the embodiments of this application, a first image signal and a second image signal are acquired. The first image signal is a high conversion gain image signal, and the second image signal is a low conversion gain image signal. An output image is then generated based on the first and second image signals, resulting in a better image quality. By acquiring two frames of image signals with different brightness levels, the dynamic range can be expanded by a factor greater than that of a single frame, eliminating motion synthesis artifacts, achieving a higher signal-to-noise ratio, and realizing a higher dynamic range. Acquiring two frames of image signals with different brightness levels significantly improves the dynamic range of the image, resulting in an image with a higher signal-to-noise ratio, achieving better noise reduction, preserving information in bright areas, and increasing the amount of image information. Because this invention does not truncate the bit width during gain processing of the image signal, or truncates it to a value greater than the brightness range of the image signal, and then normalizes the images with different value ranges through data compression, it can avoid or reduce the loss of brightness information when performing brightness compensation on the image, preserving more brightness information, improving the image processing effect, and enhancing the visual effect of the processed image. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments of this application or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings of the embodiments of this application.

[0016] Figure 1 This is a schematic diagram of the imaging device in one embodiment of this application;

[0017] Figure 2A and Figure 2B This is a schematic diagram of the imaging device in one embodiment of this application;

[0018] Figure 3A This is a schematic diagram of the structure of an image acquisition unit in one embodiment of this application;

[0019] Figure 3B This is a schematic diagram of the filter transmittance curve in one embodiment of this application;

[0020] Figure 3C This is a schematic diagram of the image sensor arrangement in one embodiment of this application;

[0021] Figure 3D This is a schematic diagram of the arrangement of RGBW sensors in one embodiment of this application;

[0022] Figure 3E This is a schematic diagram of the spectral response curve in one embodiment of this application;

[0023] Figure 3F This is a schematic diagram of an image sensor with DCG function according to one embodiment of this application;

[0024] Figures 4A-4G This is a schematic diagram of the structure of an image processing unit in one embodiment of this application;

[0025] Figure 5 This is a schematic diagram of the structure of the exposure control unit in one embodiment of this application;

[0026] Figure 6 This is a flowchart illustrating an image processing method according to one embodiment of this application;

[0027] Figure 7 This is a hardware structure diagram of an electronic device according to one embodiment of this application. Detailed Implementation

[0028] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “the,” and “the” as used in this application and claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to any and all possible combinations comprising one or more of the associated listed items.

[0029] It should be understood that although the terms first, second, third, etc., may be used to describe various information in embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" may also be interpreted as "when," "when," or "in response to a determination."

[0030] This application provides an imaging device in its embodiments, see [link to relevant documentation] Figure 1 The diagram shown is a schematic representation of the imaging device. This imaging device may include, but is not limited to, an image acquisition unit (which may be an image sensor), an image processing unit, a statistical unit, and an exposure control unit.

[0031] An image acquisition unit is used to acquire a first image signal and a second image signal based on exposure control parameters. The exposure control parameters are determined based on target statistical information corresponding to a preset number of frames acquired before the current frame. The image acquisition unit (i.e., the image sensor) can support both high and low conversion gain. It outputs the first image signal when the image sensor uses high conversion gain and outputs the second image signal when the image sensor uses low conversion gain. Since the first image signal is under high conversion gain and the second image signal is under low conversion gain, the brightness of the first image signal is higher than that of the second image signal. For example, this embodiment involves only one image sensor, and the first and second image signals are image signals acquired by the same image sensor.

[0032] The image acquisition unit is also used to output the first image signal and the second image signal to the image processing unit. For example, the image acquisition unit is used to output the first image signal and the second image signal to the image processing unit if the scene features of the imaging device meet the constraints. Alternatively, the image acquisition unit is also used to synthesize the first image signal and the second image signal into a third image signal based on processing control parameters if the scene features of the imaging device do not meet the constraints, perform nonlinear compression on the third image signal, and output the compressed image signal to the image processing unit. The processing control parameters are determined based on target statistical information corresponding to a preset number of frames of images acquired before the current frame.

[0033] The scene feature may include, but is not limited to, the available bandwidth between the image acquisition unit and the image processing unit. If the available bandwidth is greater than a preset bandwidth threshold, the scene feature is determined to meet the constraint conditions; if the available bandwidth is less than or equal to the preset bandwidth threshold, the scene feature is determined not to meet the constraint conditions.

[0034] The image processing unit, upon receiving a first image signal and a second image signal, synthesizes the first image signal and the second image signal based on processing control parameters (determined based on target statistical information corresponding to a preset number of frames of images acquired before the current frame) to obtain a synthesized image. It then performs gain processing and data compression processing on the synthesized image to obtain an output image, and outputs the output image; that is, the image processing unit can directly output the output image. Specifically, during gain processing, data truncation is not performed, or the data is truncated to a value range not less than the input image's value range; the compression parameters used during data compression processing are related to the gain parameters used during gain processing.

[0035] The image processing unit is also used to generate an output image based on the compressed image signal (i.e., the third image signal) upon receiving it, without performing a synthesis operation. For example, the compressed image signal can be subjected to gain processing and data compression processing to obtain the output image; the processing procedure is similar to that of synthesizing an image. Specifically, during gain processing, data truncation is not performed, or it is truncated to a value range not smaller than the input image's value range; the compression parameters used during data compression processing are related to the gain parameters used during gain processing. For example, a decompression process can be added before gain processing and data compression processing to restore the compressed (non-linear) third image signal to a linear signal before performing gain processing and data compression processing.

[0036] The statistical unit is used to acquire target statistical information. For example, the target statistical information includes dynamic range expansion values ​​obtained based on a first statistical image, which can be an image before or after synthesis processing, an image before gain processing, an image after gain processing, an image before data compression processing, or an image after data compression processing. Furthermore, the target statistical information includes statistical brightness values ​​obtained based on a second statistical image, which can also be an image before or after synthesis processing, an image before or after gain processing, an image before data compression processing, or an image after data compression processing. In summary, the statistical unit can obtain dynamic range expansion values ​​based on the first statistical image. For example, it can perform dynamic range statistics on the first statistical image to obtain the dynamic range expansion value, or it can calculate the dynamic range expansion value after preprocessing the first statistical image, as long as the dynamic range expansion value is obtained based on the first statistical image. Similarly, the statistical unit can obtain statistical brightness values ​​based on the second statistical image. For example, it can perform brightness statistics on the second statistical image to obtain the statistical brightness value, or it can calculate the statistical brightness value after preprocessing the second statistical image, as long as the statistical brightness value is obtained based on the second statistical image.

[0037] An exposure control unit is used to acquire target exposure control parameters and target processing control parameters based on target statistical information; wherein, the target exposure control parameters are input to the image acquisition unit to control the acquisition of a preset number of frames after the current frame; and the target processing control parameters are input to the image processing unit to control the processing of a preset number of frames after the current frame.

[0038] As can be seen from the above technical solutions, in this embodiment, a first image signal and a second image signal are acquired. The first image signal is a high conversion gain image signal, and the second image signal is a low conversion gain image signal. An output image is then generated based on the first and second image signals, resulting in a better image quality. By acquiring two frames of image signals with different brightness levels, the dynamic range can be expanded by a factor greater than that of a single frame, eliminating motion synthesis artifacts, achieving a higher signal-to-noise ratio, and realizing a higher dynamic range. Acquiring two frames of image signals with different brightness levels can significantly improve the dynamic range of the image, obtain an image with a higher signal-to-noise ratio, achieve better noise reduction, retain information in bright areas, and increase the amount of image information.

[0039] The imaging device of this application embodiment will be described below in conjunction with specific application scenarios.

[0040] See Figure 2A and Figure 2B The diagram shown is a schematic of the imaging device, which may include an image acquisition unit, an image processing unit, a statistical unit, an exposure control unit, and a buffer unit.

[0041] See Figure 2A As shown, the image acquisition unit receives light signals and exposure control parameters as inputs, and outputs a first image signal and a second image signal. The image processing unit receives the first image signal, the second image signal, and processing control parameters as inputs, and outputs the image to be output, a first statistical image, and a second statistical image. The statistics unit receives the first and second statistical images as inputs, and outputs statistical information. The exposure control unit receives statistical information as input, and outputs exposure control parameters and processing control parameters. The buffer unit receives exposure control parameters and processing control parameters as inputs, and outputs exposure control parameters and processing control parameters.

[0042] Figure 2B In this system, the image acquisition unit receives light signals and exposure control parameters as input, and outputs a third image signal. The image processing unit receives the third image signal and processing control parameters as input, and outputs the image to be output, a first statistical image, and a second statistical image. The statistics unit receives the first and second statistical images as input, and outputs statistical information. The exposure control unit receives statistical information as input, and outputs exposure control parameters and processing control parameters. The buffer unit receives exposure control parameters and processing control parameters as input, and outputs exposure control parameters and processing control parameters.

[0043] The following is a detailed description of the functions of each unit of the imaging device.

[0044] First, the image acquisition unit.

[0045] See Figure 3A The diagram shows the structure of an image acquisition unit, which may include a lens, a filter assembly, and an image sensor. Ambient light passes through the lens and becomes incident light, which then passes through the filter assembly and becomes filtered light. The image sensor acquires the filtered light (i.e., the optical signal), and based on the optical signal, it can acquire a first image signal and a second image signal.

[0046] For example, the image acquisition unit may include a lens, which is a lens module composed of multiple lenses, capable of receiving light and focusing it into an image, and the amount of light entering can be controlled by adjusting the aperture.

[0047] For example, the image acquisition unit may include a filter assembly (i.e., a filter module) capable of controlling the spectral range received by the image sensor. For instance, the filter assembly can be switched to allow both visible and infrared light to pass through, allowing the image sensor to receive both visible and infrared light. Alternatively, the filter assembly can be switched to allow only visible light to pass through, allowing the image sensor to receive only visible light.

[0048] For example, a light filter is a filter module that allows only specific wavelengths of light to pass through while blocking other wavelengths. The light filter can be controlled through coating techniques to allow light to enter the spectral range of the image sensor. See also Figure 3B As shown, the filter used in the light filtering component can pass through the visible light band of 340nm to 650nm and the near-infrared light band of 930nm to 970nm, while filtering out light of other bands. Of course, the light filtering component can also be switched to allow only visible light to pass through, blocking infrared light greater than 650nm while allowing visible light less than or equal to 650nm to pass through, without limiting the filtering range.

[0049] For example, the image acquisition unit may include an image sensor that can acquire light signals and, through DCG (Dual Coversion Gain) technology, output a first image signal and a second image signal within the same exposure time, thereby converting the light signal into an image signal.

[0050] See Figure 3C The diagram shows a schematic of image sensor arrangement. Image sensors can be RGBW sensors, Bayer sensors, or RGB-IR sensors. For the pixel arrangement of these three types of sensors, please refer to [link / reference needed]. Figure 3C As shown, other sensor arrangements can also be used. In summary, each pixel in an image sensor can sense at least one of the visible light sources: red, green, and blue.

[0051] Taking an image sensor using an RGBW sensor as an example, the arrangement of RGBW sensors can be found in [reference needed]. Figure 3D As shown, the spectral response curve of the RGBW sensor can be found in [reference needed]. Figure 3E As shown.

[0052] For example, see Figure 3F The diagram shows an image sensor with DCG functionality. The DCG function is achieved by adding a DCG switch to the pixel circuit of the image sensor.

[0053] To achieve DCG functionality, the exposure control parameters of the image acquisition unit may include aperture, exposure time, first conversion gain corresponding to the first image signal, first sensor gain corresponding to the first image signal, second conversion gain corresponding to the second image signal, and second sensor gain corresponding to the second image signal.

[0054] First, the image sensor acquires the light signal corresponding to the aperture (i.e., the current aperture, which will be updated in subsequent processes, and the latest aperture can be used each time) and the exposure time (i.e., the current exposure time, which will be updated in subsequent processes, and the latest exposure time can be used each time).

[0055] Then, after obtaining the optical signal (such as optical signal A), the dual conversion gain switch (i.e., DCG switch) can be turned off. When the DCG switch is off, the first image signal corresponding to optical signal A is determined based on the first conversion gain and the first sensor gain. For example, the image sensor can convert optical signal A into an electrical signal and then convert the electrical signal into an image signal. In this way, the brightness of the acquired image can be controlled by adjusting the first conversion gain and the first sensor gain, and the first image signal can be output.

[0056] Then, after the first image signal acquisition is complete, the dual conversion gain switch (DCG switch) can be closed. When the DCG switch is closed, the second image signal corresponding to the optical signal A is determined based on the second conversion gain and the second sensor gain. For example, the image sensor can convert the optical signal A into an electrical signal and then convert the electrical signal into an image signal. In this way, the brightness of the acquired image can be controlled by adjusting the second conversion gain and the second sensor gain, and the second image signal can be output.

[0057] For example, after obtaining optical signal A, the DCG switch can be controlled to close (i.e., close first and then open). When the DCG switch is closed, the second image signal corresponding to optical signal A is determined based on the second conversion gain and the second sensor gain. After the second image signal is acquired, the DCG switch is controlled to open, and the first image signal corresponding to optical signal A is determined based on the first conversion gain and the first sensor gain.

[0058] In summary, the effects of the first and second image signals are controlled by exposure control parameters, which may include, but are not limited to, aperture, exposure time, first conversion gain, first sensor gain, second conversion gain, and second sensor gain. Clearly, the first and second image signals are output image signals under the same exposure time and aperture. Alternatively, the first and second image signals can be output image signals under different conversion gains (the first image signal corresponds to the first conversion gain, and the second image signal corresponds to the second conversion gain) and different sensor gains (the first image signal corresponds to the first sensor gain, and the second image signal corresponds to the second sensor gain).

[0059] In the imaging process, the image sensor first acquires a light signal, then converts the light signal into an electrical signal, then into a voltage signal, and finally into a digital signal (i.e., an image signal). Based on this, the gain in the electrical-to-voltage conversion process can be called the conversion gain. The gain when the DCG switch is open is the first conversion gain corresponding to the first image signal, and the gain when the DCG switch is closed is the second conversion gain corresponding to the second image signal. Similarly, the gain in the voltage signal amplification process can be called the analog gain. The gain when the DCG switch is open is the first analog gain corresponding to the first image signal, and the gain when the DCG switch is closed is the second analog gain corresponding to the second image signal. Likewise, the gain in the digital signal amplification process can be called the digital gain. The gain when the DCG switch is open is the first digital gain corresponding to the first image signal, and the gain when the DCG switch is closed is the second digital gain corresponding to the second image signal. The first sensor gain can be determined based on the first analog gain and the first digital gain. The first sensor gain can be the first analog gain, or it can be the first digital gain, or it can be the product of the first analog gain and the first digital gain. The second sensor gain can be determined based on the second analog gain and the second digital gain. The second sensor gain can be the second analog gain, or the second sensor gain can be the second digital gain, or the second sensor gain can be the product of the second analog gain and the second digital gain.

[0060] The second conversion gain can be the same as or different from the first conversion gain. For example, the second conversion gain corresponding to the second image signal may be different from the first conversion gain corresponding to the first image signal. Conversion gain is a technical specification of the DCG function and has a higher signal-to-noise ratio than sensor gain. The first conversion gain is greater than the second conversion gain, that is, the first conversion gain is a high conversion gain, and the second conversion gain is a low conversion gain.

[0061] The gain of the second sensor can be the same as or different from that of the first sensor. For example, the gain of the second sensor corresponding to the second image signal is different from that of the first sensor corresponding to the first image signal.

[0062] See Figure 3FAs shown, when the DCG switch is open, the total capacitance in the pixel circuit of the image sensor decreases, allowing the same number of electrons to generate a larger voltage. This results in a brighter image with a higher signal-to-noise ratio. Therefore, the brighter first image signal can be acquired by adjusting the first conversion gain and the first sensor gain. Conversely, when the DCG switch is closed, the total capacitance in the pixel circuit of the image sensor increases, allowing the same number of electrons to generate a smaller voltage. This results in a darker image with a lower signal-to-noise ratio. Therefore, the darker second image signal can be acquired by adjusting the second conversion gain and the second sensor gain.

[0063] In summary, by switching the DCG switch on and off, two image signals with different brightness levels can be obtained, with the first image signal being brighter than the second. Specifically, when the DCG switch is off, the image acquisition unit achieves a higher conversion gain (high conversion gain), and the acquired image signal can be the first image signal. When the DCG switch is on, the image acquisition unit achieves a lower conversion gain (low conversion gain), and the acquired image signal can be the second image signal.

[0064] For example, after obtaining the first image signal and the second image signal, the image acquisition unit can output the first image signal and the second image signal to the image processing unit. Alternatively,

[0065] The image acquisition unit can synthesize the first image signal and the second image signal to obtain a third image signal, and output the third image signal to the image processing unit. Alternatively, the image acquisition unit can synthesize the first image signal and the second image signal to obtain a third image signal, perform nonlinear compression on the third image signal, and output the compressed image signal to the image processing unit.

[0066] For example, if the scene characteristics of the imaging device meet the constraints, the image acquisition unit can output the first image signal and the second image signal to the image processing unit. Or,

[0067] If the scene features of the imaging device do not meet the constraints, the image acquisition unit can synthesize the first image signal and the second image signal based on the processing control parameters to obtain a third image signal, and perform nonlinear compression on the third image signal, and output the compressed image signal to the image processing unit.

[0068] The scene feature may include, but is not limited to, the available bandwidth between the image acquisition unit and the image processing unit. Of course, the available bandwidth is just one example of the scene feature. There are no restrictions on this scene feature, nor are there any restrictions on the constraints under different scene features. The configuration can be made based on experience.

[0069] Taking available bandwidth as an example, if the available bandwidth is greater than a preset bandwidth threshold, it can be determined that the scene feature meets the constraints, and the first and second image signals are directly output to the image processing unit. If the available bandwidth is less than or equal to the preset bandwidth threshold, it can be determined that the scene feature does not meet the constraints, and the compressed image signal needs to be output to the image processing unit.

[0070] The method by which the image acquisition unit synthesizes the first and second image signals can be found in the synthesis method of the subsequent image processing unit, and will not be repeated here. When the image acquisition unit performs nonlinear compression on the third image signal, it can reduce the value range of the image signal, that is, obtain an image signal with a lower value range through nonlinear compression. Correspondingly, the value range is restored in the image processing unit.

[0071] For example, when the image sensor is in build-in mode, the third image signal can be a build-in high dynamic range image signal. Here, "build-in mode" refers to a wide dynamic range data acquisition technique where the image sensor acquires multiple frames, synthesizes them to obtain a high dynamic range image, and then compresses the high dynamic range image into a low dynamic range image using nonlinear mapping before outputting it. For instance, the image sensor is in build-in mode when the scene characteristics of the imaging device do not meet the constraints.

[0072] Second, the image processing unit.

[0073] An image processing unit is a logic platform that contains signal processing algorithms or programs. This logic platform can be a computer based on x86 or ARM architecture, or it can be an FPGA (Field-Programmable Gate Array) logic circuit. There are no restrictions on the implementation of this image processing unit.

[0074] When the image processing unit receives the first image signal and the second image signal, it can synthesize the first image signal and the second image signal based on processing control parameters (the processing control parameters are determined by the target statistical information corresponding to the preset number of frames of images acquired before the current frame) to obtain a synthesized image. Based on the synthesized image, it can generate an output image, for example, by performing gain processing and data compression processing on the synthesized image. Alternatively, when the image processing unit receives the compressed image signal (i.e., the third image signal), it can generate an output image based on the third image signal, for example, by performing gain processing and data compression processing on the third image signal. For example, if the third image signal is a non-linearly compressed image, a decompression process can be added to restore the value range of the third image signal. In subsequent processes, we will take the case where the third image signal is non-linearly compressed as an example.

[0075] For example, see Figure 4A The diagram shows the structure of an image processing unit, which may include a synthesis processing module, a gain processing module, and a data compression module. The synthesis processing module synthesizes the first image signal and the second image signal to obtain a synthesized image. The gain processing module performs gain processing on the synthesized image to obtain a gain-processed image. The data compression module performs data compression processing on the gain-processed image to obtain the image to be output.

[0076] During the image processing unit's processing, the value range IB1 of the first image signal and the value range IB2 of the second image signal can be the same. The value range FB1 of the synthesized image is not less than the value range IB1 of the first image signal, and the value range FB1 of the synthesized image is not less than the value range IB2 of the second image signal. The value range FB2 of the image after gain processing is not less than the value range IB1 of the first image signal, and the value range FB2 of the image after gain processing is not less than the value range IB2 of the second image signal. The value range OB1 of the image to be output is not greater than the value range FB2 of the image after gain processing.

[0077] For example, the range of values ​​can be the range of grayscale values. For instance, the range of values ​​can be expressed in the form of [T1, T2], where T1 represents the minimum grayscale value and T2 represents the maximum grayscale value.

[0078] For example, see Figure 4BThe diagram shows the structure of an image processing unit, which may include a gain processing module and a data compression module. The synthesis of the first and second image signals has already been completed in the image acquisition unit; the image processing unit obtains the third image signal (i.e., the synthesized image). After obtaining the synthesized image, the gain processing module can perform gain processing on the synthesized image to obtain a gain-processed image. The data compression module can perform data compression processing on the gain-processed image to obtain the image to be output (i.e., the compressed image is used as the output image).

[0079] For example, see Figure 4C The diagram shows the structure of an image processing unit, which may include a preprocessing module, a gain processing module, and a data compression module. After obtaining the first image signal and the second image signal, the preprocessing module performs preprocessing on both signals to obtain preprocessed first and second image signals. Preprocessing may include, but is not limited to, at least one of the following: gain processing, black level processing, noise reduction processing, white balance processing, and deblurring processing. Then, the preprocessing module can synthesize the preprocessed first and second image signals to obtain a composite image (i.e., the functionality of the synthesis processing module can be integrated within the preprocessing module).

[0080] After obtaining the synthesized image, the gain processing module performs gain processing on the synthesized image to obtain the gain-processed image. The data compression module then performs data compression processing on the gain-processed image to obtain the output image.

[0081] For example, see Figure 4D The diagram shows the structure of an image processing unit, which may include a preprocessing module, a gain processing module, and a data compression module. The synthesis of the first and second image signals has already been completed in the image acquisition unit; the image processing unit obtains the third image signal. After obtaining the third image signal, the preprocessing module can perform preprocessing on it to obtain a preprocessed image signal (i.e., the synthesized image). The gain processing module can perform gain processing on the synthesized image to obtain a gain-processed image. The data compression module can perform data compression processing on the gain-processed image to obtain the output image (i.e., the compressed image is used as the output image).

[0082] The preprocessing may include, but is not limited to, at least one of the following: gain processing, black level processing, noise reduction processing, white balance processing, and deblurring processing. Furthermore, when the image sensor is in build-in wide dynamic range mode, the third image signal is a build-in high dynamic range image signal. Based on this, the preprocessing may also include decompression processing, which can be achieved through curve mapping. For example, a preset brightness mapping curve can be used to map the input nonlinear image to a linear image with a higher value range. Moreover, the preset brightness mapping curve corresponds to the build-in data compression curve in the image sensor.

[0083] For example, see Figure 4E The diagram shows the structure of an image processing unit, which may include a synthesis processing module, a gain processing module, a data compression module, and a post-processing module. The synthesis processing module synthesizes the first image signal and the second image signal to obtain a synthesized image. The gain processing module performs gain processing on the synthesized image to obtain a gain-processed image. The data compression module performs data compression processing on the gain-processed image to obtain a compressed image. The post-processing module performs post-processing on the compressed image to obtain the image to be output. This post-processing may include, but is not limited to, at least one of the following: noise reduction, enhancement, dehazing, deblurring, and sharpening.

[0084] For example, see Figure 4F The diagram shows the structure of an image processing unit, which may include a preprocessing module, a gain processing module, a data compression module, and a post-processing module. The preprocessing module processes the first and second image signals respectively to obtain preprocessed first and second image signals, and then synthesizes the preprocessed first and second image signals to obtain a composite image. The gain processing module performs gain processing on the composite image to obtain a gain-processed image. The data compression module performs data compression processing on the gain-processed image to obtain a compressed image. The post-processing module performs post-processing on the compressed image to obtain the image to be output.

[0085] For example, the image processing unit may also include an intermediate processing module. The intermediate processing performed by this module may include, but is not limited to, at least one of the following: noise reduction, white balance processing, and deblurring. To avoid increased resource usage in the intermediate processing module due to excessively high digital gain from the gain processing module, the digital gain can be split into two parts, i.e., two gain processing modules can be deployed. Similarly, to avoid increased resource usage in the intermediate processing module due to an excessively large value range from the data compression module, data compression can be split into two parts, i.e., two data compression modules can be deployed. In summary, see [link to summary]. Figure 4GThe diagram shown is a structural schematic of an image processing unit. The image processing unit may include a synthesis processing module, a gain processing module A, a data compression module A, an intermediate processing module, a gain processing module B, and a data compression module B.

[0086] The compositing module combines the first and second image signals to obtain a composite image. Gain processing module A performs gain processing on the composite image to obtain intermediate image 1. Data compression module A performs data compression processing on intermediate image 1 to obtain intermediate image 2. Intermediate processing module performs intermediate processing on intermediate image 2 to obtain intermediate image 3. Gain processing module B performs gain processing on intermediate image 3 to obtain a gain-processed image. Data compression module B performs data compression processing on the gain-processed image to obtain a compressed image, which can then be used as the output image.

[0087] During the image processing unit's processing, the value range IB1 of the first image signal and the value range IB2 of the second image signal can be the same. The value range FB1 of the synthesized image is not less than the value range IB1 of the first image signal, and the value range FB1 is not less than the value range IB2 of the second image signal. The value range FB2 of intermediate image 1 is not less than the value range IB1 of the first image signal, and the value range FB2 is not less than the value range IB2 of the second image signal. The value range FB3 of intermediate image 2 is not greater than the value range FB2 of intermediate image 1. The value range FB4 of intermediate image 3 is equal to the value range FB3 of intermediate image 2. The value range FB5 of the image after gain processing is not less than the value range FB4 of intermediate image 3. The value range OB1 of the image to be output is not greater than the value range FB5 of the image after gain processing.

[0088] For example, Figures 4A-4G These are just a few examples of image processing units; the structure of this image processing unit is not limited and can be any combination of the modules mentioned above. For example, the image processing unit includes a synthesis processing module, a gain processing module, and a data compression module. The preprocessing module, intermediate processing module, and postprocessing module are optional and can be combined arbitrarily.

[0089] The functions of the synthesis processing module, gain processing module, and data compression module are described below.

[0090] The compositing module combines the first image signal and the second image signal based on processing control parameters to obtain a composite image. The first and second image signals can be a pair of image signals output from an image sensor, generated under the same aperture and exposure time, but with different conversion gains. The sensor gains can be the same or different. The brightness of the first image signal can be greater than that of the second image signal. The value range of the composite image is neither less than the value range of the first image signal nor less than the value range of the second image signal.

[0091] For example, the second image signal can be multiplied by a compensation gain to obtain a gain-compensated image; then, the first image signal and the gain-compensated image can be synthesized to obtain a composite image.

[0092] For example, the processing control parameters include a compensation gain, which is the ratio between a first gain value and a second gain value. The first gain value is the product of a first conversion gain and a first sensor gain (i.e., the conversion gain and sensor gain corresponding to the first image signal), and the second gain value is the product of a second conversion gain and a second sensor gain (i.e., the conversion gain and sensor gain corresponding to the second image signal). The first gain value is greater than the second gain value. Based on these processing control parameters, the synthesis processing module can determine the compensation gain and then multiply the second image signal by the compensation gain to obtain a gain-compensated image.

[0093] When the second image signal is multiplied by the compensation gain to obtain the gain-compensated image, no data truncation is performed. Therefore, the value range of the gain-compensated image is greater than or equal to the value range of the second image signal.

[0094] For example, after obtaining the gain-compensated image, the first image signal and the gain-compensated image can be synthesized to obtain a composite image. When synthesizing the first image signal and the gain-compensated image, the image region of the first image signal is divided into a first region and a second region. The brightness in the first region is no greater than a synthesis brightness threshold, while the brightness in the second region is greater than the synthesis brightness threshold. The composite image is obtained by using the first image signal in the first region and the gain-compensated image in the second region. For instance, a synthesis brightness threshold fl_thr can be pre-configured, and the first image signal and the gain-compensated image are synthesized based on the synthesis brightness threshold fl_thr.

[0095] Image synthesis refers to: judging the brightness of different regions of a first image signal; selecting regions of the first image signal with a brightness not greater than fl_thr as the image region of the synthesized image; and selecting a gain-compensated image as the image region of the synthesized image for regions with a brightness greater than fl_thr. During image synthesis, different weights can also be assigned based on the different brightness levels of the first image signal or the gain-compensated image, resulting in a weighted synthesis of the first image signal and the gain-compensated image. Of course, the above are just two examples of image synthesis using the first image signal and the gain-compensated image, and this image synthesis method is not limited.

[0096] It should be noted that the synthesis processing module can also be designed to synthesize more frames of image signals, thereby adapting to other high dynamic range data generation methods, such as three-frame or four-frame wide dynamic range.

[0097] Gain Processing Module: This module performs gain processing on the synthesized image based on processing control parameters to obtain a gain-processed image. For example, the processing control parameters may include digital gain, and gain processing involves multiplying the synthesized image by this digital gain. When multiplying the synthesized image by this digital gain, data is either not truncated or truncated to a value range not smaller than the input image's value range (i.e., after multiplying the synthesized image by the digital gain, no data truncation is performed, or the data is truncated to a value range not smaller than the input image's value range), and the value range of the gain-processed image is greater than or equal to the value range of the synthesized image. The gain is used to achieve the target brightness in the image; by not truncating, information loss in bright areas of the image is avoided. Similarly, truncating to a value range not smaller than the input image's value range also avoids the loss of information in bright areas of the image.

[0098] Assuming the statistical brightness of the synthesized image is s1, and s1 is not higher than the preset target brightness tl, the statistical brightness of the image after gain processing is s2, and s2 is equal to the preset target brightness tl. The preset target brightness tl is a brightness value that makes the image visually appealing and can be configured empirically. Based on this, the gain value g of the digital gain processing is tl / s1, meaning the digital gain can be tl / s1.

[0099] After multiplying the synthesized image by the digital gain g, without truncating the data, the range of values ​​for the resulting image after gain processing is not less than the range of values ​​for the synthesized image.

[0100] The statistical brightness s1 of a composite image can be calculated in several ways. First, it can be the average of the composite images. Second, it can be a weighted average of image regions with different brightness levels, assigned different weights to regions at different spatial locations within the image, and then calculated using the aforementioned methods. Third, it can be a weighted average of the composite image with the smaller value between a preset brightness threshold sl_thr1 and the preset brightness threshold sl_thr1. These are just a few examples of how to calculate the statistical brightness s1, and no limitation is imposed.

[0101] The calculation method for the statistical brightness s2 of the gain-processed image can be the same as or different from the calculation method for the statistical brightness s1 of the synthesized image; this will not be elaborated here. For example, the smaller value between the gain-processed image and a preset brightness threshold can be taken first, and then a weighted average can be performed.

[0102] To prevent the digital gain g of the gain processing module from becoming too large, the digital gain g should not exceed a preset gain upper limit threshold G_MAX. Therefore, when the digital gain g exceeds the preset gain upper limit threshold G_MAX, a portion of the digital gain needs to be allocated to another gain processing module, i.e., two gain processing modules are deployed. See [link to relevant documentation]. Figure 4G As shown, the two gain processing modules are gain processing module A and gain processing module B. Based on this, the digital gain of gain processing module A is G1 = G_MAX, and the digital gain of gain processing module B is G2 = tl / (s1*g).

[0103] For a scenario involving the deployment of gain processing module A and gain processing module B, assuming the statistical brightness of the synthesized image is s1, and the statistical brightness of the image after gain processing (i.e., the image processed by gain processing module B) is s5, then the statistical brightness s5 is equal to the preset target brightness tl, where the preset target brightness tl is a brightness value that makes the image visually appealing and can be configured empirically. After performing gain processing on the image (i.e., gain processing based on digital gain G2), gain processing module B does not truncate the data. The calculation method for the statistical brightness s5 of the image after gain processing can be the same as or different from the calculation method for the statistical brightness s1 of the synthesized image; this will not be elaborated further here.

[0104] In this case, the gain processing and data compression processing of the synthesized image may include, but are not limited to: if the digital gain is greater than a preset gain threshold, then the digital gain is split into a first digital gain value and a second digital gain value, the first digital gain value being equal to the preset gain threshold, and the product of the first digital gain value and the second digital gain value being equal to the digital gain; the synthesized image is processed according to the first digital gain value to obtain a first intermediate image; the first intermediate image is processed according to data compression to obtain a second intermediate image; the second intermediate image is processed according to intermediate processing to obtain a third intermediate image; the third intermediate image is processed according to the second digital gain value to obtain a gain-processed image; the gain-processed image is processed according to data compression to obtain a compressed image; and the image to be output is determined based on the compressed image.

[0105] Data Compression Module: The data compression module can compress the gain-processed image to obtain a compressed image (the compressed image serves as the output image, or the compressed image can be post-processed to obtain the output image). During data compression, a preset non-linear mapping curve can be used to map the brightness of the gain-processed image. The non-linear mapping curve is related to the digital gain g, meaning the compression parameters used in data compression are related to the gain used in gain processing. This relationship between the compression parameters and the gain used in gain processing includes: under different gains, the brightness range of the image obtained from gain processing is different. The compression parameters are adjusted according to these different brightness ranges to ensure the compressed image has the same brightness range under different gains. The value range of the compressed image is not greater than the value range of the gain-processed image. The relationship between the non-linear mapping curve and the digital gain g means that different digital gains g produce different value ranges in the gain-processed image, and the non-linear mapping curve needs to compress data from different value ranges to the same value range.

[0106] When performing data compression on the image after gain processing, the image is divided into different regions, and the mapping curve corresponding to each region is obtained. For each region of the image after gain processing, the region is mapped using the mapping curve corresponding to that region to obtain the compressed image.

[0107] If the maximum value of the synthesized image is il1_max, the maximum value of the image after gain processing is l2_max = il1_max * g, and the maximum value of the image after compression processing is ol1_max, which is a preset value based on subsequent application conditions, then the nonlinear mapping curve needs to be set so that the maximum value of the input image is l2_max and the maximum value of the output image is ol1_max. Furthermore, to improve the naturalness of the processed image's visual effect, in the nonlinear mapping curve, the portion with brightness less than the preset value lin_thr uses linear mapping, meaning the mapped value equals the original value.

[0108] Data compression can also employ local mapping, which involves using different mapping curves for different image regions to achieve better visual effects. However, the mapping curves for different regions must satisfy the condition that the maximum value of the input image is l2_max and the maximum value of the output image is ol1_max.

[0109] Other mapping methods can also be used for data compression, such as retinex and histograms, but the mapping process also needs to satisfy the condition that the maximum value of the input image is l2_max and the maximum value of the output image is ol1_max.

[0110] When two gain processing modules (gain processing module A and gain processing module B) need to be deployed, two data compression modules (data compression module A and data compression module B) also need to be deployed accordingly. In this case, the maximum value of the first image signal is il1_max, the digital gain of gain processing module A is g, and the maximum value of intermediate image 1 is l2_max = il1_max * g. The maximum value of intermediate image 2 is a preset value l3_max. Therefore, data compression module A must satisfy the condition that the maximum value of the input image is l2_max and the maximum value of the output image is l3_max. The processing method of data compression module A can adopt the same processing method described above.

[0111] The maximum value l4_max of intermediate image 3 is equal to the maximum value l3_max of intermediate image 2. The gain value of gain processing module B is g2 (i.e., G2 mentioned above), so the maximum value of the image after gain processing is l5_max = l4_max * g2. The maximum value ol1_max of the output image is a pre-set value based on subsequent application conditions. Therefore, data compression module B needs to correspondingly satisfy the maximum value of the input image as l5_max and the maximum value of the output image as ol1_max. The processing method of data compression module B can adopt the same processing method described above.

[0112] Third, statistical units.

[0113] The statistical unit can acquire target statistical information based on the statistical image, and this target statistical information may include dynamic range expansion values ​​and statistical brightness values. For example, the statistical image may include a first statistical image and a second statistical image, and the statistical unit may include a dynamic range statistics module and a brightness statistics module. For example, the first statistical image may be an image before synthesis processing, an image after synthesis processing, an image before gain processing, an image after gain processing, an image before data compression processing, or an image after data compression processing; that is, the first statistical image can be any image involved in the image processing unit, such as the image after synthesis processing. The second statistical image may be an image before synthesis processing, an image after synthesis processing, an image before gain processing, an image after gain processing, an image before data compression processing, or an image after data compression processing; that is, the second statistical image can be any image involved in the image processing unit, such as the image after data compression processing.

[0114] For example, the dynamic range statistics module can obtain the dynamic range extension value based on the first statistical image, and the brightness statistics module can obtain the statistical brightness value based on the second statistical image. For instance, the dynamic range statistics module can perform dynamic range statistics on the first statistical image to obtain the dynamic range extension value, and the brightness statistics module can perform brightness statistics on the second statistical image to obtain the statistical brightness value.

[0115] Dynamic Range Statistics Module: After obtaining the first statistical image, the dynamic range statistics module can perform dynamic range statistics on the first statistical image to obtain the dynamic range expansion value. For example, it can perform brightness statistics on the bright areas of the first statistical image to obtain a brightness statistical value; based on the difference between the brightness statistical value and the target brightness value, and a preset dynamic range expansion value threshold, the dynamic range expansion value is determined; wherein, the preset dynamic range expansion value threshold is determined based on the computing power or noise reduction capability of the image processing logic platform, so that the noise amplification of digital processing based on the dynamic range expansion value is within a preset range.

[0116] For example, a brightness statistical value br_sta can be obtained by performing brightness statistics on the highlighted areas of the first statistical image. The brightness statistical value br_sta can be calculated by averaging the pixel values ​​in the first statistical image whose brightness falls between preset brightness thresholds br_thr1 and br_thr2. Alternatively, the brightness statistical value br_sta can be calculated by taking a weighted average of the pixel values ​​in the first statistical image whose brightness falls between the preset brightness thresholds br_thr1 and br_thr2, assigning different weight values ​​to regions at different spatial locations in the image, or assigning different weight values ​​to different brightness levels. Finally, the brightness statistical value br_sta can be calculated by summing the number of pixels in the first statistical image whose brightness falls between the preset brightness thresholds br_thr1 and br_thr2. The highlight statistic br_sta can be calculated by weighting and summing the number of pixels in the first statistical image whose brightness falls between preset highlight thresholds br_thr1 and br_thr2. Different weight values ​​can be assigned to regions at different spatial locations in the image, or different weight values ​​can be assigned to the number of pixels with different brightness levels. Of course, the above is just an example of calculating the highlight statistic br_sta and is not a limitation.

[0117] Assuming the preset highlight target value is br_dst, and the highlight target value br_dst is not greater than the preset highlight threshold br_thr1, the dynamic range extension value can be calculated by determining the dynamic range extension value dr_rat based on the ratio of the highlight statistics value br_sta to the highlight target value br_dst. For example, dr_rat = br_sta / br_dst.

[0118] For example, the highlight target value br_dst refers to the brightness value that makes the highlighted areas of the image appear more suitable. The highlight target value br_dst can be configured based on experience. The larger the highlight statistics value br_sta is, the more bright areas exist in the current scene being photographed, and a larger dynamic range extension value is needed to preserve the information of the bright areas.

[0119] After obtaining the dynamic range extension value dr_rat, the final dynamic range extension value can be obtained by taking the smaller of dr_rat and a preset dynamic range threshold. The preset dynamic range threshold keeps the noise amplification caused by the digital gain within a controllable range and is set according to the computing power of the logic platform or the noise reduction level.

[0120] In summary, the dynamic range statistics module can perform dynamic range statistics on the first statistical image to obtain an initial value for dynamic range expansion. This initial value is related to the brightness distribution of the bright areas in the first statistical image. When the initial value for dynamic range expansion is greater than a preset dynamic range threshold, the dynamic range expansion value is the smaller of the initial value and the threshold. The dynamic range threshold controls the degree of noise amplification caused by digital gain within a certain range. The dynamic range threshold can be set according to the computing power of the logic platform, the level of noise reduction processing, etc. When obtaining the dynamic range expansion value, it is not less than 1. The dynamic range expansion value determines the maximum value that can be obtained from the acquired high dynamic range image.

[0121] Brightness Statistics Module: After obtaining the second statistical image, the brightness statistics module can perform brightness statistics on the second statistical image to obtain a statistical brightness value (which can be denoted as the statistical brightness value y_sta). This statistical brightness value y_sta is related to the brightness distribution of each region in the second statistical image.

[0122] For example, the statistical brightness value y_sta can be calculated by taking the average of the second statistical image. Alternatively, it can be calculated by assigning different weights to image regions of different brightness levels in the second statistical image and then performing a weighted average. Another method is to assign different weights to regions at different spatial locations in the second statistical image and then perform a weighted average. Yet another method is to preset a brightness threshold sta_thr, take the smaller value between the second statistical image and sta_thr, and then perform a weighted average on the second statistical image to obtain the statistical brightness value. This could be achieved using weighted averages obtained in the above ways (e.g., taking the average of the second statistical image; assigning different weights to image regions of different brightness levels in the second statistical image and then performing a weighted average; assigning different weights to regions at different spatial locations in the second statistical image and then performing a weighted average). Of course, these are just a few examples and are not intended to be limiting.

[0123] Fourth, the exposure control unit.

[0124] After receiving target statistical information, the exposure control unit obtains target exposure control parameters and target processing control parameters based on the target statistical information (such as the dynamic range extension value dr_rat and the statistical brightness value y_sta), and outputs these parameters to the buffer unit. The target exposure control parameters are the exposure control parameters for the Kth frame, used by the image acquisition unit to expose the Kth frame image (such as the first and second image signals of the Kth frame). The target processing control parameters are the processing control parameters for the Kth frame, used by the image processing unit to perform image processing on the Kth frame image.

[0125] For example, the exposure control unit adjusts the historical exposure control parameters and historical processing control parameters based on the statistical brightness value y_sta to obtain candidate exposure control parameters and candidate processing control parameters. The first parameter adjustment is determined based on the adjusted statistical brightness value and a preset target brightness. For instance, the first parameter adjustment is used to ensure that the difference between the adjusted statistical brightness value and the preset target brightness is less than a threshold, i.e., the adjusted statistical brightness value is close to the preset target brightness. The exposure control unit then adjusts the candidate exposure control parameters and candidate processing control parameters based on the dynamic range extension value dr_rat to obtain target exposure control parameters and target processing control parameters. The second parameter adjustment is determined based on the adjusted target gain value and the adjusted dynamic range extension value, and the target gain value is the product of the compensation gain and the digital gain. For instance, the second parameter adjustment is used to ensure that the difference between the adjusted target gain value and the adjusted dynamic range extension value is less than a threshold, i.e., the actual gain value reaches or approaches the dynamic range extension value.

[0126] For example, historical exposure control parameters are the exposure control parameters currently used by the image acquisition unit (such as the exposure control parameters of frame K-1, frame K-2, etc.), and historical processing control parameters are the processing control parameters currently used by the image processing unit (such as the processing control parameters of frame K-1, frame K-2, etc.). Candidate exposure control parameters are the exposure control parameters adjusted by the first parameter, and candidate processing control parameters are the processing control parameters adjusted by the first parameter. Target exposure control parameters are the exposure control parameters adjusted by the second parameter, which can be fed back to the image acquisition unit as the exposure control parameters for frame K, and target processing control parameters are the processing control parameters adjusted by the second parameter, which can be fed back to the image processing unit as the processing control parameters for frame K.

[0127] Regarding exposure control parameters (such as historical exposure control parameters, candidate exposure control parameters, and target exposure control parameters), these parameters may include, but are not limited to, aperture, exposure time, a first conversion gain corresponding to the first image signal, a first sensor gain corresponding to the first image signal, a second conversion gain corresponding to the second image signal, and a second sensor gain corresponding to the second image signal. Clearly, the exposure control parameters include parameters for both the first and second image signals, allowing for different sensor gains to be configured for different image signals, achieving more flexible dynamic range adjustment and a higher signal-to-noise ratio.

[0128] For processing control parameters (such as historical processing control parameters, candidate processing control parameters, and target processing control parameters), these processing control parameters may include, but are not limited to, compensation gain and digital gain. Since the compensation gain is the ratio between a first gain value and a second gain value, and the first gain value is the product of a first conversion gain and a first sensor gain, and the second gain value is the product of a second conversion gain and a second sensor gain, the compensation gain can be obtained based on these exposure control parameters after obtaining the first conversion gain, the first sensor gain, the second conversion gain, and the second sensor gain.

[0129] For example, see Figure 5 The diagram shows the structure of the exposure control unit, which may include a first exposure control module and a second exposure control module. The first exposure control module adjusts the historical exposure control parameters and historical processing control parameters based on the statistical brightness value y_sta to obtain candidate exposure control parameters and candidate processing control parameters. The second exposure control module adjusts the candidate exposure control parameters and candidate processing control parameters based on the dynamic range extension value dr_rat to obtain target exposure control parameters and target processing control parameters, and outputs these parameters. The functions of the first and second exposure control modules are described below.

[0130] First Exposure Control Module: The first exposure control module adjusts the historical exposure control parameters EXPO based on the statistical brightness value y_sta and the preset target brightness. K-N and historical processing control parameters PRO K-N (K represents the frame number of the current frame, N is greater than or equal to 1, and KN represents the frame number of the historical frame.) Perform the first adjustment to obtain the candidate exposure control parameter EXPO for the Kth frame. K_ini And the candidate processing control parameter PRO of the Kth frame K_ini The preset target brightness refers to the brightness value that makes the image more suitable for subsequent applications, and it can be configured based on experience.

[0131] For example, the candidate exposure control parameter EXPOK_ini This may include, but is not limited to, candidate aperture LR. K_ini Candidate exposure time t K_ini The candidate first sensor gain sg_l corresponding to the first image signal K_ini The candidate first conversion gain hg_l corresponding to the first image signal K_ini The candidate second sensor gain sg_s corresponding to the second image signal K_ini The candidate second conversion gain hg_s corresponding to the second image signal K_ini .

[0132] Regarding aperture and exposure time, the first exposure control module can adjust the aperture and exposure time in the historical exposure control parameters to obtain candidate aperture (lr). K_ini and candidate exposure time t K_ini It should be noted that if the image acquisition unit does not have an automatic aperture, the subsequent aperture parameter adjustment process can be omitted. In other words, the aperture is an optional parameter. The following explanation will be based on the case where an aperture parameter exists.

[0133] Regarding the first conversion gain and the second conversion gain, the conversion gain is an attribute of the image sensor. The ratio of the first conversion gain to the second conversion gain is configured at the factory, and this ratio is 1 or the maximum value hg. max If the ratio is 1, it means the first conversion gain is the same as the second conversion gain; if the ratio is the maximum value hg, it means the first conversion gain is the same as the second conversion gain. max This means that the first conversion gain is hg of the second conversion gain. max times.

[0134] If adjustment of the first conversion gain and the second conversion gain is not supported, the first conversion gain corresponding to the first image signal can be fixed at hg. max The second conversion gain corresponding to the second image signal can be fixed at 1. Based on this, the first exposure control module does not need to adjust the first and second conversion gains in the historical exposure control parameters, that is, it keeps the first and second conversion gains unchanged. Thus, the candidate first conversion gain is hg. max The candidate second conversion gain is 1. Alternatively, if adjustment of the first and second conversion gains is supported, the first exposure control module can adjust the first and second conversion gains in the historical exposure control parameters to obtain candidate first and candidate second conversion gains. For ease of description, we will take an example where adjustment of the first and second conversion gains is not supported.

[0135] Regarding the first sensor gain and the second sensor gain, different sensor gains can be configured for the first image signal and the second image signal to further adjust the dynamic range. Based on this, the first exposure control module can adjust the first sensor gain and the second sensor gain in the historical exposure control parameters to obtain the candidate first sensor gain sg_l. K_ini and candidate second sensor gain sg_s K_ini .

[0136] For example, the candidate processing control parameter PRO K_ini Including but not limited to candidate compensated gain CDG K_ini and candidate digital gain dg K_ini Regarding the compensation gain, since the compensation gain is the ratio between the first gain value and the second gain value, and the first gain value is the product of the first conversion gain and the first sensor gain, and the second gain value is the product of the second conversion gain and the second sensor gain, the candidate compensation gain (CDG) can be determined based on the candidate first conversion gain, candidate second conversion gain, candidate first sensor gain, and candidate second sensor gain. K_ini Regarding digital gain, the first exposure control module can adjust the digital gain in the historical processing control parameters to obtain the candidate digital gain dg. K_ini .

[0137] In summary, the parameters to be adjusted can include aperture, exposure time, first sensor gain, second sensor gain, and digital gain, while the first conversion gain and the second conversion gain are not adjusted (of course, in some scenarios, the first conversion gain and the second conversion gain can also be adjusted; this embodiment takes not adjusting as an example). The compensation gain is derived based on the conversion gain and sensor gain.

[0138] For example, the first exposure control module uses historical exposure control parameters EXPO K-N When making adjustments, the direction of adjustment can be to increase EXPO. K-N It can also be to reduce EXPO K-N The candidate exposure control parameter EXPO is obtained. K_ini The first exposure control module controls the historical processing parameter PRO. K-N When making adjustments, the direction of adjustment can be to increase PRO. K-N It can also be to reduce PRO K-N The candidate processing control parameter PRO is obtained. K_ini Among them, in the EXPO K-N and PRO K-NWhen making adjustments, the adjustment step size refers to the amount of change used in the adjustment direction. It can be set as a fixed step size or a dynamic step size. The fixed step size can be set according to application requirements. The dynamic step size can be set based on the difference between the statistical brightness value y_sta and the preset target brightness; the greater the difference, the larger the dynamic step size.

[0139] For example, in the case of EXPO K-N and PRO K-N When making adjustments, if the statistical brightness value y_sta is less than the preset target brightness, then EXPO can be adjusted. K-N EXPO was obtained by enlarging the process. K_ini And it can be used for PRO K-N Enlargement process to obtain PRO K_ini This ensures that the statistical brightness value after the first adjustment is close to the preset target brightness, meaning that the difference between the adjusted statistical brightness value and the preset target brightness is less than the threshold.

[0140] When the statistical brightness value y_sta is less than the preset target brightness, the parameter adjustment order is as follows: exposure time, aperture, sensor gain (e.g., first sensor gain and second sensor gain), and digital gain. Of course, this parameter adjustment order is just an example and is not limited to any particular order. For example, the parameter adjustment order could be aperture, exposure time, sensor gain, and digital gain. Or, it could be sensor gain, digital gain, aperture, and exposure time. Or, it could be digital gain, sensor gain, aperture, and exposure time, and so on; any order is possible.

[0141] For example, in the case of EXPO K-N and PRO K-N When making adjustments, if the statistical brightness value y_sta is greater than the preset target brightness, then EXPO can be adjusted. K-N EXPO is obtained by reducing its size. K_ini And it can be used for PRO K-N PRO is obtained by reducing its size. K_ini This ensures that the statistical brightness value after the first adjustment is close to the preset target brightness, meaning that the difference between the adjusted statistical brightness value and the preset target brightness is less than the threshold.

[0142] When the statistical brightness value y_sta is greater than the preset target brightness, the parameter adjustment order is as follows: digital gain, sensor gain (e.g., first sensor gain and second sensor gain), aperture, and exposure time. Of course, this parameter adjustment order is just an example and is not limited to any particular order. For example, the parameter adjustment order could be: exposure time, aperture, sensor gain, digital gain. Or, sensor gain, digital gain, aperture, exposure time. Or, digital gain, sensor gain, exposure time, aperture, and so on; any order is possible.

[0143] For example, the statistical brightness value after the first adjustment refers to the EXPO K_ini With EXPO K-N Changes between them, PRO K_ini With PRO K-N The changes between these factors collectively affect the brightness value obtained from the statistical brightness value.

[0144] For example, when the statistical brightness value y_sta equals the preset target brightness, then no parameter adjustment is needed. In this case, EXPO K_ini =EXPO K-N And PRO K_ini =PRO K-N .

[0145] For example, in DCG mode, the image acquisition unit outputs a brighter first image signal and a darker second image signal. The first and second image signals share the same exposure time and the same aperture. The first image signal corresponds to the first conversion gain and the first sensor gain, and the second image signal corresponds to the second conversion gain and the second sensor gain. In summary, the historical exposure control parameter EXPO... K-N (Taking N as 1 as an example) Includes the first conversion gain hg_l k-1 Second conversion gain hg_s k-1 First sensor gain sg_l k-1 Second sensor gain sg_s k-1 Aperture LR k-1 Exposure time t k-1 Candidate Exposure Control Parameters EXPO K_ini It may include the first conversion gain hg_l k Second conversion gain hg_s k First sensor gain sg_l k Second sensor gain sg_s k Aperture LR k Exposure time t k In addition, the historical processing control parameter PROK-N Including compensation gain CDG k-1 and digital gain dg k-1 And the candidate processing control parameter PRO K_ini Including compensation gain CDG k and digital gain dg k Compensation gain The dynamic range extension value dr_rat k =cdg k ×dg k .

[0146] It should be noted that if DCG technology only supports switching of conversion gain, hg max The values ​​in each image sensor are independent and fixed (i.e., hg is configured individually for each image sensor). max And hg max (Unchanged), the following examples all assume the conversion gain is not adjustable; however, stepless adjustment of the conversion gain frame by frame is also supported. For example, the first conversion gain is usually set to... Set to factory maximum value hg max And the second conversion gain Set to 1 if the first sensor gain Equal to the second sensor gain And digital gain dg k-1 If the value is 1, then the dynamic range extension value is... It can be hg max In subsequent processes, the gain of the first sensor can be set. Not equal to the gain of the second sensor Furthermore, the ratio between the two remains constant, allowing the dynamic range extension value to be modified in this way.

[0147] For example, by comparing the statistical brightness value y_sta with the preset target brightness y_dst, exposure control parameters and processing control parameters can be adjusted so that the statistical brightness value of subsequently acquired images is equal to or close to the preset target brightness. Here, the preset target brightness y_dst refers to the brightness value that makes the image more suitable for subsequent applications. The adjustment process of this parameter is explained below.

[0148] Case 1: If the statistical brightness value y_sta is less than the preset target brightness y_dst, the parameters can be adjusted based on the quotient of the preset target brightness y_dst and the statistical brightness value y_sta (i.e., the ratio between the two).

[0149] Step S11: Determine the first brightness adjustment amount based on the quotient of the preset target brightness y_dst and the statistical brightness value y_sta. For example, the first brightness adjustment amount can be y_dst / y_sta. Obviously, since the preset target brightness y_dst is larger, the first brightness adjustment amount is greater than 1, and the first brightness adjustment amount is used to increase the parameter.

[0150] Step S12: Obtain a first set of parameters to be adjusted. The first set of parameters to be adjusted includes at least one parameter from historical exposure control parameters and historical processing control parameters. For example, the first set of parameters to be adjusted includes, in sequence, exposure time t. k-1 Aperture LR k-1 Sensor gain (e.g., the first sensor gain sg_l) k-1 Second sensor gain sg_s k-1 ), digital gain dg k-1 Of course, this is just an example of parameter order; other parameter orders can also be used. The order of parameters within the first set of parameters to be adjusted is not restricted. We will use this parameter order as an example later; the implementation of other parameter orders is similar and will not be elaborated further. Adjusting the parameters of the first set of parameters to be adjusted based on the above parameter order can reduce noise amplification.

[0151] After obtaining the first brightness adjustment amount and the first set of parameters to be adjusted, the parameters in the first set of parameters to be adjusted can be adjusted based on the first brightness adjustment amount to obtain the adjusted parameters. For example, the first parameter in the first set of parameters to be adjusted is adjusted first. If the first brightness adjustment amount is met, the adjustment process ends. If the first brightness adjustment amount is not met, the second parameter in the first set of parameters to be adjusted is adjusted. If the first brightness adjustment amount is met, the adjustment process ends. If the first brightness adjustment amount is not met, the third parameter in the first set of parameters to be adjusted is adjusted, and so on, until the first brightness adjustment amount is met. Based on this, candidate exposure control parameters and candidate processing control parameters can be obtained based on the adjusted parameters. The following explains this process.

[0152] Step S13: If the first product of the exposure time and the first brightness adjustment amount is less than or equal to the upper limit threshold of the exposure time, then the adjusted exposure time is determined to be the first product value, and the parameter adjustment process ends.

[0153] If the first product of the exposure time and the first brightness adjustment amount is greater than the upper limit threshold of the exposure time, then the adjusted exposure time is determined to be the upper limit threshold of the exposure time, and step S14 is executed.

[0154] For example, first increase the exposure time k represents the frame number, t k-1This represents the exposure time used when the current frame image was acquired, i.e., the historical exposure time, t. k This represents the adjusted exposure time, i.e., the candidate exposure time, and y_dst / y_sta represents the first brightness adjustment amount. Based on the above adjustments, increasing the exposure time results in the adjusted statistical brightness value. That is equal to the preset target brightness.

[0155] After adjustment, the exposure time t k Afterwards, the adjusted exposure time t can be... k Compared with the upper limit threshold t_max of exposure time, the upper limit threshold t_max of exposure time can avoid the trailing of moving objects in the image caused by excessive exposure time, and also avoid the number of image frames acquired per unit time not meeting the application requirements.

[0156] If the adjusted exposure time t k If the exposure time is less than or equal to the upper limit threshold t_max, then the adjusted exposure time t is maintained. k The exposure time t remains unchanged, that is, the adjusted exposure time t k As a candidate exposure time t k No other parameters need to be adjusted; that is, parameters such as aperture, sensor gain, and digital gain remain unchanged, i.e., lr k =lr k-1 sg_s k =sg_s k-1 sg_l k =sg_l k-1 dg k =dg k-1 In this way, candidate exposure control parameters (such as candidate first conversion gain hg_l) can be obtained. k Candidate second conversion gain hg_s k Candidate first sensor gain sg_l k Candidate second sensor gain sg_s k Candidate aperture LR k Candidate exposure time t k (etc.) and candidate processing control parameters (such as candidate compensation gain cdg) k and candidate digital gain dg k wait).

[0157] If the adjusted exposure time t k If the exposure time exceeds the upper limit threshold t_max, the adjusted exposure time t will be used. k Updated to t_max, t k =t max Adjusted exposure time t max Candidate exposure time t k Proceed to step S14.

[0158] Step S14: Based on the upper limit threshold t_max of exposure time and the exposure time t k-1 The first remaining adjustment amount is determined by the first brightness adjustment amount y_dst / y_sta. For example, the first brightness adjustment amount represents the magnification factor required, while the upper limit threshold t_max of the exposure time is related to the exposure time t. k-1 The ratio between them represents the magnification factor (i.e., the magnification factor that has been achieved through exposure time). Based on the two parameters mentioned above, the remaining magnification factor can be determined. This remaining magnification factor is used as the first remaining adjustment amount, and there are no restrictions on the determination process.

[0159] If the second product of the aperture and the first remaining adjustment amount is less than or equal to the upper limit threshold of the aperture, then the adjusted exposure time is determined to be the upper limit threshold of the exposure time, and the adjusted aperture is determined to be the second product value.

[0160] If the second product of the aperture and the first remaining adjustment amount is greater than the aperture upper limit threshold, then the adjusted exposure time is determined to be the exposure time upper limit threshold, the adjusted aperture is determined to be the aperture upper limit threshold, and step S15 is executed.

[0161] For example, increasing the aperture (LR) k >lr k-1 , such as lr k-1 The product of lr and the first remaining adjustment (the first remaining adjustment is greater than 1). k-1 Indicates historical aperture, lr k This represents the adjusted aperture, i.e., the candidate aperture. Based on the above adjustments, by increasing the exposure time and aperture, the adjusted statistical brightness value is made equal to the preset target brightness; that is, by increasing the amount of light entering the camera, the adjusted statistical brightness value is made equal to the preset target brightness.

[0162] After adjustment, the aperture (lr) k Afterwards, you can adjust the aperture (lr) to... k Compare with the aperture upper limit threshold lr_max. If the adjusted aperture lr is less than or equal to the aperture upper limit threshold lr_max, then maintain the adjusted aperture lr. k The aperture remains unchanged, i.e., the aperture lr is adjusted. k As a candidate aperture LR k No further adjustments will be made to other parameters; that is, parameters such as sensor gain and digital gain will remain unchanged, i.e., sg_s k =sg_s k-1 sg_l k =sg_l k-1 dg k =dg k-1 In this way, candidate exposure control parameters and candidate processing control parameters can be obtained.

[0163] If the aperture is adjusted to lr k If the aperture value is greater than the upper limit threshold lr_max, then the aperture lr will be adjusted. k Updated to the aperture upper limit threshold lr_max, i.e., lr k =lr_max, in this case, the adjusted exposure time t max As a candidate exposure time t k The adjusted aperture lr_max is used as the candidate aperture lr. k Proceed to step S15.

[0164] Step S15: Based on the aperture upper limit threshold lr_max and aperture lr k-1 The first remaining adjustment amount determines the second remaining adjustment amount. For example, the first remaining adjustment amount represents the magnification factor required, while the aperture upper limit threshold lr_max and the aperture lr... k-1 The ratio between them indicates the magnification factor. Based on the two parameters mentioned above, the remaining magnification factor can be determined, and this remaining magnification factor can be used as the second remaining adjustment amount.

[0165] If the third product of the first sensor gain and the second remaining adjustment amount is less than or equal to the upper limit threshold of the first sensor gain, then it can be determined that the adjusted exposure time is the upper limit threshold of the exposure time, the adjusted aperture is the upper limit threshold of the aperture, the adjusted first sensor gain is this third product value, and the adjusted second sensor gain is the product value of the second sensor gain and the second remaining adjustment amount.

[0166] If the third product of the first sensor gain and the second remaining adjustment amount is greater than the upper limit threshold of the first sensor gain, then it can be determined that the adjusted exposure time is the upper limit threshold of the exposure time, the adjusted aperture is the upper limit threshold of the aperture, the adjusted first sensor gain is the upper limit threshold of the first sensor gain, and the adjusted second sensor gain is the upper limit threshold of the second sensor gain, and step S16 is executed.

[0167] For example, increase the gain of the first sensor sg_l k >sg_l k-1 , such as sg_l k-1 The product of the second remaining adjustment (which is greater than 1), sg_l k-1 Represents the historical first sensor gain, sg_l k This represents the adjusted first sensor gain, also known as the candidate first sensor gain or the long-frame sensor gain. Based on the above adjustments, the adjusted statistical brightness value is made equal to the preset target brightness.

[0168] After adjustment, the first sensor gain sg_l kAfterwards, the adjusted first sensor gain sg_l can be... k With the first sensor gain upper limit threshold sg_l max Compare the results. If the adjusted first sensor gain sg_l... k Less than or equal to the first sensor gain upper limit threshold sg_l max Then maintain the adjusted first sensor gain sg_l k The first sensor gain sg_l remains unchanged after adjustment. k As the candidate first sensor gain. During this process, the candidate second sensor gain sg_s k Maintain the gain sg_l of the first candidate sensor k The same degree of change, while not exceeding the upper limit threshold sg_s of the second sensor gain. max Therefore, the gain of the second sensor, sg_s, can also be increased. k >sg_s k-1 sg_s k This represents the adjusted gain of the second sensor, and the adjusted gain of the second sensor is sg_s. k The gain of the second sensor is sg_s k-1 The product of the second remaining adjustment amount and the second sensor gain sg_s. k Less than or equal to the upper limit threshold of the second sensor gain sg_s max Then maintain the adjusted second sensor gain sg_s k The second sensor gain sg_s remains unchanged after adjustment. k This serves as the candidate second sensor gain. Based on this, no other parameters are adjusted; that is, the digital gain remains unchanged, i.e., dg. k =dg k-1 In this way, candidate exposure control parameters and candidate processing control parameters can be obtained.

[0169] After adjustment, the first sensor gain sg_l k Afterwards, if the first sensor gain sg_l is adjusted... k Greater than the upper limit threshold of the first sensor gain sg_l max Then the adjusted first sensor gain sg_l k Updated to the first sensor gain upper limit threshold sg_l max The adjusted second sensor gain sg_s k Updated to the second sensor gain upper limit threshold sg_s max Based on this, the adjusted exposure time t can also be... max As a candidate exposure time t k The adjusted aperture lr_max is used as the candidate aperture lr.k Adjusted first sensor gain sg_l max As the candidate first sensor gain sg_l k Adjusted second sensor gain sg_s max As a candidate for the second sensor gain sg_s k In this case, proceed to step S16.

[0170] Step S16: Based on the first sensor gain upper limit threshold sg_l max First sensor gain sg_l k-1 The second residual adjustment is used to determine the third residual adjustment. For example, the second residual adjustment represents the required amplification factor, while the first sensor gain upper limit threshold sg_l... max With the first sensor gain sg_l k-1 The ratio between them indicates the magnification factor. Based on the above two parameters, the third remaining adjustment amount can be obtained.

[0171] For example, increasing the digital gain (dg) k >dg k-1 , such as dg k-1 The product of the third residual adjustment (which is greater than 1), dg k-1 Indicates historical numerical gain, dg k This represents the adjusted digital gain, i.e., the candidate digital gain. The above adjustment makes the adjusted statistical brightness value equal to the preset target brightness.

[0172] In summary, the adjusted exposure time t can be determined. k The upper limit threshold for exposure time t max Adjusted aperture (lr) k The aperture upper limit threshold lr_max and the adjusted first sensor gain sg_l k The upper limit threshold for the first sensor gain is sg_l max Adjusted second sensor gain sg_s k The upper limit threshold for the second sensor gain is sg_s max Adjusted digital gain dg k It is the product of the digital gain and the third residual adjustment.

[0173] Case 2: If the statistical brightness value y_sta is greater than the preset target brightness y_dst, the parameters can be adjusted based on the quotient of the preset target brightness y_dst and the statistical brightness value y_sta (i.e., the ratio between the two).

[0174] Step S21: Determine the second brightness adjustment amount based on the quotient of the preset target brightness y_dst and the statistical brightness value y_sta. For example, the second brightness adjustment amount can be y_dst / y_sta. Obviously, since the preset target brightness y_dst is smaller, the second brightness adjustment amount is less than 1, and the second brightness adjustment amount is used to reduce the parameter.

[0175] Step S22: Obtain a second set of parameters to be adjusted. The second set of parameters to be adjusted includes at least one parameter from the historical exposure control parameters and historical processing control parameters. For example, the second set of parameters to be adjusted may include, in sequence, the digital gain (dg). k-1 Sensor gain (e.g., the first sensor gain sg_l) k-1 Second sensor gain sg_s k-1 ), Aperture LR k-1 Exposure time t k-1 Of course, this is just an example of parameter order; other parameter orders can also be used. The order of parameters within this second set of parameters to be adjusted is not restricted. We will use this parameter order as an example later; the implementation of other parameter orders is similar and will not be elaborated further. Adjusting based on the above parameter order can reduce noise amplification and avoid flickering.

[0176] After obtaining the second brightness adjustment amount and the second set of parameters to be adjusted, the parameters in the second set of parameters to be adjusted are adjusted based on the second brightness adjustment amount to obtain the adjusted parameters. For example, the first parameter is adjusted. If the second brightness adjustment amount is satisfied, the adjustment process ends; if the second brightness adjustment amount is not satisfied, the second parameter is adjusted, and so on. Then, candidate exposure control parameters and candidate processing control parameters can be obtained based on the adjusted parameters. The following explains this process.

[0177] Step S23: If the first product of the digital gain and the second brightness adjustment amount is greater than or equal to the lower limit threshold of the digital gain, then the adjusted digital gain is determined to be the first product value, and the parameter adjustment process ends.

[0178] If the first product of the digital gain and the second brightness adjustment amount is less than the lower limit threshold of the digital gain, then the adjusted digital gain is determined to be the lower limit threshold of the digital gain, and step S24 is executed.

[0179] For example, first reduce the digital gain. k represents the frame number, dgk -1 This indicates the digital gain used during image processing in the current frame, i.e., the historical digital gain, dg k This represents the adjusted digital gain, i.e., the candidate digital gain. This indicates the second brightness adjustment amount. Based on the above adjustments, by reducing the digital magnification, the adjusted statistical brightness value is... That is equal to the preset target brightness.

[0180] After adjustment, the digital gain dg k Afterwards, the adjusted digital gain (dg) can be... k Compare with the lower limit threshold of digital gain, dg_min. If the adjusted digital gain dg... k If the digital gain is greater than or equal to the lower limit threshold dg_min, then the adjusted digital gain dg is maintained. k The digital gain dg remains unchanged, i.e., the adjusted digital gain dg k As a candidate digital gain dg k At this point, no other parameters need to be adjusted; that is, the sensor gain, aperture, exposure time, and other parameters remain unchanged, i.e., sg_s k =sg_s k-1 sg_l k =sg_l k-1 ,lr k =lr k-1 t k =t k-1 In this way, candidate exposure control parameters and candidate processing control parameters can be obtained.

[0181] If the adjusted digital gain dg k If the digital gain is less than the lower limit threshold dg_min, the adjusted digital gain dg will be... k Updated to dg_min, the adjusted digital gain dg_min is now the candidate digital gain dg. k Proceed to step S24.

[0182] Step S24: Based on the lower limit threshold of digital gain dg_min and digital gain dg k-1 The second brightness adjustment amount, y_dst / y_sta, determines the first remaining adjustment amount. For example, the second brightness adjustment amount represents the factor by which it needs to be reduced, while the digital gain lower limit threshold dg_min and the digital gain dg... k-1 The ratio between them represents the reduction factor already achieved. Based on the two parameters mentioned above, the remaining reduction factor, i.e., the first remaining adjustment amount, can be determined.

[0183] If the second product of the first sensor gain and the first remaining adjustment amount is greater than or equal to the lower limit threshold of the first sensor gain, then the adjusted digital gain is determined to be the lower limit threshold of the digital gain, the adjusted first sensor gain is determined to be the second product value, and the adjusted second sensor gain is determined to be the product of the second sensor gain and the first remaining adjustment amount. If the second product value is less than the lower limit threshold of the first sensor gain, then the adjusted digital gain is determined to be the lower limit threshold of the digital gain, the adjusted first sensor gain is determined to be the lower limit threshold of the first sensor gain, and the adjusted second sensor gain is determined to be the lower limit threshold of the second sensor gain, and step S25 is executed.

[0184] For example, reduce the gain of the first sensor to sg_l k-1 Represents the historical first sensor gain, sg_l k This represents the adjusted gain of the first sensor, i.e., the candidate gain of the first sensor. This represents the first remaining adjustment amount, which can be a value less than 1. Based on the above adjustment, the adjusted statistical brightness value can be made equal to the preset target brightness.

[0185] After adjustment, the first sensor gain sg_l k Afterwards, the adjusted first sensor gain sg_l can be... k Compared with the first sensor gain lower limit threshold sg_l min Compare the results. If the adjusted first sensor gain sg_l... k Greater than or equal to the first sensor gain lower limit threshold sg_l min Then maintain the adjusted first sensor gain sg_l k The first sensor gain sg_l remains unchanged after adjustment. k As the candidate first sensor gain. During this process, the candidate second sensor gain sg_s k Maintain the gain sg_l of the first candidate sensor k The changes are of the same degree, and do not exceed the lower limit threshold sg_s of the second sensor gain. min Therefore, the gain sg_s of the second sensor can also be reduced. k <sg_s k-1 sg_s k This represents the adjusted gain of the second sensor, and the adjusted gain of the second sensor is sg_s. k The gain of the second sensor is sg_s k-1 The product of the first remaining adjustment amount and the second sensor gain sg_s. k Greater than or equal to the second sensor gain lower limit threshold sg_s minThen, the adjusted second sensor gain sg_s remains unchanged, that is, the adjusted second sensor gain sg_s k This serves as the candidate second sensor gain. Based on this, no other parameters are adjusted; the aperture and exposure time remain unchanged, i.e., lr. k =lr k-1 t k =t k-1 In this way, candidate exposure control parameters and candidate processing control parameters can be obtained.

[0186] After adjustment, the first sensor gain sg_l k Afterwards, if the first sensor gain sg_l is adjusted... k Less than the first sensor gain lower limit threshold sg_l min Then the adjusted first sensor gain sg_l k Updated to the first sensor gain lower limit threshold sg_l min The adjusted second sensor gain sg_s k Updated to the second sensor gain lower limit threshold sg_s min Based on this, the adjusted digital gain dg_min is selected as the candidate digital gain dg. k Adjusted first sensor gain sg_l min As the candidate first sensor gain sg_l k Adjusted second sensor gain sg_s min As a candidate for the second sensor gain sg_s k Proceed to step S25.

[0187] Step S25: Based on the first sensor gain lower limit threshold sg_l min First sensor gain sg_l k-1 The second residual adjustment is determined by the first residual adjustment. For example, the first residual adjustment represents the factor by which the sensor gain needs to be reduced, while the first sensor gain lower limit threshold sg_l... min With the first sensor gain sg_l k-1 The ratio between them indicates the factor by which the adjustment has been reduced. Based on the two parameters mentioned above, the second remaining adjustment amount can be determined.

[0188] If the third product of the aperture and the second remaining adjustment amount is greater than or equal to the lower aperture threshold, then the adjusted digital gain is determined to be the lower digital gain threshold, the adjusted first sensor gain is determined to be the lower first sensor gain threshold, the adjusted second sensor gain is determined to be the lower second sensor gain threshold, and the adjusted aperture is determined to be the third product of the aperture and the second remaining adjustment amount. Alternatively, if the third product value is less than the lower aperture threshold, then the adjusted digital gain is determined to be the lower digital gain threshold, the adjusted first sensor gain is determined to be the lower first sensor gain threshold, the adjusted second sensor gain is determined to be the lower second sensor gain threshold, and the adjusted aperture is determined to be the lower aperture threshold, and step S26 continues.

[0189] For example, reducing the aperture (lr) k <lr k-1 , such as lr k-1 The product of lr and the second remaining adjustment (which is less than 1). k-1 Indicates historical aperture, lr k This represents the adjusted aperture, i.e., the candidate aperture. Based on the above adjustments, the adjusted statistical brightness value can be made equal to the preset target brightness.

[0190] After adjustment, the aperture (lr) k After that, you can adjust the aperture (lr) k Compare with the lower aperture threshold lr_min. If the adjusted aperture lr... k If the aperture value is greater than or equal to the lower aperture threshold lr_min, then the adjusted aperture lr will be maintained. k The aperture remains unchanged, i.e., the aperture lr is adjusted. k As a candidate aperture LR k And no other parameters are adjusted, such as the exposure time remains unchanged, i.e., t k =t k-1 Thus, candidate exposure control parameters and candidate processing control parameters are obtained.

[0191] If the aperture is adjusted to lr k If the adjusted statistical brightness value is still greater than the preset target brightness, and the aperture value is less than the lower limit threshold lr_min, then the adjusted aperture lr will be adjusted. k Update to the lower aperture threshold lr_min and proceed to step S26.

[0192] Step S26: Based on the aperture lower limit threshold lr_min and aperture lr k-1 The second residual adjustment is used to determine the third residual adjustment. For example, the second residual adjustment represents the factor by which the aperture lower limit threshold lr_min needs to be reduced, and the aperture lr... k-1 The ratio between them indicates the factor by which the reduction has been made, thus allowing us to determine the third remaining adjustment amount.

[0193] If the fourth product of the exposure time and the third remaining adjustment amount is greater than or equal to the lower limit threshold of the exposure time, then the adjusted digital gain is determined to be the lower limit threshold of the digital gain, the adjusted first sensor gain is determined to be the lower limit threshold of the first sensor gain, the adjusted second sensor gain is determined to be the lower limit threshold of the second sensor gain, the adjusted aperture is determined to be the lower limit threshold of the aperture, and the adjusted exposure time is determined to be the fourth product value.

[0194] For example, reducing the exposure time t k <t k-1 , such as t k-1 The product of the third residual adjustment (which is less than 1), t k-1 Indicates the historical exposure time, t k This represents the adjusted exposure time, i.e., the candidate exposure time. Based on the above adjustments, the adjusted statistical brightness value can be made equal to the preset target brightness.

[0195] After adjustment, the exposure time t k Afterwards, the adjusted exposure time t can be... k With the lower limit threshold of exposure time t min Compare the results. If the adjusted exposure time t... k Greater than or equal to the lower limit threshold t of exposure time min Then maintain the adjusted exposure time t k The exposure time t remains unchanged, meaning the adjusted exposure time t can be kept the same. k As a candidate exposure time t k Thus, the candidate exposure control parameters and candidate processing control parameters can be obtained.

[0196] In one possible implementation, if the product of the exposure time and the fourth value of the third remaining adjustment amount is less than the lower limit threshold of the exposure time, then the adjusted first sensor gain can be determined to be the lower limit threshold of the first sensor gain, the adjusted second sensor gain to be the lower limit threshold of the second sensor gain, the adjusted aperture to be the lower limit threshold of the aperture, and the adjusted exposure time to be the lower limit threshold of the exposure time. Based on this, the digital gain can be further adjusted, that is, the digital gain can be reduced based on the lower limit threshold of the digital gain, until the total adjustment amount is the second brightness adjustment amount, and the adjusted statistical brightness value is equal to the preset target brightness. In this way, the adjusted digital gain can be used as a candidate digital gain to complete the parameter adjustment process.

[0197] Case 3: If the statistical brightness value y_sta is equal to the preset target brightness y_dst, then there is no need to adjust the historical exposure control parameters and historical processing control parameters. For example, the candidate exposure control parameters and historical exposure control parameters can be the same, and the candidate processing control parameters and historical processing control parameters can be the same.

[0198] After adjusting the parameters as described above (case 1, case 2, or case 3), the statistical brightness value of the (k+1)th frame after image acquisition and processing is made equal to the preset target brightness. When the statistical brightness value of the kth frame differs significantly from the preset target brightness, a large adjustment step size may cause a large change in the image brightness of the (k+1)th frame, resulting in inter-frame flickering. To address this, the adjustment step size can be reduced, and the exposure control parameters and processing control parameters can be adjusted gradually over multiple frames to achieve the goal of the statistical brightness value equaling the preset target brightness in the (k+n)th frame (n>1). The adjustment method is similar to the method described above, except that the amount of parameter change between adjacent frames is reduced, and will not be elaborated further.

[0199] In addition, the exposure control parameters and processing control parameters used by the current frame image referenced by the first exposure control module can be parameters obtained from the km (m>=1) frame, which are determined by the buffer unit and application requirements.

[0200] Second exposure control module: The second exposure control module is based on the dynamic range extension value dr_rat k EXPO candidate exposure control parameters K_ini and candidate processing control parameter PRO K_ini A second adjustment is made to obtain the target exposure control parameter EXPO for the Kth frame. K and the target processing control parameter PRO of the Kth frame K The second adjustment can include two aspects: adjusting the direction and adjusting the step size.

[0201] Candidate Exposure Control Parameters EXPO K_ini This may include, but is not limited to, candidate aperture LR. K_ini Candidate exposure time t K_ini Candidate first sensor gain sg_l K_ini Candidate first conversion gain hg_l K_ini Candidate second sensor gain sg_s K_ini Candidate second conversion gain hg_s K_ini Target exposure control parameters EXPO K This may include, but is not limited to, the target aperture (LR). K Target exposure time t K Target first sensor gain sg_l K Target first conversion gain hg_l K Target second sensor gain sg_s K Target second conversion gain hg_s K .

[0202] Regarding aperture and exposure time, the second exposure control module can adjust the aperture and exposure time in the candidate exposure control parameters to obtain the target aperture lr. Kand target exposure time t K It should be noted that if the image acquisition unit does not have an automatic aperture, the subsequent aperture parameter adjustment process can be omitted. In other words, the aperture is an optional parameter. The following explanation will be based on the case where an aperture parameter exists.

[0203] Regarding the first and second conversion gains, if adjustment of the first and second conversion gains is not supported, the first conversion gain can be fixed at hg. max The second conversion gain is fixed at 1.

[0204] Regarding the first sensor gain and the second sensor gain, different sensor gains can be configured. The second exposure control module can adjust the first sensor gain and the second sensor gain in the candidate exposure control parameters to obtain the target first sensor gain sg_l. K and the target's second sensor gain sg_s K .

[0205] For example, the candidate processing control parameter PRO K_ini Including but not limited to candidate compensated gain CDG K_ini and candidate digital gain dg K_ini Target processing control parameter PRO K Including but not limited to the target compensation gain CDG of the synthesis processing module K (The synthesis processing module performs synthesis processing based on the compensated gain), the target digital gain dg of the gain processing module. K (The gain processing module performs gain processing based on digital gain.) Regarding the compensation gain, the target compensation gain (CDG) can be determined based on the target first conversion gain, the target second conversion gain, the target first sensor gain, and the target second sensor gain. K Regarding digital gain, the second exposure control module can adjust the digital gain in the candidate processing control parameters to obtain the target compensation gain (CDG). K .

[0206] For example, the second exposure control module applies the candidate exposure control parameter EXPO. K_ini and candidate processing control parameter PRO K_ini A second adjustment is made to obtain the target exposure control parameter EXPO. K and target processing control parameters PRO K At that time, EXPO can be used K =EXPO K_ini Priority for PRO K_ini Digital gain dg in K_ini Adjustments were made to obtain PRO. K Digital gain dg in KAmong them, digital gain dg K In dynamic range extended value dr_rat k and digital gain dg K_ini Take the larger value between them, i.e., dg K =max(dr_rat k ,dg K_ini ).

[0207] In DG K >dg K_ini If the statistical brightness value after the second adjustment is greater than the statistical brightness value after the first adjustment, then the statistical brightness value after the second adjustment is greater than the statistical brightness value after the first adjustment. The statistical brightness value after the second adjustment refers to: EXPO K With EXPO K_ini Changes between and PRO K With PRO K_ini The changes between these parameters collectively affect the brightness value obtained from the first adjusted statistical brightness value. To ensure that the second adjusted statistical brightness value equals the first adjusted statistical brightness value, the target exposure control parameter EXPO can be reduced. K In DG K =dg K_ini If the statistical brightness value after the second adjustment is equal to the statistical brightness value after the first adjustment, then no adjustment to EXPO is needed. K .

[0208] For example, the target exposure control parameter EXPO K and target processing control parameters PRO K The effective time may be delayed; for example, the target exposure control parameter EXPO generated in the k-th frame may take longer. K and target processing control parameters PRO K The actual effective time is k+1+d frames, where d represents the number of delayed frames. The number of delayed frames d can be 0 or greater than 0. In the examples below, the number of delayed frames d is 0, which means there is no delay.

[0209] For example, when dr_rat k Less than or equal to the preset maximum conversion gain hg max At that time, the gain of the first sensor, sg_l, is adjusted first. K =sg_l K_ini Second sensor gain Aperture LR K =lr K_ini Exposure time t K =t K_ini All remain unchanged, so that the statistical brightness value after the second adjustment is equal to the statistical brightness value after the first adjustment. Or, when dr_rat k Greater than the preset maximum conversion gain hg maxWhen, the adjustment method can be as follows: When Greater than or equal to the preset sensor gain lower limit threshold sg min At that time, the gain of the second sensor is adjusted first. First sensor gain sg_lK=sg_l K_ini aperture LR k =lr k_ini Exposure time t K =t K_ini All remain unchanged, so that the statistical brightness value after the second adjustment is equal to the statistical brightness value after the first adjustment. When Less than the preset sensor gain lower limit threshold sg min At that time, the gain of the second sensor is adjusted to sg_s first. K =sg min The first sensor gain sg_l K =sg_l K_ini Keep the aperture unchanged, then reduce the aperture (lr) K <lr K_ini If the aperture is lr K If the value is greater than or equal to the preset aperture threshold, the condition that the statistical brightness value after the second adjustment is equal to the statistical brightness value after the first adjustment is met. K =t K_ini Otherwise, the exposure time t needs to be adjusted further. K <t K_ini This makes the statistical brightness value after the second adjustment equal to the statistical brightness value after the first adjustment.

[0210] For example, when the dynamic range is expanded to the value dr_rat k When the value is 1, it indicates that there are no highlighted areas in the current scene, and dynamic range expansion is not required. Therefore, the candidate exposure control parameter can be directly used as the target exposure control parameter, and the candidate processing control parameter can be used as the target candidate processing control parameter. When the dynamic range expansion value is dr_rat k A value greater than 1 indicates that there are highlighted areas in the current scene, requiring dynamic range expansion to preserve the information of these highlighted areas. Subsequent calculations will use dr_rat. k For example, greater than 1.

[0211] If the dynamic range extension value is dr_rat k If the value is greater than 1, then the candidate processing control parameter PRO is obtained. K_ini Candidate compensated gain cdg K_ini and candidate digital gain dg K_ini And based on candidate compensated gain cdg K_ini and candidate digital gain dg K_ini The product value determines the initial value of the dynamic range dr_ratb (i.e., the initial value of the dynamic range extension), for example, the following formula can be used to determine dr_rat. b dr_rat b =cdg K_ini *dg K_ini .

[0212] If dr_rat b =dr_rat k If the output image of the gain processing module in the image processing unit can provide the required dynamic range, then the candidate exposure control parameter can be directly used as the target exposure control parameter, and the candidate processing control parameter can be used as the target candidate processing control parameter.

[0213] If dr_rat b With dr_rat k If they are not equal, the second exposure control module will adjust the dynamic range extension value dr_rat. k EXPO candidate exposure control parameters K_ini and candidate processing control parameter PRO K_ini A second adjustment is made to obtain the target exposure control parameter EXPO. K and target processing control parameters PRO K The target exposure control parameters enable the second image signal of the image acquisition unit to provide the required dynamic range, and the target processing control parameters enable the image processing unit to adapt to the adjusted image acquisition unit, so that the two can be matched and coordinated.

[0214] If dr_rat b With dr_rat k If they are not equal, the second exposure control module will adjust them in the following way:

[0215] Case a: If the dynamic range extension value is dr_rat k Less than the initial value of the dynamic range dr_rat b Then based on the dynamic range initial value dr_rat b With dynamic range extension value dr_rat k The quotient (i.e., the ratio of the two) is used to adjust the parameters.

[0216] Step S31: Based on the initial value dr_rat of the dynamic range b With dynamic range extension value dr_rat k The quotient determines the first dynamic range adjustment amount. For example, if dr_rat k <dr_rat b dr_rat represents the dynamic range provided by the output image of the gain processing module in the image processing unit. bThis exceeds the demand; in order for the second image signal to just provide dr_rat k To increase the dynamic range by a factor of two and simultaneously improve the signal-to-noise ratio of the second image signal, it is necessary to increase the brightness of the second image signal. This requires increasing the exposure control parameters of the second image signal by a factor equal to the first dynamic range adjustment amount, where the first dynamic range adjustment amount is [missing value]. Obviously, due to the initial value of the dynamic range dr_rat b It is larger, therefore, the first dynamic range adjustment is greater than 1.

[0217] Step S32: Obtain a third set of parameters to be adjusted. This third set of parameters to be adjusted may include, but is not limited to, at least one parameter among candidate exposure control parameters and candidate processing control parameters.

[0218] For example, the parameters to be adjusted may include candidate aperture, candidate exposure time, candidate first conversion gain, candidate first sensor gain, candidate second conversion gain, candidate second sensor gain, candidate compensation gain, and candidate digital gain from the candidate exposure control parameters, as well as candidate compensation gain and candidate digital gain from the candidate processing control parameters. Considering that the candidate first conversion gain and candidate second conversion gain are not adjusted (although they can be adjusted in some scenarios, this embodiment takes no adjustment as an example), the candidate compensation gain is derived based on the candidate conversion gain and candidate sensor gain, and the candidate digital gain does not affect the actual dynamic range of the image signal output by the image acquisition unit. Therefore, the third set of parameters to be adjusted may include the candidate aperture (denoted as lr). k Candidate exposure time (denoted as t) k ), candidate first sensor gain (denoted as sg_l) k ) and candidate second conversion gain (denoted as sg_s) k ).

[0219] Since the first and second image signals use the same exposure time and aperture, there are two ways to adjust the brightness of the second image signal, considering whether the first image signal will be affected. One method is to adjust the exposure time, aperture, and second sensor gain. The exposure time and aperture will affect the brightness of the first image signal, while the second sensor gain will not (since the first sensor gain does affect the brightness of the first image signal, it is not adjusted). The other method is to not adjust the exposure time or aperture, but only adjust the brightness of the second image signal through the second sensor gain (since the first sensor gain affects the brightness of the first image signal, it is not adjusted), thus not affecting the brightness of the first image signal. Adjusting the brightness of the first and second image signals is a trade-off, a balancing process. In this example, it is necessary to adjust the exposure time, aperture, and second sensor gain (i.e., the adjustment method where the brightness of the first image signal is affected). Therefore, the third set of parameters to be adjusted may include candidate apertures (lr). k Candidate exposure time t k Candidate second conversion gain sg_s k .

[0220] For example, the third set of parameters to be adjusted may sequentially include candidate exposure times t. k Candidate aperture LR k Candidate second conversion gain sg_s k Of course, this is just one example of parameter order; other parameter orders can also be used (such as candidate aperture lr). k Candidate second conversion gain sg_s k Candidate exposure time t k Alternatively, the candidate second conversion gain sg_s k Candidate exposure time t k Candidate aperture LR k (etc., which can be any parameter order). The order of parameters in the third set of parameters to be adjusted is not restricted. The following will take this parameter order as an example. The implementation of other parameter orders is similar, and the subsequent process will not be repeated.

[0221] For example, parameters in the third set of parameters to be adjusted can be adjusted based on a first dynamic range adjustment amount to obtain adjusted parameters. For instance, the first parameter in the third set of parameters to be adjusted can be adjusted first. If the first dynamic range adjustment amount is satisfied, the adjustment process ends. If the first dynamic range adjustment amount is not satisfied, the second parameter in the third set of parameters to be adjusted is adjusted, and so on, until the first dynamic range adjustment amount is satisfied. Based on this, target exposure control parameters and target processing control parameters can be obtained based on the adjusted parameters. The following explains this process.

[0222] Step S33: If the first product of the exposure time and the first dynamic range adjustment amount is less than or equal to the upper limit threshold of the exposure time, then the adjusted exposure time can be determined as the first product value.

[0223] If the first product of the exposure time and the first dynamic range adjustment amount is greater than the upper limit threshold of the exposure time, then the adjusted exposure time is determined to be the upper limit threshold of the exposure time, and step S34 is executed.

[0224] For example, first increase the exposure time t′ k Indicates the candidate's exposure time. Indicates the first dynamic range adjustment amount, t′ k This indicates the adjusted exposure time, i.e., the target exposure time.

[0225] After the exposure time t′ is adjusted k Afterwards, the adjusted exposure time t′ can be... k Compare with the upper limit threshold of exposure time, t_max. If the adjusted exposure time t′ is... k If the exposure time is less than or equal to the upper limit threshold t_max, it indicates that the adjustment result of the exposure time has met the target dynamic range, and the adjusted exposure time t′ should be maintained. k The exposure time t′ remains unchanged, i.e., the adjusted exposure time remains the same. k As the target exposure time, there is no need to adjust other parameters in the third set of parameters to be adjusted, that is, the candidate aperture, candidate second sensor gain and other parameters remain unchanged.

[0226] If the adjusted exposure time t′ k If the exposure time exceeds the upper limit threshold t_max, then the adjusted exposure time t′ will be used. k Updated to t_max, t′ k =t max Adjusted exposure time t max For the target exposure time, proceed to step S34.

[0227] For example, since the first image signal and the second image signal share the same exposure time and aperture, if the exposure time is increased... This indicates that the brightness of both the first and second image signals has increased simultaneously. The gain was increased by a factor of 10, but the compensation gain in the compositing module had not yet been modified at this time. Therefore, the output image of the compositing module became brighter. To reduce the brightness of the synthesized image to its original value, it is also necessary to adjust parameters (such as the digital gain corresponding to the first image signal and / or the first sensor gain corresponding to the first image signal). For example, the digital gain of the first image signal (i.e., the candidate digital gain) can be reduced first, and then the first sensor gain of the first image signal (i.e., the candidate first sensor gain) can be reduced.

[0228] For example, based on the dynamic range extended value dr_rat k With dynamic range initial value dr_rat b The quotient determines the first reverse adjustment amount; for example, the first reverse adjustment amount can be... The first reverse adjustment amount is less than 1.

[0229] If the product of the digital gain corresponding to the first image signal and the first reverse adjustment amount is greater than or equal to the target value (the target value can be the lower limit threshold of digital gain dg), min (The target value can be 1 or any other value, without restriction) then the adjusted digital gain is determined to be the first product value.

[0230] For example, reducing digital gain dg k dg′ represents the candidate digital gain of the first image signal. k This represents the adjusted digital gain. If the adjusted digital gain is dg′ k If the value is greater than or equal to the target value, then the adjusted digital gain dg′ is maintained. k The digital gain dg′ remains unchanged after adjustment. k The target digital gain of the first image signal is used (the candidate digital gain of the second image signal is not adjusted). The candidate first sensor gain of the first image signal is no longer adjusted; dg′ can be executed in the gain processing module of the image processing unit. k The digital gain multiplied by a factor of two makes the brightness of the synthesized image become... This effectively reduces the digital gain of the first image signal by a factor of 1, at which point cdg′ k =cdg k That is, the target compensation gain equals the candidate compensation gain. If the image processing unit has a preprocessing module, the first image signal can also be processed separately in the preprocessing module. The digital gain is times that of the previous one. And digital gain dg′ k =dg k It remains unchanged.

[0231] If the product of the digital gain corresponding to the first image signal and the first inverse adjustment amount is less than the target value, then a first residual adjustment amount is determined based on the target value, the digital gain, and the first inverse adjustment amount. For example, the first residual adjustment amount could be... The first remaining adjustment amount is less than 1. Indicates the first reverse adjustment amount, dg k dg′ represents the digital gain corresponding to the first image signal. k Indicates the target value.

[0232] If the second product of the first sensor gain (such as the candidate first sensor gain corresponding to the first image signal) and the first remaining adjustment amount is greater than or equal to the lower limit threshold of the first sensor gain, then the adjusted digital gain is determined to be the target value and the adjusted first sensor gain is the second product value.

[0233] For example, if the digital gain dg′ k Reduce to the target value (e.g., the lower limit threshold of digital gain dg) min Let the digital gain dg′ be... k To achieve the target value, reduce the gain of the first sensor. sg_l k sg_l′ represents the candidate first sensor gain of the first image signal. k This represents the adjusted gain of the first sensor. If the adjusted gain of the first sensor is sg_l′ k Greater than or equal to the first sensor gain lower limit threshold sg_l min Then maintain the adjusted first sensor gain sg_l′ k The first sensor gain sg_l′ remains unchanged after adjustment. k The target first sensor gain is used as the target value of the first image signal, and the target value is used as the target digital gain of the first image signal. In this case, No other parameters need to be adjusted.

[0234] In one possible approach, if the second product of the first sensor gain (e.g., a candidate first sensor gain) and the first residual adjustment is less than a lower threshold of the first sensor gain, then the second residual adjustment is determined based on the lower threshold of the first sensor gain, the first sensor gain, and the first reverse adjustment. For example, the second residual adjustment could be... The second remaining adjustment amount is less than 1. Indicates the first reverse adjustment amount, sg_l k Indicates the gain of the first candidate sensor, sg_l minThis represents the lower limit threshold of the first sensor gain. Based on this, the adjusted digital gain can be determined as the candidate digital gain (such as the candidate digital gain dg corresponding to the first image signal). k The product of the first sensor gain and the second remaining adjustment is used to determine the adjusted first sensor gain sg_l′. k The first sensor gain lower limit threshold sg_l min In this case, the adjusted digital gain can be less than the target value; that is, the digital gain can be adjusted until the dynamic range requirement is met.

[0235] In another possible approach, if the second product of the candidate first sensor gain and the first residual adjustment is less than the lower limit threshold of the first sensor gain, then the second residual adjustment is determined based on the lower limit threshold of the first sensor gain, the first sensor gain, and the first residual adjustment. For example, the second residual adjustment could be... Indicates the first remaining adjustment amount, dg k dg′ represents the candidate digital gain corresponding to the first image signal. k This represents the target value (i.e., the adjusted digital gain). Based on this, the adjusted digital gain can be determined to be the target value dg′. k The product of the second remaining adjustment amount and the adjusted first sensor gain sg_l′ is determined. k The first sensor gain lower limit threshold sg_l min .

[0236] In summary, if the first product of the exposure time and the first dynamic range adjustment is less than or equal to the upper limit threshold of the exposure time, the target exposure control parameters and the target processing control parameters can be obtained.

[0237] Step S34: Based on the upper limit threshold t_max of exposure time and the exposure time t k and the first dynamic range adjustment amount Determine the first remaining adjustment amount. For example, the first remaining adjustment amount could be...

[0238] If the second product of the aperture and the first remaining adjustment amount is less than or equal to the upper limit threshold of the aperture, then the adjusted exposure time is determined to be the upper limit threshold of the exposure time, and the adjusted aperture is determined to be the second product value.

[0239] If the second product of the aperture and the first remaining adjustment amount is greater than the aperture upper limit threshold, then the adjusted exposure time is determined to be the exposure time upper limit threshold, the adjusted aperture is determined to be the aperture upper limit threshold, and step S35 is executed.

[0240] For example, you can increase the aperture. lr k Indicates the candidate aperture. Indicates the first remaining adjustment amount, lr′ k This indicates the adjusted aperture, i.e., the target aperture.

[0241] After adjustment, the aperture lr′ k After that, you can adjust the aperture lr′. k Compare with the aperture upper limit threshold lr_max. If the adjusted aperture lr′ k If the aperture value is less than or equal to the upper limit threshold lr_max, it indicates that the aperture adjustment result has met the target dynamic range, and the adjusted aperture lr′ can be maintained. k The aperture remains unchanged, i.e., the aperture lr′ remains the same after adjustment. k As the target aperture, there is no need to adjust other parameters in the third set of parameters to be adjusted; that is, parameters such as the gain of the candidate second sensor remain unchanged. Alternatively, if the adjusted aperture lr′ k If the aperture value is greater than the upper limit threshold lr_max, then the adjusted aperture lr′ can be adjusted. k Updated to the aperture upper limit threshold lr_max, i.e., lr′ k =lr_max, then, take the adjusted aperture lr_max as the target aperture, and execute step S35.

[0242] For example, since the first image signal and the second image signal share the same exposure time and aperture, increasing the exposure time and aperture means that the brightness of both the first and second image signals increases simultaneously. In order to reduce the brightness of the synthesized image to its original value, it is also necessary to adjust the parameters (such as the digital gain corresponding to the first image signal and / or the first sensor gain corresponding to the first image signal).

[0243] For example, based on the dynamic range extended value dr_rat k With dynamic range initial value dr_rat bThe first inverse adjustment amount is determined by the quotient. If the first product of the digital gain corresponding to the first image signal and the first inverse adjustment amount is greater than or equal to the target value, the adjusted digital gain is determined to be the first product value. If the first product of the digital gain corresponding to the first image signal and the first inverse adjustment amount is less than the target value, a first residual adjustment amount is determined based on the target value, the digital gain, and the first inverse adjustment amount. If the second product of the first sensor gain and the first residual adjustment amount is greater than or equal to the first sensor gain lower limit threshold, the adjusted digital gain is determined to be the target value, and the adjusted first sensor gain is determined to be the second product value. If the second product of the first sensor gain and the first residual adjustment amount is less than the first sensor gain lower limit threshold, a second residual adjustment amount is determined based on the first sensor gain lower limit threshold, the first sensor gain, and the first inverse adjustment amount, and the adjusted digital gain is determined to be the product of the candidate digital gain and the second residual adjustment amount, and the adjusted first sensor gain sg_l′ is determined. k The first sensor gain lower limit threshold sg_l min At this point, the target exposure control parameters and target processing control parameters can be successfully obtained.

[0244] Step S35: Based on the aperture upper limit threshold lr_max and aperture lr k and the first remaining adjustment amount Determine the second remaining adjustment amount. For example, the second remaining adjustment amount could be...

[0245] If the third product of the second sensor gain (i.e., the candidate second sensor gain) and the second remaining adjustment amount is less than or equal to the upper limit threshold of the second sensor gain, then the adjusted exposure time is determined to be the upper limit threshold of the exposure time, the adjusted aperture is determined to be the upper limit threshold of the aperture, and the adjusted second sensor gain is determined to be the third product value.

[0246] For example, increasing the gain of the second sensor sg_s k For the candidate second sensor gain, sg_s′ k This refers to the adjusted gain of the second sensor, i.e., the target second sensor gain.

[0247] After adjustment, the second sensor gain sg_s′ k Afterwards, the adjusted second sensor gain sg_s′ can be... k With the second sensor gain upper limit threshold sg_s max Compare the results. If the adjusted second sensor gain is sg_s′... k Less than or equal to the upper limit threshold of the second sensor gain sg_s maxThis indicates that the adjustment result of the second sensor gain has met the target dynamic range. Therefore, the adjusted second sensor gain sg_s′ can be maintained. k The second sensor gain sg_s′ remains unchanged after adjustment. k As the target, the second sensor gain.

[0248] If the second sensor gain sg_s′ is adjusted k Less than or equal to the upper limit threshold of the second sensor gain sg_s max A first proportional value is determined based on the quotient of the exposure time and the upper limit threshold of the exposure time; a second proportional value is determined based on the quotient of the aperture and the upper limit threshold of the aperture; and the adjusted digital gain is determined to be the product of the digital gain and the first and second proportional values. dg k For candidate digital gain, dg′ k This refers to the adjusted digital gain, i.e., the target digital gain. The first proportional value, This is the second proportional value. If the adjusted gain of the second sensor is less than or equal to the upper limit threshold of the second sensor gain, then... cdg′ k This represents the adjusted compensation gain, i.e., the target compensation gain, cdg. k Indicates the candidate compensation gain, sg_s k sg_s′ represents the gain of the candidate second sensor. k This indicates the gain of the target's second sensor.

[0249] If the image processing unit has a preprocessing module, the first image signal can also be processed separately in the preprocessing module. The digital gain is increased by a factor of two, making it possible to further enhance the digital capabilities of the system.

[0250] For example, if the third product of the second sensor gain and the second remaining adjustment amount is greater than the upper limit threshold of the second sensor gain, i.e., sg_s′ k >sg_s max Then let the adjusted second sensor gain sg_s′ k Equal to the upper limit threshold of the second sensor gain sg_s max Furthermore, the adjusted exposure time and aperture are both at the upper limit threshold. In this case, there are no parameters that can be adjusted, and the only option is to increase the digital gain of the second image signal. Since the digital gain does not affect the actual dynamic range of the image signal output by the image acquisition unit, no adjustment is needed. It is assumed that the dynamic range provided by the second image signal can only exceed dr_rat. k For any excess dynamic range, it is discarded using a numerical truncation method. For example, dr_rat kSince the second image signal s is 3, after passing through the synthesis processing module and the gain processing module, it is multiplied by 4 to reach the target brightness. Therefore, the value of the segment from s×3 to s×4 can be truncated to s×3.

[0251] Case b: If the dynamic range extension value is dr_rat k Less than the initial value of the dynamic range dr_rat b Then based on the dynamic range initial value dr_rat b With dynamic range extension value dr_rat k The quotient (i.e., the ratio of the two) is used to adjust the parameters.

[0252] Step S41: Based on the dynamic range initial value dr_rat b With dynamic range extension value dr_rat k The quotient determines the first dynamic range adjustment amount. For example, the first dynamic range adjustment amount can be...

[0253] Step S42: Obtain a third set of parameters to be adjusted. This third set of parameters to be adjusted may include, but is not limited to, at least one parameter among candidate exposure control parameters and candidate processing control parameters.

[0254] For example, compared to step S32, in this example only the second sensor gain is adjusted (i.e., the adjustment method used is one in which the brightness of the first image signal is not affected). Therefore, the third set of parameters to be adjusted may include candidate second conversion gains sg_s. k The candidate second conversion gain sg_s in the third set of parameters to be adjusted is adjusted based on the first dynamic range adjustment amount. k Adjustments are made to obtain the adjusted parameters, and the target exposure control parameters and target processing control parameters are obtained based on the adjusted parameters. The following describes this process.

[0255] Step S43: If the product of the second sensor gain and the first dynamic range adjustment amount is less than or equal to the upper limit threshold of the second sensor gain, then the adjusted second sensor gain is determined to be the product value.

[0256] If the product of the second sensor gain and the first dynamic range adjustment is greater than the upper limit threshold of the second sensor gain, then the adjusted second sensor gain is determined to be the upper limit threshold of the second sensor gain.

[0257] For example, increasing the gain of the second sensor sg_s k For the candidate second sensor gain, sg_s′ k This refers to the adjusted gain of the second sensor, i.e., the target second sensor gain.

[0258] After adjustment, the second sensor gain sg_s′ k Afterwards, the adjusted second sensor gain sg_s′ can be... k With the second sensor gain upper limit threshold sg_s max Compare the results. If the adjusted second sensor gain is sg_s′... k Less than or equal to the upper limit threshold of the second sensor gain sg_s max This indicates that the adjustment result of the second sensor gain has met the target dynamic range. Therefore, the adjusted second sensor gain sg_s′ is maintained. k The second sensor gain sg_s′ remains unchanged after adjustment. k As the target second sensor gain. In this case, no other parameters need to be adjusted, and the target compensation gain is used. Target digital gain dg′ k =dg k .

[0259] If the second sensor gain sg_s′ is adjusted k Greater than the upper limit threshold of the second sensor gain sg_s max Then let the adjusted second sensor gain sg_s′ k Equal to the upper limit threshold of the second sensor gain sg_s max That is, the target second sensor gain is the upper limit threshold sg_s of the second sensor gain. max In this case, there are no parameters that can be adjusted; the only option is to increase the digital gain of the second image signal. Since the digital gain does not affect the actual dynamic range of the image signal output by the image acquisition unit, no adjustment is needed. It is assumed that the dynamic range provided by the second image signal can only exceed dr_rat. k For any excess dynamic range, it is discarded using a numerical truncation method. For example, dr_rat k The second image signal s is 3. After passing through the synthesis processing module and the gain processing module, it is multiplied by 4 to reach the target brightness. The value of the segment from s×3 to s×4 is truncated to s×3.

[0260] Case c: If the dynamic range extension value is dr_rat k greater than the initial value of the dynamic range dr_rat b Then based on the dynamic range initial value dr_rat b With dynamic range extension value dr_rat k The quotient (i.e., the ratio of the two) is used to adjust the parameters.

[0261] Step S51: Based on the initial value dr_rat of the dynamic range b With dynamic range extension value dr_rat kThe quotient determines the second dynamic range adjustment amount. For example, if dr_rat k >dr_rat b This indicates that the output image of the gain processing module in the image processing unit provides insufficient dynamic range, failing to reach dr_rat. k In order to ensure that the second image signal can provide exactly dr_rat k To achieve a dynamic range that is several times larger than the given dynamic range, the parameters of the second image signal need to be further reduced. The reduction ratio is the second dynamic range adjustment amount, and this adjustment amount can be... Furthermore, the adjustment amount of the second dynamic range is less than 1.

[0262] Step S52: Obtain a fourth set of parameters to be adjusted. This fourth set of parameters to be adjusted may include, but is not limited to, at least one parameter among candidate exposure control parameters and candidate processing control parameters.

[0263] For example, the parameters to be adjusted include candidate aperture, candidate exposure time, candidate first conversion gain, candidate first sensor gain, candidate second conversion gain, candidate second sensor gain, and candidate compensation gain and candidate digital gain from the candidate exposure control parameters, as well as candidate compensation gain and candidate digital gain from the candidate processing control parameters. Considering that the candidate first conversion gain and candidate second conversion gain are not adjusted, the candidate compensation gain is derived from the candidate conversion gain and candidate sensor gain, the candidate digital gain does not affect the actual dynamic range of the image signal output by the image acquisition unit, and the first sensor gain affects the brightness of the first image signal, therefore, the fourth set of parameters to be adjusted may include the candidate aperture lr. k Candidate exposure time t k and candidate second conversion gain sg_s k .

[0264] For example, the fourth set of parameters to be adjusted may sequentially include candidate second conversion gains sg_s k Candidate aperture LR k Candidate exposure time t k Of course, this is just one example of parameter order; other parameter orders can also be used (such as candidate exposure time t). k Candidate aperture LR k Candidate second conversion gain sg_s k (etc., which can be any parameter order). The order of parameters in the fourth set of parameters to be adjusted is not restricted. The following will take this parameter order as an example. The implementation of other parameter orders is similar, and the subsequent process will not be repeated.

[0265] For example, parameters in the fourth set of parameters to be adjusted can be adjusted based on the second dynamic range adjustment amount to obtain the adjusted parameters. For instance, the first parameter in the fourth set of parameters to be adjusted is first adjusted. If the second dynamic range adjustment amount is satisfied, the adjustment process ends. If the second dynamic range adjustment amount is not satisfied, the second parameter in the fourth set of parameters to be adjusted is adjusted, and so on, until the second dynamic range adjustment amount is satisfied. Based on this, target exposure control parameters and target processing control parameters can be obtained based on the adjusted parameters. The following explains this process.

[0266] Step S53: If the first product of the second sensor gain and the first dynamic range adjustment amount is greater than or equal to the lower limit threshold of the second sensor gain, determine that the adjusted second sensor gain is the first product value. If the first product of the second sensor gain and the first dynamic range adjustment amount is less than the lower limit threshold of the second sensor gain, determine that the adjusted second sensor gain is the lower limit threshold of the second sensor gain, and proceed to step S54.

[0267] For example, first reduce the gain of the second sensor. sg_s k Indicates the candidate second conversion gain, sg_s′ k This indicates the adjusted gain of the second sensor, i.e., the target second sensor gain.

[0268] After adjustment, the second sensor gain sg_s′ k Afterwards, the adjusted second sensor gain sg_s′ can be... k With the second sensor gain lower limit threshold sg_s min Compare the results. If the adjusted second sensor gain is sg_s′... k Greater than or equal to the second sensor gain lower limit threshold sg_s min This indicates that the adjustment result of the second sensor gain has met the target dynamic range, and the adjusted second sensor gain sg_s′ is maintained. k The second sensor gain sg_s′ remains unchanged after adjustment. k As the target second sensor gain, other parameters do not need further adjustment. At this point, the compensation gain... Digital gain dg′ k =dg k .

[0269] If the second sensor gain sg_s′ is adjusted k Less than the second sensor gain lower limit threshold sg_s min Then the adjusted second sensor gain sg_s′ k Updated to sg_s min That is, sg_s minThe target is the gain of the second sensor.

[0270] Step S54: Based on the second sensor gain lower limit threshold sg_s min Second sensor gain sg_s k and the first dynamic range adjustment amount Determine the first remaining adjustment amount. For example, the first remaining adjustment amount could be... The first remaining adjustment amount can be a value less than 1.

[0271] If the second product of the aperture and the first remaining adjustment amount is greater than or equal to the lower limit threshold of the aperture, then the adjusted second sensor gain is determined to be the lower limit threshold of the second sensor gain, and the adjusted aperture is determined to be the second product value.

[0272] If the second product value is less than the lower limit threshold of the aperture, then the adjusted second sensor gain is determined to be the lower limit threshold of the second sensor gain, and the adjusted aperture is determined to be the lower limit threshold of the aperture, and step S55 is executed.

[0273] For example, you can lower the aperture. lr k Indicates the candidate aperture. Indicates the first remaining adjustment amount, lr′ k This indicates the adjusted aperture, i.e., the target aperture.

[0274] After adjustment, the aperture lr′ k After that, you can adjust the aperture lr′. k Compare with the lower aperture threshold lr_min. If the adjusted aperture lr′ k If the aperture value is greater than or equal to the lower aperture threshold lr_min, it indicates that the aperture adjustment has met the target dynamic range and the adjusted aperture lr′ can be maintained. k The aperture remains unchanged, i.e., the aperture lr′ remains the same after adjustment. k As the target aperture, there is no need to adjust other parameters in the fourth set of parameters to be adjusted; that is, parameters such as candidate exposure time remain unchanged. At this point, the compensation gain... Digital gain That is, it can also be based on candidate compensation gain CDG. k Second sensor gain sg_s k Second sensor gain lower limit threshold sg_s min Determine the target compensation gain cdg′ k And based on candidate digital gain dg k , dynamic range initial value dr_rat b Dynamic range extension value dr_rat k Second sensor gain sg_s kSecond sensor gain lower limit threshold sg_s min Determine the target digital gain dg′ k .

[0275] If the aperture is adjusted to lr′ k If the aperture value is less than the lower aperture threshold lr_min, then the adjusted aperture lr′ will be adjusted. k Updated to the lower aperture threshold lr_min, i.e., lr′ k =lr_min, the adjusted aperture lr_min is used as the target aperture.

[0276] Step S55: Based on the aperture lower limit threshold, aperture, and first remaining adjustment amount Determine the second remaining adjustment amount. For example, the second remaining adjustment amount could be...

[0277] If the third product of the exposure time (i.e., the candidate exposure time) and the second remaining adjustment amount is greater than or equal to the lower limit threshold of the exposure time, then the adjusted second sensor gain is determined to be the lower limit threshold of the second sensor gain, the adjusted aperture is determined to be the lower limit threshold of the aperture, and the adjusted exposure time is determined to be the third product value.

[0278] For example, you can reduce the exposure time. t k Indicates the candidate exposure time, t′ k This indicates the adjusted exposure time, i.e., the target exposure time.

[0279] The adjusted exposure time t′ is obtained k Then, adjust the post-exposure time t′ k Compare with the lower limit threshold t_min of exposure time. If the adjusted exposure time t′ is... k If the value is greater than or equal to t_min, it indicates that the adjustment of the exposure time conforms to the target dynamic range, and the adjusted exposure time t′ should be maintained. k The exposure time t′ remains unchanged after adjustment. k As the target exposure time. Compensation gain. Digital gain

[0280] If the product of the exposure time and the third remaining adjustment amount is less than the lower limit threshold of the exposure time, then the adjusted second sensor gain is determined to be the lower limit threshold of the second sensor gain, the adjusted aperture to be the lower limit threshold of the aperture, and the adjusted exposure time to be the lower limit threshold of the exposure time. For example, if the adjusted exposure time t′ k If the exposure time is less than the lower limit threshold t_min, it indicates that the second image signal can no longer reduce its brightness, meaning it cannot provide the required dynamic range. In this case, t′ can be adjusted. k =t_min, This means that the target digital gain can be determined based on candidate aperture, aperture lower limit threshold, candidate exposure time, exposure time lower limit threshold, and candidate digital gain. In this case, the dynamic range extension value corresponding to the actual parameters can be...

[0281] In one possible implementation, since the first image signal and the second image signal share the same exposure time and aperture, the brightness of the first image signal decreases when the exposure time and / or aperture is adjusted. The brightness of the second image signal decreased by a factor of two. To achieve a better signal-to-noise ratio for the first image signal, the brightness of the first image signal can be increased, for example, as follows:

[0282] Increase the gain of the first sensor to If sg_l′ k Less than or equal to the first sensor gain lower limit threshold sg_l max Then there is no need to adjust other parameters to make dg′ k =dg k That's fine. If sg_l′ k Greater than the first sensor gain lower limit threshold sg_l max Then sg_l′ k =sg_l max On this basis, make Among them, sg_l k sg_l′ represents the gain of the first candidate sensor. k This represents the adjusted first sensor gain, i.e., the target's first sensor gain, dg k dg′ represents the candidate digital gain corresponding to the first image signal. k This indicates the adjusted digital gain, i.e., the target digital gain.

[0283] Fifth, cache unit.

[0284] The second exposure control module will adjust the exposure control parameters EXPO for the Kth frame. K (i.e., target exposure control parameters) and the processing control parameters PRO for the Kth frame. K The target processing control parameters (i.e., the target exposure control parameters) are output to the buffer unit. The buffer unit is used to buffer the target exposure control parameters and target processing control parameters output by the second exposure control module.

[0285] Before the exposure begins in frame K, the buffer unit can store the exposure control parameters EXPO for frame KN (N≥1). K-NThe data is input to the image acquisition unit, which then controls the exposure of the Kth frame image based on this exposure control parameter. When the image processing unit begins processing the Kth frame image data, the buffer unit can store the processing control parameter PRO for the KNth frame. K-N The data is input to the image processing unit, which processes the Kth frame image data based on the processing control parameters. The processing method is described in the above embodiment.

[0286] The exposure control parameter EXPO of the buffer unit for the Kth frame K The processing control parameter PRO for the Kth frame K When caching, the exposure control parameter EXPO that has been input to the image acquisition unit can be replaced. K-N and the processing control parameters PRO that have been input to the image processing unit K-N This reduces the consumption of storage space resources. The cache unit controls the exposure control parameter EXPO. K and processing control parameters PRO K After caching, the image is used for the acquisition and processing of the next frame, resulting in an image that balances brightness adjustment and dynamic range expansion.

[0287] In summary, the inputs to the buffer unit are exposure control parameters and processing control parameters. The buffer unit uses cache space to store the exposure control parameters and processing control parameters, and then retrieves them in time to achieve synchronization between the exposure control parameters, processing control parameters and the image acquisition unit and image processing unit.

[0288] For example, the caching unit includes a storage module and a synchronization module. Before the exposure of the Kth frame begins, the synchronization module inputs the exposure control parameters of the KNth frame (N≥1, the value of N is related to the computing power of the logic platform and application requirements) to the image acquisition unit to control the exposure of the Kth frame image. When the image processing unit begins processing the Kth frame image data, the synchronization module inputs the processing control parameters of the KNth frame to the image processing unit, which then processes the Kth frame image data based on the processing control parameters.

[0289] The storage module can cache the exposure control parameters and processing control parameters of each frame of image. The storage module can reuse storage space based on the value of N, thereby reducing resource consumption. For example, when caching the exposure control parameters and processing control parameters of the Kth frame, it can also replace the already synchronized exposure control parameters and processing control parameters of the KNth frame.

[0290] As can be seen from the above technical solutions, in this embodiment, the dual-conversion gain technology is used to obtain an image with a higher signal-to-noise ratio in the image acquisition unit, while eliminating motion blur artifacts. The image dynamic range is adaptively calculated based on the exposure statistics of the current scene, without the need for manual setting. Using the dual-conversion gain technology, two frames with different brightness can be obtained in the image acquisition unit, and the dynamic range can be expanded by a factor far greater than that of a single frame, while also eliminating motion blur artifacts. Image processing integrates multi-segment digital gain and nonlinear curve mapping, achieving a better noise reduction effect. Image data is acquired through the image acquisition unit, image processing and statistics are performed, the required dynamic range is adaptively calculated, and exposure control parameters and processing control parameters are adjusted based on the exposure strategy for the acquisition and processing of the next frame image, obtaining a high dynamic range image with stable brightness, retaining information in bright areas, and increasing the amount of image information.

[0291] The image acquisition unit supports dual conversion gain, enabling it to output two images with different conversion gains within the same exposure time. Compared to wide dynamic range mode, this eliminates motion artifacts, and compared to linear mode, it offers a higher signal-to-noise ratio. The two images can be configured with different sensor gains to achieve a higher dynamic range, further improving the signal-to-noise ratio compared to digital gain. The image acquisition unit supports build-in mode, combining two images with different conversion gains into a single image with a high value range within the unit. It also supports non-linear compression to obtain an image with a low value range, saving bandwidth resources.

[0292] The image processing unit can synthesize two frames of image data to form image data with a high dynamic range, which is then uniformly input into subsequent processing. The overall solution is highly compatible, simple, and consistent. The image processing unit can also input a single frame of image data. It supports non-truncation processing, acquiring images with a high dynamic range to compensate for dark area brightness without losing highlight information, and ensuring the output image's value range is greater than or equal to the input image's value range. The gain processing can be executed once or multiple times at different points in the image processing flow, allowing for flexible configuration based on hardware platform resources or application requirements, reducing resource consumption. The processing control parameters of the image processing unit are provided by the exposure control unit and adaptively adjusted according to the shooting scene, ensuring that images acquired with different exposure control parameters achieve a uniform target brightness, resulting in a more stable high dynamic range output image. Through data compression processing, it supports mapping images with a high dynamic range to images with a low dynamic range, minimizing changes in dark area brightness and focusing on compressing bright areas, preserving highlight information while maintaining appropriate dark area brightness. The data compression process is related to the parameters of gain processing, supporting the compression of high-value-range image data into low-value-range image data within a unified value range. This improves system compatibility, and the value range of the output image is less than or equal to the value range of the input image. Data compression can be performed once or multiple times at different points in the image processing flow, allowing for flexible configuration based on hardware platform resources or application requirements, thus reducing resource consumption.

[0293] The image processing unit can also perform processing such as interpolation, gamma mapping, and contrast enhancement to improve image quality. This processing can be selected based on the resources of the hardware platform or the application requirements.

[0294] The statistical unit can perform brightness statistics processing, analyzing the brightness distribution of the captured scene to assist in more precise exposure control for the next frame, resulting in an image with suitable brightness. The statistical unit can also perform dynamic range statistics processing, analyzing the dynamic range distribution of the captured scene to assist in adjusting the exposure control parameters and processing control parameters for the next frame, resulting in an image with suitable dynamic range. The input images for brightness and dynamic range statistics can be the same or different, allowing for flexible adjustment based on application requirements.

[0295] The exposure control unit can adjust and process exposure control parameters by statistically analyzing brightness values ​​and by adjusting dynamic range extension values, ensuring that subsequent output images simultaneously possess stable effects and adaptive high dynamic range. Exposure control parameters include, but are not limited to, conversion gain, sensor gain, exposure time, and aperture, enabling comprehensive control of the image acquisition unit. Processing control parameters include, but are not limited to, compensation gain and digital gain, ensuring coordinated operation between the image processing unit and the image acquisition unit. When extending the dynamic range, it is necessary to reduce the brightness of the acquired image. The preferred parameter adjustment order is digital gain, sensor gain, aperture, and exposure time, which reduces noise amplification and minimizes flicker. Exposure control parameters control all output image signals from the image acquisition unit, allowing for different sensor gains to be configured for different frames, achieving more flexible dynamic range adjustment and a higher signal-to-noise ratio.

[0296] The caching unit synchronizes the parameters of the image acquisition unit and the image processing unit through storage and synchronization processing, thereby making the output image more stable.

[0297] Based on the same concept as the methods described above, this application proposes an image processing method applied to an image acquisition device. (See also...) Figure 6 As shown, the image processing method may include:

[0298] Step 601: Acquire a first image signal and a second image signal based on exposure control parameters; wherein, the exposure control parameters are determined based on the target statistical information corresponding to the preset number of frames acquired before the current frame; wherein, the image sensor supports high conversion gain and low conversion gain, outputting the first image signal when using high conversion gain and outputting the second image signal when using low conversion gain.

[0299] Step 602: Based on the processing control parameters, synthesize the first image signal and the second image signal to obtain a synthesized image. Perform gain processing and data compression processing on the synthesized image to obtain the output image and output the output image. Wherein, when performing gain processing, no data truncation is performed, or the data is truncated to a value range not less than the value range of the input image. Wherein, the compression parameters used in the data compression processing are related to the gain used in the gain processing.

[0300] Step 603: Obtain target exposure control parameters and target processing control parameters based on target statistical information; wherein, the target exposure control parameters are used to control the acquisition of the preset number of frames after the current frame, and the target processing control parameters are used to control the processing of the preset number of frames after the current frame.

[0301] For example, the exposure control parameters include aperture, exposure time, a first conversion gain and a first sensor gain corresponding to the first image signal, and a second conversion gain and a second sensor gain corresponding to the second image signal; the second conversion gain may be the same as or different from the first conversion gain, and the second sensor gain may be the same as or different from the first sensor gain. Based on this, acquiring the first image signal and the second image signal based on the exposure control parameters may include, but is not limited to: acquiring light signals corresponding to the aperture and exposure time;

[0302] After receiving the optical signal, the dual-conversion gain switch of the image sensor is opened, and the first image signal corresponding to the optical signal is determined based on the first conversion gain and the first sensor gain. After the first image signal is acquired, the dual-conversion gain switch is closed, and the second image signal corresponding to the optical signal is determined based on the second conversion gain and the second sensor gain. Alternatively, after receiving the optical signal, the dual-conversion gain switch of the image sensor is closed, and the second image signal corresponding to the optical signal is determined based on the second conversion gain and the second sensor gain. After the second image signal is acquired, the dual-conversion gain switch is opened, and the first image signal corresponding to the optical signal is determined based on the first conversion gain and the first sensor gain.

[0303] For example, the processing control parameters may include a compensation gain, which can be a ratio between a first gain value and a second gain value. The first gain value is the product of a first conversion gain and a first sensor gain, and the second gain value is the product of a second conversion gain and a second sensor gain. The composite image is obtained by synthesizing the first image signal and the second image signal based on the processing control parameters, which may include, but is not limited to: if the brightness of the first image signal is greater than the brightness of the second image signal, then multiplying the second image signal by the compensation gain to obtain a gain-compensated image; wherein, no data truncation is performed when multiplying the second image signal by the compensation gain, and the value range of the gain-compensated image is greater than or equal to the value range of the second image signal; and the composite image is obtained by synthesizing the first image signal and the gain-compensated image.

[0304] For example, synthesizing a composite image by combining a first image signal and a gain-compensated image may include: dividing the image region of the first image signal into a first region and a second region, wherein the brightness in the first region is not greater than a composite brightness threshold and the brightness in the second region is greater than the composite brightness threshold; and obtaining the composite image by using the first image signal in the first region and the gain-compensated image in the second region.

[0305] For example, synthesizing a composite image by combining a first image signal and a second image signal based on processing control parameters may include, but is not limited to: preprocessing the first image signal and the second image signal respectively; and synthesizing the preprocessed first image signal and the preprocessed second image signal based on processing control parameters to obtain a composite image; wherein, the preprocessing includes at least one of the following: gain processing, black level processing, noise reduction processing, white balance processing, and deblurring processing;

[0306] The composite image is subjected to gain processing and data compression processing to obtain an output image, which may include, but is not limited to: performing gain processing on the composite image to obtain a gain-processed image; performing data compression processing on the gain-processed image to obtain a compressed image; determining the compressed image as the output image, or performing post-processing on the compressed image to obtain the output image; wherein, the post-processing includes at least one of the following: noise reduction processing, enhancement processing, dehazing processing, deblurring processing, and sharpening processing;

[0307] The processing control parameters include digital gain, which involves multiplying the synthesized image by the digital gain. When multiplying the synthesized image by the digital gain, data is not truncated or is truncated to a value range not less than that of the input image, and the value range of the image after gain processing is greater than or equal to that of the synthesized image. The data compression processing involves using a preset nonlinear mapping curve to perform brightness mapping on the image after gain processing, and the nonlinear mapping curve is associated with the digital gain.

[0308] For example, when performing data compression on the image after gain processing, the compression parameters used in the data compression process are related to the gain used in the gain processing. This includes: under different gains, the brightness value range of the image obtained by gain processing is different, and the compression parameters of the data compression process are adjusted according to different brightness value ranges so that the compressed image has the same brightness value range under different gains.

[0309] For example, performing gain processing and data compression processing on the synthesized image may include: if the digital gain is greater than a preset gain threshold, then splitting the digital gain into a first digital gain value and a second digital gain value, the first digital gain value being equal to the preset gain threshold, and the product of the first digital gain value and the second digital gain value being equal to the digital gain; performing gain processing on the synthesized image based on the first digital gain value to obtain a first intermediate image; performing data compression processing on the first intermediate image to obtain a second intermediate image; performing intermediate processing on the second intermediate image to obtain a third intermediate image; performing gain processing on the third intermediate image based on the second digital gain value to obtain a gain-processed image; performing data compression processing on the gain-processed image to obtain a compressed image; and determining the image to be output based on the compressed image.

[0310] For example, when performing data compression on the image after gain processing, the image after gain processing is divided into different regions, and the mapping curve corresponding to each region is obtained; for each region of the image after gain processing, the region is mapped using the mapping curve corresponding to that region to obtain the compressed image.

[0311] For example, the target statistical information includes a dynamic range expansion value obtained based on a first statistical image. The first statistical image can be an image before synthesis processing, an image after synthesis processing, an image before gain processing, an image after gain processing, an image before data compression processing, or an image after data compression processing. Based on this, brightness statistics can also be performed on the bright areas of the first statistical image to obtain bright statistical values. Based on the difference between the bright statistical values ​​and the bright target values, and a preset dynamic range expansion value threshold, a dynamic range expansion value is determined. The preset dynamic range expansion value threshold is determined based on the computing power or noise reduction capability of the image processing logic platform, so that the noise amplification of digital processing based on the dynamic range expansion value is within a preset range.

[0312] For example, the target statistical information includes statistical brightness values ​​obtained based on a second statistical image. The second statistical image can be an image before synthesis processing, an image after synthesis processing, an image before gain processing, an image after gain processing, an image before data compression processing, or an image after data compression processing. Based on this, brightness statistics can also be performed on the second statistical image to obtain statistical brightness values. The statistical brightness values ​​can be determined by: taking the average value of the second statistical image; or, setting different weight values ​​for image regions with different brightness in the second statistical image and performing a weighted average of the second statistical image to obtain statistical brightness values; or, setting different weight values ​​for regions at different spatial locations in the second statistical image and performing a weighted average of the second statistical image to obtain statistical brightness values; or, setting a preset brightness statistical threshold, taking the smaller value between the second statistical image and the brightness statistical threshold, and performing a weighted average of the second statistical image to obtain statistical brightness values.

[0313] For example, the target statistical information includes dynamic range expansion value and statistical brightness value; obtaining target exposure control parameters and target processing control parameters based on the target statistical information may include, but is not limited to: adjusting historical exposure control parameters and historical processing control parameters based on the statistical brightness value to obtain candidate exposure control parameters and candidate processing control parameters; the first parameter adjustment is determined based on the adjusted statistical brightness value and the preset target brightness; adjusting the candidate exposure control parameters and candidate processing control parameters based on the dynamic range expansion value to obtain target exposure control parameters and target processing control parameters; the second parameter adjustment is determined based on the adjusted target gain value and the adjusted dynamic range expansion value, and the target gain value is the product of compensation gain and digital gain.

[0314] For example, adjusting historical exposure control parameters and historical processing control parameters based on statistical brightness values ​​to obtain candidate exposure control parameters and candidate processing control parameters may include, but is not limited to: if the statistical brightness value is less than a preset target brightness, determining a first brightness adjustment amount based on the quotient of the preset target brightness and the statistical brightness value, wherein the first brightness adjustment amount is greater than 1; obtaining a first set of parameters to be adjusted, which includes at least one parameter from the historical exposure control parameters and historical processing control parameters, and adjusting the parameters in the first set of parameters to be adjusted based on the first brightness adjustment amount to obtain adjusted parameters; and obtaining candidate exposure control parameters and candidate processing control parameters based on the adjusted parameters.

[0315] For example, the historical exposure control parameters include at least aperture, exposure time, first conversion gain, first sensor gain, second conversion gain, and second sensor gain; the historical processing control parameters include at least compensation gain and digital gain; multiple parameters in the first set of parameters to be adjusted sequentially include exposure time, aperture, first sensor gain, second sensor gain, and digital gain; the parameters in the first set of parameters to be adjusted are adjusted based on a first brightness adjustment amount to obtain the adjusted parameters, which may include, but are not limited to: if the first product of exposure time and the first brightness adjustment amount is less than or equal to the upper limit threshold of exposure time, then the adjusted exposure time is determined to be the first product value; if the first product value is greater than the upper limit threshold of exposure time, then a first remaining adjustment amount is determined based on the upper limit threshold of exposure time, exposure time, and the first brightness adjustment amount; if the second product of aperture and the first remaining adjustment amount is less than or equal to the upper limit threshold of aperture, then the adjusted exposure time is determined to be the upper limit threshold of exposure time, and the adjusted aperture is determined to be the second product value. If the second product value is greater than the aperture upper limit threshold, then the second remaining adjustment amount is determined based on the aperture upper limit threshold, the aperture, and the first remaining adjustment amount. If the third product value of the first sensor gain and the second remaining adjustment amount is less than or equal to the first sensor gain upper limit threshold, then the adjusted exposure time is determined to be the exposure time upper limit threshold, the adjusted aperture is determined to be the aperture upper limit threshold, the adjusted first sensor gain is determined to be the third product value, and the adjusted second sensor gain is determined to be the product value of the second sensor gain and the second remaining adjustment amount. If the third product value is greater than the first sensor gain upper limit threshold, then the third remaining adjustment amount is determined based on the first sensor gain upper limit threshold, the first sensor gain, and the second remaining adjustment amount. The adjusted exposure time is determined to be the exposure time upper limit threshold, the adjusted aperture is determined to be the aperture upper limit threshold, the adjusted first sensor gain is determined to be the first sensor gain upper limit threshold, the adjusted second sensor gain is determined to be the second sensor gain upper limit threshold, and the adjusted digital gain is determined to be the product value of the digital gain and the third remaining adjustment amount.

[0316] For example, adjusting historical exposure control parameters and historical processing control parameters based on statistical brightness values ​​to obtain candidate exposure control parameters and candidate processing control parameters may include, but is not limited to: if the statistical brightness value is greater than a preset target brightness, determining a second brightness adjustment amount based on the quotient of the preset target brightness and the statistical brightness value, wherein the second brightness adjustment amount is less than 1; obtaining a second set of parameters to be adjusted, which includes at least one parameter from the historical exposure control parameters and historical processing control parameters, and adjusting the parameters in the second set of parameters to be adjusted based on the second brightness adjustment amount to obtain adjusted parameters; and obtaining candidate exposure control parameters and candidate processing control parameters based on the adjusted parameters.

[0317] For example, the historical exposure control parameters include at least aperture, exposure time, first conversion gain, first sensor gain, second conversion gain, and second sensor gain; the historical processing control parameters include at least compensation gain and digital gain; multiple parameters in the second set of parameters to be adjusted sequentially include digital gain, first sensor gain, second sensor gain, aperture, and exposure time; the parameters in the second set of parameters to be adjusted are adjusted based on a second brightness adjustment amount to obtain the adjusted parameters, which may include, but are not limited to: if the first product of digital gain and the second brightness adjustment amount is greater than or equal to a lower limit threshold of digital gain, then the adjusted digital gain is determined to be the first product value; if the first product value is less than the lower limit threshold of digital gain, then a first remaining adjustment amount is determined based on the lower limit threshold of digital gain, digital gain, and second brightness adjustment amount; if the second product of the first sensor gain and the first remaining adjustment amount is greater than or equal to the lower limit threshold of first sensor gain, then the adjusted digital gain is determined to be the lower limit threshold of digital gain, the adjusted first sensor gain is determined to be the second product value, and the adjusted second sensor gain is determined to be the second sensor gain and the second brightness adjustment amount. The product of the first remaining adjustment amount; if the second product value is less than the first sensor gain lower limit threshold, then the second remaining adjustment amount is determined based on the first sensor gain lower limit threshold, the first sensor gain, and the first remaining adjustment amount; if the third product value of the aperture and the second remaining adjustment amount is greater than or equal to the aperture lower limit threshold, then the adjusted digital gain is determined to be the digital gain lower limit threshold, the adjusted first sensor gain is determined to be the first sensor gain lower limit threshold, the adjusted second sensor gain is determined to be the second sensor gain lower limit threshold, and the adjusted aperture is determined to be the aperture and the third product value of the second remaining adjustment amount; if the third product value is less than the aperture lower limit threshold, then the third remaining adjustment amount is determined based on the aperture lower limit threshold, the aperture, and the second remaining adjustment amount; if the fourth product value of the exposure time and the third remaining adjustment amount is greater than or equal to the exposure time lower limit threshold, then the adjusted digital gain is determined to be the digital gain lower limit threshold, the adjusted first sensor gain is determined to be the first sensor gain lower limit threshold, the adjusted second sensor gain is determined to be the second sensor gain lower limit threshold, the adjusted aperture is determined to be the aperture lower limit threshold, and the adjusted exposure time is determined to be the fourth product value.

[0318] For example, adjusting the candidate exposure control parameters and candidate processing control parameters based on the dynamic range extension value to obtain the target exposure control parameters and target processing control parameters may include, but is not limited to: if the dynamic range extension value is greater than 1, and the candidate processing control parameters include compensation gain and digital gain, then determining the initial dynamic range value based on the product of compensation gain and digital gain; if the dynamic range extension value is less than the initial dynamic range value, then determining the first dynamic range adjustment amount based on the quotient of the initial dynamic range value and the dynamic range extension value; obtaining a third set of parameters to be adjusted, the third set of parameters to be adjusted including at least one parameter from the candidate exposure control parameters and candidate processing control parameters, and adjusting the parameters in the third set of parameters to be adjusted based on the first dynamic range adjustment amount to obtain the adjusted parameters; and obtaining the target exposure control parameters and target processing control parameters based on the adjusted parameters.

[0319] For example, candidate exposure control parameters may include at least aperture, exposure time, first conversion gain, first sensor gain, second conversion gain, and second sensor gain; candidate processing control parameters may include at least compensation gain and digital gain; the parameters in the third set of parameters to be adjusted may sequentially include exposure time, aperture, and second sensor gain; the parameters in the third set of parameters to be adjusted are adjusted based on the first dynamic range adjustment amount to obtain the adjusted parameters, which may include, but are not limited to: if the first product value of exposure time and the first dynamic range adjustment amount is less than or equal to the upper limit threshold of exposure time, then the adjusted exposure time is determined to be the first product value; if the first product value is greater than the upper limit threshold of exposure time, then... A first remaining adjustment amount is determined based on the upper limit threshold of exposure time, the exposure time, and the first dynamic range adjustment amount. If the second product of the aperture and the first remaining adjustment amount is less than or equal to the upper limit threshold of aperture, the adjusted exposure time is determined to be the upper limit threshold of exposure time, and the adjusted aperture is determined to be the second product value. If the second product value is greater than the upper limit threshold of aperture, a second remaining adjustment amount is determined based on the upper limit threshold of aperture, the aperture, and the first remaining adjustment amount. If the third product of the second sensor gain and the second remaining adjustment amount is less than or equal to the upper limit threshold of second sensor gain, the adjusted exposure time is determined to be the upper limit threshold of exposure time, the adjusted aperture is determined to be the upper limit threshold of aperture, and the adjusted second sensor gain is determined to be the third product value.

[0320] For example, if the first product value is less than or equal to the upper limit threshold of exposure time, or if the first product value is greater than the upper limit threshold of exposure time and the second product value is less than or equal to the upper limit threshold of aperture, the first inverse adjustment amount can be determined based on the quotient of the dynamic range extension value and the initial value of dynamic range; if the first product value of the digital gain corresponding to the first image signal and the first inverse adjustment amount is greater than or equal to the target value, then the adjusted digital gain is determined to be the first product value; if the first product value is less than the target value, the first residual adjustment amount is determined based on the target value, the digital gain, and the first inverse adjustment amount; if the second product value of the first sensor gain and the first residual adjustment amount is greater than or equal to the lower limit threshold of the first sensor gain, then the adjusted digital gain is determined to be the target value and the adjusted first sensor gain is determined to be the second product value; if the second product value is less than the lower limit threshold of the first sensor gain, then the second residual adjustment amount is determined based on the lower limit threshold of the first sensor gain, the first sensor gain, and the first inverse adjustment amount; the adjusted digital gain is determined to be the product value of the digital gain and the second residual adjustment amount, and the adjusted first sensor gain is determined to be the lower limit threshold of the first sensor gain.

[0321] If the first product value is greater than the upper limit threshold of exposure time, the second product value is greater than the upper limit threshold of aperture, and the third product value is less than or equal to the upper limit threshold of second sensor gain, the first proportional value can be determined based on the quotient of exposure time and the upper limit threshold of exposure time, the second proportional value can be determined based on the quotient of aperture and the upper limit threshold of aperture, and the adjusted digital gain can be determined as the product of digital gain and the first proportional value and the second proportional value.

[0322] For example, the candidate exposure control parameters include at least aperture, exposure time, first conversion gain, first sensor gain, second conversion gain, and second sensor gain; the candidate processing control parameters include at least compensation gain and digital gain; the parameters in the third set of parameters to be adjusted include the second sensor gain; the parameters in the third set of parameters to be adjusted are adjusted based on the first dynamic range adjustment amount to obtain the adjusted parameters, which may include, but are not limited to: if the product of the second sensor gain and the first dynamic range adjustment amount is less than or equal to the upper limit threshold of the second sensor gain, then the adjusted second sensor gain is determined to be the product value; if the product of the second sensor gain and the first dynamic range adjustment amount is greater than the upper limit threshold of the second sensor gain, then the adjusted second sensor gain is determined to be the upper limit threshold of the second sensor gain.

[0323] For example, adjusting the candidate exposure control parameters and candidate processing control parameters based on the dynamic range extension value to obtain the target exposure control parameters and target processing control parameters may include, but are not limited to: if the dynamic range extension value is greater than 1, and the candidate processing control parameters include compensation gain and digital gain, then determining the initial dynamic range value based on the product of compensation gain and digital gain; if the dynamic range extension value is greater than the initial dynamic range value, then determining the second dynamic range adjustment amount based on the quotient of the initial dynamic range value and the dynamic range extension value; obtaining a fourth set of parameters to be adjusted, which includes at least one parameter from the candidate exposure control parameters and candidate processing control parameters, and adjusting the parameters in the fourth set of parameters to be adjusted based on the second dynamic range adjustment amount to obtain the adjusted parameters; and obtaining the target exposure control parameters and target processing control parameters based on the adjusted parameters.

[0324] For example, candidate exposure control parameters include at least aperture, exposure time, first conversion gain, first sensor gain, second conversion gain, and second sensor gain; candidate processing control parameters include at least compensation gain and digital gain; the parameters in the fourth set of parameters to be adjusted sequentially include second sensor gain, aperture, and exposure time; the parameters in the fourth set of parameters to be adjusted are adjusted based on the second dynamic range adjustment amount to obtain the adjusted parameters, including: if the first product of the second sensor gain and the first dynamic range adjustment amount is greater than or equal to the lower limit threshold of the second sensor gain, then the adjusted second sensor gain is determined to be the first product value; if the first product value is less than the lower limit threshold of the second sensor gain, then a first residual adjustment is determined based on the lower limit threshold of the second sensor gain, the second sensor gain, and the first dynamic range adjustment amount. The adjustment is performed as follows: If the second product of the aperture and the first remaining adjustment amount is greater than or equal to the lower aperture threshold, then the adjusted second sensor gain is determined to be the lower aperture threshold and the adjusted aperture is determined to be the second product value; if the second product value is less than the lower aperture threshold, then the second remaining adjustment amount is determined based on the lower aperture threshold, the aperture, and the first remaining adjustment amount; if the third product of the exposure time and the second remaining adjustment amount is greater than or equal to the lower exposure time threshold, then the adjusted second sensor gain is determined to be the lower aperture threshold, the adjusted aperture is determined to be the lower aperture threshold, and the adjusted exposure time is determined to be the third product value; if the third product value is less than the lower exposure time threshold, then the adjusted second sensor gain is determined to be the lower aperture threshold, the adjusted aperture is determined to be the lower aperture threshold, and the adjusted exposure time is determined to be the lower exposure time threshold.

[0325] Based on the same application concept as the above method, this application proposes an electronic device (i.e., an image acquisition device), see [link to relevant documentation]. Figure 7As shown, the electronic device includes a processor 71 and a machine-readable storage medium 72, the machine-readable storage medium 72 storing machine-executable instructions that can be executed by the processor 71; the processor 71 is used to execute the machine-executable instructions to implement the image processing method of the example above in this application.

[0326] Based on the same concept as the above method, this application also provides a machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, can implement the image processing method disclosed in the above examples of this application.

[0327] The aforementioned machine-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0328] The systems, devices, modules, or units described in the above embodiments can be implemented by a computer entity or by a product with a certain function. A typical implementation device is a computer, which can be a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0329] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0330] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0331] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0332] Furthermore, these computer program instructions can also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0333] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0334] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An image processing method, characterized in that, The method includes: A first image signal and a second image signal are acquired based on exposure control parameters; wherein the exposure control parameters are determined based on target statistical information corresponding to a preset number of frames of images acquired before the current frame; wherein the image sensor supports high conversion gain and low conversion gain, outputting a first image signal when using the high conversion gain and outputting a second image signal when using the low conversion gain; Based on processing control parameters, the first image signal and the second image signal are synthesized to obtain a synthesized image. Gain processing and data compression processing are then performed on the synthesized image to obtain an output image, which is then output. Specifically, during gain processing, data truncation is not performed, or the data is truncated to a value range not smaller than the input image's value range. The compression parameters used during data compression processing are related to the gain used during gain processing. Under different gains, the brightness value range of the image obtained through gain processing is different. The compression parameters of the data compression processing are adjusted according to different brightness value ranges so that the compressed image has the same brightness value range under different gains. Target exposure control parameters and target processing control parameters are obtained based on target statistical information; wherein, the target exposure control parameters are used to control the acquisition of a preset number of frames after the current frame, and the target processing control parameters are used to control the processing of a preset number of frames after the current frame.

2. The method according to claim 1, characterized in that, The exposure control parameters include aperture, exposure time, first conversion gain and first sensor gain corresponding to the first image signal, and second conversion gain and second sensor gain corresponding to the second image signal; the second conversion gain may be the same as or different from the first conversion gain, and the second sensor gain may be the same as or different from the first sensor gain. The acquisition of the first image signal and the second image signal based on exposure control parameters includes: Acquire light signals corresponding to the aperture and the exposure time; After obtaining the optical signal, the dual conversion gain switch of the image sensor is opened, and the first image signal corresponding to the optical signal is determined based on the first conversion gain and the first sensor gain; after the first image signal is acquired, the dual conversion gain switch is closed, and the second image signal corresponding to the optical signal is determined based on the second conversion gain and the second sensor gain. Alternatively, after obtaining the optical signal, the dual conversion gain switch of the image sensor is closed, and a second image signal corresponding to the optical signal is determined based on the second conversion gain and the second sensor gain; after the second image signal is acquired, the dual conversion gain switch is opened, and a first image signal corresponding to the optical signal is determined based on the first conversion gain and the first sensor gain.

3. The method according to claim 2, characterized in that, The processing control parameters include a compensation gain, which is the ratio between a first gain value and a second gain value. The first gain value is the product of a first conversion gain and a first sensor gain, and the second gain value is the product of a second conversion gain and a second sensor gain. The method of synthesizing the first image signal and the second image signal based on processing control parameters to obtain a composite image includes: if the brightness of the first image signal is greater than the brightness of the second image signal, then multiplying the second image signal by the compensation gain to obtain a gain-compensated image; wherein, when the second image signal is multiplied by the compensation gain, no data truncation is performed, and the value range of the gain-compensated image is greater than or equal to the value range of the second image signal. The first image signal and the gain-compensated image are combined to obtain the composite image.

4. The method according to claim 3, characterized in that, The process of synthesizing the first image signal and the gain-compensated image to obtain the synthesized image includes: The image region of the first image signal is divided into a first region and a second region, wherein the brightness in the first region is not greater than the composite brightness threshold, and the brightness in the second region is greater than the composite brightness threshold. The composite image is obtained by using the first image signal in the first region and the gain-compensated image in the second region.

5. The method according to claim 1 or 3, characterized in that, The process of synthesizing the first image signal and the second image signal based on processing control parameters to obtain a synthesized image includes: The first image signal and the second image signal are preprocessed respectively; A composite image is obtained by synthesizing the pre-processed first image signal and the pre-processed second image signal based on the processing control parameters; wherein, the pre-processing includes at least one of the following: gain processing, black level processing, noise reduction processing, white balance processing, and deblurring processing; The synthesized image is subjected to gain processing and data compression processing to obtain the output image, including: The synthesized image is subjected to gain processing to obtain a gain-processed image; the gain-processed image is subjected to data compression processing to obtain a compressed image; the compressed image is determined as the image to be output, or the compressed image is subjected to post-processing to obtain the image to be output; wherein, the post-processing includes at least one of the following: noise reduction processing, enhancement processing, dehazing processing, deblurring processing, and sharpening processing; The processing control parameters include digital gain, and the gain processing involves multiplying the synthesized image by the digital gain. When the synthesized image is multiplied by the digital gain, no data truncation is performed or the data is trunculated to a value range that is not less than the value range of the input image, and the value range of the image after the gain processing is greater than or equal to the value range of the synthesized image. The data compression process involves using a preset nonlinear mapping curve to perform brightness mapping on the gain-processed image, and the nonlinear mapping curve is associated with the digital gain.

6. The method according to claim 5, characterized in that, The gain processing and data compression processing of the synthesized image include: If the digital gain is greater than a preset gain threshold, the digital gain is split into a first digital gain value and a second digital gain value, the first digital gain value is equal to the preset gain threshold, and the product of the first digital gain value and the second digital gain value is equal to the digital gain. The first intermediate image is obtained by performing gain processing on the synthesized image based on the first digital gain value. The first intermediate image is compressed to obtain the second intermediate image; The second intermediate image is processed to obtain the third intermediate image; The third intermediate image is subjected to gain processing based on the second digital gain value to obtain the gain-processed image; The image after gain processing is compressed to obtain a compressed image. The image to be output is determined based on the compressed image.

7. The method according to claim 1, characterized in that, When performing data compression on the gain-processed image, the gain-processed image is divided into different regions, and the mapping curve corresponding to each region is obtained; for each region of the gain-processed image, the region is mapped using the mapping curve corresponding to that region to obtain the compressed image.

8. The method according to claim 1, characterized in that, The target statistical information includes a dynamic range expansion value obtained based on a first statistical image, wherein the first statistical image is an image before synthesis processing, an image after synthesis processing, an image before gain processing, an image after gain processing, an image before data compression processing, or an image after data compression processing. The method further includes: Brightness statistics are obtained by performing brightness statistics on the bright areas of the first statistical image; The dynamic range expansion value is determined based on the difference between the brightness statistics and the brightness target value, and a preset dynamic range expansion value threshold. The preset dynamic range expansion value threshold is determined based on the computing power or noise reduction capability of the image processing logic platform, so that the noise amplification of digital processing based on the dynamic range expansion value is within a preset range.

9. The method according to claim 1, characterized in that, The target statistical information includes statistical brightness values ​​obtained based on a second statistical image, wherein the second statistical image is an image before synthesis processing, an image after synthesis processing, an image before gain processing, an image after gain processing, an image before data compression processing, or an image after data compression processing. The method further includes: The second statistical image is subjected to brightness statistics to obtain the statistical brightness value; The statistical brightness value is determined as follows: The statistical brightness value is obtained by calculating the mean of the second statistical image; or, Different weight values ​​are assigned to image regions of different brightness levels in the second statistical image, and the statistical brightness value is obtained by performing a weighted average of the second statistical image; or... Different weight values ​​are assigned to regions at different spatial locations in the second statistical image, and the statistical brightness value is obtained by performing a weighted average of the second statistical image; or... A preset brightness statistics threshold is set, and the second statistical image is taken as the smaller of the brightness statistics threshold and the second statistical image is then weighted to obtain the statistical brightness value.

10. The method according to claim 1, characterized in that, The target statistical information includes dynamic range expansion value and statistical brightness value; the acquisition of target exposure control parameters and target processing control parameters based on the target statistical information includes: Based on the statistical brightness value, the historical exposure control parameters and historical processing control parameters are adjusted by a first parameter to obtain candidate exposure control parameters and candidate processing control parameters; wherein, the first parameter adjustment is determined based on the adjusted statistical brightness value and the preset target brightness. The candidate exposure control parameters and the candidate processing control parameters are adjusted by a second parameter based on the dynamic range extension value to obtain the target exposure control parameters and the target processing control parameters; wherein, the second parameter adjustment is determined based on the adjusted target gain value and the adjusted dynamic range extension value, and the target gain value is the product of the compensation gain and the digital gain.

11. The method according to claim 10, characterized in that, The step of adjusting the candidate exposure control parameters and candidate processing control parameters based on the dynamic range extension value to obtain the target exposure control parameters and target processing control parameters includes: If the dynamic range extension value is greater than 1, and the candidate processing control parameters include compensation gain and digital gain, then the initial value of the dynamic range is determined based on the product of the compensation gain and the digital gain. If the dynamic range extension value is less than the dynamic range initial value, then a first dynamic range adjustment amount is determined based on the quotient of the dynamic range initial value and the dynamic range extension value; Obtain a third set of parameters to be adjusted, the third set of parameters to be adjusted including at least one of the candidate exposure control parameters and the candidate processing control parameters, and adjust the parameters in the third set of parameters to be adjusted based on the first dynamic range adjustment amount to obtain the adjusted parameters; Based on the adjusted parameters, the target exposure control parameters and target processing control parameters are obtained.

12. The method according to claim 11, characterized in that, The candidate exposure control parameters include at least aperture, exposure time, first conversion gain, first sensor gain, second conversion gain and second sensor gain, and the candidate processing control parameters include at least compensation gain and digital gain. The parameters in the third set of parameters to be adjusted sequentially include the exposure time, the aperture, and the second sensor gain; adjusting the parameters in the third set of parameters to be adjusted based on the first dynamic range adjustment amount to obtain the adjusted parameters includes: If the first product of the exposure time and the first dynamic range adjustment amount is less than or equal to the upper limit threshold of the exposure time, then the adjusted exposure time is determined to be the first product value. If the first product value is greater than the upper limit threshold of the exposure time, then a first remaining adjustment amount is determined based on the upper limit threshold of the exposure time, the exposure time, and the first dynamic range adjustment amount; If the second product of the aperture and the first remaining adjustment amount is less than or equal to the upper limit threshold of the aperture, then the adjusted exposure time is determined to be the upper limit threshold of the exposure time and the adjusted aperture is determined to be the second product value. If the second product value is greater than the aperture upper limit threshold, then the second remaining adjustment amount is determined based on the aperture upper limit threshold, the aperture, and the first remaining adjustment amount; If the third product of the second sensor gain and the second remaining adjustment amount is less than or equal to the upper limit threshold of the second sensor gain, then the adjusted exposure time is determined to be the upper limit threshold of the exposure time, the adjusted aperture is determined to be the upper limit threshold of the aperture, and the adjusted second sensor gain is determined to be the third product value.

13. The method according to claim 12, characterized in that, If the first product value is less than or equal to the upper limit threshold of exposure time, or if the first product value is greater than the upper limit threshold of exposure time and the second product value is less than or equal to the upper limit threshold of aperture, the method further includes: The first reverse adjustment amount is determined based on the quotient of the dynamic range extension value and the dynamic range initial value; If the first product of the digital gain corresponding to the first image signal and the first inverse adjustment amount is greater than or equal to the target value, then the adjusted digital gain is determined to be the first product value. If the first product value is less than the target value, then a first remaining adjustment amount is determined based on the target value, the digital gain, and the first reverse adjustment amount; if the second product value of the first sensor gain and the first remaining adjustment amount is greater than or equal to the first sensor gain lower limit threshold, then the adjusted digital gain is determined to be the target value and the adjusted first sensor gain is determined to be the second product value. If the second product value is less than the first sensor gain lower limit threshold, then a second remaining adjustment amount is determined based on the first sensor gain lower limit threshold, the first sensor gain, and the first reverse adjustment amount; the adjusted digital gain is determined to be the product of the digital gain and the second remaining adjustment amount, and the adjusted first sensor gain is determined to be the first sensor gain lower limit threshold.

14. The method according to claim 12, characterized in that, If the first product value is greater than the upper limit threshold of the exposure time, and the second product value is greater than the upper limit threshold of the aperture, and the third product value is less than or equal to the upper limit threshold of the second sensor gain, the method further includes: A first ratio value is determined based on the quotient of the exposure time and the upper limit threshold of the exposure time; a second ratio value is determined based on the quotient of the aperture and the upper limit threshold of the aperture; and the adjusted digital gain is determined to be the product of the digital gain and the first ratio value and the second ratio value.

15. The method according to claim 11, characterized in that, The candidate exposure control parameters include at least aperture, exposure time, first conversion gain, first sensor gain, second conversion gain, and second sensor gain; the candidate processing control parameters include at least compensation gain and digital gain; the parameters in the third set of parameters to be adjusted include the second sensor gain; adjusting the parameters in the third set of parameters to be adjusted based on the first dynamic range adjustment amount to obtain the adjusted parameters includes: If the product of the second sensor gain and the first dynamic range adjustment is less than or equal to the upper limit threshold of the second sensor gain, then the adjusted second sensor gain is determined to be the product value. If the product of the second sensor gain and the first dynamic range adjustment amount is greater than the upper limit threshold of the second sensor gain, then the adjusted second sensor gain is determined to be the upper limit threshold of the second sensor gain.

16. The method according to claim 10, characterized in that, The step of adjusting the candidate exposure control parameters and candidate processing control parameters based on the dynamic range extension value to obtain the target exposure control parameters and target processing control parameters includes: If the dynamic range extension value is greater than 1, and the candidate processing control parameters include compensation gain and digital gain, then the initial value of the dynamic range is determined based on the product of the compensation gain and the digital gain. If the dynamic range extension value is greater than the dynamic range initial value, then a second dynamic range adjustment amount is determined based on the quotient of the dynamic range initial value and the dynamic range extension value; Obtain a fourth set of parameters to be adjusted, the fourth set of parameters to be adjusted including at least one parameter from the candidate exposure control parameters and the candidate processing control parameters, and adjust the parameters in the fourth set of parameters to be adjusted based on the second dynamic range adjustment amount to obtain the adjusted parameters; Based on the adjusted parameters, the target exposure control parameters and target processing control parameters are obtained.

17. The method according to claim 16, characterized in that, The candidate exposure control parameters include at least aperture, exposure time, first conversion gain, first sensor gain, second conversion gain and second sensor gain, and the candidate processing control parameters include at least compensation gain and digital gain. The parameters in the fourth set of parameters to be adjusted include, in order, the second sensor gain, the aperture, and the exposure time; The step of adjusting the parameters in the fourth set of parameters to be adjusted based on the second dynamic range adjustment amount to obtain the adjusted parameters includes: If the first product of the second sensor gain and the second dynamic range adjustment amount is greater than or equal to the lower limit threshold of the second sensor gain, then the adjusted second sensor gain is determined to be the first product value. If the first product value is less than the second sensor gain lower limit threshold, then the first remaining adjustment amount is determined based on the second sensor gain lower limit threshold, the second sensor gain, and the second dynamic range adjustment amount; If the second product of the aperture and the first remaining adjustment amount is greater than or equal to the lower limit threshold of the aperture, then the adjusted second sensor gain is determined to be the lower limit threshold of the second sensor gain, and the adjusted aperture is determined to be the second product value. If the second product value is less than the lower aperture threshold, then the second remaining adjustment amount is determined based on the lower aperture threshold, the aperture, and the first remaining adjustment amount; If the third product of the exposure time and the second remaining adjustment amount is greater than or equal to the lower limit threshold of the exposure time, then the adjusted second sensor gain is determined to be the lower limit threshold of the second sensor gain, the adjusted aperture is determined to be the lower limit threshold of the aperture, and the adjusted exposure time is determined to be the third product value; if the third product value is less than the lower limit threshold of the exposure time, then the adjusted second sensor gain is determined to be the lower limit threshold of the second sensor gain, the adjusted aperture is determined to be the lower limit threshold of the aperture, and the adjusted exposure time is determined to be the lower limit threshold of the exposure time.

18. An imaging device, characterized in that, The imaging device includes: An image acquisition unit is used to acquire a first image signal and a second image signal based on exposure control parameters; wherein, the exposure control parameters are determined based on target statistical information corresponding to a preset number of frames of images acquired before the current frame; the image acquisition unit supports high conversion gain and low conversion gain, outputting the first image signal when using high conversion gain and outputting the second image signal when using low conversion gain; The first image signal and the second image signal are output to the image processing unit; An image processing unit is configured to synthesize a composite image from a first image signal and a second image signal based on processing control parameters, perform gain processing and data compression processing on the composite image to obtain an output image, and output the output image. Specifically, during gain processing, data truncation is not performed, or the data is truncated to a value range not less than the input image's value range. The compression parameters used during data compression processing are related to the gain used during gain processing. Under different gains, the brightness value range of the image obtained through gain processing is different; the compression parameters of the data compression processing are adjusted according to different brightness value ranges so that the compressed image has the same brightness value range under different gains. Statistical unit, used to obtain target statistical information; An exposure control unit is used to acquire target exposure control parameters and target processing control parameters based on the target statistical information; wherein, the target exposure control parameters are input to the image acquisition unit to control the acquisition of a preset number of frames after the current frame; and the target processing control parameters are input to the image processing unit to control the processing of a preset number of frames after the current frame.

19. The apparatus according to claim 18, characterized in that, The image acquisition unit is configured to output the first image signal and the second image signal to the image processing unit if the scene features of the imaging device meet the constraint conditions. The image acquisition unit is further configured to, if the scene features do not meet the constraints, synthesize the first image signal and the second image signal into a third image signal based on the processing control parameters, perform nonlinear compression on the third image signal, and output the compressed image signal to the image processing unit, wherein the image processing unit performs gain processing and data compression processing based on the compressed image signal; The scene features include the available bandwidth between the image acquisition unit and the image processing unit. If the available bandwidth is greater than a preset bandwidth threshold, the scene features satisfy the constraint conditions. If the available bandwidth is less than or equal to the preset bandwidth threshold, the scene features do not satisfy the constraint conditions.

20. An electronic device, characterized in that, The electronic device includes: a memory, a processor, and an image processing program stored in the memory and executable on the processor, wherein the image processing program, when executed by the processor, implements the image processing method as described in any one of claims 1 to 17.

21. A storage medium, characterized in that, The storage medium stores an image processing program, which, when executed by a processor, implements the image processing method as described in any one of claims 1 to 17.

Images