Image processing method, imaging device and equipment

Abstract

The invention provides an image processing method, an imaging device and equipment. The method comprises the following steps: acquiring a first image signal and a second image signal based on exposure control parameters; the first image signal is output when the high conversion gain is adopted, and the second image signal is output when the low conversion gain is adopted; synthesizing the first image signal and the second image signal based on the processing control parameter to obtain a synthesized image, performing gain processing and data compression processing on the synthesized image to obtain a to-be-output image, and outputting the to-be-output image; obtaining a target exposure control parameter and a target processing control parameter based on the target statistical information; wherein the target exposure control parameter is used for controlling acquisition of the preset number of frame images after the current frame, and the target processing control parameter is used for controlling processing of the preset number of frame images after the current frame. Through the technical scheme of the invention, the dynamic range of the image is improved, the image with a higher signal-to-noise ratio is obtained, a better noise reduction effect is realized, and the information amount of the image is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to an image processing method, imaging device and equipment. Background Art

[0002] After an image acquisition device (such as a camera, etc.) acquires an image, the image can usually be processed to improve the image quality. For example, the image can be subjected to gain processing, noise reduction processing, black level processing, white balance processing, deblurring processing, sharpening processing, etc. Among them, gain processing is used to increase the brightness of the image. Noise reduction processing is used to reduce the noise of the image. Black level processing is used to ensure the dark details of the image. White balance processing is used to control the hue of the image. Deblurring processing is used to repair the blurred area of ​​the image. Sharpening processing is used to compensate for the contour of the image, enhance the edge and grayscale jump part of the image, and improve the clarity of the image.

[0003] Although the image can be subjected to operations such as gain processing, noise reduction processing, black level processing, white balance processing, deblurring processing, and sharpening processing, there are still problems such as poor image effects. Summary of the invention

[0004] The present application provides an image processing method, the method comprising:

[0005] The first image signal and the second image signal are collected based on an exposure control parameter; wherein the exposure control parameter is determined based on target statistical information corresponding to a preset number of frames of images collected before the current frame; wherein the image sensor supports a high conversion gain and a low conversion gain, outputs the first image signal when the high conversion gain is adopted, and outputs the second image signal when the low conversion gain is adopted;

[0006] The first image signal and the second image signal are synthesized based on the processing control parameter to obtain a synthesized image, the synthesized image is subjected to gain processing and data compression processing to obtain an image to be output, and the image to be output is output; wherein, when performing the gain processing, no data truncation is performed, or the data is truncation to a certain value range that is not less than the value range of the input image; wherein, the compression parameter used in the data compression processing is related to the gain used in the gain processing;

[0007] Target exposure control parameters and target processing control parameters are obtained based on target statistical information; wherein the target exposure control parameters are used to control the acquisition of a preset number of image frames after the current frame, and the target processing control parameters are used to control the processing of a preset number of image frames after the current frame.

[0008] The present application provides an imaging device, the imaging device comprising:

[0009] An image acquisition unit, configured to acquire a first image signal and a second image signal based on an exposure control parameter; wherein the exposure control parameter is determined based on target statistical information corresponding to a preset number of frames of images acquired before a current frame; the image acquisition unit supports a high conversion gain and a low conversion gain, outputs a first image signal when a high conversion gain is adopted, and outputs a second image signal when a low conversion gain is adopted;

[0010] outputting the first image signal and the second image signal to an image processing unit;

[0011] an image processing unit, configured to synthesize the first image signal and the second image signal based on a processing control parameter to obtain a synthesized image, perform gain processing and data compression processing on the synthesized image to obtain an image to be output, and output the image to be output; wherein, when performing the gain processing, no data truncation is performed, or the data is truncation to a certain value range that is not less than the value range of the input image; wherein, the compression parameter used in the data compression processing is related to the gain used in the gain processing;

[0012] A statistical unit, used to obtain target statistical information;

[0013] An exposure control unit, used for acquiring a target exposure control parameter and a target processing control parameter based on the target statistical information; wherein the target exposure control parameter is input to the image acquisition unit to control the acquisition of a preset number of image frames after the current frame; and the target processing control parameter is input to the image processing unit to control the processing of a preset number of image frames after the current frame.

[0014] It can be seen from the above technical scheme that in the embodiment of the present application, the first image signal and the second image signal are collected, the first image signal is an image signal with high conversion gain, and the second image signal is an image signal with low conversion gain, so that the image to be output is generated based on the first image signal and the second image signal, and the image effect of the image to be output is better. By collecting two frames of image signals with different brightness, the expandable dynamic range multiple is greater than that of a single frame, and motion synthesis defects can be eliminated, with a higher signal-to-noise ratio, and a higher dynamic range can be achieved. By collecting two frames of image signals with different brightness, the dynamic range of the image can be significantly improved, an image with a higher signal-to-noise ratio can be obtained, a better noise reduction effect can be achieved, the highlight area information can be retained, and the image information amount can be increased. Since the present invention does not perform bit width truncation in the gain processing of the image signal or truncates to a brightness value range greater than the image signal in the gain processing, and then normalizes the images with different value ranges through data compression, it can avoid or reduce the loss of brightness information when the image is compensated for brightness, retain more brightness information, improve the image processing effect, and improve the visual effect of the processed image. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments of the present application or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings of the embodiments of the present application.

[0016] Figure 1 is a schematic structural diagram of an imaging device in one embodiment of the present application;

[0017] Figure 2A and Figure 2B is a schematic structural diagram of an imaging device in one embodiment of the present application;

[0018] Figure 3A is a structural schematic diagram of an image acquisition unit in one embodiment of the present application;

[0019] Figure 3B is a schematic diagram of a filter pass rate curve in one embodiment of the present application;

[0020] Figure 3C is a schematic diagram of an arrangement of image sensors in one embodiment of the present application;

[0021] Figure 3D is a schematic diagram of an arrangement of RGBW sensors in one embodiment of the present application;

[0022] Figure 3E is a schematic diagram of a spectral response curve in one embodiment of the present application;

[0023] Figure 3F is a schematic diagram of an image sensor with a DCG function in one embodiment of the present application;

[0024] Figure 4A-4G is a schematic diagram of the structure of an image processing unit in one embodiment of the present application;

[0025] Figure 5 is a schematic structural diagram of an exposure control unit in one embodiment of the present application;

[0026] Figure 6 is a flowchart of an image processing method in one embodiment of the present application;

[0027] Figure 7 It is a hardware structure diagram of an electronic device in one embodiment of the present application. DETAILED DESCRIPTION

[0028] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, rather than limiting the present application. The singular forms of "a", "said" and "the" used in the present application and claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to any or all possible combinations of one or more associated listed items.

[0029] It should be understood that, although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, in addition, the word "if" used may be interpreted as "at..." or "when..." or "in response to determination".

[0030] In the embodiment of the present application, an imaging device is proposed, see Figure 1 FIG. 1 is a schematic diagram of the structure of the imaging device. The imaging device may include, but is not limited to: an image acquisition unit (the image acquisition unit may be an image sensor), an image processing unit, a statistical unit and an exposure control unit.

[0031] An image acquisition unit is used to acquire a first image signal and a second image signal based on an exposure control parameter; wherein the exposure control parameter is determined based on target statistical information corresponding to a preset number of frames of images acquired before the current frame. The image acquisition unit (i.e., the image sensor) can support high conversion gain and low conversion gain, and output the first image signal when the image sensor adopts a high conversion gain, and output the second image signal when the image sensor adopts a low conversion gain. Since the first image signal is an image signal under a high conversion gain and the second image signal is an image signal under a low conversion gain, the brightness of the first image signal is higher than the brightness of the second image signal. Exemplarily, this embodiment involves only one image sensor, and the first image signal and the second image signal are image signals acquired by the same image sensor.

[0032] The image acquisition unit is further used to output the first image signal and the second image signal to the image processing unit. For example, the image acquisition unit is used to output the first image signal and the second image signal to the image processing unit if the scene characteristics of the imaging device meet the constraint conditions. Alternatively, the image acquisition unit is also used to synthesize the first image signal and the second image signal based on the processing control parameters to obtain a third image signal, perform nonlinear compression on the third image signal, and output the compressed image signal to the image processing unit if the scene characteristics of the imaging device do not meet the constraint conditions. The processing control parameters are determined based on target statistical information corresponding to a preset number of frames of images acquired before the current frame.

[0033] The scene feature may include but is not limited to the available bandwidth between the image acquisition unit and the image processing unit. If the available bandwidth is greater than a preset bandwidth threshold, it is determined that the scene feature meets the constraint condition; if the available bandwidth is less than or equal to the preset bandwidth threshold, it is determined that the scene feature does not meet the constraint condition.

[0034] The image processing unit is used to synthesize the first image signal and the second image signal based on the processing control parameter (the processing control parameter is determined based on the target statistical information corresponding to the preset number of frames of images collected before the current frame) to obtain a synthesized image, perform gain processing and data compression processing on the synthesized image to obtain an image to be output, and output the image to be output, that is, the image processing unit can directly output the image to be output. Wherein, when performing the gain processing, no data truncation is performed, or the data is truncation to a certain value range that is not less than the value range of the input image; when performing the data compression processing, the compression parameter used is related to the gain used when performing the gain processing.

[0035] The image processing unit is also used to generate an image to be output based on the compressed image signal when receiving the compressed image signal (i.e., the third image signal), without performing a synthesis operation. For example, the compressed image signal is subjected to gain processing and data compression processing to obtain the image to be output, and the processing process is similar to the processing process of the synthesized image. Among them, when performing the gain processing, no data truncation is performed, or the data is truncated to a certain value range that is not less than the value range of the input image; the compression parameter used in the data compression processing is related to the gain used in the gain processing. Exemplarily, before the gain processing and data compression processing, a decompression processing can be added to restore the compressed (non-linear) third image signal to a linear signal, and then perform the gain processing and data compression processing.

[0036] A statistical unit is used to obtain target statistical information. For example, the target statistical information includes a dynamic range extension value obtained based on a first statistical image, and the first statistical image is an image before synthesis processing, an image after synthesis processing, an image before gain processing, an image after gain processing, an image before data compression processing, or an image after data compression processing. In addition, the target statistical information includes a statistical brightness value obtained based on a second statistical image, and the second statistical image is an image before synthesis processing, an image after synthesis processing, an image before gain processing, an image after gain processing, an image before data compression processing, or an image after data compression processing. In summary, it can be seen that the statistical unit can obtain a dynamic range extension value based on the first statistical image, for example, the dynamic range extension value can be obtained by performing dynamic range statistics on the first statistical image, or the dynamic range extension value can be obtained by performing pre-processing on the first statistical image, as long as the dynamic range extension value is obtained based on the first statistical image. The statistical unit can obtain a statistical brightness value based on the second statistical image, for example, the statistical brightness value can be obtained by performing brightness statistics on the second statistical image, or the statistical brightness value can be obtained by performing pre-processing on the second statistical image, as long as the statistical brightness value is obtained based on the second statistical image.

[0037] An exposure control unit is used to obtain target exposure control parameters and target processing control parameters based on target statistical information; wherein the target exposure control parameters are input to the image acquisition unit to control the acquisition of a preset number of image frames after the current frame; and the target processing control parameters are input to the image processing unit to control the processing of a preset number of image frames after the current frame.

[0038] It can be seen from the above technical solution that in the embodiment of the present application, the first image signal and the second image signal are collected, the first image signal is an image signal with a high conversion gain, and the second image signal is an image signal with a low conversion gain, so that the image to be output is generated based on the first image signal and the second image signal, and the image effect of the image to be output is better. By collecting two frames of image signals with different brightness, the expandable dynamic range multiple is greater than that of a single frame, motion synthesis defects can be eliminated, a higher signal-to-noise ratio is provided, and a higher dynamic range is achieved. By collecting two frames of image signals with different brightness, the dynamic range of the image can be significantly improved, an image with a higher signal-to-noise ratio can be obtained, a better noise reduction effect can be achieved, the highlight area information can be retained, and the amount of image information can be increased.

[0039] The imaging device according to the embodiment of the present application is described below in conjunction with specific application scenarios.

[0040] See also Figure 2A and Figure 2B , which is a schematic diagram of the structure of an imaging device, the imaging device may include an image acquisition unit, an image processing unit, a statistical unit, an exposure control unit and a cache unit.

[0041] See also Figure 2A As shown, the input of the image acquisition unit is the light signal and the exposure control parameter, and the output of the image acquisition unit is the first image signal and the second image signal. The input of the image processing unit is the first image signal, the second image signal and the processing control parameter, and the output of the image processing unit is the image to be output, the first statistical image and the second statistical image. The input of the statistical unit is the first statistical image and the second statistical image, and the output of the statistical unit is statistical information. The input of the exposure control unit is statistical information, and the output of the exposure control unit is exposure control parameters and processing control parameters. The input of the cache unit is exposure control parameters and processing control parameters, and the output of the cache unit is exposure control parameters and processing control parameters.

[0042] Figure 2B In the embodiment, the input data of the image acquisition unit is the light signal and the exposure control parameter, and the output data is the third image signal. The input data of the image processing unit is the third image signal and the processing control parameter, and the output data is the image to be output, the first statistical image, and the second statistical image. The input data of the statistical unit is the first statistical image and the second statistical image, and the output data is statistical information. The input data of the exposure control unit is statistical information, and the output data is the exposure control parameter and the processing control parameter. The input data of the cache unit is the exposure control parameter and the processing control parameter, and the output data is the exposure control parameter and the processing control parameter.

[0043] The detailed functions of each functional unit of the imaging device are described below.

[0044] First, the image acquisition unit.

[0045] See also Figure 3A FIG. 1 is a schematic diagram of the structure of an image acquisition unit, which may include a lens, a filter assembly, and an image sensor. The ambient light becomes incident light after passing through the lens, and the incident light becomes filtered light after passing through the filter assembly. The image sensor may acquire the filtered light (i.e., the optical signal), and based on the optical signal, the image sensor may acquire a first image signal and a second image signal.

[0046] Exemplarily, the image acquisition unit may include a lens, which is a lens module composed of multiple lenses, which can receive light and focus on an image, and the amount of light entering can be controlled by adjusting the aperture.

[0047] Exemplarily, the image acquisition unit may include a filter component (i.e., a filter module), which can control the spectral range received by the image sensor. For example, the filter component can be switched to allow visible light and infrared light to pass through, so that the image sensor receives visible light and infrared light. For another example, the filter component can be switched to allow only visible light to pass through, so that the image sensor receives visible light.

[0048] For example, a filter assembly is a filter module that allows only light of a specific wavelength to pass through while filtering out light of other wavelengths. The filter assembly can control the spectral range that can enter the image sensor through coating means. Figure 3B As shown, the filter used in the filter assembly can pass the visible light band of 340nm to 650nm and the near infrared light band of 930nm to 970nm, and filter out light in other bands. Of course, the filter assembly can also be switched to allow only visible light to pass, blocking infrared light greater than 650nm from passing, and allowing visible light less than or equal to 650nm to pass, and there is no limit to this filter range.

[0049] Exemplarily, the image acquisition unit may include an image sensor, which can acquire a light signal and output a first image signal and a second image signal within the same exposure time through a DCG (Dual Coversion Gain) technology, thereby converting the light signal into an image signal.

[0050] See also Figure 3C As shown in FIG. 1 , it is a schematic diagram of the arrangement of image sensors. The image sensor can be an RGBW sensor, a Bayer sensor, or an RGB-IR sensor. The pixel arrangement of these three sensors is shown in FIG. Figure 3C As shown, of course, other arrangements of sensors may also be used. In summary, each pixel in the image sensor can sense at least one of red, green and blue visible light.

[0051] Taking the image sensor using RGBW sensor as an example, the arrangement of RGBW sensor can be seen in Figure 3D As shown, the spectral response curve of the RGBW sensor can be found in Figure 3E shown.

[0052] For example, see Figure 3F , which is a schematic diagram of an image sensor with a DCG function. The DCG function is achieved by adding a DCG switch in the pixel circuit of the image sensor.

[0053] In order to realize the DCG function, the exposure control parameters of the image acquisition unit may include aperture, exposure time, a first conversion gain corresponding to the first image signal, a first sensor gain corresponding to the first image signal, a second conversion gain corresponding to the second image signal, and a second sensor gain corresponding to the second image signal.

[0054] First, the image sensor collects light signals corresponding to the aperture (i.e., the current aperture, which will be updated in the subsequent process, and the latest aperture can be used each time) and the exposure time (i.e., the current exposure time, which will be updated in the subsequent process, and the latest exposure time can be used each time).

[0055] Then, after obtaining the optical signal (such as optical signal A), the dual conversion gain switch (i.e., DCG switch) can be controlled to be turned off. When the DCG switch is turned off, the first image signal corresponding to the optical signal A is determined based on the first conversion gain and the first sensor gain. For example, the image sensor can convert the optical signal A into an electrical signal, and convert the electrical signal into an image signal. In this way, the brightness of the collected image can be controlled by adjusting the first conversion gain and the first sensor gain, and the first image signal can be output.

[0056] Then, after the first image signal is acquired, the dual conversion gain switch (i.e., DCG switch) can be controlled to be closed. When the DCG switch is closed, the second image signal corresponding to the light signal A is determined based on the second conversion gain and the second sensor gain. For example, the image sensor can convert the light signal A into an electrical signal, and convert the electrical signal into an image signal. In this way, the brightness of the acquired image can be controlled by adjusting the second conversion gain and the second sensor gain, and the second image signal can be output.

[0057] For another example, after obtaining the light signal A, the DCG switch can be controlled to be closed (i.e., closed first and then opened), and when the DCG switch is closed, the second image signal corresponding to the light signal A is determined based on the second conversion gain and the second sensor gain. After the second image signal is acquired, the DCG switch is controlled to be opened, and the first image signal corresponding to the light signal A is determined based on the first conversion gain and the first sensor gain.

[0058] In summary, it can be seen that the effects of the first image signal and the second image signal are regulated by the exposure control parameters, and the exposure control parameters may include but are not limited to: aperture, exposure time, first conversion gain, first sensor gain, second conversion gain, and second sensor gain. Obviously, the first image signal and the second image signal are image signals output at the same exposure time and the same aperture. The first image signal and the second image signal are image signals output at different conversion gains (the first image signal corresponds to the first conversion gain, and the second image signal corresponds to the second conversion gain) and different sensor gains (the first image signal corresponds to the first sensor gain, and the second image signal corresponds to the second sensor gain).

[0059] For the imaging process, the image sensor first collects the light signal, then converts the light signal into an electrical signal, then converts the electrical signal into a voltage signal, and then converts the voltage signal into a digital signal (i.e., an image signal). Based on this, the gain in the process of converting the electrical signal to the voltage signal can be called the conversion gain, the gain when the DCG switch is disconnected is the first conversion gain corresponding to the first image signal, and the gain when the DCG switch is closed is the second conversion gain corresponding to the second image signal. The gain in the process of amplifying the voltage signal can be called the analog gain, the gain when the DCG switch is disconnected is the first analog gain corresponding to the first image signal, and the gain when the DCG switch is closed is the second analog gain corresponding to the second image signal. The gain in the process of amplifying the digital signal can be called the digital gain, the gain when the DCG switch is disconnected is the first digital gain corresponding to the first image signal, and the gain when the DCG switch is closed is the second digital gain corresponding to the second image signal. The first sensor gain can be determined based on the first analog gain and the first digital gain, and the first sensor gain can be the first analog gain, or the first sensor gain can be the first digital gain, or the first sensor gain can be the product value of the first analog gain and the first digital gain. The second sensor gain may be determined based on the second analog gain and the second digital gain. The second sensor gain may be the second analog gain, or the second sensor gain may be the second digital gain, or the second sensor gain may be the product value of the second analog gain and the second digital gain.

[0060] The second conversion gain may be the same as or different from the first conversion gain. For example, the second conversion gain corresponding to the second image signal is different from the first conversion gain corresponding to the first image signal. Conversion gain is a technical indicator of the DCG function and has a higher signal-to-noise ratio than sensor gain. The first conversion gain is greater than the second conversion gain, that is, the first conversion gain is a high conversion gain and the second conversion gain is a low conversion gain.

[0061] The second sensor gain may be the same as or different from the first sensor gain. For example, the second sensor gain corresponding to the second image signal is different from the first sensor gain corresponding to the first image signal.

[0062] See also Figure 3FAs shown, when the DCG switch is disconnected, the total capacitance in the pixel circuit of the image sensor becomes smaller, and the same number of electrons can be converted into a larger voltage, which is reflected in the image signal as a brighter image, and at the same time has a higher signal-to-noise ratio. In this way, the first conversion gain and the first sensor gain can be adjusted to control the acquisition of a brighter first image signal. Conversely, when the DCG switch is closed, the total capacitance in the pixel circuit of the image sensor becomes larger, and the same number of electrons can be converted into a smaller voltage, which is reflected in the image signal as a darker image, and at the same time has a lower signal-to-noise ratio. In this way, the second conversion gain and the second sensor gain can be adjusted to control the acquisition of a darker second image signal.

[0063] In summary, by turning the DCG switch on and off, two image signals with different brightness can be obtained, and the brightness of the first image signal is higher than that of the second image signal. When the DCG switch is disconnected, the image acquisition unit will obtain a higher conversion gain, that is, the image acquisition unit adopts a high conversion gain, and the image signal collected when the high conversion gain is adopted can be the first image signal. When the DCG switch is closed, the image acquisition unit will obtain a lower conversion gain, that is, the image acquisition unit adopts a low conversion gain, and the image signal collected when the low conversion gain is adopted can be the second image signal.

[0064] Exemplarily, after obtaining the first image signal and the second image signal, the image acquisition unit may output the first image signal and the second image signal to the image processing unit. Or,

[0065] The image acquisition unit may synthesize the first image signal and the second image signal to obtain a third image signal, and output the third image signal to the image processing unit. Alternatively, the image acquisition unit may synthesize the first image signal and the second image signal to obtain a third image signal, perform nonlinear compression on the third image signal, and output the compressed image signal to the image processing unit.

[0066] For example, if the scene characteristics of the imaging device meet the constraint conditions, the image acquisition unit can output the first image signal and the second image signal to the image processing unit. Or,

[0067] If the scene characteristics of the imaging device do not meet the constraint conditions, the image acquisition unit can synthesize the first image signal and the second image signal based on the processing control parameters to obtain a third image signal, perform nonlinear compression on the third image signal, and output the compressed image signal to the image processing unit.

[0068] The scene feature may include but is not limited to the available bandwidth between the image acquisition unit and the image processing unit. Of course, the available bandwidth is only an example of a scene feature, and there is no restriction on this scene feature, and there is no restriction on the constraints under different scene features, and it can be configured based on experience.

[0069] Taking the available bandwidth as an example, if the available bandwidth is greater than the preset bandwidth threshold, it can be determined that the scene feature meets the constraint condition, and the first image signal and the second image signal are directly output to the image processing unit. If the available bandwidth is less than or equal to the preset bandwidth threshold, it can be determined that the scene feature does not meet the constraint condition, and the compressed image signal needs to be output to the image processing unit.

[0070] The implementation method of synthesizing the first image signal and the second image signal by the image acquisition unit can refer to the synthesis method of the subsequent image processing unit, which will not be repeated here. When the image acquisition unit performs nonlinear compression on the third image signal, the value range of the image signal can be reduced, that is, an image signal with a lower value range is obtained through nonlinear compression, and accordingly, the value range is restored in the image processing unit.

[0071] Exemplarily, when the image sensor turns on the build-in mode, the third image signal can be a build-in high dynamic range image signal. The "build-in mode" refers to a wide dynamic data acquisition technology in which the image sensor obtains a high value range image by acquiring multiple frames of images and then synthesizing the images, and finally compresses the high value range image into a low value range image through nonlinear mapping data before outputting it. For example, when the image sensor turns on the build-in mode, it means that the scene characteristics of the imaging device do not meet the constraint conditions.

[0072] Second, the image processing unit.

[0073] The image processing unit is a logic platform that includes a signal processing algorithm or program. The logic platform can be a computer based on X86 or ARM architecture, or an FPGA (Field-Programmable Gate Array) logic circuit. There is no restriction on the implementation of this image processing unit.

[0074] When the image processing unit receives the first image signal and the second image signal, it can synthesize the first image signal and the second image signal based on the processing control parameter (the processing control parameter is determined by the target statistical information corresponding to the preset number of frames of images collected before the current frame) to obtain a synthesized image, and generate an image to be output based on the synthesized image, for example, performing gain processing and data compression processing on the synthesized image to obtain the image to be output. Alternatively, when the image processing unit receives a compressed image signal (i.e., a third image signal), it can generate an image to be output based on the third image signal, for example, performing gain processing and data compression processing on the third image signal to obtain the image to be output. For example, if the third image signal is a nonlinearly compressed image, a decompression process can be added to restore the value range of the third image signal. In the subsequent process, the third image signal is taken as an example that has been nonlinearly compressed.

[0075] For example, see Figure 4A FIG. 1 is a schematic diagram of the structure of an image processing unit, and the image processing unit may include a synthesis processing module, a gain processing module, and a data compression module. The synthesis processing module synthesizes the first image signal and the second image signal to obtain a synthesized image. The gain processing module performs gain processing on the synthesized image to obtain a gain-processed image. The data compression module performs data compression processing on the gain-processed image to obtain an image to be output.

[0076] During the processing of the image processing unit, the value range IB1 of the first image signal and the value range IB2 of the second image signal may be the same, the value range FB1 of the synthesized image is not less than the value range IB1 of the first image signal, and the value range FB1 of the synthesized image is not less than the value range IB2 of the second image signal. The value range FB2 of the image after gain processing is not less than the value range IB1 of the first image signal, and the value range FB2 of the image after gain processing is not less than the value range IB2 of the second image signal. The value range OB1 of the image to be output is not greater than the value range FB2 of the image after gain processing.

[0077] Exemplarily, the value range may be a value range of grayscale values. For example, the value range may be expressed in the form of [T1, T2], where T1 represents the minimum value of the grayscale value and T2 represents the maximum value of the grayscale value.

[0078] For example, see Figure 4BAs shown, it is a schematic diagram of the structure of the image processing unit, and the image processing unit may include a gain processing module and a data compression module. The synthesis operation of the first image signal and the second image signal has been completed in the image acquisition unit, and the image processing unit obtains a third image signal (i.e., a synthesized image). After obtaining the synthesized image, the gain processing module may perform gain processing on the synthesized image to obtain a gain-processed image. The data compression module may perform data compression processing on the gain-processed image to obtain an image to be output (i.e., the compressed image is used as the image to be output).

[0079] For example, see Figure 4C As shown, it is a schematic diagram of the structure of the image processing unit, and the image processing unit may include a pre-processing module, a gain processing module and a data compression module. After obtaining the first image signal and the second image signal, the pre-processing module pre-processes the first image signal and the second image signal respectively to obtain the pre-processed first image signal and the pre-processed second image signal. The pre-processing may include but is not limited to at least one of the following: gain processing, black level processing, noise reduction processing, white balance processing, and deblurring processing. Then, the pre-processing module may synthesize the pre-processed first image signal and the pre-processed second image signal to obtain a synthesized image (that is, the function of the synthesis processing module may be integrated in the pre-processing module).

[0080] After obtaining the composite image, the gain processing module performs gain processing on the composite image to obtain a gain-processed image. The data compression module performs data compression processing on the gain-processed image to obtain an image to be output.

[0081] For example, see Figure 4D As shown, it is a schematic diagram of the structure of the image processing unit, and the image processing unit may include a pre-processing module, a gain processing module and a data compression module. The synthesis operation of the first image signal and the second image signal has been completed in the image acquisition unit, and the image processing unit obtains the third image signal. After obtaining the third image signal, the pre-processing module may perform pre-processing on the third image signal to obtain a pre-processed image signal (i.e., a synthesized image). The gain processing module may perform gain processing on the synthesized image to obtain a gain-processed image. The data compression module may perform data compression processing on the gain-processed image to obtain an image to be output (i.e., the compressed image is used as the image to be output).

[0082] Among them, the pre-processing may include but is not limited to at least one of the following: gain processing, black level processing, noise reduction processing, white balance processing, and deblurring processing. In addition, when the image sensor turns on the built-in wide dynamic mode, the third image signal is a built-in high dynamic range image signal. Based on this, the pre-processing may also include decompression processing, and the decompression processing may be achieved through curve mapping. For example, the input nonlinear image can be mapped to a linear image with a higher value range through a preset brightness mapping curve, and the preset brightness mapping curve corresponds to the built-in data compression curve in the image sensor.

[0083] For example, see Figure 4E As shown, it is a schematic diagram of the structure of the image processing unit, and the image processing unit may include a synthesis processing module, a gain processing module, a data compression module and a post-processing module. The synthesis processing module synthesizes the first image signal and the second image signal to obtain a synthesized image. The gain processing module performs gain processing on the synthesized image to obtain a gain-processed image. The data compression module performs data compression processing on the gain-processed image to obtain a compressed image. The post-processing module performs post-processing on the compressed image to obtain an image to be output, wherein the post-processing may include but is not limited to at least one of the following: noise reduction processing, enhancement processing, defogging processing, deblurring processing, and sharpening processing.

[0084] For example, see Figure 4F As shown, it is a schematic diagram of the structure of the image processing unit, and the image processing unit may include a pre-processing module, a gain processing module, a data compression module and a post-processing module. The pre-processing module performs pre-processing on the first image signal and the second image signal respectively to obtain the pre-processed first image signal and the pre-processed second image signal, and synthesizes the pre-processed first image signal and the pre-processed second image signal to obtain a synthesized image. The gain processing module performs gain processing on the synthesized image to obtain a gain-processed image. The data compression module performs data compression processing on the gain-processed image to obtain a compressed image. The post-processing module performs post-processing on the compressed image to obtain an image to be output.

[0085] For example, the image processing unit may further include an intermediate processing module. The intermediate processing performed by the intermediate processing module may include but is not limited to at least one of the following: noise reduction processing, white balance processing, and deblurring processing. In order to avoid the increase in resources of the intermediate processing module due to the excessive digital gain of the gain processing module, the digital gain can be split into two parts, that is, two gain processing modules are deployed. In order to avoid the increase in resources of the intermediate processing module due to the excessive value range of the data compression module, the data compression can be split into two parts, that is, two data compression modules are deployed. In summary, see Figure 4GAs shown, it is a structural schematic diagram of an image processing unit. The image processing unit may include a synthesis processing module, a gain processing module A, a data compression module A, an intermediate processing module, a gain processing module B and a data compression module B.

[0086] The synthesis processing module synthesizes the first image signal and the second image signal to obtain a synthesized image. The gain processing module A performs gain processing on the synthesized image to obtain an intermediate image 1. The data compression module A performs data compression processing on the intermediate image 1 to obtain an intermediate image 2. The intermediate processing module performs intermediate processing on the intermediate image 2 to obtain an intermediate image 3. The gain processing module B performs gain processing on the intermediate image 3 to obtain a gain-processed image. The data compression module B performs data compression processing on the gain-processed image to obtain a compressed image, and then the compressed image can be used as the image to be output.

[0087] During the processing of the image processing unit, the value range IB1 of the first image signal and the value range IB2 of the second image signal may be the same. The value range FB1 of the composite image is not less than the value range IB1 of the first image signal, and the value range FB1 is not less than the value range IB2 of the second image signal. The value range FB2 of the intermediate image 1 is not less than the value range IB1 of the first image signal, and the value range FB2 is not less than the value range IB2 of the second image signal. The value range FB3 of the intermediate image 2 is not greater than the value range FB2 of the intermediate image 1. The value range FB4 of the intermediate image 3 is equal to the value range FB3 of the intermediate image 2. The value range FB5 of the image after gain processing is not less than the value range FB4 of the intermediate image 3. The value range OB1 of the image to be output is not greater than the value range FB5 of the image after gain processing.

[0088] For example, Figure 4A-4G These are just a few examples of image processing units. There is no limitation on the structure of the image processing units, which can be any combination of the above modules. For example, the image processing unit includes a synthesis processing module, a gain processing module, and a data compression module. The pre-processing module, the intermediate processing module, and the post-processing module are optional, and the pre-processing module, the intermediate processing module, and the post-processing module can be combined in any way.

[0089] The functions of the synthesis processing module, the gain processing module and the data compression module are described below.

[0090] Synthesis processing module: The synthesis processing module can synthesize the first image signal and the second image signal based on the processing control parameters to obtain a synthesized image. The first image signal and the second image signal can be two pairs of image signals output by the image sensor, which are image signals generated under the same aperture and the same exposure time, but with different conversion gains, and the sensor gains can be the same or different. The brightness of the first image signal can be greater than the brightness of the second image signal. The value range of the synthesized image is not less than the value range of the first image signal, nor is it less than the value range of the second image signal.

[0091] Exemplarily, the second image signal may be multiplied by the compensation gain to obtain a gain-compensated image; then, the first image signal and the gain-compensated image may be synthesized to obtain a synthesized image.

[0092] For example, the processing control parameter includes a compensation gain, which is a ratio between a first gain value and a second gain value, and the first gain value is a product value of a first conversion gain and a first sensor gain (i.e., a conversion gain and a sensor gain corresponding to a first image signal), and the second gain value is a product value of a second conversion gain and a second sensor gain (i.e., a conversion gain and a sensor gain corresponding to a second image signal), and the first gain value is greater than the second gain value. On this basis, based on the processing control parameter, the synthesis processing module can obtain the compensation gain, and then multiply the second image signal by the compensation gain to obtain a gain-compensated image.

[0093] When the second image signal is multiplied by the compensation gain to obtain the gain compensated image, no data truncation is performed, and therefore, the value range of the gain compensated image is greater than or equal to the value range of the second image signal.

[0094] Exemplarily, after obtaining the gain-compensated image, the first image signal and the gain-compensated image may be synthesized to obtain a synthesized image. When synthesizing the first image signal and the gain-compensated image to obtain the synthesized image, the image area of ​​the first image signal is divided into a first area and a second area, the brightness in the first area is not greater than the synthesized brightness threshold, and the brightness in the second area is greater than the synthesized brightness threshold; the first image signal is used in the first area and the gain-compensated image is used in the second area to obtain the synthesized image. For example, the synthesized brightness threshold fl_thr may be pre-configured, and the first image signal and the gain-compensated image may be synthesized according to the synthesized brightness threshold fl_thr.

[0095] Image synthesis means: judging the brightness of different areas of the first image signal, selecting the first image signal as the image area of ​​the synthesized image for areas with brightness not greater than fl_thr, and selecting the gain-compensated image as the image area of ​​the synthesized image for areas with brightness greater than fl_thr. When performing image synthesis, different weights may also be set according to different brightness of the first image signal or the gain-compensated image, and weighted synthesis may be performed on the first image signal and the gain-compensated image. Of course, the above are only two examples of performing image synthesis on the first image signal and the gain-compensated image, and there is no limitation on this image synthesis method.

[0096] It should be noted that the synthesis processing module can also be designed to access more frames of image signals for synthesis, so as to adapt to other high dynamic data generation methods, such as three-frame or four-frame wide dynamic.

[0097] Gain processing module: The gain processing module can perform gain processing on the composite image according to the processing control parameters to obtain a gain-processed image. For example, the processing control parameters may include a digital gain, and the gain processing is to multiply the composite image by the digital gain. When the composite image is multiplied by the digital gain, the data is not truncate or is truncated to a certain value range that is not less than the value range of the input image (that is, after the composite image is multiplied by the digital gain, the data is not truncate or is truncated to a certain value range that is not less than the value range of the input image), and the value range of the image after gain processing is greater than or equal to the value range of the composite image. Among them, the gain is used to make the image reach the target brightness, and the loss of information in the highlight area of ​​the image is avoided by not truncating. Similarly, the loss of information in the highlight area of ​​the image can also be avoided by truncating to a certain value range that is not less than the value range of the input image.

[0098] Assume that the statistical brightness of the synthesized image is s1, and the statistical brightness s1 is not higher than the preset target brightness t1, the statistical brightness of the image after gain processing is s2, and the statistical brightness s2 is equal to the preset target brightness t1, and the preset target brightness t1 is a brightness value that makes the image visual perception more suitable, which can be configured based on experience. On this basis, the gain value g of the digital gain processing is t1 / s1, that is, the digital gain can be t1 / s1.

[0099] After the synthesized image is multiplied by the digital gain g, no data truncation processing is performed on the data, and the value range of the obtained gain-processed image is not less than the value range of the synthesized image.

[0100] The statistical brightness s1 of the composite image may be calculated by taking the average of the composite image. The statistical brightness s1 of the composite image may also be calculated by setting different weight values ​​for image areas of different brightness, and then performing weighted averaging on the image. The statistical brightness s1 of the composite image may also be calculated by setting different weight values ​​for areas of different spatial positions of the image, and then performing weighted averaging on the image. The statistical brightness s1 of the composite image may also be calculated by presetting a brightness threshold sl_thr1, taking the smaller of the composite image and the preset brightness threshold sl_thr1, and then using the weighted average result obtained by the above method. Of course, the above are just a few examples of calculating the statistical brightness s1, and there is no limitation to this.

[0101] The calculation method of the statistical brightness s2 of the image after gain processing can be the same as the calculation method of the statistical brightness s1 of the synthetic image, or it can be different from the calculation method of the statistical brightness s1, which will not be repeated here. For example, the smaller of the image after gain processing and the preset brightness threshold is first taken, and then weighted average is performed.

[0102] In order to prevent the digital gain g of the gain processing module from being too large, the digital gain g is not greater than the preset gain upper limit threshold G_MAX. Based on this, when the digital gain g is greater than the preset gain upper limit threshold G_MAX, part of the digital gain needs to be allocated to another gain processing module, that is, deploying two gain processing modules, see Figure 4G As shown, the two gain processing modules are gain processing module A and gain processing module B. Based on this, the digital gain G1 of gain processing module A=G_MAX, and the digital gain G2 of gain processing module B=tl / (s1*g).

[0103] For the scenario where the gain processing module A and the gain processing module B are deployed, assuming that the statistical brightness of the composite image is s1, and the statistical brightness of the image after gain processing (i.e., the image processed by the gain processing module B) is s5, then the statistical brightness s5 is equal to the preset target brightness tl, and the preset target brightness tl is a brightness value that makes the image visual perception more suitable, which can be configured based on experience. After the gain processing module B performs gain processing on the image (i.e., performs gain processing based on the digital gain G2), no data truncation processing is performed on the data. The calculation method of the statistical brightness s5 of the image after gain processing can be the same as the calculation method of the statistical brightness s1 of the composite image, or it can be different from the calculation method of the statistical brightness s1, which will not be repeated here.

[0104] In this case, gain processing and data compression processing are performed on the composite image, which may include but is not limited to: if the digital gain is greater than a preset gain threshold, splitting the digital gain into a first digital gain value and a second digital gain value, the first digital gain value is equal to the preset gain threshold, and the product of the first digital gain value and the second digital gain value is equal to the digital gain; gain processing is performed on the composite image according to the first digital gain value to obtain a first intermediate image; data compression processing is performed on the first intermediate image to obtain a second intermediate image; intermediate processing is performed on the second intermediate image to obtain a third intermediate image; gain processing is performed on the third intermediate image according to the second digital gain value to obtain a gain-processed image; data compression processing is performed on the gain-processed image to obtain a compressed image; and an image to be output is determined based on the compressed image.

[0105] Data compression module: The data compression module can perform data compression processing on the gain-processed image to obtain a compressed image (the compressed image is used as the image to be output, or the compressed image is post-processed to obtain the image to be output). When performing data compression processing, a preset nonlinear mapping curve can be used to map the brightness of the gain-processed image. The nonlinear mapping curve is associated with the digital gain g, that is, the compression parameter used in the data compression processing is related to the gain used in the gain processing. When performing data compression processing on the gain-processed image, the compression parameter used in the data compression processing is related to the gain used in the gain processing, including: under different gains, the brightness value range of the image obtained by the gain processing is different, and the compression parameter of the data compression processing is adjusted according to the different brightness value ranges, so that the compressed image has the same brightness value range under different gains. Among them, the value range of the compressed image is not greater than the value range of the gain-processed image. The association of the nonlinear mapping curve with the digital gain g means that the value range of the gain-processed image generated by different digital gains g is different, and the nonlinear mapping curve needs to compress data with different value ranges to the same value range.

[0106] When data compression is performed on the gain-processed image, the gain-processed image is divided into different regions, and a mapping curve corresponding to each region is obtained; for each region of the gain-processed image, mapping processing is performed on the region using the mapping curve corresponding to the region to obtain a compressed image.

[0107] If the maximum value of the synthesized image is il1_max, the maximum value of the image after gain processing is l2_max=il1_max*g, and the maximum value of the image after compression processing is ol1_max, which is a value preset in advance according to the subsequent application situation, the nonlinear mapping curve needs to be set so that the maximum value of the input image is l2_max and the maximum value of the output image is ol1_max. In addition, in order to enhance the naturalness of the visual effect of the processed image, in the nonlinear mapping curve, the part with brightness less than the preset value lin_thr adopts linear mapping, that is, the value after mapping is equal to the value before mapping.

[0108] Data compression processing can also use local mapping processing, that is, different mapping curves are used for mapping processing for different image areas to obtain better visual effects, but the mapping curves of different areas need to satisfy the maximum value of the input image is l2_max, and the maximum value of the output image is ol1_max.

[0109] Data compression processing can also use other mapping methods, such as retinex, histogram, etc., but the mapping processing also needs to satisfy that the maximum value of the input image is l2_max, and the maximum value of the output image is ol1_max.

[0110] When two gain processing modules (gain processing module A and gain processing module B) need to be deployed, two data compression modules (data compression module A and data compression module B) also need to be deployed accordingly. In this case, the maximum value of the first image signal is il1_max, the digital gain of the gain processing module A is g, and the maximum value of the intermediate image 1 is l2_max=il1_max*g. The maximum value of the intermediate image 2 is the preset value l3_max, then the data compression module A needs to satisfy that the maximum value of the input image is l2_max, and the maximum value of the output image is l3_max. The processing method of the data compression module A can be performed in the same manner as described above.

[0111] The maximum value l4_max of the intermediate image 3 is equal to the maximum value l3_max of the intermediate image 2. The gain value of the gain processing module B is g2 (i.e., G2 mentioned above), and the maximum value l5_max of the image after gain processing is l4_max*g2. The maximum value ol1_max of the image to be output is a value preset in advance according to the subsequent application situation, and the data compression module B needs to satisfy the maximum value of the input image is l5_max, and the maximum value of the output image is ol1_max. The processing method of the data compression module B can be performed in the same manner as described above.

[0112] Third, statistical unit.

[0113] The statistical unit can obtain target statistical information based on the statistical image, and the target statistical information can include a dynamic range extension value and a statistical brightness value. For example, the statistical image can include a first statistical image and a second statistical image, and the statistical unit can include a dynamic range statistical module and a brightness statistical module. For example, the first statistical image is an image before synthesis processing, an image after synthesis processing, an image before gain processing, an image after gain processing, an image before data compression processing, or an image after data compression processing, that is, the first statistical image can be any image involved in the image processing unit, such as an image after synthesis processing. The second statistical image is an image before synthesis processing, an image after synthesis processing, an image before gain processing, an image after gain processing, an image before data compression processing, or an image after data compression processing, that is, the second statistical image can be any image involved in the image processing unit, such as an image after data compression processing, etc.

[0114] Exemplarily, the dynamic range statistics module can obtain the dynamic range extension value based on the first statistical image, and the brightness statistics module can obtain the statistical brightness value based on the second statistical image. For example, the dynamic range statistics module can perform dynamic range statistics on the first statistical image to obtain the dynamic range extension value, and the brightness statistics module can perform brightness statistics on the second statistical image to obtain the statistical brightness value.

[0115] Dynamic range statistics module: After obtaining the first statistical image, the dynamic range statistics module can perform dynamic range statistics on the first statistical image to obtain a dynamic range extension value. For example, the highlight statistical value can be obtained by performing brightness statistics on the highlight area of ​​the first statistical image; the dynamic range extension value is determined based on the difference between the highlight statistical value and the highlight target value, and the preset dynamic range extension value threshold; wherein the preset dynamic range extension value threshold is determined based on the computing power or noise reduction capability of the image processing logic platform, so that the noise amplification of the digital processing based on the dynamic range extension value is within a preset range.

[0116] Exemplarily, the highlight area of ​​the first statistical image may be subjected to brightness statistics to obtain a highlight statistical value br_sta. The highlight statistical value br_sta may be calculated by taking the average of the pixel values ​​in the first statistical image whose brightness is between the preset highlight threshold br_thr1 and the preset highlight threshold br_thr2. The highlight statistical value br_sta may be calculated by taking the weighted average of the pixel values ​​in the first statistical image whose brightness is between the preset highlight threshold br_thr1 and the preset highlight threshold br_thr2, setting different weight values ​​for areas at different spatial positions of the image, and setting different weight values ​​for different brightness. The highlight statistical value br_sta may be calculated by summing the number of pixels in the first statistical image whose brightness is between the preset highlight threshold br_thr1 and the preset highlight threshold br_thr2. The highlight statistic br_sta may be calculated by weighting and summing the number of pixels whose brightness is between a preset highlight threshold br_thr1 and a preset highlight threshold br_thr2 in the first statistical image, setting different weight values ​​for regions at different spatial positions of the image, or setting different weight values ​​for the number of pixels with different brightness. Of course, the above is only an example of calculating the highlight statistic br_sta, and there is no limitation to this.

[0117] Assuming that the preset highlight target value is br_dst, and the highlight target value br_dst is not greater than the preset highlight threshold br_thr1, the dynamic range extension value may be calculated by determining the dynamic range extension value dr_rat based on the ratio of the highlight statistical value br_sta to the highlight target value br_dst, for example, dr_rat=br_sta / br_dst.

[0118] Exemplarily, the highlight target value br_dst refers to the brightness value of the highlight area of ​​the image that is more suitable for viewing. The highlight target value br_dst can be configured based on experience. The larger the highlight statistical value br_sta is, the more highlights there are in the current scene, and a larger dynamic range expansion value is needed to retain the highlight area information.

[0119] After obtaining the dynamic range extension value dr_rat, the dynamic range extension value dr_rat and the preset dynamic range threshold are smaller to obtain the final dynamic range extension value. The preset dynamic range threshold makes the noise amplification degree caused by the digital gain within a controllable range and is set according to the computing power or noise reduction level of the logic platform.

[0120] In summary, the dynamic range statistics module can perform dynamic range statistics on the first statistical image to obtain an initial value of dynamic range expansion, and the initial value of dynamic range expansion is related to the brightness distribution of the highlight area in the first statistical image. When the initial value of dynamic range expansion is greater than the preset dynamic range threshold, the dynamic range expansion value is the smaller one between the initial value of dynamic range expansion and the dynamic range threshold. The dynamic range threshold can control the degree of noise amplification caused by digital gain within a certain range. The dynamic range threshold can be set according to the computing power of the logic platform, the level of noise reduction processing, etc. When the dynamic range expansion value is obtained, the dynamic range expansion value is not less than 1. The dynamic range expansion value determines the maximum value that can be obtained by the acquired high dynamic range image.

[0121] Brightness statistics module: After obtaining the second statistical image, the brightness statistics module can perform brightness statistics on the second statistical image to obtain a statistical brightness value (which can be recorded as a statistical brightness value y_sta). The statistical brightness value y_sta is related to the brightness distribution of each area in the second statistical image.

[0122] Exemplarily, the calculation method of the statistical brightness value y_sta may be the result obtained by averaging the second statistical image. The calculation method of the statistical brightness value y_sta may be the result obtained by setting different weight values ​​for image areas of different brightness of the second statistical image, and then performing weighted averaging on the image. The calculation method of the statistical brightness value y_sta may be the result obtained by setting different weight values ​​for areas of different spatial positions of the second statistical image, and then performing weighted averaging on the image. The calculation method of the statistical brightness value y_sta may be to preset a brightness statistical threshold sta_thr, take the smaller of the second statistical image and sta_thr, and perform weighted averaging on the second statistical image to obtain a statistical brightness value, such as the weighted averaging result obtained by the above method (such as the result obtained by averaging the second statistical image; the result obtained by setting different weight values ​​for image areas of different brightness of the second statistical image, and then performing weighted averaging on the image; the result obtained by setting different weight values ​​for areas of different spatial positions of the second statistical image, and then performing weighted averaging on the image). Of course, the above are just a few examples and are not limited to this.

[0123] Fourth, exposure control unit.

[0124] After receiving the target statistical information, the exposure control unit obtains the target exposure control parameter and the target processing control parameter based on the target statistical information (such as the dynamic range extension value dr_rat and the statistical brightness value y_sta), and outputs the target exposure control parameter and the target processing control parameter to the cache unit. The target exposure control parameter is the exposure control parameter of the Kth frame, which is used by the image acquisition unit to expose the Kth frame image (such as the first image signal and the second image signal of the Kth frame). The target processing control parameter is the processing control parameter of the Kth frame, which is used by the image processing unit to perform image processing on the Kth frame image.

[0125] Exemplarily, the exposure control unit performs a first parameter adjustment on the historical exposure control parameter and the historical processing control parameter based on the statistical brightness value y_sta to obtain a candidate exposure control parameter and a candidate processing control parameter; wherein the first parameter adjustment is determined based on the adjusted statistical brightness value and the preset target brightness, for example, the first parameter adjustment is used to make the difference between the adjusted statistical brightness value and the preset target brightness less than a threshold value, that is, the adjusted statistical brightness value is close to the preset target brightness. The exposure control unit performs a second parameter adjustment on the candidate exposure control parameter and the candidate processing control parameter based on the dynamic range extension value dr_rat to obtain a target exposure control parameter and a target processing control parameter; wherein the second parameter adjustment is determined based on the adjusted target gain value and the adjusted dynamic range extension value, and the target gain value is the product value between the compensation gain and the digital gain. For example, the second parameter adjustment is used to make the difference between the adjusted target gain value and the adjusted dynamic range extension value less than a threshold value, that is, to make the actual gain value reach or approach the dynamic range extension value.

[0126] Exemplarily, the historical exposure control parameter is the exposure control parameter currently used by the image acquisition unit (such as the exposure control parameter of the K-1th frame, the exposure control parameter of the K-2th frame, etc.), and the historical processing control parameter is the processing control parameter currently used by the image processing unit (such as the processing control parameter of the K-1th frame, the processing control parameter of the K-2th frame, etc.). The candidate exposure control parameter is the exposure control parameter adjusted by the first parameter, and the candidate processing control parameter is the processing control parameter adjusted by the first parameter. The target exposure control parameter is the exposure control parameter adjusted by the second parameter, which can be fed back to the image acquisition unit as the exposure control parameter of the Kth frame, and the target processing control parameter is the processing control parameter adjusted by the second parameter, which can be fed back to the image processing unit as the processing control parameter of the Kth frame.

[0127] For exposure control parameters (such as historical exposure control parameters, candidate exposure control parameters, and target exposure control parameters), the exposure control parameters may include, but are not limited to, aperture, exposure time, a first conversion gain corresponding to the first image signal, a first sensor gain corresponding to the first image signal, a second conversion gain corresponding to the second image signal, and a second sensor gain corresponding to the second image signal. Obviously, the exposure control parameters include parameters for the first image signal and the second image signal, and different sensor gains may be configured for different image signals to achieve more flexible dynamic range adjustment and a higher signal-to-noise ratio.

[0128] For the processing control parameters (such as historical processing control parameters, candidate processing control parameters, and target processing control parameters), the processing control parameters may include, but are not limited to, compensation gain and digital gain. Since the compensation gain is a ratio between the first gain value and the second gain value, the first gain value is a product value of the first conversion gain and the first sensor gain, and the second gain value is a product value of the second conversion gain and the second sensor gain, after obtaining the exposure control parameters such as the first conversion gain, the first sensor gain, the second conversion gain, and the second sensor gain, the compensation gain may be obtained based on these parameters.

[0129] For example, see Figure 5 As shown, it is a schematic diagram of the structure of the exposure control unit, and the exposure control unit may include a first exposure control module and a second exposure control module. The first exposure control module performs a first parameter adjustment on the historical exposure control parameter and the historical processing control parameter based on the statistical brightness value y_sta to obtain a candidate exposure control parameter and a candidate processing control parameter. The second exposure control module performs a second parameter adjustment on the candidate exposure control parameter and the candidate processing control parameter based on the dynamic range extension value dr_rat to obtain a target exposure control parameter and a target processing control parameter, and outputs the target exposure control parameter and the target processing control parameter. The functions of the first exposure control module and the second exposure control module are described below.

[0130] First exposure control module: The first exposure control module controls the historical exposure control parameter EXPO based on the statistical brightness value y_sta and the preset target brightness K-N and history processing control parameters PRO K-N (K represents the frame number of the current frame image, N is greater than or equal to 1, and KN represents the frame number of the historical frame image) perform the first adjustment to obtain the candidate exposure control parameter EXPO of the Kth frame K_ini and the candidate processing control parameter PRO of the Kth frame K_ini The preset target brightness refers to the brightness value that makes the image more suitable for subsequent applications and can be configured based on experience.

[0131] Exemplarily, the candidate exposure control parameter EXPOK_ini Can include but not limited to candidate aperture lr K_ini , candidate exposure time t K_ini , a candidate first sensor gain sg_1 corresponding to the first image signal K_ini , the candidate first conversion gain hg_1 corresponding to the first image signal K_ini , the candidate second sensor gain sg_s corresponding to the second image signal K_ini , the candidate second conversion gain hg_s corresponding to the second image signal K_ini .

[0132] Regarding the aperture and exposure time, the first exposure control module can adjust the aperture and exposure time in the historical exposure control parameters to obtain a candidate aperture lr K_ini and candidate exposure time t K_ini It should be noted that if there is no automatic aperture in the image acquisition unit, the subsequent aperture parameter adjustment process can be omitted, that is, the aperture is an optional parameter, and the following description will be given with the aperture parameter as an example.

[0133] Regarding the first conversion gain and the second conversion gain, the conversion gain is a property of the image sensor. When the image sensor leaves the factory, the ratio of the first conversion gain to the second conversion gain has been configured, and the ratio is 1 or the maximum value hg max If the ratio is 1, it means that the first conversion gain is the same as the second conversion gain. If the ratio is the maximum value hg max , then the first conversion gain is hg of the second conversion gain max times.

[0134] If the adjustment of the first conversion gain and the second conversion gain is not supported, the first conversion gain corresponding to the first image signal can be fixed to hg max , the second conversion gain corresponding to the second image signal can be fixed to 1. Based on this, the first exposure control module may not adjust the first conversion gain and the second conversion gain in the historical exposure control parameters, that is, keep the first conversion gain and the second conversion gain unchanged. In this way, the candidate first conversion gain is hg max , the candidate second conversion gain is 1. Alternatively, if the adjustment of the first conversion gain and the second conversion gain is supported, the first exposure control module can adjust the first conversion gain and the second conversion gain in the historical exposure control parameters to obtain the candidate first conversion gain and the candidate second conversion gain. For the convenience of description, the case where the adjustment of the first conversion gain and the second conversion gain is not supported is taken as an example.

[0135] Regarding the first sensor gain and the second sensor gain, different sensor gains can be configured for the first image signal and the second image signal, thereby further adjusting the dynamic range. Based on this, the first exposure control module can adjust the first sensor gain and the second sensor gain in the historical exposure control parameters to obtain a candidate first sensor gain sg_l K_ini and the candidate second sensor gain sg_s K_ini .

[0136] Exemplarily, the candidate process control parameter PRO K_ini including but not limited to candidate compensation gain cdg K_ini and the candidate digital gain dg K_ini Regarding the compensation gain, since the compensation gain is a ratio value between the first gain value and the second gain value, the first gain value is a product value of the first conversion gain and the first sensor gain, and the second gain value is a product value of the second conversion gain and the second sensor gain, the candidate compensation gain cdg can be determined based on the candidate first conversion gain, the candidate second conversion gain, the candidate first sensor gain, and the candidate second sensor gain. K_ini Regarding the digital gain, the first exposure control module can adjust the digital gain in the historical processing control parameter to obtain a candidate digital gain dg K_ini .

[0137] In summary, it can be seen that the parameters to be adjusted may include aperture, exposure time, first sensor gain, second sensor gain and digital gain, while the first conversion gain and the second conversion gain are not adjusted (of course, in some scenarios, the first conversion gain and the second conversion gain can also be adjusted, and this embodiment takes no adjustment as an example), and the compensation gain is derived based on the conversion gain and the sensor gain.

[0138] Exemplarily, the first exposure control module processes the historical exposure control parameter EXPO K-N When adjusting, the adjustment direction can be to increase EXPO K-N , or it can reduce EXPO K-N , get the candidate exposure control parameter EXPO K_ini The first exposure control module processes the historical control parameter PRO K-N When adjusting, the adjustment direction can be to increase PRO K-N , or it can be to reduce PRO K-N , get the candidate processing control parameters PRO K_ini Among them, in the EXPO K-N and PRO K-NWhen performing adjustment processing, the adjustment step size refers to the amount of change in the adjustment direction, which can be set as a fixed step size or a dynamic change step size. The fixed step size can be set according to application requirements. The dynamic change step size can be set according to the difference between the statistical brightness value y_sta and the preset target brightness. The greater the difference, the larger the dynamic change step size.

[0139] For example, in the EXPO K-N and PRO K-N When adjusting, if the statistical brightness value y_sta is less than the preset target brightness, then EXPO K-N Perform enlargement to obtain EXPO K_ini , and can be PRO K-N Pro K_ini In this way, the statistical brightness value after the first adjustment is close to the preset target brightness, that is, the difference between the adjusted statistical brightness value and the preset target brightness is less than the threshold.

[0140] When the statistical brightness value y_sta is less than the preset target brightness, the order of parameter adjustment is exposure time, aperture, sensor gain (such as the first sensor gain and the second sensor gain), and digital gain. Of course, this parameter adjustment order is just an example, and there is no restriction on this parameter adjustment order. For example, the parameter adjustment order is aperture, exposure time, sensor gain, and digital gain. Alternatively, the parameter adjustment order is sensor gain, digital gain, aperture, and exposure time. Alternatively, the parameter adjustment order is digital gain, sensor gain, aperture, exposure time, and so on, which can be any order.

[0141] For example, in the EXPO K-N and PRO K-N When adjusting, if the statistical brightness value y_sta is greater than the preset target brightness, then EXPO K-N Reduce the processing to get EXPO K_ini , and can be PRO K-N Reduce the processing to get PRO K_ini In this way, the statistical brightness value after the first adjustment is close to the preset target brightness, that is, the difference between the adjusted statistical brightness value and the preset target brightness is less than the threshold.

[0142] When the statistical brightness value y_sta is greater than the preset target brightness, the parameter adjustment order is digital gain, sensor gain (such as the first sensor gain and the second sensor gain), aperture, and exposure time. Of course, this parameter adjustment order is just an example, and there is no restriction on this parameter adjustment order. For example, the parameter adjustment order is exposure time, aperture, sensor gain, and digital gain. Alternatively, the parameter adjustment order is sensor gain, digital gain, aperture, and exposure time. Alternatively, the parameter adjustment order is digital gain, sensor gain, exposure time, aperture, and so on, which can be any order.

[0143] For example, the statistical brightness value after the first adjustment refers to the EXPO K_ini EXPO K-N Changes between PRO K_ini With PRO K-N The changes between them act together on the statistical brightness value to obtain the brightness value.

[0144] For example, when the statistical brightness value y_sta is equal to the preset target brightness, then no parameter adjustment is required. In this case, EXPO K_ini =EXPO K-N , and PRO K_ini =PRO K-N .

[0145] Exemplarily, in the DCG mode, the image acquisition unit outputs a brighter first image signal and a darker second image signal, the first image signal and the second image signal share the same exposure time, and the first image signal and the second image signal share the same aperture. The first image signal corresponds to the first conversion gain and the first sensor gain, and the second image signal corresponds to the second conversion gain and the second sensor gain. In summary, the historical exposure control parameter EXPO K-N (Take N as 1 as an example) including the first conversion gain hg_1 k-1 , the second conversion gain hg_s k-1 , the first sensor gain sg_1 k-1 , the second sensor gain sg_s k-1 , aperture lr k-1 , exposure time t k-1 . Candidate exposure control parameters EXPO K_ini The first conversion gain hg_1 may be included k , the second conversion gain hg_s k , the first sensor gain sg_1 k , the second sensor gain sg_s k , aperture lr k , exposure time t k In addition, the history processing control parameter PROK-N Including compensation gain cdg k-1 and digital gain dg k-1 , and the candidate processing control parameter PRO K_ini Including compensation gain cdg k and digital gain dg k . Compensation Gain The dynamic range extension value dr_rat k =cdg k ×dg k .

[0146] It should be noted that if DCG technology only supports switching of conversion gain, hg max The value in each image sensor is independent and fixed (i.e. hg is configured separately for each image sensor). max , and hg max The following takes the conversion gain as an example, but the conversion gain can also be adjusted frame by frame. Set to factory maximum value hg max , and the second conversion gain Set to 1, if the first sensor gain Equal to the second sensor gain And the digital gain dg k-1 is 1, then the dynamic range expansion value Can be hg max In the subsequent process, the first sensor gain can be set Not equal to the second sensor gain And the ratio between the two is constant, in this way the dynamic range extension value is modified

[0147] Exemplarily, by comparing the statistical brightness value y_sta with the preset target brightness y_dst, the exposure control parameters and the processing control parameters can be adjusted so that the statistical brightness value of the subsequently acquired image is equal to or close to the preset target brightness. The preset target brightness y_dst refers to a brightness value that makes the image more suitable for subsequent applications. The parameter adjustment process is described below.

[0148] Case 1: If the statistical brightness value y_sta is less than the preset target brightness y_dst, the parameters may be adjusted based on the quotient of the preset target brightness y_dst and the statistical brightness value y_sta (ie, the ratio of the two).

[0149] Step S11: Determine a first brightness adjustment amount based on the quotient of the preset target brightness y_dst and the statistical brightness value y_sta. For example, the first brightness adjustment amount may be y_dst / y_sta. Obviously, since the preset target brightness y_dst is larger, the first brightness adjustment amount is greater than 1, and the first brightness adjustment amount is used to increase the parameter.

[0150] Step S12: Obtain a first set of parameters to be adjusted, the first set of parameters to be adjusted including at least one of a historical exposure control parameter and a historical processing control parameter. For example, the first set of parameters to be adjusted includes exposure time t k-1 , aperture lr k-1 , sensor gain (such as the first sensor gain sg_l k-1 , the second sensor gain sg_s k-1 ), digital gain dg k-1 Of course, this is just an example of the parameter order, and other parameter orders can also be used. There is no limitation on the parameter order in the first parameter set to be adjusted. This parameter order is used as an example later, and the implementation methods of other parameter orders are similar, which will not be described later. When the parameters of the first parameter set to be adjusted are adjusted based on the above parameter order, noise amplification can be reduced.

[0151] After obtaining the first brightness adjustment amount and the first set of parameters to be adjusted, the parameters in the first set of parameters to be adjusted can be adjusted based on the first brightness adjustment amount to obtain adjusted parameters. For example, the first parameter in the first set of parameters to be adjusted is adjusted first. If the first brightness adjustment amount is satisfied, the adjustment process ends. If the first brightness adjustment amount is not satisfied, the second parameter in the first set of parameters to be adjusted is adjusted. If the first brightness adjustment amount is satisfied, the adjustment process ends. If the first brightness adjustment amount is not satisfied, the third parameter in the first set of parameters to be adjusted is adjusted, and so on, until the first brightness adjustment amount is satisfied. On this basis, candidate exposure control parameters and candidate processing control parameters can be obtained based on the adjusted parameters. The process is described below.

[0152] Step S13: If the first product value of the exposure time and the first brightness adjustment amount is less than or equal to the exposure time upper limit threshold, the adjusted exposure time is determined to be the first product value, and the parameter adjustment process ends.

[0153] If the first product value of the exposure time and the first brightness adjustment amount is greater than the exposure time upper limit threshold, the adjusted exposure time is determined to be within the exposure time upper limit threshold, and step S14 is executed.

[0154] For example, first increase the exposure time k represents the frame number, t k-1Indicates the exposure time used when acquiring the current frame image, that is, the historical exposure time, t k represents the adjusted exposure time, that is, the candidate exposure time, and y_dst / y_sta represents the first brightness adjustment amount. Based on the above adjustment, after increasing the exposure time, the adjusted statistical brightness value That is equal to the preset target brightness.

[0155] After the adjustment, the exposure time t k Afterwards, the adjusted exposure time t k Compared with the upper limit threshold t_max of the exposure time, the upper limit threshold t_max of the exposure time can avoid the tailing of moving objects in the image caused by excessive exposure time, and also avoid the number of image frames collected per unit time not meeting the application requirements.

[0156] If the adjusted exposure time t k If it is less than or equal to the upper limit threshold of exposure time t_max, the adjusted exposure time t k unchanged, that is, the adjusted exposure time t k As the candidate exposure time t k , and there is no need to adjust other parameters, that is, the aperture, sensor gain, digital gain and other parameters remain unchanged, that is, lr k =lr k-1 、sg_s k =sg_s k-1 、sg_l k =sg_l k-1 、dg k =dg k-1 In this way, the candidate exposure control parameters (such as the candidate first conversion gain hg_1) can be obtained. k , candidate second conversion gain hg_s k , candidate first sensor gain sg_1 k , candidate second sensor gain sg_s k , candidate aperture lr k , candidate exposure time t k etc.) and candidate processing control parameters (such as candidate compensation gain cdg k and the candidate digital gain dg k wait).

[0157] If the adjusted exposure time t k If it is greater than the upper threshold of exposure time t_max, the exposure time t k Updated to t_max, t k =t max , adjusted exposure time t max is the candidate exposure time t k , execute step S14.

[0158] Step S14: based on the exposure time upper limit threshold t_max and the exposure time t k-1 The first remaining adjustment amount is determined by the first brightness adjustment amount y_dst / y_sta. For example, the first brightness adjustment amount indicates the magnification factor required, and the exposure time upper limit threshold t_max is related to the exposure time t k-1 The ratio between represents the magnification factor (ie, the magnification factor that has been magnified by the exposure time). Based on the above two parameters, the remaining magnification factor can be obtained, and this remaining magnification factor is used as the first remaining adjustment amount. There is no restriction on this determination process.

[0159] If the second product value of the aperture and the first remaining adjustment amount is less than or equal to the aperture upper limit threshold, the adjusted exposure time is determined to be the exposure time upper limit threshold and the adjusted aperture is determined to be the second product value.

[0160] If the second product value of the aperture and the first remaining adjustment amount is greater than the aperture upper limit threshold, the adjusted exposure time is determined to be the exposure time upper limit threshold, the adjusted aperture is determined to be the aperture upper limit threshold, and step S15 is executed.

[0161] For example, increasing the aperture lr k >lr k-1 , such as lr k-1 The product value of the first residual adjustment amount (the first residual adjustment amount is greater than 1), lr k-1 Represents historical aperture, lr k Represents the adjusted aperture, i.e., the candidate aperture. Based on the above adjustment, after increasing the exposure time and aperture, the adjusted statistical brightness value is equal to the preset target brightness, i.e., the adjusted statistical brightness value is equal to the preset target brightness by increasing the amount of incoming light.

[0162] After adjusting the aperture lr k After that, you can also adjust the aperture lr k Compare with the aperture upper limit threshold lr_max. If the adjusted aperture lr is less than or equal to the aperture upper limit threshold lr_max, keep the adjusted aperture lr k unchanged, that is, the aperture lr after adjustment k As a candidate aperture lr k , and no other parameters are adjusted, that is, the sensor gain, digital gain and other parameters remain unchanged, that is, sg_s k =sg_s k-1 、sg_l k =sg_l k-1 、dg k =dg k-1 In this way, candidate exposure control parameters and candidate processing control parameters can be obtained.

[0163] If you adjust the aperture lr k If it is greater than the aperture upper limit threshold lr_max, the aperture lr will be adjusted k Updated to the aperture upper limit threshold lr_max, i.e. lr k = lr_max, in this case, the adjusted exposure time t max As the candidate exposure time t k , the adjusted aperture lr_max is used as the candidate aperture lr k , execute step S15.

[0164] Step S15: based on the aperture upper limit threshold lr_max and the aperture lr k-1 For example, the first remaining adjustment amount indicates the magnification factor required, and the aperture upper limit threshold lr_max is related to the aperture lr k-1 The ratio between represents the magnification factor that has been magnified. Based on the above two parameters, the remaining magnification factor can be obtained, and this remaining magnification factor can be used as the second remaining adjustment amount.

[0165] If the third product value of the first sensor gain and the second remaining adjustment amount is less than or equal to the first sensor gain upper limit threshold, then it can be determined that the adjusted exposure time is the exposure time upper limit threshold, the adjusted aperture is the aperture upper limit threshold, the adjusted first sensor gain is the third product value, and the adjusted second sensor gain is the product value of the second sensor gain and the second remaining adjustment amount.

[0166] If the third product value of the first sensor gain and the second remaining adjustment amount is greater than the first sensor gain upper limit threshold, then it can be determined that the adjusted exposure time is the exposure time upper limit threshold, the adjusted aperture is the aperture upper limit threshold, the adjusted first sensor gain is the first sensor gain upper limit threshold, and the adjusted second sensor gain is the second sensor gain upper limit threshold, and step S16 is executed.

[0167] For example, increase the first sensor gain sg_l k >sg_l k-1 , such as sg_l k-1 The product value of sg_1 and the second remaining adjustment amount (the second remaining adjustment amount is greater than 1) is k-1 Indicates the historical first sensor gain, sg_l k represents the adjusted first sensor gain, ie, the candidate first sensor gain, which is also called the long frame sensor gain. Based on the above adjustment, the adjusted statistical brightness value is equal to the preset target brightness.

[0168] After adjustment, the first sensor gain sg_l kAfterwards, the adjusted first sensor gain sg_l can also be k and the first sensor gain upper limit threshold sg_l max If the first sensor gain sg_l is adjusted k Less than or equal to the first sensor gain upper limit threshold sg_1 max , then keep the adjusted first sensor gain sg_l k unchanged, that is, the adjusted first sensor gain sg_l k As the candidate first sensor gain. In this process, the candidate second sensor gain sg_s k Maintain the candidate first sensor gain sg_l k The same degree of change, while not exceeding the second sensor gain upper threshold sg_s max Therefore, the second sensor gain sg_s can also be increased k >sg_s k-1 , sg_s k represents the adjusted second sensor gain, and the adjusted second sensor gain sg_s k is the second sensor gain sg_s k-1 The product value of the second residual adjustment amount. If the adjusted second sensor gain sg_s k Less than or equal to the second sensor gain upper threshold sg_s max , then keep the adjusted second sensor gain sg_s k unchanged, that is, the adjusted second sensor gain sg_s k As a candidate for the second sensor gain. On this basis, no other parameters are adjusted, that is, the digital gain remains unchanged, that is, dg k =dg k-1 In this way, candidate exposure control parameters and candidate processing control parameters can be obtained.

[0169] After adjustment, the first sensor gain sg_l k Afterwards, if the adjusted first sensor gain sg_l k Greater than the first sensor gain upper limit threshold sg_1 max , then the adjusted first sensor gain sg_l k Updated to the first sensor gain upper limit threshold sg_l max , the second sensor gain sg_s will be adjusted k Updated to the second sensor gain upper threshold sg_s max On this basis, the adjusted exposure time t max As the candidate exposure time t k , the adjusted aperture lr_max is used as the candidate aperture lrk , after adjustment, the gain of the first sensor sg_l max As a candidate first sensor gain sg_1 k , adjusted second sensor gain sg_s max As a candidate for the second sensor gain sg_s k , in this case, continue to step S16.

[0170] Step S16: Based on the first sensor gain upper limit threshold sg_1 max , the first sensor gain sg_1 k-1 The third remaining adjustment amount is determined by the second remaining adjustment amount. For example, the second remaining adjustment amount indicates the multiple required for amplification, and the first sensor gain upper limit threshold sg_1 max With the first sensor gain sg_l k-1 The ratio between represents the magnification factor. Based on the above two parameters, the third remaining adjustment amount can be obtained.

[0171] For example, increasing the digital gain dg k >dg k-1 , such as dg k-1 The product value of the third residual adjustment amount (the third residual adjustment amount is greater than 1), dg k-1 represents the historical digital gain, dg k The above adjustment makes the adjusted statistical brightness value equal to the preset target brightness.

[0172] In summary, the adjusted exposure time t can be determined k is the upper threshold of exposure time t max , adjust the aperture lr k is the aperture upper limit threshold lr_max, the adjusted first sensor gain sg_l k is the first sensor gain upper limit threshold sg_1 max , adjusted second sensor gain sg_s k The second sensor gain upper threshold sg_s max , after adjustment, the digital gain dg k is the product of the digital gain and the third residual adjustment amount.

[0173] Case 2: If the statistical brightness value y_sta is greater than the preset target brightness y_dst, the parameters may be adjusted based on the quotient of the preset target brightness y_dst and the statistical brightness value y_sta (ie, the ratio of the two).

[0174] Step S21: Determine a second brightness adjustment amount based on the quotient of the preset target brightness y_dst and the statistical brightness value y_sta. For example, the second brightness adjustment amount may be y_dst / y_sta. Obviously, since the preset target brightness y_dst is smaller, the second brightness adjustment amount is less than 1, and is used to reduce the parameter.

[0175] Step S22: Obtain a second set of parameters to be adjusted, the second set of parameters to be adjusted including at least one of the historical exposure control parameters and the historical processing control parameters. For example, the second set of parameters to be adjusted includes the digital gain dg k-1 , sensor gain (such as the first sensor gain sg_l k-1 , the second sensor gain sg_s k-1 ), aperture lr k-1 , exposure time t k-1 Of course, this is just an example of the parameter order, and other parameter orders can also be used. There is no restriction on the parameter order in the second parameter set to be adjusted. This parameter order is used as an example in the following, and the implementation methods of other parameter orders are similar, which will not be repeated in the following. When adjusting based on the above parameter order, noise amplification can be reduced and stroboscopic phenomenon can be avoided.

[0176] After obtaining the second brightness adjustment amount and the second set of parameters to be adjusted, the parameters in the second set of parameters to be adjusted are adjusted based on the second brightness adjustment amount to obtain adjusted parameters. For example, the first parameter is adjusted. If the second brightness adjustment amount is satisfied, the adjustment process ends. If the second brightness adjustment amount is not satisfied, the second parameter continues to be adjusted, and so on. Then, candidate exposure control parameters and candidate processing control parameters can be obtained based on the adjusted parameters. The process is described below.

[0177] Step S23: If the first product value of the digital gain and the second brightness adjustment amount is greater than or equal to the digital gain lower limit threshold, the adjusted digital gain is determined to be the first product value, and the parameter adjustment process ends.

[0178] If the first product value of the digital gain and the second brightness adjustment amount is less than the digital gain lower limit threshold, the adjusted digital gain is determined to be the digital gain lower limit threshold, and step S24 is executed.

[0179] For example, first reduce the digital gain k represents the frame number, dgk -1 Indicates the digital gain used in the current frame image processing, that is, the historical digital gain, dg k represents the adjusted digital gain, i.e. the candidate digital gain, Based on the above adjustment, by reducing the digital magnification, the adjusted statistical brightness value is That is equal to the preset target brightness.

[0180] After the adjustment, the digital gain dg k After that, the adjusted digital gain dg k Compare with the digital gain lower limit threshold dg_min. If the adjusted digital gain dg k If the digital gain is greater than or equal to the lower limit threshold dg_min, the adjusted digital gain dg is maintained. k unchanged, that is, the adjusted digital gain dg k As a candidate digital gain dg k At this time, there is no need to adjust other parameters, that is, the sensor gain, aperture, exposure time and other parameters remain unchanged, that is, sg_s k =sg_s k-1 、sg_l k =sg_l k-1 , lr k =lr k-1 ,t k =t k-1 In this way, candidate exposure control parameters and candidate processing control parameters can be obtained.

[0181] If the adjusted digital gain dg k is less than the digital gain lower limit threshold dg_min, the adjusted digital gain dg k Updated to dg_min, the adjusted digital gain dg_min is the candidate digital gain dg k , execute step S24.

[0182] Step S24: based on the digital gain lower limit threshold dg_min, the digital gain dg k-1 The first remaining adjustment amount is determined by the second brightness adjustment amount y_dst / y_sta. For example, the second brightness adjustment amount indicates the multiple that needs to be reduced, and the digital gain lower limit threshold dg_min is related to the digital gain dg k-1 The ratio between represents the multiple of reduction. Based on the above two parameters, the remaining reduction multiple, that is, the first remaining adjustment amount, can be obtained.

[0183] If the second product value of the first sensor gain and the first remaining adjustment amount is greater than or equal to the first sensor gain lower limit threshold, the adjusted digital gain is determined to be the digital gain lower limit threshold, the adjusted first sensor gain is the second product value, and the adjusted second sensor gain is the product value between the second sensor gain and the first remaining adjustment amount. If the second product value is less than the first sensor gain lower limit threshold, the adjusted digital gain is determined to be the digital gain lower limit threshold, the adjusted first sensor gain is the first sensor gain lower limit threshold, and the adjusted second sensor gain is the second sensor gain lower limit threshold, and step S25 is executed.

[0184] For example, reducing the first sensor gain to sg_l k-1 Indicates the historical first sensor gain, sg_l k represents the adjusted first sensor gain, i.e., the candidate first sensor gain, represents a first remaining adjustment amount, which may be a value less than 1. Based on the above adjustment, the adjusted statistical brightness value can be equal to the preset target brightness.

[0185] After adjustment, the first sensor gain sg_l k Afterwards, the adjusted first sensor gain sg_l can also be k and the first sensor gain lower limit threshold sg_l min If the first sensor gain sg_l is adjusted k Greater than or equal to the first sensor gain lower limit threshold sg_1 min , then keep the adjusted first sensor gain sg_l k unchanged, that is, the adjusted first sensor gain sg_l k As the candidate first sensor gain. In this process, the candidate second sensor gain sg_s k Maintain the candidate first sensor gain sg_l k The same degree of change, while not exceeding the second sensor gain lower limit threshold sg_s min , therefore, the second sensor gain sg_s can also be reduced k <sg_s k-1 , sg_s k represents the adjusted second sensor gain, and the adjusted second sensor gain sg_s k is the second sensor gain sg_s k-1 The product value of the first residual adjustment amount. If the adjusted second sensor gain sg_s k Greater than or equal to the second sensor gain lower limit threshold sg_s min, then keep the adjusted second sensor gain sg_s unchanged, that is, the adjusted second sensor gain sg_s k As a candidate for the second sensor gain. On this basis, no other parameters are adjusted, and the aperture and exposure time remain unchanged, that is, lr k =lr k-1 ,t k =t k-1 In this way, candidate exposure control parameters and candidate processing control parameters can be obtained.

[0186] After adjustment, the first sensor gain sg_l k Afterwards, if the adjusted first sensor gain sg_l k Less than the first sensor gain lower limit threshold sg_1 min , then the first sensor gain sg_l will be adjusted k Updated to the first sensor gain lower limit threshold sg_1 min , the second sensor gain sg_s will be adjusted k Updated to the second sensor gain lower limit threshold sg_s min On this basis, the adjusted digital gain dg_min is used as the candidate digital gain dg k , after adjustment, the gain of the first sensor sg_l min As a candidate first sensor gain sg_1 k , adjusted second sensor gain sg_s min As a candidate for the second sensor gain sg_s k , execute step S25.

[0187] Step S25: based on the first sensor gain lower limit threshold sg_1 min , the first sensor gain sg_1 k-1 For example, the first remaining adjustment amount indicates the multiple that needs to be reduced, and the first sensor gain lower limit threshold sg_1 min With the first sensor gain sg_l k-1 The ratio between represents the multiple of reduction. Based on the above two parameters, the second remaining adjustment amount can be obtained.

[0188] If the third product value of the aperture and the second remaining adjustment amount is greater than or equal to the aperture lower limit threshold, it can be determined that the adjusted digital gain is the digital gain lower limit threshold, the adjusted first sensor gain is the first sensor gain lower limit threshold, the adjusted second sensor gain is the second sensor gain lower limit threshold, and the adjusted aperture is the third product value of the aperture and the second remaining adjustment amount. Alternatively, if the third product value is less than the aperture lower limit threshold, it can be determined that the adjusted digital gain is the digital gain lower limit threshold, the adjusted first sensor gain is the first sensor gain lower limit threshold, the adjusted second sensor gain is the second sensor gain lower limit threshold, and the adjusted aperture is the aperture lower limit threshold, and step S26 is continued.

[0189] For example, reducing the aperture lr k <lr k-1 , such as lr k-1 The product value of the second residual adjustment amount (the second residual adjustment amount is less than 1), lr k-1 Represents historical aperture, lr k represents the aperture after adjustment, that is, the candidate aperture. Based on the above adjustment, the adjusted statistical brightness value can be equal to the preset target brightness.

[0190] After adjusting the aperture lr k After that, you can adjust the aperture lr k Compare with the aperture lower limit threshold lr_min. k If it is greater than or equal to the aperture lower limit threshold lr_min, the adjusted aperture lr is maintained k unchanged, that is, the aperture lr after adjustment k As a candidate aperture lr k , and other parameters are not adjusted, such as the exposure time remains unchanged, that is, t k =t k-1 , thereby obtaining candidate exposure control parameters and candidate processing control parameters.

[0191] If you adjust the aperture lr k If the adjusted statistical brightness value is still greater than the preset target brightness, the aperture lr will be adjusted. k Update to the aperture lower limit threshold lr_min, and execute step S26.

[0192] Step S26: based on the aperture lower limit threshold lr_min, aperture lr k-1 The third remaining adjustment amount is determined by the second remaining adjustment amount. For example, the second remaining adjustment amount indicates the multiple that needs to be reduced, and the aperture lower limit threshold lr_min is equal to the aperture lr k-1 The ratio between represents the multiple of reduction, so the third remaining adjustment amount can be determined.

[0193] If the fourth product value of the exposure time and the third remaining adjustment amount is greater than or equal to the exposure time lower limit threshold, it can be determined that the adjusted digital gain is the digital gain lower limit threshold, the adjusted first sensor gain is the first sensor gain lower limit threshold, the adjusted second sensor gain is the second sensor gain lower limit threshold, the adjusted aperture is the aperture lower limit threshold, and the adjusted exposure time is the fourth product value.

[0194] For example, reducing the exposure time t k <t k-1 , such as t k-1 The product value of the third residual adjustment amount (the third residual adjustment amount is less than 1), t k-1 represents the historical exposure time, t k represents the adjusted exposure time, ie, the candidate exposure time. Based on the above adjustment, the adjusted statistical brightness value can be equal to the preset target brightness.

[0195] After the adjustment, the exposure time t k Afterwards, the adjusted exposure time t k With the exposure time lower limit threshold t min If the adjusted exposure time t k Greater than or equal to the lower limit threshold of exposure time t min , then keep the adjusted exposure time t k unchanged, that is, the adjusted exposure time t k As the candidate exposure time t k , so far, candidate exposure control parameters and candidate processing control parameters can be obtained.

[0196] In a possible implementation, if the fourth product value of the exposure time and the third remaining adjustment amount is less than the exposure time lower limit threshold, it can be determined that the adjusted first sensor gain is the first sensor gain lower limit threshold, the adjusted second sensor gain is the second sensor gain lower limit threshold, the adjusted aperture is the aperture lower limit threshold, and the adjusted exposure time is the exposure time lower limit threshold. On this basis, the digital gain can continue to be adjusted, that is, based on the digital gain lower limit threshold, the digital gain is reduced and adjusted until the total adjustment amount is the second brightness adjustment amount, and the adjusted statistical brightness value is equal to the preset target brightness. In this way, the adjusted digital gain can be used as a candidate digital gain to complete the parameter adjustment process.

[0197] Case 3: If the statistical brightness value y_sta is equal to the preset target brightness y_dst, there is no need to adjust the historical exposure control parameters and the historical processing control parameters. For example, the candidate exposure control parameters can be the same as the historical exposure control parameters, and the candidate processing control parameters can be the same as the historical processing control parameters.

[0198] After the above parameter adjustment (case 1, case 2 or case 3), the statistical brightness value of the k+1 frame after image acquisition and image processing is equal to the preset target brightness. When the statistical brightness value of the kth frame is significantly different from the preset target brightness, the adjustment step is large, which may cause a large change in the image brightness of the k+1 frame and cause inter-frame flickering. In this regard, the adjustment step can be reduced, and the exposure control parameters and processing control parameters can be gradually adjusted for multiple frames to achieve the purpose of making the statistical brightness value equal to the preset target brightness in the k+n (n>1) frame. The adjustment method is similar to the above method, except that the parameter change between adjacent frames is reduced, and it will not be expanded in detail.

[0199] In addition, the exposure control parameters and processing control parameters used by the current frame image referenced by the first exposure control module may be parameters acquired in the km (m>=1)th frame, which are determined by the cache unit and application requirements.

[0200] Second exposure control module: The second exposure control module is based on the dynamic range extension value dr_rat k For candidate exposure control parameters EXPO K_ini and candidate process control parameters PRO K_ini Make a second adjustment to get the target exposure control parameter EXPO for the Kth frame K and the target processing control parameter PRO of the Kth frame K For the second adjustment, the adjustment process may include two aspects: adjustment direction and adjustment step size.

[0201] Candidate exposure control parameters EXPO K_ini Can include but not limited to candidate aperture lr K_ini , candidate exposure time t K_ini , candidate first sensor gain sg_1 K_ini , candidate first conversion gain hg_1 K_ini , candidate second sensor gain sg_s K_ini , candidate second conversion gain hg_s K_ini Target exposure control parameter EXPO K May include but not limited to target aperture lr K , target exposure time t K , target first sensor gain sg_l K , target first conversion gain hg_l K , target second sensor gain sg_s K , target second conversion gain hg_s K .

[0202] Regarding the aperture and exposure time, the second exposure control module can adjust the aperture and exposure time in the candidate exposure control parameters to obtain the target aperture lr Kand target exposure time t K It should be noted that if there is no automatic aperture in the image acquisition unit, the subsequent aperture parameter adjustment process can be omitted, that is, the aperture is an optional parameter, and the following description will be given with the aperture parameter as an example.

[0203] Regarding the first conversion gain and the second conversion gain, if the adjustment of the first conversion gain and the second conversion gain is not supported, the first conversion gain can be fixed to hg max , the second conversion gain is fixed to 1.

[0204] Regarding the first sensor gain and the second sensor gain, different sensor gains can be configured, and the second exposure control module can adjust the first sensor gain and the second sensor gain in the candidate exposure control parameters to obtain the target first sensor gain sg_l K and the target second sensor gain sg_s K .

[0205] Exemplarily, the candidate process control parameter PRO K_ini including but not limited to candidate compensation gain cdg K_ini and the candidate digital gain dg K_ini . Target processing control parameters PRO K Including but not limited to the target compensation gain cdg of the synthesis processing module K (the synthesis processing module performs synthesis processing based on the compensation gain), the target digital gain dg of the gain processing module K (The gain processing module performs gain processing based on the digital gain). Regarding the compensation gain, a target compensation gain cdg may be determined based on the target first conversion gain, the target second conversion gain, the target first sensor gain, and the target second sensor gain. K Regarding the digital gain, the second exposure control module can adjust the digital gain in the candidate processing control parameter to obtain the target compensation gain cdg K .

[0206] Exemplarily, the second exposure control module processes the candidate exposure control parameter EXPO K_ini and candidate process control parameters PRO K_ini Make a second adjustment to get the target exposure control parameter EXPO K and target process control parameters PRO K When you can use EXPO K =EXPO K_ini , give priority to PRO K_ini The digital gain dg in K_ini Make adjustments, get PRO K Digital gain dg in KAmong them, the digital gain dg K In the dynamic range extension value dr_rat k and digital gain dg K_ini Take the larger one, that is, dg K =max(dr_rat k ,dg K_ini ).

[0207] In dg K >dg K_ini , the second adjusted statistical brightness value is greater than the first adjusted statistical brightness value. The second adjusted statistical brightness value means: K EXPO K_ini Changes between PRO K With PRO K_ini The changes between the two together act on the statistical brightness value after the first adjustment to obtain the brightness value. In order to ensure that the statistical brightness value after the second adjustment is equal to the statistical brightness value after the first adjustment, the target exposure control parameter EXPO can also be reduced. K In dg K =dg K_ini When the second adjustment is made, the statistical brightness value is equal to the statistical brightness value after the first adjustment, that is, there is no need to adjust EXPO K .

[0208] For example, the target exposure control parameter EXPO K and target process control parameters PRO K The effective time may be delayed. For example, the target exposure control parameter EXPO generated in the kth frame K and target process control parameters PRO K The actual effective time is k+1+d frames, where d represents the number of delayed frames. The number of delayed frames d can be 0 or greater than 0. In the following examples, the number of delayed frames d is 0, indicating no delay.

[0209] For example, when dr_rat k Less than or equal to the preset maximum conversion gain hg max When the first sensor gain sg_l is adjusted first K =sg_l K_ini , the second sensor gain Aperture lr K =lr K_ini , exposure time t K =t K_ini remain unchanged, so that the statistical brightness value after the second adjustment is equal to the statistical brightness value after the first adjustment. k Greater than the preset maximum conversion gain hg maxThe adjustment method can be as follows: Greater than or equal to the preset sensor gain lower limit threshold sg min When the second sensor gain is adjusted to First sensor gain sg_lK=sg_l K_ini , aperture lr k =lr k_ini , exposure time t K =t K_ini remain unchanged, so that the statistical brightness value after the second adjustment is equal to the statistical brightness value after the first adjustment. Less than the preset sensor gain lower limit threshold sg min When the second sensor gain is adjusted to sg_s K =sg min , the first sensor gain sg_l K =sg_l K_ini Keep it the same and then reduce the aperture lr K <lr K_ini , if the aperture lr K is greater than or equal to the preset aperture threshold, and the condition that the statistical brightness value after the second adjustment is equal to the statistical brightness value after the first adjustment is satisfied, t K =t K_ini Otherwise, you need to continue to adjust the exposure time t K <t K_ini , so that the statistical brightness value after the second adjustment is equal to the statistical brightness value after the first adjustment.

[0210] For example, when the dynamic range extension value dr_rat k When it is 1, it means that there is no highlight area in the current scene and no dynamic range expansion is required. Based on this, the candidate exposure control parameter can be directly used as the target exposure control parameter, and the candidate processing control parameter can be used as the target candidate processing control parameter. k When it is greater than 1, it means that there is a highlight area in the current scene and dynamic range expansion is needed to preserve the highlight area information. k Take greater than 1 as an example.

[0211] If the dynamic range extension value dr_rat k If it is greater than 1, the candidate processing control parameter PRO is obtained. K_ini Candidate compensation gain cdg in K_ini and the candidate digital gain dg K_ini , and based on the candidate compensation gain cdg K_ini and the candidate digital gain dg K_ini The product value determines the dynamic range initial value dr_ratb (i.e. the initial value of the dynamic range extension value), for example, dr_rat can be determined by the following formula b :dr_rat b =cdg K_ini *dg K_ini .

[0212] If dr_rat b =dr_rat k , it means that the output image of the gain processing module in the image processing unit can just provide the required dynamic range. Therefore, the candidate exposure control parameters can be directly used as the target exposure control parameters, and the candidate processing control parameters can be used as the target candidate processing control parameters.

[0213] If dr_rat b With dr_rat k If they are not equal, the second exposure control module will adjust the dynamic range according to the dynamic range extension value dr_rat k For candidate exposure control parameters EXPO K_ini and candidate process control parameters PRO K_ini Make a second adjustment to get the target exposure control parameter EXPO K and target process control parameters PRO K The target exposure control parameter enables the second image signal of the image acquisition unit to provide the required dynamic range, and the target processing control parameter enables the image processing unit to adapt to the adjusted image acquisition unit, and the two can be matched and coordinated.

[0214] If dr_rat b With dr_rat k If they are not equal, the second exposure control module makes adjustments in the following manner:

[0215] Case a: If the dynamic range extension value dr_rat k Less than the initial dynamic range value dr_rat b , then based on the dynamic range initial value dr_rat b and the dynamic range extension value dr_rat k The quotient of (i.e. the ratio of the two) is used to adjust the parameters.

[0216] Step S31: Based on the dynamic range initial value dr_rat b and the dynamic range extension value dr_rat k The quotient of determines the first dynamic range adjustment amount. For example, if dr_rat k <dr_rat b , which represents the dynamic range dr_rat provided by the output image of the gain processing module in the image processing unit bIt has exceeded the requirement. In order to make the second image signal just enough to provide dr_rat k times the dynamic range and at the same time improve the signal-to-noise ratio of the second image signal, it is necessary to increase the brightness of the second image signal, that is, to increase the exposure control parameter of the second image signal, the increase ratio is the first dynamic range adjustment amount, and the first dynamic range adjustment amount is Obviously, due to the dynamic range initial value dr_rat b is greater, therefore, the first dynamic range adjustment amount is greater than 1.

[0217] Step S32: Acquire a third set of parameters to be adjusted. The third set of parameters to be adjusted may include but is not limited to at least one parameter among the candidate exposure control parameters and the candidate processing control parameters.

[0218] For example, the parameters to be adjusted may include the candidate aperture, the candidate exposure time, the candidate first conversion gain, the candidate first sensor gain, the candidate second conversion gain, the candidate second sensor gain, the candidate compensation gain and the candidate digital gain in the candidate processing control parameters. Considering that the candidate first conversion gain and the candidate second conversion gain are not adjusted (they may be adjusted in some scenarios, and this embodiment takes no adjustment as an example), the candidate compensation gain is derived based on the candidate conversion gain and the candidate sensor gain, and the candidate digital gain does not affect the actual dynamic range of the image signal output by the image acquisition unit, therefore, the third set of parameters to be adjusted may include the candidate aperture (denoted as lr k ), candidate exposure time (denoted as t k ), candidate first sensor gain (denoted as sg_1 k ) and the candidate second conversion gain (denoted as sg_s k ).

[0219] Since the first image signal and the second image signal use the same exposure time and aperture, there are two ways to adjust the brightness of the second image signal from the perspective of whether the first image signal will be affected. One adjustment method is to adjust the exposure time, aperture, and second sensor gain. The exposure time and aperture will affect the brightness of the first image signal, while the second sensor gain will not affect the brightness of the first image signal (the first sensor gain will affect the brightness of the first image signal, so it is not adjusted). Another adjustment method is not to adjust the exposure time and aperture, but only adjust the brightness of the second image signal through the second sensor gain (the first sensor gain will affect the brightness of the first image signal, so it is not adjusted), which does not affect the brightness of the first image signal. The brightness adjustment of the first image signal and the second image signal is a game and balance process. In this example, it is necessary to adjust the exposure time, aperture, and second sensor gain (that is, the adjustment method that the brightness of the first image signal will be affected is adopted). Therefore, the third set of parameters to be adjusted can include the candidate aperture lr k , candidate exposure time t k , candidate second conversion gain sg_s k .

[0220] Exemplarily, the third set of parameters to be adjusted may include candidate exposure times t k , candidate aperture lr k , candidate second conversion gain sg_s k Of course, this is just an example of the parameter order, and other parameter orders can also be used (such as candidate aperture lr k , candidate second conversion gain sg_s k , candidate exposure time t k , or, the candidate second conversion gain sg_s k , candidate exposure time t k , candidate aperture lr k The order of parameters in the third parameter set to be adjusted is not limited. This parameter order is used as an example in the following. The implementation methods of other parameter orders are similar, and the subsequent process will not be repeated.

[0221] Exemplarily, the parameters in the third set of parameters to be adjusted can be adjusted based on the first dynamic range adjustment amount to obtain adjusted parameters. For example, the first parameter in the third set of parameters to be adjusted is adjusted first. If the first dynamic range adjustment amount is satisfied, the adjustment process is terminated. If the first dynamic range adjustment amount is not satisfied, the second parameter in the third set of parameters to be adjusted is adjusted continuously, and so on, until the first dynamic range adjustment amount is satisfied. On this basis, the target exposure control parameter and the target processing control parameter can be obtained based on the adjusted parameters. The process is described below.

[0222] Step S33: If the first product value of the exposure time and the first dynamic range adjustment amount is less than or equal to the exposure time upper limit threshold, it can be determined that the adjusted exposure time is the first product value.

[0223] If the first product value of the exposure time and the first dynamic range adjustment amount is greater than the exposure time upper limit threshold, the adjusted exposure time is determined to be the exposure time upper limit threshold, and step S34 is executed.

[0224] For example, first increase the exposure time t′ k represents the candidate exposure time, represents the first dynamic range adjustment, t′ k Indicates the adjusted exposure time, that is, the target exposure time.

[0225] After the adjustment, the exposure time t′ k Afterwards, the adjusted exposure time t′ can be k Compare with the exposure time upper limit threshold t_max. If the adjusted exposure time t′ k If it is less than or equal to the upper threshold of exposure time t_max, it indicates that the exposure time adjustment result has met the target dynamic range, and the adjusted exposure time t′ is maintained. k unchanged, that is, the adjusted exposure time t′ k As the target exposure time, there is no need to adjust other parameters in the third parameter set to be adjusted, that is, parameters such as the candidate aperture and the candidate second sensor gain remain unchanged.

[0226] If the exposure time t′ is adjusted k If it is greater than the upper threshold of exposure time t_max, the adjusted exposure time t′ k Update to t_max, t′ k =t max , adjusted exposure time t max is the target exposure time, and step S34 is executed.

[0227] For example, since the first image signal and the second image signal share the same exposure time and aperture, if the exposure time is increased This means that the brightness of the first image signal and the second image signal are increased at the same time. times, and at this time the compensation gain in the synthesis processing module has not been modified, so the output image of the synthesis processing module becomes brighter In order to reduce the brightness of the synthesized image to the original value, it is also necessary to adjust the parameters (such as the digital gain corresponding to the first image signal and / or the first sensor gain corresponding to the first image signal). For example, the digital gain of the first image signal (i.e., the candidate digital gain) may be reduced first, and then the first sensor gain of the first image signal (i.e., the candidate first sensor gain) may be reduced.

[0228] For example, based on the dynamic range extension value dr_rat k With the dynamic range initial value dr_rat b The first reverse adjustment amount is determined by the quotient of The first reverse adjustment amount is less than 1.

[0229] If the first product value of the digital gain corresponding to the first image signal and the first reverse adjustment amount is greater than or equal to the target value (the target value may be the digital gain lower limit threshold dg min , the target value can be 1 or other values, which are not limited), then the adjusted digital gain is determined to be the first product value.

[0230] For example, lowering the digital gain dg k represents the candidate digital gain of the first image signal, dg′ k Indicates the adjusted digital gain. If the adjusted digital gain dg′ k If it is greater than or equal to the target value, the adjusted digital gain dg′ is maintained. k unchanged, that is, the adjusted digital gain dg′ k As the target digital gain of the first image signal (the candidate digital gain of the second image signal is not adjusted). Without adjusting the candidate first sensor gain of the first image signal, dg′ can be executed in the gain processing module of the image processing unit. k times the digital gain, making the brightness of the synthesized image times, which is equivalent to reducing the digital gain of the first image signal. At this time, cdg′ k =cdg k , that is, the target compensation gain is equal to the candidate compensation gain. If the image processing unit has a pre-processing module, a separate processing can also be performed on the first image signal in the pre-processing module. times the digital gain, The digital gain dg′ k =dg k Remain unchanged.

[0231] If the first product value of the digital gain corresponding to the first image signal and the first reverse adjustment amount is less than the target value, a first remaining adjustment amount is determined based on the target value, the digital gain and the first reverse adjustment amount. For example, the first remaining adjustment amount can be The first remaining adjustment amount is less than 1. Indicates the first reverse adjustment, dg k Indicates the digital gain corresponding to the first image signal, dg′ k Indicates the target value.

[0232] If the second product value of the first sensor gain (such as the candidate first sensor gain corresponding to the first image signal) and the first remaining adjustment amount is greater than or equal to the first sensor gain lower limit threshold, the adjusted digital gain is determined to be the target value and the adjusted first sensor gain is the second product value.

[0233] For example, if the digital gain dg′ k Reduce to the target value (such as the digital gain lower limit threshold dg min ), then let the digital gain dg′ k To the target value, reduce the gain of the first sensor sg_l k Represents a candidate first sensor gain of the first image signal, sg_l′ k Represents the adjusted first sensor gain. If the adjusted first sensor gain sg_l′ k Greater than or equal to the first sensor gain lower limit threshold sg_1 min , then keep the adjusted first sensor gain sg_l′ k unchanged, that is, the first sensor gain sg_l′ after adjustment k As the target first sensor gain of the first image signal, and the target value as the target digital gain of the first image signal. In this case, No other parameters need to be adjusted.

[0234] In one possible manner, if the second product value of the first sensor gain (such as the candidate first sensor gain) and the first remaining adjustment amount is less than the first sensor gain lower limit threshold, the second remaining adjustment amount is determined based on the first sensor gain lower limit threshold, the first sensor gain and the first reverse adjustment amount. For example, the second remaining adjustment amount can be The second remaining adjustment amount is less than 1. Represents the first reverse adjustment amount, sg_l k represents the candidate first sensor gain, sg_l minrepresents the lower limit threshold of the first sensor gain. On this basis, the adjusted digital gain can be determined as the candidate digital gain (such as the candidate digital gain dg corresponding to the first image signal). k ) and the product value of the second remaining adjustment amount, and determine the adjusted first sensor gain sg_l′ k is the first sensor gain lower limit threshold sg_1 min In this case, the adjusted digital gain may be smaller than the target value, that is, the digital gain is adjusted until the dynamic range requirement is met.

[0235] In another possible manner, if the second product value of the candidate first sensor gain and the first remaining adjustment amount is less than the first sensor gain lower limit threshold, the second remaining adjustment amount is determined based on the first sensor gain lower limit threshold, the first sensor gain and the first remaining adjustment amount. For example, the second remaining adjustment amount can be Indicates the first remaining adjustment, dg k represents the candidate digital gain corresponding to the first image signal, dg′ k represents the target value (i.e. the adjusted digital gain). On this basis, the adjusted digital gain can be determined as the target value dg′ k The product value of the second residual adjustment amount is used to determine the adjusted first sensor gain sg_l′ k is the first sensor gain lower limit threshold sg_1 min .

[0236] In summary, if the first product value of the exposure time and the first dynamic range adjustment amount is less than or equal to the exposure time upper limit threshold, the target exposure control parameter and the target processing control parameter can be obtained.

[0237] Step S34: based on the exposure time upper limit threshold t_max and the exposure time t k and the first dynamic range adjustment amount Determine a first remaining adjustment amount. For example, the first remaining adjustment amount can be

[0238] If the second product value of the aperture and the first remaining adjustment amount is less than or equal to the aperture upper limit threshold, the adjusted exposure time is determined to be the exposure time upper limit threshold and the adjusted aperture is determined to be the second product value.

[0239] If the second product value of the aperture and the first remaining adjustment amount is greater than the aperture upper limit threshold, the adjusted exposure time is determined to be the exposure time upper limit threshold, the adjusted aperture is determined to be the aperture upper limit threshold, and step S35 is executed.

[0240] For example, you can increase the aperture 1. k represents the candidate aperture, Represents the first remaining adjustment, lr′ k Indicates the adjusted aperture, that is, the target aperture.

[0241] After adjusting the aperture lr′ k After that, you can also adjust the aperture lr′ k Compare with the aperture upper limit threshold lr_max. If the aperture lr′ is adjusted k If it is less than or equal to the aperture upper limit threshold lr_max, it indicates that the aperture adjustment result has met the target dynamic range and the adjusted aperture lr′ can be maintained. k unchanged, that is, the aperture lr′ after adjustment k As the target aperture, there is no need to adjust other parameters in the third parameter set to be adjusted, that is, the parameters such as the candidate second sensor gain remain unchanged. k If it is greater than the aperture upper limit threshold lr_max, the adjusted aperture lr′ k Update to the aperture upper limit threshold lr_max, i.e. lr′ k =lr_max, then, the adjusted aperture lr_max is used as the target aperture, and step S35 is executed.

[0242] For example, since the first image signal and the second image signal share the same exposure time and aperture, if the exposure time and aperture are increased, it means that the brightness of the first image signal and the second image signal are simultaneously increased. In order to reduce the brightness of the synthesized image to the original value, it is also necessary to adjust the parameters (such as the digital gain corresponding to the first image signal and / or the first sensor gain corresponding to the first image signal) downward.

[0243] For example, based on the dynamic range extension value dr_rat k With the dynamic range initial value dr_rat bThe first reverse adjustment amount is determined by the quotient of , and if the first product value of the digital gain corresponding to the first image signal and the first reverse adjustment amount is greater than or equal to the target value, the adjusted digital gain is determined to be the first product value. If the first product value of the digital gain corresponding to the first image signal and the first reverse adjustment amount is less than the target value, the first residual adjustment amount is determined based on the target value, the digital gain and the first reverse adjustment amount. If the second product value of the first sensor gain and the first residual adjustment amount is greater than or equal to the first sensor gain lower limit threshold, the adjusted digital gain is determined to be the target value, and the adjusted first sensor gain is the second product value. If the second product value of the first sensor gain and the first residual adjustment amount is less than the first sensor gain lower limit threshold, the second residual adjustment amount is determined based on the first sensor gain lower limit threshold, the first sensor gain and the first reverse adjustment amount, and the adjusted digital gain is determined to be the product value of the candidate digital gain and the second residual adjustment amount, and the adjusted first sensor gain sg_l′ is determined. k is the first sensor gain lower limit threshold sg_1 min At this point, the target exposure control parameters and target processing control parameters can be successfully obtained.

[0244] Step S35: based on the aperture upper limit threshold lr_max, aperture lr k and the first remaining adjustment Determine the second remaining adjustment amount. For example, the second remaining adjustment amount can be

[0245] If the third product value of the second sensor gain (i.e., the candidate second sensor gain) and the second remaining adjustment amount is less than or equal to the second sensor gain upper limit threshold, the adjusted exposure time is determined to be the exposure time upper limit threshold, the adjusted aperture is the aperture upper limit threshold, and the adjusted second sensor gain is determined to be the third product value.

[0246] For example, increase the gain of the second sensor sg_s k is the candidate second sensor gain, sg_s′ k is the adjusted second sensor gain, that is, the target second sensor gain.

[0247] After adjustment, the second sensor gain sg_s′ k Afterwards, the adjusted second sensor gain sg_s′ can also be k The second sensor gain upper threshold sg_s max If the adjusted second sensor gain sg_s′ k Less than or equal to the second sensor gain upper threshold sg_s max, it indicates that the adjustment result of the second sensor gain has met the target dynamic range, so the adjusted second sensor gain sg_s′ can be maintained k unchanged, that is, the adjusted second sensor gain sg_s′ k As target second sensor gain.

[0248] If the adjusted second sensor gain sg_s′ k Less than or equal to the second sensor gain upper threshold sg_s max , determining a first proportional value based on the quotient of the exposure time and the upper limit threshold of the exposure time, determining a second proportional value based on the quotient of the aperture and the upper limit threshold of the aperture, and determining the adjusted digital gain as the product of the digital gain and the first proportional value and the second proportional value, dg k is the candidate digital gain, dg′ k is the adjusted digital gain, that is, the target digital gain. is the first ratio value, is the second proportional value. If the adjusted second sensor gain is less than or equal to the second sensor gain upper limit threshold, then cdg′ k Indicates the adjusted compensation gain, that is, the target compensation gain, cdg k represents the candidate compensation gain, sg_s k represents the candidate second sensor gain, sg_s′ k Indicates the target second sensor gain.

[0249] If the image processing unit has a pre-processing module, a separate processing can be performed on the first image signal in the pre-processing module. times the digital gain, and

[0250] Exemplarily, if the third product of the second sensor gain and the second remaining adjustment amount is greater than the second sensor gain upper limit threshold, ie, sg_s′ k >sg_s max , then let the adjusted second sensor gain sg_s′ k Equal to the second sensor gain upper threshold sg_s max , and the adjusted exposure time is the upper limit threshold of the exposure time, and the adjusted aperture is the upper limit threshold of the aperture. In this case, there are no parameters that can be adjusted, and the digital gain of the second image signal can only be increased. Since the digital gain does not affect the actual dynamic range of the image signal output by the image acquisition unit, it does not need to be adjusted. It is believed that the dynamic range provided by the second image signal can only exceed dr_rat k , the extra dynamic range is discarded by digital truncation. For example, dr_rat kis 3. After the second image signal s passes through the synthesis processing module and the gain processing module, it is multiplied by 4 in total to reach the target brightness. Therefore, the value of the section from s×3 to s×4 can be truncated to s×3.

[0251] Case b: If the dynamic range extension value dr_rat k Less than the initial dynamic range value dr_rat b , then based on the dynamic range initial value dr_rat b and the dynamic range extension value dr_rat k The quotient of (i.e. the ratio of the two) is used to adjust the parameters.

[0252] Step S41: Based on the dynamic range initial value dr_rat b and the dynamic range extension value dr_rat k The first dynamic range adjustment amount is determined by the quotient of

[0253] Step S42: Acquire a third set of parameters to be adjusted. The third set of parameters to be adjusted may include but is not limited to at least one parameter among the candidate exposure control parameters and the candidate processing control parameters.

[0254] For example, compared with step S32, in this example, only the second sensor gain is adjusted (i.e., an adjustment method is adopted in which the brightness of the first image signal is not affected). Therefore, the third set of parameters to be adjusted may include the candidate second conversion gain sg_s k Based on the first dynamic range adjustment amount, the candidate second conversion gain sg_s in the third parameter set to be adjusted is adjusted. k Adjustments are made to obtain adjusted parameters, and target exposure control parameters and target processing control parameters are obtained based on the adjusted parameters. The process is described below.

[0255] Step S43: If the product value of the second sensor gain and the first dynamic range adjustment amount is less than or equal to the second sensor gain upper limit threshold, determine that the adjusted second sensor gain is the product value.

[0256] If the product value of the second sensor gain and the first dynamic range adjustment amount is greater than the second sensor gain upper limit threshold, the adjusted second sensor gain is determined to be the second sensor gain upper limit threshold.

[0257] For example, increase the gain of the second sensor sg_s k is the candidate second sensor gain, sg_s′ k is the adjusted second sensor gain, that is, the target second sensor gain.

[0258] After adjustment, the second sensor gain sg_s′ k Afterwards, the adjusted second sensor gain sg_s′ can also be k The second sensor gain upper threshold sg_s max If the adjusted second sensor gain sg_s′ k Less than or equal to the second sensor gain upper threshold sg_s max , it indicates that the adjustment result of the second sensor gain has met the target dynamic range, so the adjusted second sensor gain sg_s′ is maintained. k unchanged, that is, the adjusted second sensor gain sg_s′ k As the target second sensor gain. In this case, other parameters do not need to be adjusted further, and the target compensation gain Target digital gain dg′ k =dg k .

[0259] If the adjusted second sensor gain sg_s′ k Greater than the second sensor gain upper threshold sg_s max , then let the adjusted second sensor gain sg_s′ k Equal to the second sensor gain upper threshold sg_s max , that is, the target second sensor gain is the second sensor gain upper limit threshold sg_s max In this case, there is no parameter that can be adjusted, and the only option is to increase the digital gain of the second image signal. Since the digital gain does not affect the actual dynamic range of the image signal output by the image acquisition unit, there is no need to adjust it. It is assumed that the dynamic range provided by the second image signal can only exceed dr_rat k , the extra dynamic range is discarded by digital truncation. k is 3. After the second image signal s passes through the synthesis processing module and the gain processing module, it is multiplied by 4 in total to reach the target brightness. The value of the section from s×3 to s×4 can be truncated to s×3.

[0260] Case c: If the dynamic range extension value dr_rat k Greater than the initial dynamic range value dr_rat b , then based on the dynamic range initial value dr_rat b and the dynamic range extension value dr_rat k The quotient of (i.e. the ratio of the two) is used to adjust the parameters.

[0261] Step S51: Based on the dynamic range initial value dr_rat b and the dynamic range extension value dr_rat kThe quotient of determines the second dynamic range adjustment amount. For example, if dr_rat k >dr_rat b , indicating that the dynamic range provided by the output image of the gain processing module in the image processing unit is insufficient and cannot reach dr_rat k , in order to make the second image signal just enough to provide dr_rat k times the dynamic range, it is necessary to further reduce the parameters of the second image signal to obtain a larger dynamic range. The reduction ratio is the second dynamic range adjustment amount, and the second dynamic range adjustment amount can be And the second dynamic range adjustment amount is less than 1.

[0262] Step S52: Acquire a fourth set of parameters to be adjusted. The fourth set of parameters to be adjusted may include but is not limited to at least one parameter among the candidate exposure control parameters and the candidate processing control parameters.

[0263] For example, the parameters to be adjusted include the candidate aperture, the candidate exposure time, the candidate first conversion gain, the candidate first sensor gain, the candidate second conversion gain, the candidate second sensor gain, the candidate compensation gain and the candidate digital gain in the candidate processing control parameters. Considering that the candidate first conversion gain and the candidate second conversion gain are not adjusted, the candidate compensation gain is derived based on the candidate conversion gain and the candidate sensor gain, the candidate digital gain does not affect the actual dynamic range of the image signal output by the image acquisition unit, and the first sensor gain affects the brightness of the first image signal, therefore, the fourth set of parameters to be adjusted may include the candidate aperture lr k , candidate exposure time t k and the candidate second conversion gain sg_s k .

[0264] Exemplarily, the fourth set of parameters to be adjusted may include the candidate second conversion gains sg_s in sequence. k , candidate aperture lr k , candidate exposure time t k Of course, this is just an example of the parameter order, and other parameter orders can also be used (such as candidate exposure time t k , candidate aperture lr k , candidate second conversion gain sg_s k The order of parameters may be any order), and there is no restriction on the order of parameters in the fourth parameter set to be adjusted. This parameter order is taken as an example later, and the implementation methods of other parameter orders are similar, and the subsequent process will not be repeated.

[0265] Exemplarily, the parameters in the fourth set of parameters to be adjusted can be adjusted based on the second dynamic range adjustment amount to obtain adjusted parameters. For example, the first parameter in the fourth set of parameters to be adjusted is adjusted first. If the second dynamic range adjustment amount is satisfied, the adjustment process is terminated. If the second dynamic range adjustment amount is not satisfied, the second parameter in the fourth set of parameters to be adjusted is adjusted continuously, and so on, until the second dynamic range adjustment amount is satisfied. On this basis, the target exposure control parameter and the target processing control parameter can be obtained based on the adjusted parameters. The process is described below.

[0266] Step S53: If the first product value of the second sensor gain and the first dynamic range adjustment amount is greater than or equal to the second sensor gain lower limit threshold, determine that the adjusted second sensor gain is the first product value. If the first product value of the second sensor gain and the first dynamic range adjustment amount is less than the second sensor gain lower limit threshold, determine that the adjusted second sensor gain is the second sensor gain lower limit threshold, and execute step S54.

[0267] For example, first reduce the gain of the second sensor sg_s k represents the candidate second conversion gain, sg_s′ k Indicates the adjusted second sensor gain, that is, the target second sensor gain.

[0268] After adjustment, the second sensor gain sg_s′ k Afterwards, the adjusted second sensor gain sg_s′ can also be k The second sensor gain lower limit threshold sg_s min If the adjusted second sensor gain sg_s′ k Greater than or equal to the second sensor gain lower limit threshold sg_s min , indicating that the adjustment result of the second sensor gain has met the target dynamic range, and the adjusted second sensor gain sg_s′ is maintained k unchanged, that is, the adjusted second sensor gain sg_s′ k As the target second sensor gain, other parameters do not need to be adjusted further. At this time, the compensation gain Digital gain dg′ k =dg k .

[0269] If the adjusted second sensor gain sg_s′ k Less than the second sensor gain lower limit threshold sg_s min , then the adjusted second sensor gain sg_s′ k Updated to sg_s min , that is, sg_s minis the target second sensor gain.

[0270] Step S54: based on the second sensor gain lower limit threshold sg_s min , the second sensor gain sg_s k and the first dynamic range adjustment amount Determine a first remaining adjustment amount. For example, the first remaining adjustment amount can be The first remaining adjustment amount may be a value less than 1.

[0271] If the second product value of the aperture and the first remaining adjustment amount is greater than or equal to the aperture lower limit threshold, it is determined that the adjusted second sensor gain is the second sensor gain lower limit threshold and the adjusted aperture is the second product value.

[0272] If the second product value is less than the aperture lower limit threshold, it is determined that the adjusted second sensor gain is the second sensor gain lower limit threshold, the adjusted aperture is the aperture lower limit threshold, and step S55 is executed.

[0273] For example, you can lower the aperture 1. k represents the candidate aperture, Represents the first remaining adjustment, lr′ k Indicates the adjusted aperture, that is, the target aperture.

[0274] After adjusting the aperture lr′ k After that, you can also adjust the aperture lr′ k Compare with the aperture lower limit threshold lr_min. If the aperture lr′ is adjusted k If it is greater than or equal to the aperture lower limit threshold lr_min, it indicates that the aperture adjustment result has met the target dynamic range and the adjusted aperture lr′ can be maintained. k unchanged, that is, the aperture lr′ after adjustment k As the target aperture, there is no need to adjust other parameters in the fourth parameter set to be adjusted, that is, the parameters such as the candidate exposure time remain unchanged. Digital Gain That is, it is also possible to calculate the compensation gain cdg based on the candidate k , the second sensor gain sg_s k and the second sensor gain lower limit threshold sg_s min Determine the target compensation gain cdg′ k , and based on the candidate digital gain dg k , dynamic range initial value dr_rat b , dynamic range extension value dr_rat k , the second sensor gain sg_s kand the second sensor gain lower limit threshold sg_s min Determine the target digital gain dg′ k .

[0275] If the rear aperture lr′ is adjusted k If it is less than the aperture lower limit threshold lr_min, the adjusted aperture lr′ k Update to the aperture lower limit threshold lr_min, i.e. lr′ k =lr_min, the adjusted aperture lr_min is used as the target aperture.

[0276] Step S55: based on the aperture lower limit threshold, the aperture and the first remaining adjustment amount Determine the second remaining adjustment amount. For example, the second remaining adjustment amount can be

[0277] If the third product value of the exposure time (i.e., the candidate exposure time) and the second remaining adjustment amount is greater than or equal to the exposure time lower limit threshold, the adjusted second sensor gain is determined to be the second sensor gain lower limit threshold, the adjusted aperture is the aperture lower limit threshold, and the adjusted exposure time is determined to be the third product value.

[0278] For example, you can reduce the exposure time t k represents the candidate exposure time, t′ k Indicates the adjusted exposure time, that is, the target exposure time.

[0279] Get the adjusted exposure time t′ k After that, the exposure time t′ is adjusted k Compare with the exposure time lower limit threshold t_min. If the adjusted exposure time t′ k If it is greater than or equal to t_min, it indicates that the exposure time adjustment result meets the target dynamic range, and the adjusted exposure time t′ is maintained. k unchanged, adjusted exposure time t′ k As the target exposure time. Compensation gain Digital Gain

[0280] If the third product of the exposure time and the second remaining adjustment amount is less than the exposure time lower limit threshold, the adjusted second sensor gain is determined to be the second sensor gain lower limit threshold, the adjusted aperture is the aperture lower limit threshold, and the adjusted exposure time is the exposure time lower limit threshold. For example, if the adjusted exposure time t′ k is less than the lower limit threshold value t_min of the exposure time, it indicates that the second image signal can no longer reduce the brightness, that is, it cannot provide the required dynamic range. At this time, t′ k = t_min, That is, the target digital gain can be determined based on the candidate aperture, the aperture lower limit threshold, the candidate exposure time, the exposure time lower limit threshold and the candidate digital gain. The dynamic range extension value corresponding to the actual parameters in this case can be

[0281] In a possible implementation manner, since the first image signal and the second image signal share the same exposure time and aperture, when the exposure time and / or the aperture are adjusted, the brightness of the first image signal is reduced. times, the brightness of the second image signal is reduced In order to obtain a better signal-to-noise ratio for the first image signal, the brightness of the first image signal may be increased, such as by using the following method:

[0282] Increase the gain of the first sensor to If sg_l′ k Less than or equal to the first sensor gain lower limit threshold sg_1 max , then there is no need to adjust other parameters, so that dg′ k =dg k If sg_l′ k Greater than the first sensor gain lower limit threshold sg_1 max , then sg_l′ k =sg_l max On this basis, Among them, sg_l k represents the candidate first sensor gain, sg_l′ k Indicates the adjusted first sensor gain, that is, the target first sensor gain, dg k represents the candidate digital gain corresponding to the first image signal, dg′ k Indicates the adjusted digital gain, that is, the target digital gain.

[0283] Fifth, cache unit.

[0284] The second exposure control module sets the exposure control parameter EXPO of the Kth frame K (i.e., target exposure control parameter) and the processing control parameter PRO of the Kth frame K The target exposure control parameters (ie, target processing control parameters) are output to the cache unit. The cache unit is used to cache the target exposure control parameters and target processing control parameters output by the second exposure control module.

[0285] Before the Kth frame begins to be exposed, the buffer unit may set the exposure control parameter EXPO of the KNth (N≥1)th frame. K-NThe image acquisition unit controls the exposure of the Kth frame image based on the exposure control parameter. When the image processing unit starts to process the Kth frame image data, the cache unit can process the KNth frame with the control parameter PRO K-N The image data is input to the image processing unit, and the image processing unit processes the Kth frame image data based on the processing control parameter. The processing method is as described in the above embodiment.

[0286] The exposure control parameter EXPO of the cache unit for the Kth frame K and the processing control parameter PRO of the Kth frame K When caching, the exposure control parameters EXPO that have been input to the image acquisition unit can be replaced K-N and the processing control parameters PRO which have been input to the image processing unit K-N , thereby reducing the consumption of storage space resources. The cache unit controls the exposure control parameters EXPO K and process control parameters PRO K After being cached, it is used for the collection and processing of the next frame of image to obtain an image that takes into account both image brightness adjustment and image dynamic range expansion.

[0287] From the above, it can be seen that the input of the cache unit is the exposure control parameters and the processing control parameters. The cache unit uses the cache space to store the exposure control parameters and the processing control parameters, and then takes them out at an appropriate time to achieve synchronization between the exposure control parameters, the processing control parameters and the image acquisition unit and the image processing unit.

[0288] Exemplarily, the cache unit includes a storage module and a synchronization module. Before the Kth frame starts to be exposed, the synchronization module inputs the exposure control parameters of the KNth frame (N≥1, the value of N is related to the computing power of the logic platform and the application requirements) to the image acquisition unit to control the exposure of the Kth frame image. When the image processing unit starts to process the Kth frame image data, the synchronization module inputs the processing control parameters of the KNth frame to the image processing unit, and the image processing unit processes the Kth frame image data based on the processing control parameters.

[0289] The storage module can cache the exposure control parameters and processing control parameters of each frame of the image, and the storage module can realize storage space reuse according to the value of N, thereby reducing resource consumption. For example, when caching the exposure control parameters and processing control parameters of the Kth frame and the Kth frame, the exposure control parameters and processing control parameters of the KNth frame that have been synchronously output can also be replaced.

[0290] It can be seen from the above technical solutions that in the embodiment of the present application, the dual conversion gain technology is used to obtain an image with a higher signal-to-noise ratio in the image acquisition unit, while eliminating motion synthesis defects. The image dynamic range is adaptively calculated based on the exposure statistics of the current scene, and no manual setting is required. Using the dual conversion gain technology, two frames of images with different brightness can be obtained in the image acquisition unit, and the dynamic range that can be expanded is much larger than that of a single frame, and motion synthesis defects can be eliminated. Image processing combines multi-segment digital gain with nonlinear curve mapping to achieve better noise reduction effects. Image data is collected by the image acquisition unit, image processing and statistics are performed, the dynamic range required for expansion is adaptively calculated, and exposure control parameters and processing control parameters are adjusted based on the exposure strategy for the acquisition and processing of the next frame of the image, to obtain a high dynamic range image with stable brightness, retain highlight area information, and increase the amount of image information.

[0291] The image acquisition unit supports dual conversion gain function, which can output two frames of images with different conversion gains within the same exposure time. Compared with the wide dynamic mode, it eliminates motion artifacts and has a higher signal-to-noise ratio than the linear mode. Among them, the two frames of images can be configured with different sensor gains to achieve a higher dynamic range, which further improves the signal-to-noise ratio compared to digital gain. The image acquisition unit can support build-in mode, synthesizing two frames of images with different conversion gains into a frame of high value range image inside the image acquisition unit, and supports nonlinear compression to obtain low value range images, which can save bandwidth resources.

[0292] The image processing unit can synthesize two frames of image data to form image data with a high value range, and then uniformly access the subsequent processing. The overall solution has good compatibility, is simple and unified. The image processing unit can access one frame of image data. The image processing unit supports data non-truncation processing, obtains images with a high value range, compensates for the brightness of dark areas, and does not lose the information of highlight areas, and the value range of the output image is greater than or equal to the value range of the input image. The gain processing process can be executed once or multiple times at multiple different locations in the image processing process, and can be flexibly configured according to the resources or application requirements of the hardware platform to reduce resource consumption. The processing control parameters of the image processing unit are provided by the exposure control unit, and are adaptively adjusted according to the captured scene, so that the images collected according to different exposure control parameters reach a unified target brightness, making the output high dynamic range image more stable. Through data compression processing, it supports mapping images with a high value range to images with a low value range, the brightness of the dark area changes less, and the bright area is compressed with emphasis, while ensuring that the brightness of the dark area is appropriate, while retaining the information of the highlight area. The participation of data compression processing is related to the parameters of gain processing, which supports compressing high-value range image data of different value ranges to low-value range image data of a unified value range, improving the compatibility of the system, and the value range of the output image is less than or equal to the value range of the input image. Data compression processing can be performed once or multiple times at different locations in the image processing process, and can be flexibly configured according to the resources of the hardware platform or application requirements to reduce resource consumption.

[0293] The image processing unit can also perform processing such as interpolation, gamma mapping, contrast enhancement, etc. to improve the image effect. The processing process can be selected according to the resources of the hardware platform or the application requirements.

[0294] The statistics unit can perform brightness statistics processing, count the brightness distribution of the captured scene, assist in more accurate exposure control of the next frame of image, and obtain an image with appropriate brightness. The statistics unit can perform dynamic range statistics processing, count the dynamic range distribution of the captured scene, assist in adjusting the exposure control parameters and processing control parameters of the next frame of image, and obtain an image with appropriate dynamic range. The input images for brightness statistics and dynamic range statistics can be the same or different, and can be flexibly adjusted according to application requirements.

[0295] The exposure control unit can adjust the exposure control parameters and processing control parameters by statistical brightness values, and adjust the exposure control parameters and processing control parameters by dynamic range expansion values, so that the subsequent output images have both stable effects and adaptive high dynamic range. Exposure control parameters include but are not limited to conversion gain, sensor gain, exposure time, and aperture, so as to achieve comprehensive control of the image acquisition unit. Processing control parameters include but are not limited to compensation gain and digital gain, so that the image processing unit and the image acquisition unit work in coordination. When performing dynamic range expansion, it is necessary to reduce the brightness of the captured image. The preferred parameter adjustment order is digital gain, sensor gain, aperture, and exposure time, which can reduce noise amplification and avoid flicker as much as possible. The exposure control parameters include control of all output image signals of the image acquisition unit. Different sensor gains can be configured for different frames to achieve more flexible dynamic range adjustment and higher signal-to-noise ratio.

[0296] The cache unit realizes parameter synchronization between the image acquisition unit and the image processing unit through storage processing and synchronization processing, thereby making the effect of subsequent output images more stable.

[0297] Based on the same application concept as the above method, an image processing method is proposed in the embodiment of the present application and applied to an image acquisition device, see Figure 6 As shown, the image processing method may include:

[0298] Step 601, collecting a first image signal and a second image signal based on an exposure control parameter; wherein the exposure control parameter is determined based on target statistical information corresponding to a preset number of frames of images collected before a current frame; wherein the image sensor supports high conversion gain and low conversion gain, outputs a first image signal when a high conversion gain is used, and outputs a second image signal when a low conversion gain is used.

[0299] Step 602: synthesize the first image signal and the second image signal based on the processing control parameter to obtain a synthesized image, perform gain processing and data compression processing on the synthesized image to obtain an image to be output, and output the image to be output; wherein, when performing the gain processing, no data truncation is performed, or the data is truncate to a certain value range that is not less than the value range of the input image; wherein, the compression parameter used in the data compression processing is related to the gain used in the gain processing.

[0300] Step 603, obtaining target exposure control parameters and target processing control parameters based on target statistical information; wherein the target exposure control parameters are used to control the acquisition of a preset number of image frames after the current frame, and the target processing control parameters are used to control the processing of a preset number of image frames after the current frame.

[0301] Exemplarily, the exposure control parameters include aperture, exposure time, a first conversion gain and a first sensor gain corresponding to the first image signal, and a second conversion gain and a second sensor gain corresponding to the second image signal; the second conversion gain is the same as or different from the first conversion gain, and the second sensor gain is the same as or different from the first sensor gain. Based on this, collecting the first image signal and the second image signal based on the exposure control parameters may include but is not limited to: collecting light signals corresponding to the aperture and the exposure time;

[0302] After obtaining the light signal, the dual conversion gain switch of the image sensor is controlled to be opened, and a first image signal corresponding to the light signal is determined based on the first conversion gain and the first sensor gain; after the first image signal is acquired, the dual conversion gain switch is controlled to be closed, and a second image signal corresponding to the light signal is determined based on the second conversion gain and the second sensor gain; or, after obtaining the light signal, the dual conversion gain switch of the image sensor is controlled to be closed, and the second image signal corresponding to the light signal is determined based on the second conversion gain and the second sensor gain; after the second image signal is acquired, the dual conversion gain switch is controlled to be opened, and the first image signal corresponding to the light signal is determined based on the first conversion gain and the first sensor gain.

[0303] Exemplarily, the processing control parameter may include a compensation gain, and the compensation gain may be a ratio between a first gain value and a second gain value, wherein the first gain value is a product value of a first conversion gain and a first sensor gain, and the second gain value is a product value of a second conversion gain and a second sensor gain. The first image signal and the second image signal are synthesized based on the processing control parameter to obtain a synthesized image, which may include but is not limited to: if the brightness of the first image signal is greater than the brightness of the second image signal, the second image signal is multiplied by the compensation gain to obtain a gain-compensated image; wherein, when the second image signal is multiplied by the compensation gain, data truncation is not performed, and the value range of the gain-compensated image is greater than or equal to the value range of the second image signal; and the first image signal and the gain-compensated image are synthesized to obtain a synthesized image.

[0304] Exemplarily, synthesizing the first image signal and the gain compensation image to obtain a composite image may include: dividing the image area of ​​the first image signal into a first area and a second area, the brightness in the first area is not greater than a composite brightness threshold, and the brightness in the second area is greater than the composite brightness threshold; and obtaining the composite image by using the first image signal in the first area and the gain compensation image in the second area.

[0305] Exemplarily, synthesizing the first image signal and the second image signal based on the processing control parameter to obtain the synthesized image may include, but is not limited to: pre-processing the first image signal and the second image signal respectively; synthesizing the pre-processed first image signal and the pre-processed second image signal based on the processing control parameter to obtain the synthesized image; wherein the pre-processing includes at least one of the following: gain processing, black level processing, noise reduction processing, white balance processing, and deblurring processing;

[0306] Performing gain processing and data compression processing on the synthesized image to obtain an image to be output may include but is not limited to: performing gain processing on the synthesized image to obtain a gain-processed image; performing data compression processing on the gain-processed image to obtain a compressed image; determining the compressed image as the image to be output, or performing post-processing on the compressed image to obtain the image to be output; wherein the post-processing includes at least one of the following: noise reduction processing, enhancement processing, defogging processing, deblurring processing, and sharpening processing;

[0307] Among them, the processing control parameters include digital gain, and the gain processing is to multiply the synthesized image by the digital gain; when the synthesized image is multiplied by the digital gain, no data truncation is performed or the data is truncation is performed to a certain value range that is not less than the value range of the input image, and the value range of the image after gain processing is greater than or equal to the value range of the synthesized image; wherein the data compression processing is to use a preset nonlinear mapping curve to perform brightness mapping on the image after gain processing, and the nonlinear mapping curve is associated with the digital gain.

[0308] Exemplarily, when data compression is performed on an image after gain processing, the compression parameters used in the data compression processing are related to the gains used in the gain processing, including: under different gains, the brightness value range of the image obtained by the gain processing is different, and the compression parameters of the data compression processing are adjusted according to the different brightness value ranges so that the compressed image has the same brightness value range under different gains.

[0309] Exemplarily, performing gain processing and data compression processing on a composite image may include: if the digital gain is greater than a preset gain threshold, splitting the digital gain into a first digital gain value and a second digital gain value, the first digital gain value being equal to the preset gain threshold, and the product of the first digital gain value and the second digital gain value being equal to the digital gain; performing gain processing on the composite image according to the first digital gain value to obtain a first intermediate image; performing data compression processing on the first intermediate image to obtain a second intermediate image; performing intermediate processing on the second intermediate image to obtain a third intermediate image; performing gain processing on the third intermediate image according to the second digital gain value to obtain a gain-processed image; performing data compression processing on the gain-processed image to obtain a compressed image; and determining an image to be output based on the compressed image.

[0310] Exemplarily, when performing data compression processing on the gain-processed image, the gain-processed image is divided into different regions, and a mapping curve corresponding to each region is obtained; for each region of the gain-processed image, mapping processing is performed on the region using the mapping curve corresponding to the region to obtain a compressed image.

[0311] Exemplarily, the target statistical information includes a dynamic range extension value obtained based on a first statistical image, where the first statistical image is an image before synthesis processing, an image after synthesis processing, an image before gain processing, an image after gain processing, an image before data compression processing, or an image after data compression processing. Based on this, brightness statistics of a highlight area of ​​the first statistical image can also be performed to obtain a highlight statistical value; based on the difference between the highlight statistical value and the highlight target value, and a preset dynamic range extension value threshold, the dynamic range extension value is determined; wherein the preset dynamic range extension value threshold is determined based on the computing power or noise reduction capability of the image processing logic platform, so that the noise amplification of the digital processing based on the dynamic range extension value is within a preset range.

[0312] Exemplarily, the target statistical information includes a statistical brightness value obtained based on a second statistical image, and the second statistical image is an image before synthesis processing, an image after synthesis processing, an image before gain processing, an image after gain processing, an image before data compression processing, or an image after data compression processing. Based on this, brightness statistics can also be performed on the second statistical image to obtain a statistical brightness value; wherein the statistical brightness value is determined by: taking the average of the second statistical image to obtain the statistical brightness value; or, setting different weight values ​​for image areas of different brightness in the second statistical image, and performing weighted averaging on the second statistical image to obtain the statistical brightness value; or, setting different weight values ​​for areas at different spatial positions in the second statistical image, and performing weighted averaging on the second statistical image to obtain the statistical brightness value; or, presetting a brightness statistical threshold, taking the smaller of the second statistical image and the brightness statistical threshold, and performing weighted averaging on the second statistical image to obtain the statistical brightness value.

[0313] Exemplarily, the target statistical information includes a dynamic range extension value and a statistical brightness value; the acquisition of the target exposure control parameter and the target processing control parameter based on the target statistical information may include but is not limited to: performing a first parameter adjustment on the historical exposure control parameter and the historical processing control parameter based on the statistical brightness value to obtain a candidate exposure control parameter and a candidate processing control parameter; the first parameter adjustment is determined based on the adjusted statistical brightness value and a preset target brightness; performing a second parameter adjustment on the candidate exposure control parameter and the candidate processing control parameter based on the dynamic range extension value to obtain a target exposure control parameter and a target processing control parameter; the second parameter adjustment is determined based on the adjusted target gain value and the adjusted dynamic range extension value, and the target gain value is the product value between the compensation gain and the digital gain.

[0314] Exemplarily, a first parameter adjustment is performed on historical exposure control parameters and historical processing control parameters based on statistical brightness values ​​to obtain candidate exposure control parameters and candidate processing control parameters, which may include but is not limited to: if the statistical brightness value is less than a preset target brightness, a first brightness adjustment amount is determined based on a quotient of the preset target brightness and the statistical brightness value, and the first brightness adjustment amount is greater than 1; a first set of parameters to be adjusted is obtained, the first set of parameters to be adjusted includes at least one parameter among the historical exposure control parameters and the historical processing control parameters, and the parameters in the first set of parameters to be adjusted are adjusted based on the first brightness adjustment amount to obtain adjusted parameters; and candidate exposure control parameters and candidate processing control parameters are obtained based on the adjusted parameters.

[0315] Exemplarily, the historical exposure control parameters include at least aperture, exposure time, first conversion gain, first sensor gain, second conversion gain and second sensor gain, and the historical processing control parameters include at least compensation gain and digital gain; the multiple parameters in the first parameter set to be adjusted include exposure time, aperture, first sensor gain, second sensor gain and digital gain in sequence; the parameters in the first parameter set to be adjusted are adjusted based on the first brightness adjustment amount to obtain adjusted parameters, which may include but are not limited to: if the first product value of the exposure time and the first brightness adjustment amount is less than or equal to the exposure time upper limit threshold, then the adjusted exposure time is determined to be the first product value; if the first product value is greater than the exposure time upper limit threshold, then the first remaining adjustment amount is determined based on the exposure time upper limit threshold, the exposure time and the first brightness adjustment amount; if the second product value of the aperture and the first remaining adjustment amount is less than or equal to the aperture upper limit threshold, then the adjusted exposure time is determined to be the exposure time upper limit threshold, and the adjusted aperture is determined to be the second product value ; If the second product value is greater than the aperture upper limit threshold, the second remaining adjustment amount is determined based on the aperture upper limit threshold, the aperture and the first remaining adjustment amount; If the third product value of the first sensor gain and the second remaining adjustment amount is less than or equal to the first sensor gain upper limit threshold, the adjusted exposure time is determined to be the exposure time upper limit threshold, the adjusted aperture is the aperture upper limit threshold, the adjusted first sensor gain is the third product value, and the adjusted second sensor gain is the product value of the second sensor gain and the second remaining adjustment amount; If the third product value is greater than the first sensor gain upper limit threshold, the third remaining adjustment amount is determined based on the first sensor gain upper limit threshold, the first sensor gain and the second remaining adjustment amount; The adjusted exposure time is determined to be the exposure time upper limit threshold, the adjusted aperture is the aperture upper limit threshold, the adjusted first sensor gain is the first sensor gain upper limit threshold, the adjusted second sensor gain is the second sensor gain upper limit threshold, and the adjusted digital gain is the product value of the digital gain and the third remaining adjustment amount.

[0316] Exemplarily, a first parameter adjustment is performed on historical exposure control parameters and historical processing control parameters based on statistical brightness values ​​to obtain candidate exposure control parameters and candidate processing control parameters, which may include but is not limited to: if the statistical brightness value is greater than a preset target brightness, a second brightness adjustment amount is determined based on a quotient of the preset target brightness and the statistical brightness value, and the second brightness adjustment amount is less than 1; a second set of parameters to be adjusted is obtained, the second set of parameters to be adjusted includes at least one parameter among the historical exposure control parameters and the historical processing control parameters, and the parameters in the second set of parameters to be adjusted are adjusted based on the second brightness adjustment amount to obtain adjusted parameters; and the candidate exposure control parameters and candidate processing control parameters are obtained based on the adjusted parameters.

[0317] Exemplarily, the historical exposure control parameters include at least aperture, exposure time, first conversion gain, first sensor gain, second conversion gain and second sensor gain, and the historical processing control parameters include at least compensation gain and digital gain; the multiple parameters in the second parameter set to be adjusted include digital gain, first sensor gain, second sensor gain, aperture and exposure time in sequence; the parameters in the second parameter set to be adjusted are adjusted based on the second brightness adjustment amount to obtain adjusted parameters, which may include but are not limited to: if the first product value of the digital gain and the second brightness adjustment amount is greater than or equal to the digital gain lower limit threshold, then the adjusted digital gain is determined to be the first product value; if the first product value is less than the digital gain lower limit threshold, then the first remaining adjustment amount is determined based on the digital gain lower limit threshold, the digital gain and the second brightness adjustment amount; if the second product value of the first sensor gain and the first remaining adjustment amount is greater than or equal to the first sensor gain lower limit threshold, then the adjusted digital gain is determined to be the digital gain lower limit threshold, the adjusted first sensor gain is determined to be the second product value, and the adjusted second sensor gain is determined to be the second sensor gain and a product value between the first and second remaining adjustment amounts; if the second product value is less than the first sensor gain lower limit threshold, the second remaining adjustment amount is determined based on the first sensor gain lower limit threshold, the first sensor gain and the first remaining adjustment amount; if the third product value of the aperture and the second remaining adjustment amount is greater than or equal to the aperture lower limit threshold, the adjusted digital gain is determined to be the digital gain lower limit threshold, the adjusted first sensor gain is the first sensor gain lower limit threshold, the adjusted second sensor gain is the second sensor gain lower limit threshold, and the adjusted aperture is the third product value of the aperture and the second remaining adjustment amount; if the third product value is less than the aperture lower limit threshold, the third remaining adjustment amount is determined based on the aperture lower limit threshold, the aperture and the second remaining adjustment amount; if the fourth product value of the exposure time and the third remaining adjustment amount is greater than or equal to the exposure time lower limit threshold, the adjusted digital gain is determined to be the digital gain lower limit threshold, the adjusted first sensor gain is the first sensor gain lower limit threshold, the adjusted second sensor gain is the second sensor gain lower limit threshold, the adjusted aperture is the aperture lower limit threshold, and the adjusted exposure time is the fourth product value.

[0318] Exemplarily, performing a second parameter adjustment on candidate exposure control parameters and candidate processing control parameters based on a dynamic range extension value to obtain target exposure control parameters and target processing control parameters may include but is not limited to: if the dynamic range extension value is greater than 1, and the candidate processing control parameters include a compensation gain and a digital gain, determining an initial dynamic range value based on a product value of the compensation gain and the digital gain; if the dynamic range extension value is less than the initial dynamic range value, determining a first dynamic range adjustment amount based on a quotient of the initial dynamic range value and the dynamic range extension value; obtaining a third set of parameters to be adjusted, the third set of parameters to be adjusted including at least one parameter among the candidate exposure control parameters and the candidate processing control parameters, and adjusting the parameters in the third set of parameters to be adjusted based on the first dynamic range adjustment amount to obtain adjusted parameters; and obtaining target exposure control parameters and target processing control parameters based on the adjusted parameters.

[0319] Exemplarily, the candidate exposure control parameters may include at least an aperture, an exposure time, a first conversion gain, a first sensor gain, a second conversion gain, and a second sensor gain, and the candidate processing control parameters may include at least a compensation gain and a digital gain; the parameters in the third parameter set to be adjusted may include, in sequence, the exposure time, the aperture, and the second sensor gain; the parameters in the third parameter set to be adjusted are adjusted based on the first dynamic range adjustment amount to obtain adjusted parameters, which may include but are not limited to: if a first product value of the exposure time and the first dynamic range adjustment amount is less than or equal to an upper limit threshold of the exposure time, then determining that the adjusted exposure time is the first product value; if the first product value is greater than the ... A first remaining adjustment amount is determined based on an upper threshold value of exposure time, the exposure time and a first dynamic range adjustment amount; if a second product value of the aperture and the first remaining adjustment amount is less than or equal to the upper threshold value of the aperture, the adjusted exposure time is determined to be the upper threshold value of the exposure time, and the adjusted aperture is determined to be the second product value; if the second product value is greater than the upper threshold value of the aperture, the second remaining adjustment amount is determined based on the upper threshold value of the aperture, the aperture and the first remaining adjustment amount; if a third product value of the second sensor gain and the second remaining adjustment amount is less than or equal to the upper threshold value of the second sensor gain, the adjusted exposure time is determined to be the upper threshold value of the exposure time, the adjusted aperture is the upper threshold value of the aperture, and the adjusted second sensor gain is determined to be the third product value.

[0320] Exemplarily, if the first product value is less than or equal to the upper threshold of the exposure time, or the first product value is greater than the upper threshold of the exposure time and the second product value is less than or equal to the upper threshold of the aperture, the first reverse adjustment amount can be determined based on the quotient of the dynamic range extension value and the initial value of the dynamic range; if the first product value of the digital gain corresponding to the first image signal and the first reverse adjustment amount is greater than or equal to the target value, the adjusted digital gain is determined to be the first product value; if the first product value is less than the target value, the first remaining adjustment amount is determined based on the target value, the digital gain and the first reverse adjustment amount; if the second product value of the first sensor gain and the first remaining adjustment amount is greater than or equal to the first sensor gain lower threshold, the adjusted digital gain is determined to be the target value and the adjusted first sensor gain is the second product value; if the second product value is less than the first sensor gain lower threshold, the second remaining adjustment amount is determined based on the first sensor gain lower threshold, the first sensor gain and the first reverse adjustment amount; the adjusted digital gain is determined to be the product value of the digital gain and the second remaining adjustment amount, and the adjusted first sensor gain is determined to be the first sensor gain lower threshold.

[0321] If the first product value is greater than the exposure time upper limit threshold, the second product value is greater than the aperture upper limit threshold, and the third product value is less than or equal to the second sensor gain upper limit threshold, the first proportional value can be determined based on the quotient of the exposure time and the exposure time upper limit threshold, the second proportional value can be determined based on the quotient of the aperture and the aperture upper limit threshold, and the adjusted digital gain is determined to be the product value of the digital gain and the first proportional value and the second proportional value.

[0322] Exemplarily, the candidate exposure control parameters include at least aperture, exposure time, first conversion gain, first sensor gain, second conversion gain and second sensor gain, and the candidate processing control parameters include at least compensation gain and digital gain; the parameters in the third parameter set to be adjusted include the second sensor gain; the parameters in the third parameter set to be adjusted are adjusted based on the first dynamic range adjustment amount to obtain adjusted parameters, which may include but are not limited to: if the product value of the second sensor gain and the first dynamic range adjustment amount is less than or equal to the second sensor gain upper limit threshold, then the adjusted second sensor gain is determined to be the product value; if the product value of the second sensor gain and the first dynamic range adjustment amount is greater than the second sensor gain upper limit threshold, then the adjusted second sensor gain is determined to be the second sensor gain upper limit threshold.

[0323] Exemplarily, performing a second parameter adjustment on candidate exposure control parameters and candidate processing control parameters based on the dynamic range extension value to obtain target exposure control parameters and target processing control parameters may include but is not limited to: if the dynamic range extension value is greater than 1, and the candidate processing control parameters include a compensation gain and a digital gain, determining an initial dynamic range value based on a product value of the compensation gain and the digital gain; if the dynamic range extension value is greater than the initial dynamic range value, determining a second dynamic range adjustment amount based on a quotient of the initial dynamic range value and the dynamic range extension value; obtaining a fourth set of parameters to be adjusted, the fourth set of parameters to be adjusted including at least one parameter among the candidate exposure control parameters and the candidate processing control parameters, and adjusting the parameters in the fourth set of parameters to be adjusted based on the second dynamic range adjustment amount to obtain adjusted parameters; and obtaining target exposure control parameters and target processing control parameters based on the adjusted parameters.

[0324] Exemplarily, the candidate exposure control parameters include at least aperture, exposure time, first conversion gain, first sensor gain, second conversion gain and second sensor gain, and the candidate processing control parameters include at least compensation gain and digital gain; the parameters in the fourth parameter set to be adjusted include the second sensor gain, aperture and exposure time in sequence; the parameters in the fourth parameter set to be adjusted are adjusted based on the second dynamic range adjustment amount to obtain adjusted parameters, including: if a first product value of the second sensor gain and the first dynamic range adjustment amount is greater than or equal to the second sensor gain lower limit threshold, determining that the adjusted second sensor gain is the first product value; if the first product value is less than the second sensor gain lower limit threshold, determining a first residual adjustment value based on the second sensor gain lower limit threshold, the second sensor gain and the first dynamic range adjustment amount. an integral amount; if the second product value of the aperture and the first remaining adjustment amount is greater than or equal to the aperture lower limit threshold, it is determined that the adjusted second sensor gain is the second sensor gain lower limit threshold, and the adjusted aperture is the second product value; if the second product value is less than the aperture lower limit threshold, the second remaining adjustment amount is determined based on the aperture lower limit threshold, the aperture and the first remaining adjustment amount; if the third product value of the exposure time and the second remaining adjustment amount is greater than or equal to the exposure time lower limit threshold, it is determined that the adjusted second sensor gain is the second sensor gain lower limit threshold, the adjusted aperture is the aperture lower limit threshold, and the adjusted exposure time is the third product value; if the third product value is less than the exposure time lower limit threshold, it is determined that the adjusted second sensor gain is the second sensor gain lower limit threshold, the adjusted aperture is the aperture lower limit threshold, and the adjusted exposure time is the exposure time lower limit threshold.

[0325] Based on the same application concept as the above method, an electronic device (i.e., an image acquisition device) is proposed in the embodiment of the present application, see Figure 7As shown, the electronic device includes a processor 71 and a machine-readable storage medium 72, wherein the machine-readable storage medium 72 stores machine-executable instructions that can be executed by the processor 71; the processor 71 is used to execute the machine-executable instructions to implement the image processing method of the above example of the present application.

[0326] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the image processing method disclosed in the above example of the present application can be implemented.

[0327] The above-mentioned machine-readable storage medium may be any electronic, magnetic, optical or other physical storage device, which may contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium may be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as CD, DVD, etc.), or similar storage medium, or a combination thereof.

[0328] The systems, devices, modules or units described in the above embodiments may be implemented by a computer entity or by a product having a certain function. A typical implementation device is a computer, and the specific form of the computer may be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver device, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0329] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0330] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0331] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0332] Moreover, these computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0333] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0334] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: The first image signal and the second image signal are collected based on an exposure control parameter; wherein the exposure control parameter is determined based on target statistical information corresponding to a preset number of frames of images collected before the current frame; wherein the image sensor supports a high conversion gain and a low conversion gain, outputs the first image signal when the high conversion gain is adopted, and outputs the second image signal when the low conversion gain is adopted; The first image signal and the second image signal are synthesized based on the processing control parameter to obtain a synthesized image, the synthesized image is subjected to gain processing and data compression processing to obtain an image to be output, and the image to be output is output; wherein, when performing the gain processing, no data truncation is performed, or the data is truncation to a certain value range that is not less than the value range of the input image; wherein, the compression parameter used in the data compression processing is related to the gain used in the gain processing; Target exposure control parameters and target processing control parameters are obtained based on target statistical information; wherein the target exposure control parameters are used to control the acquisition of a preset number of image frames after the current frame, and the target processing control parameters are used to control the processing of a preset number of image frames after the current frame.

2. The method according to claim 1, characterized in that The exposure control parameters include aperture, exposure time, a first conversion gain and a first sensor gain corresponding to the first image signal, and a second conversion gain and a second sensor gain corresponding to the second image signal; the second conversion gain is the same as or different from the first conversion gain, and the second sensor gain is the same as or different from the first sensor gain; The collecting of the first image signal and the second image signal based on the exposure control parameter comprises: collecting a light signal corresponding to the aperture and the exposure time; After obtaining the light signal, controlling the dual conversion gain switch of the image sensor to be opened, and determining a first image signal corresponding to the light signal based on the first conversion gain and the first sensor gain; after the first image signal is acquired, controlling the dual conversion gain switch to be closed, and determining a second image signal corresponding to the light signal based on the second conversion gain and the second sensor gain; Alternatively, after obtaining the light signal, the dual conversion gain switch of the image sensor is controlled to be closed, and a second image signal corresponding to the light signal is determined based on the second conversion gain and the second sensor gain; after the second image signal is acquired, the dual conversion gain switch is controlled to be opened, and a first image signal corresponding to the light signal is determined based on the first conversion gain and the first sensor gain.

3. The method according to claim 2, characterized in that The processing control parameter includes a compensation gain, the compensation gain is a ratio between a first gain value and a second gain value, the first gain value is a product value of a first conversion gain and a first sensor gain, and the second gain value is a product value of a second conversion gain and a second sensor gain; The synthesizing the first image signal and the second image signal based on the processing control parameter to obtain a synthesized image includes: if the brightness of the first image signal is greater than the brightness of the second image signal, multiplying the second image signal by the compensation gain to obtain a gain-compensated image; wherein no data truncation is performed when the second image signal is multiplied by the compensation gain, and a value range of the gain-compensated image is greater than or equal to a value range of the second image signal; The first image signal and the gain compensated image are synthesized to obtain the synthesized image.

4. The method according to claim 3, characterized in that The synthesizing the first image signal and the gain compensated image to obtain the synthesized image includes: dividing the image area of ​​the first image signal into a first area and a second area, wherein the brightness in the first area is not greater than a composite brightness threshold, and the brightness in the second area is greater than the composite brightness threshold; The synthesized image is obtained by using the first image signal in the first area and using the gain compensated image in the second area.

5. The method according to claim 1 or 3, characterized in that: The synthesizing the first image signal and the second image signal based on the processing control parameter to obtain a synthesized image includes: Pre-processing the first image signal and the second image signal respectively; The pre-processed first image signal and the pre-processed second image signal are synthesized based on the processing control parameter to obtain a synthesized image; wherein the pre-processing includes at least one of the following: gain processing, black level processing, noise reduction processing, white balance processing, and deblurring processing; Performing gain processing and data compression processing on the synthesized image to obtain an image to be output, including: Performing gain processing on the composite image to obtain a gain-processed image; performing data compression processing on the gain-processed image to obtain a compressed image; determining the compressed image as the image to be output, or performing post-processing on the compressed image to obtain the image to be output; wherein the post-processing includes at least one of the following: noise reduction processing, enhancement processing, defogging processing, deblurring processing, and sharpening processing; Wherein, the processing control parameter includes a digital gain, and the gain processing is to multiply the synthesized image by the digital gain; when the synthesized image is multiplied by the digital gain, data truncation is not performed or is truncated to a certain value range that is not less than the value range of the input image, and the value range of the image after the gain processing is greater than or equal to the value range of the synthesized image; The data compression process is to use a preset nonlinear mapping curve to perform brightness mapping on the gain-processed image, and the nonlinear mapping curve is associated with the digital gain.

6. The method according to claim 1 or 5, characterized in that: When data compression is performed on the gain-processed image, the compression parameters used in the data compression processing are related to the gains used in the gain processing, including: under different gains, the brightness value range of the image obtained by the gain processing is different, and the compression parameters of the data compression processing are adjusted according to the different brightness value ranges, so that the compressed image has the same brightness value range under different gains.

7. The method according to claim 1 or 5, characterized in that: The step of performing gain processing and data compression processing on the synthesized image comprises: If the digital gain is greater than a preset gain threshold, splitting the digital gain into a first digital gain value and a second digital gain value, wherein the first digital gain value is equal to the preset gain threshold, and the product of the first digital gain value and the second digital gain value is equal to the digital gain; Performing gain processing on the composite image according to the first digital gain value to obtain a first intermediate image; Performing data compression processing on the first intermediate image to obtain a second intermediate image; performing intermediate processing on the second intermediate image to obtain a third intermediate image; Performing gain processing on the third intermediate image according to the second digital gain value to obtain a gain-processed image; Performing data compression processing on the gain-processed image to obtain a compressed image; The image to be output is determined based on the compressed image.

8. The method according to claim 1 or 5, characterized in that: When data compression is performed on the gain-processed image, the gain-processed image is divided into different regions, and a mapping curve corresponding to each region is obtained; for each region of the gain-processed image, mapping processing is performed on the region using the mapping curve corresponding to the region to obtain a compressed image.

9. The method according to claim 1, characterized in that: The target statistical information includes a dynamic range extension value obtained based on a first statistical image, wherein the first statistical image is an image before synthesis processing, an image after synthesis processing, an image before gain processing, an image after gain processing, an image before data compression processing, or an image after data compression processing, and the method further includes: Performing brightness statistics on the highlight area of ​​the first statistical image to obtain a highlight statistical value; The dynamic range extension value is determined based on the difference between the highlight statistical value and the highlight target value, and a preset dynamic range extension value threshold; wherein the preset dynamic range extension value threshold is determined based on the computing power or noise reduction capability of the image processing logic platform, so that the noise amplification of the digital processing based on the dynamic range extension value is within a preset range.

10. The method according to claim 1, characterized in that The target statistical information includes a statistical brightness value obtained based on a second statistical image, wherein the second statistical image is an image before synthesis processing, an image after synthesis processing, an image before gain processing, an image after gain processing, an image before data compression processing, or an image after data compression processing, and the method further includes: Performing brightness statistics on the second statistical image to obtain the statistical brightness value; The statistical brightness value is determined as follows: Calculating the average value of the second statistical image to obtain the statistical brightness value; or, Setting different weight values ​​for image regions of different brightness of the second statistical image, and performing weighted averaging on the second statistical image to obtain the statistical brightness value; or, Setting different weight values ​​for regions at different spatial positions of the second statistical image, and performing weighted averaging on the second statistical image to obtain the statistical brightness value; or, A brightness statistical threshold is preset, the smaller of the second statistical image and the brightness statistical threshold is taken, and the second statistical image is weighted averaged to obtain the statistical brightness value.

11. The method according to claim 1, characterized in that: The target statistical information includes a dynamic range extension value and a statistical brightness value; and obtaining a target exposure control parameter and a target processing control parameter based on the target statistical information includes: Performing a first parameter adjustment on the historical exposure control parameter and the historical processing control parameter based on the statistical brightness value to obtain a candidate exposure control parameter and a candidate processing control parameter; wherein the first parameter adjustment is determined based on the adjusted statistical brightness value and a preset target brightness; A second parameter adjustment is performed on the candidate exposure control parameter and the candidate processing control parameter based on the dynamic range extension value to obtain a target exposure control parameter and a target processing control parameter; wherein the second parameter adjustment is determined based on an adjusted target gain value and an adjusted dynamic range extension value, and the target gain value is a product value between a compensation gain and a digital gain.

12. The method according to claim 11, characterized in that The performing a second parameter adjustment on the candidate exposure control parameter and the candidate processing control parameter based on the dynamic range extension value to obtain a target exposure control parameter and a target processing control parameter includes: If the dynamic range extension value is greater than 1, and the candidate processing control parameters include a compensation gain and a digital gain, determining a dynamic range initial value based on a product value of the compensation gain and the digital gain; If the dynamic range extension value is less than the dynamic range initial value, determining a first dynamic range adjustment amount based on a quotient of the dynamic range initial value and the dynamic range extension value; Acquire a third set of parameters to be adjusted, the third set of parameters to be adjusted including at least one parameter among the candidate exposure control parameters and the candidate processing control parameters, and adjust the parameters in the third set of parameters to be adjusted based on the first dynamic range adjustment amount to obtain adjusted parameters; A target exposure control parameter and a target processing control parameter are obtained based on the adjusted parameters.

13. The method according to claim 12, characterized in that The candidate exposure control parameters include at least an aperture, an exposure time, a first conversion gain, a first sensor gain, a second conversion gain, and a second sensor gain, and the candidate processing control parameters include at least a compensation gain and a digital gain; The parameters in the third set of parameters to be adjusted include the exposure time, the aperture, and the second sensor gain in sequence; and adjusting the parameters in the third set of parameters to be adjusted based on the first dynamic range adjustment amount to obtain adjusted parameters includes: If a first product value of the exposure time and the first dynamic range adjustment amount is less than or equal to an upper limit threshold of the exposure time, determining that the adjusted exposure time is the first product value; If the first product value is greater than the exposure time upper limit threshold, determining a first remaining adjustment amount based on the exposure time upper limit threshold, the exposure time and the first dynamic range adjustment amount; If a second product value of the aperture and the first remaining adjustment amount is less than or equal to an aperture upper limit threshold, determining that the adjusted exposure time is the exposure time upper limit threshold and the adjusted aperture is the second product value; If the second product value is greater than the aperture upper limit threshold, determining a second remaining adjustment amount based on the aperture upper limit threshold, the aperture and the first remaining adjustment amount; If the third product value of the second sensor gain and the second remaining adjustment amount is less than or equal to the second sensor gain upper limit threshold, the adjusted exposure time is determined to be the exposure time upper limit threshold, the adjusted aperture is the aperture upper limit threshold, and the adjusted second sensor gain is determined to be the third product value.

14. The method according to claim 13, characterized in that If the first product value is less than or equal to an upper limit threshold of exposure time, or the first product value is greater than the upper limit threshold of exposure time and the second product value is less than or equal to an upper limit threshold of aperture, the method further includes: determining a first reverse adjustment amount based on a quotient of the dynamic range extension value and the dynamic range initial value; If a first product value of the digital gain corresponding to the first image signal and the first reverse adjustment amount is greater than or equal to a target value, determining that the adjusted digital gain is the first product value; If the first product value is less than the target value, a first remaining adjustment amount is determined based on the target value, the digital gain and the first reverse adjustment amount; if a second product value of the first sensor gain and the first remaining adjustment amount is greater than or equal to a first sensor gain lower limit threshold, the adjusted digital gain is determined to be the target value and the adjusted first sensor gain is determined to be the second product value; If the second product value is less than the first sensor gain lower limit threshold, a second remaining adjustment amount is determined based on the first sensor gain lower limit threshold, the first sensor gain and the first reverse adjustment amount; the adjusted digital gain is determined to be the product value of the digital gain and the second remaining adjustment amount, and the adjusted first sensor gain is determined to be the first sensor gain lower limit threshold.

15. The method according to claim 13, characterized in that If the first product value is greater than the exposure time upper limit threshold, and the second product value is greater than the aperture upper limit threshold, and the third product value is less than or equal to the second sensor gain upper limit threshold, the method further includes: A first proportional value is determined based on the quotient of the exposure time and the upper limit threshold of the exposure time, a second proportional value is determined based on the quotient of the aperture and the upper limit threshold of the aperture, and an adjusted digital gain is determined as a product value of the digital gain and the first proportional value and the second proportional value.

16. The method according to claim 12, characterized in that The candidate exposure control parameters include at least an aperture, an exposure time, a first conversion gain, a first sensor gain, a second conversion gain, and a second sensor gain; the candidate processing control parameters include at least a compensation gain and a digital gain; the parameters in the third set of parameters to be adjusted include the second sensor gain; and the parameters in the third set of parameters to be adjusted based on the first dynamic range adjustment amount are adjusted to obtain adjusted parameters, including: If the product value of the second sensor gain and the first dynamic range adjustment amount is less than or equal to the second sensor gain upper limit threshold, determining the adjusted second sensor gain to be the product value; If the product value of the second sensor gain and the first dynamic range adjustment amount is greater than the second sensor gain upper limit threshold, the adjusted second sensor gain is determined to be the second sensor gain upper limit threshold.

17. The method according to claim 11, characterized in that The performing a second parameter adjustment on the candidate exposure control parameter and the candidate processing control parameter based on the dynamic range extension value to obtain a target exposure control parameter and a target processing control parameter includes: If the dynamic range extension value is greater than 1, and the candidate processing control parameters include a compensation gain and a digital gain, determining a dynamic range initial value based on a product value of the compensation gain and the digital gain; If the dynamic range extension value is greater than the dynamic range initial value, determining a second dynamic range adjustment amount based on a quotient of the dynamic range initial value and the dynamic range extension value; Acquire a fourth set of parameters to be adjusted, the fourth set of parameters to be adjusted including at least one parameter among the candidate exposure control parameters and the candidate processing control parameters, and adjust the parameters in the fourth set of parameters to be adjusted based on the second dynamic range adjustment amount to obtain adjusted parameters; A target exposure control parameter and a target processing control parameter are obtained based on the adjusted parameters.

18. The method according to claim 17, characterized in that The candidate exposure control parameters include at least an aperture, an exposure time, a first conversion gain, a first sensor gain, a second conversion gain, and a second sensor gain, and the candidate processing control parameters include at least a compensation gain and a digital gain; The parameters in the fourth set of parameters to be adjusted include the second sensor gain, the aperture, and the exposure time in sequence; The adjusting the parameters in the fourth set of parameters to be adjusted based on the second dynamic range adjustment amount to obtain adjusted parameters includes: If a first product value of the second sensor gain and the first dynamic range adjustment amount is greater than or equal to a second sensor gain lower limit threshold, determining that the adjusted second sensor gain is the first product value; If the first product value is less than a second sensor gain lower limit threshold, determining a first remaining adjustment amount based on the second sensor gain lower limit threshold, the second sensor gain and the first dynamic range adjustment amount; If the second product value of the aperture and the first remaining adjustment amount is greater than or equal to the aperture lower limit threshold, determining that the adjusted second sensor gain is the second sensor gain lower limit threshold and the adjusted aperture is the second product value; If the second product value is less than the aperture lower limit threshold, determining a second remaining adjustment amount based on the aperture lower limit threshold, the aperture and the first remaining adjustment amount; If the third product value of the exposure time and the second remaining adjustment amount is greater than or equal to the lower limit threshold of the exposure time, the adjusted second sensor gain is determined to be the second sensor gain lower limit threshold, the adjusted aperture is the aperture lower limit threshold, and the adjusted exposure time is the third product value; if the third product value is less than the lower limit threshold of the exposure time, the adjusted second sensor gain is determined to be the second sensor gain lower limit threshold, the adjusted aperture is the aperture lower limit threshold, and the adjusted exposure time is the exposure time lower limit threshold.

19. An imaging device, characterized in that: The imaging device comprises: An image acquisition unit, configured to acquire a first image signal and a second image signal based on an exposure control parameter; wherein the exposure control parameter is determined based on target statistical information corresponding to a preset number of frames of images acquired before a current frame; the image acquisition unit supports a high conversion gain and a low conversion gain, outputs a first image signal when a high conversion gain is adopted, and outputs a second image signal when a low conversion gain is adopted; outputting the first image signal and the second image signal to an image processing unit; an image processing unit, configured to synthesize the first image signal and the second image signal based on a processing control parameter to obtain a synthesized image, perform gain processing and data compression processing on the synthesized image to obtain an image to be output, and output the image to be output; wherein, when performing the gain processing, no data truncation is performed, or the data is truncation to a certain value range that is not less than the value range of the input image; wherein, the compression parameter used in the data compression processing is related to the gain used in the gain processing; A statistical unit, used to obtain target statistical information; An exposure control unit, used for acquiring a target exposure control parameter and a target processing control parameter based on the target statistical information; wherein the target exposure control parameter is input to the image acquisition unit to control the acquisition of a preset number of image frames after the current frame; and the target processing control parameter is input to the image processing unit to control the processing of a preset number of image frames after the current frame.

20. The device according to claim 19, characterized in that The image acquisition unit is configured to output the first image signal and the second image signal to an image processing unit if the scene feature of the imaging device satisfies a constraint condition; The image acquisition unit is further configured to synthesize the first image signal and the second image signal based on a processing control parameter to obtain a third image signal if the scene feature does not satisfy a constraint condition, perform nonlinear compression on the third image signal, and output the compressed image signal to the image processing unit, wherein the image processing unit performs gain processing and data compression processing based on the compressed image signal; Among them, the scene feature includes the available bandwidth between the image acquisition unit and the image processing unit. If the available bandwidth is greater than a preset bandwidth threshold, the scene feature meets the constraint condition; if the available bandwidth is less than or equal to the preset bandwidth threshold, the scene feature does not meet the constraint condition.

21. An electronic device, characterized in that: The electronic device comprises: a memory, a processor, and an image processing program stored in the memory and executable on the processor, wherein the image processing program implements the image processing method according to any one of claims 1 to 18 when executed by the processor.

22. A storage medium, characterized in that: An image processing program is stored on the storage medium, and when the image processing program is executed by the processor, the image processing method according to any one of claims 1 to 18 is implemented.

Images