Image processing system, image processing method, image signal processor, storage medium, and program product

Abstract

The invention relates to an image processing system and method, an image signal processor, a storage medium and a program product. The image processing system includes a sensor and an image signal processor. The image signal processor is configured to: calculate a sensor parameter and an image signal processor parameter to be applied in a subsequent frame based on image data of a current frame every time a predetermined number of frames has elapsed; the sensor parameter is written to the sensor, and the image signal processor parameter is written to the image signal processor in a frame in which the written sensor parameter takes effect in the sensor. And processing the image data by applying the written image signal processor parameters, and outputting the processed image data.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image processing system, method, image signal processor, storage medium and program product. Background Art

[0002] In the image processing system, the image sensor (Image Sensor) and the image signal processor (Image Signal Processor, ISP) work closely together. As the source of image data, the performance and adjustment method of the image sensor have an important impact on the effect of the entire system, and the ISP processing flow plays a key role in improving image quality. At present, in the ISP processing flow, the adjustment of the image sensor is completed through the configuration of the register. The current image sensor register writing and response cannot be fully synchronized, that is, the register parameter modification cannot take effect immediately, and the ISP parameter takes effect in real time after being written to the ISP, which leads to the problem of the image sensor and the ISP writing parameter taking effect time being out of sync. On the other hand, the inability to fully synchronize the image sensor register writing and response affects the sensor's response speed, which in turn affects the convergence speed of the ISP image processing process. Therefore, the image data stream output by the ISP has a certain delay in the presentation of real-time scene changes. Summary of the invention

[0003] The present application provides an image processing system, method, image signal processor, storage medium and program product.

[0004] One aspect of the present application relates to an image processing system, comprising: a sensor, configured to collect multiple frames of image data, wherein sensor parameters are effective after at least one frame has passed after being written into the sensor; and an image signal processor, wherein image signal processor parameters are effective immediately in the written frame after being written into the image signal processor, wherein the image signal processor comprises: an acquisition unit, configured to acquire each frame of image data collected by the sensor; a calculation unit, configured to calculate sensor parameters and image signal processor parameters to be applied in subsequent frames based on image data of a current frame after a predetermined number of frames have passed; a writing unit, configured to write sensor parameters calculated based on image data of a current frame into the sensor, and write the image signal processor parameters into the image signal processor in the frame in which the written sensor parameters are effective in the sensor; and a processing unit, configured to process the image data using the written image signal processor parameters, and output the processed image data.

[0005] Another aspect of the present application relates to an image processing system, comprising: a sensor, configured to collect multiple frames of image data, wherein sensor parameters are effective after at least one frame has passed after being written into the sensor; and an image signal processor, wherein image signal processor parameters are effective immediately in the written frame after being written into the image signal processor, wherein the image signal processor comprises: an acquisition unit, configured to acquire each frame of image data collected by the sensor; a calculation unit, configured to calculate sensor parameters and image signal processor parameters to be applied in subsequent frames based on image data of a current frame after a predetermined number of frames have passed; an image signal processor compensation value is calculated based on the sensor parameters calculated based on the image data of the current frame and the sensor parameters calculated in the previous time based on image data of a previous frame; a writing unit, configured to write the sensor parameters calculated based on the image data of the current frame into the sensor, and write the image signal processor parameters into the image signal processor after combining the image signal processor compensation value in the frame before the written sensor parameters take effect in the sensor; a processing unit, configured to process the image data using the written image signal processor parameters combined with the image signal processor compensation value, and output the processed image data.

[0006] In some embodiments, the calculation unit is further configured to calculate the image signal processor compensation value as a change between a sensor parameter calculated based on image data of a current frame and a sensor parameter previously calculated based on image data of a previous frame.

[0007] In some implementations, the calculation unit is further configured to calculate the image signal processor compensation value by combining the sensor parameters calculated based on the image data of the current frame and the sensor parameters calculated in the previous time based on the image data of the previous frame; and the writing unit is further configured to write the image signal processor after multiplying the image signal processor parameter and the image signal processor compensation value.

[0008] In some embodiments, the writing unit is further configured to: determine whether the sensor parameters calculated based on the image data of the current frame are the same as the sensor parameters calculated based on the image data of the previous frame in the previous time; if the judgment result is not the same, write the sensor parameters calculated based on the image data of the current frame into the sensor, and in the frame before the written sensor parameters take effect in the sensor, write the image signal processor parameters into the image signal processor after combining them with the image signal processor compensation value.

[0009] In some embodiments, the writing unit is further configured to: write the sensor parameters calculated based on the image data of the current frame into the sensor a predetermined number of frames after the current frame, and write the image signal processor parameters combined with the image signal processor compensation value into the image signal processor.

[0010] In some embodiments, the image processing system also includes a memory, which is configured to temporarily store multiple frames of image data collected by the sensor; wherein the predetermined number is greater than or equal to 2; the sensor parameters become effective after x frames after being written to the sensor, and x is a positive integer less than the predetermined number; the acquisition unit is configured to: acquire each frame of image data collected by the sensor from the memory; and the writing unit is configured to, in the frame where the written sensor parameters are effective in the sensor, rewrite the image signal processor parameters into the image signal processor without combining the image signal processor compensation value.

[0011] In some embodiments, the acquisition unit is configured to: directly acquire from the sensor each frame of image data collected by the sensor; and the predetermined number is 1.

[0012] In some embodiments, the image processing system includes a plurality of the sensors; the image signal processor arranges a processing timing for processing the image data collected by the plurality of sensors according to a calculation cycle for calculating the image data collected by each sensor.

[0013] Another aspect of the present application relates to an image processing method for processing a stream including multiple frames of image data, including: acquiring each frame of image data collected by a sensor; calculating sensor parameters and image signal processor parameters to be applied in subsequent frames based on the image data of the current frame after every predetermined number of frames; writing the sensor parameters calculated based on the image data of the current frame into the sensor, and writing the image signal processor parameters into the image signal processor in the frame in which the written sensor parameters are effective in the sensor; applying the written image signal processor parameters to process the image data, and outputting the processed image data.

[0014] Another aspect of the present application relates to an image processing method for processing a stream including multiple frames of image data, including: acquiring each frame of image data collected by a sensor; calculating sensor parameters and image signal processor parameters to be applied in subsequent frames based on the image data of the current frame after every predetermined number of frames; calculating image signal processor compensation values ​​based on the sensor parameters calculated based on the image data of the current frame and the sensor parameters calculated in the previous time based on the image data of the previous frame; writing the sensor parameters calculated based on the image data of the current frame into the sensor, and writing the image signal processor parameters into the image signal processor after combining them with the image signal processor compensation values ​​in a frame before the written sensor parameters take effect in the sensor; applying the written image signal processor parameters combined with the image signal processor compensation values ​​to process the image data, and outputting the processed image data.

[0015] Another aspect of the present application relates to an image signal processor, comprising a logic circuit capable of executing a computer program stored in a memory, wherein the logic circuit implements the above-mentioned image processing method when executing the computer program.

[0016] Another aspect of the present application relates to a non-transitory computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned image processing method when executed by a processor.

[0017] Another aspect of the present application relates to a computer program product, comprising a computer program, which implements the above-mentioned image processing method when executed by a processor.

[0018] According to the image processing system, method, image signal processor, storage medium and program product of the present application, the sensor parameters are effective after several frames after being written into the sensor, while the ISP parameters are effective immediately in the written frame after being written into the corresponding module of the ISP. On the one hand, by writing the ISP parameters into the ISP in the frame in which the written sensor parameters are effective in the sensor, the image signal processor parameters and the ISP parameters can be synchronized and effective, thereby improving the image processing quality. On the other hand, by utilizing the real-time response characteristics of the ISP to the written parameters, the ISP compensation value corresponding to the parameter change in the sensor before and after the written sensor parameters are effective in the sensor is written into the corresponding module of the ISP, and the part of the sensor delayed effective is compensated in real time in the image stream output by the ISP. When the image parameters change, the processing delay of the sensor hardware is compensated by adding a compensation mechanism for the delay in writing the register of the sensor to the software algorithm of the ISP, and the effective frame of the parameters in the image stream output by the ISP is at least 1 frame earlier than the effective frame in the image stream collected by the sensor. As a result, the processed image stream output by the ISP can reflect the changes of the real-time scene more quickly. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A block diagram of an image processing system according to an embodiment of the present application is shown;

[0020] Figure 2 is a schematic diagram of the operation of the image processing system according to the first embodiment of the present application;

[0021] Figure 3 A block diagram of an image processing system according to an embodiment of the present application is shown;

[0022] Figure 4 is another operation schematic diagram of the image processing system according to the first embodiment of the present application;

[0023] Figure 5 is a schematic diagram of the operation of the image processing system according to the second embodiment of the present application;

[0024] Figure 6 is another operation schematic diagram of the image processing system according to the second embodiment of the present application;

[0025] Figure 7 is a flowchart of an image processing method according to an embodiment of the present application;

[0026] Figure 8 is a flowchart of an image processing method according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] In the image processing system, the image sensor works closely with the image signal processor (ISP). Especially in the vehicle environment, the ISP faces more special and stringent requirements. With the development of vehicle technology, autonomous driving and assisted driving functions are increasingly dependent on accurate and real-time image data processing. Since vehicles often travel at higher speeds and the surrounding environment changes rapidly, this requires that the data stream based on the ISP output can quickly respond to changes in real-time scenes. Since the register write and response of the image sensor are not synchronized, the strict real-time requirements in the vehicle environment cannot be met. This may pose a potential threat to the safe implementation of autonomous driving and assisted driving functions, because inaccurate or delayed image data may cause the system to misjudge the surrounding environment, thereby affecting the accuracy and safety of driving decisions. Based on this, the embodiment of the present application proposes an image processing system that can reflect the changes in real-time scenes in real time, and can perform low-latency and fast-convergence processing on large-scale data collected by the image sensor.

[0029] One aspect of the present application provides an image processing system 100 . Figure 1 FIG. 1 is a block diagram of an image processing system 100 according to an embodiment of the present application. In this embodiment, the image processing system 100 includes a sensor 10 and an ISP 20 .

[0030] The sensor 10 may be an image sensor configured to collect multiple frames of image data, where the image data may be understood as raw image data collected by the sensor 10. For example, the sensor 10 may continuously collect multiple frames of image data to form an image data stream. Exemplarily, the image processing system 100 further includes an optical lens, the sensor 10 is connected to the optical lens, the optical lens focuses light onto the sensor 10, and the sensor 10 converts the optical signal into an electrical signal.

[0031] The adjustment of the sensor 10 is completed through the configuration of the register. As a storage unit, the register can store various parameters related to the sensor 10, including but not limited to exposure time, gain, white balance, resolution, etc. Correspondingly, the register of the sensor 10 includes registers for controlling various functions, such as an exposure register for controlling automatic exposure, an analog gain register, a white balance gain register required for white balance, etc. By configuring the values ​​in the register, the working state of the image sensor can be accurately controlled to meet different application requirements. For example, adjusting the value of the exposure time register can change the sensitivity of the image sensor to light, thereby affecting the brightness of the image.

[0032] The configuration of the registers can be adjusted in real time while the image processing system 100 is running, which enables the sensor 10 to adapt to different lighting conditions, scene changes, and user needs. For example, in automatic exposure control (AEC), the image processing system 100 can adjust sensor parameters such as exposure time and gain in real time according to the brightness information of the image by modifying the values ​​in the corresponding registers. In other words, the sensor parameters are converted into the values ​​of the corresponding registers after calculation and written into the corresponding registers, thereby writing the sensor parameters into the sensor 10.

[0033] According to the characteristics of the sensor 10, the value of the register takes effect after at least one frame has passed after the value is written into the register. That is, the writing and response of the register of the sensor 10 cannot be completely synchronized, and the modification of the register cannot take effect immediately. In other words, the sensor parameters take effect after at least one frame has passed after being written into the sensor 10. Sensors 10 of different manufacturers and models exhibit different characteristics in this regard. For example, the exposure and gain effectiveness mechanism of the sensors 10 of manufacturers such as Ominivision, On semiconductor and Sony is that the sensor parameters take effect after two frames after being written into the sensor 10 (the sensor parameters are written in the Nth frame, and the written sensor parameters take effect in the N+2th frame), and the white balance gain effectiveness mechanism of the sensor 10 of the Ominivision manufacturer is that the sensor parameters take effect after one frame after being written into the sensor 10 (the sensor parameters are written in the Nth frame, and the written sensor parameters take effect in the N+1th frame). In practical applications, this may cause delays in image processing and fail to adapt to changes in the scene in a timely manner, thereby affecting the quality of the image output by the image processing system 100 and the performance of the image processing system 100.

[0034] The sensor 10 in the present application is a common image sensor in the technical field of image processing. Those skilled in the art can understand its specific configuration, which will not be described in detail here.

[0035] The image signal processor (ISP) 20 is a device for processing the output data of the image sensor, such as automatic exposure control (AEC), automatic gain control (AGC), automatic white balance (AWB), color correction and other functions. The ISP is a device including a central processing unit (such as an MCU for calculating parameters), a data communication control bus, a data transmission interface, general peripherals, a signal adjustment circuit, an image signal processing logic circuit and other components. In the present application, the image signal processor may also be referred to as an image signal processing device, which can be understood as a system on chip (SOC).

[0036] The ISP 20 is connected to the sensor through an I / O interface, for example, and is configured to obtain each frame of image data (electrical signal) collected by the sensor 10. In the case where the sensor 10 continuously collects multiple frames of image data streams, the sensor 10 transmits each frame of image data collected in real time to the ISP 20 via the I / O interface. The ISP 20 processes the electrical signal obtained by the sensor 10 to obtain a visible image, and the ISP 20 outputs the processed image data to obtain an image stream, such as a video. The image stream output by the ISP 20 can be displayed by a display or stored by a memory.

[0037] The functions of ISP 20 are implemented by the ISP logic circuit and the firmware running on it. In addition to completing the algorithm processing of the firmware, the ISP logic circuit can also count the real-time information of the current image. The firmware obtains the image statistical information of the ISP logic circuit. The ISP 20 implements the functions of each processing module through the firmware running on it, forming an ISP PIPE (processing function pipeline), and then realizes functions such as automatic exposure control (AEC), automatic gain control (AGC), and automatic white balance (AWB), and performs corresponding feedback control on the ISP logic circuit, thereby performing corresponding feedback control on the optical lens and sensor 10, so as to achieve the purpose of automatically adjusting the image quality. The firmware is a software algorithm program running on the ISP logic circuit, which can be stored in the internal memory of the ISP (such as non-volatile memory such as flash memory, EPROM, etc.), or in a storage device outside the ISP.

[0038] The architecture of the ISP 20 in the present application is a common ISP architecture in the technical field of image processing. Those skilled in the art can understand its specific structure, which will not be described in detail here.

[0039] ISP 20 needs to write the corresponding parameters into ISP 20 to enable it to perform the corresponding functions. For example, the automatic exposure control (AEC) algorithm in the firmware will automatically calculate the appropriate exposure parameters based on the brightness information of the image, and the automatic gain control (AGC) algorithm in the firmware will automatically calculate the gain value based on the strength of the input signal. These automatically calculated ISP parameters will be written into ISP 20 so that ISP 20 can adjust the processing of the image. The way to write ISP parameters into ISP 20 is, for example, to write into the corresponding registers in ISP20. After the ISP parameters are written into ISP 20, they take effect immediately in the written frame. For example, ISP 20 writes the ISP parameters in the Nth frame (in this application, writing in the Nth frame can be understood as writing after the N-1th frame and before the start of the Nth frame, or writing at the start of the Nth frame), and the written ISP parameters take effect in the Nth frame, and ISP 20 processes the image data of the Nth frame with the written ISP parameters.

[0040] In the present application, ISP 20 may include an acquisition unit 220, a calculation unit 240, a writing unit 260 and a processing unit 280, and ISP 20 implements the functions of the acquisition unit 220, the calculation unit 240, the writing unit 260 and the processing unit 280 through software algorithms running thereon.

[0041] Figure 2 is a schematic diagram of the operation of the image processing system according to the first embodiment of the present application, and the following reference is made to Figure 2 The operations performed by the image processing system according to the first embodiment of the present application are described in detail. The operations performed by the ISP 20 can be implemented by software algorithms running thereon.

[0042] The acquisition unit 220 of the ISP 20 may be configured to acquire each frame of image data collected by the sensor 10 .

[0043] Depend on Figure 1 As can be seen from the above description, the sensor 10 is configured to collect multiple frames of image data. Figure 2 The acquisition unit 220 of the ISP 20 is configured to acquire each frame of image data acquired by the sensor 10, and process the acquired image data to generate processed image data, which is processed in Figure 2 Each frame in the ISP image data stream is obtained after the corresponding frame in the sensor image data stream is processed by the ISP 20 using ISP parameters.

[0044] The calculation unit 240 of the ISP 20 may be configured to calculate sensor parameters and ISP parameters to be applied in subsequent frames based on image data of a current frame every time a predetermined number of frames pass.

[0045] The sensor parameters and ISP parameters have initial values, which are used to apply when processing the first frame of image. The sensor parameters to be applied in subsequent frames are parameters applied when the sensor 10 collects image data of subsequent frames, such as exposure time, gain, etc. The ISP parameters to be applied in subsequent frames are parameters applied when the ISP 20 processes images of subsequent frames, which include parameters of each functional module of the ISP 20, such as exposure parameters, gain values, etc., and parameters for overall control of each functional module of the ISP 20. It is easy to understand that the sensor parameters and ISP parameters include parameters of corresponding functions, for example, the sensor parameters include exposure time parameters, and the ISP parameters correspondingly include exposure parameters.

[0046] Exemplarily, the predetermined number of frames may be one frame, in which case ISP 20 calculates the sensor parameters and ISP parameters for each frame; the predetermined number of frames may be two frames, in which case ISP 20 performs calculations every two frames to calculate the sensor parameters and ISP parameters.

[0047] Specifically, when calculating the sensor parameters and ISP parameters, the ISP logic circuit in the ISP 20 performs parameter statistics on the image data of the current frame, and the functional module of the ISP 20 calculates the sensor parameters and ISP parameters to be applied in the subsequent frames based on the statistical parameters. The sensor parameters calculated each time are cached in a specific storage part of the ISP 20. It is easy to understand that the sensor parameters calculated each time can also be cached in a storage part located outside the ISP 20.

[0048] The writing unit 260 of the ISP 20 may be configured to write the sensor parameters calculated based on the image data of the current frame into the sensor 10 .

[0049] Specifically, the ISP converts the calculated sensor parameters into the values ​​of the registers corresponding to the sensor 10, and writes the values ​​of the registers into the corresponding registers to write the calculated sensor parameters into the sensor 10. It is easy to understand that the sensor parameters are written into the frame after the current frame, for example, the sensor parameters are written into the sensor 10 1 frame, 2 frames or more frames after the current frame.

[0050] The sensor parameters written into the sensor 10 are used for the sensor 10 to collect image data of subsequent frames. However, as mentioned above, according to the characteristics of the sensor 10, the sensor parameters written into the sensor 10 cannot take effect immediately. The sensor parameters written into the Fn frame usually take effect in Fn+1 or Fn+2 or even subsequent frames to adjust the image data collection effect of the sensor 10.

[0051] The writing unit 260 of the ISP 20 may also be configured to write the ISP parameters into the ISP 20 after combining them with the ISP compensation values ​​in a frame in which the written sensor parameters are effective in the sensor 10 .

[0052] For example, when the sensor parameters written in the Fn frame take effect in the Fn+1 frame, the ISP parameters can be written into the ISP 20 after combining with the ISP compensation values ​​in the Fn+1 frame. The ISP parameters can be written into the ISP 20 by addressing the corresponding registers in the ISP 20, which correspond to the functional modules in the ISP 20, such as the AEC and AGC modules, so as to play a role in these functional modules, so that the ISP 20 processes the image. It should be understood that the functions implemented by writing the sensor parameters into the sensor 20 correspond to the functions implemented by writing the ISP parameters into the ISP 20. It is easy to understand that the frames in the sensor 10 correspond to the frames in the ISP 20, and the frames with the same timing in the sensor 10 and the ISP 20 indicate the same frame.

[0053] The processing unit 280 of the ISP 20 may be configured to process the image data using the written ISP parameters and output the processed image data.

[0054] exist Figure 2 In the example shown, at the Nth frame (e.g., F0' frame), the ISP 20 calculates the sensor parameter S1 and the ISP parameter I1 based on the image data of the F0' frame. At the N+1th frame (e.g., F1 frame), the calculated sensor parameter S1 is written to the sensor 10, and the written sensor parameter S1 takes effect in the sensor 10 after 1 frame. The calculated ISP parameter I1 is written to the ISP 20 after the frame in which the written sensor parameter takes effect in the sensor 10 (i.e., the N+2th frame (e.g., F2' frame)). The ISP 20 processes the image data using the ISP parameter I1 and outputs the processed image data, i.e., outputs the processed image data of the F2' frame. As a result, the written sensor parameter S1 and the ISP parameter I1 are both effective at the N+2th frame (e.g., F2 (or F2') frame), that is, the image sensor parameter and the ISP parameter take effect synchronously. In this example, a calculation can be performed once per frame to calculate the sensor parameter and the ISP parameter, and the effective period of the sensor parameter and the ISP parameter is 2 frames.

[0055] According to the above embodiment, by writing ISP parameters into the ISP in the frame in which the written sensor parameters take effect in the sensor, the image sensor parameters and the ISP parameters can be synchronously effective, thereby improving the image processing quality.

[0056] In some embodiments, Figure 3 As shown, the image processing system 100 further includes a memory 30 , which is configured to temporarily store multiple frames of image data collected by the sensor 10 . Figure 3 ISP 20 and Figure 1The ISP 20 in can have the same architecture. It is easy for those skilled in the art to understand that the way of connecting the image data collected by the sensor 10 to the ISP 20 includes an online (Inline) mode and an offline (Offline) mode. For the Inline mode, the ISP 20 directly obtains the image data collected by the sensor 10 from the sensor 10. Specifically, the image data flows out of the sensor 10 and immediately enters the ISP 20 to be processed by the ISPPIPE. This mode has the characteristic of low latency. For the Offline mode, the image data to be processed is stored in the memory 30 (for example, a DDR memory) in units of frames. After being temporarily stored in the memory 30, the image data is read from the memory 30 by the ISP 20 so that the image data is processed by the ISP PIPE. The image processing system according to the present application is applicable to the Inline mode and the Offline mode.

[0057] For Inline mode, apply to Figure 2 The operation flow shown; for the Offline mode, considering that it takes a certain amount of time for image data to be transmitted through the memory, the parameter calculation cycle is slightly different from the writing cycle.

[0058] Figure 4 Another operation schematic diagram of the image processing system according to the first embodiment of the present application is shown, and the operation adopts the Offline mode. Figure 4 In the example, the acquisition unit 220 of ISP 20 acquires each frame of image data collected by the sensor 10 from the memory 30. At the Nth frame (for example, the F0' frame), the ISP 20 calculates the sensor parameter S1 and the ISP parameter I1 based on the image data of the F0' frame. At the N+2th frame (for example, the F2 frame), the calculated sensor parameter S1 is written to the sensor 10, and the written sensor parameter S1 takes effect in the sensor 10 after 1 frame. The calculated ISP parameter I1 is written to ISP 20 after the frame in which the written sensor parameter takes effect in the sensor 10 (that is, the N+3th frame (for example, the F3' frame)). ISP 20 processes the image data using the ISP parameter I1 and outputs the processed image data, that is, outputs the processed image data of the F3' frame. As a result, the written sensor parameter S1 and ISP parameter I1 are both effective at the N+3th frame (for example, the F3 (or F3') frame), that is, the image sensor parameters and ISP parameters are synchronously effective. In this example, calculation may be performed every 3 frames to calculate sensor parameters and ISP parameters, and the ISP parameters may be written every 3 frames.

[0059] The present application also provides an image processing system 100. The image processing system according to the second embodiment has a similar configuration to the image processing system according to the first embodiment. Except that the configurations of the computing unit, the writing unit, and the processing unit are different from those of the image processing system of the first embodiment, the rest of the parts are the same as those of the image processing system of the first embodiment. The description of the same parts is omitted, and only the differences from the image processing system of the first embodiment are described below.

[0060] Figure 5 is a schematic diagram of the operation of the image processing system according to the second embodiment of the present application, and the following reference is made to Figure 5 The operations performed by the image processing system according to the embodiment of the present application are described in detail.

[0061] The calculation unit 240 of the ISP 20 may also be configured such that the ISP 20 calculates the ISP compensation value based on the sensor parameters calculated based on the image data of the current frame and the sensor parameters previously calculated based on the image data of the previous frame.

[0062] Specifically, for example, when the predetermined number of frames is one frame, the previous calculation is a calculation based on the image data of the previous frame of the current frame; when the predetermined number of frames is two frames, the previous calculation is a calculation based on the image data of the previous two frames of the current frame. It is easy to understand that the previous frame refers to the frame before the current frame, and is not limited to the previous frame of the current frame. That is, the ISP compensation value is calculated based on the sensor parameters currently calculated and the sensor parameters calculated last time. Since the sensor parameters calculated each time are cached in a specific storage part of the ISP 20, the sensor parameters calculated last time can be obtained from the specific storage part to calculate the ISP compensation value. When the current calculation is the first calculation, the initial value of the sensor parameter is used as the sensor parameter calculated last time. The initial value of the sensor parameter is the default value assigned to the sensor 10 by the image processing system 100 when it is initialized. In this case, the writing unit 260 of the ISP 20 can also be configured to write the ISP parameters into the ISP 20 after combining the ISP compensation value in the frame before the written sensor parameters take effect in the sensor 10.

[0063] For example, when the sensor parameters written in the Fn frame take effect in the Fn+2 frame, the ISP parameters can be written into the ISP 20 after being combined with the ISP compensation value in the frame before the Fn+2 frame (for example, the Fn frame or the Fn+1 frame). The ISP compensation value can be a multiple (coefficient) of scaling the previously calculated ISP parameters. The ISP compensation value is used to compensate the parameters in the sensor 10 before the written sensor parameters take effect in the sensor 10 to the parameters in the sensor 10 after the written sensor parameters take effect in the sensor 10 in the ISP image data stream.

[0064] Advantageously, in the same frame in the timing of writing the sensor parameters to the sensor 10, the ISP 20 writes the ISP parameters in combination with the ISP compensation value into the ISP 20. The same frame in the timing refers to a frame in the ISP image data stream that corresponds to the sensor image data stream in the timing. Writing the ISP parameters in combination with the ISP compensation value into the ISP 20 may be writing the ISP parameters into the ISP 20 after multiplying the ISP parameters by the ISP compensation value. It is easy to understand that the ISP parameters written into the ISP 20 at this time are the parameters compensated by the ISP compensation value. The way of writing the parameters into the ISP 20 may be writing into the corresponding registers in the ISP 20 by addressing, which correspond to the functional modules in the ISP 20, such as the AEC and AGC modules, so as to play a role in these functional modules. It should be understood that the functions implemented by writing the sensor parameters into the sensor 20 correspond to the functions implemented by writing the ISP parameters into the ISP 20. Writing the ISP parameters in combination with the ISP compensation value into the ISP 20 and writing the sensor parameters into the sensor 10 are performed in the same frame in the timing, in other words, the writing of the ISP parameters and the writing of the sensor parameters are performed roughly synchronously.

[0065] In this case, the processing unit 280 of the ISP 20 may be configured to process the image data using the written ISP parameters combined with the ISP compensation value and output the processed image data.

[0066] As described above, according to the characteristics of the ISP 20, the ISP parameters written into the ISP 20 take effect immediately in the written frame after being written into the ISP 20. That is, the image data is processed by applying the parameters in the frame where the ISP parameters are written, so that the changes of the parameters are reflected in real time in the frame where the ISP parameters are written, thereby reflecting the changes of the real-time scene captured by the sensor 10.

[0067] For example, in Figure 5In the example shown, the ISP 20 calculates the sensor parameters and the ISP parameters in each frame, that is, the predetermined number of frames is 1 frame. At the F0' frame, the ISP 20 calculates the sensor parameters S1 and the ISP parameters I1 based on the image data of the F0' frame. Since the calculation based on the image data of the F0' frame is the first calculation, the initial value of the sensor parameter is used as the sensor parameter calculated last time. Then, the ISP compensation value C1 is calculated based on the sensor parameter S1 calculated based on the image data of the F0' frame and the initial value of the sensor parameter. In the F1 frame, the calculated sensor parameter S1 is written to the sensor 10, and the calculated ISP parameter I1 is written to the ISP 20 after being combined with the calculated ISP compensation value C1 in the same frame in time sequence (that is, the F1' frame). The ISP 20 processes the image data using the ISP parameter I1 combined with the ISP compensation value C1, and outputs the processed image data, that is, outputs the processed image data of the F1' frame. In the sensor image data stream, the sensor parameter S1 calculated based on the image data of the F0' frame can only take effect in the F2 frame; while in the ISP image data stream, the parameters calculated based on the image data of the F0' frame have been applied to process the image data in the F1' frame and present the processed image data. Therefore, the effective frame of the relevant parameters in the ISP image data stream is one frame earlier than that of the sensor image data stream.

[0068] At the F1' frame, the ISP 20 calculates the sensor parameters S2 and the ISP parameters I2 based on the image data of the F1' frame. The calculation based on the image data of the F0' frame is not the first calculation. The sensor parameters calculated last time are the sensor parameters S1 calculated based on the image data of the F0' frame. The sensor parameters S1 are cached in a specific storage part of the ISP 20, and the ISP 20 can directly obtain the sensor parameters S1 calculated last time from the specific storage part. Then, the ISP compensation value C1 is calculated based on the sensor parameters S2 calculated based on the image data of the F1' frame and the sensor parameters S1 calculated based on the image data of the F0' frame. At the F2 frame, the calculated sensor parameters S1 are written to the sensor 10, and the calculated ISP parameters I2 are written to the ISP 20 after being combined with the calculated ISP compensation value C2 at the same frame in the time sequence (i.e., the F2' frame). The ISP 20 processes the image data using the ISP parameters I2 combined with the ISP compensation value C2, and outputs the processed image data, i.e., outputs the processed image data of the F2' frame. In the sensor image data stream, the sensor parameter S2 calculated based on the image data of the F1' frame can only take effect in the F3 frame; while in the ISP image data stream, the parameters calculated based on the image data of the F1' frame have been applied to process the image data in the F2' frame and present the processed image data. Therefore, the effective frame of the relevant parameters in the ISP image data stream is one frame earlier than that of the sensor image data stream.

[0069] Figure 5 The processing for subsequent frames is similar to the above process, which is not repeated in the figure and will not be described here.

[0070] According to the above technical solution, the sensor parameters take effect after several frames after being written into the sensor, while the ISP parameters take effect immediately in the written frames when they are written into the corresponding functional modules of ISP 20. By utilizing the real-time response characteristics of ISP 20 to the written parameters, the ISP compensation values ​​corresponding to the parameter changes in the sensor before and after the written sensor parameters take effect in the sensor are written into the corresponding functional modules of ISP 20, and the delayed effective part of the sensor is compensated in real time in the image stream output by ISP 20. When the image parameters change, the processing delay of the sensor hardware is compensated by adding a compensation mechanism for the delay in writing the registers of the sensor to the software algorithm of ISP 20, and the effective frame of the parameters in the image stream output by ISP 20 is at least 1 frame earlier than the effective frame in the image stream collected by the sensor. As a result, the processed image stream output by ISP 20 can reflect the changes in the real-time scene more quickly.

[0071] In some embodiments, the calculation unit 240 of the ISP 20 is further configured to calculate the image signal processor compensation value as a change between the sensor parameters calculated based on the image data of the current frame and the sensor parameters previously calculated based on the image data of the previous frame.

[0072] By writing the ISP compensation value corresponding to the parameter change of the sensor before and after the written sensor parameter takes effect in the sensor into the ISP corresponding module, the part of the sensor delayed effect is compensated in real time in the image stream output by the ISP.

[0073] The change between the sensor parameters calculated based on the image data of the current frame (hereinafter referred to as the currently calculated sensor parameters) and the sensor parameters previously calculated based on the image data of the previous frame (hereinafter referred to as the previously calculated sensor parameters) may be the difference between the currently calculated sensor parameters and the previously calculated sensor parameters, or may be the ratio between the currently calculated sensor parameters and the previously calculated sensor parameters.

[0074] Exemplarily, the calculation unit 240 may be configured to calculate the ISP compensation value based on the ratio of the currently calculated sensor parameters to the previously calculated sensor parameters; in this case, the writing unit 260 may be configured to write the ISP 20 after multiplying the ISP parameters by the ISP compensation value. In other words, the ISP compensation value is based on the ratio of the currently calculated sensor parameters to the previously calculated sensor parameters; the ISP parameters are combined with the ISP compensation value in the form of multiplying the ISP parameters by the ISP compensation value. For example, for the calculation of parameters of automatic exposure control (AEC) and automatic gain control (AGC), the ratio between the currently calculated sensor parameters and the previously calculated sensor parameters may be calculated as the ISP compensation value.

[0075] See again Figure 5 At the F1' frame, the ISP compensation value C2 can be determined as the ratio (S2:S1) of the sensor parameter S2 calculated based on the image data of the F1' frame and the sensor parameter S1 calculated based on the image data of the F0' frame. In this case, the ISP compensation value can be a multiple (coefficient) of the previously calculated ISP parameter scaled, that is, the ISP parameter is combined with the ISP compensation value in the manner of multiplying the ISP parameter with the ISP compensation value, and the parameter written to the ISP 20 at the F2' frame should be I2*C2=I1*(S2:S1). In this way, the ISP compensation value corresponds to the parameter change in the sensor before and after the written sensor parameter takes effect in the sensor.

[0076] For example, see Figure 5 , the initial values ​​of the sensor parameters include the exposure time parameter E0 and the gain parameter G0; the sensor parameter S1 calculated in the F0' frame includes the exposure time parameter E1 and the gain parameter G1; the sensor parameter S2 calculated in the F1' frame includes the exposure time parameter E2 and the gain parameter G2. In this case, the ISP compensation value C1 calculated in the F0' frame = (E1*G1) / (E0*G0); the ISP compensation value C2 calculated in the F1' frame = (E2*G2) / (E1*G1). It is easy to know that E0*G0*C1 = E1*G1, that is, the effect of applying the effective sensor parameters to acquire the image in the F2 frame is equal to the effect of the ISP 20 outputting the processed image in the F1' frame; E1*G1*C2 = E2*G2, that is, the effect of applying the effective sensor parameters to acquire the image in the F3 frame is equal to the effect of the ISP 20 outputting the processed image in the F2' frame. Therefore, the effective frame of the relevant parameters in the ISP image data stream is one frame earlier than the sensor image data stream.

[0077] In other examples, when the change between the currently calculated sensor parameter and the previously calculated sensor parameter is the difference between the currently calculated sensor parameter and the previously calculated sensor parameter, the writing unit 260 may be configured to add the ISP parameter and the ISP compensation value and then write them to the ISP 20. For example, in the case where the parameter is converted to be operated in the log-domain, the difference between the currently calculated sensor parameter and the previously calculated sensor parameter may be calculated as the ISP compensation value.

[0078] In some embodiments, the ISP 20 writing unit 260 can be configured to determine whether the sensor parameters calculated based on the image data of the current frame are the same as the sensor parameters calculated based on the image data of the previous frame; if the judgment result is not the same, the sensor parameters calculated based on the image data of the current frame are written into the sensor 10, and in the frame before the written sensor parameters take effect in the sensor 10, the ISP parameters are combined with the ISP compensation values ​​and written into ISP20.

[0079] Specifically, see again Figure 5 , at frame F1', the sensor parameter calculated based on the image data of the current frame is sensor parameter S2, and the sensor parameter calculated in the previous time based on the image data of the previous frame is sensor parameter S1. As can be seen from the foregoing description, the sensor parameter S1 calculated in the previous time is cached in a specific storage part of ISP 20, and ISP 20 can directly obtain the sensor parameter S1 calculated in the previous time from the specific storage part for comparison. ISP 20 determines (for example, compares) whether sensor parameter S2 is the same as sensor parameter S1. When the judgment result is that they are not the same (S2≠S1), the sensor parameter S2 calculated based on the image data of the current frame is written into sensor 10, and in the frame before the written sensor parameter S2 takes effect in sensor 10, Figure 5 In the example, in the F2' frame, the ISP parameter I2 is combined with the ISP compensation value C2 and written into ISP 20.

[0080] On the other hand, if the judgment result is different, in the frame before the written sensor parameters take effect in the sensor 10, the ISP parameters are combined with the ISP compensation value and written into the ISP 20, without writing the sensor parameters calculated based on the image data of the current frame into the sensor 10. If the judgment result is the same (S2=S1), the writing process of the sensor parameter S2 into the sensor 10 is not performed, and only the ISP parameter I2 is written into the ISP 20, and the writing timing may be a predetermined number of frames after the frame in which the ISP parameters and the sensor parameters are calculated; in this way, the sensor 10 directly uses the sensor parameter S1 written last time, saving the system resource overhead of writing the sensor parameter into the sensor 10. In this case, when the ISP compensation value C2 is the ratio (S2:S1) of the sensor parameter S2 calculated based on the image data of the current frame (F1') and the sensor parameter S1 calculated at the previous time based on the image data of the previous frame (F0'), when the judgment results are the same, the ISP compensation value C2 is 1; the ISP compensation value C2 serves as a scaling multiple of the calculated ISP parameter I2. When the ISP compensation value C2 is 1, it indicates that the ISP parameter I2 at this time does not need to be compensated and the ISP parameter can be directly written into ISP 20.

[0081] In this embodiment, before writing the sensor parameters calculated based on the image data of the current frame into the sensor 10, it is first determined whether the sensor parameters calculated based on the image data of the current frame are the same as the sensor parameters calculated based on the image data of the previous frame at the previous time. Only when the determination result is that they are not the same, the sensor parameters calculated based on the image data of the current frame are written into the sensor 10. The determination of whether the sensor parameters calculated based on the image data of the current frame are the same as the sensor parameters calculated based on the image data of the previous frame at the previous time can be performed before or after calculating the ISP compensation value, as long as the determination is performed after obtaining the sensor parameters calculated based on the image data of the current frame and the sensor parameters calculated based on the image data of the previous frame at the previous time. In addition, as an additional implementation, the ISP 20 can convert the sensor parameters calculated based on the image data of the current frame into the values ​​of the corresponding registers to prepare for writing into the corresponding registers, and determine whether the sensor parameters calculated based on the image data of the current frame are the same as the sensor parameters calculated based on the image data of the previous frame at the previous time by determining whether the value of the register currently converted is the same as the value of the register that is in effect in the sensor 10.

[0082] In some embodiments, the writing unit 260 of the ISP 20 may also be configured to write the sensor parameters calculated based on the image data of the current frame into the sensor 10 and write the ISP parameters combined with the ISP compensation values ​​into the ISP 20 a predetermined number of frames after the current frame.

[0083] According to the above description, after every predetermined number of frames, ISP 20 calculates the sensor parameters and ISP parameters to be applied in the subsequent frames based on the image data of the current frame. That is, the predetermined number of frames is the calculation cycle of ISP 20. For example, ISP 20 performs a calculation once every x frames to calculate the sensor parameters and ISP parameters to be applied in the subsequent frames, and the current frame calculated by ISP 20 is the Nth frame; in this case, the writing timing of writing the sensor parameters into the sensor 10 and the writing timing of writing the ISP parameters into ISP 20 after combining the ISP compensation values ​​should be the N+xth frame. For example, in Figure 5 In the example shown, the calculation cycle of ISP 20 is 1 frame, that is, calculation is performed every frame, and ISP 20 writes the calculated parameters to sensor 10 and ISP 20 1 frame after the frame in which the calculation is performed. According to the above scheme, the calculation cycle of ISP 20 is kept consistent with the writing cycle, so that the processing flow of the image processing system can be efficiently controlled.

[0084] The image processing system 100 according to the second embodiment is also applicable to the Inline mode and the Offline mode.

[0085] In some embodiments, the image processing system 100 also includes a memory 30, which is configured to temporarily store multiple frames of image data collected by the sensor 10; the predetermined number is greater than or equal to 2; the sensor parameters are written to the sensor 10 and take effect after x frames, where x is a positive integer less than the predetermined number; the acquisition unit 220 of the ISP 20 can also be configured to: obtain each frame of image data collected by the sensor 10 from the memory 30; and the writing unit 260 of the ISP 20 can also be configured to, in the frame where the written sensor parameters are effective in the sensor 10, rewrite the ISP parameters into the ISP 20 without combining them with the ISP compensation values.

[0086] In this embodiment, the image processing system 100 adopts the Offline mode. Figure 3The image processing system 100 is shown. The memory 30 temporarily stores multiple frames of image data collected by the sensor 10, and each frame of image data can be fused by multiple exposure data. For example, a high dynamic image (HDR) can be a frame output by fusion of data of different exposures (long, medium, and short exposures). The ISP 20 obtains each frame of image data collected by the sensor 10 from the memory 30. In this case, the predetermined number is greater than or equal to 2, that is, the sensor parameters and ISP parameters are not calculated continuously for each frame, and there are several frames between the two calculations. The sensor parameters are written to the sensor 10 and take effect after x frames, where x is a positive integer less than the predetermined number. For example, when the predetermined number is 3, the ISP 20 performs a calculation every 3 frames to calculate the sensor parameters and ISP parameters; when the sensor parameters are written to the sensor 10 in the Nth frame, the sensor parameters take effect in the N+xth frame, because x is a positive integer less than 3, so it can take effect in the N+1th frame or the N+2th frame. In this case, the ISP 20 may be configured to rewrite the ISP parameters into the ISP 20 without combining the ISP compensation values ​​in the frame where the written sensor parameters take effect in the sensor 10, that is, in the N+xth frame, that is, only rewrite the ISP parameters into the ISP 20 without combining the ISP compensation values. In short, the ISP parameters are written into the ISP 20 in combination with the ISP compensation values ​​in the frame before the written sensor parameters take effect in the sensor 10; and the ISP parameters are rewritten into the ISP 20 without combining the ISP compensation values ​​in the frame where the written sensor parameters take effect in the sensor 10. In the case where the ISP 20 also writes the ISP parameters into the ISP 20 in combination with the ISP compensation values ​​in the Nth frame (the same frame in the timing of writing the sensor parameters into the sensor 10), in the xth frame after the ISP parameters are written into the ISP 20 in combination with the ISP compensation values, the written sensor parameters have taken effect in the sensor image data stream, and therefore it is no longer necessary to apply the ISP compensation values ​​to perform software compensation in the ISP processing flow.

[0087] Exemplarily, the following further explanation is given by taking the predetermined number of frames as 2 frames and x=1 as an example. ISP 20 calculates sensor parameters and ISP parameters once every 2 frames. In the case where the current frame is the Nth frame to calculate sensor parameters and ISP parameters, the calculated sensor parameters and ISP parameters (after combining with ISP compensation values) can be written into sensor 10 and ISP 20, respectively, in the N+2th frame. The sensor parameters become effective after x (x=1) frames after being written into sensor 10, that is, the sensor parameters calculated in the Nth frame become effective in the N+2+x frame (that is, in the N+3th frame); and the ISP parameters combined with ISP compensation values ​​become effective in the N+2 frame. In the N+2+xth frame (that is, in the N+3th frame), the calculated ISP parameters are written into ISP 20 again without combining with ISP compensation values, because in the N+3th frame, the written sensor parameters have been effective in sensor 10, so the N+3th frame image data obtained by ISP 20 from sensor 10 is image data whose parameters have been effective, and there is no need to apply ISP compensation values ​​for compensation in the ISP processing flow.

[0088] For example, see Figure 6 , Figure 6 This is another operation diagram of the image processing system according to the second embodiment of the present application, which adopts the Offline mode. ISP 20 calculates the sensor parameters S1 (including exposure time parameters E1 and gain parameters G1) and ISP parameters in the F0' frame, and calculates the ISP compensation value C1. The sensor parameters S1 (including exposure time parameters E1 and gain parameters G1) are written to the sensor 10 in the F2 frame. In the sensor image data stream, the exposure time parameters E1 and gain parameters G1 take effect in the F3 frame. The ISP parameters I1 (combined with the ISP compensation value) are written to the ISP 20 in the F2' frame and take effect in the F2' frame. The image parameters presented in the F2' frame output by the ISP 20 are E0*G0*C1, and their values ​​are equal to E1*G1. Since the effective parameters of the sensor 10 in the F3 frame in the sensor image data stream are E1 and G1, that is, the calculated parameters have been effective in the sensor 10 in the F3 frame, in the ISP image data stream, the effective E1 and G1 do not need to be compensated in the corresponding F3' frame, so the ISP parameter I1 is rewritten into the ISP 20 in the F3' frame without combining the ISP compensation value C1, and the image parameter presented in the F3' frame output by the ISP 20 is E1*G1. The ISP 20 performs the next calculation in the F2' frame to calculate the new sensor parameters and ISP parameters.

[0089] Figure 6 The processing for subsequent frames is similar to the above process, which is not repeated in the figure and will not be described here.

[0090] exist Figure 6In the example, since it takes a certain amount of time for the image data to be transmitted through the memory, and it also takes a certain amount of time for the ISP to calculate the sensor parameters and the ISP parameters, the sensor parameters and the ISP parameters calculated at F0' are written into the sensor 10 and the ISP 20 in the F2 (or F2') frame, respectively. Ideally, regardless of the transmission time and the calculation time, the sensor parameters and the ISP parameters calculated at F0' can also be immediately written into the sensor 10 and the ISP 20 in the F1 (or F1') frame. In other embodiments, the sensor parameters and the ISP parameters can also be written into the sensor 10 and the ISP 20 in more frames after the calculation.

[0091] In some embodiments, the image processing system 100 adopts the Inline mode, that is, the acquisition unit 220 of the ISP 20 can be configured to directly acquire each frame of image data collected by the sensor 10 from the sensor 10; in this case, the predetermined number of frames can be 1 frame, for example, see Figure 5 .

[0092] In this case, ISP 20 directly obtains each frame of image data collected by sensor 10 from sensor 10 to calculate sensor parameters, ISP parameters and ISP compensation values. Specifically, ISP 20 calculates sensor parameters, ISP parameters, and ISP compensation values ​​for each frame. That is, the calculation cycle of ISP 20 is 1 frame, and accordingly, the writing cycle of ISP 20 is also 1 frame. For example, in the case where the current frame is the Nth frame to calculate sensor parameters and ISP parameters, the calculated sensor parameters and ISP parameters (after combining with the ISP compensation values) can be written to sensor 10 and ISP 20 respectively in the N+1th frame. In the ISP processing flow, ISP parameters (after combining with the ISP compensation values) are written to each frame, and the ISP parameters take effect immediately in the written frame, so the overall effective period of the image data output by ISP 20 is 1 frame. According to this embodiment, ISP 20 calculates and writes compensation values ​​for each frame, which further reduces the delay in Inline mode, and the processed image stream output by ISP 20 can reflect the changes in real-time scenes more quickly.

[0093] In some embodiments, the image processing system 100 includes a plurality of sensors 10 , and the ISP 20 arranges a processing timing for processing the image data collected by the plurality of sensors according to a calculation cycle for calculating the image data collected by each sensor 10 .

[0094] Specifically, the image processing system 100 includes a plurality of sensors 10, and the ISP 20 needs to process a plurality of frames of image data collected by each sensor 10, and output the processed image data streams respectively. In order to avoid data congestion in the ISP 20, the ISP 20 arranges the processing sequence according to the calculation cycle for calculating the image data collected by each sensor 10. Since the ISP 20 performs a calculation once for each image data of each sensor 10 after a predetermined number of frames, the calculation cycle is the aforementioned predetermined number of frames, for example, 1 frame, 2 frames, 3 frames... In addition, the parameter processing speed of the ISP 20 (such as the calculation speed and the writing speed) can also be considered to arrange the processing sequence for processing the image data collected by the plurality of sensors 10, so that only processing is to be performed at the same time, further improving the efficiency of the image processing system 100.

[0095] The image processing system according to the present application has significant progress compared to the prior art. In the prior art, taking Sony Sensor automatic exposure as an example, the response time of exposure / gain is the N+2th frame after the register is written (Nth frame), and the ambient light changes from dark to bright, and the exposure / gain of the ISP will enter the next steady state from one steady state; then in the conventional processing flow, the convergence cycle of the ISP processing is consistent with the register effectiveness cycle of the image sensor, and the convergence cycle of the entire process is an integer multiple of 2 frames. The image output by the ISP has a significant delay relative to the changes in the real-time scene. The present application fully considers the writing and response cycle of the register of the image sensor, and by adopting dynamic compensation in the ISP image data stream, it improves the video convergence speed introduced by the dynamic changes of the scene in the ISP processing flow, and further improves the ISP processing performance in the entire process.

[0096] In actual verification, when the ambient light suddenly becomes brighter, the image processing system automatically adjusts the exposure and gain, and the image converges from a stable state to the next stable state. Under the same conditions, the convergence speed of the image processing system of the present application is accelerated by about 50%. For example, under the same conditions, the convergence period according to the conventional scheme is approximately 27 frames, that is, when the ambient light suddenly changes, the output image stream enters a new stable state after 27 frames; and according to the image processing system of the present application, the convergence period is approximately 16 frames, that is, when the ambient light suddenly changes, the output image stream enters a new stable state after 16 frames, and no obvious flicker is observed during the convergence process.

[0097] Another aspect of the present application provides an image processing method for processing a stream including multiple frames of image data. Figure 7 , Figure 7is a flowchart of an image processing method according to an embodiment of the present application, and the image processing method includes the following steps S710-S740. It is easy to understand that the image processing method is executed by an ISP, for example, the ISP 20 in the image processing system 100 in the above-mentioned embodiment is executed.

[0098] In S710, each frame of image data collected by the sensor is obtained;

[0099] In S720, after every predetermined number of frames, sensor parameters and image signal processor parameters to be applied in subsequent frames are calculated based on image data of the current frame;

[0100] In S730, the sensor parameters calculated based on the image data of the current frame are written into the sensor, and the image signal processor parameters are written into the image signal processor in the frame in which the written sensor parameters are effective in the sensor;

[0101] In S740, the image data is processed using the written image signal processor parameters, and the processed image data is output.

[0102] Another aspect of the present application provides an image processing method for processing a stream including multiple frames of image data. Figure 8 , Figure 8 is a flowchart of an image processing method according to an embodiment of the present application, and the image processing method includes the following steps S810-S850. It is easy to understand that the image processing method is executed by an ISP, for example, the ISP 20 in the image processing system 100 in the above-mentioned embodiment is executed.

[0103] In S810, each frame of image data collected by the sensor is obtained;

[0104] In S820, after every predetermined number of frames, sensor parameters and image signal processor parameters to be applied in subsequent frames are calculated based on image data of the current frame;

[0105] At S830, an image signal processor compensation value is calculated based on the sensor parameters calculated based on the image data of the current frame and the sensor parameters calculated at a previous time based on the image data of the previous frame;

[0106] In S840, the sensor parameters calculated based on the image data of the current frame are written into the sensor, and the image signal processor parameters are combined with the image signal processor compensation value and written into the image signal processor in a frame before the written sensor parameters take effect in the sensor;

[0107] In S850, the image data is processed using the written ISP parameters combined with the ISP compensation values, and the processed image data is output.

[0108] In some embodiments, S830 specifically includes: calculating the image signal processor compensation value as a change between the sensor parameter calculated based on the image data of the current frame and the sensor parameter calculated last time based on the image data of the previous frame.

[0109] In some embodiments, S830 specifically includes: calculating a ratio of a sensor parameter calculated based on the image data of the current frame to a sensor parameter previously calculated based on the image data of the previous frame as an image signal processor compensation value; and

[0110] Step S840 specifically includes: writing the sensor parameters calculated based on the image data of the current frame into the sensor, and in a frame before the written sensor parameters take effect in the sensor, multiplying the image signal processor parameters by the image signal processor compensation value and writing them into the image signal processor.

[0111] In some embodiments, step S840 specifically includes:

[0112] Determining whether the sensor parameters calculated based on the image data of the current frame are the same as the sensor parameters previously calculated based on the image data of the previous frame;

[0113] If the judgment result is different, the sensor parameters calculated based on the image data of the current frame are written into the sensor, and in the frame before the written sensor parameters take effect in the sensor, the image signal processor parameters are combined with the image signal processor compensation value and written into the image signal processor.

[0114] In some embodiments, step S840 specifically includes: a predetermined number of frames after the current frame, writing sensor parameters calculated based on image data of the current frame into the sensor, and writing image signal processor parameters combined with image signal processor compensation values ​​into the image signal processor.

[0115] In some embodiments, the predetermined number is greater than or equal to 2; the sensor parameters become effective after x frames after being written to the sensor, where x is a positive integer less than the predetermined number; step S810 specifically includes: acquiring each frame of image data collected by the sensor from the memory; and the image processing method also includes: S860, in the frame where the written sensor parameters are effective in the sensor, rewriting the image signal processor parameters without combining the image signal processor compensation value into the image signal processor.

[0116] In some embodiments, the predetermined number is 1; step S810 specifically includes: directly acquiring from the sensor each frame of image data collected by the sensor.

[0117] The specific details of the image processing method according to the present application can be referred to the above description of the image processing system according to the present application, which will not be repeated here.

[0118] Another aspect of the present application provides an image signal processor, comprising a logic circuit capable of executing a computer program stored in a memory, wherein the logic circuit implements the image processing method according to an embodiment of the present application when executing the computer program.

[0119] The specific details of the image signal processor according to the present application can be referred to the above description of the image processing system according to the present application, which will not be repeated here.

[0120] Another aspect of the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the image processing method according to the embodiment of the present application is implemented.

[0121] The specific details of the non-transitory computer-readable storage medium according to the present application can be referred to the above description of the image processing system according to the present application, which will not be repeated here.

[0122] Another aspect of the present application provides a computer program product, including a computer program, which implements the image processing method according to the embodiment of the present application when the computer program is executed by a processor.

[0123] The specific details of the computer program product according to the present application can be referred to the above description of the image processing system according to the present application, which will not be repeated here.

[0124] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0125] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. Image processing system, including: A sensor is configured to collect multiple frames of image data, and sensor parameters written into the sensor take effect after at least one frame has passed; as well as An image signal processor, wherein after image signal processor parameters are written into the image signal processor, the parameters take effect immediately in the written frame, and the image signal processor comprises: An acquisition unit, configured to acquire each frame of image data collected by the sensor; a calculation unit configured to calculate sensor parameters and image signal processor parameters to be applied in subsequent frames based on image data of a current frame every time a predetermined number of frames have passed; a writing unit configured to write sensor parameters calculated based on image data of a current frame into the sensor, and write the image signal processor parameters into the image signal processor in a frame in which the written sensor parameters are effective in the sensor; The processing unit is configured to apply the written image signal processor parameters to process the image data and output the processed image data.

2. Image processing system, including: A sensor is configured to collect multiple frames of image data, and sensor parameters written into the sensor take effect after at least one frame has passed; as well as An image signal processor, wherein after image signal processor parameters are written into the image signal processor, the parameters take effect immediately in the written frame, and the image signal processor comprises: An acquisition unit, configured to acquire each frame of image data collected by the sensor; Computing unit, configured as: After each predetermined number of frames, calculating sensor parameters and image signal processor parameters to be applied in subsequent frames based on image data of a current frame; Calculating an image signal processor compensation value based on a sensor parameter calculated based on image data of a current frame and a sensor parameter previously calculated based on image data of a previous frame; a writing unit configured to write the sensor parameters calculated based on the image data of the current frame into the sensor, and write the image signal processor parameters combined with the image signal processor compensation value into the image signal processor in a frame before the written sensor parameters take effect in the sensor; The processing unit is configured to process the image data by applying the written image signal processor parameters combined with the image signal processor compensation value, and output the processed image data.

3. The image processing system according to claim 2, wherein: The calculation unit is further configured to calculate the image signal processor compensation value as a change between a sensor parameter calculated based on image data of a current frame and a sensor parameter previously calculated based on image data of a previous frame.

4. The image processing system according to claim 3, wherein The calculation unit is further configured to calculate the image signal processor compensation value by combining the sensor parameters calculated based on the image data of the current frame and the sensor parameters calculated at a previous time based on the image data of the previous frame; The writing unit is further configured to write the multiplied image signal processor parameter and the image signal processor compensation value into the image signal processor.

5. The image processing system according to claim 2, wherein: The writing unit is further configured as: Determining whether the sensor parameters calculated based on the image data of the current frame are the same as the sensor parameters calculated previously based on the image data of the previous frame; When the judgment result is that they are not the same, the sensor parameters calculated based on the image data of the current frame are written into the sensor, and in the frame before the written sensor parameters take effect in the sensor, the image signal processor parameters are combined with the image signal processor compensation values ​​and written into the image signal processor.

6. The image processing system according to any one of claims 2 to 5, wherein: The writing unit is further configured as: The sensor parameters calculated based on the image data of the current frame are written into the sensor, and the image signal processor parameters combined with the image signal processor compensation values ​​are written into the image signal processor, the predetermined number of frames after the current frame.

7. The image processing system according to claim 6, further comprising a memory configured to temporarily store multiple frames of image data collected by the sensor; wherein The predetermined number is greater than or equal to 2; The sensor parameters are written into the sensor and become effective after x frames have passed, where x is a positive integer less than the predetermined number; The acquisition unit is configured to acquire each frame of image data collected by the sensor from the memory; and The writing unit is configured to rewrite the image signal processor parameters into the image signal processor without combining the image signal processor compensation value in a frame in which the written sensor parameters are effective in the sensor.

8. The image processing system according to claim 6, wherein: The acquisition unit is configured to: directly acquire each frame of image data collected by the sensor from the sensor; The predetermined number is 1.

9. The image processing system according to any one of claims 1 to 5, wherein: The image processing system includes a plurality of the sensors; The image signal processor arranges a processing timing for processing the image data collected by the plurality of sensors according to a calculation cycle for calculating the image data collected by each sensor.

10. An image processing method for processing a stream including multiple frames of image data, comprising: Get each frame of image data collected by the sensor; After each predetermined number of frames, calculating sensor parameters and image signal processor parameters to be applied in subsequent frames based on image data of a current frame; Writing sensor parameters calculated based on image data of a current frame into the sensor, and writing image signal processor parameters into the image signal processor in a frame in which the written sensor parameters are effective in the sensor; The image data is processed using the written image signal processor parameters, and the processed image data is output.

11. An image processing method for processing a stream including multiple frames of image data, comprising: Get each frame of image data collected by the sensor; After each predetermined number of frames, calculating sensor parameters and image signal processor parameters to be applied in subsequent frames based on image data of a current frame; Calculating an image signal processor compensation value based on a sensor parameter calculated based on image data of a current frame and a sensor parameter previously calculated based on image data of a previous frame; Writing the sensor parameters calculated based on the image data of the current frame into the sensor, and writing the image signal processor parameters combined with the image signal processor compensation value into the image signal processor in a frame before the written sensor parameters take effect in the sensor; The image data is processed using the written image signal processor parameters combined with the image signal processor compensation values, and the processed image data is output.

12. An image signal processor comprising a logic circuit capable of executing a computer program stored in a memory, wherein the logic circuit implements the image processing method according to claim 10 or 11 when executing the computer program.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the image processing method according to claim 10 or 11 when executed by a processor.

14. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the image processing method according to claim 10 or 11 is implemented.

Images