Six-stage water bayer image dynamic range correction circuit and correction method

Abstract

This invention discloses a six-stage pipelined Bayer image dynamic range correction circuit and method, relating to the field of video image processing technology. The method includes: performing PQ decomposition on the luminance data converted from the Bayer image to obtain P data and Q data; scaling and buffering the decomposed P data; filtering the buffered P data; further buffering the filtered P data; amplifying the P data from the previous frame and fusing it with the Q data of the current frame; and finally converting the fused data back to Bayer data. This solution implements on-chip image dynamic range correction with low latency.

Description

Technical Field

[0001] This invention relates to video processing technology, and more particularly to a dynamic range correction circuit and correction method for video images implemented on a chip. Background Technology

[0002] In security scenarios, the light sources are complex and often experience sudden changes in brightness. However, the radiation fluctuation range of the photosensitive chip is limited. This can lead to overexposure or underexposure in certain areas under specific exposure intensities, resulting in image distortion.

[0003] Feng Qingchun, Wang Xiu, Li Junhui, et al. Color Correction Method for Greenhouse Tomato Plant Images Based on High Dynamic Range Imaging [J / OL]. Transactions of the Chinese Society for Agricultural Machinery, 2020, 51(11): 235-242. In this paper, the brightness information of four images with exposure times of 0.01ms, 0.05ms, 0.08ms, and 0.1ms is extracted respectively to obtain the relationship between brightness and exposure. Then, the high dynamic range is compressed using the S-curve to perform color correction and reconstruction on the image. This method uses multi-frame information, and four frames of images need to be stored in the circuit implementation, resulting in a large memory overhead.

[0004] Wu Guotao. Research and Implementation of Dynamic Range Enhancement Method for Single-Frame Images Based on Lightweight Networks [D]. Shandong University, 2020. The method adopted in this paper is a single-frame enhancement algorithm based on neural networks. However, the computational complexity is very high and it relies on powerful computing capabilities. This scheme uses a high-performance processor to process a 12-megapixel image in 3 seconds, but it cannot process real-time video images.

[0005] Therefore, current approaches to implementing image dynamic range correction on-chip are relatively limited, and when processing real-time dynamic video images, they often involve significant input-output latency. Solving this problem requires powerful computing capabilities and additional memory. Summary of the Invention

[0006] This invention addresses the shortcomings of existing technologies by providing a six-stage pipeline design scheme to achieve on-chip image dynamic range correction with low latency.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0008] The six-stage pipelined Bayer image dynamic range correction circuit employs a six-stage pipelined processing circuit, with the first to sixth stages connected sequentially, including:

[0009] The first-stage circuit takes Bayer image data as input, converts it into luminance data, and then decomposes it into P data and Q data.

[0010] The second-stage circuit is connected to the output of the first-stage circuit. It scales the P data and outputs the scaled P data.

[0011] The third-stage circuit connects to the output of the second-stage circuit and includes an electrically connected Gaussian filter and a time-domain filter.

[0012] The fourth-stage circuit, connected to the output of the third-stage circuit, includes a first buffer unit connected to the output of the time-domain filter to receive the previous frame P data output by the time-domain filter; it also has a first output and a second output. The first output is connected to the time-domain filter to transmit the previous frame P data to the time-domain filter, and the second output outputs the previous frame P data to the fifth-stage circuit.

[0013] The fifth stage circuit includes an amplification unit and a fusion unit. It takes the previous frame P data from the buffer unit and sends it to the amplification unit. The amplification unit amplifies the P data to the original image size. The first input terminal of the fusion unit is connected to the output terminal of the amplification unit, and the second input terminal of the fusion unit receives the Q data of the current frame.

[0014] The sixth-stage circuit converts the output data of the fifth-stage circuit into Bayer data.

[0015] As a preferred embodiment, the fifth-level circuit also includes a weighting unit. After the P data and Q data are fused, the weighting unit weights the fused data and the current frame brightness data and outputs the result to the sixth-level circuit.

[0016] As a preferred embodiment, it also includes a second cache unit, through which the second-level circuit and the third-level circuit are connected. The scaled P data is stored in the second cache unit, and the third-level circuit retrieves the cached P data from the second cache unit.

[0017] As a preferred embodiment, the third-stage circuit includes an electrically connected Gaussian filter and a time-domain filter.

[0018] The output of the Gaussian filter is connected to the first input of the time-domain filter. The second input of the time-domain filter is connected to the first output of the buffer unit of the fourth-stage circuit to obtain the previous frame P data in the buffer unit. The output of the time-domain filter is connected to the input of the buffer unit to output the current frame to the buffer unit.

[0019] As a preferred embodiment, the first-stage circuit includes:

[0020] The luminance conversion unit is used to convert Bayer data into luminance data;

[0021] A brightness curve adjustment unit is connected to the brightness conversion unit to adjust the brightness curve and output the adjusted brightness data sF(i) and the original brightness data F(i);

[0022] The relationship between sF(i) and F(i) is: sF(i) = F(i) * 2 - 1.

[0023] Based on the above circuit structure, a six-stage pipelined Bayer image dynamic range correction method is provided, comprising the following steps:

[0024] 1) Convert the Bayer image into luminance data, adjust the luminance transformation curve according to the scene, and perform PQ decomposition to obtain P data and Q data;

[0025] 2) Scale and cache the decomposed P data, then filter the cached P data and cache the filtered P data again.

[0026] 3) After reading the P data from the previous frame from the buffer and amplifying it, fuse it with the Q data from the current frame;

[0027] 4) After merging, switch back to Bayer data.

[0028] As a preferred option, the method in step 2) specifically includes:

[0029] The decomposed P data is scaled and then cached in the second cache unit;

[0030] The P data is filtered using a Gaussian filter and a time-domain filter. The p data is taken from the second buffer unit and input into the Gaussian filter. The Gaussian filter outputs the data to the time-domain filter.

[0031] The first input terminal of the time-domain filter is connected to the Gaussian filter, and the output terminal of the time-domain filter is connected to the first buffer unit to buffer the current frame P data. The first buffer unit outputs the previous frame P data to the second input terminal of the time-domain filter.

[0032] As a preferred embodiment, the method for scaling the decomposed P data includes:

[0033] Perform a scaling operation on the decomposed P data with an optional scaling factor;

[0034] Divide the P data into blocks according to the scaling factor;

[0035] Calculate the summation of blocks in the horizontal direction, and then sum the results row by row in the subsequent column directions.

[0036] After the block operation is completed, the results of the block operation are averaged to obtain the scaled data.

[0037] As a preferred approach, the luminance data after fusion in step 3) is weighted and then converted back to Bayer data.

[0038] As a preferred embodiment, the method for adjusting the brightness transformation curve according to the scene includes the following steps:

[0039] Adjust the brightness curve of the converted brightness data and output the adjusted brightness data sF(i) and the original brightness data F(i).

[0040] The relationship between sF(i) and F(i) is: sF(i) = F(i) * 2 - 1;

[0041] The maximum and minimum values ​​of sF(i) are between [-1, 1].

[0042] The beneficial effects of this invention are:

[0043] The circuit disclosed in this invention adopts a six-stage pipeline structure, with corresponding row buffers storing the results before and after each stage of processing. Each stage of pipeline processing is independent of each other, and the calculation process does not require processor intervention.

[0044] This circuit adds a scaling process before Gaussian filtering, which allows intermediate calculation results to be stored in an on-chip cache, eliminating the need for external caches (such as DDR) for intermediate data. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is the overall block diagram of the six-stage flow Bayer image dynamic range correction circuit;

[0047] Figure 2 This is a circuit structure diagram of each stage of the six-stage Bayer image dynamic range correction circuit;

[0048] Figure 3 This is a schematic diagram of the first-stage circuit converting Bayer data to luminance data;

[0049] Figure 4 This is a schematic diagram of the brightness data curve adjustment for the first-stage circuit;

[0050] Figure 5 This is the structure diagram of solving the P data from the PQ decomposition of the first-stage circuit;

[0051] Figure 6This is the structure diagram for solving the Q data from the PQ decomposition of the first-stage circuit;

[0052] Figure 7 This is a schematic diagram of the operation process of the second-stage circuit P data scaling circuit;

[0053] Figure 8 This is a schematic diagram of the filtering operation of the 9×9 filter structure in the third-stage circuit;

[0054] Figure 9 This is a schematic diagram of the ping-pong buffer operation process in the third-level circuit;

[0055] Figure 10 This is a schematic diagram of the data amplification and operation process of the fifth-stage circuit P. Detailed Implementation

[0056] The present invention will be further described in detail below with reference to the embodiments. The following embodiments are explanations of the present invention, but the present invention is not limited to the following embodiments.

[0057] This application proposes a pipelined processing technology that divides the pipeline structure according to functional characteristics, enabling real-time data processing without waiting. After each stage finishes processing and outputting data, it can continue to process subsequent data, achieving a maximum real-time data processing capability of 4K@30fps.

[0058] The following presents a Bayer image dynamic range correction method and a six-stage pipeline circuit for this method;

[0059] like Figure 1 and Figure 2 It includes a first-level circuit, a second-level circuit, a third-level circuit, a fourth-level circuit, a fifth-level circuit, and a sixth-level circuit. The first-level circuit to the sixth-level circuit are electrically connected in sequence, wherein:

[0060] The first-stage circuit 100 takes Bayer image data as input, converts it into luminance data, and then decomposes it into P data and Q data.

[0061] The second-stage circuit 200 is connected to the output of the first-stage circuit, performs scaling processing on the P data, and outputs the scaled P data.

[0062] The third-stage circuit 300 is connected to the output of the second-stage circuit and includes a Gaussian filter and a time-domain filter that are electrically connected.

[0063] The fourth-stage circuit 400 is connected to the output of the third-stage circuit. It includes a first buffer unit, which is connected to the output of the time-domain filter and receives the previous frame P data output by the time-domain filter. It also has a first output and a second output. The first output is connected to the time-domain filter and transmits the previous frame P data to the time-domain filter. The second output outputs the previous frame P data to the fifth-stage circuit.

[0064] The fifth-stage circuit 500 includes an amplification unit and a fusion unit. It takes the previous frame P data from the buffer unit and sends it to the amplification unit. The amplification unit amplifies the P data to the original image size. The first input terminal of the fusion unit is connected to the output terminal of the amplification unit, and the second input terminal of the fusion unit receives the Q data of the current frame.

[0065] The sixth-stage circuit 600 converts the output data of the fifth-stage circuit into Bayer data.

[0066] This circuit employs a six-stage pipeline structure, with corresponding row buffers storing the results before and after each stage. Each pipeline stage operates independently, and the computation process requires no processor intervention.

[0067] This circuit adds a scaling process before Gaussian filtering, which allows intermediate calculation results to be stored in an on-chip cache, eliminating the need for external caches (such as DDR) for intermediate data.

[0068] Real-time image processing: First, the image line and field pipeline cannot be interrupted, which is not possible if the CPU needs to be interrupted to participate in the calculation. This solution effectively solves this problem. Second, the processing time of this solution is short.

[0069] 100, the first-stage circuit includes:

[0070] The luminance conversion unit converts Bayer data into luminance data. It buffers the Bayer data using two rows and performs a convolution operation on the Bayer image data using three rows of data and a preset convolution kernel to obtain the luminance data, such as... Figure 3 As shown;

[0071] The brightness curve adjustment unit is connected to the brightness conversion unit to perform brightness curve adjustment and outputs F(i) and sF(i).

[0072] like Figure 4 (a) is the brightness curve F(i), such as Figure 4 (b) is the brightness curve sF(i);

[0073] The input data is divided into 32 segments for interpolation to obtain a new brightness curve F(i). sF(i) is obtained by doubling the size of F(i) and then shifting it downwards by half along the vertical axis so that the maximum and minimum values ​​of the output are between [-1,1]. That is, the formula is sF(i)=F(i)*2-1.

[0074] Output the original brightness data F(i) (Y' in the figure) and the brightness data sF(i) (Y” in the figure) after curve adjustment. sF(i) is input into the second-level circuit to complete the processing from the second-level circuit to the fifth-level circuit, until the fusion of P data box and Q data is completed and then the brightness data F(i) is added.

[0075] The PQ decomposition method is as follows:

[0076] The formulas for solving P0 to P8 are as follows:

[0077] P0(x)=1;P1(x)=x;P n (x)=a n P n-2 (x)+b n *x*P n-1 (x), a n and b n It is to calculate P n The required constant circuit structure is as follows: Figure 5 As shown

[0078] The calculation formulas for Q1 to Q7 are as follows:

[0079] Circuit structure as follows Figure 6 As shown.

[0080] 200, Second-stage circuit:

[0081] Scaling of P data for the 7 channels P1-P7, such as Figure 7 As shown: Assuming there is data P from line 0 to line N, the data P is divided into blocks according to the scaling factor; the scaling factor is optional, such as 16×16, 32×32, 64×64, 128×128 which can be configured;

[0082] Calculate the summation of blocks in the horizontal direction, and then sum the results row by row in the subsequent column directions.

[0083] After the block operation is completed, the results of the block operation are averaged to obtain the scaled data.

[0084] It also includes a second cache unit. The second-level circuit and the third-level circuit are connected through the second cache unit. The scaled P data is stored in the second cache unit, and the third-level circuit retrieves the cached P data from the second cache unit.

[0085] The optimized circuit structure scales the P data first and then caches it. When needed, it is retrieved and scaled up again, which reduces the amount of intermediate data processing, saves on-chip cache, eliminates the need for additional circuitry to read and write external DDR, and eliminates the need for external storage to cache intermediate data, further reducing design complexity and making the structure simpler.

[0086] 300, third-level circuit:

[0087] It includes a Gaussian filter and a time-domain filter that are electrically connected. The output of the Gaussian filter is connected to the first input of the time-domain filter. The second input of the time-domain filter is connected to the first output of the buffer unit of the fourth-stage circuit to obtain the previous frame P data in the first buffer unit. The output of the time-domain filter is connected to the input of the first buffer unit to output the current frame to the buffer unit.

[0088] The Gaussian filter uses a 9×9 filtering structure, such as... Figure 8 As shown, the scaled real-time data is first subjected to a horizontal 1×9 filter point by point, and 9 rows of horizontal filtering results are cached. Then, the data is retrieved one by one and subjected to a 9×1 filter.

[0089] By utilizing the characteristics of real-time image streaming and taking advantage of the time difference, the processing time of horizontal filtering and vertical filtering is separated (horizontal first, then vertical). The horizontal and vertical filtering reuses 9 multipliers, and the entire circuit only requires 63 multipliers.

[0090] 400, fourth-level circuit:

[0091] like Figure 9 The fourth-stage pipeline is the first buffer unit, specifically a ping-pong buffer for one frame of P data. Data retrieval for right-side image magnification occurs within the valid region of the image, while data access for left-side storage occurs within the blanking region. The fourth-stage pipeline's frame buffer caches the processing result of the previous frame. The current frame retrieves and merges the P data from the previous frame with the Q data of the current frame, requiring only synchronization with the Q data of the current frame. The delay is the delay of the UP (magnification) processing. Using a ping-pong buffer structure, subsequent fusion calculations read the result from the previous frame, reducing line delay. The module requires only two lines of delay from input to output.

[0092] 500, fifth-level circuit:

[0093] To amplify the P data, based on the scaling method described above, two-point bilinear interpolation is used for calculation. Vertical amplification is performed first, followed by horizontal amplification. The two-point bilinear interpolation requires buffering two rows of data. The circuit module structure is as follows: Figure 10 As shown.

[0094] It also includes a weighting unit. After the P data and Q data are fused, the weighting unit weights the fused data and the current frame brightness data and outputs it to the sixth-level circuit.

[0095] Example 2:

[0096] The Bayer image dynamic range correction method disclosed in Example 1 includes the following steps:

[0097] 1) Convert the Bayer image into luminance data, adjust the luminance transformation curve according to the scene, and perform PQ decomposition to obtain P data and Q data;

[0098] 2) Scale and cache the decomposed P data, then filter the cached P data and cache the filtered P data again.

[0099] 3) After reading the P data from the previous frame from the buffer and amplifying it, fuse it with the Q data from the current frame;

[0100] 4) After merging, switch back to Bayer data.

[0101] In step 1), the Bayer image data is converted into luminance data, the luminance curve is adaptively adjusted according to the scene, and then the luminance data is decomposed into PQ.

[0102] Specifically, this includes converting Bayer data into luminance data. This is achieved by performing a convolution operation on the input Bayer image using a preset convolution kernel to obtain the luminance data, such as... Figure 3 As shown;

[0103] Further, the brightness curve is adaptively adjusted according to the scene, the brightness curve of the converted brightness data is adjusted, and the adjusted brightness data sF(i) and the original brightness data F(i) are output.

[0104] like Figure 4 (a) is the brightness curve F(i), such as Figure 4 (b) is the brightness curve sF(i);

[0105] The input data is divided into 32 intervals for interpolation to obtain a new brightness curve F(i). sF(i) is obtained by doubling the size of F(i) and then shifting it downwards by half along the vertical axis, so that the maximum and minimum values ​​of the output are between [-1,1], i.e., the formula sF(i)=F(i)*2-1.

[0106] Data sF(i) is input into the second-stage circuit, and the processing is completed from the second-stage circuit to the fifth-stage circuit until the fusion of p data and q data is completed and then added to the data F(i). The PQ decomposition method is detailed in Example 1 and will not be repeated here.

[0107] Step 2) specifically includes the following methods:

[0108] Methods for scaling the decomposed P data include:

[0109] Perform a scaling operation on the decomposed P data with an optional scaling factor;

[0110] Divide the P data into blocks according to the scaling factor;

[0111] Calculate the summation of blocks in the horizontal direction, and then sum the results row by row in the subsequent column directions.

[0112] After the block operation is completed, the results of the block operation are averaged to obtain the scaled data.

[0113] The method in step 2) further includes:

[0114] The decomposed P data is scaled and buffered. The P data is filtered using a Gaussian filter and a time-domain filter. The buffered P data is input into the Gaussian filter, and the Gaussian filter outputs the current frame P data to the time-domain filter.

[0115] The output of the time-domain filter is the previous frame's P data, which is then fed to the Gaussian filter. The output of the time-domain filter is the previous frame's P data, which is then magnified to the original image and fused with the Q data of the current frame.

[0116] Step 3) involves reading the P data from the previous frame from the buffer, amplifying it, and then fusing it with the Q data from the current frame.

[0117] To enlarge the P data, based on the scaling method described above, the data enlargement is performed using two-point bilinear interpolation, first vertically, then horizontally, as follows: Figure 10 As shown.

[0118] It also includes weighted processing. After the P data and Q data are fused, the fused brightness data is multiplied by a weighting coefficient and then added to the unsigned data before being output to the sixth-level circuit.

[0119] Furthermore, it should be noted that the shapes and names of the parts and components described in the specific embodiments described in this specification may differ. All equivalent or simple variations made to the structure, features, and principles described in this patent concept are included within the protection scope of this patent. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to replace them, as long as they do not depart from the structure of this invention or exceed the scope defined in these claims, they should all fall within the protection scope of this invention.

Claims

1. A six-stage flow-through Bayer image dynamic range correction circuit, characterized in that, A six-stage pipeline processing circuit is adopted, with the first stage circuit to the sixth stage circuit being electrically connected sequentially, including: The first-stage circuit takes Bayer image data as input, converts it into luminance data, and then decomposes it into P data and Q data. Specifically, a preset convolution kernel is used to perform convolution operations on the input Bayer image data to obtain luminance data. The second-stage circuit is connected to the output of the first-stage circuit. It scales the P data and outputs the scaled P data. The third-stage circuit connects to the output of the second-stage circuit and includes an electrically connected Gaussian filter and a time-domain filter. The fourth-stage circuit, connected to the output of the third-stage circuit, includes a first buffer unit connected to the output of the time-domain filter, receiving the current frame P data output by the time-domain filter; it also has a first output and a second output. The first output is connected to the time-domain filter, transmitting the previous frame P data to the time-domain filter, and the second output outputs the previous frame P data to the fifth-stage circuit. The first buffer unit stores the current frame P data output by the time-domain filter and outputs it as the previous frame P data to the time-domain filter in the next frame. The fifth stage circuit includes an amplification unit and a fusion unit. It takes the previous frame P data from the buffer unit and sends it to the amplification unit. The amplification unit amplifies the P data to the original image size. The first input terminal of the fusion unit is connected to the output terminal of the amplification unit, and the second input terminal of the fusion unit receives the Q data of the current frame. The sixth-stage circuit converts the output data of the fifth-stage circuit into Bayer data.

2. The six-stage flow-through Bayer image dynamic range correction circuit according to claim 1, characterized in that, The fifth-level circuit also includes a weighting unit. After the P data and Q data are fused, the weighting unit weights the fused data and the current frame brightness data and outputs it to the sixth-level circuit.

3. The six-stage flowing Bayer image dynamic range correction circuit according to claim 1, characterized in that, It also includes a second cache unit, the second-level circuit and the third-level circuit are connected through the second cache unit, the scaled P data is stored in the second cache unit, and the third-level circuit retrieves the cached P data from the second cache unit.

4. The six-stage flow-through Bayer image dynamic range correction circuit according to claim 1, characterized in that, The third-stage circuit includes an electrically connected Gaussian filter and a time-domain filter. The output of the Gaussian filter is connected to the first input of the time-domain filter. The second input of the time-domain filter is connected to the first output of the buffer unit of the fourth-stage circuit to obtain the previous frame P data in the buffer unit. The output of the time-domain filter is connected to the input of the buffer unit to output the current frame to the buffer unit.

5. The six-stage flow-through Bayer image dynamic range correction circuit according to claim 1, characterized in that, The first-stage circuit includes: The luminance conversion unit is used to convert Bayer data into luminance data; A brightness curve adjustment unit is connected to the brightness conversion unit to adjust the brightness curve and output the adjusted brightness data sF(i) and F(i). The relationship between sF(i) and F(i) is: sF(i) = F(i) * 2 - 1.

6. The Bayer image dynamic range correction method, characterized in that, Includes the following steps: 1) Convert the Bayer image into luminance data, adjust the luminance transformation curve according to the scene, and perform PQ decomposition to obtain P data and Q data. Specifically, a preset convolution kernel is used to perform convolution operation on the Bayer image to convert it into luminance data. 2) Scale and buffer the decomposed P data, filter the buffered P data, store the filtered current frame P data output by the time domain filter, and output it as the previous frame P data to the time domain filter in the next frame. 3) After reading the P data from the previous frame from the buffer and amplifying it, fuse it with the Q data of the current frame; 4) After merging, the data is converted back to Bayer data.

7. The Bayer image dynamic range correction method according to claim 6, characterized in that, Step 2) specifically includes the following methods: The decomposed P data is scaled and then cached in the second cache unit; The P data is filtered using a Gaussian filter and a time-domain filter. The P data is taken from the second buffer unit and input into the Gaussian filter. The Gaussian filter outputs the data to the time-domain filter. The first input terminal of the time-domain filter is connected to the Gaussian filter, and the output terminal of the time-domain filter is connected to the first buffer unit to buffer the current frame P data. The first buffer unit outputs the previous frame P data to the second input terminal of the time-domain filter.

8. The Bayer image dynamic range correction method according to claim 6, characterized in that, The method for scaling the decomposed P data includes: Perform a scaling operation on the decomposed P data with an optional scaling factor; Divide the P data into blocks by selecting an appropriate scaling factor based on the image size; Calculate the summation of blocks in the horizontal direction, and then sum the results row by row in the subsequent column directions. After the block operation is completed, the results of the block operation are averaged to obtain the scaled data.

9. The Bayer image dynamic range correction method according to claim 6, characterized in that, The luminance data after fusion in step 3) is weighted and then converted back to Bayer data.

10. The Bayer image dynamic range correction method according to claim 6, characterized in that, The method for adjusting the brightness transformation curve according to the scene includes the following steps: Adjust the brightness curve of the converted brightness data and output the adjusted brightness data sF(i) and F(i). The relationship between sF(i) and F(i) is: sF(i) = F(i) * 2 - 1; The maximum and minimum values ​​of sF(i) are between [-1, 1].

Images