Image enhancement system, method and medium

Abstract

The present invention proposes an image enhancement system, method and medium. The system includes: (N‑1) row cache FIFOs connected in series, N row cache registers, a demosaic module, a G component cache FIFO, and a G component cache register. The present application reasonably controls the reading and writing of the row cache FIFO and the register, so that each clock and each row of image data always outputs at least 5 pixels when transmitting any number of pixel data, so as to meet the calculation requirements of the demosaic algorithm of the present invention. The present application balances and optimizes the number of combinational logic levels to be executed in each clock cycle and the total register occupancy. In the pipeline design of the demosaic algorithm, the limited hardware internal resources and the timing convergence problem of high clock frequency are fully considered, and the combinational logic level optimization and pipeline level optimization are combined to achieve better FPGA resource occupancy of various types, thereby achieving efficient calculation. Based on the calculation strategy proposed in the present application, the image enhancement process is finally realized, and a high-quality color display image is obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of real-time image processing, and in particular to an image enhancement system, method and medium. Background Art

[0002] A single CCD or CMOS color image sensor is formed by covering a color filter array (CFA) on its surface. The Bayer array is a special filter arrangement that uses red, green, and blue filters to form a mosaic layout.

[0003] In the Bayer array, the number of green pixels is the sum of the number of red and blue pixels, because the human eye is more sensitive to green. Common Bayer array arrangements include RGGB, GRBG, GBRG, and BGRG.

[0004] The image output by the image sensor covered with the Bayer array is a Bayer image, which is actually a monochrome image without demosaicing, and it lacks the sampling of the spatial color channel. Each pixel in the Bayer image has only one component among R, G, and B. In order to realize the color display of the image, it is necessary to use the Demosaic algorithm to perform Bayer image interpolation calculation, restore each pixel to the complete RGB three-component data, realize image enhancement, and finally obtain a high-quality color image.

[0005] Existing de-mosaicing algorithms can be roughly divided into two categories: the first category is linear interpolation represented by classic bilinear interpolation, which is prone to loss and blurring of edge information. It can improve throughput from a hardware perspective, but the restored image has poor subjective visual quality and is prone to aliasing. The second category is edge detection interpolation represented by Hamilton and Kimmel, which introduces directional operators. The introduction of edge detection first restores the green component and then restores the red and blue components, which greatly improves the quality of the reconstructed image. However, there are color moiré fringes in areas with tight edges or fine textures. At the same time, due to its relative complexity and the fact that the image restoration process does not conform to the real-time processing of hardware logic, it is not easy to implement in hardware.

[0006] Chinese patent CN 118158551 B discloses a de-mosaicing method, system and medium based on FPGA. Based on the de-mosaicing algorithm provided by the patent, the hardware implementation of the algorithm needs to occupy FPGA storage resources with a capacity of 9 lines of pixel data. The de-mosaicing algorithm used in this application is different from the idea of ​​judging the interpolation method, and the algorithm recorded in the present invention uses fewer hardware resources.

[0007] Chinese patent CN 109104595 B provides an FPGA implementation method for Hamilton adaptive interpolation in real-time image processing. The patent uses 6 row RAMs to circularly buffer the original Bayer data, and uses 2-way G component recovery parallel operation, 3-way R and B component parallel operation, pipeline processing, data synchronization and other methods to implement the Hamilton adaptive interpolation algorithm on the FPGA. The hardware architecture set up in this application and the de-mosaic algorithm used are different from those of the patent, and the hardware implementation saves more resources.

[0008] Chinese patent CN 116797679 A discloses a Bayer image decoding system based on FPGA. Based on the edge perception algorithm, the patent modifies the original green pixel value of the target pixel, improving the smoothness of the decoded image and the de-mosaicing effect. This application proposes the setting of registers to solve the problem that the patent cannot meet the real-time processing of any number of pixel data; and the de-mosaicing algorithm of this solution is different from that of the patent. Summary of the invention

[0009] The present invention provides an image enhancement system, method and medium, which can solve at least one of the above technical problems.

[0010] To achieve the above object, the present invention proposes the following technical solutions:

[0011] An image enhancement system, comprising:

[0012] (N-1) row buffer FIFOs connected in series, for receiving input Bayer data row by row, and outputting row image data to corresponding row buffer registers; wherein N≥3;

[0013] N row buffer registers, used for buffering the row image data currently input and the row image data output by the row buffer FIFO; based on each clock cycle, any row buffer FIFO and the T pixels transmitted by the currently input row image data respectively, the number of buffer levels is set to output N rows of image data at the same time and any row of image data is at least five pixels;

[0014] The Demosaic module is used to receive the image data output by the line buffer register, calculate the G component line by line; receive and reuse the G component output by the G component buffer FIFO, calculate the non-G component line by line; and output the RGB components of each line of image data;

[0015] The G component cache FIFO is used to receive the G component of any row calculated by the Demosaic module and output it to the corresponding G component register; the capacity of each G component cache FIFO is half of the capacity of any row cache FIFO;

[0016] The G component cache register is used to cache the G component of any row output by the G component cache FIFO and output it to the Demosaic module; the capacity of each G component cache register is half of the capacity of any row cache register.

[0017] Furthermore, the simultaneously outputting N lines of image data and any line of image data having at least five pixels comprises:

[0018] When T=1, the row data valid signal of any row cache register and the cached row image data are cached at four levels, so that when the valid signal is 1, in the row image data output in any clock cycle, any pixel point of the second-level cache of the row cache register has at least 2 adjacent pixels on the left and 2 adjacent pixels on the right.

[0019] Furthermore, the valid signal includes a row data valid signal, a row data valid signal of a first-level cache of a row buffer register, a row data valid signal of a second-level cache of a row buffer register, a row data valid signal of a third-level cache of a row buffer register, and a row data valid signal of a fourth-level cache of a row buffer register.

[0020] Furthermore, the simultaneously outputting N lines of image data and any line of image data having at least five pixels comprises:

[0021] T≥2, the row data valid signal of any row cache register and the cached row image data are cached in two levels, so that when the valid signal is 1, in the row image data output in any clock cycle, any pixel point of the first-level cache of the row cache register has at least 2 adjacent pixels on the left and 2 adjacent pixels on the right.

[0022] Furthermore, the valid signal includes a row data valid signal, a row data valid signal of a first level cache of a row buffer register, and a row data valid signal of a second level cache of a row buffer register.

[0023] Furthermore, if there are multiple G component buffer FIFOs, the G component buffer FIFOs are connected in series to simultaneously output the G component of any line of image data to the corresponding G component buffer register.

[0024] Furthermore, the Demosaic module also includes: based on the first two rows and the last two rows of the input Bayer data, expanding the Bayer data to obtain virtual Bayer data; based on the virtual Bayer data, outputting the RGB components corresponding to the first three rows and the last three rows of the Bayer data respectively.

[0025] The present application also proposes an image enhancement method, which is applicable to the above-mentioned system, comprising:

[0026] Receive input Bayer data line by line, read N lines of image data at the same time, and any line of image data has at least five pixels;

[0027] Based on N lines of image data, the color difference change value and interpolation result of each interpolation direction of any non-G pixel point are calculated in parallel line by line, and the G component interpolation direction and G component of the line are determined in the same clock cycle;

[0028] Based on N lines of image data and the G component of the previous line of image data, the chromatic aberration change value and the interpolation result in each interpolation direction of any non-G pixel point in any line of image data, and the interpolation result in each interpolation direction of any G pixel point are calculated in parallel line by line, and the missing component of the non-G pixel point and the non-G component of the G pixel point in the same clock cycle are determined.

[0029] Furthermore, it also includes: before determining, performing data overflow protection on the calculation result of the color difference change value and / or the interpolation result of each pixel point to obtain the color difference change value and / or the interpolation result.

[0030] The present application also proposes a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the image enhancement method as described above is implemented.

[0031] The beneficial effects of the present invention are as follows:

[0032] The image enhancement system disclosed in the present invention reasonably controls the reading and writing of the line cache FIFO and the register, so that each line of image data in each clock always outputs at least 5 pixels when transmitting any number of pixel data, so as to meet the calculation requirements of the demosaic algorithm of the present invention.

[0033] This application balancedly optimizes the number of combinational logic levels to be executed in each clock cycle and the total register occupancy. In the pipeline design of the Demosaic algorithm, the limited internal resources of the hardware and the timing convergence problem under high clock frequency are fully considered. The combinational logic level optimization and pipeline level optimization are combined to achieve better FPGA resource occupancy and realize efficient computing.

[0034] In order to save the number of pipeline stages, that is, to save the FPGA register resources consumed by the pipeline, and to reduce the overall Demosaic algorithm delay, in the implementation method of the present application, the G component interpolation direction judgment and G component calculation of non-G pixel points are calculated in parallel, so that in the fourth stage of the pipeline, both the judgment of the interpolation direction of the non-G pixel points is completed and the pixel value of the G component is output; similarly, the calculation of the non-G component of the G pixel points of the same row of pixel data and the calculation of the missing component of the non-G pixel points are calculated in parallel; the non-G components of the pixel points of the row are obtained at the same time in the same clock cycle, the register resources of the FPGA are reasonably used, and invalid pipeline stages are not introduced to achieve efficient calculation.

[0035] While balancing and optimizing the number of combinatorial logic levels to be executed in each clock cycle and the total register occupancy, the limited internal resources of the hardware are fully considered in the design of the pipeline. The calculation formula of the non-G component is split into four-stage pipeline step calculation to avoid too many combinatorial logic levels in a single clock cycle, which leads to the failure of timing convergence at high clock frequency. At the same time, it is also necessary to ensure that the pipeline levels of the calculation of the non-G components of the G pixels and the calculation of the missing components of the non-G pixels match, so as to minimize the invalid occupation of FPGA register resources.

[0036] At the same time, along with the input data flow of the Nth row, the R&B components of the G pixels and the B / R components of the R / B pixels of the N-3th row can be continuously output with a certain number of pipeline stages delayed.

[0037] The image enhancement method disclosed in the present invention receives the Bayer format image line by line, and performs subsequent de-mosaic processing and output while receiving the buffer, thereby providing a real-time and efficient implementation method based on FPGA.

[0038] In the Demosaic algorithm disclosed in the present invention, the interpolation calculation of the non-G component reuses a large number of intermediate calculation results of the G component, which reduces the demand for buffer space and reduces the calculation cost; and in the interpolation calculation of the G component of the non-G pixel point, a new calculation formula for the interpolation direction is adopted to avoid the situation where the gradient information of the edge changes greatly, resulting in obvious differences between the interpolated result and the original pixel point.

[0039] In addition, the present invention fully considers the problem that the top 3 rows and the bottom 3 rows of the image cannot implement the Demosaic algorithm through the pipeline design of the present application, and sets virtual data so that the present application can implement all RGB Demosaic calculations starting from the first row.

[0040] Based on the Demosaic algorithm proposed in the embodiment of the present invention, the FPGA implementation method only needs 4 row cache FIFOs and 2 G component cache FIFOs (each only needs the capacity of half a row of data), and a total of only FPGA storage resources with the capacity of 5 rows of pixel data, saving a large amount of hardware resources.

[0041] Based on the calculation strategy proposed in this application, the image enhancement process is finally realized and a high-quality color display image is obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a schematic diagram of an image enhancement system in Embodiment 1 of the present invention;

[0043] Figure 2 is a timing diagram of a row buffer register in Embodiment 1 of the present invention;

[0044] Figure 3 It is a schematic diagram of the process of the image enhancement method in the present invention;

[0045] Figure 4 yes Figure 2 Schematic diagram of pixel data acquired in any clock cycle;

[0046] Figure 5 is a schematic diagram of Bayer image data;

[0047] Figure 6 It is a schematic diagram of virtual data of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Example 1

[0049] like Figure 1 As shown, the present invention proposes an image enhancement system, comprising:

[0050] (N-1) row buffer FIFOs connected in series are used to receive input Bayer data row by row and output row image data to corresponding row buffer registers; wherein N≥3.

[0051] N row cache registers are used to cache the currently input row image data and the row image data output by the row cache FIFO; based on each clock cycle, any row cache FIFO and the currently input row image data transmit T pixels respectively, and the cache level is set to output N rows of image data at the same time and any row of image data is at least five pixels.

[0052] like Figure 1 As shown, in the present application, a row cache FIFO queue consisting of (N-1) row cache FIFOs connected in series is constructed to achieve the acquisition of multiple rows of Bayer data at the same time.

[0053] In this embodiment, four row buffer FIFOs are set in the system. Taking the simultaneous reading of five rows of Bayer data as an example, the reading process is as follows:

[0054] Receive the currently input Bayer data stream and cache the Bayer data line by line;

[0055] When the first line of image data is input, line buffer FIFO1 is written but not read; the remaining line buffer FIFOs are neither written nor read;

[0056] When the second line of image data is input, the line buffer FIFO1 writes the second line of image data and synchronously reads out the first line of image data; at the same time, the line buffer FIFO2 writes the first line of image data and does not read it out; the remaining line buffer FIFOs do not write or read out;

[0057] When the third line of image data is input, the line buffer FIFO1 writes the third line of image data and reads out the second line of image data at the same time; the line buffer FIFO2 writes the second line of image data and reads out the first line of image data at the same time; the line buffer FIFO3 writes the first line of image data and does not read it out; the remaining line buffer FIFOs do not write or read out;

[0058] Similarly, when the fifth line of image data is input, line cache FIFO1 writes the fifth line of image data and reads out the fourth line of image data to line cache register 1; line cache FIFO2 writes the fourth line of image data and reads out the third line of image data to line cache register 2; line cache FIFO3 writes the third line of image data and reads out the second line of image data to line cache register 3; line cache FIFO4 writes the second line of image data and reads out the first line of image data to line cache register 4.

[0059] The above-mentioned line buffer FIFOs are all provided with corresponding line buffer registers for storing each line of image data output by the line buffer FIFO. In addition, a line buffer register 0 for storing the current input line image data is also provided, so that 5 lines of image data can be read simultaneously.

[0060] In summary, when the Nth row of image data (N≥5) is input, row cache FIFO1 reads out the N-1th row of image data to row cache register 1, row cache FIFO2 reads out the N-2th row of image data to row cache register 2, row cache FIFO3 reads out the N-3th row of image data to row cache register 3, row cache FIFO4 reads out the N-4th row of image data to row cache register 4, and the currently input row image data is cached to row register 0, so that 5 rows of original Bayer data streams can be obtained at the same time.

[0061] The input Bayer data stream is generally input into the Demosaic module for several clock cycles per row, that is, T pixels are input per clock (T=1, T=2, T=4, T=32, etc.).

[0062] If the bit width of the data written to or read from any row buffer FIFO is the same as the bit width of the currently input row image data, then the input data stream and any row buffer FIFO output T pixels per clock cycle, respectively, where bit width = T×W, and W is the depth of each pixel.

[0063] Because in the Demosaic algorithm, for the calculation of the RGB components of each pixel, the two adjacent pixels on the left and the two adjacent pixels on the right are also required.

[0064] Therefore, for each row of data stream input by each clock, if T≤4, then for all pixels in each clock, using only the pixel data of this clock is not sufficient to implement the Demosaic algorithm; if T>4, then for the first two pixels and the last two pixels of each clock, using only the pixel data of this clock is not sufficient to implement the Demosaic algorithm; for the middle pixels except the first two pixels and the last two pixels in this row of image data, the Demosaic algorithm can be implemented by using only the pixel data of this clock.

[0065] In order to solve the problem that the pixel data of each clock is insufficient to implement the Demosaic algorithm, this application proposes the following solution:

[0066] When T=1, the row data valid signal of any row cache register and the cached row image data are cached at four levels, so that when the valid signal is 1, in the row image data output in any clock cycle, any pixel point of the second-level cache of the row cache register has at least 2 adjacent pixels on the left and 2 adjacent pixels on the right.

[0067] The valid signals include a row data valid signal, a row data valid signal of the first level cache of the row cache register, a row data valid signal of the second level cache of the row cache register, a row data valid signal of the third level cache of the row cache register, and a row data valid signal of the fourth level cache of the row cache register.

[0068] T≥2, the row data valid signal of any row cache register and the cached row image data are cached in two levels, so that when the valid signal is 1, in the row image data output in any clock cycle, any pixel point of the first-level cache of the row cache register has at least 2 adjacent pixels on the left and 2 adjacent pixels on the right.

[0069] The valid signals include a row data valid signal, a row data valid signal of a first level cache of a row buffer register, and a row data valid signal of a second level cache of a row buffer register.

[0070] This embodiment takes T≥2 as an example to illustrate the output control of the row buffer register.

[0071] like Figure 2 As shown, for any row buffer register's row data valid signal LVAL and cached row image data DATA, two levels of cache are used. LVAL_r and DATA_r are respectively the row data valid signal and row image data of the first level cache of the row buffer register, and LVAL_rr and DATA_rr are respectively the row data valid signal and row image data of the second level cache of the row buffer register. The first level of cache is represented by LVAL_r and DATA_r, both of which are delayed by one clock cycle compared to LVAL and DATA, and the second level of cache is represented by LVAL_rr and DATA_rr, both of which are delayed by two clock cycles compared to LVAL and DATA.

[0072] When the valid signal is 1, that is, the row data valid signal LVAL, the row data valid signal LVAL_r of the first level cache of the row buffer register, and the row data valid signal LVAL_rr of the second level cache of the row buffer register are all 1, which are all represented as rising edges in the figure.

[0073] When the valid signal is 1, it indicates that the data in the clock cycle is valid image data; if it is 0, it indicates that it is not valid image data.

[0074] The data DATA includes the currently input row image data and the row image data read out from each row buffer FIFO. In a cycle when LVAL is 1, the corresponding valid data DATA includes D0, D1, D2, and D3. Each row transmits T pixels, D0 is the pixel data of P(0)-P(T-1); D1 is the pixel data of P(T)-P(2T-1); and D2 is the pixel data of P(2T)-P(3T-1).

[0075] For the first pixel point P(T) in D1, its two left pixel values ​​P(T-1) and P(T-2) can be obtained from D0, and for the last pixel P(2T-1), its two right pixel values ​​P(2T) and P(2T+1) can be obtained from D2. At this time, any pixel data in DATA_r has two adjacent left pixel points and two adjacent right pixel points, which meets the calculation basis of the Demosaic algorithm.

[0076] For the two leftmost pixels P(0) and P(1) in D0, the leftmost pixel P(0) is used as the two left pixels of P(0) and P(1) by default; for the two rightmost pixels of D2 in the current clock cycle, no default processing is performed. Since the input of row image data is processed continuously, in the next clock cycle, the two right pixels of D2 can obtain the two adjacent right pixel values ​​in D3. If D3 is the last transmission unit of the row image data, for the two rightmost pixels of D3 P(4T-2) and P(4T-1), P(4T-1) is used as the two rightmost pixel values ​​of P(4T-2) and P(4T-1) by default.

[0077] Preferably, the two leftmost pixels and the two rightmost pixels of any row of image data may be discarded, and their RGB components are not calculated.

[0078] Through the above-mentioned setting of the row buffer FIFO queue and the row buffer register, at least 5 rows of image data can be obtained simultaneously, and when the valid signal is 1, at least 5 pixels are output for any row of image data in each clock cycle.

[0079] The row buffer register outputs at least 5 rows of image data to the Demosaic module simultaneously.

[0080] The Demosaic module is used to receive the image data output by the line buffer register and calculate the G component line by line; receive and multiplex the G component output by the G component buffer FIFO and calculate the non-G component line by line; and output the RGB components of each line of image data.

[0081] The demosaic module of this embodiment may use the Hamilton interpolation algorithm described in patent CN 109104595 B to calculate the RGB components of the pixel points; other interpolation algorithms may also be used to calculate the RGB components.

[0082] This embodiment also proposes a simplified Demosaic algorithm, which specifically includes:

[0083] Calculate the G component interpolation direction and G component, including:

[0084] Based on any non-G pixel point and its two adjacent left pixels and two left pixels, calculate the color difference change value and interpolation result in the horizontal direction; based on any non-G pixel point and its two adjacent upper pixels and two lower pixels, calculate the color difference change value and interpolation result in the vertical direction; compare the color difference change value in the horizontal direction with the color difference change value in the vertical direction, take the interpolation direction with the smaller color difference change value as the interpolation direction of the G component, and assign the corresponding interpolation result to the G component.

[0085] Calculate the missing components of non-G pixels, including:

[0086] Using any non-G pixel and its adjacent upper left pixel and lower left pixel, calculate the color difference change value and interpolation result in the vertical direction; using any non-G pixel and its adjacent upper left pixel and upper right pixel, calculate the color difference change value and interpolation result in the horizontal direction; compare the color difference change value in the horizontal direction with the color difference change value in the vertical direction, take the interpolation direction with the smaller color difference change value as the interpolation direction of the missing component, and assign the corresponding interpolation result as the missing component.

[0087] Calculate the non-G component of the G pixel, including:

[0088] Using any G pixel point and its two adjacent upper and lower pixels, calculate the interpolation result of the first non-G component of the G pixel point; using any G pixel point and its two adjacent left and right pixels, calculate the interpolation result of the second non-G component of the G pixel point; based on the arrangement of the Bayer array, determine the first non-G component and the second non-G component.

[0089] Since the calculation of non-G components depends on the calculation result of G components, in order to save FPGA resources, in this embodiment, the calculation of non-G components is arranged after the calculation of G components and is arranged to be parallel calculated to improve the calculation efficiency.

[0090] In order to further realize the multiplexing of the G component, the system is also provided with a G component buffer FIFO and a G component buffer register for storing the G component of the previous line of image data to facilitate the calculation of the RGB component of the current line.

[0091] The G component cache FIFO is used to receive the G component of any row calculated by the Demosaic module and output it to the corresponding G component register; the capacity of each G component cache FIFO is half of the capacity of any row cache FIFO;

[0092] The G component cache register is used to cache the G component of any row output by the G component cache FIFO and output it to the Demosaic module; the capacity of each G component cache register is half of the capacity of any row cache register.

[0093] In this embodiment, two G component buffer FIFOs and a G component buffer register are provided, and the two G component buffer FIFOs are connected in series.

[0094] If there are multiple G component buffer FIFOs, the G component buffer FIFOs are connected in series to simultaneously output the G component of any line of image data to the corresponding G component buffer register.

[0095] The Demosaic module outputs the G component of the (N-2)th row of image data. The G component cache FIFO1 performs writing but does not read out. The G component cache FIFO2 does not write or read out.

[0096] The Demosaic module outputs the G component of the (N-3)th row of image data, and the G component cache FIFO1 writes the G component of the (N-3)th row of image data, and reads the G component of the (N-2)th row of image data to the Demosaic module at the same time; the G component cache FIFO2 writes the G component of the (N-2)th row of image data and does not read it out;

[0097] The Demosaic module outputs the G component of the (N-4)th row of image data, the G component cache FIFO1 writes the G component of the (N-4)th row of image data, and simultaneously reads the G component of the (N-3)th row of image data to the Demosaic module; the G component cache FIFO2 writes the G component of the (N-3)th row of image data, and reads the G component of the (N-2)th row of image data to the Demosaic module.

[0098] In summary, the Demosaic module can obtain the G component of two different lines of image data output by the G component cache register in the same clock cycle, and the G component of one line of image data currently calculated, that is, the G component of the (N-2)th line to the (N-4)th line, and then start calculating the R&B component calculation of the (N-3)th line.

[0099] The Demosaic module also includes: based on the first two rows and the last two rows of the input Bayer data, expanding the Bayer data to obtain virtual Bayer data; based on the virtual Bayer data, the RGB components corresponding to the first three rows and the last three rows of the Bayer data can be output.

[0100] like Figure 3 As shown, based on the above FPGA implementation system, this embodiment proposes an image enhancement method, which is as follows:

[0101] Receive input Bayer data line by line, read N lines of image data at the same time, and any line of image data has at least five pixels;

[0102] Based on the settings of the row cache FIFO and the row cache register in this embodiment, when the 5th row of image data is input, the Demosaic module can read 5 rows of image data at the same time, and the output pixel data of any row of image data is at least 5, that is, the Demosaic module can obtain at least 5×5 pixel data.

[0103] When the fifth line of image data is input, the line buffer FIFO1 writes the fifth line of image data and reads the fourth line of image data to the line buffer register 1; the line buffer FIFO2 writes the fourth line of image data and reads the third line of image data to the line buffer register 2; the line buffer FIFO3 writes the third line of image data and reads the second line of image data to the line buffer register 3; the line buffer FIFO4 writes the second line of image data and reads the first line of image data to the line buffer register 4. The current input line image data (fifth line of image data) is stored in the line buffer register 0, so that 5 lines of image data can be read at the same time.

[0104] This embodiment takes T=4 as an example, that is, 4 pixel data are transmitted per row of image data per clock cycle.

[0105] The row data valid signal LVAL of any row buffer register and the cached row image data DATA are cached at two levels. Figure 4 Taking the output data shown as an example, in this clock cycle, D0 is obtained at DATA_rr, D1 is obtained at DATA_r, and D2 is obtained at DATA. D0 represents P(0)-P(3), D1 represents P(4)-P(7), and D2 represents P(8)-P(11).

[0106] The first pixel P(4) in D1 can obtain its two left pixel values ​​P(2) and P(3) from D0, and the last pixel P(7) can obtain its two right pixel values ​​P(8) and P(9) from D2. At this time, any pixel data in DATA_r has two adjacent left pixel points and two adjacent right pixel points, which meets the calculation basis of the Demosaic algorithm.

[0107] For the two leftmost pixels in each row, that is, P(0) and P(1) in D0. If P(0) does not have two pixels on its left, P(0) is assumed to be the value of the two pixels on its left. For P(1), P(0) is the value of the first pixel on its left, and P(0) is assumed to be the value of the second pixel on its left. Similarly, for the two rightmost pixels, that is, P(10) and P(11) in D2, P(11) is assumed to be the two pixels on the right of P(10) and P(11).

[0108] At this time, any pixel point in the pixel data D0, D1, and D2 acquired in this clock cycle has two adjacent pixels on the left and two adjacent pixels on the right, which meets the calculation basis of the Demosaic algorithm.

[0109] This embodiment takes T=2, and the demosaic module obtains 5×6 pixel data in each clock cycle as an example to illustrate the calculation process of RGB components.

[0110] Based on the storage settings of the row buffer FIFO and row buffer register in the present application, for the case of T≥2, when the valid signal of any row buffer register is 1, it continuously transmits pixel data for four clock cycles and outputs image data of a row with at least 5 pixels. Considering the timing of the row image data input, this embodiment uses four clock cycles as the cycle of G component calculation, so that the Demosaic module obtains pixel data and maintains continuity with the G component calculation, thereby improving the system operation efficiency.

[0111] In this embodiment, the simplified Demosaic algorithm is taken as an example to calculate the RGB components.

[0112] Based on N lines of image data, starting from the (N-2)th line, the color difference change value and interpolation result of each interpolation direction of any non-G pixel point are calculated in parallel, and the G component interpolation direction and G component of the line are determined in the same clock cycle. Before determination, the calculation results of the color difference change value and / or interpolation result of each pixel point are protected from data overflow to obtain the color difference change value and / or interpolation result.

[0113] like Figure 5 As shown, the R pixel point in the (N-2)th row is taken as an example to illustrate the process of determining the G component.

[0114] Based on the above simplified Demosaic algorithm, the process of determining the interpolation direction of the G component is to decompose the color difference change value calculation formula of each interpolation direction into a three-stage pipeline, each stage using only one operation to obtain the color difference change value of each interpolation direction; the fourth stage pipeline compares the color difference change value of each interpolation direction.

[0115] Taking the vertical color difference change value Dv and the horizontal color difference change value Dh of R(N-2,2) as examples, the idea of ​​the corresponding color difference change value calculation formula is explained:

[0116] The first color difference Cv1 in the vertical direction of R(N-2,2) R(N-2,2) =[G(N-3,2)+G(N-1,2)] / 2-R(N-2,2)

[0117] The color difference Cv between the vertically adjacent G pixel point G(N-3,2) of R(N-2,2) G(N-3,2)、Cv of G(N-1,2) G(N-1,2) They are:

[0118] Cv G(N-3,2) =G(N-3,2)-[R(N-4,2)+R(N-2,2)] / 2

[0119] Cv G(N-1,2) =G(N-1,2)-[R(N,2)+R(N-2,2)] / 2

[0120] Then the vertical color difference change value Dv of R(N-2,2)=|Cv1 R(N-2,2) -Cv G(N-3,2) |+|Cv G(N-1,2) -Cv1 R(N-2,2) |, simplifying it, we get:

[0121] Dv=|-R(N-4,2)+G(N-3,2)+R(N-2,2)-G(N-1,2)|+|-G(N-3,2)+R(N-2,2)+G(N-1,2)-R(N,2)|.

[0122] Similarly, calculate the first color difference Ch1 in the horizontal direction of R(N-2,2) R(N-2,2) , and the color difference Ch of its horizontally adjacent G pixel point G(N-2,1) G(N-2,1) 、G(N-2,3) color difference Ch G(N-2,3) ;

[0123] Then the horizontal color difference change value of R(N-2,2) Dh=|Ch1 R(N-2,2) -Ch G(N-2,1) |+|Ch G(N-2,3) -Ch1 R(N-2,2) |, simplifying it, we get:

[0124] Dh=|-R(N-2,0)+G(N-2,1)+R(N-2,2)-G(N-2,3)|+|-G(N-2,1)+R(N-2,2)+G(N-2,3)-R(N-2,4)|.

[0125] The calculation process of the G component is as follows: the interpolation result calculation formula of each interpolation direction is decomposed into two-stage pipelines, each stage uses only one operation, and the same operation method exists in the color difference change value calculation formula, and is placed in the same stage of the pipeline to obtain the G component difference of each interpolation direction; the third stage of the pipeline performs overflow protection on the G component difference of each interpolation direction respectively to obtain the interpolation result of each interpolation direction; the fourth stage of the pipeline selects the interpolation direction corresponding to the smaller color difference change value based on the comparison result of the color difference change value of each interpolation direction, and assigns the G component of the non-G pixel point to the corresponding interpolation result; wherein, the interpolation direction includes the horizontal direction and the vertical direction.

[0126] Taking the vertical interpolation result Gv and the horizontal interpolation result Gh of R(N-2,2) as examples, the calculation ideas of the calculation formula are explained:

[0127] The second color difference Cv2 in the vertical direction of R(N-2,2) R(N-2,2) =[Cv G(N-3,2) +Cv G(N-1,2) ] / 2

[0128] R(N-2,2) vertical color difference Cv R(N-2,2) =(Cv1 R(N-2,2) +Cv2 R(N-2,2) ) / 2

[0129] Gv=R(N-2,2)+Cv R(N-2,2) , we can simplify it to get:

[0130] Gv=(-R(N-4,2)+4G(N-3,2)+2R(N-2,2)+4G(N-1,2)-R(N,2)) / 8.

[0131] Similarly, calculate the second color difference Ch2 in the horizontal direction of R(N-2,2) R(N-2,2) =(Ch G(N-2,1) +Ch G(N-2,3) ) / 2;

[0132] R(N-2,2) horizontal color difference Ch R(N-2,2) =(Ch1 R(N-2,2) +Ch2 R(N-2,2) ) / 2

[0133] Gh=R(N-2,2)+Ch R(N-2,2) , we can simplify it to get:

[0134] Gh=(-R(N-2,0)+4G(N-2,1)+2R(N-2,2)+4G(N-2,3)-R(N-2,4)) / 8.

[0135] In summary, the interpolation direction judgment of the G component at R(N-2,2) and the calculation formula of the G component are as follows:

[0136] Dv=|-R(N-4,2)+G(N-3,2)+R(N-2,2)-G(N-1,2)|+|-G(N-3,2)+R(N-2,2)+G(N-1,2)-R(N,2)|

[0137] Dh=|-R(N-2,0)+G(N-2,1)+R(N-2,2)-G(N-2,3)|+|-G(N-2,1)+R(N-2,2)+G(N-2,3)-R(N-2,4)|

[0138] Gv=(-R(N-4,2)+4G(N-3,2)+2R(N-2,2)+4G(N-1,2)-R(N,2)) / 8

[0139] Gh=(-R(N-2,0)+4G(N-2,1)+2R(N-2,2)+4G(N-2,3)-R(N-2,4)) / 8

[0140] If(Dv≥Dh) Ge=Gh else Ge=Gv

[0141] Among them, Dv is the color difference change value of R(N-2,2) in the vertical direction, that is, the color difference change value of the 2nd column from the N-4th row to the Nth row along the vertical direction; Dh is the color difference change value of R(N-2,2) in the horizontal direction, that is, the RG color difference change value of the N-2th row from the 0th column to the 4th column along the horizontal direction; Gv is the vertical interpolation result of the G component of R(N-2,2), and Gh is the horizontal interpolation result of the G component of R(N-2,2).

[0142] In the above formula calculation, operations such as division by 8, multiplication by 4, and multiplication by 2 can be implemented in a shifting manner. However, the large number of additions, subtractions, and absolute value calculations involved require a large number of combinatorial logic levels when the pixel width W (the default T=1, in which case the depth W of a single pixel is used as the pixel width) is large. For example, due to limited DSP resources, LUT resources are used in the FPGA to implement operations such as addition and subtraction. In this case, it is difficult to achieve timing convergence if the Demosaic algorithm is to be implemented at a higher clock frequency. Therefore, the above formula can be split into multiple pipeline steps for calculation.

[0143] When using pipeline calculation, the increase in the number of pipeline stages will lead to an increase in the number of FPGA register resources used. In addition, since the judgment of the interpolation direction and the interpolation result of the G component jointly determine the final G component pixel value calculation result, it is best to design a pipeline solution to allow the two calculations to be performed in parallel and complete the calculations at the same level of the pipeline to output the final result without introducing additional invalid pipeline stages.

[0144] In summary, in this embodiment, the pipeline design proposed for G component calculation is:

[0145] In the first-stage pipeline, according to the interpolation direction of the G component and the calculation formula of the G component, an adder is used to calculate the sum in the calculation formula, as follows:

[0146] Dv11=G(N-3,2)+R(N-2,2); Dv12=R(N-4,2)+G(N-1,2);

[0147] Dv13=R(N-2,2)+G(N-1,2); Dv14=G(N-3,2)+R(N,2);

[0148] Dh11=G(N-2,1)+R(N-2,2); Dh12=R(N-2,0)+G(N-2,3);

[0149] Dh13=R(N-2,2)+G(N-2,3); Dh14=G(N-2,1)+R(N-2,4);

[0150] Gv11=2G(N-3,2)+R(N-2,2)+2G(N-1,2); Gv12=R(N-4,2)+R(N,2);

[0151] Gh11=2G(N-2,1)+R(N-2,2)+2G(N-2,3); Gh12=R(N-2,0)+R(N-2,4).

[0152] In the second-stage pipeline, according to the interpolation direction of the G component and the calculation formula of the G component, a subtractor is used to calculate the difference in the calculation formula; and the absolute value of the difference in the judgment of the interpolation direction of the G component is taken, as follows:

[0153] Dv21=|Dv11-Dv12|=|-R(N-4,2)+G(N-3,2)+R(N-2,2)-G(N-1,2)|

[0154] Dv22=|Dv13-Dv14|=|-G(N-3,2)+R(N-2,2)+G(N-1,2)-R(N,2)|

[0155] Dh21=|Dh11-Dh12|=|-R(N-2,0)+G(N-2,1)+R(N-2,2)-G(N-2,3)|

[0156] Dh22=|Dh13-Dh14|=|-G(N-2,1)+R(N-2,2)+G(N-2,3)-R(N-2,4)|

[0157] Gv2=2Gv11-Gv12=(-R(N-4,2)+4G(N-3,2)+2R(N-2,2)+4G(N-1,2)-R(N,2)

[0158] Gh2=2Gh11-Gh12=(-R(N-2,0)+4G(N-2,1)+2R(N-2,2)+4G(N-2,3)-R(N-2,4))

[0159] In the third-level pipeline, the calculation of the G component interpolation direction is to calculate the sum of the two absolute values ​​Dh21 and Dh22 in the horizontal direction as the color difference change value Dh3=Dh21+Dh22 in the horizontal direction; calculate the sum of the two absolute values ​​Dv21 and Dv22 in the vertical direction as the color difference change value Dv3=Dv21+Dv22 in the vertical direction;

[0160] In the third-stage pipeline, the G component is calculated by performing overflow protection on Gv2 and Gh2, assigning values ​​and rounding them off to obtain the interpolation result Gh3 in the horizontal direction and the interpolation result Gv3 in the vertical direction;

[0161] Taking Gv2 as an example, when the [W+4] bit of Gv2 is high, it means that Gv2 is a negative number, and Gv3 is assigned all 0s; when the [W+3] bit of Gv2 is high, it means that Gv2 exceeds the full value of the pixel, and Gv3 is assigned {W{1'b1}} (full value); when [W+2:3] of Gv2 is all high, Gv3 is assigned {W{1'b1}} (full value); in other cases, Gv3 is assigned Gv2[W+2:3]+Gv2[2] (rounding off the decimal part).

[0162] {W{1'b1}} means concatenating W 1s. "1'b1" represents a single bit with a binary value of 1, which means a "true" or "high" logical value. "1" represents a binary value, and "b'" represents the base of the binary value.

[0163] In the fourth stage of the pipeline, Dv3 and Dh3 are compared; if Dv3 ≥ Dh3, Ge is assigned the value of Gh3, otherwise, Ge is assigned the value of Gv3.

[0164] According to the above four-stage pipeline, the G component calculation of the R / B pixel can be completed while balancing and optimizing the number of combinational logic levels to be executed in each clock cycle and the total register occupancy, and the G component data stream of the R / B pixel of the N-2th row can be continuously output with the input data stream of the Nth row while delaying a certain number of pipeline stages.

[0165] The calculation of the RGB components of the next row of image data depends on the G component of the previous row of image data. The G component of the (N-2)th row of image data is used as the data basis for calculating the non-G components of the (N-3)th row of image data.

[0166] Based on N lines of image data and the G component of the previous line of image data, starting from the (N-3)th line, the chromatic aberration change value and the interpolation result in each interpolation direction of any non-G pixel point in any line of image data, and the interpolation result in each interpolation direction of any G pixel point are calculated in parallel, and the missing components of the non-G pixel points and the non-G components of the G pixel points in the same clock cycle are determined.

[0167] like Figure 5 As shown, the pixel points in the (N-3)th row are taken as an example to illustrate the process of determining the non-G component.

[0168] Since the calculation of the non-G components of the G pixels and non-G pixels in the (N-3)th row of image data depends on the G components in the (N-2)th row, and the calculation of the non-G components of the adjacent G pixels and B pixels in the (N-3)th row requires the use of the same pixel color difference, in order to save FPGA resources, the calculation of the non-G components of the G pixels and non-G pixels in the (N-3)th row of image data are processed in parallel.

[0169] Preferably, if hardware resources are limited, parallel calculation of G pixels and non-G components of non-G pixels may not be designed. The calculation of G components of G pixels and non-G pixels does not affect each other, and the calculation order can be set according to actual needs.

[0170] In this embodiment, in order to realize parallel processing of the calculation of the G component of the (N-4)th row and the R&B component of the (N-3)th row, the calculation of the non-G component is also designed as a four-stage pipeline.

[0171] Due to the limited internal resources of the hardware, this embodiment splits the calculation formula of the G component into multiple pipeline steps to avoid too many combinational logic levels in a single clock cycle, which leads to the failure of timing convergence at high clock frequency. At the same time, it is also necessary to ensure that the pipeline levels of the calculation of the non-G components of the G pixel points and the calculation of the missing components of the non-G pixel points match, so as to minimize the ineffective occupation of FPGA register resources.

[0172] The pipeline design of the non-G component of the G pixel is to decompose the interpolation result calculation formula of each interpolation direction into three-stage pipeline, each stage uses only one operation to obtain the interpolation result of each interpolation direction; the fourth stage pipeline performs overflow protection assignment and rounding on the interpolation result of each interpolation direction to obtain the non-G component corresponding to each interpolation direction.

[0173] Taking the calculation of the non-G component of G(N-3,2) as an example, the calculation formulas for its R component Re and B component Be are:

[0174] Re=(R(N-4,2)-Ge(N-4,2)+R(N-2,2)-Ge(N-2,2)) / 2+G(N-3,2)

[0175] Be=(B(N-3,1)-Ge(N-3,1)+B(N-3,3)-Ge(N-3,3)) / 2+G(N-3,2)

[0176] The pipeline design for the missing components of non-G pixels is to decompose the interpolation result calculation formula and the color difference change value calculation formula for each interpolation direction into two-stage pipeline operations, so that the third-stage pipeline obtains the interpolation result and the color difference change value of each interpolation direction; the fourth-stage pipeline compares the color difference change value of each interpolation direction; the sum of each interpolation direction is assigned with overflow protection and rounded to obtain the missing component of each interpolation direction; the interpolation direction corresponding to the smaller color difference change value is selected, and the missing component of the non-G pixel is assigned to the missing component of the corresponding interpolation direction.

[0177] Taking the determination of the interpolation direction and calculation of the R component of B(N-3,3) as an example, the details are as follows:

[0178] The upper left non-G pixel adjacent to B(N-3,3) is R(N-4,2), the lower left non-G pixel is R(N-2,2), and the upper right non-G pixel is R(N-4,4).

[0179] The absolute value of the difference between the color difference of R(N-4,2) and the color difference of R(N-2,2) is used as the missing component color difference change value (RG color difference change value) in the vertical direction of B(N-3,3), and the absolute value of the difference between the color difference of R(N-4,2) and the color difference of R(N-4,4) is used as the RG color difference change value in the horizontal direction of B(N-3,3); the interpolation direction of the R component of B(N-3,3) is determined by comparing the RG color difference change values ​​in the horizontal direction and the vertical direction.

[0180] The average of the color difference of R(N-4,2) and the color difference of R(N-2,2) is used as the missing component color difference (RG color difference) in the vertical direction of B(N-3,3), and the average of the color difference of R(N-4,2) and the color difference of R(N-4,4) is used as the RG color difference in the horizontal direction of B(N-3,3); based on the color difference constancy criterion, the horizontal R component and the vertical R component of B(N-3,3) are calculated using the RG color difference in each direction of B(N-3,3) and the G component of B(N-3,3); based on the judgment result of the R component interpolation direction, the final R component of B(N-3,3) is determined.

[0181] Then the interpolation direction of the R component of B(N-3,3) is determined and the calculation formula of the R component is as follows:

[0182] CDv=|Ge(N-4,2)-R(N-4,2)-Ge(N-2,2)+R(N-2,2)|

[0183] CDh=|Ge(N-4,2)-R(N-4,2)-Ge(N-4,4)+R(N-4,4)|

[0184] RBv=(R(N-4,2)-Ge(N-4,2)+R(N-4,4)-Ge(N-4,4)) / 2+Ge(N-3,3)

[0185] RBh=(R(N-4,2)-Ge(N-4,2)+R(N-2,2)-Ge(N-2,2)) / 2+Ge(N-3,3)

[0186] If(CDh≥CDv) RBe=RBv else RBe=RBh

[0187] Among them, CDv is the RG color difference change value of B(N-3,3) in the vertical direction, CDh is the RG color difference change value of B(N-3,3) in the horizontal direction, RBv is the horizontal R component of B(N-3,3); RBv is the vertical R component of B(N-3,3).

[0188] The direction with the smallest RG color difference change value is used as the interpolation direction of the R component.

[0189] When CDh≥CDv, the vertical direction is used as the interpolation direction of the R component, that is, the vertical direction R component RBv is used as the final R component pixel value RBe; otherwise, the horizontal direction is used as the interpolation direction of the R component, that is, the horizontal direction interpolation result RBh is used as the final R component pixel value RBe.

[0190] The interpolation direction of the R component of B(N-3,3) and the calculation formula of the R component can also be adjusted as follows:

[0191] CDv=|(R(N-4,2)-Ge(N-4,2))-(R(N-2,2)-Ge(N-2,2))|=|R51-R52|

[0192] CDh=|(R(N-4,2)-Ge(N-4,2))-(R(N-4,4)-Ge(N-4,4))|=|R51-(R(N-4,4)-Ge(N-4,4))|

[0193] RBv=(R51+(R(N-4,4)-Ge(N-4,4))) / 2+Ge(N-3,3)

[0194] RBh=(R51+R52) / 2+Ge(N-3,3)

[0195] The purpose of adjusting the above formula is to reasonably use the intermediate calculation results R51 and R52 of the G pixel point to calculate the non-G component to calculate CDv, CDh, RBv, RBh, and the intermediate calculation result R(N-4,4)-Ge(N-4,4) of CDh to calculate RBv, so as to save FPGA resources by reusing the intermediate calculation results.

[0196] In summary, the four-stage pipeline design proposed for non-G component calculation in the Demosaic algorithm of this application is:

[0197] In the first-level pipeline, the calculation of the non-G component of the G pixel is to calculate the color difference of the non-G pixel adjacent to G(N-3,2), as follows:

[0198] The color difference of B(N-3,1) is B51=B(N-3,1)-Ge(N-3,1)

[0199] The color difference of B(N-3,3) is B52=B(N-3,3)-Ge(N-3,3)

[0200] The color difference of R(N-4,2) is R51=R(N-4,2)-Ge(N-4,2)

[0201] The color difference of R(N-2,2) is R52=R(N-2,2)-Ge(N-2,2)

[0202] Among them, since G(N-3,2) is not the first G pixel point in the current row, its previous G(N-3,0) has completed the calculation of the non-G component. Therefore, B51 of G(N-3,2) can reuse the calculation result of B52 in the non-G component calculated by G(N-3,0) to more fully realize resource reuse.

[0203] In the first-stage pipeline, the missing components of non-G pixels are calculated by calculating the color difference of the non-G pixels adjacent to the upper left, upper right, and lower left of B(N-3,3), as follows: CDh5=R(N-4,4)-Ge(N-4,4);

[0204] Since the calculation of the non-G component of the G pixel and the missing component of the non-G pixel is a parallel calculation, the color difference of the same non-G pixel is calculated only once in the first stage of the pipeline.

[0205] In the second-stage pipeline, the non-G component of the G pixel is calculated by calculating the sum of the color difference in the vertical direction R6=R51+R52; calculating the sum of the color difference in the horizontal direction B6=B51+B52;

[0206] In the second-stage pipeline, the missing component of the non-G pixel is calculated by calculating the difference between the color difference of the adjacent upper left pixel and the color difference of the upper right and lower left non-G pixels, as follows:

[0207] CDv6=R51-R52; CDh6=R51-CDh5

[0208] Calculate the sum of the color differences in the horizontal direction RBh6=R51+CDh5, and the sum of the color differences in the vertical direction R6;

[0209] Similarly, since the calculation of the non-G component of the G pixel and the missing component of the non-G pixel is a parallel calculation, the sum of the color difference of the same non-G pixel is calculated only once in the second stage of the pipeline.

[0210] In the third-level pipeline, the calculation of the non-G component of the G pixel is to calculate the interpolation result R7=(R6+2G(N-3,2)) in the vertical direction; and calculate the interpolation result B7=(B6+2G(N-3,2)) in the horizontal direction.

[0211] In the third-stage pipeline, the missing components of non-G pixels are calculated as follows:

[0212] Calculate the color difference change value CDv7=|CDv6| in the horizontal direction; the color difference change value CDh7=|CDh6| in the vertical direction;

[0213] The interpolation result in the horizontal direction is calculated as RBh7=RBh6+2Ge(N-3,3); the interpolation result in the vertical direction is calculated as RBv7=R6+2Ge(N-3,3).

[0214] In the fourth-stage pipeline, the calculation of the non-G component of the G pixel is to perform overflow protection, assignment and rounding on R7 and G7 respectively to obtain the R component Re and B component Be of G(N-3,2);

[0215] Both R7 and B7 registers need a width of W+3 bits to ensure that data overflow does not occur. At this time, the highest bit ([W+2] bit) is the negative overflow bit (sign bit), [W+1] bit is the positive overflow bit, [W:1] is the valid data bit, and [0] is the decimal place (because the final calculation result needs to be divided by 2);

[0216] Taking R7 as an example, the overflow protection value assignment judgment basis is that when the [W+2] bit of R7 is high, it means that R7 is a negative number, and Re is assigned to all 0s; when the [W+1] bit of R7 is high, it means that R7 exceeds the full value of the pixel, and Re is assigned to {W{1'b1}} (full value); when [W:1] of R7 is all high, Re is assigned to {W{1'b1}} (full value); in other cases, Re is assigned to R7[W:1]+R7[0] (rounding off the decimal part). The assignment process of Be is similar.

[0217] Preferably, in the above example content, the Nth row is the RG row. If the Nth row is the GB row, then the N-3th row is the RG row, and at this time, the R&B component calculation formula changes, Re becomes the horizontal interpolation, and Be becomes the vertical interpolation. At this time, there is no need to build a set of combinational logic for calculation, and only the assignment relationship between Re and Be in the fourth stage of the pipeline for calculating the non-G component of the G pixel point needs to be swapped, and the R&B component calculation of the G pixel point of the N-3th row of the RG row type can be completed under the condition of using the same set of combinational logic.

[0218] In the fourth stage of the pipeline, overflow protection, assignment and rounding of RBv7 and RBh7, and determination of the interpolation direction are combined into one step, as follows:

[0219] Overflow protection, value assignment and rounding are performed on RBv7 and RBh7 respectively. Both require a register width of W+3 to ensure no overflow. The highest bit ([W+2] bit) is the negative overflow bit (sign bit), [W+1] bit is the positive overflow bit, [W:1] is the valid data bit, and [0] is the decimal place (because the final calculation result needs to be divided by 2).

[0220] When CDh7≥CDv7, RBe is assigned RBv7, as follows:

[0221] When the [W+2] bit of RBv7 is high, it means that RBv7 is a negative number, and RBe is assigned to all 0s; when the [W+1] bit of RBv7 is high, it means that RBv7 exceeds the full value of the pixel, and RBe is assigned to {W{1'b1}} (full value); when [W:1] of RBv7 is all high, RBe is assigned to {W{1'b1}} (full value); in other cases, RBe is assigned to RBv7[W:1]+RBv7[0] (rounding off the decimal part).

[0222] When CDh7 < CDv7, RBe is assigned the value of RBh7, specifically as follows:

[0223] When the [W + 2] bit of RBh7 is high, it indicates that RBh7 is negative, and RBe is assigned all 0s; when the [W + 1] bit of RBh7 is high, it indicates that RBh7 exceeds the pixel full value, and RBe is assigned {W{1’b1}} (full value); when the [W:1] of RBh7 is all high, RBe is assigned {W{1’b1}} (full value); in other cases, RBe is assigned RBh7[W:1] + RBh7[0] (rounding the fractional part).

[0224] According to the above four-stage pipeline for non-G components, while evenly optimizing the number of combinational logic levels to be executed in each clock cycle and the total register occupancy, the pixel values of the R and B components of the G pixel points (i.e., the non-G components of the G pixel points) and the pixel values of the B / R components of the R / B pixel points (i.e., the missing components of the non-G pixel points) are calculated. It can be achieved that with the input data stream of the Nth row, and with a certain pipeline stage delay, the R&B component data stream of the G pixel points in the (N - 3)th row and the B / R component data stream of the R / B pixel points can be continuously output.

[0225] After the implementation of the Demosaic algorithm through the above G component calculation and non-G component calculation, when the data stream of the Nth row is input and passes through an eight-stage pipeline, the entire RGB component data stream of the (N - 3)th row can be obtained.

[0226] Preferably, for different Bayer formats, the Bayer image data in non-RGGB arrangement can be converted to the RGGB format after being offset horizontally or vertically.

[0227] In addition, the above content only solves the G component calculation starting from the (N - 2)th row and the R&B component calculation starting from the (N - 3)th row.

[0228] When the input original Bayer image data is the last row N, only the entire RGB calculation results of the (N - 3)th row can be output, and the three rows from the (N - 2)th row to the Nth row cannot obtain all the calculation results for output. For the top 3 rows and the bottom 3 rows of the image, the Demosaic algorithm cannot be implemented through the above pipeline design.

[0229] Preferably, the image data of the top 3 rows and the bottom 3 rows can be discarded without performing demosaicing calculation on them.

[0230] Preferably, before the Demosaic module calculates the G component, the Bayer data can also be extended to obtain virtual data to solve the problem that the RGB components of the top 3 rows and the bottom 3 rows of the image cannot be calculated through the above pipeline.

[0231] The image enhancement system proposed in this embodiment only requires 4 line cache FIFOs and 2 G component cache FIFOs (each of which only requires the capacity of half a line of data), and only occupies FPGA storage resources with the capacity of 5 lines of pixel data in total. In addition, full consideration has been given to the optimization of the number of combinational logic levels and the number of pipeline levels to achieve better occupancy of various FPGA resources.

[0232] The FPGA-based de-mosaicing method proposed in patent CN 118158551 B requires 6 row cache FIFOs, 2 R / B pixel G component cache FIFOs (each requiring only half a row of data capacity), and 2 G pixel R / B component cache FIFOs (each requiring only one row of data capacity), for a total of FPGA storage resources with a capacity of 9 rows of pixel data.

[0233] In addition, if you want to save computing units for some intermediate results, you also need to consume additional storage resources for caching. The judgment result of the interpolation direction (although only 1 bit wide) also needs to be cached to occupy storage resources. Compared with the technical solution described in patent CN 118158551 B, this application can save more than 4 / 9 of storage resources. Example 2

[0234] Based on the content of Example 1, this embodiment proposes the following content for 3≤N<5:

[0235] In this embodiment, only two line buffers FIGIO, three line buffer registers, one G component buffer FIFO and one G buffer register are used to simultaneously acquire three lines of image data for implementing the demosaicing process.

[0236] The difference is that the Demosaic module also includes: based on the first two rows and the last two rows of the input Bayer data, expanding the Bayer data to obtain virtual Bayer data; based on the virtual Bayer data, the RGB components corresponding to the first three rows and the last three rows of the Bayer data can be output.

[0237] like Figure 6As shown, when the 3rd row arrives, the 2nd row of the input original Bayer image data is used as the virtual data of the 0th row, and the 1st row of the original Bayer image data is used as the virtual data of the -1th row, so as to realize the calculation of the G component of the R / B pixel point of the 1st row. When the 4th row arrives, the G component of the 1st row is simulated as the virtual G component of the 0th row, and the calculation of the R / B component of the 1st row is realized accordingly. In summary, all RGB Demosaic calculations starting from the 1st row can be realized. This solves the problem that when the input Bayer data stream is less than 5 rows, the row image data simultaneously output by the row buffer register cannot meet the calculation of the Demosaic algorithm.

[0238] The present invention further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the above-mentioned image enhancement method.

[0239] The present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the image enhancement method as described above is implemented.

[0240] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0241] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image enhancement system, characterized in that: include: (N-1) row buffer FIFOs connected in series, for receiving input Bayer data row by row, and outputting row image data to corresponding row buffer registers; wherein N≥3; N row buffer registers, used for buffering the row image data currently input and the row image data output by the row buffer FIFO; based on each clock cycle, any row buffer FIFO and the T pixels transmitted by the currently input row image data respectively, the number of buffer levels is set to output N rows of image data at the same time and any row of image data is at least five pixels; The Demosaic module is used to receive the image data output by the line buffer register, calculate the G component line by line; receive and reuse the G component output by the G component buffer FIFO, calculate the non-G component line by line; and output the RGB components of each line of image data; The calculation process of the G component is as follows: the interpolation result calculation formula of each interpolation direction is decomposed into two-stage pipelines, each stage uses only one operation, and the same operation method exists in the color difference change value calculation formula, and is placed in the same stage pipeline to obtain the G component difference of each interpolation direction; the third stage pipeline performs overflow protection on the G component difference of each interpolation direction respectively to obtain the interpolation result of each interpolation direction; the fourth stage pipeline selects the interpolation direction corresponding to the smaller color difference change value based on the comparison result of the color difference change value of each interpolation direction, and assigns the G component of the non-G pixel point to the corresponding interpolation result; wherein the interpolation direction includes the horizontal direction and the vertical direction; In order to realize the parallel processing of the calculation of the G component of the (N-4)th row and the R&B component of the (N-3)th row, the calculation of the non-G component is also designed as a four-stage pipeline; The pipeline design of the non-G component of the G pixel point is to decompose the calculation formula of the interpolation result of each interpolation direction into three-stage pipelines, each stage uses only one operation to obtain the interpolation result of each interpolation direction; the fourth stage pipeline performs overflow protection assignment and rounding on the interpolation result of each interpolation direction to obtain the non-G component corresponding to each interpolation direction; The pipeline design of the missing component of the non-G pixel point is to decompose the interpolation result calculation formula and the color difference change value calculation formula of each interpolation direction into two-stage pipeline operations, so that the third-stage pipeline obtains the interpolation result and the color difference change value of each interpolation direction; the fourth-stage pipeline compares the color difference change value of each interpolation direction; the sum of each interpolation direction is assigned overflow protection and rounded to obtain the missing component of each interpolation direction; the interpolation direction corresponding to the smaller color difference change value is selected, and the missing component of the non-G pixel point is assigned to the missing component of the corresponding interpolation direction; The G component cache FIFO is used to receive the G component of any row calculated by the Demosaic module and output it to the corresponding G component register; the capacity of each G component cache FIFO is half of the capacity of any row cache FIFO; The G component buffer register is used to buffer the G component of any row output by the G component buffer FIFO and output it to the Demosaic module; the capacity of each G component buffer register is half of the capacity of any row buffer register; If there are multiple G component buffer FIFOs, the G component buffer FIFOs are connected in series to simultaneously output the G component of any line of image data to the corresponding G component buffer register; The Demosaic module obtains the G components of two different lines of image data output by the G component cache register and the G component of one line of image data currently obtained by calculation in the same clock cycle.

2. The image enhancement system according to claim 1, characterized in that: The method of simultaneously outputting N lines of image data and any line of image data having at least five pixels comprises: When T=1, the row data valid signal of any row cache register and the cached row image data are cached at four levels, so that when the valid signal is 1, in the row image data output in any clock cycle, any pixel point of the second-level cache of the row cache register has at least 2 adjacent pixels on the left and 2 adjacent pixels on the right.

3. The image enhancement system according to claim 2, characterized in that: The valid signals include a row data valid signal, a row data valid signal of a first-level cache of a row buffer register, a row data valid signal of a second-level cache of a row buffer register, a row data valid signal of a third-level cache of a row buffer register, and a row data valid signal of a fourth-level cache of a row buffer register.

4. The image enhancement system according to claim 1, characterized in that: The method of simultaneously outputting N lines of image data and any line of image data having at least five pixels comprises: T≥2, the row data valid signal of any row cache register and the cached row image data are cached in two levels, so that when the valid signal is 1, in the row image data output in any clock cycle, any pixel point of the first-level cache of the row cache register has at least 2 adjacent pixels on the left and 2 adjacent pixels on the right.

5. The image enhancement system according to claim 4, characterized in that: The valid signals include a row data valid signal, a row data valid signal of a first level cache of a row buffer register, and a row data valid signal of a second level cache of a row buffer register.

6. The image enhancement system according to claim 1, characterized in that: The Demosaic module also includes: based on the first two rows and the last two rows of the input Bayer data, expanding the Bayer data to obtain virtual Bayer data; based on the virtual Bayer data, the RGB components corresponding to the first three rows and the last three rows of the Bayer data can be output.

7. An image enhancement method, applicable to the system according to any one of claims 1 to 6, characterized in that: include: Receive input Bayer data line by line, read N lines of image data at the same time, and any line of image data has at least five pixels; Based on N lines of image data, the color difference change value and interpolation result of each interpolation direction of any non-G pixel point are calculated in parallel line by line, and the G component interpolation direction and G component of the line are determined in the same clock cycle; Based on N lines of image data and the G component of the previous line of image data, the chromatic aberration change value and the interpolation result in each interpolation direction of any non-G pixel point in any line of image data, and the interpolation result in each interpolation direction of any G pixel point are calculated in parallel line by line, and the missing component of the non-G pixel point and the non-G component of the G pixel point in the same clock cycle are determined.

8. The image enhancement method according to claim 7, characterized in that: Also includes: Before the determination, data overflow protection is performed on the calculation result of the color difference change value and / or the interpolation result of each pixel point to obtain the color difference change value and / or the interpolation result.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image enhancement method according to claim 7 or 8 is implemented.

Images