A high-precision image scaling method based on FPGA

Abstract

The present invention provides a high-precision image scaling method based on FPGA, comprising: calculating a column scaling weight coefficient through columns of pixels of an original image and columns of pixels of a target image; caching the original image in BRAM in sequence by rows, and sending the BRAM data and the original image data to a calculation unit to obtain image intermediate data of all columns of the target image; calculating a row scaling weight coefficient through rows of pixels of the original image and rows of pixels of the target image; caching the image intermediate data in BRAM in sequence by rows, and sending the image intermediate data to a calculation unit to obtain images of all rows of the target image after reading the image intermediate data; adopting a pipeline method to process logical operations in real time, and the scaling processing of the original image is all real-time, thereby improving the real-time performance of the system.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a high-precision image scaling method based on FPGA. Background Art

[0002] With the development of technology, people's demand for 4K high-definition video is getting stronger and stronger. Scaling low-resolution video to 4K high-definition video has become a major issue. With the increasing application of FPGA, the conversion of low-definition video to high-definition video based on FPGA has gradually become the mainstream.

[0003] In the existing technical background, the commonly used scaling algorithms include the nearest neighbor algorithm, the unilinear interpolation algorithm, the bilinear interpolation algorithm, the trilinear interpolation algorithm, the Lanczos algorithm, etc. Among them, the nearest neighbor difference algorithm and the unilinear interpolation algorithm are relatively simple and have poor effects; the trilinear interpolation algorithm and the Lancos algorithm are relatively complex and consume too many resources. The bilinear interpolation algorithm has become a more mainstream scaling algorithm.

[0004] Patent publication number CN111935416A proposes a method for storing scaled images based on FPGA, which focuses on the implementation method of writing the scaled image to DDR4 through the fifo cache;

[0005] Patent publication number CN112053283A proposes an image scaling method, a storage medium and an electronic device, which focuses on saving storage resources and time to achieve image scaling;

[0006] Patent publication number CN108986033A also proposes an algorithm for implementing bilinear interpolation based on FPGA, which also focuses on optimizing resources, but the display effect is not good after image processing.

[0007] The FPGA-based bilinear interpolation amplification algorithm proposed in patent publication number CN104869284A focuses on controlling the number of multiplier resources to save resources.

[0008] The existing processing methods need to consume a lot of logic resources to ensure the processing accuracy. The present invention saves logic resources and improves the processing accuracy to ensure the quality of the image after scaling, while balancing the logic resources and the calculation accuracy. Summary of the invention

[0009] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and provide a high-precision image scaling method based on FPGA, optimize the multiplication and division operations, and utilize multiplier resources to perform scaling operations after fixing the image data, thereby improving the calculation accuracy.

[0010] To achieve the above object, the present invention adopts the following technical solutions:

[0011] The present invention provides a high-precision image scaling method based on FPGA, comprising the following steps:

[0012] S1, calculating the weight coefficient of column scaling by the columns of pixels of the original image and the columns of pixels of the target image;

[0013] S2, caching the original image into BRAM row by row, reading out the BRAM data and sending it to the calculation unit together with the original image data, and obtaining the image intermediate data of all columns of the target image;

[0014] S3, calculating a row scaling weight coefficient using rows of pixels of the original image and rows of pixels of the target image;

[0015] S4, buffering the image intermediate data in BRAM row by row, and after reading out the image intermediate data, sending it to the calculation unit to obtain the image of all rows of the target image.

[0016] Further, the S1 comprises the following steps:

[0017] S11, obtaining the resolution of the original image at the source end, and filtering out rows and columns of pixels to be cached and involved in interpolation calculation, wherein the rows of pixels to be cached and involved in interpolation calculation are marked as m, and the columns of pixels to be cached and involved in interpolation calculation are marked as n;

[0018] S12, marking the row and column of the target pixel, wherein the row of the target pixel is marked as M, and the column of the target pixel is marked as N;

[0019] S13, calculate the column scaling factor step_h = n / N, a multiplier of XILINX FPGA realizes the multiplication of two 18-bit data, optimizes the division operation, and sets step_h = A*2 -18 , calculate the value of A.

[0020] Further, the S2 comprises the following steps:

[0021] S21, writing the first row of the original image into the first BRAM, and setting the depth of the BRAM according to the total number of columns of the original image;

[0022] S22, after the first row of data of the original image is completely written into the first BRAM and the second BRAM, read out the data in the BRAM;

[0023] S23, calculate the amplification 2 in step S13 18The product of the value of A and the column pixel x of the target image; the high 13 bits of the product result are recorded as an integer f0, and the low 18 bits are recorded as a decimal coe0; the f0 is used as the address to read the BRAM, and the read data is recorded as dou0;

[0024] S24, using f0 in S22 as the first BRAM read address, and reading out data dout0, and using f0+1 as the second BRAM read address, and reading out data dout1; sending dout0, dout1 and weights coe0, ~coe0+1 to the calculation unit, and calculating dout0*coe0, dout1*(~coe0+1) respectively through two multipliers;

[0025] S25, using an adder to complete the accumulation operation of dout0, dout1 and weights coe0, ~coe0+1, discarding the lower 18 bits of data, obtaining the image intermediate data of all columns of the first row, and obtaining the image intermediate data p0(x, y);

[0026] S26, repeating S21, S22, S23, S24 and S25, that is, completing the scaling operation of the original image to the target image sequence, and obtaining the image intermediate data p1(x, y), p2(x, y) ..., pn(x, y);

[0027] Further, the S3 comprises the following steps:

[0028] S31, obtaining the resolution of the original image at the source end, filtering out rows and columns of pixels to be cached and involved in interpolation calculation, wherein the rows of pixels to be cached and involved in interpolation calculation are marked as m1, and the columns of pixels to be cached and involved in interpolation calculation are marked as n1;

[0029] S32, marking the row and column of the target pixel, wherein the row of the marked target pixel is marked as M1, and the column of the marked target pixel is marked as N1;

[0030] S33, calculate the column scaling factor step_w = m1 / M1, a multiplier of XILINX FPGA can realize the multiplication of two 18-bit data, optimize the division operation, set step_w = B*2 -18 , calculate the value of B.

[0031] Further, the S4 comprises the following steps:

[0032] S41, after the first row of the image intermediate data is completely written into the third BRAM, the product of step_w and the column pixel y of the target image is calculated;

[0033] S42, the high 13 bits of the product result are recorded as integer f1, and the low 18 bits are recorded as decimal coe1; as y increases;

[0034] If the value of f1 calculated for the next row is different from the value of f1 corresponding to the previous row, the next row of image intermediate data is written to the fourth BRAM;

[0035] Otherwise, the fourth BRAM is not written;

[0036] By analogy, only when the corresponding f1 is different, the ping-pong cache will be executed to update the image intermediate data to the third BRAM and the fourth BRAM in sequence;

[0037] S43, the value of f1 calculated for the next row is different from the value of f1 corresponding to the previous row, and the image data of the third BRAM and the intermediate image data to be written into the fourth BRAM are read out at the same time, and sent to the calculation unit with the weights coe1,~coe1+1 of B, and dou0*coe0, dout1*(~coe0+1) are calculated respectively through two multipliers;

[0038] An adder is used to complete the accumulation operation of the two data of dou0*coe0 and dout1*(~coe0+1), and the 18-bit data is discarded to obtain all the target image data of the first row after interpolation, thereby obtaining the target image data p(x, y);

[0039] S44, repeating S41, S42 and S43, that is, completing the scaling operation from the original image to the target image sequence, and obtaining all the target image data.

[0040] The beneficial effects of the present invention are as follows: scaling an original image to a target image. The weight coefficient of the scaling column is calculated by the respective columns of the pixels of the original image and the pixels of the target image; the original image data is written into the BRAM by row, the first row is written into the first BRAM, and the row data is written into the second BRAM at the same time, and then the image data of the first BRAM and the image data of the second BRAM and the respective weight coefficients are read out and sent to the calculation unit to obtain the image intermediate data of the first row of the target image, and the image intermediate data of the second row, the third row, and the last row are calculated by analogy; the weight coefficient of the scaling row is calculated by the respective rows of the pixels of the original image and the pixels of the target image; the image intermediate data of the first row of the target image obtained above is written into the third BRAM, and after judging again, the image intermediate data of the second row is written into the fourth BRAM when the requirements are met, and the writing is done in a ping-pong cache manner, and then the image intermediate data of the third BRAM and the image intermediate data of the second row are read out and the respective weight coefficients are sent to the calculation unit to obtain the image of the first row of the target image, and the image data of the second row, the third row, and the last row are calculated by analogy.

[0041] The ping-pong cache method is used to optimize cache resources. All original data and intermediate processing data are cached on the chip through the ping-pong cache method, which improves the reading and writing speed and calculation speed of the calculation process; the pipeline method is used to process logical operations in real time, and the scaling of original images is real-time, which improves the real-time performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A block diagram of a high-precision image scaling method based on FPGA of the present invention;

[0043] Figure 2 This is an image timing diagram of a high-precision image scaling method based on FPGA of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0045] See also Figure 1 and Figure 2 , a high-precision image scaling method based on FPGA, comprising the following steps:

[0046] S1, calculating the weight coefficient of column scaling by the columns of pixels of the original image and the columns of pixels of the target image;

[0047] Get the resolution of the original image on the source side, filter out the rows and columns of pixels to be cached and involved in the interpolation calculation, and calculate the column scaling factor.

[0048] The S1 comprises the following steps:

[0049] S11, obtaining the resolution of the original image at the source end, and filtering out rows and columns of pixels to be cached and involved in interpolation calculation, wherein the rows of pixels to be cached and involved in interpolation calculation are marked as m, and the columns of pixels to be cached and involved in interpolation calculation are marked as n;

[0050] S12, marking the row and column of the target pixel, wherein the row of the target pixel is marked as M, and the column of the target pixel is marked as N;

[0051] S13, calculate the column scaling factor step_h = n / N, a multiplier of XILINX FPGA realizes the multiplication of two 18-bit data, optimizes the division operation, and sets step_h = A*2 -18 , calculate the value of A.

[0052] S2, caching the original image into BRAM row by row, reading out the BRAM data and sending it to the calculation unit together with the original image data, and obtaining the image intermediate data of all columns of the target image;

[0053] By optimizing multiplication and division operations, the representation of integers and decimals is completed in a fixed-point manner, and the calculation speed is improved, and the calculation accuracy is increased to 2 18 At the same time, by caching column data simultaneously, consuming two BRAMs, one multiplier, and one DSP, all columns can be cached and calculated.

[0054] The S2 comprises the following steps:

[0055] S21, writing the first row of the original image into the first BRAM, and setting the depth of the BRAM according to the total number of columns of the original image;

[0056] S22, after the first row of data of the original image is completely written into the first BRAM and the second BRAM, read out the data in the BRAM;

[0057] S23, calculate the amplification 2 in step S13 18 The product of the value of A and the column pixel x of the target image; the high 13 bits of the product result are recorded as an integer f0, and the low 18 bits are recorded as a decimal coe0; the f0 is used as the address to read the BRAM, and the read data is recorded as dou0;

[0058] S24, using f0 in S22 as the first BRAM read address, and reading out data dout0, and using f0+1 as the second BRAM read address, and reading out data dout1; sending dout0, dout1 and weights coe0, ~coe0+1 to the calculation unit, and calculating dout0*coe0, dout1*(~coe0+1) respectively through two multipliers;

[0059] S25, using an adder to complete the accumulation operation of dout0, dout1 and weights coe0, ~coe0+1, discarding the lower 18 bits of data, obtaining the image intermediate data of all columns of the first row, and obtaining the image intermediate data p0(x, y);

[0060] S26. Repeat S21, S22, S23, S24 and S25 to complete the scaling operation from the original image to the target image sequence, and obtain image intermediate data p1(x, y), p2(x, y) ..., pn(x, y).

[0061] The depth of BRAM is reasonably set according to the total number of columns of the original image to improve the utilization rate of BRAM resources; after optimizing the multiplication and division in the calculation, the calculation is performed in a fixed-point manner, which saves the number of multipliers and dividers used. At the same time, only one multiplier is used, which expands the data by 2 18 After multiplying, the integer and decimal parts are calculated using fixed-point method, and the accuracy is increased to 2 18 ; Use a DSP to complete the calculation process of multiplying two sets of data and then accumulating them, consuming one multiplier and one DSP resource to increase the calculation accuracy to 2 18 ;

[0062] S3, calculating a row scaling weight coefficient using rows of pixels of the original image and rows of pixels of the target image;

[0063] Get the resolution of the original image on the source side, filter out the rows and columns of pixels to be cached and involved in the interpolation calculation, and calculate the row scaling factor.

[0064] The S3 comprises the following steps:

[0065] S31, obtaining the resolution of the original image at the source end, filtering out rows and columns of pixels to be cached and involved in interpolation calculation, wherein the rows of pixels to be cached and involved in interpolation calculation are marked as m1, and the columns of pixels to be cached and involved in interpolation calculation are marked as n1;

[0066] S32, marking the row and column of the target pixel, wherein the row of the marked target pixel is marked as M1, and the column of the marked target pixel is marked as N1;

[0067] S33, calculate the column scaling factor step_w = m1 / M1, a multiplier of XILINX FPGA can realize the multiplication of two 18-bit data, optimize the division operation, set step_w = B*2 -18 , calculate the value of B.

[0068] S4, buffering the image intermediate data in BRAM row by row, and after reading out the image intermediate data, sending it to the calculation unit to obtain the image of all rows of the target image.

[0069] By optimizing multiplication and division operations, the representation of integers and decimals is completed in a fixed-point manner, and the calculation speed is improved, while the calculation accuracy is increased to 2 18 At the same time, through the ping-pong cache, two BRAMs, a multiplier and a DSP are consumed to complete the cache and calculation of all rows.

[0070] The S4 comprises the following steps:

[0071] S41, after the first row of the image intermediate data is completely written into the third BRAM, the product of step_w and the column pixel y of the target image is calculated;

[0072] S42, the high 13 bits of the product result are recorded as integer f1, and the low 18 bits are recorded as decimal coe1; as y increases;

[0073] If the value of f1 calculated for the next row is different from the value of f1 corresponding to the previous row, the next row of image intermediate data is written to the fourth BRAM;

[0074] Otherwise, the fourth BRAM is not written;

[0075] By analogy, only when the corresponding f1 is different, the ping-pong cache will be executed to update the image intermediate data to the third BRAM and the fourth BRAM in sequence;

[0076] S43, the value of f1 calculated for the next row is different from the value of f1 corresponding to the previous row, and the image data of the third BRAM and the intermediate image data to be written into the fourth BRAM are read out at the same time, and sent to the calculation unit with the weights coe1,~coe1+1 of B, and dou0*coe0, dout1*(~coe0+1) are calculated respectively through two multipliers;

[0077] An adder is used to complete the accumulation operation of the two data of dou0*coe0 and dout1*(~coe0+1), and the lower 18 bits of data are discarded to obtain all the target image data of the first row after interpolation, thereby obtaining the target image data p(x, y);

[0078] S44, repeating S41, S42 and S43, that is, completing the scaling operation from the original image to the target image sequence, and obtaining all the target image data.

[0079] The depth of BRAM is reasonably set according to the total number of columns of the intermediate data of the image to improve the utilization rate of BRAM resources; after optimizing the multiplication and division in the calculation, the calculation is performed in a fixed-point manner, which saves the number of multipliers and dividers used. At the same time, only one multiplier is used, which expands the data by 2 18 After multiplying, the integer and decimal parts are calculated using fixed-point method, and the accuracy is increased to 2 -18 ;

[0080] Only two BRAMs can be used to complete the scaling calculation of all rows of the entire frame through ping-pong caching, without using BRAM to cache a complete frame, and without using external DDR to cache intermediate calculation data, which greatly simplifies the use of BRAM; at the same time, a DSP is used to complete the calculation process of multiplying and accumulating two sets of data, consuming one multiplier and one DSP resource, and the calculation accuracy is increased to 2 -18 ;

[0081] Calculate the weight coefficient of column scaling through the pixels of the original image and the pixels of the target image respectively;

[0082] The original image is cached into BRAM row by row, and the BRAM data is read out and sent to the calculation unit together with the original image data to obtain the image intermediate data of all columns of the target image;

[0083] Calculate the weight coefficient of row scaling by the pixels of the original image and the pixels of the target image respectively;

[0084] The image intermediate data is cached in BRAM row by row, and after reading out the image intermediate data, it is sent to the calculation unit to obtain the image of all rows of the target image;

[0085] The fixed-point calculation method is used to optimize the multiplication and division operations with decimals in the calculation process, and the calculation accuracy is increased to 2 without increasing the logic resources. 18 ;

[0086] The above-mentioned embodiments only express the implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.

Claims

1. A high-precision image scaling method based on FPGA, comprising the following steps: S1, calculating the weight coefficient of column scaling by the columns of pixels of the original image and the columns of pixels of the target image; S2, caching the pixels of the original image in BRAM row by row, and sending the BRAM data and the weight coefficient of the column scaling to the calculation unit after reading out to obtain the image intermediate data of all columns of the target image; S3, calculating a row scaling weight coefficient using rows of pixels of the original image and rows of pixels of the target image; S4, cache the image intermediate data in BRAM row by row, read out the image intermediate data, and send it to the calculation unit to obtain the image of all rows of the target image; The S1 comprises the following steps: S11, obtaining the resolution of the original image at the source end, and filtering out the rows and columns of pixels to be cached and involved in the interpolation calculation, wherein: The row of the pixel to be cached and involved in the interpolation calculation is marked as m, and the column of the pixel to be cached and involved in the interpolation calculation is marked as n; S12, marking the row and column of the target pixel, wherein the row of the target pixel is marked as M, and the column of the target pixel is marked as N; S13, calculate the column scaling factor step_h = n / N, a multiplier of XILINX FPGA realizes the multiplication of two 18-bit data, optimizes the division operation, and sets step_h = A*2 -18 , calculate the value of A; The S2 comprises the following steps: S21, writing the first row of the original image into the first BRAM, and setting the depth of the BRAM according to the total number of columns of the original image; S22, after the first row of data of the original image is completely written into the first BRAM and the second BRAM, the data in the BRAM is read out; S23, calculate the amplification 2 in step S13 18 The product of the value of A and the column pixel x of the target image; the high 13 bits of the product result are recorded as an integer f0, and the low 18 bits are recorded as a decimal coe0; the f0 is used as the address to read the BRAM, and the read data is recorded as dou0; S24, using f0 in S23 as the first BRAM read address, and reading out data dou0, and using f0+1 as the second BRAM read address, and reading out data dou1; sending dou0, dou1 and weights coe0, ~coe0+1 to the calculation unit, and calculating dou0*coe0, dou1*(~coe0+1) respectively through two multipliers; S25, using an adder to complete the accumulation operation of dou0, dou1 and weights coe0, ~coe0+1, discarding the lower 8 bits of data, obtaining the image intermediate data of all columns of the first row, and obtaining the image intermediate data p0(x, y); S26, repeat S21, S22, S23, S24 and S25, that is, complete the scaling operation of the original image to the target image sequence, and obtain the image intermediate data p1(x, y), p2(x, y) ..., pn(x, y); The S3 comprises the following steps: S31, obtaining the resolution of the original image at the source end, filtering out rows and columns of pixels to be cached and involved in interpolation calculation, wherein the rows of pixels to be cached and involved in interpolation calculation are marked as m1, and the columns of pixels to be cached and involved in interpolation calculation are marked as n1; S32, marking the row and column of the target pixel, wherein the row of the marked target pixel is marked as M1, and the column of the marked target pixel is marked as N1; S33, calculate the row scaling factor step_w = m1 / M1, a multiplier of XILINX FPGA can realize the multiplication of two 18-bit data, optimize the division operation, set step_w = B *2 -18 , calculate the value of B; The S4 comprises the following steps: S41, after the first row of the image intermediate data is completely written into the third BRAM, calculate the magnification 2 in step S33 18 The product of the value of B and the row pixel y of the target image; S42, the high 13 bits of the product result are recorded as integer f1, and the low 18 bits are recorded as decimal coe0; as y increases; If the value of f1 calculated for the next row is different from the value of f1 corresponding to the previous row, the next row of image intermediate data is written to the fourth BRAM; Otherwise, the fourth BRAM is not written; By analogy, only when the corresponding f1 is different, the ping-pong cache will be executed to update the image intermediate data to the third BRAM and the fourth BRAM in sequence; S43, the f1 calculated for the next row is different from the value of f1 corresponding to the previous row, and the image data of the third BRAM and the intermediate image data to be written to the fourth BRAM are read out at the same time, and the weight coe0 of B is compared. ~ coe0+1 is sent to the calculation unit, and dou0*coe0, dou1*( ~ coe0+1); Then use an adder to complete the dou0*coe0, dou1*( ~ coe0+1) two data are added, the lower 8 bits of data are discarded, and all the target image data of the first row after interpolation are obtained to obtain the target image data p(x, y); S44, repeating S41, S42 and S43, that is, completing the scaling operation from the original image to the target image row, and obtaining all the target image data.

Images