A convolution template-based image processing method

Abstract

The present invention belongs to the field of image processing technology, and specifically relates to an image processing method based on a convolution template. The present invention provides an image processing method based on a convolution template, by making the convolution template dynamic, the convolution template can set the size range of the convolution template according to the convolution operation processing requirements of the image, so that the size of the convolution template can be changed during the image processing without recompiling. The dynamic nature of the convolution template makes the size of the convolution template no longer fixed, thereby solving the problem that the convolution operation under different circumstances has a large demand for convolution templates, reducing the design difficulty, shortening the development cycle and other related issues.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to an image processing method based on a convolution template. Background Art

[0002] With the development of image processing technology, different fields have higher and higher requirements for algorithms, resulting in higher and higher complexity. The convolution operation in the image processing algorithm can enhance or weaken the specific effect of the image. By blurring and sharpening the image, certain specific features in the image can be found, and the edge information of the target image can be found. It is widely applicable to the needs of different fields. The image convolution operation processing requires the processing of huge matrix data. The implementation of a single template size is not too complicated, but the design difficulty of the template with multiple size requirements is high, the development cycle is long, and the application direction is relatively single. Summary of the invention

[0003] The technical problem to be solved by the present invention is that for existing convolution operations, the size of convolution templates is fixed in a wide range of application fields, while the demand for convolution templates in different fields is large, the design is difficult, and the development cycle is long. An image processing method based on convolution template is provided.

[0004] The present invention provides an image processing method based on a convolution template, comprising:

[0005] Step S1, constructing a convolution operation module, setting the size range of the convolution template according to the requirements of the convolution operation processing of the image, and storing the total number of pixel points in the size range of the convolution template into the line buffer array;

[0006] Step S2, performing one-dimensional row data processing according to the convolution template in step S1, performing convolution operation processing on each pixel point of each row according to the column index value of the input image and the size of the convolution template, and outputting the pixel point scanning to the column cache array of the image after the one-dimensional row data processing;

[0007] Step S3, performing one-dimensional column data processing according to the column cache array of the image after the one-dimensional row data processing in step S2, by inputting the row index value of the image and according to the size of the convolution template, performing convolution operation processing on each pixel point of each column of the one-dimensional row data processed image in the column cache number, and outputting the result of the convolution operation processing of the image after all the pixel points are scanned;

[0008] Step S4: perform image edge processing according to the result of the convolution operation in step S3. After the image edge processing is completed, the image processing is completed.

[0009] Optionally, in step S1, the convolution template is made dynamic by setting the size range of the convolution template according to the convolution operation processing of the image, and initializing the size and variables of the cache array required for image processing, rather than the fixed convolution template size setting in the prior art. This method solves the problems related to the prior art in which the convolution operation in different situations has a large demand for convolution templates, reduces the design difficulty, shortens the development cycle, and the like.

[0010] Optionally, in step S2, the one-dimensional row data processing, wherein the column index value of the image is determined based on the total number of pixels in the size range of the convolution template, and the image with the determined total number of pixels is stored in a row cache array, and then the one-dimensional row data processing is entered based on the image.

[0011] Optionally, in step S3, the one-dimensional column data processing needs to store the pixels required for the next one-dimensional column data processing while processing each column of the one-dimensional column data, and move the result of the current one-dimensional column data processing upward by the corresponding number of pixels to ensure the normal processing of the next row of one-dimensional column data.

[0012] Optionally, in step S4, the image edge processing is performed differently according to different project requirements. The edge portion of the image can be directly supplemented with constants for processing. This method has a faster image processing speed, but sacrifices some accuracy.

[0013] The edge of the image needs to be calculated based on the column index value combined with the row index value. For the result of the convolution operation, the number of pixels corresponding to each edge of the image is supplemented. This method improves accuracy compared to the more common method of supplementing constants.

[0014] Optionally, the convolution template supports different sizes, and different convolution template sizes are supported during image processing. Changing the size of the convolution template does not require recompilation, thereby reducing the waste of time, cost and resources caused by performance testing of fixed convolution template sizes.

[0015] Optionally, each step of the image data processing is designed according to a coarse-grained pipeline. At each stage of image processing, when the pixels processed in the first step meet the conditions for the next step, they can be operated in parallel. This coarse-grained pipeline design significantly improves the performance of image processing.

[0016] Optionally, each step of the image data processing is designed according to a fine-grained pipeline, and each data operation of the image processing is completed, and the operations of each step overlap, and multiple functional components work in parallel to shorten the program execution time and improve the operation efficiency and throughput.

[0017] Optionally, the interface design of the convolution template can be designed according to the required targeting, and the port usage type and the protocol used by a certain port can be specified, such as supporting a high-speed AXI4 interface.

[0018] Optionally, the convolution template-based image processing method is implemented using a Vivado HLS tool. The optimization strategy of the Vivado HLS tool can optimize the resource consumption of the module implemented by the RTL and improve the data throughput rate, and the verification mechanism can ensure the functional correctness. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a data flow chart of an image processing method based on a convolution template of the present invention;

[0020] Figure 2 It is a flow chart of image processing of an image processing method based on a convolution template of the present invention;

[0021] Figure 3 It is a schematic diagram of a coarse-grained pipeline design layout of an image processing method based on a convolution template of the present invention;

[0022] Figure 4 It is a schematic diagram of the fine-grained pipeline design layout of an image processing method based on a convolution template of the present invention. DETAILED DESCRIPTION

[0023] The present invention will be described in more detail below in conjunction with the accompanying drawings and specific embodiments.

[0024] In one embodiment, a convolution template-based image processing method is provided, by making the convolution template dynamic. Dynamic means setting the size range of the convolution template according to the convolution operation processing of the image, initializing the size of the cache array and the variables required for image processing, rather than the fixed convolution template size setting in the prior art. The size length of the convolution template is generally an odd number, such as 3, 5, 7, etc. Compared with an even number, the cardinality template has a center point, is more sensitive to edges and lines, and can extract edge information more effectively. Among them, a large-sized convolution template can bring a larger perception field of view, but has more parameters and calculations, while a small-sized template is the opposite. The size of the convolution template is generally selected according to different application scenarios to set the size of the convolution template. The factors that usually affect the size of the convolution template include the image background and the specific distribution of the target. The size range of the convolution template is set according to the actual image needs, and its maximum and minimum values ​​[KMIN, KMAX] are determined. At the same time, the number of rows ROW and the number of columns COL of the processed image are set. The storage cache group required for row processing is defined as h_pbuf[KMAX], the storage cache group required for column processing is defined as v_linebuf[KMAX][COL], and the storage cache group required for edge processing is defined as b_pbuf[COL-KMIN+1]. Temporary variables are flexibly set according to the needs of the operation. The reasonable use of data bit width can save a lot of resources.

[0025] In one embodiment, Figure 1 As shown in the figure, after the convolution template is determined, the steps of image processing are divided into three stages. The two-dimensional data processing is divided into one-dimensional array row data processing (horizontal processing abbreviated as HP) and one-dimensional array column data processing (vertical processing abbreviated as VP). After completing the convolution operation, the image edge processing (border processing abbreviated as BP) is performed.

[0026] In one embodiment, Figure 2 As shown, the specific implementation of each image processing stage is as follows:

[0027] The steps of one-dimensional array row data processing are: according to the column index value widx of the input image src and the size range of the convolution template, determine the size of the image to be processed, the number of rows ROW and the number of columns COL, multiply the number of rows and columns to obtain the total number of pixels ksize corresponding to the processed image, and store the total number of pixels ksize into the cache array h_pbuf[KMAX], where KMAX is the maximum value of the size range required to support the convolution template. The number of cycles of this processing can be obtained according to the value of KMAX. One-dimensional array row data processing performs convolution operations on each pixel on each row. When k is greater than 0 and k is equal to ksize-1, it means that the operation has reached the cumulative amount of operations required for the next image processing, and the current loop is jumped out. If it is not satisfied, the operation continues. Whether the output of this row convolution operation is valid depends on whether the column index value widx satisfies widx greater than ksize-1. If this condition is met, it means that the output of the convolution operation is valid. Otherwise, continue to obtain new pixels from the input image to participate in the operation until the image is scanned horizontally row by row from the upper left corner to the lower right corner to complete the convolution operation of each row of the image.

[0028] The steps of one-dimensional array column data processing are: according to the column index value widx, the result of the previous convolution operation is obtained and stored in the cache array v_linebuf[KMAX][COL]. It is a memory composed of multi-dimensional shift registers, which can store pixel data of KMAX rows and perform convolution operations on ksize pixels in each column. Similarly, KMAX here is the maximum value of the required support convolution template, and the value of KMAX can also be used to obtain the number of cycles of this processing. One-dimensional array column data processing is aimed at the image output by the convolution operation of the previous data processing, and each pixel on each column is convolved. When j is greater than 0 and j is equal to ksize-1, it means that the operation has reached the required cumulative operation amount and jumps out of the current loop. If it is not satisfied, the operation continues. Whether the output of this column convolution operation is valid depends on whether the row index value hidx satisfies hidx greater than ksize-1. If this condition is met, it means that the output of the convolution operation is valid, otherwise it continues to obtain new pixels from the convolution operation of the previous data processing to participate in the operation. Column data processing requires storing ksize-1 rows of data at the same time. Whenever a pixel point is obtained in the ksize-th row, the convolution operation is performed until the end of this row. At the same time, the data of ksize-1 points in each column are moved upward to ensure that the ksize-1 data on each column are reused before obtaining the next row of data.

[0029] The steps of image edge processing are: Due to the characteristics of convolution operation, the area processed by the convolution template cannot completely match the actual area of ​​the image, so when processing the edge pixels of the image, the image edge needs to be properly processed. The processing method will be determined according to the needs of the project. The image edge part can be directly supplemented with constants for image edge processing, or the border part of the result after the previous column convolution operation can be copied for image edge processing. The former method does not require an additional row cache array, and the latter needs to use the cache array b_pbuf[COL-KMIN+1] to save the result after the previous column convolution operation. The data is scanned horizontally from the upper left corner to the lower right corner row by row according to the column index value widx. Here, KMIN is the minimum value required to support the convolution template. The number of pixels that need to be padded on the edge of the image is defined as num, which is obtained based on the total number of pixels ksize, and is calculated as num=ksize / 2. Based on the column index value widx combined with the row index value hidx, after obtaining the result of the previous column convolution operation, num columns of pixels are padded on the left and right sides of the output image, and num rows of pixels are padded on the top and bottom of the image respectively.

[0030] In one embodiment, the size of the convolution template changes during image processing without recompiling. In existing designs, the size of the convolution template is generally set to a fixed value or defined as a macro. To change this size to meet the needs of image processing, the entire design needs to be recompiled and synthesized to go through all the previous processes to obtain the corresponding RTL or netlist, and sufficient tests are required to ensure that the results are correct and the performance meets the requirements, which results in a waste of time, cost and resources. The present invention does not require recompilation and synthesis, but only needs to set the convolution template range [KMIN, KMAX] that needs to be supported. The user can dynamically change the size, that is, the same design supports different convolution template sizes.

[0031] In one embodiment, Figure 3As shown in the figure, each step of image processing is designed according to a coarse-grained pipeline. One-dimensional array row data processing, one-dimensional array column data processing and image edge processing are the core steps of the entire image processing. The loop nesting of the image algorithm itself makes it impossible to process multiple input pixels in parallel, and they can only be executed sequentially. However, different image processing stages can be executed in parallel, so different image processing stages can be designed according to a coarse-grained pipeline. In essence, it is a FIFO or ping-pong cache structure arrangement processing process. The data required for the next row data processing HP does not affect the previous column data processing VP, and can be operated in parallel. The next image edge processing BP does not affect the previous column data processing VP, and can be operated in parallel. The output is only valid after the rows and columns corresponding to the template size are met. This coarse-grained pipeline design significantly improves the performance of image processing. The overall image processing performance is not a simple accumulation of each stage, but depends on the number of clocks required for the last image processing stage.

[0032] In one embodiment, Figure 4 As shown in the figure, each step of image processing is designed according to a fine-grained pipeline. The coarse-grained pipeline design makes the overall performance depend on which image processing stage requires the longest number of clocks. The fine-grained pipeline design improves the execution speed of each specific operation in each cycle of each processing stage, so that the next execution of the same operation does not have to wait until the end of the previous cycle, and the interval time for receiving the next input data (initiation interval is also II) is reduced as much as possible, so as to improve the data processing throughput. Each data operation of data reading (DATA_R), data processing (DATA_P) and data writing (DATA_W) is completed, and each step of operation overlaps. Through the parallel operation of multiple functional components, the program execution time is shortened, and the operation efficiency and throughput are improved.

[0033] In one embodiment, the interface design of the convolution template is user-selectable and supports the AXI4 interface. According to the user's design requirements, the port usage type and the protocol used by a certain port can be specified, such as supporting the high-speed AXI4 interface.

[0034] In one embodiment, a convolution template-based image processing method is implemented using the Vivado HLS tool. The entire design process, functional verification, performance testing, timing and resource configuration are all implemented by the Vivado HLS tool. The optimization strategy of the Vivado HLS tool can optimize the resource consumption of the module implemented by RTL and improve the data throughput rate, and the verification mechanism can ensure the functional correctness. First, the most original matlab or C / C++ algorithm code is obtained from the user, and then the original algorithm is evaluated. In particular, when the user has high requirements for extreme performance, it is necessary to improve the algorithm structure of the original algorithm to achieve the purpose of extreme performance. Therefore, in order to achieve a balance between performance and resources, it is necessary to reasonably configure the module interface, internal function, and core computing unit based on the selected device; once the structure is built, it is necessary to write different C / C++ test samples and complete C / C++ / RTL simulation to ensure the correctness of the function. According to the comprehensive report of the Vivado HLS tool, combined with the report of the Vivado RTL-level code test, the module configuration timing and resource report are summarized, and finally the most accurate performance report is obtained based on the board-level test, and the correctness of the function is verified in more depth. In response to the requirements of these functions, the fast and efficient implementation characteristics of Vivado HLS are used, and it is iterated several times to achieve convergence of various indicators.

[0035] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An image processing method based on a convolution template, characterized in that: The specific steps include: Step S1, constructing a convolution operation module, setting the size range of the convolution template according to the convolution operation processing requirements of the image, and storing the total number of pixel points in the size range of the convolution template into the line buffer array; the size range of the convolution template described in step S1 can be set according to the actual image requirements, and its maximum and minimum values ​​can be determined; Step S2, performing one-dimensional row data processing according to the convolution template in step S1, by inputting the column index value of the image, and performing convolution operation processing on each pixel point of each row according to the size of the convolution template, and outputting the pixel point scanning to the column cache array of the image after the one-dimensional row data processing; the one-dimensional row data processing described in step S2, wherein the column index value of the image is determined according to the total number of pixels in the size range of the convolution template, and the image with the determined total number of pixels is stored in the row cache array, and then the one-dimensional row data processing is entered according to the image; Step S3, performing one-dimensional column data processing according to the column cache array of the image after the one-dimensional row data processing in step S2, by inputting the row index value of the image and according to the size of the convolution template, performing convolution operation processing on each pixel point of each column of the one-dimensional row data processed image in the column cache number, and outputting the result of the convolution operation processing of the image after all the pixel points are scanned; Step S4: perform image edge processing according to the result of the convolution operation in step S3. After the image edge processing is completed, the image processing is completed.

2. The image processing method based on convolution template according to claim 1, characterized in that: In the one-dimensional column data processing described in step S3, when processing each column of one-dimensional column data, it is necessary to store the pixels required for the next column of one-dimensional column data processing, and move the result of the current one-dimensional column data processing upward by the corresponding number of pixels to ensure the normal processing of the next row of one-dimensional column data.

3. The image processing method based on convolution template according to claim 1, characterized in that: The image edge processing described in step S4 is performed differently according to different project requirements, and the image edge portion can be directly supplemented with constants for processing; The edge of the image needs to be processed based on the column index value combined with the row index value, and the number of pixels corresponding to each edge of the image is supplemented for the result of the convolution operation.

4. The image processing method based on convolution template according to claim 1, characterized in that: The convolution template supports different sizes. Different convolution template sizes are supported during image processing. Changing the size of the convolution template does not require recompilation.

5. The image processing method based on convolution template according to claim 1, characterized in that: Each step of the image data processing is designed according to a coarse-grained pipeline.

6. The image processing method based on convolution template according to claim 1, characterized in that: Each step of the image data processing is internally designed according to a fine-grained pipeline.

7. The image processing method based on convolution template according to claim 1, characterized in that: The interface design of the convolution template can be designed specifically as required.

8. The image processing method based on convolution template according to claim 1, characterized in that: The image processing method based on convolution template is implemented using Vivado HLS tool.

Images