Image processing method and chip based on convolution algorithm

Abstract

The application discloses an image processing method and chip based on a convolution algorithm, and the image processing method comprises the following steps: step A, using fast Fourier transform and fast inverse Fourier transform in sequence to control an image template and an image to be processed to obtain a convolution result element in a corresponding spatial domain in the form of multiplication in a corresponding frequency domain; and step B, if a convolution result element meets a preset gray value condition, setting a gray value of a pixel point corresponding to the convolution result element in the image to be processed as a preset target gray value. The complexity of the calculation time is reduced, the data amount of the pixel point processing is also reduced, and the real-time performance of the image processing is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to an image processing method based on a convolution algorithm and a chip. BACKGROUND

[0002] The dilation algorithm is a basic algorithm in various image morphological algorithms. The dilation algorithm merges all background pixel points in contact with a region to be processed into the region to be processed, so that the boundary of the region to be processed expands outward. Therefore, the main uses of the dilation algorithm include eliminating noise, segmenting independent image pixel points, connecting adjacent pixel points in an image, finding obvious maximum or minimum value regions in an image, and calculating the gradient of an image.

[0003] As understood by those skilled in the art, a structural element is designed as the most basic component of the dilation and erosion operation, which is used to test the output image and is usually much smaller than the image to be processed. The structural element of a two-dimensional plane is composed of a matrix, and the origin of the structural element specifies the pixel range to be processed in the image.

[0004] In the image region covered by the structural element, a pixel point (x, y) with a gray value of A can be taken as a search center, and the gray values of all pixel points in the neighborhood of the search center are set to B. At this time, the conventional way to set the gray value to B is to set the gray values of all pixel points in the image region covered by the structural element to B row by row and column by column, or to perform specific logical operations on the region corresponding to the structural element and the binary image (converted from the image to be processed) covered by the structural element to set the gray values of the pixel points in the neighborhood of the pixel point (x, y) with the gray value of A to B, which often causes the pixel points with the gray value of B to be relatively enlarged by one circle. However, the time complexity generated by the foregoing algorithm when applied to traverse the entire image increases exponentially, so that the time spent on dilation calculation on a large image is relatively long, and the algorithm cannot be effectively applied to the image processing field with high real-time and variability requirements. SUMMARY

[0005] In order to reduce the time complexity consumed by image dilation calculation and associated image processing algorithms, the present application discloses an image processing method based on a convolution algorithm and a chip to improve the real-time performance of image processing. The specific technical solutions are as follows:

[0006] The application discloses a kind of based on convolution algorithm's image processing method, comprising: step A, in succession using fast Fourier transform and fast Fourier inverse transform, to control image template and the form that is multiplied in corresponding frequency domain with to-be-processed image obtains convolution result element in corresponding spatial domain;Step B, if there is a convolution result element meets preset gray value condition, then the gray value of the pixel point corresponding to the convolution result element in to-be-processed image is set as preset target gray value.

[0007] Further, the step B specifically includes: if the numerical value of the convolution result element is greater than 0 is detected, then the gray value of the pixel point covered by the center of the image template corresponding to the convolution result element in the to-be-processed image is set as the preset target gray value.

[0008] Further, in the step B, including: if the numerical value of the convolution result element is greater than 0 is detected, then it is determined that there is a pixel point with a preset search value in the image region framed by the image template corresponding to the convolution result element in the to-be-processed image, and then it is determined that one convolution result element obtained by the currently executed step A meets the preset gray value condition;Wherein, the image template corresponding to the convolution result element is a convolution kernel configured to support translation in the to-be-processed image for the purpose of matrix convolution operation.

[0009] Further, the image template is a convolution kernel in the form of a matrix, and there are matrix elements with a value of 1 and matrix elements with a value of 0 in the image template, wherein the position characteristics of the matrix elements with a value of 1 in the image template determine the pixel points in the to-be-processed image that actually participate in image processing.

[0010] Further, in the image template, the shape composed of the matrix elements with a value of 1 is defined as the shape of the image template.

[0011] Further, the preset search value is greater than 0;Or the mapping value converted according to a preset pixel value function is 1.

[0012] Further, the step A specifically comprises: transforming the image to be processed from a spatial domain to a frequency domain by using a two-dimensional discrete Fourier transform to obtain a first parameter matrix; transforming the image template from the spatial domain to the frequency domain by using a two-dimensional discrete Fourier transform to obtain a second parameter matrix; multiplying each matrix element of the first parameter matrix with a matrix element at a corresponding matrix position in the second parameter matrix to obtain a third parameter matrix; transforming the third parameter matrix from the frequency domain to the spatial domain by using a two-dimensional inverse discrete Fourier transform to obtain a convolution result matrix, so as to realize multiplication of the image template and the image to be processed in the corresponding frequency domain to obtain a convolution result element in the corresponding spatial domain; wherein the convolution result element in the corresponding spatial domain is arranged as a convolution result matrix; a relative position of the convolution result element in the convolution result matrix is the same as a relative position of a center of the image template corresponding to the convolution result element in the image to be processed; and the relative position is a position relationship of a matrix element relative to a fixed corner point position of the matrix.

[0013] Further, the image template is equivalently configured such that, during translation of the image template in the image to be processed, when the center of the image template covers one pixel point in the image to be processed, a matrix convolution operation is performed on elements of the image template itself and pixel points in an image region framed by the image template to obtain a convolution result element; after the center of the image template traverses all pixel points in the image to be processed, all convolution result elements obtained through the matrix convolution operation form an output matrix, which is equal to the convolution result matrix; wherein all elements in the image template are pre-configured; and all elements in the image to be processed are also pre-configured.

[0014] Further, when it is detected that a value of a convolution result element is greater than 0, there is a pixel point with the preset search value in a neighborhood of a pixel point covered by a center of the image template corresponding to the convolution result element in the image to be processed.

[0015] Further, each matrix element in the first parameter matrix is a Fourier coefficient described in a two-dimensional discrete Fourier transform form; each matrix element in the second parameter matrix is a Fourier coefficient described in a two-dimensional discrete Fourier transform form; a number of rows of the third parameter matrix, a number of rows of the second parameter matrix, and a number of rows of the first parameter matrix are equal; a number of columns of the third parameter matrix, a number of columns of the second parameter matrix, and a number of columns of the first parameter matrix are equal; a number of rows of the convolution result matrix is equal to a number of rows of a pixel matrix corresponding to the image to be processed; and a number of columns of the convolution result matrix is equal to a number of columns of the pixel matrix corresponding to the image to be processed.

[0016] A chip, which stores program code corresponding to the image processing method based on the convolution algorithm.

[0017] Further, the chip integrates a convolution hardware accelerator, which is used to convert the matrix convolution operation between the image to be processed and the image template into a convolution operation between two sequences; wherein each element in each sequence is a result of one-dimensional discrete Fourier transform of a row of elements of a corresponding matrix by the convolution hardware accelerator, or a result of one-dimensional discrete Fourier transform of a column of elements of a corresponding matrix by the convolution hardware accelerator; each element in each sequence is obtained by parallel processing of the convolution hardware accelerator.

[0018] Further, the chip integrates a convolution hardware accelerator, which is used to, after multiplying the transformed value of the image to be processed in the frequency domain with the transformed value of the image template corresponding to the matrix convolution operation in the corresponding frequency domain and outputting a parameter matrix, first perform one-dimensional inverse discrete Fourier transform on each row of elements of the parameter matrix in parallel to obtain an intermediate result sequence, then perform one-dimensional inverse discrete Fourier transform on each column of elements of the intermediate result sequence in parallel, and finally obtain a convolution result matrix to complete two-dimensional inverse discrete Fourier transform; or, after multiplying the transformed value of the image to be processed in the frequency domain with the transformed value of the image template corresponding to the matrix convolution operation in the corresponding frequency domain and outputting a parameter matrix, first perform one-dimensional inverse discrete Fourier transform on each column of elements of the parameter matrix in parallel to obtain an intermediate result sequence, then perform one-dimensional inverse discrete Fourier transform on each row of elements of the intermediate result sequence in parallel, and finally obtain a convolution result matrix.

[0019] Compared with the prior art, the beneficial technical effects of the present application are that: on the one hand, when there is a pixel point of a preset search value in a specific neighborhood of the center of the image template (an image region framed by the image template) in the equivalent convolution result obtained by fast Fourier transform of the image template and the image to be processed, the gray value of the center of the image template is configured as a preset target gray value. Compared with the various gray value assignment processes mentioned in the background art, the present application does not need to assign values to the pixel points outside the boundary of the image to be processed, avoids excessive expansion of the boundary points of the image to be processed, reduces the amount of pixel processing data, and ensures the real-time performance of image processing.

[0020] On the other hand, when the size of the image to be processed is relatively large, in order to reduce the huge calculation time cost brought by the traditional convolution operation or the dilated search operation, the present application uses fast Fourier transform and fast inverse Fourier transform successively to speed up the convolution processing procedure of the image to be processed and the image template, so as to speed up the obtaining of the pixel points in the corresponding image region which need to be re-assigned, and then speed up the process of the image dilation calculation, thereby reducing the calculation times, especially the more the discrete pixel points in the image to be processed, the more significant the saving degree of the calculation time of the present application. The fast Fourier transform is used to replace the convolution operation, mainly to reduce the operation amount and the storage cost, and it is particularly advantageous for hardware implementation. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 A flow chart of the image processing method based on the convolution algorithm is disclosed for an embodiment of the present application. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings of the embodiments of the present application. It should be understood that the specific embodiments described below are only used to explain the present application, and are not used to limit the present application. In the following description, specific details are given to provide a thorough understanding of the embodiments. However, one of ordinary skill in the art will understand that the embodiments can be implemented without these specific details. For example, circuits can be shown in block diagrams to avoid obscuring the embodiments in unnecessary details. In other cases, well-known circuits, structures and techniques can not be shown in detail to avoid obscuring the embodiments.

[0023] As an embodiment, the present application discloses an image processing method based on a convolution algorithm, as shown in Figure 1 specifically comprising:

[0024] Step A, using fast Fourier transform and fast inverse Fourier transform to control the image template and the image to be processed to obtain the convolution result elements in the corresponding spatial domain in the form of multiplication in the corresponding frequency domain; and then entering step B.

[0025] Specifically, the present embodiment sets the image template as a basic operation unit for the dilation or other morphological image processing of the image to be processed, which is usually smaller than the image to be processed. The image template is a convolution template configured to support translation in the image to be processed for the purpose of matrix convolution operation, that is, the image template is a convolution kernel for scanning the image to be processed.

[0026] As can be known from the background art, the image template can frame the pixel range to be processed in the image to be processed, and the image template and the binary image covered thereby can be subjected to an AND operation, which is equivalent to a matrix convolution operation of the image template and the image to be processed, i.e. multiplication and addition operations of all pixel points in the image region currently framed by the image template, if the results of the one-time convolution operation are all 0, the gray value of the pixel point corresponding to the non-0 element of the image template in the image region currently framed by the image template is 0; otherwise, the pixel points in the image region currently framed by the image template are all assigned a value of 1 or greater than 0, thereby expanding the binary image (the image after binarization of the image to be processed) by one circle, achieving the purpose of image dilation.

[0027] As can be understood by those skilled in the art, in the matrix convolution operation, each element in the image template is multiplied by and added to the value represented by the pixel point at the corresponding position covered thereby, to obtain a convolution result element, wherein the value represented by the pixel point at the position covered by the element is set to 0 if there is no pixel point at the position; after the center of the image template scans all the pixel points of the image to be processed, each pixel point of the image to be processed is subjected to the multiplication and addition operation, to obtain a matrix composed of elements as the image template, r is a positive integer, and in order to obtain the convolution result element corresponding to the pixel point covered by the center of the image template, the number of calculations consumed by each convolution result element is in a certain proportional relationship with the square of r; further, the pixel matrix is composed of n pixels (n is greater than 2r+1), and each time the center of the image template scans a pixel point of the image to be processed, the pixel matrix is offset by 2n units, and in order to obtain a convolution result matrix, ((2r+1)2+(2r)2)n2 operations are cumulatively performed, so that the complexity of the calculation time increases in a quadratic manner, which has an impact on the real-time performance of the image convolution operation, and restricts the speed of the related image processing, including the improvement of the speed of the image dilation operation.

[0028] In summary, in order to reduce the time complexity consumed by the image dilation calculation and the associated image processing algorithm, fast Fourier transform and inverse fast Fourier transform are used in step A to control the multiplication of the conversion coefficient of the image template in the corresponding frequency domain and the conversion coefficient of the image to be processed in the corresponding frequency domain, to replace the convolution operation between the image template and the image to be processed (i.e. the matrix convolution operation between two pixel matrices), and obtain the same operation result, i.e. the convolution result element in the corresponding spatial domain.

[0029] Specifically, the fast Fourier transform, namely the high-efficiency and fast calculation method for calculating the discrete Fourier transform (DFT) by using a computer, is collectively referred to as FFT. The discrete Fourier transform can sample the discrete points of the frequency spectrum of an image at equal intervals in the interval of 0 degrees to 360 degrees, that is, the discretization of the sequence spectrum, which is the physical meaning of DFT. Both the image template and the image to be processed are composed of discrete point matrices; in the form of the discrete Fourier transform, the sequences at both ends (in the time domain and the frequency domain) are of finite length, and in fact, both sequences should be considered as the principal value sequences of discrete periodic signals. Even for the DFT of a discrete signal of finite length, it should be considered as a periodic signal after period extension before transformation. In the present embodiment, the fast Fourier transform is used to efficiently calculate the DFT.

[0030] It should be noted that the Fourier transform is to convert an image from the spatial domain to the frequency domain, and the inverse transform is to convert an image from the frequency domain to the spatial domain. In other words, the physical meaning of the Fourier transform is to transform the gray scale distribution function of an image into the frequency distribution function of the image, and the inverse Fourier transform is to transform the frequency distribution function of the image into the gray scale distribution function. The fast Fourier transform is a fast algorithm for the discrete Fourier transform, and has obvious advantages when the data source is large (more than 1000).

[0031] A pixel matrix composed of n*n pixels, wherein n and r are both positive integers, and n is greater than 2r+1; the image template and the image to be processed corresponding to the convolution are both converted from the spatial domain to the frequency domain by using the fast Fourier transform, then the transformed value of the image template at the frequency domain position is multiplied by the transformed value of the image to be processed corresponding to the convolution at the corresponding frequency domain position, and then the fast Fourier transform is used to reduce the number of operations from n2 to (2r+1)nlog2((2r+1)n), thereby reducing the complexity of the calculation time and accelerating the convolution of the two-dimensional pixel points.

[0032] Therefore, when the size of the image to be processed is relatively large, in order to reduce the huge calculation time cost caused by the traditional convolution operation or the dilation search operation, the fast Fourier transform and the fast inverse Fourier transform are used in succession to accelerate the convolution process of the image to be processed and the image template, so as to accelerate the acquisition of the pixel points in the corresponding image region that need to be re-assigned, and further accelerate the process of the image dilation calculation, thereby reducing the number of calculations, and the more the discrete pixel points in the image to be processed, the more significant the saving of the calculation time of the present application. The fast Fourier transform is used to replace the convolution operation, which mainly reduces the amount of calculation and storage overhead, and is particularly advantageous for hardware implementation.

[0033] Step B, if one of the convolution result elements meets the preset gray value condition, the gray value of the pixel point corresponding to the convolution result element in the image to be processed is set to the preset target gray value; otherwise, the gray value of the pixel point corresponding to the convolution result element in the image to be processed remains the original gray value. In this embodiment, if it is detected that the value of one of the convolution result elements in the matrix output by the inverse fast Fourier transform meets the preset gray value condition, it is determined that the center of the image template is equivalent to the corresponding position of the image to be processed, and there is a pixel point with a specific gray value near the corresponding position and detected by the image template, and the gray value of the pixel point at the corresponding position is set to the preset target gray value, so as to achieve the purpose of expanding the pixel point with the preset target gray value in the image to be processed.

[0034] As an implementation, step B specifically includes: if it is detected that the value of the convolution result element is greater than 0, the gray value of the pixel point covered by the center of the image template corresponding to the convolution result element in the image to be processed is set to the preset target gray value, wherein the preset target gray value can be 0 or greater than 0. If it is detected that the value of the convolution result element is not greater than 0, the gray value of the pixel point covered by the center of the image template corresponding to the convolution result element in the image to be processed remains unchanged. In this embodiment, the value of the convolution result element greater than 0 is configured as the preset gray value condition met by the convolution result element; whether the image template has a value of 1 and a value of 0 or not, it will not cause misjudgment of whether the value of the convolution result element is greater than 0, because the parameter of the image template after the fast Fourier transform is equivalent to the parameter after the period extension processing, and the image template and the image to be processed are periodic signals in the process of fast Fourier transform and inverse fast Fourier transform, which is equivalent to the translation operation of the image template in the image to be processed in the matrix convolution operation process. Therefore, the value 0 is used to exclude the influence of the area with a gray value of 0 (black area) in the image to be processed on image processing, which can be understood as excluding the influence of the invalid area in the image to be processed, and then extracting the pixel point whose gray value needs to be configured as the preset target gray value, and accelerating the image inflation operation to set the pixel point to the pixel point with the preset target gray value, so as to realize the area increasing operation of the corresponding connected domain in the image to be processed; wherein the invalid area in the image to be processed is kept in the neighborhood range of the pixel point covered by the center of the image template corresponding to the convolution result element, and the center of the image template corresponding to the convolution result element is the center of the image template which actually performs convolution operation.

[0035] In summary, for the equivalent convolution results of the image template and the image to be processed after fast Fourier transform and inverse transform, when there is a pixel point with a preset search value in a specific neighborhood of the center of the image template (image region framed by the image template), the gray value of the center of the image template is configured as a preset target gray value. Compared with the assignment processing of various gray values mentioned in the background art, the present application does not need to assign values to the pixel points outside the boundary of the image to be processed, avoids excessive expansion of the boundary points of the image to be processed, reduces the amount of pixel processing data, and ensures the real-time performance of image processing.

[0036] As an embodiment, in the step B, if it is detected that the value of the convolution result element is greater than 0, it is determined that there is a pixel point with a preset search value in the image region framed by the image template in the image to be processed, and it is further determined that one of the convolution result elements obtained by the currently executed step A satisfies the preset gray value condition. It can be understood that the pixel point with the preset search value participates in the step A, and the convolution result element output by the step A satisfies the preset gray value condition. It can also be understood that when the pixel point with the preset search value is framed by the image template, one of the convolution results obtained after participating in the equivalent matrix convolution operation satisfies the preset gray value condition. If it is detected that the value of the convolution result element is not greater than 0, it is determined that there is no pixel point with a preset search value in the image region framed by the image template in the image to be processed, and it is further determined that the convolution result element output by the step A does not satisfy the preset gray value condition. The preset search value is set to a value greater than 0 or a gray value greater than 0 (generally, the preset search value is set to 255). When the image template is applied to the image dilation operation, the pixel point with the preset search value is scanned and obtained by convolution, which is necessary to form a connected domain, i.e. by setting the gray value of the pixel point covered by the center of the image template corresponding to the convolution result element to the preset target gray value, the pixel point can be merged with the pixel point with the preset search value existing in the neighborhood to become a pixel point of the same type, or the pixel point can form a white connected domain with the pixel point with the preset target gray value existing in the neighborhood, which can form the boundary of the image to be processed or smooth the original image boundary. Preferably, the gray value of the pixel point covered by the center of the image template corresponding to the convolution result element can be the preset search value, and then be configured as the preset target gray value, wherein the preset search value can be equal to the preset target gray value.

[0037] It should be noted that the image template corresponding to the convolution result element is a convolution template configured to support translation within the to-be-processed image for the purpose of matrix convolution operation. Notably, the definition of the image template here is in relation to the correspondence between the convolution result element and the pixel point participating in the aforementioned fast Fourier transform within the to-be-processed image. The image template does not undergo translation operation in this embodiment, specifically, it does not translate within the to-be-processed image to avoid the multiplication calculation of each small window required in the convolution operation.

[0038] Preferably, the image template is a convolution kernel in the form of a matrix, and the image template includes matrix elements with a value of 1 and matrix elements with a value of 0. The positions of the matrix elements with a value of 1 in the image template determine the pixel points in the to-be-processed image that actually participate in image processing. When the image template is used as a structural element, the elements with a value of 1 in the image template determine whether the pixel point covered by the center of the image template or the neighborhood pixel points of the pixel point covered by the element in the to-be-processed image need to participate in the calculation when performing dilation. The calculation complexity of the image template as a convolution kernel participating in the aforementioned matrix convolution operation or the aforementioned fast Fourier transform is simplified.

[0039] On the basis of the above embodiment, the shape formed by the matrix elements with a value of 1 in the image template is defined as the shape of the image template. The image template is a basic operator of morphology, and reasonable selection of the image template directly affects the effect and quality of image processing. The shape of the image template includes but is not limited to a circular shape, a square shape, a diamond shape, a hexagonal shape, and a line segment shape. The circular image template can obtain a direction-independent operation result due to isotropy, and the square shape and the diamond shape can be regarded as variations of the circular shape. The operation results of structural elements with different shapes will be different, and the structural element should be selected according to the geometric shape of the to-be-processed image.

[0040] Specifically, the image template as a convolution kernel has a shape that actually has a calculation effect, which is a partial region formed by the elements with a value of 1. Within the image template, if the elements with a value of 1 form a circular shape, the shape that actually has a calculation effect in the image template is a circular shape. If the elements with a value of 1 form a diamond shape within the image template, the shape that actually has a calculation effect in the image template is a diamond shape. Thus, whenever it is detected that the value of the convolution result element is greater than 0, the gray value of the pixel point covered by the center of the image template corresponding to the convolution result element within the to-be-processed image is set to the preset target gray value. Furthermore, after all the convolution result elements output in step A are traversed, the shape formed by connecting the pixel points with the preset target gray value within the to-be-processed image is the same as or adapted to the shape that actually has a calculation effect in the image template.

[0041] Specifically, the preset search value is greater than 0; or the mapping value converted by the preset pixel value function is 1, i.e. the mapping value is a logic value 1, so as to convert the to-be-processed image into a binary image, at this time, the gray values of all pixel points with a gray value greater than 0 in the to-be-processed image are mapped to logic 1, and the gray values of all pixel points with a gray value equal to 0 in the to-be-processed image are mapped to logic 0. In some embodiments, the to-be-processed image can be regarded as a two-dimensional array, and then the to-be-processed image is converted into a binary image, where the binary image is a two-dimensional logic array with only 0 and 1; at the same time, the image template is configured as a convolution kernel with all element values being 1, the center of the image template currently scans a specific pixel of the image, and the image template and the binary image covered thereby are subjected to an AND operation, if the result of the current output of the AND operation is 1, the pixel point currently covered by the center of the image template corresponds to the result of the AND operation, the result of the AND operation can reflect that there is an element with a value of 1 in the pixel point itself and its neighborhood currently covered by the center of the image template, and then the value of the pixel point currently covered by the center of the image template is further set to 1, so that the pixel point with a value of 1 in the binary image is increased by one; if the result of the current output of the AND operation is 0, the result of the AND operation can reflect that there is no element with a value of 1 in the image region currently framed by the image template (the pixel point itself and its neighborhood currently covered by the center of the image template), and the pixel point currently covered by the center of the image template does not need to be re-assigned, and the original gray value remains unchanged.

[0042] As an embodiment, the step A specifically comprises: transforming the image to be processed from a spatial domain to a frequency domain by using a two-dimensional discrete Fourier transform to obtain a first parameter matrix, wherein each matrix element in the first parameter matrix is a Fourier coefficient obtained by the transformation; it should be noted that the fast Fourier transform uses a two-dimensional discrete Fourier transform to process the image, which is a kind of matrix signal, by using a computer device. Similarly, the image template is transformed from the spatial domain to the frequency domain by using a two-dimensional discrete Fourier transform to obtain a second parameter matrix, wherein each matrix element in the second parameter matrix is a Fourier coefficient obtained by the transformation; the image template is a two-dimensional array and is used as a matrix signal for Fourier transform; it should be noted that, based on the result of the periodic extension processing involved in the Fourier transform process, the number of Fourier coefficients in the second parameter matrix is equal to the number of Fourier coefficients in the first parameter matrix. Then, each matrix element of the first parameter matrix is multiplied by the matrix element at the corresponding matrix position in the second parameter matrix to obtain a third parameter matrix, and it is worth noting that the multiplication of the matrix elements at this time is different from the multiplication between two matrices. Then, the third parameter matrix is transformed from the frequency domain to the spatial domain by using a two-dimensional discrete inverse Fourier transform to obtain a convolution result matrix, so as to realize the multiplication of the control image template and the image to be processed in the corresponding frequency domain to obtain the convolution result element in the corresponding spatial domain. Therefore, when the image convolution operation is required, the foregoing fast Fourier transform and its inverse transform are used to reduce the operation time of the matrix elements, so that real-time processing of the image signal is possible, mainly to reduce the operation amount and storage overhead, which is particularly advantageous for hardware implementation.

[0043] It should be noted that the convolution result elements in the corresponding spatial domain are arranged as a convolution result matrix, and are filled into the corresponding row and column positions of the convolution result matrix; the relative position of a convolution result element in the convolution result matrix is the same as the relative position of the center of the image template corresponding to the convolution result element in the image to be processed, and it is worth noting that the relative position is the position relationship of a matrix element relative to a fixed corner point position of the matrix, and the fixed corner point position preferably includes but is not limited to the pixel point at the top left corner of the image to be processed, the pixel point at the top right corner of the image to be processed, and the pixel point at the bottom left corner of the image to be processed, and the pixel point at the bottom right corner of the image to be processed; the image to be processed exists in the form of a pixel matrix, including the gray value of each pixel.

[0044] It should be noted that the foregoing two-dimensional discrete Fourier transform and its inverse transform is essentially an algorithm for converting a polynomial represented by a coefficient into a point value representation, thereby reducing the complexity of the calculation time from quadratic to logarithmic, and the reduction of the complexity of the calculation time is more obvious when the number of pixels processed is large.

[0045] In the foregoing embodiment, the image template is equivalently configured as: during the translation of the image template in the to-be-processed image, when the center of the image template covers one pixel point of the to-be-processed image, matrix convolution operation is performed between the elements existing in the image template itself and the pixel points in the image region currently framed by the image template, to obtain one convolution result element. The specific calculation steps of the matrix convolution operation refer to the foregoing embodiment, and will not be described herein. When the center of the image template traverses all pixel points in the to-be-processed image, all convolution result elements obtained through the matrix convolution operation form an output matrix, which is equal to the convolution result matrix. It should be noted that the configuration of the action of the image template in this embodiment is only used to illustrate the technical effect generated by the matrix convolution operation that can be performed between the image template and the to-be-processed image, and points out some technical features that can be used for two-dimensional discrete Fourier transform, and cannot represent that the image template actually translates in the related embodiments. Thus, a position matching relationship is established between each matrix element of the foregoing converted convolution result matrix and the pixel point participating in the fast Fourier transform in the to-be-processed image.

[0046] It should be noted that all elements in the image template are pre-configured; and all elements in the to-be-processed image are also pre-configured. The to-be-processed image can be a map, and then the matrix elements of the matrix of the map are the values at the coordinate positions of the map or the gray values of the map image at the corresponding pixel points. The image template can also be composed of a pixel matrix, and the elements in the image template are the gray values of the pixel points at the corresponding positions.

[0047] Preferably, when it is detected that the value of one convolution result element is greater than 0, the pixel point with the preset search value exists in the neighborhood of the pixel point covered by the center of the image template corresponding to the convolution result element in the to-be-processed image. The preferred example cannot determine that the pixel point with the preset search value is covered by the center of the image template in the to-be-processed image, but determines that one convolution result element obtained through the currently executed step A satisfies the preset gray value condition. It can be understood that the pixel point with the preset search value participates in step A, and the convolution result element output by the step A satisfies the preset gray value condition. It can also be understood that when the pixel point with the preset search value is framed by the image template, one convolution result obtained through the equivalent matrix convolution operation satisfies the preset gray value condition.

[0048] As an embodiment, each matrix element in the first parameter matrix is a Fourier coefficient described in the form of two-dimensional discrete Fourier transform, which can be understood as the result of periodical extension of the gray value of a relevant pixel point in the image to be processed; each matrix element in the second parameter matrix is a Fourier coefficient described in the form of two-dimensional discrete Fourier transform, which can be understood as the result of periodical extension of the value of a relevant element in the image template. In the process of the aforementioned fast Fourier transform and fast inverse Fourier transform, the number of rows of the third parameter matrix, the number of rows of the second parameter matrix and the number of rows of the first parameter matrix are equal, and at the same time, the number of columns of the third parameter matrix, the number of columns of the second parameter matrix and the number of columns of the first parameter matrix are equal; the number of rows of the final output convolution result matrix is equal to the number of rows of the pixel matrix corresponding to the image to be processed, and the number of columns of the final output convolution result matrix is equal to the number of columns of the pixel matrix corresponding to the image to be processed. Thus, the second parameter matrix and the first parameter matrix are combined by using the symmetry and periodicity of the exponential factor in the DFT calculation formula.

[0049] The whole process of the aforementioned two-dimensional discrete Fourier transform includes: first, calculating one-dimensional transform along each row of pixel points of the input image to transform an intermediate result, at this time the intermediate result forms an intermediate result sequence; then, calculating one-dimensional transform along each column of pixel points of the intermediate result sequence, and finally obtaining an output matrix. It should be noted that the above order can be changed, i.e., performing corresponding one-dimensional transform first by column and then by row. The one-dimensional transform here belongs to one-dimensional discrete Fourier transform applicable to periodic signals. Correspondingly, the above calculation and transform process is also applicable to the calculation of two-dimensional discrete inverse Fourier transform, except that the aforementioned one-dimensional transform is changed to one-dimensional inverse transform. In summary, the pixel matrix of the image to be processed and the image template are respectively decomposed into a series of short sequences in this embodiment, and the symmetry and periodicity of the exponential factor in the DFT calculation formula are fully utilized to further calculate the corresponding DFT of these short sequences and properly combine them, so as to delete repeated calculation, reduce multiplication operation and simplify image convolution calculation.

[0050] It is to be noted that in the process of processing the image to be processed or the image template by using one-dimensional discrete Fourier transform, the process is divided into two parts, each part is calculated by the same number of elements, one part is set as the odd part of the discrete Fourier transform, and the other part is set as the even part of the discrete Fourier transform, the sum of the odd part of the discrete Fourier transform and the even part of the discrete Fourier transform is one half of the Fourier coefficients of the discrete Fourier transform, and the difference between the odd part of the discrete Fourier transform and the even part of the discrete Fourier transform is the other half of the Fourier coefficients of the discrete Fourier transform. Therefore, in the frequency domain, the result of the discrete Fourier transform of one half of the matrix elements of the image to be processed is calculated each time the image to be processed is traced back, and the result of the discrete Fourier transform of the other half of the matrix elements of the image to be processed is calculated. Thus, the transformation calculation is reduced, the calculation complexity is reduced, the order of magnitude of the calculation amount before and after the transformation is reduced from the power of magnitude to the logarithm of magnitude, and the difference is obvious.

[0051] In summary, after the applicant inputs the image to be processed and the image template, the image processing method based on the convolution algorithm is repeatedly executed according to the foregoing embodiment; when the image to be processed and the image template are output by performing the convolution matrix operation to output the convolution result matrix, it takes about 1 second to accumulate, and the image processing method based on the convolution algorithm is executed according to the foregoing embodiment, and the convolution result matrix can be output in about 200 ms.

[0052] The application further discloses a chip which stores program code corresponding to the image processing method based on the convolution algorithm disclosed in the foregoing embodiment. The pixel processing data amount is reduced, and the real-time performance of the chip in processing images is improved.

[0053] As an embodiment, the chip integrates a convolution hardware accelerator, which is used to convert the matrix convolution operation between the image to be processed and the image template into a convolution operation between two sequences; each element in each sequence is a result of one-dimensional discrete Fourier transform of a row element (each element in a row of a matrix) of a corresponding matrix performed by the convolution hardware accelerator or a result of one-dimensional discrete Fourier transform of a column element (each element in a column of a matrix) of the corresponding matrix performed by the convolution hardware accelerator; each element in each sequence is obtained by parallel processing of the convolution hardware accelerator, one element is configured with one register to participate in one-dimensional discrete Fourier transform, different elements are configured with registers which are executed in parallel, and the transmission paths of each register are not interfered with each other. Thus, the speed of the convolution hardware accelerator in performing two-dimensional discrete Fourier transform on matrix signals is accelerated.

[0054] As an embodiment, the chip integrated convolution hardware accelerator is configured to, after multiplying the transform value of the image to be processed in the frequency domain with the transform value of the image template corresponding to the matrix convolution operation in the corresponding frequency domain and outputting a parameter matrix, first perform one-dimensional inverse discrete Fourier transform on each row element of the parameter matrix in parallel to obtain an intermediate result sequence, then perform one-dimensional inverse discrete Fourier transform on each column element of the intermediate result sequence in parallel, and finally obtain a convolution result matrix to complete two-dimensional inverse discrete Fourier transform; or, after multiplying the transform value of the image to be processed in the frequency domain with the transform value of the image template corresponding to the matrix convolution operation in the corresponding frequency domain and outputting a parameter matrix, first perform one-dimensional inverse discrete Fourier transform on each column element of the parameter matrix in parallel to obtain an intermediate result sequence, then perform one-dimensional inverse discrete Fourier transform on each row element of the intermediate result sequence in parallel, and finally obtain a convolution result matrix. One register is configured for inverse transform operation of each row element, the registers configured for different row elements are executed in parallel, and the signal transmission paths of each register do not interfere with each other; the registers configured for all row elements process corresponding input signals in parallel; one register is configured for inverse transform operation of each column element, and the registers configured for different column elements are executed in parallel, and the signal transmission paths of each register do not interfere with each other. The speed of the convolution hardware accelerator for performing two-dimensional inverse discrete Fourier transform on the matrix signal is accelerated.

[0055] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. These programs can be stored in a computer readable storage medium (such as ROM, RAM, magnetic disc or optical disc, and various media that can store program codes). When the program is executed, the steps of the above-mentioned method embodiments are executed. Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the above embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of image processing based on a convolution algorithm, characterized in that, The method comprises: Step A, using fast Fourier transform and inverse fast Fourier transform to control the image template and the image to be processed to multiply in the corresponding frequency domain to obtain the convolution result element in the corresponding spatial domain; The step A specifically comprises: using two-dimensional discrete Fourier transform to transform the image to be processed from the spatial domain to the frequency domain to obtain a first parameter matrix; using two-dimensional discrete Fourier transform to transform the image template from the spatial domain to the frequency domain to obtain a second parameter matrix; multiplying each matrix element of the first parameter matrix with the matrix element at the corresponding matrix position in the second parameter matrix to obtain a third parameter matrix; using two-dimensional discrete inverse Fourier transform to transform the third parameter matrix from the frequency domain to the spatial domain to obtain a convolution result matrix, so as to control the image template and the image to be processed to multiply in the corresponding frequency domain to obtain the convolution result element in the corresponding spatial domain; wherein the convolution result element in the corresponding spatial domain is arranged as a convolution result matrix; the relative position of one convolution result element in the convolution result matrix is the same as the relative position of the center of the image template corresponding to the convolution result element in the image to be processed; the relative position is the position relationship of one matrix element relative to the position of a fixed corner point of the matrix; Step B, if there is a convolution result element satisfying a preset gray value condition, setting the gray value of the pixel point corresponding to the convolution result element in the image to be processed to a preset target gray value; when the image template is applied to the image dilation operation, the pixel point with the gray value of the preset search value is scanned and obtained by convolution, which is necessary to form a connected domain; In the step B, it comprises: if it is detected that the value of the convolution result element is greater than 0, it is determined that there is a pixel point with the gray value of the preset search value in the image region framed by the image template corresponding to the convolution result element in the image to be processed, and then it is determined that one convolution result element obtained by the currently executed step A satisfies the preset gray value condition; wherein the image template corresponding to the convolution result element is a convolution template configured to support translation in the image to be processed for the purpose of matrix convolution operation; the image template is a convolution kernel existing in the form of a matrix, and the image template comprises matrix elements with a value of 1 and matrix elements with a value of 0, wherein the position feature of the matrix elements with a value of 1 distributed in the image template determines the pixel points actually participating in image processing in the image to be processed.

2. The image processing method of claim 1, wherein, The step B specifically comprises: if it is detected that the value of the convolution result element is greater than 0, setting the gray value of the pixel point covered by the center of the image template corresponding to the convolution result element in the image to be processed to the preset target gray value.

3. The image processing method of claim 2, wherein, In the image template, the shape composed of the matrix elements with a value of 1 is defined as the shape of the image template.

4. The image processing method of claim 2, wherein, The preset search value is greater than 0; or the mapping value calculated according to the preset pixel value function is 1.

5. The image processing method of claim 4, wherein, The image template is equivalently configured: during the translation of the image template in the to-be-processed image, when the center of the image template covers one pixel point of the to-be-processed image, matrix convolution operation is performed between the elements existing in the image template itself and the pixel points in the image region currently framed by the image template, and one convolution result element is obtained; After the center of the image template traverses all pixel points in the to-be-processed image, all convolution result elements obtained through matrix convolution operation form an output matrix, which is equal to the convolution result matrix; Wherein, all elements in the image template are pre-configured; all elements in the to-be-processed image are also pre-configured.

6. The image processing method of claim 5, wherein, When it is detected that the value of a convolution result element is greater than 0, the pixel point covered by the center of the image template corresponding to the convolution result element exists in the neighborhood of the pixel point with the preset search value in the to-be-processed image.

7. The image processing method of claim 4, wherein, Each matrix element in the first parameter matrix is a Fourier coefficient described in the form of two-dimensional discrete Fourier transform; Each matrix element in the second parameter matrix is a Fourier coefficient described in the form of two-dimensional discrete Fourier transform; The number of rows of the third parameter matrix, the number of rows of the second parameter matrix and the number of rows of the first parameter matrix are equal; The number of columns of the third parameter matrix, the number of columns of the second parameter matrix and the number of columns of the first parameter matrix are equal; The number of rows of the convolution result matrix is equal to the number of rows of the pixel matrix corresponding to the to-be-processed image; The number of columns of the convolution result matrix is equal to the number of columns of the pixel matrix corresponding to the to-be-processed image.

8. A chip, characterized by The chip stores program code corresponding to the image processing method based on the convolution algorithm in any one of claims 1 to 7; The chip integrates a convolution hardware accelerator, which is used to convert the matrix convolution operation between the to-be-processed image and the image template into convolution operation between two sequences; Wherein, each element in each sequence is the result of one-dimensional discrete Fourier transform of one row of elements of the corresponding matrix by the convolution hardware accelerator, or the result of one-dimensional discrete Fourier transform of one column of elements of the corresponding matrix by the convolution hardware accelerator; each element in each sequence is obtained through parallel processing by the convolution hardware accelerator.

9. The chip of claim 8, wherein, The chip integrates a convolution hardware accelerator, which is used to, after multiplying the transformed value of the to-be-processed image in the frequency domain with the transformed value of the image template corresponding to the matrix convolution operation in the corresponding frequency domain and outputting a parameter matrix, first perform one-dimensional inverse discrete Fourier transform on each row of elements of the parameter matrix in parallel to obtain an intermediate result sequence, then perform one-dimensional inverse discrete Fourier transform on each column of elements of the intermediate result sequence in parallel, and finally obtain a convolution result matrix to complete two-dimensional inverse discrete Fourier transform; Or, after multiplying the transform value of the image to be processed in the frequency domain with the transform value of the image template corresponding to the matrix convolution operation in the corresponding frequency domain and outputting a parameter matrix, the elements of each column of the parameter matrix are first subjected to one-dimensional inverse discrete Fourier transform in parallel to obtain an intermediate result sequence, then the elements of each row of the intermediate result sequence are subjected to one-dimensional inverse discrete Fourier transform in parallel, and finally a convolution result matrix is obtained.

Images