Image processing method and device, equipment and medium
By executing image processing methods in a vector processor, obtaining input matrix and output matrix, and performing convolution kernel determination operations, generating convolution kernels of different scales, the problems of code redundancy and difficulty in maintaining during convolution kernel processing in the prior art are solved, and efficient development and maintenance are achieved.
Patent Information
- Application Number
- CN202510277224.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-10
- Publication Date
- 2025-06-13
AI Technical Summary
When the prior art deals with convolution kernels of different scales in convolutional neural networks, different algorithms need to be written, resulting in tight development cycles, redundant codes and difficult to maintain.
By implementing the image processing method in a vector processor, obtaining the input matrix and output matrix, and performing a convolution kernel determination operation to generate convolution kernels of different sizes, the required convolution kernel is generated directly in the vector processor.
This method enables the vector processor to directly generate convolution kernels of different scales, reducing the amount of algorithm code, saving development time, and solving the problems of code redundancy and difficulty in maintaining.
Smart Images
Figure CN120147164A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of deep learning, and particularly to an image processing method, apparatus, device and medium. Background Art
[0002] In a Convolutional Neural Networks (CNN) architecture, the convolutional layer undertakes more than 90% of the overall computational load. Therefore, accelerating and optimizing the convolutional layer has become the key to improving network performance. The convolutional layer involves convolutional kernels (kernels) of various sizes such as 3x3, 4x4, 5x5, etc., and operations with different strides such as s1, s2, which constitute operators of different scales. Currently, when processing these convolutional kernels on a vector processor, it mainly relies on past experience and immediate user needs to implement each algorithm one by one, that is, different scales of convolutional kernels are pre-loaded into the vector processor through different algorithms for use. However, in the face of sudden demands and the lack of a pre-set algorithm to implement the corresponding convolutional kernel, it is often necessary to urgently organize a development team to write the corresponding algorithm, which not only leads to a tight development cycle, but also as the types of convolutional kernels increase, the amount of algorithm code also expands, making the entire project appear redundant and difficult to maintain. Summary of the Invention
[0003] In view of this, the purpose of the present invention is to provide an image processing method, apparatus, device and medium, which can enable a vector processor to directly generate convolutional kernels of different scales, solve the problem of writing different algorithms for different scales of convolutional kernels, reduce the amount of algorithm code, save development time, and effectively solve the problems of code redundancy and difficulty in maintenance. The specific solutions are as follows:
[0004] In a first aspect, the present application provides an image processing method applied to a vector processor, including:
[0005] When the target convolutional kernel does not exist in the vector processor, obtain an input matrix and an output matrix; the input matrix is a matrix obtained by pre-transforming an input image into a matrix; the output matrix is a matrix obtained by pre-convolving the input matrix with the target convolutional kernel;
[0006] Perform a convolutional kernel determination operation on the input matrix and the output matrix to obtain the target convolutional kernel, and process the image input into the vector processor through the target convolutional kernel;
[0007] Wherein, the convolutional kernel determination operation includes:
[0008] Based on a preset step size and the size of the target convolution kernel, determine the sliding positions of each element in the target convolution kernel on the input matrix, and based on the output matrix, the sliding positions of each element in the target convolution kernel on the input matrix, and the element values at the corresponding sliding positions in the input matrix, determine the element values of each element in the target convolution kernel, so as to determine the target convolution kernel based on the element values of each element in the target convolution kernel.
[0009] Optionally, the obtaining the input matrix and the output matrix includes:
[0010] Obtain the input matrix and the output matrix from the main board memory based on a direct memory access component, and store the input matrix in a scalar memory and the output matrix in an array memory; wherein, the scalar memory is located in the scalar processing unit of the vector processor; the array memory is located in the vector processing unit of the vector processor;
[0011] Correspondingly, after performing the convolution kernel determination operation on the input matrix and the output matrix to obtain the target convolution kernel, it further includes:
[0012] Store the target convolution kernel in the array memory.
[0013] Optionally, when the matrix size of the input matrix is greater than the maximum matrix size that the scalar memory can accommodate, obtaining the input matrix and the output matrix and performing the convolution kernel determination operation on the input matrix and the output matrix to obtain the target convolution kernel includes:
[0014] Based on the direct memory access component, sequentially obtain the input sub-matrices after the input matrix is partitioned from the main board memory, and obtain the output sub-matrices obtained by convolving the currently obtained input sub-matrices with the target convolution kernel in advance;
[0015] Perform the convolution kernel determination operation on the currently obtained input sub-matrix and the output sub-matrix to obtain an initial convolution kernel, and after all the input sub-matrices after the input matrix is partitioned have performed the convolution kernel determination operation, determine the target convolution kernel based on each of the initial convolution kernels.
[0016] Optionally, the input sub-matrix is a sub-matrix obtained by partitioning the input matrix using the maximum matrix size.
[0017] Optionally, when the number of channels of the input matrix and the output matrix is the same and greater than 1, the performing the convolution kernel determination operation on the input matrix and the output matrix to obtain the target convolution kernel includes:
[0018] Select the input data of a preset number of channels from the input matrix in sequence according to the channel order, and select the output data of the same channel as the currently selected input data from the output matrix;
[0019] Perform a convolution kernel determination operation on the input data of the preset number of channels currently selected and the output data of the same channel in parallel to obtain convolution kernels of a preset number of channels, and after the input data of all channels in the input matrix have been selected, determine the target convolution kernel based on the convolution kernels of all channels.
[0020] Optionally, the convolution kernel determination operation includes:
[0021] Use a preset matrix dimension and based on the size of the target convolution kernel, sequentially determine the elements participating in this time from the target convolution kernel; the preset matrix dimension is a dimension determined based on a preset number of matrix rows and matrix columns; the number of elements of the elements participating in this time is equal to the product result of the number of matrix rows and the number of matrix columns;
[0022] Based on the preset step size and the size of the target convolution kernel, determine the sliding position of the elements participating in this time on the input matrix;
[0023] According to the output matrix, the sliding position of the elements participating in this time on the input matrix, and the element values at the corresponding sliding positions in the input matrix, determine the element values of the elements participating in this time, and after determining the element values of all elements in the target convolution kernel, determine the target convolution kernel based on the element values of all elements in the target convolution kernel.
[0024] Optionally, when the height in the size of the target convolution kernel is not a multiple of the number of matrix rows or the width in the size of the target convolution kernel is not a multiple of the number of matrix columns, before using the preset matrix dimension and based on the size of the target convolution kernel to sequentially determine the elements participating in this time from the target convolution kernel, further include:
[0025] Perform zero-padding on the target convolution kernel using the preset matrix dimension to obtain a padded target convolution kernel; wherein, the height in the size of the padded target convolution kernel is a multiple of the number of matrix rows, and the width in the size of the padded target convolution kernel is a multiple of the number of matrix columns.
[0026] In a second aspect, the present application provides an image processing device, applied to a vector processor, including:
[0027] A matrix acquisition module, configured to acquire an input matrix and an output matrix when a target convolution kernel does not exist in the vector processor; the input matrix is a matrix obtained by pre-transforming an input image; the output matrix is a matrix obtained by pre-convolving the input matrix with the target convolution kernel;
[0028] A convolution kernel determination module, configured to perform a convolution kernel determination operation on the input matrix and the output matrix to obtain the target convolution kernel, and process an image input into the vector processor through the target convolution kernel;
[0029] Wherein, the convolution kernel determination operation includes:
[0030] Based on a preset step size and the size of the target convolution kernel, determine the sliding positions of each element in the target convolution kernel on the input matrix, and determine the element values of each element in the target convolution kernel according to the output matrix, the sliding positions of each element in the target convolution kernel on the input matrix, and the element values at the corresponding sliding positions in the input matrix, so as to determine the target convolution kernel based on the element values of each element in the target convolution kernel.
[0031] In a third aspect, the present application provides an electronic device, including:
[0032] A memory, configured to store a computer program;
[0033] A processor, configured to execute the computer program to implement the foregoing image processing method.
[0034] In a fourth aspect, the present application provides a computer-readable storage medium, configured to store a computer program, and when the computer program is executed by a processor, the foregoing image processing method is implemented.
[0035] In this application, when the target convolution kernel does not exist in the vector processor, the input matrix and the output matrix are obtained; the input matrix is a matrix obtained by pre-transforming the input image into a matrix; the output matrix is a matrix obtained by pre-convolving the input matrix with the target convolution kernel; a convolution kernel determination operation is performed on the input matrix and the output matrix to obtain the target convolution kernel, and the image input into the vector processor is processed by the target convolution kernel; wherein, the convolution kernel determination operation includes: based on a preset step size and the size of the target convolution kernel, determining the sliding positions of each element in the target convolution kernel on the input matrix, and according to the output matrix, the sliding positions of each element in the target convolution kernel on the input matrix, and the element values at the corresponding sliding positions in the input matrix, determining the element values of each element in the target convolution kernel, so as to determine the target convolution kernel based on the element values of each element in the target convolution kernel.
[0036] It can be seen that when the convolution kernel of the required scale does not exist in the vector processor in this application, the input matrix and the output matrix related to the convolution kernel of the required scale are obtained, wherein the output matrix is a matrix obtained by pre-convolving the input matrix with the convolution kernel of the required scale, and then the convolution kernel of the required scale can be obtained by performing a convolution kernel determination operation on the input matrix and the output matrix; in this way, the method proposed in this application can enable the vector processor to have the ability to directly generate convolution kernels of different scales, that is, for convolution kernels of different scales, the method proposed in this application can be implemented, solving the problem that different algorithms need to be written for convolution kernels of different scales, which is not only simple to operate, but also reduces the amount of algorithm code and saves a lot of development time, effectively solving the problems of code redundancy and difficulty in maintenance. Description of the Drawings
[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.
[0038] Figure 1 It is a schematic structural diagram of a vector processor disclosed in this application;
[0039] Figure 2 It is a flowchart of an image processing method disclosed in this application;
[0040] Figure 3 It is a schematic diagram of matrix partitioning disclosed in this application;
[0041] Figure 4 A schematic diagram of channel division disclosed in this application;
[0042] Figure 5 A schematic diagram of a sliding position disclosed in this application;
[0043] Figure 6 Another schematic diagram of a sliding position disclosed in this application;
[0044] Figure 7 Yet another schematic diagram of a sliding position disclosed in this application;
[0045] Figure 8 A schematic diagram of the structure of an image processing device disclosed in this application;
[0046] Figure 9 A structural diagram of an electronic device disclosed in this application. Detailed implementation manners
[0047] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.
[0048] When processing these convolution kernels on a vector processor, it mainly relies on past experience and immediate user needs to implement each algorithm one by one; however, in the face of sudden demands and the lack of a pre-set algorithm to implement the corresponding convolution kernel, it is often necessary to urgently organize a development team to write the corresponding algorithm, which not only leads to a tight development cycle, but also as the types of convolution kernels increase, the amount of algorithm code also expands accordingly, making the entire project appear redundant and difficult to maintain. Therefore, this application provides an image processing method, which can enable the vector processor to directly generate convolution kernels of different scales, solves the problem of writing different algorithms for convolution kernels of different scales, reduces the amount of algorithm code, saves development time, and effectively solves the problems of code redundancy and difficulty in maintenance.
[0049] A vector processor is an advanced computing architecture, and its structural diagram is as Figure 1As shown in the figure, the vector processor integrates a scalar processing unit (SPU, Standard Processing Unit) for performing scalar operations, a vector processing unit (VPU, Vector Processing Unit) for performing vector operations, and a direct memory access (DMA) component responsible for efficient data transfer, etc. The scalar processing unit is composed of a scalar processing component (SPE, Standard Processing Element) and a scalar memory (SM, Standard Memory), which is specifically used to process scalar data operations. The vector processing unit is composed of L vector processing components (VPE, Vector Processing Element) and an array memory (AM, Associative Memory). The L vector processing components operate collaboratively in a single instruction multiple data (SIMD, Single Instruction Multiple Data) manner, supporting the turning on and off of specified vector processing components, but not supporting data interaction between multiple vector processing components. Moreover, a single vector processing component can process 1 8-byte data at a time, such as FP64 (Floating Point 64, 64-bit floating point number), Int64 (64-bit integer), or can also process 2 4-byte data, such as FP32 (Floating Point 32, 32-bit floating point number), Int32 (32-bit integer). The direct memory access component plays an important role in data transfer in the vector processor. It is responsible for performing efficient data transfer between the scalar memory and the DDR (Double Data Rate, double rate) memory, and between the array memory and the DDR memory, generally transferring data in units of 1 byte.
[0050] Convolution is a basic operation in signal processing and image processing, which describes the interaction between two functions (or sequences). In image processing, the convolution operation usually refers to the process of multiplying and summing an image (or a part of the image) with a smaller matrix (i.e., the convolution kernel). The convolution kernel is a small matrix, usually with an odd number × odd number size (such as 3x3, 5x5, etc.), which is used to perform convolution operations on the image. The element values of the convolution kernel determine the effect of the convolution operation. Different convolution kernels can achieve different image processing effects, such as smoothing, sharpening, edge detection, etc.
[0051] See Figure 2 As shown in the figure, an embodiment of the present invention discloses an image processing method applied to a vector processor, including:
[0052] Step S11: When the target convolution kernel does not exist in the vector processor, obtain the input matrix and the output matrix; the input matrix is the matrix obtained by pre-transforming the input image into a matrix; the output matrix is the matrix obtained by pre-convolving the input matrix with the target convolution kernel.
[0053] In this embodiment, when the target convolution kernel of the required scale does not exist in the vector processor, obtain the input matrix obtained by pre-transforming the input image into a matrix, and obtain the output matrix obtained by pre-convolving the input matrix with the target convolution kernel of the required scale; that is, the input matrix and the output matrix are two matrices related to the target convolution kernel of the required scale and used to calculate the target convolution kernel of the required scale.
[0054] It should be noted that assume the height (number of rows) of the input matrix is Hi, and the width (number of columns) is Wi. The height of the target convolution kernel of the required scale is Hf, the width is Wf, the stride of the target convolution kernel moving up and down on the input matrix is stride_h, the stride of the target convolution kernel moving left and right on the input matrix is stride_w, and the number of padding steps at the left and right edges of the input matrix are left_padding and right_padding respectively, and the number of padding steps at the top and bottom edges of the input matrix are top_padding and bottom_padding respectively. The height (number of rows) of the output matrix is Ho, and the width (number of columns) is Wo. Therefore, the relationship among the input matrix, the output matrix, and the target convolution kernel is as follows:
[0055] Ho = [(top_padding + Hi + bottom_padding - Hf) / stride_h] + 1;
[0056] Wo = [(left_padding + Wi + right_padding - Wf) / stride_w] + 1.
[0057] Regarding the storage of the input matrix, the output matrix, and the target convolution kernel, the output matrix and the target convolution kernel can be stored in the vector processing unit of the vector processor, specifically in the array memory of the vector processing unit, and the input matrix is stored in the scalar processing unit of the vector processor, specifically in the scalar memory of the scalar processing unit. Based on this, it is necessary to pre-partition the array memory to store the output matrix and the convolution kernel in different storage areas.
[0058] Specifically, for the acquisition of the input matrix and the output matrix, the input matrix and the output matrix can be obtained from the main board memory based on the direct memory access component, and then the input matrix is stored in the scalar memory, and the output matrix is stored in the array memory.
[0059] Step S12: Perform a convolution kernel determination operation on the input matrix and the output matrix to obtain the target convolution kernel, and process the image input into the vector processor through the target convolution kernel; wherein, the convolution kernel determination operation includes: based on a preset step size and the size of the target convolution kernel, determine the sliding positions of each element in the target convolution kernel on the input matrix, and according to the output matrix, the sliding positions of each element in the target convolution kernel on the input matrix, and the element values at the corresponding sliding positions in the input matrix, determine the element values of each element in the target convolution kernel, so as to determine the target convolution kernel based on the element values of each element in the target convolution kernel.
[0060] In this embodiment, after obtaining the input matrix and the output matrix, the input matrix is broadcast from the scalar processing unit to the vector processing unit based on a broadcast instruction, and then a convolution kernel determination operation is performed on the input matrix and the output matrix to obtain a target convolution kernel of the required scale, and the target convolution kernel is stored in the array memory. Then, the target convolution kernel can be used to process the image input into the vector processor to achieve corresponding image processing effects, such as smoothing, sharpening, edge detection, etc.
[0061] Specifically, the convolution kernel determination operation includes: based on a preset step size and the size of the target convolution kernel, determine the sliding positions of each element in the target convolution kernel on the input matrix, and according to the output matrix, the sliding positions of each element in the target convolution kernel on the input matrix, and the element values at the corresponding sliding positions in the input matrix, determine the element values of each element in the target convolution kernel, so as to determine the target convolution kernel of the required scale based on the size of the target convolution kernel and the element values of each element in the target convolution kernel.
[0062] The convolution kernel determination operation in this application adopts a self-innovative General Direct Convolution Algorithm (GDC), so that the vector processor has the ability to directly generate convolution kernels of different scales through a unified algorithm, that is, convolution kernels of different scales can be implemented through a unified algorithm, solving the problem of writing different algorithms for convolution kernels of different scales, reducing the amount of algorithm code, and effectively preventing code redundancy.
[0063] It should be noted that since the storage space of the scalar memory is limited, it is necessary to determine whether to perform a block operation on the input matrix according to the size relationship between the size of the input matrix and the maximum matrix size that the scalar memory can accommodate.
[0064] When the matrix size of the input matrix is less than or equal to the maximum matrix size that can be accommodated by the scalar memory, the complete input matrix can be directly obtained from the main board memory based on the direct memory access component. Correspondingly, the complete output matrix corresponding to the input matrix is also obtained from the main board memory; then, the convolution kernel determination operation is directly performed on the complete input matrix and output matrix to obtain the target convolution kernel.
[0065] Exemplarily, when the maximum matrix size that can be accommodated by the scalar memory is 100×240, if the matrix size of the input matrix is a 6×6 matrix, since the matrix size of the input matrix is less than the maximum matrix size that can be accommodated by the scalar memory, the complete input matrix can be obtained from the main board memory at one time. If the matrix size of the input matrix is 120×120, since the matrix size of the input matrix is greater than the maximum matrix size that can be accommodated by the scalar memory, each input sub-matrix after the input matrix is divided into blocks needs to be obtained from the main board memory in multiple times.
[0066] When the matrix size of the input matrix is greater than the maximum matrix size that can be accommodated by the scalar memory, the input sub-matrices after the input matrix is divided into blocks are sequentially obtained from the main board memory based on the direct memory access component, and the output sub-matrices obtained by convolving the currently obtained input sub-matrices with the target convolution kernel in advance are obtained; then, the convolution kernel determination operation is performed on the currently obtained input sub-matrix and the output sub-matrix to obtain the initial convolution kernel, and after the convolution kernel determination operation has been performed on all the input sub-matrices after the input matrix is divided into blocks, the target convolution kernel is determined based on each initial convolution kernel.
[0067] That is, the first input sub-matrix after the input matrix is divided into blocks is obtained from the main board memory based on the direct memory access component, and the output sub-matrix obtained by convolving the first input sub-matrix with the target convolution kernel in advance is obtained, and then the convolution kernel determination operation is performed on the obtained first input sub-matrix and the corresponding output sub-matrix to obtain the initial convolution kernel. Then, the second input sub-matrix after the input matrix is divided into blocks is obtained from the main board memory based on the direct memory access component, and the output sub-matrix obtained by convolving the second input sub-matrix with the target convolution kernel in advance is obtained, and then the convolution kernel determination operation is performed on the obtained second input sub-matrix and the corresponding output sub-matrix to obtain the initial convolution kernel, and so on, until the convolution kernel determination operation has been performed on all the input sub-matrices after the input matrix is divided into blocks, and then the final target convolution kernel is determined based on the initial convolution kernels corresponding to each input sub-matrix respectively.
[0068] It should be noted that the input sub-matrix is a sub-matrix obtained by dividing the complete input matrix using the maximum matrix size that can be accommodated by the scalar memory, and the matrix size of the input sub-matrix is not greater than the maximum matrix size that can be accommodated by the scalar memory.
[0069] For the number of blocks of the input matrix, specifically, the number of rows and columns of the input matrix can be determined according to the size of the input matrix, and the maximum number of rows and the maximum number of columns can be determined according to the maximum matrix size that the scalar memory can accommodate. Then, the first ratio between the number of rows of the input matrix and the maximum number of rows is rounded up to obtain the first value, and the second ratio between the number of columns of the input matrix and the maximum number of columns is rounded up to obtain the second value. Finally, the product of the first value and the second value is the number of blocks of the input matrix. As Figure 3 shown, taking the size of the input matrix as 512×512 and the maximum matrix size that the scalar memory can accommodate as 100×240 as an example, the first value at this time is the integer part of 512 / 100 plus 1, that is, 6, and the second value is the integer part of 512 / 240 plus 1, that is, 3. Therefore, the total number of blocks of the input matrix is 6×3 = 18, that is, a total of 18 blocks can complete the processing of all input sub-matrices after the input matrix is blocked.
[0070] Specifically, the input matrix in the main board memory is blocked according to the number of blocks of the input matrix to obtain input sub-matrices with the corresponding number of blocks. Then, a current row index and a current column index are created and both are initialized to 0. Then, using the current row index and the current column index and based on the maximum matrix size that the scalar memory can accommodate, the corresponding input sub-matrix is obtained from the main board memory, and a convolution kernel determination operation is performed on the currently obtained input sub-matrix and the corresponding output sub-matrix to obtain the initial convolution kernel. Then, the maximum number of columns is added to the current column index to obtain a new current column index, and it is judged whether the current column index is not less than the number of columns of the input matrix. If the current column index is less than the number of columns of the input matrix, it jumps back to the step of obtaining the corresponding input sub-matrix from the main board memory using the current row index and the current column index and based on the maximum matrix size that the scalar memory can accommodate; if the current column index is not less than the number of columns of the input matrix, the maximum number of rows is added to the current row index to obtain a new current row index, and the current column index is reset to 0, and then it is judged whether the current row index is not less than the number of rows of the input matrix. If the current row index is less than the number of rows of the input matrix, it jumps back to the step of obtaining the corresponding input sub-matrix from the main board memory using the current row index and the current column index and based on the maximum matrix size that the scalar memory can accommodate until the current row index is not less than the number of rows of the input matrix, at which time it indicates that all input sub-matrices after the input matrix is blocked have been processed.
[0071] Furthermore, in order to improve the parallelism of the general direct convolution algorithm, when the input matrix is a multi-channel matrix in this application, the channel dimension is taken as the focus of parallelism, so that through the division of the channel dimension, the vector processor can process the data of multiple channels at one time, which can significantly improve the efficiency of the general direct convolution algorithm compared with processing the data of one channel at a time.
[0072] Specifically, when the number of channels of the input matrix and the output matrix is the same and greater than 1, a preset number of channels of input data are sequentially selected from the input matrix according to the channel order, and the output data of the same channel as the currently selected input data is selected from the output matrix; then, the convolution kernel determination operation is performed in parallel on the preset number of channels of input data and the output data of the same channel to obtain the convolution kernels of the preset number of channels. After all the input data of all channels in the input matrix are selected, the target convolution kernel is determined based on the convolution kernels of all channels. As Figure 4 shown, taking the number of channels of the input matrix as 32, that is, c == 32, and taking the example that two channels of data can be processed at a time, all the data of all channels in the input matrix can be processed in 16 times in total.
[0073] In this embodiment, for the convolution kernel determination operation, specifically, the preset matrix dimension is used and based on the size of the target convolution kernel, the elements participating in this time are sequentially determined from the target convolution kernel; among them, the preset matrix dimension is the dimension determined based on the preset number of matrix rows and matrix columns; the number of elements of the elements participating in this time is equal to the product result of the number of matrix rows and matrix columns; then, based on the preset step length and the size of the target convolution kernel, the sliding position of the elements participating in this time on the input matrix is determined, and according to the output matrix, the sliding position of the elements participating in this time on the input matrix, and the element values at the corresponding sliding positions in the input matrix, the element values of the elements participating in this time are determined. After all the element values of all the elements in the target convolution kernel are determined, the target convolution kernel is determined based on the size of the target convolution kernel and all the element values in the target convolution kernel.
[0074] In one case, the preset matrix dimension is 1×1, that is, each time the element participating in this time determined from the target convolution kernel is one. In this case, the general direct convolution algorithm proposed in this application can be implemented for input matrices and convolution kernels of any scale.
[0075] As Figure 5 and Figure 6As shown, taking the preset matrix dimension as 1×1, the input matrix as a 6×6 matrix, the target convolution kernel as a 2×2 matrix, and the preset stride as 2 as an example, using the preset matrix dimension and based on the size of the target convolution kernel, determine the first participating element from the target convolution kernel, that is, 0-0 in the target convolution kernel; then based on the preset stride and the size of the target convolution kernel, determine the sliding position of the first participating element on the input matrix, that is, the position where 0-0 is located in the input matrix, and determine the element value of the first participating element according to the output matrix, the sliding position of the first participating element on the input matrix, and the element values at the corresponding sliding positions in the input matrix. After that, using the preset matrix dimension and based on the size of the target convolution kernel, determine the second participating element from the target convolution kernel, that is, 0-1 in the target convolution kernel; then based on the preset stride and the size of the target convolution kernel, determine the sliding position of the second participating element on the input matrix, that is, the position where 0-1 is located in the input matrix, and determine the element value of the second participating element according to the output matrix, the sliding position of the second participating element on the input matrix, and the element values at the corresponding sliding positions in the input matrix. And so on, until the element values of all elements in the target convolution kernel are determined, then determine the target convolution kernel based on the size of the target convolution kernel and the element values of all elements in the target convolution kernel.
[0076] It should be noted that for the determination of the element value of any participating element in the target convolution kernel, the element value of any participating element can be obtained by accumulating the product results of the element values at the corresponding sliding positions in the input matrix corresponding to any participating element and the output matrix.
[0077] In another case, the preset matrix dimension is not limited to 1×1, that is, the number of matrix rows and / or matrix columns used to determine the preset matrix dimension is greater than 1. For example, the preset matrix dimension can be 1×2, 2×2, etc. At this time, the number of the current participating elements determined from the target convolution kernel each time is more than one, so that through parallel calculation of multiple elements, the calculation efficiency of the target convolution kernel can be improved.
[0078] Such as Figure 7As shown, taking the preset matrix dimension as 1×2, the input matrix as a 6×6 matrix, the target convolution kernel as a 2×2 matrix, and the preset stride as 2 as an example, using the preset matrix dimension and based on the size of the target convolution kernel, the first participating elements are determined from the target convolution kernel, that is, 0-0 and 0-1 in the target convolution kernel; then based on the preset stride and the size of the target convolution kernel, the sliding positions of each element in the first participating elements on the input matrix are determined, that is, the positions where 0-0 is located in the input matrix and the position where 0-1 is located in the input matrix, and according to the output matrix, the sliding positions of each element in the first participating elements on the input matrix, and the element values at the corresponding sliding positions in the input matrix, the element values of each element in the first participating elements are determined. And so on, until the element values of all elements in the target convolution kernel are determined, the target convolution kernel is determined based on the size of the target convolution kernel and the element values of all elements in the target convolution kernel.
[0079] However, considering that sometimes the preset matrix dimension does not match the size of the target convolution kernel, that is, the number of rows of the matrix used to determine the preset matrix dimension is not a multiple of the height in the size of the target convolution kernel, or the number of columns of the matrix used to determine the preset matrix dimension is not a multiple of the width in the size of the target convolution kernel. At this time, before successively determining the elements participating in this time from the target convolution kernel using the preset matrix dimension and based on the size of the target convolution kernel, it is necessary to first perform zero-padding on the target convolution kernel using the preset matrix dimension to obtain the padded target convolution kernel, so that the height in the size of the padded target convolution kernel is a multiple of the number of rows of the matrix, and the width in the size of the padded target convolution kernel is a multiple of the number of columns of the matrix.
[0080] For example, when the preset matrix dimension is 1×3 and the size of the target convolution kernel is 2×2, at this time, it is necessary to perform zero-padding on the target convolution kernel according to the preset matrix dimension to obtain the padded target convolution kernel, and the size of the padded target convolution kernel is 2×3. Then, using the preset matrix dimension and based on the size of the padded target convolution kernel, the elements participating in this time are successively determined from the padded target convolution kernel; then based on the preset stride and the size of the padded target convolution kernel, the sliding positions of the elements participating in this time on the input matrix are determined, and according to the output matrix, the sliding positions of the elements participating in this time on the input matrix, and the element values at the corresponding sliding positions in the input matrix, the element values of the elements participating in this time are determined, until the element values of all elements in the padded target convolution kernel are determined, and then based on the size of the target convolution kernel 2×2 and the element values of all elements in the padded target convolution kernel, the final required 2×2 scale target convolution kernel is determined. In this way, no matter what the preset matrix dimension is, through the zero-padding operation, the general direct convolution algorithm proposed in this application can be implemented for input matrices and convolution kernels of any scale.
[0081] It can be seen that when there is no convolution kernel of the required scale in the vector processor in this application, the input matrix and the output matrix related to the convolution kernel of the required scale are obtained, where the output matrix is the matrix obtained by convolving the input matrix with the convolution kernel of the required scale in advance. Then, the convolution kernel determination operation can be performed on the input matrix and the output matrix to obtain the convolution kernel of the required scale. In this way, through the method proposed in this application, the vector processor can be made to have the ability to directly generate convolution kernels of different scales, that is, for convolution kernels of different scales, the method proposed in this application can be used to achieve it, solving the problem that different algorithms need to be written for convolution kernels of different scales. It is not only simple to operate, but also reduces the amount of algorithm code, saves a lot of development time, and effectively solves the problems of code redundancy and difficulty in maintenance.
[0082] See Figure 8 As shown, an embodiment of the present invention discloses an image processing device applied to a vector processor, including:
[0083] A matrix acquisition module 11, configured to acquire an input matrix and an output matrix when there is no target convolution kernel in the vector processor; the input matrix is the matrix obtained by pre-converting the input image; the output matrix is the matrix obtained by convolving the input matrix with the target convolution kernel in advance;
[0084] A convolution kernel determination module 12, configured to perform a convolution kernel determination operation on the input matrix and the output matrix to obtain the target convolution kernel, and process the image input into the vector processor with the target convolution kernel;
[0085] Among them, the convolution kernel determination operation includes:
[0086] Based on a preset step size and the size of the target convolution kernel, determine the sliding position of each element in the target convolution kernel on the input matrix, and determine the element value of each element in the target convolution kernel according to the output matrix, the sliding position of each element in the target convolution kernel on the input matrix, and the element value at the corresponding sliding position in the input matrix, so as to determine the target convolution kernel based on the element value of each element in the target convolution kernel.
[0087] It can be seen that when there is no convolution kernel of the required scale in the vector processor in this application, the input matrix and the output matrix related to the convolution kernel of the required scale are obtained, where the output matrix is a matrix obtained by previously performing convolution on the input matrix with the convolution kernel of the required scale. Then, the convolution kernel determination operation can be performed on the input matrix and the output matrix to obtain the convolution kernel of the required scale. In this way, through the method proposed in this application, the vector processor can be made to have the ability to directly generate convolution kernels of different scales, that is, for convolution kernels of different scales, they can all be implemented through the method proposed in this application, solving the problem of needing to write different algorithms for different scales of convolution kernels. It is not only simple to operate, but also reduces the amount of algorithm code, saves a lot of development time, and effectively solves the problems of code redundancy and difficulty in maintenance.
[0088] In some specific embodiments, the matrix acquisition module 11 includes:
[0089] A matrix storage unit, configured to obtain the input matrix and the output matrix from the main board memory based on a direct memory access component, store the input matrix in a scalar memory, and store the output matrix in an array memory; wherein, the scalar memory is located in the scalar processing unit of the vector processor; the array memory is located in the vector processing unit of the vector processor;
[0090] Correspondingly, the image processing device further includes:
[0091] A convolution kernel storage unit, configured to store the target convolution kernel in the array memory.
[0092] In some specific embodiments, when the matrix size of the input matrix is greater than the maximum matrix size that the scalar memory can accommodate, the image processing device includes:
[0093] A sub-matrix acquisition unit, configured to sequentially obtain the input sub-matrices obtained by partitioning the input matrix from the main board memory based on the direct memory access component, and obtain the output sub-matrices obtained by previously performing convolution on the currently obtained input sub-matrices with the target convolution kernel;
[0094] A first convolution kernel determination unit, configured to perform a convolution kernel determination operation on the currently obtained input sub-matrix and the output sub-matrix to obtain an initial convolution kernel, and based on each of the initial convolution kernels, determine the target convolution kernel after all the input sub-matrices obtained by partitioning the input matrix have performed the convolution kernel determination operation.
[0095] In some specific embodiments, when the number of channels of the input matrix and the output matrix are the same and both are greater than 1, the convolution kernel determination module 12 includes:
[0096] A channel data selection unit, configured to sequentially select input data of a preset number of channels from the input matrix in the channel order, and select output data of the same channel as the currently selected input data from the output matrix;
[0097] A second convolution kernel determination unit, configured to perform a convolution kernel determination operation on the input data of the preset number of channels currently selected and the output data of the same channel in parallel, to obtain convolution kernels of the preset number of channels, and after all the input data of all channels in the input matrix have been selected, determine the target convolution kernel based on the convolution kernels of all channels.
[0098] In some specific embodiments, the convolution kernel determination operation in the convolution kernel determination module 12 includes:
[0099] An element participation determination unit, configured to sequentially determine the elements participating in this time from the target convolution kernel by using a preset matrix dimension and based on the size of the target convolution kernel; the preset matrix dimension is a dimension determined based on a preset number of matrix rows and matrix columns; the number of elements of the elements participating in this time is equal to the product result of the number of matrix rows and the number of matrix columns;
[0100] A sliding position determination unit, configured to determine the sliding position of the elements participating in this time on the input matrix based on the preset step size and the size of the target convolution kernel;
[0101] A third convolution kernel determination unit, configured to determine the element values of the elements participating in this time according to the output matrix, the sliding position of the elements participating in this time on the input matrix, and the element values at the corresponding sliding positions in the input matrix, and after determining the element values of all the elements in the target convolution kernel, determine the target convolution kernel based on the element values of all the elements in the target convolution kernel.
[0102] In some specific embodiments, when the height in the size of the target convolution kernel is not a multiple of the number of matrix rows or the width in the size of the target convolution kernel is not a multiple of the number of matrix columns, the convolution kernel determination operation in the convolution kernel determination module 12 further includes:
[0103] A zero-padding unit, configured to perform zero-padding on the target convolution kernel by using the preset matrix dimension to obtain a padded target convolution kernel; wherein, the height in the size of the padded target convolution kernel is a multiple of the number of matrix rows, and the width in the size of the padded target convolution kernel is a multiple of the number of matrix columns.
[0104] Furthermore, an embodiment of the present application also discloses an electronic device, Figure 9It is a structural diagram of an electronic device 20 shown according to an exemplary embodiment. The content in the figure should not be regarded as any limitation on the scope of use of this application.
[0105] Figure 9 This is a schematic structural diagram of an electronic device 20 provided by an embodiment of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the image processing method disclosed in any of the foregoing embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.
[0106] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows is any communication protocol applicable to the technical solution of this application, and no specific limitation is imposed on it here; the input / output interface 25 is used to obtain external input data or output data to the outside, and its specific interface type can be selected according to specific application needs, and no specific limitation is made here.
[0107] In addition, as a carrier for resource storage, the memory 22 may be a read-only memory, a random access memory, a magnetic disk, or an optical disc, etc. The resources stored thereon may include an operating system 221, a computer program 222, etc., and the storage method may be temporary storage or permanent storage.
[0108] Among them, the operating system 221 is used to manage and control each hardware device on the electronic device 20 and the computer program 222, and it may be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the image processing method executed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program that can be used to complete other specific tasks.
[0109] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the image processing method disclosed above. For the specific steps of this method, reference may be made to the corresponding content disclosed in the foregoing embodiments, and details are not repeated here.
[0110] In the present specification, the various embodiments are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the various embodiments, reference can be made to each other. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple. For the relevant parts, reference can be made to the description in the method section.
[0111] Those skilled in the art can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of the examples have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.
[0112] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be directly implemented by hardware, software modules executed by a processor, or a combination of the two. The software modules can be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field.
[0113] Finally, it should also be noted that in this document, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising a..." does not exclude the existence of additional identical elements in the process, method, article or device comprising the element.
[0114] The above has introduced the technical solutions provided by this application in detail. Specific examples are used herein to elaborate on the principles and implementation manners of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to this application.
Claims
1. An image processing method, characterized in that: Applications in vector processors include: When the target convolution kernel does not exist in the vector processor, an input matrix and an output matrix are obtained; the input matrix is a matrix obtained by performing matrix conversion on the input image in advance; the output matrix is a matrix obtained by convolving the input matrix with the target convolution kernel in advance; Performing a convolution kernel determination operation on the input matrix and the output matrix to obtain the target convolution kernel, and processing the image input into the vector processor by using the target convolution kernel; The convolution kernel determination operation includes: Based on a preset step size and the size of the target convolution kernel, the sliding position of each element in the target convolution kernel on the input matrix is determined, and according to the output matrix, the sliding position of each element in the target convolution kernel on the input matrix and the element value at the corresponding sliding position in the input matrix, the element value of each element in the target convolution kernel is determined to determine the target convolution kernel based on the element value of each element in the target convolution kernel.
2. The image processing method according to claim 1, characterized in that: The step of obtaining an input matrix and an output matrix comprises: Based on the direct memory access component, an input matrix and an output matrix are obtained from the mainboard memory, and the input matrix is stored in a scalar memory, and the output matrix is stored in an array memory; wherein the scalar memory is located in a scalar processing unit of the vector processor; and the array memory is located in a vector processing unit of the vector processor; Correspondingly, after performing the convolution kernel determination operation on the input matrix and the output matrix to obtain the target convolution kernel, the method further includes: The target convolution kernel is stored in the array memory.
3. The image processing method according to claim 2, characterized in that: When the matrix size of the input matrix is greater than the maximum matrix size that the scalar memory can accommodate, acquiring the input matrix and the output matrix, and performing a convolution kernel determination operation on the input matrix and the output matrix to obtain the target convolution kernel, including: Based on the direct memory access component, sequentially obtain the input sub-matrix after the input matrix is divided into blocks from the mainboard memory, and obtain the output sub-matrix obtained by convolving the currently obtained input sub-matrix with the target convolution kernel in advance; A convolution kernel determination operation is performed on the currently acquired input sub-matrix and the output sub-matrix to obtain an initial convolution kernel, and after all input sub-matrices after the input matrix is divided into blocks have performed the convolution kernel determination operation, the target convolution kernel is determined based on each of the initial convolution kernels.
4. The image processing method according to claim 3, characterized in that: The input sub-matrix is a sub-matrix obtained by dividing the input matrix into blocks using the maximum matrix size.
5. The image processing method according to claim 1, characterized in that: When the number of channels of the input matrix and the output matrix is the same and both are greater than 1, performing a convolution kernel determination operation on the input matrix and the output matrix to obtain the target convolution kernel includes: Selecting input data of a preset number of channels from the input matrix in sequence according to the channel order, and selecting output data of the same channel as the currently selected input data from the output matrix; The convolution kernel determination operation is performed in parallel on the input data of the preset number of channels currently selected and the output data of the same channel to obtain the convolution kernels of the preset number of channels, and after the input data of all channels in the input matrix are selected, the target convolution kernel is determined based on the convolution kernels of all channels.
6. The image processing method according to any one of claims 1 to 5, characterized in that: The convolution kernel determination operation includes: Determine the participating elements from the target convolution kernel in sequence by using a preset matrix dimension and based on the size of the target convolution kernel; the preset matrix dimension is a dimension determined based on a preset number of matrix rows and a preset number of matrix columns; the number of elements of the participating elements is equal to the product of the number of matrix rows and the number of matrix columns; Based on the preset step size and the size of the target convolution kernel, determining the sliding position of the participating element on the input matrix; The element value of the participating element is determined according to the output matrix, the sliding position of the participating element on the input matrix, and the element value at the corresponding sliding position in the input matrix. After determining the element values of all elements in the target convolution kernel, the target convolution kernel is determined based on the element values of all elements in the target convolution kernel.
7. The image processing method according to claim 6, characterized in that: When the height of the target convolution kernel size is not a multiple of the number of matrix rows or the width of the target convolution kernel size is not a multiple of the number of matrix columns, before determining the participating elements from the target convolution kernel in sequence by using the preset matrix dimension and based on the size of the target convolution kernel, the method further includes: The target convolution kernel is zero-filled using the preset matrix dimension to obtain a filled target convolution kernel; wherein the height of the size of the filled target convolution kernel is a multiple of the number of matrix rows, and the width of the size of the filled target convolution kernel is a multiple of the number of matrix columns.
8. An image processing device, characterized in that: Applications in vector processors include: A matrix acquisition module, used for acquiring an input matrix and an output matrix when the target convolution kernel does not exist in the vector processor; the input matrix is a matrix obtained by performing matrix conversion on the input image in advance; the output matrix is a matrix obtained by convolving the input matrix with the target convolution kernel in advance; A convolution kernel determination module, used to perform a convolution kernel determination operation on the input matrix and the output matrix to obtain the target convolution kernel, and process the image input into the vector processor through the target convolution kernel; The convolution kernel determination operation includes: Based on a preset step size and the size of the target convolution kernel, the sliding position of each element in the target convolution kernel on the input matrix is determined, and according to the output matrix, the sliding position of each element in the target convolution kernel on the input matrix and the element value at the corresponding sliding position in the input matrix, the element value of each element in the target convolution kernel is determined to determine the target convolution kernel based on the element value of each element in the target convolution kernel.
9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the image processing method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed by a processor, implements the image processing method according to any one of claims 1 to 7.