Image recognition method and device based on implicit general matrix multiplication

Abstract

The application discloses an image recognition method and device based on implicit general matrix multiplication, and the method comprises the following steps: acquiring dimension information of an expected output matrix according to structure parameters of an input image and structure parameters of a convolution kernel; sequentially taking original data points of N*N order in the expected output matrix as aggregated data points to acquire an aggregated output matrix corresponding to the expected output matrix; wherein N is a positive even number; taking data points in the aggregated output matrix as block base points, acquiring the expected output matrix based on implicit general matrix multiplication, and recognizing the input image according to the expected output matrix. The technical scheme of the embodiment of the application realizes the multiplexing of data loading when the input matrix reads data from the physical layer, reduces the data loading time, improves the calculation efficiency of the heterogeneous hardware accelerator in executing the convolution operation, and avoids the performance decline problem caused by the different data loading logics of the boundary points and the non-boundary points.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to an image recognition method and apparatus based on implicit universal matrix multiplication. Background Technology

[0002] With the continuous advancement of convolutional neural network technology, heterogeneous hardware accelerators have developed rapidly. As the most densely packed operator in heterogeneous hardware accelerator computation, the optimization of convolutional operators has become a top priority for improving the computing performance of heterogeneous hardware accelerators.

[0003] Implicit general matrix multiplication (igemm) is a commonly used convolution algorithm in heterogeneous hardware accelerators, and it is also the best performing algorithm under some convolution parameters (e.g., 2D convolution kernel size of 3×3 and stride of 2). igemm is based on the matrix block method in general matrix multiplication (gemm) combined with the im2col method. It performs an implicit transformation on the im2col process, and then uses the block method of gemm to optimize the convolution process on the transformed gemm matrix.

[0004] However, existing igemm algorithms lack optimization for the input data loading process, resulting in low data reuse rate and excessively long data loading time. Consequently, this leads to low computational efficiency and insufficient performance optimization of heterogeneous hardware accelerators. Summary of the Invention

[0005] This invention provides an image recognition method and apparatus based on implicit universal matrix multiplication to solve the problem of low computational efficiency when heterogeneous hardware accelerators perform convolution operations.

[0006] According to one aspect of the present invention, an image recognition method based on implicit universal matrix multiplication is provided, comprising:

[0007] Based on the structural parameters of the input image and the structural parameters of the convolution kernel, obtain the dimensionality information of the expected output matrix;

[0008] The original N×N data points in the desired output matrix are sequentially used as aggregated data points to obtain the aggregated output matrix corresponding to the desired output matrix; where N is a positive even number.

[0009] The data points in the aggregated output matrix are used as the block base points. The expected output matrix is ​​obtained based on implicit universal matrix multiplication, and the input image is recognized based on the expected output matrix.

[0010] The step of sequentially using the N×N original data points in the desired output matrix as aggregated data points to obtain the aggregated output matrix corresponding to the desired output matrix includes: obtaining the matching N value based on memory resource utilization. While making reasonable use of memory resources and avoiding excessive memory resource utilization due to an excessive number of threads, the efficiency of convolution operations on heterogeneous accelerators is improved through multi-threaded parallel computing.

[0011] The step of sequentially using the N×N original data points in the expected output matrix as aggregated data points to obtain the aggregated output matrix corresponding to the expected output matrix includes: if the expected output matrix, in addition to the aggregated data points, also includes remaining original data points that failed to aggregate, then determining whether the number of remaining original data points is greater than a first preset threshold; if the number of remaining original data points is greater than the first preset threshold, then padding the expected output matrix with zero elements to convert the remaining original data points into aggregated data points. While reducing the number of startup threads, each thread can use the aggregated data points disclosed in this embodiment as the basis for matrix partitioning during convolution operations, thereby achieving data reuse for loading the partitioned input matrix and reducing the amount of convolution operations within the thread.

[0012] After determining whether the number of remaining original data points is greater than or equal to a first preset threshold, the method further includes: if the number of remaining original data points is less than or equal to the first preset threshold, then based on the aggregated data points and the remaining original data points, obtaining the aggregated output matrix corresponding to the desired output matrix. Without inserting boundary elements to avoid additional computational overhead, the remaining original data points can still undergo convolution calculations based on the matrix block method of the existing igemm algorithm, ensuring the data integrity of the obtained desired output matrix.

[0013] The step of sequentially using the N×N original data points in the desired output matrix as aggregated data points to obtain the aggregated output matrix corresponding to the desired output matrix includes: if the desired output matrix contains remaining original data points that failed to aggregate successfully, in addition to the aggregated data points, the value of N is reduced to continue aggregating the remaining original data points until the value of N is reduced to 2 or there are no remaining original data points. By continuously reducing the value of N, the aggregation of original data points is maximized. In particular, when the value of N is reduced to 2, at most only 1 column and / or 1 row of original data points remain. At this time, except for the remaining 1 column and / or 1 row of original data points, the block input matrices in other threads have achieved data reuse, maximizing data loading efficiency and reducing a large number of invalid multiplication operations.

[0014] After further aggregating the remaining original data points by reducing the value of N, the process includes: when N is reduced to 2, if there are still remaining original data points, then the expected output matrix is ​​padded with zero elements to transform the remaining original data points into aggregated data points; or, based on the aggregated data points and the remaining original data points, an aggregated output matrix corresponding to the expected output matrix is ​​obtained. This achieves the aggregation of the remaining original data points with only a small increase in computational overhead, allowing the reuse of data loading from the block-based input matrix to be achieved in the thread containing the remaining original data points, thus improving data loading efficiency; or, without inserting boundary elements to avoid additional overhead, the remaining original data points can still be used for convolution calculations based on the existing igemm algorithm's matrix block method, ensuring the data integrity of the obtained expected output matrix.

[0015] The process of using data points in the aggregated output matrix as block base points, obtaining the desired output matrix based on implicit universal matrix multiplication, and recognizing the input image according to the desired output matrix includes: fusing the block input matrices corresponding to the aggregated data points in the aggregated output matrix according to the structural parameters of the convolution kernel to obtain a fused block input matrix; and obtaining the pixel values ​​of the corresponding aggregated data points based on the block input fusion matrix and the convolution kernel. This significantly reduces the amount of data computation, avoids a large amount of invalid computation (i.e., multiplying 0 values ​​with the convolution kernel), and further improves the computational efficiency of heterogeneous hardware accelerators.

[0016] According to another aspect of the present invention, an image recognition device based on implicit universal matrix multiplication is provided, comprising:

[0017] The dimension information acquisition module is used to obtain the dimension information of the expected output matrix based on the structural parameters of the input image and the structural parameters of the convolution kernel.

[0018] The data aggregation execution module is used to sequentially take the original N×N data points in the desired output matrix as aggregation data points to obtain the aggregated output matrix corresponding to the desired output matrix; where N is a positive even number.

[0019] The image recognition execution module is used to take the data points in the aggregated output matrix as the block base points, obtain the expected output matrix based on implicit universal matrix multiplication, and recognize the input image according to the expected output matrix.

[0020] According to another aspect of the present invention, a heterogeneous hardware accelerator is provided for executing the image recognition method based on implicit universal matrix multiplication as described in any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the image recognition method based on implicit universal matrix multiplication as described in any embodiment of the present invention.

[0022] The technical solution of this invention, based on the dimension information of the expected output matrix, sequentially uses the original N×N data points in the expected output matrix as aggregated data points to obtain the aggregated output matrix corresponding to the expected output matrix. Then, the data points in the aggregated output matrix are used as block base points, and the expected output matrix is ​​obtained based on implicit universal matrix multiplication. This enables the reuse of loaded data when the input matrix reads data from the physical layer, reduces data loading time, improves the computational efficiency of convolution operations performed by heterogeneous hardware accelerators, and avoids the performance degradation problem caused by different data loading logic between boundary points and non-boundary points due to encoding branches.

[0023] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1A This is a flowchart of an image recognition method based on implicit universal matrix multiplication provided in Embodiment 1 of the present invention;

[0026] Figure 1B This is a schematic diagram of the aggregation process of raw data points provided in Embodiment 1 of the present invention;

[0027] Figure 1C This is a schematic diagram of loading input data before optimization of the convolution operator according to Embodiment 1 of the present invention;

[0028] Figure 1D This is a schematic diagram showing the location of the data repetition loading points of the input data before the convolution operator optimization provided in Embodiment 1 of the present invention;

[0029] Figure 1E This is a schematic diagram of loading input data after optimization of the convolution operator according to Embodiment 1 of the present invention;

[0030] Figure 1FThis is a flowchart of obtaining the block input fusion matrix according to Embodiment 1 of the present invention;

[0031] Figure 2 This is a flowchart of an image recognition method based on implicit universal matrix multiplication according to Embodiment 2 of the present invention;

[0032] Figure 3 This is a flowchart of an image recognition method based on implicit universal matrix multiplication provided in Embodiment 3 of the present invention;

[0033] Figure 4 This is a schematic diagram of the structure of an image recognition device based on implicit universal matrix multiplication according to Embodiment 4 of the present invention. Detailed Implementation

[0034] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0035] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0036] Example 1

[0037] Figure 1A This is a flowchart of an image recognition method based on implicit universal matrix multiplication provided in Embodiment 1 of the present invention. This embodiment is applicable to improving the convolution operation efficiency of heterogeneous hardware accelerators by optimizing the convolution operator of the igemm algorithm. This method can be executed by an image recognition device based on implicit universal matrix multiplication, which can be implemented in hardware and / or software and can be configured in a heterogeneous hardware accelerator. Figure 1A As shown, the method includes:

[0038] S101. Obtain the dimension information of the desired output matrix based on the structural parameters of the input image and the structural parameters of the convolution kernel.

[0039] After acquiring the image to be recognized, the Central Processing Unit (CPU) transmits the image to a heterogeneous hardware accelerator for image recognition-related data processing. When the heterogeneous hardware accelerator performs convolution operations on the image using the igemm algorithm, it first obtains the number of rows and columns of the desired output matrix based on the structural parameters of the input image and the structural parameters of the convolution kernel. The heterogeneous hardware accelerator may include various functional components such as a Graphics Processing Unit (GPU). In this embodiment of the invention, the type of heterogeneous hardware accelerator is not specifically limited.

[0040] As shown in Table 1, the structural parameters of the input image include its height, width, number of channels, and batch size. The height of the input image represents the number of pixels in the vertical direction, the width represents the number of pixels in the horizontal direction, and the batch size represents the number of images in the batch. Black and white images have 1 channel, and RGB color images have 3 channels. The structural parameters of the input image also include the number of padding edges in the height direction and the number of padding edges in the width direction. The structural parameters of the convolution kernel include the number of kernels, their height and width, the stride of the convolution kernel in the height direction of the input image, and the stride of the convolution kernel in the width direction of the input image.

[0041] Table 1. Structural parameters of the input image and convolution kernel.

[0042] Structural parameters logo Structural parameters logo The height of the input image h Batch size of input images n The width of the input image w Number of channels in the input image c Number of convolution kernels k Number of padding edges in the height direction of the input image <![CDATA[t1]]> The height of the convolution kernel r Number of padding edges in the width direction of the input data <![CDATA[t2]]> width of the convolution kernel s The stride of the convolution kernel in the height direction of the input image <![CDATA[d1]]> The stride of the convolution kernel in the width direction of the input image <![CDATA[d2]]>

[0043] Wherein, the number of channels of the convolution kernel = the number of channels of the input image c; the number of output data = the number of convolution kernels k; the height of the output data p = (h + 2 * t1 - r) / d1 + 1; the width of the output data q = (w + 2 * t2 - s) / d2 + 1.

[0044] Based on the above structural parameters, in the igemm algorithm, the input image is transformed into an input matrix M (n*p*q, r*s*c), and the convolution kernel is transformed into a convolution matrix N (r*s*c, k). Thus, the number of rows and columns corresponding to the output data (i.e. the expected output matrix) can be obtained as n*p*q ​​and k, respectively. The expected output matrix is ​​the target matrix to be obtained, which reflects the image features after the convolution operation.

[0045] S102. Sequentially use the original N×N data points in the desired output matrix as aggregated data points to obtain the aggregated output matrix corresponding to the desired output matrix; where N is a positive even number.

[0046] like Figure 1B As shown, the left side is a schematic diagram of the data points in the desired output matrix. Each square in the left-hand diagram represents a data point (i.e., an original data point) in the desired output matrix. Starting from the top left corner of the matrix, N×N adjacent data points are grouped together as a single data point; where N is a positive integer multiple of 2. Figure 1B The example shown is a 2×2 adjacent data point; the right side is a schematic diagram of the data points in the aggregated output matrix, where each square on the right represents a data point in the aggregated output matrix (i.e., an aggregated data point).

[0047] S103. Using the data points in the aggregated output matrix as the block base points, the expected output matrix is ​​obtained based on implicit universal matrix multiplication, and the input image is recognized according to the expected output matrix.

[0048] In the igemm algorithm, the input image is divided into blocks based on the matrix in the aggregated output moment. When the convolution kernel on each channel slides, the corresponding position is stretched into a one-dimensional vector. Then, each slide of the batch size, width, and height of the input image is concatenated into a larger input matrix M (n*p*q, r*s*c). The above transformation is only at the logical level, and its physical storage address remains unchanged. Therefore, the input matrix M only exists at the logical level, and the input matrix M still needs to read the data of each data point from the corresponding physical location.

[0049] Taking a 3×3 2D convolution kernel with a stride of 2 for backpropagation, and an input layout in nchw mode (i.e., taking data in the c-direction first, then the w-direction, then the h-direction, and finally the n-direction) as an example; according to the basic principle of backpropagation, it is necessary to insert blank rows and columns with a stride of 1 into the input data and convert it into a convolution operation with a stride of 1. In the actual execution of the convolution operation, it is necessary to slide two steps in both the height and width directions of the input data. Taking the above technical solution as an example, the input matrix corresponding to the input image is as follows:

[0050]

[0051] Taking the sliding of the convolution slider between data points M11, M12, M21, and M22 as an example; Figure 1C This is a schematic diagram illustrating the input data loading process in the existing iGeMM algorithm; each time the convolution slider slides, data needs to be read from 9 points in the input data, i.e., ... Figure 1C Read the data within the shaded area, then for Figure 1CFor the four adjacent data points M11, M12, M21, and M22, the above data loading operation requires a total of 4 × 9 = 36 data points to be read from the physical layer, that is, a total of 36 data points need to be loaded; the corresponding number of times the input data is multiplied with the convolution kernel is also 36, that is, the data of each point needs to be multiplied with the corresponding data in the convolution kernel, so a total of 36 data multiplication calculations are performed.

[0052] In fact, during the aforementioned sliding process of the convolution slider, as Figure 1D The area enclosed by the dashed line represents the points in the input data that are repeatedly loaded during the sliding operation of the convolution slider. In existing igemm algorithms, due to boundary issues—that is, the data loading logic for boundary points differs from that for non-boundary points—these repeatedly loaded data cannot be removed. Improper boundary handling during parallel computation on heterogeneous hardware accelerators can lead to a sharp performance drop. Furthermore, because existing igemm algorithms use a single original data point as the basis for matrix partitioning, they cannot resolve the inconsistency in data loading logic between boundary and non-boundary data points. This inevitably leads to encoding decision branches, i.e., determining whether the current data point is a boundary or non-boundary point. Such decision logic significantly reduces the computational performance of heterogeneous hardware accelerators. Therefore, existing igemm algorithms cannot reuse loaded data and must rely on the aforementioned method of reading all data.

[0053] In this embodiment of the invention, when N×N aggregated data points are used as the base points of the block matrix, there are no boundary data points when the convolution slider slides in the input image. The heterogeneous hardware accelerator does not need to execute a branch to determine whether the current data point is a boundary or non-boundary data point. Each aggregated data point is executed with the same data reading logic, thus providing a basis for data reuse. Figure 1E As shown, in this embodiment of the invention, the input data only needs to load the data of the above four data points M11, M12, M21, and M22, and the rest are 0 values. Compared with the existing igemm algorithm, which requires a total of 36 data points to be loaded, this obviously greatly reduces the total amount of data loaded from the physical layer for the input matrix M.

[0054] When the convolution kernel is transformed into a convolution matrix N(r*s*t, k) based on the block-based method in the expected output matrix, the transformation is also at the logical level, and its physical storage address remains unchanged. Therefore, the convolution matrix N only exists at the logical level, and it also needs to read the data of each data point from the corresponding physical location. Then, the heterogeneous hardware accelerator performs convolution operation on each block input matrix with the corresponding block convolution matrix through multi-threaded parallel computing. The result is the corresponding block output matrix. The block output matrices from each thread are then concatenated to form the complete expected output matrix. Finally, based on the image features reflected by the expected output matrix, the input image is identified and classified to obtain the specific type of the input image.

[0055] Optionally, in this embodiment of the invention, the step of using the data points in the aggregated output matrix as block base points, obtaining the expected output matrix based on implicit universal matrix multiplication, and recognizing the input image according to the expected output matrix includes: fusing the block input matrices corresponding to the aggregated data points in the aggregated output matrix according to the structural parameters of the convolution kernel to obtain a fused block input fusion matrix; and obtaining the pixel values ​​of the corresponding aggregated data points according to the block input fusion matrix and the convolution kernel.

[0056] Specifically, taking the above technical solution as an example, after the input matrix has loaded the data, the product method for convolution operation between it and the convolution kernel is as follows: Figure 1F As shown, in the first data loading graph, after the input matrix loads the data, the data points that need to be multiplied with the convolution kernel are those numbered 1, 3, 7, and 9. The remaining positions are all 0 values ​​and do not need to be multiplied with the convolution kernel; even if they were, the result would be 0, not affecting the final summation. In the second data loading graph, after loading the data, the data points that need to be multiplied with the convolution kernel are those numbered 2 and 8. In the third data loading graph, after loading the data, the data points that need to be multiplied with the convolution kernel are those numbered 4 and 6. In the fourth data loading graph, after loading the data, the data points that need to be multiplied with the convolution kernel are... The kernel performs the product operation on the data at position 5. After loading the above data, the data points are fused into a 3×3 input fusion matrix with the same dimension as the currently used convolution kernel. Only 9 points in the input fusion matrix need to be multiplied with the corresponding data in the convolution kernel, that is, only 9 product operations are required. Compared with the above technical solution, the existing igemm algorithm requires 36 product operations, which obviously greatly reduces the amount of data computation, avoids a lot of invalid computation (i.e., multiplying 0 values ​​with the convolution kernel), and further improves the computational efficiency of heterogeneous hardware accelerators.

[0057] Optionally, in this embodiment of the invention, the step of sequentially using the N×N original data points in the desired output matrix as aggregated data points to obtain the aggregated output matrix corresponding to the desired output matrix includes: obtaining a matching N value based on the memory resource occupancy rate. Specifically, the heterogeneous hardware accelerator performs parallel computation on each block matrix by starting multiple threads. The smaller the value of N, the more threads are started, the less data is loaded in each thread, the less convolution computation is required, and the higher the data processing efficiency. However, the more threads there are, the more memory resources are required. Therefore, if the current memory resource occupancy rate is high, it indicates that the available memory resources are low, so N is set to a larger value, such as 6 and 8; if the current memory resource occupancy rate is low, it indicates that the available memory resources are high, so N is set to a smaller value, such as 2 and 4. Thus, the matching N value can be obtained based on the value range of the memory resource occupancy rate, so as to reasonably utilize memory resources and avoid excessive memory resource occupancy caused by too many threads, while improving the convolution operation efficiency of the heterogeneous accelerator through multi-threaded parallel computation.

[0058] The technical solution of this invention, based on the dimension information of the expected output matrix, sequentially uses the original N×N data points in the expected output matrix as aggregated data points to obtain the aggregated output matrix corresponding to the expected output matrix. Then, the data points in the aggregated output matrix are used as block base points, and the expected output matrix is ​​obtained based on implicit universal matrix multiplication. This enables the reuse of loaded data when the input matrix reads data from the physical layer, reduces data loading time, improves the computational efficiency of convolution operations performed by heterogeneous hardware accelerators, and avoids the performance degradation problem caused by different data loading logic between boundary points and non-boundary points due to encoding branches.

[0059] Example 2

[0060] Figure 2 This is a flowchart of an image recognition method based on implicit universal matrix multiplication provided in Embodiment 2 of the present invention. The relationship between this embodiment and the above embodiments is that the output matrix is ​​expected to include the remaining original data points that failed to be aggregated. Figure 2 As shown, the method includes:

[0061] S201. Obtain the dimension information of the desired output matrix based on the structural parameters of the input image and the structural parameters of the convolution kernel; execute S202.

[0062] S202. Sequentially use the original N×N data points in the desired output matrix as aggregated data points to obtain the aggregated output matrix corresponding to the desired output matrix; where N is a positive even number; execute S203.

[0063] S203. If the expected output matrix includes, in addition to the aggregated data points, remaining original data points that failed to be aggregated, then determine whether the number of remaining original data points is greater than a first preset threshold; if yes, execute S204; if no, execute S205.

[0064] S204. Complete the boundary of the desired output matrix by using zero elements to transform the remaining original data points into aggregated data points; execute S206.

[0065] If the number of rows and / or columns of the expected output matrix is ​​not a positive integer multiple of the current value N, then for the expected output matrix, there are remaining original data points that have not been fully aggregated at its right and / or lower boundaries. When the number of remaining original data points is large, if each original data point is retained as an independent data point, that is, if the remaining original data points are also treated as aggregated data points, and each remaining original data point is convolved by a separate thread, not only will a large number of threads be started, consuming excessive memory resources, but each thread corresponding to the remaining original data points will also have issues with loading data that cannot be reused and convolution. The computational complexity of the operation is relatively large. However, by using boundary completion, only a small number of zero elements need to be inserted to combine the remaining original data points into new aggregated data points. This reduces the number of threads that need to be started. When each thread performs convolution operations, it can use the aggregated data points disclosed in this embodiment as the basis for matrix partitioning, thereby realizing the reuse of loading data for the block input matrix and reducing the amount of convolution operations within the thread. For example, when N is 8, if there are 7 columns of original data points remaining, only 1 column of zero elements needs to be inserted on the right boundary to combine the above 7 columns of original data points with the newly inserted 1 column of zero elements to form aggregated data points.

[0066] S205. Based on the aggregated data points and the remaining original data points, obtain the aggregated output matrix corresponding to the desired output matrix; execute S206.

[0067] When the number of remaining original data points is small, if the value of N is large, a large number of zero-element boundary points need to be inserted to complete the boundary completion. For example, in the above technical solution, when the value of N is 8, if there is 1 column of original data points remaining, 7 columns of zero elements need to be inserted on the right boundary to form an aggregated data point with the 1 column of remaining original data points and the newly inserted 7 columns of zero elements. Inserting a large number of zero-element boundary points undoubtedly increases the extra overhead in the calculation process. At the same time, since the number of remaining original data points is small, the number of threads that need to be started is also small. The remaining original data points can be directly retained as an independent data point, that is, the remaining original data points are also regarded as aggregated data points, and convolution calculation is performed by an independent thread. Without inserting boundary elements to avoid increasing the extra computational overhead, the remaining original data points can still be convolved based on the matrix block method of the existing igemm algorithm, ensuring the data integrity of the obtained expected output matrix.

[0068] S206. Using the data points in the aggregated output matrix as the block base points, the expected output matrix is ​​obtained based on implicit universal matrix multiplication, and the input image is recognized according to the expected output matrix.

[0069] In the technical solution of this invention embodiment, when the expected output matrix includes, in addition to the aggregated data points, remaining original data points that failed to aggregate, if the number of remaining original data points is large, the expected output matrix is ​​padded with zero elements. This reduces the number of threads started, and each thread can reuse the loading data of the block input matrix during convolution operations, reducing the amount of convolution operations within the thread. If the number of remaining original data points is small, the aggregated output matrix corresponding to the expected output matrix is ​​obtained based on the aggregated data points and the remaining original data points. Without inserting boundary elements to avoid additional computational overhead, the remaining original data points can still be convolved based on the matrix block method of the existing igemm algorithm, ensuring the data integrity of the obtained expected output matrix.

[0070] Example 3

[0071] Figure 3 This is a flowchart of an image recognition method based on implicit universal matrix multiplication provided in Embodiment 3 of the present invention. The relationship between this embodiment and the above embodiments is that when the desired output matrix includes, in addition to the aggregated data points, remaining original data points that failed to be aggregated, the value of N is reduced to continue aggregating the remaining original data points. Figure 3 As shown, the method includes:

[0072] S301. Obtain the dimension information of the desired output matrix based on the structural parameters of the input image and the structural parameters of the convolution kernel.

[0073] S302. Sequentially use the original N×N data points in the desired output matrix as aggregated data points.

[0074] S303. If the expected output matrix includes, in addition to the aggregated data points, remaining original data points that failed to be aggregated, then by reducing the value of N, the remaining original data points are aggregated until the value of N is reduced to 2 or there are no remaining original data points.

[0075] For example, when the initial value of N is 8, if 7 columns of original data points remain, the value of N can be reduced to 4, and aggregation can continue on the remaining 7 columns. After aggregation again, 3 columns of original data points remain, and the value of N is reduced to 2, and aggregation can continue on the remaining 3 columns. After aggregation again, only 1 column of original data points remains, and the value of N has been reduced to 2, at which point aggregation stops. Alternatively, when the initial value of N is 8, if 4 columns of original data points remain, the value of N can also be reduced to 4, and aggregation can continue on the remaining 4 columns, exactly completing the aggregation of all remaining original data points, at which point aggregation stops. By continuously reducing the value of N, the aggregation of original data points is maximized. In particular, when the value of N is reduced to 2, at most only 1 column and / or 1 row of original data points remain. At this point, apart from the aforementioned remaining 1 column and / or 1 row of original data points, the block input matrices in other threads achieve data reuse, maximizing data loading efficiency and reducing a large number of invalid multiplication operations.

[0076] S304. When the value of N is reduced to 2, if there are still remaining original data points, the expected output matrix is ​​filled with zero elements to transform the remaining original data points into aggregated data points, or an aggregated output matrix corresponding to the expected output matrix is ​​obtained based on the aggregated data points and the remaining original data points.

[0077] As described in the above technical solution, when the value of N is reduced to 2, at most only 1 column and / or 1 row of original data points remain. At this time, only 1 column and / or 1 row of zero elements need to be inserted to achieve aggregation of the remaining original data points with only a small amount of additional computational overhead. This allows the thread containing the remaining original data points to also achieve data reuse for loading the block input matrix, improving data loading efficiency. Alternatively, the remaining 1 column and / or 1 row of original data points can be retained, treating them as aggregated data points and performing convolution calculations in an independent thread. Without inserting boundary elements to avoid additional overhead, the remaining original data points can still be used for convolution calculations based on the existing igemm algorithm's matrix block method, ensuring the data integrity of the obtained desired output matrix.

[0078] S305. Using the data points in the aggregated output matrix as the block base points, the expected output matrix is ​​obtained based on implicit universal matrix multiplication, and the input image is recognized according to the expected output matrix.

[0079] The technical solution of this invention addresses the situation where, in addition to the aggregated data points, the output matrix also includes remaining original data points that failed to aggregate. By continuously reducing the value of N, the original data points are aggregated to the maximum extent. In particular, when the value of N is reduced to 2, at most only 1 column and / or 1 row of original data points remain. At this time, apart from the aforementioned remaining 1 column and / or 1 row of original data points, the block input matrices in other threads have all achieved data reuse, maximizing data loading efficiency and reducing a large number of invalid multiplication operations.

[0080] Example 4

[0081] Figure 4 This is a structural block diagram of an image recognition device based on implicit universal matrix multiplication provided in Embodiment 4 of the present invention. The device specifically includes:

[0082] The dimension information acquisition module 401 is used to acquire the dimension information of the expected output matrix based on the structural parameters of the input image and the structural parameters of the convolution kernel.

[0083] The data aggregation execution module 402 is used to sequentially take the original N×N data points in the expected output matrix as aggregation data points to obtain the aggregated output matrix corresponding to the expected output matrix; where N is a positive even number.

[0084] The image recognition execution module 403 is used to take the data points in the aggregated output matrix as the block base points, obtain the expected output matrix based on implicit universal matrix multiplication, and recognize the input image according to the expected output matrix.

[0085] The technical solution of this invention, based on the dimension information of the expected output matrix, sequentially uses the original N×N data points in the expected output matrix as aggregated data points to obtain the aggregated output matrix corresponding to the expected output matrix. Then, the data points in the aggregated output matrix are used as block base points, and the expected output matrix is ​​obtained based on implicit universal matrix multiplication. This enables the reuse of loaded data when the input matrix reads data from the physical layer, reduces data loading time, improves the computational efficiency of convolution operations performed by heterogeneous hardware accelerators, and avoids the performance degradation problem caused by different data loading logic between boundary points and non-boundary points due to encoding branches.

[0086] Optionally, the data aggregation execution module 402 is specifically used to obtain the matching N value based on the memory resource utilization rate.

[0087] Optionally, the data aggregation execution module 402 is further configured to determine whether the number of remaining original data points is greater than a first preset threshold if the expected output matrix includes remaining original data points that failed to be aggregated, in addition to the aggregated data points; if the number of remaining original data points is greater than the first preset threshold, the expected output matrix is ​​filled with zero elements to convert the remaining original data points into aggregated data points.

[0088] Optionally, the data aggregation execution module 402 is further configured to, if the number of remaining original data points is less than or equal to a first preset threshold, obtain an aggregated output matrix corresponding to the desired output matrix based on the aggregated data points and the remaining original data points.

[0089] Optionally, the data aggregation execution module 402 is further configured to, if the expected output matrix includes, in addition to the aggregated data points, remaining original data points that failed to be aggregated, continue to aggregate the remaining original data points by reducing the value of N until the value of N is reduced to 2 or there are no remaining original data points.

[0090] Optionally, the data aggregation execution module 402 is further configured to, when the value of N is reduced to 2, if there are still remaining original data points, fill the boundary of the expected output matrix with zero elements to transform the remaining original data points into aggregated data points, or obtain the aggregated output matrix corresponding to the expected output matrix based on the aggregated data points and the remaining original data points.

[0091] Optionally, the image recognition execution module 403 is specifically used to perform data fusion on the block input matrix corresponding to the aggregated data points in the aggregated output matrix according to the structural parameters of the convolution kernel, so as to obtain the fused block input fusion matrix; and to obtain the pixel value of the corresponding aggregated data point according to the block input fusion matrix and the convolution kernel.

[0092] The above-described apparatus can execute the image recognition method based on implicit universal matrix multiplication provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the image recognition method based on implicit universal matrix multiplication provided in any embodiment of the present invention.

[0093] Example 5

[0094] Embodiment 5 of the present invention also provides a heterogeneous hardware accelerator, which is used to execute the image recognition method based on implicit universal matrix multiplication described in any embodiment of the present invention.

[0095] In some embodiments, the image recognition method based on implicit universal matrix multiplication can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as a storage unit. In some embodiments, part or all of the computer program can be loaded and / or installed on a heterogeneous hardware accelerator via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by a processor, one or more steps of the image recognition method based on implicit universal matrix multiplication described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform the image recognition method based on implicit universal matrix multiplication by any other suitable means (e.g., by means of firmware).

[0096] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0097] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0098] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0099] To provide user interaction, the systems and techniques described herein can be implemented on a heterogeneous hardware accelerator, which includes: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the heterogeneous hardware accelerator. Other types of devices can also be used to provide user interaction; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback); and input from the user can be received in any form (including sound input, voice input, or haptic input).

[0100] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0101] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0102] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0103] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An image recognition method based on implicit universal matrix multiplication, characterized in that, include: Based on the structural parameters of the input image and the structural parameters of the convolution kernel, obtain the dimensionality information of the expected output matrix; The original N×N data points in the desired output matrix are sequentially used as aggregated data points to obtain the aggregated output matrix corresponding to the desired output matrix; where N is a positive even number. The step of sequentially using the N×N original data points in the desired output matrix as aggregated data points to obtain the aggregated output matrix corresponding to the desired output matrix includes: If the expected output matrix includes, in addition to the aggregated data points, remaining original data points that failed to aggregate, then it is determined whether the number of remaining original data points is greater than a first preset threshold. If the number of remaining original data points is greater than the first preset threshold, the expected output matrix is ​​filled with zero elements to transform the remaining original data points into aggregated data points. If the number of remaining original data points is less than or equal to the first preset threshold, then based on the aggregated data points and the remaining original data points, obtain the aggregated output matrix corresponding to the expected output matrix; The data points in the aggregated output matrix are used as the block base points. The expected output matrix is ​​obtained based on implicit universal matrix multiplication, and the input image is recognized based on the expected output matrix.

2. The method according to claim 1, characterized in that, The step of sequentially using the N×N original data points in the desired output matrix as aggregated data points to obtain the aggregated output matrix corresponding to the desired output matrix includes: Obtain the matching N value based on memory resource utilization.

3. The method according to claim 1, characterized in that, The step of sequentially using the N×N original data points in the desired output matrix as aggregated data points to obtain the aggregated output matrix corresponding to the desired output matrix includes: If the expected output matrix includes, in addition to the aggregated data points, remaining original data points that failed to aggregate, then by reducing the value of N, the remaining original data points are aggregated again until the value of N is reduced to 2 or there are no remaining original data points.

4. The method according to claim 3, characterized in that, After further aggregating the remaining original data points by reducing the value of N, the process also includes: When the value of N is reduced to 2, if there are still remaining original data points, the expected output matrix is ​​filled with zero elements to transform the remaining original data points into aggregated data points, or an aggregated output matrix corresponding to the expected output matrix is ​​obtained based on the aggregated data points and the remaining original data points.

5. The method according to any one of claims 1-4, characterized in that, The step of using data points in the aggregated output matrix as block base points, obtaining the expected output matrix based on implicit universal matrix multiplication, and recognizing the input image based on the expected output matrix includes: Based on the structural parameters of the convolution kernel, the block input matrices corresponding to the aggregated data points in the aggregated output matrix are fused to obtain the fused block input fusion matrix. The pixel values ​​of the corresponding aggregated data points are obtained based on the block input fusion matrix and the convolution kernel.

6. An image recognition device based on implicit universal matrix multiplication, characterized in that, include: The dimension information acquisition module is used to obtain the dimension information of the expected output matrix based on the structural parameters of the input image and the structural parameters of the convolution kernel. The data aggregation execution module is used to sequentially take the original N×N data points in the desired output matrix as aggregation data points to obtain the aggregated output matrix corresponding to the desired output matrix; where N is a positive even number. The data aggregation execution module is specifically used to determine whether the number of remaining original data points is greater than a first preset threshold if the expected output matrix includes remaining original data points that failed to aggregate, in addition to the aggregated data points; if the number of remaining original data points is greater than the first preset threshold, the expected output matrix is ​​padded with zero elements to convert the remaining original data points into aggregated data points; if the number of remaining original data points is less than or equal to the first preset threshold, the aggregated output matrix corresponding to the expected output matrix is ​​obtained based on the aggregated data points and the remaining original data points. The image recognition execution module is used to take the data points in the aggregated output matrix as the block base points, obtain the expected output matrix based on implicit universal matrix multiplication, and recognize the input image according to the expected output matrix.

7. A heterogeneous hardware accelerator, characterized in that, The heterogeneous hardware accelerator is used to execute the image recognition method based on implicit universal matrix multiplication as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the image recognition method based on implicit universal matrix multiplication as described in any one of claims 1-5.

Images