Convolution operation method, device, matrix expansion device and graphics processor

Abstract

The present application relates to a convolution operation method, device, matrix expansion device, and graphics processor. The method includes: for any sub-feature map in the original feature map, loading at least one target feature tile used to constitute the sub-feature map from a preset memory layout; the memory layout is obtained by writing at least one feature tile into the memory according to a preset data arrangement; the at least one feature tile is obtained by tiling the original feature map; according to the convolution parameters of the convolution layer, the feature map composed of the at least one target feature tile is expanded to obtain a target expansion matrix; the target expansion matrix is ​​matrix multiplied with the expansion matrix corresponding to the convolution kernel to obtain the convolution operation result of the original feature map. The use of this method can improve the efficiency of convolution operations.

Description

Technical Field

[0001] The present application relates to the field of general computing technology, and in particular to a convolution operation method, device, matrix expansion device, graphics processor, storage medium and computer program product. Background Art

[0002] Convolution operation is an important part of convolutional neural network (CNN). The training and inference time of convolutional neural network (CNN) are often affected by the speed of convolution operation.

[0003] The convolutional neural network (CNN) in traditional technology often multiplies and accumulates the elements inside the convolution kernel filter with the corresponding elements in the input feature map to obtain an element in the output feature map, then moves to the next step according to the step size and repeats the above operation until all elements of the output feature map are obtained. The convolution operation is inefficient.

[0004] Therefore, there is a problem of reduced efficiency of convolution operations in traditional technologies. Summary of the Invention

[0005] Based on this, it is necessary to provide a convolution operation method, device, computer equipment, computer-readable storage medium and computer program product that can improve the efficiency of convolution operations in response to the above technical problems.

[0006] In a first aspect, the present application provides a convolution operation method, the method comprising:

[0007] For any sub-feature map in the original feature map, at least one target feature tile used to constitute the sub-feature map is loaded from a preset memory layout; the memory layout is obtained by writing at least one feature tile into the memory according to a preset data arrangement; the at least one feature tile is obtained by tiling the original feature map;

[0008] Expanding the feature map composed of the at least one target feature tile according to the convolution parameters of the convolution layer to obtain a target expanded matrix;

[0009] Perform a matrix multiplication operation on the target expansion matrix and the expansion matrix corresponding to the convolution kernel to obtain the convolution operation result of the original feature map.

[0010] In one embodiment, after the step of reading the original feature map for performing the convolution operation, the method further includes:

[0011] Tiling the original feature map to obtain the at least one feature tile;

[0012] According to the data arrangement method, each feature tile is written into the memory in sequence to obtain the memory layout; wherein the data arrangement method includes at least a batch processing dimension, a channel dimension and a position dimension of the feature tile in the original feature map.

[0013] In one embodiment, writing each of the feature tiles into the memory in sequence according to the data arrangement to obtain the memory layout includes:

[0014] At least one feature tile at the same target position in the original feature map is written into the memory in sequence along a direction corresponding to the channel dimension to obtain a feature tile block corresponding to the target position.

[0015] In one embodiment, tiling the original feature map to obtain the at least one feature tile includes:

[0016] Obtaining a tile template used for tiling the original feature map;

[0017] determining a size of the tile template in at least one direction;

[0018] Performing matrix zero padding on the original feature map so that the size of the zero-padded feature map in the direction satisfies a multiple relationship with the size of the tile template in the direction;

[0019] The zero-padded feature map is tiled according to the tile template to obtain the at least one feature tile.

[0020] In one embodiment, each of the feature tiles has a corresponding tile index coordinate in the memory layout, and for any sub-feature map in the original feature map, loading at least one target feature tile for constituting the sub-feature map from a preset memory layout includes:

[0021] Obtaining the expanded matrix position coordinates corresponding to any sub-feature map; the expanded matrix position coordinates are used to represent the position information of the target expanded matrix in the expanded matrix corresponding to the original feature map;

[0022] Mapping the unfolded matrix position coordinates to target tile index coordinates; the target tile index coordinates are tile index coordinates corresponding to at least one target feature tile constituting any sub-feature map in the memory layout;

[0023] The feature tile corresponding to the target tile index coordinate is loaded into the memory layout to obtain the target feature tile.

[0024] In one embodiment, the step of expanding the feature map composed of the at least one target feature tile according to the convolution parameters of the convolution layer to obtain a target expansion matrix includes:

[0025] Expanding the feature map composed of the at least one target feature tile according to a convolution parameter of the convolution layer to obtain an expanded matrix;

[0026] A transposition operation is performed on the expanded matrix to obtain the target expanded matrix.

[0027] In one embodiment, before the step of expanding the feature map composed of the at least one target feature tile according to the convolution parameters of the convolution layer to obtain a target expanded matrix, the method further includes:

[0028] Get the convolution layer to which the current convolution operation belongs;

[0029] The convolution pattern of the convolution layer is analyzed to determine the convolution parameters of the convolution layer.

[0030] In a second aspect, the present application further provides a convolution operation device, comprising:

[0031] A reading module is used to read the original feature map used for convolution operation;

[0032] A loading module is configured to load, for any sub-feature map in an original feature map, at least one target feature tile used to constitute the sub-feature map from a preset memory layout; the memory layout is obtained by writing the at least one feature tile into a memory according to a preset memory layout; the at least one feature tile is obtained by tiling the original feature map; the memory layout includes at least a batch dimension, a channel dimension, and a position dimension of the feature tile in the original feature map;

[0033] an expansion module, configured to expand the feature map composed of the at least one target feature tile according to a convolution parameter of the convolution layer to obtain a target expansion matrix;

[0034] An operation module is used to perform a matrix multiplication operation on the target expansion matrix and the expansion matrix corresponding to the convolution kernel to obtain a convolution operation result for the original feature map.

[0035] In a third aspect, the present application further provides a matrix expansion device, comprising: a tile collection module, a pattern parsing module, a matrix processing module, and a matrix cache module, wherein:

[0036] The tile collection module is configured to execute acquisition of at least one target feature tile used to constitute any sub-feature map in the original feature map from a texture unit; the target feature tile is loaded by the texture unit from a preset memory layout;

[0037] The pattern parsing module is configured to obtain convolution parameters of the convolution layer;

[0038] The matrix processing module is configured to perform expansion processing on the feature map composed of the at least one target feature tile according to the convolution parameters to obtain a target expansion matrix;

[0039] The matrix cache module is further configured to execute and cache the target expansion matrix so that the execution unit can generate the convolution operation result of the original feature map based on the target expansion matrix.

[0040] In one embodiment, the matrix processing module includes a matrix expansion submodule and a matrix transposition submodule, wherein:

[0041] The matrix expansion submodule is configured to expand the feature map composed of the at least one target feature tile according to the convolution parameters to obtain an expanded matrix;

[0042] The matrix transposition submodule is configured to perform a transposition operation on the expanded matrix to obtain the target expanded matrix.

[0043] In one embodiment, the convolution parameters include a convolution step size and a convolution kernel size, and the matrix expansion submodule is specifically configured to execute, according to the convolution step size and the convolution kernel size, converting the feature map composed of the at least one target feature tile into at least one row vector in sequence according to the position of the original image, and splicing the at least one row vector into a feature map matrix to obtain the expanded matrix.

[0044] In one embodiment, the pattern parsing module is specifically configured to obtain the convolution layer to which the current convolution operation belongs; parse the convolution pattern of the convolution layer, and determine the convolution parameters of the convolution layer.

[0045] In one embodiment, the matrix cache module is further configured to transfer the target unfolded matrix to a high-speed shared memory of the execution unit.

[0046] In a fourth aspect, the present application further provides a graphics processor, comprising: a texture unit, an execution unit, and the above-mentioned matrix expansion device, wherein:

[0047] The texture unit is configured to load, for any sub-feature map in the original feature map, at least one target feature tile used to constitute the sub-feature map from a preset memory layout; and is further configured to transmit the at least one target feature tile to the matrix expansion device;

[0048] The execution unit is configured to execute the target expansion matrix received and transmitted by the matrix expansion device, and perform a matrix multiplication operation on the target expansion matrix and the expansion matrix corresponding to the convolution kernel to obtain the convolution operation result of the original feature map.

[0049] In one embodiment, the execution unit is configured to execute sending the expanded matrix position coordinates to the texture unit; the expanded matrix position coordinates are used to represent the position information of the target expanded matrix in the expanded matrix corresponding to the original feature map;

[0050] The texture unit is configured to execute mapping of the unfolded matrix position coordinates into target tile index coordinates; the target tile index coordinates are tile index coordinates corresponding to at least one target feature tile constituting any sub-feature map in the memory layout; and the feature tile corresponding to the target tile index coordinates is loaded into the memory layout to obtain the target feature tile.

[0051] In one embodiment, the graphics processor is configured to perform tiling on the original feature map to obtain the at least one feature tile; write each of the feature tiles into the memory in sequence according to the data arrangement method to obtain the memory layout; wherein the data arrangement method includes at least a batch dimension, a channel dimension, and a position dimension of the feature tile in the original feature map.

[0052] In one embodiment, the graphics processor is configured to write at least one feature tile at the same target position in the original feature map into the memory in sequence along a direction corresponding to the channel dimension to obtain a feature tile block corresponding to the target position.

[0053] In one embodiment, the graphics processor is configured to execute the steps of obtaining a tile template for tiling the original feature map; determining the size of the tile template in at least one direction; performing matrix zero padding on the original feature map so that the size of the zero-padded feature map in the direction satisfies a multiple relationship with the size of the tile template in the direction; and tiling the zero-padded feature map according to the tile template to obtain the at least one feature tile.

[0054] The above-mentioned convolution operation method, device, matrix expansion device, graphics processor, storage medium and computer program product, for any sub-feature map in the original feature map, loads at least one target feature tile used to constitute any sub-feature map from a preset memory layout; wherein the memory layout is obtained by writing at least one feature tile into the memory according to a preset data arrangement method; at least one feature tile is obtained by tiling the original feature map; and according to the convolution parameters of the convolution layer, the feature map composed of at least one target feature tile is expanded to obtain a target expansion matrix, so as to perform the target expansion matrix on the expansion matrix corresponding to the convolution kernel. The row matrix multiplication operation is performed to obtain the convolution operation result of the original feature map; in this way, it is possible to quickly and accurately perform batch matrix multiplication operations on each sub-feature map in the original feature map to obtain the convolution operation result of the original feature map, and realize the simultaneous execution of matrix expansion and matrix multiplication operations in the same operation kernel (kernel). There is no need to wait until the original feature map is expanded into an ultra-large matrix and then perform matrix multiplication on the ultra-large matrix with the expanded matrix corresponding to the convolution kernel, which greatly improves the convolution operation efficiency and operation execution efficiency; at the same time, since there is no need to store the ultra-large matrix corresponding to the original feature map, the storage space of the matrix is ​​also greatly reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 1 is a flow chart of a convolution operation method in one embodiment;

[0056] Figure 2a is a schematic diagram of a memory layout in one embodiment;

[0057] Figure 2b is a schematic diagram of a data arrangement method in one embodiment;

[0058] Figure 3 Schematic diagram of an expanded matrix corresponding to an original feature map in one embodiment;

[0059] Figure 4 is a structural block diagram of a graphics processor in one embodiment;

[0060] Figure 5 is a schematic flow chart of a coordinate calculation process in one embodiment;

[0061] Figure 6 1 is a schematic flow chart of an extraction process in one embodiment;

[0062] Figure 7 is a flowchart of a reading process in one embodiment;

[0063] Figure 8 is a structural block diagram of a controller in one embodiment;

[0064] Figure 9 is a structural block diagram of a buffer in one embodiment;

[0065] Figure 10 is a structural block diagram of a matrix expansion device in one embodiment;

[0066] Figure 11 is a schematic diagram of a matrix expansion process in one embodiment;

[0067] Figure 12 is a schematic diagram of a transposition operation process in one embodiment;

[0068] Figure 13 A schematic diagram of a matrix writing process in one embodiment;

[0069] Figure 14 is a schematic diagram of a target feature tile loading process in one embodiment;

[0070] Figure 15 1 is a flow chart of a convolution operation method according to another embodiment;

[0071] Figure 16 2 is a structural block diagram of a convolution operation device in one embodiment. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0073] In one embodiment, Figure 1 As shown, a convolution operation method is provided, which includes the following steps:

[0074] Step S110: For any sub-feature map in the original feature map, at least one target feature tile used to constitute any sub-feature map is loaded from a preset memory layout.

[0075] The original feature map may refer to a feature map (FeatureMap) that requires a convolution kernel to perform a convolution operation.

[0076] The memory layout is obtained by writing at least one feature tile into the memory according to a preset data arrangement.

[0077] Among them, at least one feature tile is obtained by tiling the original feature map.

[0078] In practical applications, the original feature map can be tiled to obtain multiple feature tiles that constitute the original feature map. Then, the multiple feature tiles that constitute the original feature map are written into the memory according to a preset memory layout (data arrangement) to form a memory layout for the original feature map. Among them, each feature tile has a corresponding coordinate in the memory layout of the original feature map.

[0079] In a specific implementation, when the graphics processor performs a convolution operation on the original feature map, the texture unit of the graphics processor can load at least one target feature tile used to constitute any sub-feature map in the original feature map from a preset memory layout.

[0080] Specifically, the texture unit of the graphics processor can obtain the position information of any sub-feature map in the original feature map and map the position information to corresponding tile coordinate information. The tile coordinate information may include the coordinates of at least one feature tile constituting any sub-feature map in the memory layout. The graphics processor then loads at least one target feature tile used to constitute any sub-feature map into the memory layout based on the tile coordinate information. The texture unit of the graphics processor then sends the at least one target feature tile to the matrix expansion device of the graphics processor.

[0081] Step S120 : Expand the feature map composed of at least one target feature tile according to the convolution parameters of the convolution layer to obtain a target expansion matrix.

[0082] In a specific implementation, after loading at least one target feature tile that constitutes any sub-feature map, the graphics processor can obtain the convolution parameters of the convolution layer (such as the size of the convolution kernel), and use the img2col (feature map to matrix) algorithm to expand the feature map composed of at least one target feature tile to obtain the target expanded matrix.

[0083] Specifically, after receiving the target feature tiles from the texture unit, the GPU's matrix expansion device can use the img2col algorithm to expand the feature map composed of each target feature tile to obtain an initial expansion matrix. The GPU then transposes the initial expansion matrix to obtain a target expansion matrix. The GPU's matrix expansion device then sends the target expansion matrix to a cache in the GPU's execution unit for use in matrix multiplication.

[0084] Step S130: Perform a matrix multiplication operation on the target expansion matrix and the expansion matrix corresponding to the convolution kernel to obtain the convolution operation result of the original feature map.

[0085] In the specific implementation, after the execution unit of the graphics processor obtains the target expanded matrix, the arithmetic unit (ALU) in the execution unit of the image processor performs a matrix multiplication operation on the target expanded matrix and the expanded matrix corresponding to the convolution kernel to obtain a matrix multiplication operation result; then, the graphics processor uses the col2img algorithm (the inverse operation of the img2col algorithm) to convert the matrix multiplication operation result into an output feature map as the convolution operation result for the original feature map.

[0086] The technical solution of this embodiment is to load at least one target feature tile used to constitute any sub-feature map in the original feature map from a preset memory layout; wherein the memory layout is obtained by writing at least one feature tile into the memory according to a preset data arrangement method; at least one feature tile is obtained by tiling the original feature map; and according to the convolution parameters of the convolution layer, the feature map composed of at least one target feature tile is expanded to obtain a target expanded matrix, and the target expanded matrix is ​​matrix multiplied with the expanded matrix corresponding to the convolution kernel to obtain the convolution operation result of the original feature map; in this way, it is possible to quickly and accurately perform batch matrix multiplication operations on each sub-feature map in the original feature map to obtain the convolution operation result of the original feature map, and to realize the simultaneous execution of matrix expansion and matrix multiplication operations in the same operation kernel, without having to wait until the original feature map is expanded into an ultra-large matrix and then perform matrix multiplication on the ultra-large matrix with the expanded matrix corresponding to the convolution kernel, thereby greatly improving the convolution operation efficiency and operation execution efficiency; at the same time, since there is no need to store the ultra-large matrix corresponding to the original feature map, the storage space of the matrix is ​​also greatly reduced.

[0087] In another embodiment, the method further includes: tiling the original feature map to obtain at least one feature tile; writing each feature tile into the memory in sequence according to the data arrangement method to obtain a memory layout; wherein the arrangement dimension of the data arrangement method includes at least a batch dimension, a channel dimension, and a position dimension of the feature tile in the original feature map.

[0088] At least one feature tile at the same target position in the original feature map is written into the memory in sequence along the direction corresponding to the channel dimension to obtain a feature tile block corresponding to the target position.

[0089] In a specific implementation, the graphics processor can tile the original feature map to obtain at least one feature tile; then, the graphics processor can write each feature tile into memory in sequence according to the batch dimension, channel dimension, and position dimension of the feature tile in the original feature map to obtain a memory layout. Among them, the graphics processor can write at least two feature tiles that are at the same target position in the original feature map and have different channels into memory in sequence along the direction corresponding to the channel dimension to obtain a feature tile block (Brick) corresponding to the target position. In practical applications, the feature tile block can also be named tile, brick, etc.

[0090] In practical applications, the above data arrangement can be named 4D-Brick. The graphics processor first divides the input original feature map into tiles according to the tile size of axb. At the same time, the first dimension height and second dimension width of the original feature map are also aligned according to the tile size. Then, the tiled feature tiles are stored along the channel direction. The tiles at the same position in all channels are stored together and called a Brick. For the convenience of those skilled in the art, please refer to Figure 2a , Figure 2a The figure shows an example of a 4D-Brick memory layout when the batch size is 1. The original feature map of each channel includes axb feature tiles. Feature tiles at the same target position but different channels in the original feature map constitute a feature tile block (Brick).

[0091] To facilitate understanding by those skilled in the art, Figure 2b A schematic diagram of the data arrangement is also provided as an example; see Figure 2b , the order of storing the characteristic data of 4D-Brick can be: Tiles of all channels of Brick0 are stored in the order of CH a W b Store, that is, Brick0.C0[a,b],Brick0.C1[a,b],…,Brick0.C in-1 [a, b], then move along W-Dim to the next Tile until all Bricks of W-Dim are stored, and finally move along H-Dim until all bricks of the current batch (C, Aligned_H, Aligned_W) are stored.

[0092] If the batch size N>1, repeat the above steps until all batches are stored, until all bricks in Figure 2(a) are stored.

[0093] The technical solution of this embodiment is to obtain a memory layout by tiling the original feature map and writing the feature tiles obtained after tiling into the memory in sequence according to the data arrangement method, and the arrangement dimensions of the data arrangement method adopted include at least the batch processing dimension, the channel dimension and the position dimension of the feature tiles in the original feature map; in this way, the original feature map can be stored in the form of a four-dimensional channel block, which is convenient for the subsequent rapid loading of the feature tiles used to construct the sub-feature map.

[0094] In another embodiment, the original feature map is tiled to obtain at least one feature tile, including: obtaining a tile template used to tile the original feature map; determining the size of the tile template in at least one direction; performing matrix zero padding on the original feature map so that the size of the zero-padded feature map in the direction satisfies a multiple relationship with the size of the tile template in the direction; and tiling the zero-padded feature map according to the tile template to obtain at least one feature tile.

[0095] In a specific implementation, when the graphics processor tiles the original feature map to obtain at least one feature tile, the graphics processor may obtain the tile template used to tile the original feature map and determine the size of the tile template in at least one direction. The graphics processor may then determine whether the size of the original feature map in at least one direction satisfies a multiple relationship with the size of the tile template in that direction; if not, the graphics processor performs matrix zero padding on the original feature map so that the size of the zero-padded feature map in that direction satisfies a multiple relationship with the size of the tile template in that direction. Finally, the graphics processor uses the tile template to tile the zero-padded feature map to obtain at least one feature tile.

[0096] For example, let's assume the original feature map has a size of 40x37 and the tile template has a size of 4x4: that is, the original feature map has a size of 40 in the x-direction and a size of 37 in the y-direction; the tile template has a size of 4 in the x-direction and a size of 4 in the y-direction. It can be seen that the size of the original feature map in the x-direction is a multiple of the size of the tile template in the x-direction, but the size of the original feature map in the y-direction is not a multiple of the size of the tile template in the y-direction. Therefore, the original feature map is padded with zeros, resulting in a feature map with a size of 40x40 after zero padding. It can be seen that the size of the feature map in the x-direction after zero padding is a multiple of the size of the tile template in the x-direction, and the size of the feature map in the y-direction after zero padding is a multiple of the size of the tile template in the y-direction.

[0097] In this way, by performing matrix zero padding on the original feature map, the size of the zero-padded feature map in the direction and the size of the tile template in the direction satisfy a multiple relationship, so that the tile template can be successfully used to tile the zero-padded feature map into an integer number of feature tiles.

[0098] In another embodiment, for any sub-feature map in the original feature map, at least one target feature tile used to constitute any sub-feature map is loaded from a preset memory layout, including: obtaining the expanded matrix position coordinates corresponding to any sub-feature map; mapping the expanded matrix position coordinates to target tile index coordinates; loading the feature tile corresponding to the target tile index coordinates in the memory layout to obtain the target feature tile.

[0099] Among them, the expanded matrix position coordinates are used to represent the position information of the target expanded matrix in the expanded matrix corresponding to the original feature map.

[0100] The target tile index coordinate is the tile index coordinate corresponding to at least one target feature tile constituting any sub-feature map in the memory layout.

[0101] Among them, each feature tile has a corresponding tile index coordinate in the memory layout.

[0102] In a specific implementation, when the GPU loads at least one target feature tile used to constitute any sub-feature map from a preset memory layout, it can obtain the unfolded matrix position coordinates corresponding to the sub-feature map. The unfolded matrix position coordinates are then mapped to target tile index coordinates. Finally, the GPU loads the feature tile corresponding to the target tile index coordinates from the memory layout to obtain the target feature tile.

[0103] The technical solution of this embodiment obtains the position coordinates of the expanded matrix corresponding to any sub-feature map, that is, the position coordinates of the required expanded matrix in the original feature map, maps the expanded matrix position coordinates to the target tile index coordinates, and loads the feature tiles corresponding to the target tile index coordinates from the memory layout, thereby accurately loading the target feature tiles constituting any sub-feature map in the memory layout.

[0104] In another embodiment, a feature map composed of at least one target feature tile is expanded according to the convolution parameters of the convolution layer to obtain a target expanded matrix, including: expanding the feature map composed of at least one target feature tile according to the convolution parameters of the convolution layer to obtain an expanded matrix; and performing a transpose operation on the expanded matrix to obtain a target expanded matrix.

[0105] In a specific implementation, the graphics processor expands the feature map composed of at least one target feature tile according to the convolution parameters of the convolution layer to obtain a target expanded matrix. The graphics processor can expand the feature map composed of at least one target feature tile according to the convolution parameters of the convolution layer to obtain an expanded matrix; finally, the graphics processor performs a transpose operation on the expanded matrix to obtain the target expanded matrix.

[0106] The technical solution of this embodiment is to expand the feature map composed of at least one target feature tile according to the convolution parameters of the convolution layer, obtain the expanded matrix, and then perform a transpose operation on the expanded matrix, so that the obtained target expanded matrix can meet the matrix form required for subsequent matrix multiplication operations.

[0107] In another embodiment, the method further includes: obtaining a convolution layer to which the current convolution operation belongs; parsing a convolution pattern of the convolution layer to determine convolution parameters of the convolution layer.

[0108] Among them, the convolution parameters include the size, step size, and padding of the convolution kernel filter.

[0109] In a specific implementation, the GPU can obtain the convolution layer to which the current convolution operation belongs; then, the GPU analyzes the convolution pattern of the convolution layer and determines the convolution parameters of the convolution layer. Specifically, the GPU only needs to analyze the convolution pattern of the same convolution layer once, and the data of the remaining feature tiles of the convolution layer will have the same convolution pattern.

[0110] The technical solution of this embodiment, by obtaining the convolution layer to which the current convolution operation belongs and parsing the convolution pattern of the convolution layer, can accurately determine the convolution parameters of the convolution layer corresponding to the target feature tiles in the same convolution layer, and perform an expansion operation on the feature map composed of the above target feature tiles based on the convolution parameters.

[0111] To facilitate understanding by those skilled in the art, Figure 3 A schematic diagram of the unfolded matrix corresponding to the original feature map is provided; see Figure 3 , matrix A is the complete matrix expanded by img2col, [P, R] is a sub-matrix of A, which can be mapped to the 4D-Brick through the upper left corner coordinate (X, Y) of the sub-matrix. TU loads the tile in the Brick into the MDC for img2col expansion through the mapping address. Figure 3 P and R can be configured according to the required target matrix size.

[0112] like Figure 4 As shown, Figure 4A graphics processor is provided. The graphics processor includes a texture unit (TextureUnit), a matrix development device (MDC) and an execution unit (Execute Unit).

[0113] The texture unit may include an extractor (Brick Extractor), a controller (BrickController), a cache (Brick Cache) and a sender (Brick Sender); wherein the controller (BrickController) includes a tile loading subunit (Tile Loader).

[0114] Among them, the execution unit includes the SMP (Sampling) module, the arithmetic unit (ALU) and the high-speed shared cache (SM).

[0115] In the specific implementation, the tile size axb = 4x8, P = R = 32 is used as an example for explanation. In the process of the graphics processor mapping the expanded matrix position coordinates to the target tile index coordinates, the graphics processor obtains the expanded matrix position coordinates corresponding to any sub-feature map, that is, the coordinates (X, Y)∈[C IN *kh*kw,N*H out *W out ], the execution unit of the graphics processor calculates (Quo x ,Rem x ,batch idx ,h in_off ,w in_off ) coordinates and sent to the Brick Extractor in the texture unit through the SMP module. The specific calculation process of the coordinates is as follows Figure 5 As shown: Among them, (stride h ,stride w ) and (pad h ,pad w ) is a constant for the same convolutional layer in CNN, and different convolutional layers (stride h ,stride w ) and (pad h ,pad w ) may be different.

[0116] The Brick Extractor receives the Quo x ,Rem x ,batch idx ,h in_off ,w in_off) coordinates, the Brick Extractor extracts the above coordinate information to obtain the Brick in_off and Rem X , and send the extracted information to the controller (Brick Controller). The extraction process is as follows Figure 6 shown.

[0117] See Figure 7 When loading tile data, the tile loader first searches the Brick Cache. If the target tile data is in the Brick Cache, it sends the corresponding data directly from the Brick Cache to the Brick Sender. If not, the tile loader requests the data from the L2 cache.

[0118] Specifically, after the Brick Controller receives the information from the Brick Extractor, it will load the tiles through the Tile Loader in the Brick Controller according to the Rem X Calculate the tile data in the Brick that needs to be loaded; the process of Tile Loader loading the tile data in the Brick is as follows Figure 8 As shown, the length of the R direction in [[P, R] is 32. Since the output Feature Map in the P direction is also divided according to Tile axb, after img2col is expanded, P=axb=4x8=32 in this embodiment.

[0119] See Figure 9 As shown, the cache (Brick Cache) sends the tile data requested by the controller (Brick Controller) directly to the sender (BrickSender) if it is in the cache; if not, the tile in the Brick returned by the second-level cache L2 is stored in the cache and the tile data (i.e., the target feature tile) is sent to the sender (BrickSender).

[0120] The Brick Sender is responsible for sending the target feature tile to the matrix expansion device MDC. Specifically, the texture unit can be based on the stride of the current convolution layer. h ,stride w ) and (padh ,pad w ), combined with the Brick in the controller Brick Controller in_off From the 4D-Brick memory layout, the target feature tile in the corresponding brick is loaded and sent to the matrix expansion device (MDC) via the Brick Sender. The MDC dynamically performs img2col on the received tile, expanding it into a [P, Q] matrix, or a [32, 32] matrix in this embodiment. It should be noted that if a texture unit is out-of-bound (OOB) during loading, the corresponding portion of the returned tile can be filled with zeros.

[0121] In another embodiment, Figure 10 As shown, Figure 10 A matrix expansion device (MDC) 420 is provided, comprising: a tile collection module (Tile Collector) 1010, a pattern parsing module (Pattern Parser) 1020, a matrix processing module 1030 and a matrix buffer module (Matrix Buffer) 1040, wherein the matrix processing module includes a matrix expansion submodule (Matrix DeCompress Engine) 1031 and a matrix transpose submodule (Matrix Transpose Ctrl) 1032.

[0122] In a specific implementation, the Tile Collector is configured to retrieve at least one target feature tile from the texture unit's Brick Sender, used to construct any sub-feature map within the original feature map. The target feature tile is loaded from a preset memory layout by the texture unit. Specifically, the Tile Collector receives the data (target feature tile) corresponding to the output Feature Map of a given tile axb size.

[0123] Among them, the corresponding input Feature Map size that the tile collection module needs to collect can be calculated by the following formula:

[0124] h input =(h output -1)*stride h -(2*pad h -kh)

[0125] =(a-1)*stride h -(2*pad h -kh)

[0126] w input =(w output -1)*stride w -(2*pad w -kw)

[0127] =(b-1)*stride w -(2*pad w -kw)

[0128] Tile collection module collection (h input ,w input ) size Tile data, the matrix expansion submodule can perform img2col expansion on the feature map composed of the target feature tiles.

[0129] The pattern parsing module (Pattern Parser) is configured to execute the acquisition of the convolution parameters of the convolution layer; specifically, the matrix expansion submodule is specifically configured to execute the conversion of the feature map composed of at least one target feature tile into at least one row vector according to the position of the original image according to the convolution step size and the convolution kernel size, and splice the at least one row vector into a feature map matrix to obtain the expanded matrix.

[0130] In practical applications, the matrix expansion device (MDC) can support common convolution parameters in CNN models, such as convolution kernel filters (Filter kh*kw): 1x1, 3x3, 5x5, 7x7, 1x7, 7x1, 1x3, 3x1, etc., stride: 1, 2, etc., padding height and width: 0x0, 1x1, 2x2, 3x3, 0x3, 3x0, 0x1, 1x0, etc.

[0131] The matrix processing module is configured to perform an expansion operation on a feature map composed of at least one target feature tile according to the convolution parameters to obtain a target expanded matrix. The matrix expansion submodule is configured to perform an expansion operation on the feature map composed of at least one target feature tile according to the convolution parameters to obtain an expanded matrix. The matrix transposition submodule is configured to perform a transposition operation on the expanded matrix to obtain a target expanded matrix.

[0132] After the tile collection module collects the tile data and the convolution parameters parsed by the pattern parsing module, the matrix expansion submodule (Matrix DeCompress Engine) can perform img2col expansion. input,winput) performs img2col expansion according to the parsed convolution parameters. Specifically, based on the convolution step size and kernel size, the feature map composed of at least one target feature tile can be converted into at least one row vector in sequence according to the position of the original image. The at least one row vector is then concatenated into a feature map matrix to obtain an expanded matrix.

[0133] Figure 11 Taking the common convolution kernel filter in CNN as 3x3, step size as 1, and padding as 0 as an example, the process of img2col expansion by the matrix expansion submodule is explained. It should be noted that Figure 11 Only one channel of the corresponding tile data is expanded by img2col to a matrix of [9,32]. The remaining convolution modes can be derived similarly and will not be described here.

[0134] To facilitate understanding by those skilled in the art, Figure 12 A schematic diagram of the transposition operation process is provided, see Figure 12 After the matrix expansion submodule (Matrix DeCompress Engine) performs img2col expansion to obtain the expanded matrix (DestMatrix), the matrix transpose submodule (Matrix Transpose Control) will perform a transpose operation on the expanded matrix to obtain the target expanded matrix, and write the target expanded matrix to the matrix cache module, that is, the internal matrix buffer (Matrix Buffer).

[0135] The matrix buffer module is also configured to cache the target expanded matrix and transmit the target expanded matrix to the high-speed shared memory of the execution unit, so that the execution unit can generate the convolution operation result of the original feature map based on the target expanded matrix. The matrix is ​​written to the high-speed shared memory according to the current data grid. For the convenience of understanding of those skilled in the art, please refer to Figure 13 , Figure 13 Taking the 8-bit data format as an example, the process of MDC writing [P, R] (this embodiment takes [32, 32] as an example) to the high-speed shared memory is described.

[0136] To facilitate understanding by those skilled in the art, this embodiment provides an example of loading a target feature tile; see Figure 14 , Figure 14This example shows how a Texture Unit loads tile data from a 4D-Brick memory layout and performs img2col expansion on the tile data using the MDC. For example, the target matrix [P,Q] for the expansion is [32,32]. This example performs a convolution operation on an input feature map with (N,C,Aligned_H,Aligned_W) of (1,7,8,16) using a 3x3 filter, a stride of 1, and a pad of 0; the output feature map is (1,1,6,14).

[0137] In another embodiment, Figure 15 As shown, a convolution operation method is provided, comprising the following steps:

[0138] Step S1510: Tiling the original feature map to obtain the at least one feature tile.

[0139] Step S1520: Write each of the feature tiles into the memory in sequence according to the data arrangement method to obtain the memory layout; wherein the arrangement dimensions of the data arrangement method include at least a batch dimension, a channel dimension, and a position dimension of the feature tile in the original feature map.

[0140] Step S1530: For any sub-feature map in the original feature map, at least one target feature tile used to constitute the sub-feature map is loaded from a preset memory layout.

[0141] Step S1540: Expand the feature map composed of the at least one target feature tile according to the convolution parameters of the convolution layer to obtain an expanded matrix.

[0142] Step S1550: performing a transpose operation on the expanded matrix to obtain a target expanded matrix.

[0143] Step S1560: Perform a matrix multiplication operation on the target expansion matrix and the expansion matrix corresponding to the convolution kernel to obtain a convolution operation result of the original feature map.

[0144] It should be noted that the specific limitations of the above steps can be found in the specific limitations of a convolution operation method above, which will not be repeated here.

[0145] In another embodiment, a graphics processor is provided, comprising: a texture unit, an execution unit, and the above-mentioned matrix expansion device, wherein:

[0146] The texture unit is configured to load, for any sub-feature map in the original feature map, at least one target feature tile used to constitute any sub-feature map from a preset memory layout; and is further configured to transmit the at least one target feature tile to the matrix expansion device;

[0147] The execution unit is configured to execute the target expansion matrix transmitted by the receiving matrix expansion device, and perform matrix multiplication operation on the target expansion matrix and the expansion matrix corresponding to the convolution kernel to obtain the convolution operation result of the original feature map.

[0148] In another embodiment, the execution unit is configured to execute sending the expanded matrix position coordinates to the texture unit; the expanded matrix position coordinates are used to represent position information of the target expanded matrix in the expanded matrix corresponding to the original feature map;

[0149] The texture unit is configured to perform mapping of the unfolded matrix position coordinates to target tile index coordinates; the target tile index coordinates are tile index coordinates corresponding to at least one target feature tile constituting any sub-feature map in the memory layout; and the feature tile corresponding to the target tile index coordinates is loaded in the memory layout to obtain the target feature tile.

[0150] In another embodiment, a graphics processor is configured to perform tiling on an original feature map to obtain at least one feature tile; write each feature tile into a memory in sequence according to a data arrangement method to obtain a memory layout; wherein the data arrangement method includes at least a batch dimension, a channel dimension, and a position dimension of the feature tile in the original feature map.

[0151] In another embodiment, the graphics processor is configured to write at least one feature tile at the same target position in the original feature map into the memory in sequence along a direction corresponding to the channel dimension to obtain a feature tile block corresponding to the target position.

[0152] In another embodiment, a graphics processor is configured to execute the steps of obtaining a tile template for tiling an original feature map; determining the size of the tile template in at least one direction; performing matrix zero padding on the original feature map so that the size of the zero-padded feature map in the direction satisfies a multiple relationship with the size of the tile template in the direction; and tiling the zero-padded feature map according to the tile template to obtain at least one feature tile.

[0153] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0154] Based on the same inventive concept, embodiments of the present application also provide a convolution operation device for implementing the aforementioned convolution operation method. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations of one or more convolution operation device embodiments provided below can be found in the above-mentioned limitations of the convolution operation method and will not be repeated here.

[0155] In one embodiment, Figure 16 As shown, a convolution operation device is provided, comprising:

[0156] A reading module 1610 is configured to read an original feature map for performing a convolution operation;

[0157] A loading module 1620 is configured to load, for any sub-feature map in the original feature map, at least one target feature tile used to constitute the sub-feature map from a preset memory layout; the memory layout is obtained by writing at least one feature tile into a memory according to a preset memory layout; the at least one feature tile is obtained by tiling the original feature map; the memory layout includes at least a batch dimension, a channel dimension, and a position dimension of the feature tile in the original feature map;

[0158] An expansion module 1630 is configured to expand the feature map composed of the at least one target feature tile according to a convolution parameter of the convolution layer to obtain a target expansion matrix;

[0159] The operation module 1640 is used to perform a matrix multiplication operation on the target expansion matrix and the expansion matrix corresponding to the convolution kernel to obtain a convolution operation result for the original feature map.

[0160] In one embodiment, the device is further used to tile the original feature map to obtain the at least one feature tile; write each feature tile into the memory in sequence according to the data arrangement method to obtain the memory layout; wherein the data arrangement method includes at least a batch dimension, a channel dimension and a position dimension of the feature tile in the original feature map.

[0161] In one embodiment, the device is further used to write at least one feature tile at the same target position in the original feature map into the memory in sequence along a direction corresponding to the channel dimension to obtain a feature tile block corresponding to the target position.

[0162] In one embodiment, the device is also used to obtain a tile template used for tiling the original feature map; determine the size of the tile template in at least one direction; perform matrix zero padding on the original feature map so that the size of the zero-padded feature map in the direction satisfies a multiple relationship with the size of the tile template in the direction; and tile the zero-padded feature map according to the tile template to obtain the at least one feature tile.

[0163] In one embodiment, each of the feature tiles has a corresponding tile index coordinate in the memory layout, and the loading module 1620 is specifically used to obtain the expanded matrix position coordinates corresponding to any sub-feature map; the expanded matrix position coordinates are used to represent the position information of the target expanded matrix in the expanded matrix corresponding to the original feature map; the expanded matrix position coordinates are mapped to target tile index coordinates; the target tile index coordinates are the tile index coordinates corresponding to at least one target feature tile constituting any sub-feature map in the memory layout; the feature tile corresponding to the target tile index coordinate is loaded in the memory layout to obtain the target feature tile.

[0164] In one embodiment, the expansion module 1630 is specifically used to expand the feature map composed of the at least one target feature tile according to the convolution parameters of the convolution layer to obtain an expanded matrix; and perform a transpose operation on the expanded matrix to obtain the target expanded matrix.

[0165] In one embodiment, the device is further used to obtain a current convolutional layer to which the convolution operation belongs; analyze the convolution mode of the current convolutional layer, and determine the convolution parameters of the convolutional layer.

[0166] Each module in the convolution operation device described above can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0167] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0168] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0169] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A convolution operation method, characterized in that: The method comprises: For any sub-feature map in the original feature map, at least one target feature tile used to constitute the sub-feature map is loaded from a preset memory layout; the memory layout is obtained by writing at least one feature tile into the memory according to a preset data arrangement; the memory layout includes feature tile blocks corresponding to multiple target positions, and each feature tile block corresponding to the target position corresponds to a row of data in the memory layout; the feature tile block corresponding to each target position is obtained by writing at least two feature tiles at the same target position in the original feature map and with different channels into the memory in sequence along a direction corresponding to the channel dimension; the feature tiles are obtained by tiling the original feature map; Expanding the feature map composed of the at least one target feature tile according to the convolution parameters of the convolution layer to obtain a target expanded matrix; Perform a matrix multiplication operation on the target expansion matrix and the expansion matrix corresponding to the convolution kernel to obtain the convolution operation result of the original feature map.

2. The method according to claim 1, characterized in that The method further comprises: Tiling the original feature map to obtain the at least one feature tile; According to the data arrangement method, each feature tile is written into the memory in sequence to obtain the memory layout; wherein the arrangement dimension of the data arrangement method includes at least a batch processing dimension, a channel dimension and a position dimension of the feature tile in the original feature map.

3. The method according to claim 2, characterized in that Writing each of the feature tiles into the memory in sequence according to the data arrangement to obtain the memory layout includes: At least one feature tile at the same target position in the original feature map is written into the memory in sequence along a direction corresponding to the channel dimension to obtain a feature tile block corresponding to the target position.

4. The method according to claim 2, characterized in that The tiling of the original feature map to obtain the at least one feature tile includes: Obtaining a tile template used for tiling the original feature map; determining a size of the tile template in at least one direction; Performing matrix zero padding on the original feature map so that the size of the zero-padded feature map in the direction satisfies a multiple relationship with the size of the tile template in the direction; The zero-padded feature map is tiled according to the tile template to obtain the at least one feature tile.

5. The method according to claim 1, wherein Each of the feature tiles has a corresponding tile index coordinate in the memory layout, and for any sub-feature map in the original feature map, loading at least one target feature tile for constituting the sub-feature map from a preset memory layout includes: Obtaining the expanded matrix position coordinates corresponding to any sub-feature map; the expanded matrix position coordinates are used to represent the position information of the target expanded matrix in the expanded matrix corresponding to the original feature map; Mapping the unfolded matrix position coordinates to target tile index coordinates; the target tile index coordinates are tile index coordinates corresponding to at least one target feature tile constituting any sub-feature map in the memory layout; The feature tile corresponding to the target tile index coordinate is loaded into the memory layout to obtain the target feature tile.

6. The method according to claim 1, characterized in that The step of expanding the feature map composed of the at least one target feature tile according to the convolution parameters of the convolution layer to obtain a target expansion matrix includes: Expanding the feature map composed of the at least one target feature tile according to a convolution parameter of the convolution layer to obtain an expanded matrix; A transposition operation is performed on the expanded matrix to obtain the target expanded matrix.

7. The method according to claim 1, characterized in that The method further comprises: Get the convolution layer to which the current convolution operation belongs; The convolution pattern of the convolution layer is analyzed to determine the convolution parameters of the convolution layer.

8. A convolution operation device, characterized in that: The device comprises: A reading module is used to read the original feature map used for convolution operation; A loading module is used to load at least one target feature tile used to constitute any sub-feature map in the original feature map from a preset memory layout; the memory layout is obtained by writing at least one feature tile into the memory according to a preset memory layout method; the memory layout includes feature tile blocks corresponding to multiple target positions, and each feature tile block corresponding to the target position corresponds to a row of data in the memory layout; the feature tile block corresponding to each target position is obtained by writing at least two feature tiles that are at the same target position in the original feature map and have different channels into the memory in sequence along a direction corresponding to the channel dimension; the feature tiles are obtained by tiling the original feature map; the memory layout method includes at least a batch dimension, a channel dimension, and a position dimension of the feature tile in the original feature map; an expansion module, configured to expand the feature map composed of the at least one target feature tile according to a convolution parameter of the convolution layer to obtain a target expansion matrix; An operation module is used to perform a matrix multiplication operation on the target expansion matrix and the expansion matrix corresponding to the convolution kernel to obtain a convolution operation result for the original feature map.

9. The device according to claim 8, characterized in that The device is also used to tile the original feature map to obtain the at least one feature tile; write each feature tile into the memory in sequence according to the data arrangement method to obtain the memory layout; wherein the arrangement dimension of the data arrangement method includes at least a batch processing dimension, a channel dimension and a position dimension of the feature tile in the original feature map.

10. The device according to claim 9, characterized in that The device is also used to write at least one feature tile at the same target position in the original feature map into the memory in sequence along a direction corresponding to the channel dimension to obtain a feature tile block corresponding to the target position.

11. The device according to claim 9, characterized in that The device is also used to obtain a tile template used for tiling the original feature map; determine the size of the tile template in at least one direction; perform matrix zero padding on the original feature map so that the size of the zero-padded feature map in the direction satisfies a multiple relationship with the size of the tile template in the direction; and tile the zero-padded feature map according to the tile template to obtain the at least one feature tile.

12. The device according to claim 8, characterized in that Each of the feature tiles has a corresponding tile index coordinate in the memory layout, and the loading module is specifically used to obtain the expanded matrix position coordinates corresponding to any sub-feature map; the expanded matrix position coordinates are used to represent the position information of the target expanded matrix in the expanded matrix corresponding to the original feature map; Mapping the expanded matrix position coordinates to target tile index coordinates; the target tile index coordinates are tile index coordinates corresponding to at least one target feature tile constituting any sub-feature map in the memory layout; loading the feature tile corresponding to the target tile index coordinates in the memory layout to obtain the target feature tile.

13. The device according to claim 8, characterized in that The expansion module is specifically used to expand the feature map composed of the at least one target feature tile according to the convolution parameters of the convolution layer to obtain an expanded matrix; and perform a transpose operation on the expanded matrix to obtain the target expanded matrix.

14. The device according to claim 8, characterized in that The device is also used to obtain the convolution layer to which the current convolution operation belongs; analyze the convolution mode of the convolution layer, and determine the convolution parameters of the convolution layer.

15. A matrix expansion device, characterized in that: include: Tile collection module, pattern parsing module, matrix processing module and matrix cache module, where: The tile collection module is configured to execute acquisition from a texture unit of at least one target feature tile used to constitute any sub-feature map in the original feature map; the target feature tile is obtained by the texture unit loading from a preset memory layout; the memory layout includes a plurality of feature tile blocks corresponding to target positions, each feature tile block corresponding to the target position corresponds to a row of data in the memory layout; the feature tile block corresponding to each target position is obtained by writing at least two feature tiles at the same target position in the original feature map and having different channels into the memory in sequence along a direction corresponding to the channel dimension; The feature tile is obtained by tiling the original feature map; The pattern parsing module is configured to obtain convolution parameters of the convolution layer; The matrix processing module is configured to perform expansion processing on the feature map composed of the at least one target feature tile according to the convolution parameters to obtain a target expansion matrix; The matrix cache module is further configured to execute and cache the target expansion matrix so that the execution unit can generate the convolution operation result of the original feature map based on the target expansion matrix.

16. The device according to claim 15, characterized in that The matrix processing module includes a matrix expansion submodule and a matrix transposition submodule, wherein: The matrix expansion submodule is configured to expand the feature map composed of the at least one target feature tile according to the convolution parameters to obtain an expanded matrix; The matrix transposition submodule is configured to perform a transposition operation on the expanded matrix to obtain the target expanded matrix.

17. The device according to claim 16, characterized in that The convolution parameters include the convolution step size and the convolution kernel size. The matrix expansion submodule is specifically configured to execute, according to the convolution step size and the convolution kernel size, the feature map composed of the at least one target feature tile into at least one row vector in sequence according to the position of the original image, and splice the at least one row vector into a feature map matrix to obtain the expanded matrix.

18. The device according to claim 15, characterized in that The pattern parsing module is specifically configured to execute and obtain the current convolution layer to which the convolution operation belongs; parse the convolution pattern of the current convolution layer, and determine the convolution parameters of the convolution layer.

19. The device according to claim 15, characterized in that The matrix cache module is further configured to execute transmission of the target unfolded matrix to the high-speed shared memory of the execution unit.

20. A graphics processor, characterized in that: include: Texture unit, execution unit and matrix expansion device, wherein: The texture unit is configured to execute, for any sub-feature map in the original feature map, loading at least one target feature tile used to constitute the sub-feature map from a preset memory layout; and is further configured to execute transmitting the at least one target feature tile to the matrix expansion device; the memory layout includes feature tile blocks corresponding to a plurality of target positions, and the feature tile block corresponding to each target position corresponds to a row of data in the memory layout; the feature tile block corresponding to each target position is obtained by writing at least two feature tiles at the same target position in the original feature map and having different channels into the memory in sequence along a direction corresponding to the channel dimension; the feature tiles are obtained by tiling the original feature map; The matrix expansion device includes: a tile collection module, a pattern analysis module, a matrix processing module and a matrix cache module, wherein: the tile collection module is configured to obtain the at least one target feature tile from the texture unit; the pattern analysis module is configured to obtain convolution parameters of a convolution layer; the matrix processing module is configured to perform expansion processing on a feature map composed of the at least one target feature tile according to the convolution parameters to obtain a target expansion matrix; the matrix cache module is further configured to cache the target expansion matrix; The execution unit is configured to execute the target expansion matrix received and transmitted by the matrix expansion device, and perform a matrix multiplication operation on the target expansion matrix and the expansion matrix corresponding to the convolution kernel to obtain the convolution operation result of the original feature map.

21. The graphics processor according to claim 20, wherein: The execution unit is configured to execute sending the expanded matrix position coordinates to the texture unit; the expanded matrix position coordinates are used to represent the position information of the target expanded matrix in the expanded matrix corresponding to the original feature map; The texture unit is configured to execute mapping of the unfolded matrix position coordinates into target tile index coordinates; the target tile index coordinates are tile index coordinates corresponding to at least one target feature tile constituting any sub-feature map in the memory layout; and the feature tile corresponding to the target tile index coordinates is loaded into the memory layout to obtain the target feature tile.

22. The graphics processor according to claim 20, wherein: The graphics processor is configured to perform tiling on the original feature map to obtain the at least one feature tile; write each feature tile into the memory in sequence according to a data arrangement method to obtain the memory layout; wherein the data arrangement method includes at least a batch dimension, a channel dimension, and a position dimension of the feature tile in the original feature map.

23. The graphics processor according to claim 22, wherein: The graphics processor is configured to write at least one feature tile at the same target position in the original feature map into the memory in sequence along a direction corresponding to the channel dimension to obtain a feature tile block corresponding to the target position.

24. The graphics processor according to claim 22, wherein: The graphics processor is configured to obtain a tile template used for tiling the original feature map; determine a size of the tile template in at least one direction; Matrix zero padding is performed on the original feature map so that the size of the zero-padded feature map in the direction satisfies a multiple relationship with the size of the tile template in the direction; and the zero-padded feature map is tiled according to the tile template to obtain the at least one feature tile.

Images