Intelligent computing architecture-oriented data storage and computing collaborative optimization method

Abstract

The invention relates to an intelligent computing architecture-oriented data storage and computing collaborative optimization method and device, computer equipment and a readable storage medium. The method comprises the following steps: segmenting a feature graph in a convolutional neural network according to channel dimensions to obtain feature sub-graphs, and segmenting the feature sub-graphs into feature planes according to height dimensions; segmenting the feature plane according to the width dimension to obtain each feature matrix slice; storing each feature matrix slice in a general storage unit according to a feature map format rule; and sequentially reading each feature matrix slice in the general storage unit, and performing matrix operation on each feature matrix slice to obtain a convolution kernel matrix. By adopting the method, the efficiency of the intelligent computing architecture-oriented data storage and computing collaborative optimization method can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence hardware design technology, and in particular to a method, apparatus, computer equipment, computer-readable storage medium, and computer program product for collaborative optimization of data storage and computing for intelligent computing architecture. Background Art

[0002] To improve the accuracy of convolutional neural networks, they are often trained before being applied and updated using the backpropagation algorithm. The backpropagation calculation process of a convolutional neural network includes the reverse error graph calculation process, the reverse weight gradient calculation process, and the gradient update calculation process.

[0003] In traditional techniques, during the calculation of reverse weight gradients, the feature maps of a convolutional neural network are stored as data tensors in a universal storage unit of a universal storage layer. The feature maps are then read from the universal storage unit, formatted, and weighted to obtain the convolution kernel matrix.

[0004] However, in traditional technologies, format conversion of feature maps consumes a lot of time, resulting in a long calculation time for reverse weight gradients. This, in turn, leads to low efficiency of current data processing methods. Summary of the Invention

[0005] Based on this, it is necessary to provide a data storage and computing collaborative optimization method, device, computer equipment, computer-readable storage medium and computer program product for intelligent computing architecture to address the above technical problems.

[0006] In a first aspect, the present application provides a data storage and computing collaborative optimization method for an intelligent computing architecture, comprising:

[0007] The feature graph in the convolutional neural network is divided according to the channel dimension to obtain a feature subgraph, and the feature subgraph is divided into feature planes according to the height dimension;

[0008] Slicing the feature plane according to the width dimension to obtain each feature matrix slice;

[0009] Storing each of the feature matrix slices in a universal storage unit according to feature map format rules;

[0010] The feature matrix slices in the universal storage unit are sequentially read, and matrix operations are performed on the feature matrix slices to obtain a convolution kernel matrix.

[0011] In one embodiment, the feature map includes an error map and an activation map.

[0012] In one embodiment, the feature subgraph includes an activation subgraph and an error subgraph, and the feature subgraphs are obtained by dividing the feature graph in the convolutional neural network according to the channel dimension, including:

[0013] The activation map and error map in the convolutional neural network are split serially or in parallel according to the channel dimension to obtain the activation submaps and the error submaps.

[0014] In one embodiment, storing each of the feature matrix slices in the universal storage unit according to the feature map format rule includes:

[0015] Determining a starting feature matrix slice in each of the feature matrix slices;

[0016] Starting from the starting feature matrix slice, each feature matrix slice is stored in a universal storage unit of a universal storage level in the order of width from small to large, height from high to low, and channel dimension from small to large.

[0017] In one embodiment, sequentially reading each of the feature matrix slices in the universal storage unit and performing matrix operations on each of the feature matrix slices to obtain a convolution kernel matrix includes:

[0018] Sequentially reading feature matrix slices from the general storage unit and storing the feature matrix slices in a high-speed storage unit;

[0019] Matrix multiplication operations are performed on the feature matrix slices in the high-speed storage unit to obtain intermediate results, and a convolution kernel matrix is ​​constructed according to each of the intermediate results.

[0020] In one embodiment, after storing each of the feature matrix slices in the universal storage unit according to the feature map format rule, the method further includes:

[0021] In response to a feature matrix slice read request, the feature matrix slices are sequentially read from the universal storage unit and the feature matrix slices are multiplexed.

[0022] In a second aspect, the present application also provides a data storage and computing collaborative optimization device for an intelligent computing architecture, comprising:

[0023] The first segmentation module is used to segment the feature map in the convolutional neural network according to the channel dimension to obtain a feature sub-map, and segment the feature sub-map into a feature plane according to the height dimension;

[0024] A second segmentation module is used to segment the feature plane according to the width dimension to obtain feature matrix slices;

[0025] A storage module, configured to store each of the feature matrix slices in a universal storage unit according to a feature map format rule;

[0026] An operation module is used to sequentially read each of the feature matrix slices in the universal storage unit and perform matrix operations on each of the feature matrix slices to obtain a convolution kernel matrix.

[0027] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0028] The feature graph in the convolutional neural network is divided according to the channel dimension to obtain a feature subgraph, and the feature subgraph is divided into feature planes according to the height dimension;

[0029] Slicing the feature plane according to the width dimension to obtain feature matrix slices;

[0030] Storing each of the feature matrix slices in a universal storage unit according to feature map format rules;

[0031] The feature matrix slices in the universal storage unit are sequentially read, and matrix operations are performed on the feature matrix slices to obtain a convolution kernel matrix.

[0032] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0033] The feature graph in the convolutional neural network is divided according to the channel dimension to obtain a feature subgraph, and the feature subgraph is divided into feature planes according to the height dimension;

[0034] Slicing the feature plane according to the width dimension to obtain feature matrix slices;

[0035] Storing each of the feature matrix slices in a universal storage unit according to feature map format rules;

[0036] The feature matrix slices in the universal storage unit are sequentially read, and matrix operations are performed on the feature matrix slices to obtain a convolution kernel matrix.

[0037] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the following steps:

[0038] The feature graph in the convolutional neural network is divided according to the channel dimension to obtain a feature subgraph, and the feature subgraph is divided into feature planes according to the height dimension;

[0039] Slicing the feature plane according to the width dimension to obtain feature matrix slices;

[0040] Storing each of the feature matrix slices in a universal storage unit according to feature map format rules;

[0041] The feature matrix slices in the universal storage unit are sequentially read, and matrix operations are performed on the feature matrix slices to obtain a convolution kernel matrix.

[0042] The above-mentioned data storage and computing collaborative optimization method, device, computer equipment, computer-readable storage medium and computer program product for intelligent computing architecture divides the feature map in the convolutional neural network according to the channel dimension to obtain feature subgraphs, and divides the feature subgraphs into feature planes according to the height dimension; divides the feature planes according to the width dimension to obtain each feature matrix slice; stores each feature matrix slice in a general storage unit according to the feature map format rule; sequentially reads each feature matrix slice in the general storage unit, and performs matrix operations on each feature matrix slice to obtain a convolution kernel matrix. Using this method, the feature map is divided in the order of the channel dimension, height dimension and width dimension to obtain each feature matrix slice, and each feature matrix slice is stored according to the feature map format rule, so that the feature matrix slices can be directly read sequentially and matrix operations can be performed in the subsequent operation process, avoiding the format conversion of the feature map during the operation process, reducing the reverse weight gradient calculation time, and improving the efficiency of the data storage and computing collaborative optimization method for intelligent computing architecture. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 A schematic diagram of performing convolution calculation on a convolution map on an error map according to an exemplary embodiment;

[0045] Figure 2 1. A flowchart of a method for collaborative optimization of data storage and computing for an intelligent computing architecture in one embodiment;

[0046] Figure 3 Schematic diagram of a process for storing feature matrix slices in one embodiment;

[0047] Figure 4 is a schematic diagram of storing a feature map in a universal storage unit in an exemplary embodiment;

[0048] Figure 5 1 is a schematic diagram of a process for calculating feature matrix slices in one embodiment;

[0049] Figure 6 is a schematic diagram of reading a feature matrix slice in an exemplary embodiment;

[0050] Figure 7 is a schematic diagram of performing matrix operations in an exemplary embodiment;

[0051] Figure 8 A structural block diagram of a data storage and computing collaborative optimization device for an intelligent computing architecture in one embodiment;

[0052] Figure 9 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0054] With the rapid development of deep learning technology, the scale and complexity of neural networks continue to increase, and the demand for computing and storage resources is also growing exponentially. Backpropagation, a core algorithm in neural network training, requires extensive storage and computation of intermediate activation values, gradients, and weights. This computationally and storage-intensive nature not only limits training speed but can also lead to performance bottlenecks due to insufficient memory bandwidth or inefficient storage.

[0055] To improve the accuracy of convolutional neural networks, they are often trained and updated using the backpropagation algorithm before they are applied. The backpropagation process for convolutional neural networks includes the reverse error map calculation, the reverse weight gradient calculation, and the gradient update calculation. The reverse error map calculation involves performing tensor calculations on the error map and weights to obtain the error map for the previous layer. The reverse weight gradient calculation primarily involves performing tensor calculations on the neural network's forward activation map and the error map for that layer to obtain the weight gradient map.

[0056] The reverse gradient calculation process of the convolutional neural network is to perform tensor operations between the activation map and the error map. The error map performs a process similar to the convolution calculation on the activation map. Specifically, Figure 1 Schematic diagram of an exemplary embodiment of performing convolution calculation on the error map on the convolution map. Figure 1 As shown, Figure 1The vector contains an error map of dimension [gh, gw, F] and an input activation tensor (activation map) of dimension [H, W, C]. C represents the input channel and F represents the output channel. The error map is split into multiple two-dimensional error maps along the output channel dimension. Each two-dimensional error map undergoes a two-dimensional convolution on C planes of the input activation map. This process is repeated C times, resulting in C independent weight gradient maps. At this point, the error map is replaced along the F dimension to obtain the next batch of C independent weight gradient maps. In summary, after this process is repeated C times for the error and activation maps, a weight gradient tensor of dimension [ky, kx, F, C] is obtained.

[0057] In traditional techniques, during the calculation of reverse weight gradients, the feature maps of a convolutional neural network are stored as data tensors in a universal storage unit of a universal storage layer. The feature maps are then read from the universal storage unit, formatted, and weighted to obtain the convolution kernel matrix.

[0058] However, in traditional technologies, format conversion of feature maps consumes a lot of time, resulting in a long calculation time for reverse weight gradients. This, in turn, leads to low efficiency of current data processing methods.

[0059] Furthermore, traditional techniques typically optimize computation and data storage as independent modules, ignoring the synergy between them. This separate optimization approach makes it difficult to fully exploit the potential of hardware resources, especially when processing large-scale models, where scheduling conflicts between storage, data transfer, and computation tasks increase significantly. Therefore, achieving efficient synergy between data storage and computation has become a key research direction for improving the efficiency of neural network training.

[0060] The present application provides a data storage and computing collaborative optimization method, apparatus, computer equipment, computer-readable storage medium, and computer program product for an intelligent computing architecture. The method segments a feature map in the order of channel dimension, height dimension, and width dimension to obtain feature matrix slices, and stores each feature matrix slice according to the feature map format rules, so that the feature matrix slices can be directly read sequentially and matrix operations can be performed in subsequent operations, thereby avoiding format conversion of the feature map during the operation process, reducing the reverse weight gradient calculation time, and improving the efficiency of the data storage and computing collaborative optimization method for an intelligent computing architecture.

[0061] In addition, this application optimizes the storage process of the feature map before the convolution operation, so that the data calculation process no longer needs format conversion, coordinates the data calculation and data storage processes, and improves the training efficiency of the convolutional neural network.

[0062] In one embodiment, Figure 2As shown, a data storage and computing collaborative optimization method for an intelligent computing architecture is provided. The embodiment of the present application takes the method applied to a computer device as an example for description. The embodiment of the present application does not limit the execution device of the data storage and computing collaborative optimization method for an intelligent computing architecture, and includes the following steps 202 to 208:

[0063] In step 202, the feature graph in the convolutional neural network is divided according to the channel dimension to obtain a feature subgraph, and the feature subgraph is divided into feature planes according to the height dimension.

[0064] In implementation, a dual-cached storage hierarchy is provided in the computer device, specifically comprising general-purpose storage units in a general-purpose storage hierarchy and high-speed storage units in a high-speed storage hierarchy. The general-purpose storage units in the general-purpose storage hierarchy are used to store the input and output matrices or weight matrices of the convolutional neural network during the reverse gradient calculation process, as well as intermediate results during the execution of the convolutional neural network. The high-speed storage hierarchy is also known as the local cache hierarchy. The computer device obtains the feature map of the convolutional neural network from the general-purpose storage hierarchy. This feature map is a three-dimensional tensor. The computer device segments the feature map according to the channel dimension to obtain feature subgraphs. The computer device then segments each feature subgraph into feature planes according to the height dimension.

[0065] Specifically, during the training process of a convolutional neural network, the parameters of the convolutional neural network need to be updated. Updating the parameters of the convolutional neural network requires performing reverse gradient calculations on the convolutional neural network. The computer device obtains the feature map of the convolutional neural network from the general storage unit of the general storage layer. To avoid reprocessing the feature map during the calculation process, the computer device needs to split the feature map into vectors for storage. The computer device splits the feature map according to the input channel dimension or the output channel dimension to obtain each feature subgraph. Then, for each feature subgraph, the computer device splits the feature subgraph into two-dimensional feature planes according to the height dimension.

[0066] Step 204 : Slice the feature plane according to the width dimension to obtain feature matrix slices.

[0067] During implementation, the computer device divides each feature plane again according to the width dimension to obtain feature matrix slices.

[0068] Specifically, the feature plane includes an activation plane and an error plane. A computer device performs planar slicing of each activation plane and error plane, either serially or in parallel, along the width dimension to obtain activation matrix slices and error matrix slices. The feature map includes an error map and an activation map. An error matrix slice is an error vector in an error map, and an activation matrix slice is an activation vector in an activation map.

[0069] Step 206: Store each feature matrix slice in a universal storage unit according to the feature map format rule.

[0070] The feature map format rules include the format of the feature vector and the operation order of the feature map.

[0071] In an implementation, the computer device determines a starting feature matrix slice in each feature matrix slice, and then, starting from the starting feature matrix slice, stores each feature matrix slice in a universal storage unit of the universal storage level according to the operation order of the feature map and the format of the feature vector.

[0072] Specifically, the feature map includes an error map and an activation map. The order of operations on the feature map is the order in which tensor operations are performed on the error map on the activation map. The computer device determines the feature matrix slice with the smallest width, highest height, and smallest channel dimension among each feature matrix slice as the starting feature matrix slice. Then, starting from the starting feature matrix slice, the computer device stores each feature matrix slice in a universal storage unit of the universal storage layer according to the order of tensor operations and the format of the feature vector.

[0073] Step 208 , sequentially read each feature matrix slice in the universal storage unit, and perform matrix operations on each feature matrix slice to obtain a convolution kernel matrix.

[0074] In implementation, the computer device sequentially reads each feature matrix slice from the storage unit of the general storage layer and stores each feature matrix slice in the high-speed storage unit. The computer device then performs matrix operations on the feature matrix slices in the high-speed storage unit to obtain a convolution kernel matrix.

[0075] Specifically, the computer device continuously reads each feature matrix slice from the general storage unit according to the storage order and preset read granularity, obtaining each matrix block. The computer device then stores each feature matrix block in a high-speed storage unit of the high-speed storage layer (local cache layer). The computer device then performs matrix multiplication on the feature matrix slices in the high-speed storage unit to obtain intermediate results. The computer device constructs the convolution kernel matrix based on each intermediate result.

[0076] In the above-mentioned data storage and computing collaborative optimization method for intelligent computing architecture, the feature map is segmented in the order of channel dimension, height dimension and width dimension to obtain each feature matrix slice, and each feature matrix slice is stored according to the feature map format rules, so that the feature matrix slices can be directly read sequentially in the subsequent calculation process and matrix operations can be performed, avoiding the format conversion of the feature map during the calculation process, reducing the reverse weight gradient calculation time, and improving the efficiency of the data storage and computing collaborative optimization method for intelligent computing architecture.

[0077] In an exemplary embodiment, the feature map includes an error map and an activation map.

[0078] In implementation, the feature maps in a convolutional neural network include an error map and an activation map. During the training of a convolutional neural network, it is necessary to perform reverse gradient calculation on the feature maps in the convolutional neural network. The reverse gradient calculation process of the feature map includes performing tensor calculation on the error map on the activation map to obtain the convolution kernel matrix. Therefore, the computer device reads the error map and activation map from a common storage unit. Both the error map and the activation map are three-dimensional tensors. The activation map contains three dimensions, namely the input channel dimension, the height dimension, and the width dimension. The error map also contains three dimensions, namely the output channel dimension, the height dimension, and the width dimension. For example, the activation map is a three-dimensional tensor of [H, W, C], where H represents the height of the activation map, W represents the width of the activation map, and C represents the input channel of the activation map. The error map is a three-dimensional tensor of [gh, gw, F]. Where gh is the height of the error map, gw is the width of the error map, and F is the output channel of the error map.

[0079] In this embodiment, by reading the activation map and the error map, it is convenient to perform reverse weight calculation on the error map and the activation map, obtain the convolution kernel matrix, and realize the training and updating of the convolutional neural network.

[0080] In an exemplary embodiment, the feature subgraph includes an activation subgraph and an error subgraph. The specific process of splitting the feature graph in the convolutional neural network according to the channel dimension to obtain the feature subgraph in step 202 includes:

[0081] The activation map and error map in the convolutional neural network are split serially or in parallel according to the channel dimension to obtain activation submaps and error submaps.

[0082] Among them, the channel dimension includes the input channel dimension and the output channel dimension.

[0083] In implementations, the computer device segments the activation map of the convolutional neural network along the input channel dimension to obtain activation subgraphs. The computer device then segments the error map of the convolutional neural network along the output channel dimension to obtain error subgraphs. Alternatively, while segmenting the activation map, the computer device also segments the error map of the convolutional neural network along the output channel dimension to obtain error subgraphs.

[0084] In an optional embodiment, the computer device segments the activation subgraph into activation planes according to the height dimension of the activation graph. The computer device then segments the activation planes into activation matrix slices according to the width dimension of the activation graph. Simultaneously with or after segmenting the activation subgraphs, the computer device segments the error subgraph into error planes according to the height dimension of the error graph, and segments each error plane into error matrix slices according to the width dimension of the error graph.

[0085] In an exemplary embodiment, the activation map is a three-dimensional tensor of [H, W, C], where H represents the height of the activation map, W represents the width of the activation map, and C represents the input channel of the activation map. The error map is a three-dimensional tensor of [gh, gw, F]. Where gh is the height of the error map, gw is the width of the error map, and F is the output channel of the error map. The computer device divides the activation map with C0 as the granularity in the input channel dimension, that is, divides the WC plane into several WC0 matrices to obtain each activation sub-map. The activation sub-map is a three-dimensional tensor of [H, W, C0]. The computer device divides the activation sub-map into activation plane maps according to the height dimension of the activation map. In this case, the activation plane map is a two-dimensional tensor of [W, C0]. Then, the computer device divides the activation plane map into activation matrix slices according to the width dimension of the activation map. In this case, the activation matrix slices are a one-dimensional tensor of [C0]. Then / simultaneously, the computer device segments the error map at the output channel dimension using the granularity of F0, dividing the wF plane into several wF0 matrices to obtain error subgraphs. Each error subgraph is a three-dimensional tensor of [H, W, F0]. The computer device segments the error subgraph into error plane graphs according to the height dimension of the error map. The error plane graph is a two-dimensional tensor of [W, F0]. The computer device then segments each error plane graph into error matrix slices according to the width dimension of the error map. At this point, the error matrix slices are one-dimensional tensors of [F0].

[0086] In this embodiment, the feature map is segmented in the order of channel dimension, height dimension and width dimension to obtain slices of each feature matrix, so that continuous storage of the width dimension can be achieved in the subsequent storage process, which facilitates the direct processing of the feature matrix slices in the storage order in the subsequent calculation process, avoids the format conversion of the feature map during the calculation process, reduces the reverse weight gradient calculation time, and improves the efficiency of the data storage and computing collaborative optimization method for the intelligent computing architecture.

[0087] In an exemplary embodiment, Figure 3 As shown, the specific processing process of step 206 includes steps 302 to 304. Among them:

[0088] Step 302: Determine a starting feature matrix slice among the feature matrix slices.

[0089] Among them, the feature matrix slices include activation matrix slices and error matrix slices.

[0090] During implementation, the computer device determines, among the feature matrix slices, a feature matrix slice with the smallest width, the highest height, and the smallest channel dimension as the starting feature matrix slice.

[0091] Specifically, the computer device determines the activation matrix slice with the smallest width, the highest height, and the smallest input channel dimension among the activation matrix slices as the starting activation matrix slice. Simultaneously, or subsequently, the computer device determines the error matrix slice with the smallest width, the highest height, and the smallest output channel dimension among the error matrix slices as the starting error matrix slice.

[0092] Step 304 , starting from the initial feature matrix slice, each feature matrix slice is stored in a general storage unit of the general storage level in the order of width from small to large, height from high to low, and channel dimension from small to large.

[0093] During implementation, the computer device starts from the starting feature matrix slice and stores each feature matrix slice into the general storage unit of the general storage level in the order of width from small to large, height from high to low, and channel dimension from small to large.

[0094] Specifically, the computer device stores the current feature plane where the starting feature matrix slice is located in the universal storage unit in the order of width from small to large. Then, the computer device updates the next feature plane of the current feature plane to the current feature plane in the order of height from high to low. Then, the computer device stores each feature matrix slice where the current feature plane is located in the universal storage unit of the universal storage level in the order of width from small to large, until the full amount of feature planes in the current channel dimension where the starting feature matrix slice is located is stored. Then, the computer device updates the current channel dimension to the next channel dimension of the current channel dimension in the order of channel dimensions from small to large, and stores each feature matrix slice in the current channel dimension in the order of width from small to large and height from high to low, until the feature matrix slices in all channel dimensions are stored.

[0095] In an exemplary embodiment, the starting feature matrix slice includes a starting activation matrix slice and a starting error matrix slice. The computer device stores the current activation plane where the starting activation matrix slice is located in a universal storage unit in the order of width from small to large. Then, the computer device updates the next activation plane of the current activation plane to the current activation plane in the order of height from high to low. Then, the computer device stores each activation matrix slice where the current activation plane is located in the universal storage unit of the universal storage level in the order of width from small to large until the full amount of activation planes in the current input channel dimension where the starting activation matrix slice is located is stored. Then, the computer device updates the current input channel dimension to the next input channel dimension of the current input channel dimension in the order of input channel dimensions from small to large, and stores each activation matrix slice in the current input channel dimension in the order of width from small to large and height from high to low until the activation matrix slices in all input channel dimensions are stored.

[0096] While / after storing the activation matrix slices, the computer device stores the current error plane where the starting error matrix slice is located in the universal storage unit in the order of width from small to large. Then, the computer device updates the next error plane of the current error plane to the current error plane in the order of height from high to low. Then, the computer device stores each error matrix slice where the current error plane is located in the universal storage unit of the universal storage level in the order of width from small to large, until the full amount of error planes in the current output channel dimension where the starting error matrix slice is located is stored. Then, the computer device updates the current output channel dimension to the next output channel dimension of the current output channel dimension in the order of output channel dimensions from small to large, and stores each error matrix slice in the current output channel dimension in the order of width from small to large and height from high to low, until the error matrix slices in all output channel dimensions are stored.

[0097] In an exemplary embodiment, Figure 4 FIG. 1 is a schematic diagram of storing a feature map in a universal storage unit in an exemplary embodiment. Figure 4 As shown in , the arrows in the activation diagram are the storage order in the width and height dimensions. Figure 4 As shown, the computer device divides the activation map of the three-dimensional tensor in the format of [H, W, C] into activation matrix slices. The activation matrix slice is a one-dimensional tensor of [C0]. At the same time, the computer device divides the error map of the three-dimensional tensor in the format of [H, W, F] into error matrix slices. The error matrix slice is a one-dimensional tensor of [F0]. The computer device first stores a row of activation matrix slices [C0] at a granularity of C0. Then, the computer device stores W rows of activation matrix slices. The computer device stores the activation matrix slices in (H-1) WC0 matrices (activation planes) in the order of the input channel dimensions from smallest to largest in the same manner as the activation matrix slices are stored.

[0098] While / after storing the error matrix slices, the computer device first stores a row of error matrix slices [F0] at a granularity of F0. The computer device then stores W rows of error matrix slices. The computer device stores the (H-1) error matrix slices in the WF0 matrix (error plane) in the order of the error matrix slice storage format and output channel dimensions from smallest to largest. The error matrix slices and activation matrix slices are stored in this manner so that the minimum granularity that can be read each time the activation map and error map are subsequently read is a basic matrix block. Figure 4This also shows that the present application essentially maps the activation map and the error map. The activation map data is stored in the order in which the error map performs tensor calculations on the activation map, the (1) matrix in the activation map is extended to a vector, and then the data used in the calculation process is extended to vector (1)-vector (9) in sequence. When the error map is read, the two-dimensional data is also read in sequence in vector form. In fact, a mapping and conversion is performed during the storage and reading process. Then the activation matrix and the error matrix data are matrix multiplied to obtain the output convolution kernel matrix.

[0099] In this embodiment, the memory access characteristics of reverse gradient data are utilized to store each feature matrix slice in the order of width from small to large, height from high to low, and channel dimension from small to large, thereby optimizing the reverse calculation data access mode, avoiding unnecessary data conversion overhead from reverse calculation to matrix multiplication, reducing the data transmission frequency and transmission volume across hierarchical storage, and thus reducing the performance limitations of bandwidth bottlenecks.

[0100] In an exemplary embodiment, Figure 5 As shown, the specific processing process of step 208 includes steps 502 to 504. Among them:

[0101] Step 502 : sequentially read feature matrix slices from the general storage unit and store the feature matrix slices in the high-speed storage unit.

[0102] In implementation, the computer device sequentially reads the feature matrix slices from the general storage unit according to a preset read granularity, and stores the feature matrix slices in the high-speed storage unit of the high-speed storage layer (local cache layer).

[0103] Specifically, the feature matrix slices include activation matrix slices and error matrix slices. The computer device synchronously and sequentially reads the activation matrix slices and error matrix slices from the general storage unit at a preset read granularity. The computer device then synchronously stores the activation matrix slices and error matrix slices in a high-speed storage unit of a high-speed storage layer (local cache layer).

[0104] In an exemplary embodiment, FIG6 is a schematic diagram of reading a feature matrix slice in an exemplary embodiment. Figure 6 As shown, after the computer device stores each activation matrix slice in the universal storage unit, a virtual activation matrix is ​​formed. At the same time, after the computer device stores each error matrix slice in the universal storage unit, a virtual error matrix is ​​formed in the universal storage unit. Figure 6The arrows in the figure indicate the order in which the computer reads the activation matrix and error matrix, which is also the order in which the weight gradient matrix is ​​calculated (stored). The computer sequentially reads activation matrix slices from the activation matrix in the universal storage unit at a preset activation read granularity, obtaining activation matrix blocks. Simultaneously, the computer sequentially reads error matrix slices from the error matrix in the universal storage unit at a preset error read granularity, obtaining error matrix blocks.

[0105] For example, the activation matrix is ​​a two-dimensional matrix of [H*W, C]. The error matrix is ​​a two-dimensional matrix of [H*W, F]. The activation reading granularity is [C0, P0]. Among them, gh×gw×N= H*W, gh =H, gw =W, P0 represents gh×gw, that is, a one-dimensional vector spliced ​​together by gh and gw. The error reading granularity is [P0, F0]. Among them, P0 represents gh×gw, that is, a one-dimensional vector spliced ​​together by gh and gw. The computer device sequentially reads activation matrix slices from the activation matrix of the general storage unit according to the activation reading granularity of [C0, P0] to obtain activation matrix blocks. The activation matrix blocks are two-dimensional matrices of [C0, P0]. At the same time, the computer device sequentially reads error matrix slices from the error matrix of the general storage unit according to the error reading granularity of [P0, F0] to obtain error matrix blocks. The error matrix blocks are two-dimensional matrices of [P0, F0].

[0106] Step 504: perform matrix multiplication on the feature matrix slices in the high-speed storage unit to obtain intermediate results, and construct a convolution kernel matrix based on the intermediate results.

[0107] Among them, the feature matrix slices include activation matrix slices and error matrix slices.

[0108] In implementation, the computing device performs matrix multiplication operations (reverse gradient operations) on the activation matrix slices and error matrix slices in the high-speed memory cells to obtain intermediate results. The computing device then stores each intermediate result in the general memory cells of the general memory layer to obtain the convolution kernel matrix.

[0109] In an exemplary embodiment, Figure 7 FIG. 1 is a schematic diagram of performing matrix operations in an exemplary embodiment. Figure 7As shown, the computer device reads activation matrix slices from the general storage unit at a preset read granularity, obtaining the n+2th (n is a non-zero natural number) activation matrix block. Simultaneously, the computer device reads error matrix slices from the general storage unit at a preset read granularity, obtaining the n+2th error matrix block. While the computer device reads the n+2 activation matrix blocks and error matrix blocks, it performs matrix multiplication on the nth activation matrix block and the nth error matrix block in the height storage unit, obtaining an intermediate result and storing the intermediate value in the general storage unit. Simultaneously, the height storage unit also stores the already read n+1th activation matrix block and the n+1th error matrix block. Local cache 1 (high-speed storage unit 1) is used to store activation matrix blocks, local cache 2 (high-speed storage unit 2) is used to store error matrix blocks, and the intermediate value cache is used to store partial sum matrices (intermediate results) of the matrix multiplication calculations.

[0110] The calculation and processing process of the activation map and error map by the computer device during the reverse gradient calculation process is specifically as follows:

[0111] 1) The basic granularity of the local cache level is [F0, P0] * [P0, C0];

[0112] 2) Complete Figure 6 Block matrix multiplication and accumulation operations in the P dimension;

[0113] 3) Complete Figure 6 Computation switching on the middle C dimension;

[0114] 4) Complete the x-dimension calculation of the output gradient map;

[0115] 5) Complete the calculation of the y dimension of the output gradient map;

[0116] 6) Complete the C-dimensional calculation of the output gradient map.

[0117] In this embodiment, by sequentially slicing each feature matrix and performing matrix operations on each feature matrix slice, feature map format conversion is avoided, the time required for reverse gradient calculation is reduced, and the efficiency of the data storage and computational collaborative optimization method for intelligent computing architecture is improved. Furthermore, through the joint design of computation and storage, the utilization of hardware resources (including computational units and storage units) is maximized, fully utilizing the matrix multiplication computation unit to complete the reverse gradient calculation performance, and avoiding performance loss caused by uneven resource utilization.

[0118] In an exemplary embodiment, the feature matrix slices can be reused after the feature matrix slices are stored. After step 206 is executed, the specific processing process of the data storage and computing collaborative optimization method for intelligent computing architecture further includes:

[0119] In response to a feature matrix slice read request, each feature matrix slice is sequentially read from the general storage unit and multiplexed.

[0120] In implementation, when feature matrix slices need to be reused, the computer device receives a feature matrix slice read request, then re-reads each feature matrix slice from the universal storage unit based on the feature matrix slice read request, and reuses each feature matrix slice.

[0121] In an exemplary embodiment, a computer device needs to migrate the current convolutional neural network to another terminal. Specifically, the computer device needs to read all data in the current convolutional neural network and copy all the data to the other terminal. For slices of feature matrix slices, the computer device accepts a feature matrix slice read request and, based on the feature matrix slice read request, rereads each feature matrix slice from a universal storage unit. The computer device then copies each feature matrix slice to the other terminal. The other terminal can reuse the feature matrix slices to retrain the convolutional neural network.

[0122] In this embodiment, by storing and reusing feature matrix slices, the convolutional neural network can be continued to be trained starting from the current feature matrix slice, avoiding retraining the convolutional neural network from scratch, reducing the time for training the convolutional neural network, and improving training efficiency.

[0123] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily performed in sequence in the order indicated by the arrows. Unless clearly stated herein, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps. It is understandable that the various steps in different embodiments can be freely combined as needed, and the various non-contradictory schemes formed by the combination all fall within the scope of protection of this application.

[0124] Based on the same inventive concept, the embodiments of the present application also provide a data storage and computing collaborative optimization device for intelligent computing architectures for implementing the aforementioned data storage and computing collaborative optimization method for intelligent computing architectures. The implementation solution provided by the device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations in the one or more data processing device embodiments provided below can be found in the above-mentioned limitations on the data storage and computing collaborative optimization method for intelligent computing architectures, and will not be repeated here.

[0125] In an exemplary embodiment, Figure 8 As shown, a data storage and computing collaborative optimization device 800 for intelligent computing architecture is provided, including: a first segmentation module 801, a second segmentation module 802, a storage module 803 and an operation module 804, wherein.

[0126] The first segmentation module 801 is used to segment the feature map in the convolutional neural network according to the channel dimension to obtain a feature sub-map, and segment the feature sub-map into feature planes according to the height dimension.

[0127] The second segmentation module 802 is configured to segment the feature plane according to a width dimension to obtain feature matrix slices.

[0128] The storage module 803 is configured to store each of the feature matrix slices in a universal storage unit according to a feature map format rule.

[0129] The operation module 804 is used to sequentially read each of the feature matrix slices in the universal storage unit, and perform matrix operations on each of the feature matrix slices to obtain a convolution kernel matrix.

[0130] In an exemplary embodiment, the feature map includes an error map and an activation map.

[0131] In an exemplary embodiment, the feature subgraph includes an activation subgraph and an error subgraph, and the first segmentation module 801 includes a first segmentation submodule and a second segmentation submodule. The first segmentation submodule 801 is specifically configured to segment the activation graph and the error graph in the convolutional neural network serially or in parallel according to the channel dimension to obtain the activation subgraphs and the error subgraphs.

[0132] In an exemplary embodiment, the storage module 803 is specifically used to determine a starting feature matrix slice in each of the feature matrix slices; starting from the starting feature matrix slice, each of the feature matrix slices is stored in a general storage unit of the general storage level in the order of width from small to large, height from high to low, and channel dimension from small to large.

[0133] In an exemplary embodiment, the operation module 804 is specifically used to sequentially read feature matrix slices from the general storage unit and store the feature matrix slices in the high-speed storage unit; perform matrix multiplication operations on the feature matrix slices in the high-speed storage unit to obtain intermediate results, and construct a convolution kernel matrix based on each of the intermediate results.

[0134] In an exemplary embodiment, the data storage and computing collaborative optimization device 800 for intelligent computing architecture is also used to respond to a feature matrix slice read request, sequentially read each feature matrix slice from the general storage unit, and reuse each feature matrix slice.

[0135] Each module in the aforementioned data storage and computing collaborative optimization device for intelligent computing architecture can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a computer device's memory in software form, so that the processor can call and execute the corresponding operations of each module.

[0136] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 9 As shown. The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via wired or wireless means, and the wireless means can be implemented via Wi-Fi, a mobile cellular network, near-field communication (NFC), or other technologies. When executed by the processor, the computer program implements a data storage and computing collaborative optimization method for an intelligent computing architecture. The display unit of the computer device is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse.

[0137] Those skilled in the art will understand that Figure 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0138] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0139] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0140] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0141] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0142] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0143] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A data storage and computing collaborative optimization method for intelligent computing architecture, characterized in that: The method comprises: The feature graph in the convolutional neural network is divided according to the channel dimension to obtain a feature subgraph, and the feature subgraph is divided into feature planes according to the height dimension; Slicing the feature plane according to the width dimension to obtain each feature matrix slice; Storing each of the feature matrix slices in a universal storage unit according to feature map format rules; The feature matrix slices in the universal storage unit are sequentially read, and matrix operations are performed on the feature matrix slices to obtain a convolution kernel matrix.

2. The method according to claim 1, characterized in that The feature map includes an error map and an activation map.

3. The method according to claim 2, characterized in that The feature subgraph includes an activation subgraph and an error subgraph. The feature subgraph is obtained by dividing the feature graph in the convolutional neural network according to the channel dimension, including: The activation map and error map in the convolutional neural network are split serially or in parallel according to the channel dimension to obtain the activation submaps and the error submaps.

4. The method according to claim 1, wherein Storing each of the feature matrix slices in the universal storage unit according to the feature map format rule includes: Determining a starting feature matrix slice in each of the feature matrix slices; Starting from the starting feature matrix slice, each feature matrix slice is stored in a universal storage unit of a universal storage level in the order of width from small to large, height from high to low, and channel dimension from small to large.

5. The method according to claim 1, wherein The sequentially reading each of the feature matrix slices in the universal storage unit and performing matrix operations on each of the feature matrix slices to obtain a convolution kernel matrix includes: Sequentially reading feature matrix slices from the general storage unit and storing the feature matrix slices in a high-speed storage unit; Matrix multiplication operations are performed on the feature matrix slices in the high-speed storage unit to obtain intermediate results, and a convolution kernel matrix is ​​constructed according to each of the intermediate results.

6. The method according to claim 1, characterized in that After storing each of the feature matrix slices in the universal storage unit according to the feature map format rule, the method further includes: In response to a feature matrix slice read request, the feature matrix slices are sequentially read from the universal storage unit and the feature matrix slices are multiplexed.

7. A data storage and computing collaborative optimization device for intelligent computing architecture, characterized in that: The device comprises: The first segmentation module is used to segment the feature map in the convolutional neural network according to the channel dimension to obtain a feature sub-map, and segment the feature sub-map into a feature plane according to the height dimension; A second segmentation module is used to segment the feature plane according to the width dimension to obtain feature matrix slices; A storage module, configured to store each of the feature matrix slices in a universal storage unit according to a feature map format rule; An operation module is used to sequentially read each of the feature matrix slices in the universal storage unit and perform matrix operations on each of the feature matrix slices to obtain a convolution kernel matrix.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images