A GCN-reRAM vertex feature selective updating method based on staggered mapping

Abstract

The application discloses a GCN-ReRAM vertex feature selective updating method based on staggered mapping. The application aims at the problem that when a graph convolutional neural network is trained by using a ReRAM-based memory-computing integrated system, the total delay of GCN related writing operation is limited to a unit maximum writing delay due to the random distribution of vertices satisfying the power law distribution, a "short board effect" is caused, and the performance of the accelerator is seriously affected. The importance of vertex features is considered in the application, the vertex features with high importance are updated each time back propagation is performed, and the vertex features with low importance are updated multiple times interval back propagation is performed, so that the writing times of the Crossbar are reduced, the writing delay is reduced, and the service life of the Crossbar is prolonged. Meanwhile, the staggered mapping mode adopted in the method can ensure that the updating time of each Crossbar is consistent, and prevents the overall performance of the system from being reduced due to the long updating time of individual Crossbars.

Description

Technical Field

[0001] The present invention belongs to the field of computer science and artificial intelligence systems, and relates to a GCN-ReRAM vertex feature selective update method based on interleaved mapping. Background Art

[0002] The emergence of Graph Neural Networks (GNNs) and related technologies has effectively overcome the limitations of traditional neural network models in Euclidean data and is one of the hottest research directions today. First, due to the irregularity and disorder of graph data, traditional deep learning methods can only process input features in a certain order. GNNs, on the other hand, adopt a vertex-based message propagation mechanism, ignoring the input order of vertex features, effectively solving this problem. Second, due to the interdependence between vertices in graph data, traditional deep learning methods are unable to record the graph structure, resulting in serious information loss. GNNs, on the other hand, rely on the connections between vertices to transmit information, which can more completely preserve the graph structure characteristics and computational accuracy.

[0003] The computation of graph convolutional neural networks mainly consists of two stages: vertex feature update (Combination) and vertex message passing (Aggregation). The Combination operation uses a multi-layer perceptron to perform a linear transformation on vertex features, that is, matrix multiplication of the vertex feature matrix and the weight matrix is ​​performed to update the vertex features. The Aggregation operation transmits vertex feature information along the direction from the source vertex to the destination vertex, that is, each vertex stage collects the updated feature information of its source vertex, aggregates it with its own information, and finally activates it through an activation function to obtain the output features of the layer. The former is a dense matrix-vector multiplication operation. Its static and regular calculation mode gives it a relatively stable computational cost. The latter is the dot product between a sparse adjacency matrix and a dense vertex feature matrix. It has dynamic and irregular characteristics. Its computational cost is greatly affected by the size of the graph data and often increases exponentially with the number of vertices in the graph.

[0004] The PIM accelerator is an emerging hardware device primarily used to accelerate the training or inference process of compute- and storage-intensive applications, including traditional neural networks (CNNs). The core unit of this device is a ReRAM crossbar, which can be used to simulate matrix vector multiplication (MVM) operations. Its structure includes multiple crossbar cells, digital-to-analog converters (DACs), analog-to-digital converters (ADCs), shift and adders (S&A), and sample and hold (S&H).

[0005] When simulating the linear mapping transformation of a neural network, the encoded weight matrix (WeightMatrix, W) can be converted into a conductance map and applied to the ReRAM cells at the intersection of each row and column. The input feature matrix (InputFeatureMap, IFM) is then converted into a voltage by a DAC and applied to each word line. Finally, the current output on each bit line is converted by an ADC, which is the product of the input feature matrix and the weight matrix: IFM*W. The hardware's simulation of the MVM operation reduces its computational time complexity from the original O(n) to O(1), greatly reducing computational overhead and improving the training or inference efficiency of the neural network model.

[0006] When training a GCN model using ReRAM, vertex features must first be mapped to the ReRAM Crossbar, and then the adjacency matrix is ​​used as input to complete the aggregation phase. Therefore, after each backpropagation, the vertex features mapped to the ReRAM Crossbar must be updated.

[0007] Existing work typically maps vertex features by sequence number and updates all vertex features after backpropagation. This full update method based on sequential mapping fails to consider the importance of distinguishing vertex features and generates a large number of crossbar write operations. ReRAM crossbars have high write latency and low write endurance. Therefore, this method results in long crossbar write latency and significantly reduces the crossbar's lifespan. Summary of the Invention

[0008] In view of the above-mentioned deficiencies in the prior art, the object of the present invention is to provide a GCN-ReRAM vertex feature selective update method based on interleaved mapping.

[0009] The present invention achieves this objective through the following technical solution: a GCN-ReRAM vertex feature selective update method based on interleaved mapping, which distinguishes the importance of each vertex according to its vertex degree, then interleavedly maps vertices of different importance on each ReRAM Crossbar array, the core computing unit of the Processing In Memory (PIM) chip, and finally selectively updates the features of vertices with higher importance. The method specifically includes the following steps:

[0010] Step 1: Assume that the number of vertices in the graph data is N V ,The size of each ReRAM Crossbar is S*S. Calculate the degrees of all vertices and sort them in descending order to obtain a descending list of vertex degrees;

[0011] Step 2: Divide the descending list into S intervals in order, with no more than vertices, each interval has the same or similar importance, and different intervals represent vertex clusters of different importance levels;

[0012] Step 3: For each ReRAM Crossbar to be mapped, interleaved mapping is performed. A vertex is selected from each interval and mapped to the ReRAM Crossbar. Each vertex can only be mapped once. This ensures that the vertices mapped on each ReRAM Crossbar come from different intervals, making the degree distribution of the mapped vertices on each ReRAM Crossbar similar.

[0013] Step 4: Perform Graph Convolution Neural Networks (GCN) model training, complete a round of forward propagation and backpropagation, and obtain updated vertex features;

[0014] Step 5: For each vertex mapped on the ReRAM Crossbar, based on the degree of the vertex, selectively update the low-degree vertices within M percent of the descending list, and update the remaining high-degree vertices every period. The parameter M can be dynamically adjusted according to the characteristics of the graph dataset used to train the graph convolutional neural network;

[0015] Step 6. Repeat steps 4 and 5 until the training of the graph convolutional neural network model is completed.

[0016] Furthermore, the selective update described in step 5 is specifically as follows: reducing the update frequency of the low-degree vertices without losing model accuracy or with an acceptable loss of model accuracy, so as to speed up the training process of the graph neural network without significantly reducing accuracy; using a predefined ratio to divide the vertices into important and unimportant categories, with important vertices updated in each period and unimportant vertices updated periodically; before selectively updating vertex features, in order to ensure that the update delays of each ReRAM Crossbar are as balanced as possible, an interleaved mapping method is used to evenly map vertices of different degrees to each ReRAM Crossbar to avoid the short board effect.

[0017] Furthermore, the combination of interleaved mapping and selective update not only makes the number of updated rows of the two ReRAM Crossbars equal, but also ensures that the two ReRAM Crossbars discard the same number of low-degree non-important vertices, thereby improving system performance while avoiding excessive degradation of model accuracy.

[0018] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the GCN-ReRAM vertex feature selective update method based on interleaved mapping.

[0019] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the GCN-ReRAM vertex feature selective update method based on interleaved mapping.

[0020] The beneficial effect of the present invention is that, compared with the full update method based on sequential mapping, the present invention takes into account the importance of vertex features. Vertex features with higher importance are updated each time through backpropagation, while unimportant vertices are updated multiple times through backpropagation at intervals. This not only reduces the number of Crossbar writes and its write latency, but also increases the service life of the Crossbar. At the same time, the interleaved mapping method adopted by this method can ensure that the update time of each Crossbar is consistent, preventing the update time of individual Crossbars from being too long, which reduces the overall performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Schematic diagram of the graph convolutional neural network calculation mode; (a) is the graph data structure diagram, (b) is the calculation mode diagram of the combination stage of the graph convolutional neural network, and (c) is the calculation mode diagram of the aggregation stage of the graph convolutional neural network;

[0022] Figure 2This is the ReRAM Crossbar structure and calculation principle diagram;

[0023] Figure 3 Schematic diagram of interleaved mapping strategy;

[0024] Figure 4 Schematic diagram of selective update based on interleaved mapping;

[0025] Figure 5 Flowchart of the GCN-ReRAM vertex feature selective update method based on interleaved mapping. DETAILED DESCRIPTION

[0026] The present invention will be further described below with reference to embodiments and accompanying drawings.

[0027] like Figure 5 As shown, an embodiment of the present invention provides a GCN-ReRAM vertex feature selective update method based on interleaved mapping, comprising the following steps:

[0028] Step 1. Assume that the number of vertices in the graph data is N V ,The size of each ReRAM Crossbar is S×S. Calculate the degrees of all vertices and sort them in descending order to obtain a descending list of vertex degrees.

[0029] Step 2. Divide the list into S intervals in order, each interval has Vertices in the same interval have similar degrees, so they can be considered to be of equal or similar importance. Different intervals represent vertex clusters of different importance levels. That is, each ReRAM Crossbar sequentially interleaves the vertex features from different intervals, and vertices of equal importance (located in the same interval) are mapped to different ReRAM Crossbars.

[0030] Step 3. For each ReRAM Crossbar with data to be mapped, select a vertex from each interval to map to the ReRAM Crossbar, and each vertex can only be mapped once. This strategy ensures that the vertices mapped on each ReRAM Crossbar come from different intervals, so that the degree distribution of the mapped vertices on each ReRAM Crossbar is close. The structure of ReRAM Crossbar is as follows Figure 2 As shown in FIG, a row-by-row mapping method is adopted when mapping vertex features, that is, each row maps the features of a vertex, and each dimension of the feature vector is mapped to one or more Crossbar units.

[0031] Step 4. Perform graph convolution neural model training, complete a round of forward propagation and back propagation, and obtain the updated vertex features. Figure 1 As shown in Figures (b) and (c), the forward propagation of the graph convolutional neural network is divided into two stages: combination and aggregation. Figure 1 In the graph dataset structure shown in Figure (a), the combination phase uses the weight matrix to update the features of each vertex, such as the feature of vertex A at layer l-1. After the weight matrix is ​​updated, it becomes the intermediate feature of layer l In the Aggregation stage, each vertex aggregates the intermediate features of its source vertex to obtain the output result of this layer. For example, the source vertices of vertex A are vertices B, C, and E, so vertex A aggregates The features of layer l are obtained The back propagation phase of the graph convolutional neural network starts from the last layer and calculates the gradient of the weight matrix and vertex features in reverse layer by layer for subsequent updates. The essence of the combination and aggregation operations is matrix multiplication. The characteristics of ReRAMCrossbar enable it to efficiently simulate matrix multiplication operations, such as Figure 2 As shown, it is assumed that the conductance of the ReRAM cell at the intersection of the i-th word line and the j-th bit line is G i,j , the current applied on the i-th word line is V i According to Ohm's law and Kirchhoff's law, the current on the j-th bit line is The result is the product of the input voltage of each row and the conductance on the same column. After each calculation, the result output on each bit line is the product of a single vector and the matrix mapped on the ReRAM Crossbar. The result of multiple input calculations is the product of the two matrices.

[0032] Step 5. For each vertex mapped on the ReRAM Crossbar, based on the degree of the vertex, selectively update the low-degree vertices within M percent of the descending list, and update the remaining high-degree vertices every period. The parameter M can be dynamically adjusted according to the characteristics of the graph dataset used to train the graph convolutional neural network.

[0033] Specifically, selective updating involves the following: in graph data, the degrees of vertices follow a power-law distribution, with the vast majority of vertices being low-degree vertices and only a very small number of vertices having higher degrees. Low-degree vertices have less impact on the training accuracy of the graph neural network due to the small amount of information they receive and transmit. However, a single low-degree vertex requires the same update time as a high-degree vertex. Therefore, the present invention proposes reducing the update frequency of these low-degree vertices without sacrificing model accuracy or with an acceptable loss of model accuracy, thereby accelerating the training process of the graph neural network without significantly reducing accuracy. Vertices are divided into important and unimportant categories using a predefined ratio. Important vertices are updated in each period, while unimportant vertices are updated periodically. Before selectively updating vertex features, to ensure that the update delays of each ReRAM Crossbar are as balanced as possible, the present invention also uses an interleaved mapping method to evenly map vertices of different degrees to each ReRAM Crossbar to avoid the short board effect.

[0034] Step 6. Repeat steps 4-5 until the training of the graph convolutional neural model is completed.

[0035] The interleaved mapping proposed in the present invention is specifically as follows: the interleaved mapping strategy is as follows Figure 3 As shown, assuming that the graph data has a total of N V Vertices, the size of each ReRAM Crossbar is S×S, and the interleaving mapping strategy works as follows. First, the degrees of all vertices are calculated and sorted in descending order. Then, the reordered vertex list is divided into S intervals, each interval contains vertices. Because vertices within the same interval have similar degrees, they can be considered equally important. Different intervals represent groups of vertices with varying levels of importance. During vertex feature mapping, a vertex is selected from each interval and assigned to a specific ReRAM Crossbar, and each vertex can only be assigned once. This strategy ensures that each ReRAM Crossbar maps vertices from different intervals, achieving a more balanced distribution of vertex degrees across all ReRAM Crossbars.

[0036] The present invention combines interleaved mapping with selective updating, which not only makes the number of updated rows of the two ReRAM Crossbars equal, but also ensures that the two ReRAM Crossbars discard the same number of low-degree non-important vertices, thereby improving system performance while avoiding excessive degradation of model accuracy. Figure 4As shown, assuming that the first 50% of the vertex degrees in the descending order are high-degree vertices, then V2, V4, V1, and V3, with degrees of 500, 450, 300, and 250, respectively, are determined to be high-degree vertices. Vertex degrees with higher degrees represent more source vertices and richer information collected, thus being more important and having a greater impact on model accuracy, requiring backpropagation updates each time. Conversely, vertex degrees with lower degrees represent less importance and can be updated with backpropagation multiple times. In this embodiment, these four high-degree vertices are classified as important groups and evenly mapped to all ReRAM crossbars (two vertices per ReRAM crossbar). The degrees of the discarded vertices on the two ReRAM crossbars are 15, 2, 10, and 1, respectively, all of which are unimportant, low-degree vertices.

[0037] An embodiment of the present invention also provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the GCN-ReRAM vertex feature selective update method based on interleaved mapping.

[0038] An embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the method for selectively updating GCN-ReRAM vertex features based on interleaved mapping is implemented.

[0039] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

Claims

1. A GCN-ReRAM vertex feature selective update method based on interleaved mapping, characterized in that: The importance of each vertex is distinguished by its degree. Then, vertices of different importance are interleaved on each ReRAM Crossbar of the ReRAM Crossbar array, the core computing unit of the in-memory PIM chip. Finally, the features of the more important vertices are selectively updated. The specific steps include the following: (1) Assume that the number of vertices in the graph data is N V ,The size of each ReRAM Crossbar is S*S. Calculate the degrees of all vertices and sort them in descending order to obtain a descending list of vertex degrees; (2) Divide the descending list into S intervals in order, and each interval has no more than vertices, each interval has the same or similar importance, and different intervals represent vertex clusters of different importance levels; (3) For each ReRAM Crossbar to be mapped, interleaved mapping is performed. A vertex is selected from each interval and mapped to the ReRAM Crossbar. Each vertex can only be mapped once. It is ensured that the vertices mapped on each ReRAM Crossbar come from different intervals, so that the degree distribution of the mapped vertices on each ReRAM Crossbar is close. (4) Conduct graph convolutional neural network model training, complete a round of forward propagation and backpropagation, and obtain updated vertex features; (5) For each vertex mapped on the ReRAM Crossbar, based on the degree of the vertex, the low-degree vertices within M percent of the descending list are selectively updated, and the remaining high-degree vertices are updated every period. The parameter M can be dynamically adjusted according to the characteristics of the graph dataset used to train the graph convolutional neural network; (6) Repeat steps (4) to (5) until the training of the graph convolutional neural network model is completed.

2. A GCN-ReRAM vertex feature selective update method based on interleaved mapping according to claim 1, characterized in that: The selective update in step (5) is specifically as follows: reducing the update frequency of the low-degree vertices without losing the model accuracy, so as to speed up the training process of the graph neural network without reducing the accuracy; using a predefined ratio to divide the vertices into important and unimportant categories, important vertices are updated in each period, and unimportant vertices are updated regularly; before selectively updating the vertex features, in order to ensure that the update delay of each ReRAM Crossbar is as balanced as possible, an interleaved mapping method is used to evenly map vertices of different degrees to each ReRAM Crossbar to avoid the short board effect.

3. An electronic device comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement a GCN-ReRAM vertex feature selective update method based on interleaved mapping as described in any one of claims 1-2.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the GCN-ReRAM vertex feature selective update method based on interleaved mapping is implemented.

Images