Fpga-based graph convolutional neural network sparse matrix multiplication distribution system

Abstract

The application belongs to the technical field of CPU data processing, and particularly relates to a sparse matrix multiplication distribution system of a graph convolutional neural network based on FPGA. The application is characterized in that a sparse matrix preprocessing module is arranged in a CPU calculator, and an FPGA module is arranged in a memory; an input sparse matrix in a CSC format is sent to the memory, a CSC format file compresses the sparse matrix into three arrays, i.e., a row index record array, a column index record array and a non-zero value array, the sparse matrix preprocessing module retrieves a data stream from the memory, and performs calculation on the row index record array, the column index record array and the non-zero value array of the sparse matrix, thereby realizing sparse matrix multiplication distribution. The CPU data processing rate of the sparse matrix multiplication distribution system of the graph convolutional neural network based on FPGA is fast.

Description

Technical Field

[0001] This invention relates to the field of CPU data processing technology, and in particular to a sparse matrix multiplication allocation system based on FPGA-based graph convolutional neural networks. Background Technology

[0002] In fields such as electromagnetism, semiconductors, robotics, image processing, networks and graphics, molecular dynamics, and fluid dynamics, when using corresponding software to process data, the central processing unit (CPU) is typically used in conjunction with the program. The core algorithms used in these applications are mostly derived from sparse linear algebra, which involves matrix operations. When the number of non-zero elements in a matrix is ​​very small, it is compressed into a special sparse matrix format. This compressed sparse matrix significantly reduces its storage space and speeds up access. Sparse matrix computation is a method for processing large sparse matrices. This method typically uses FPGAs (Field-Programmable Gate Arrays), which are further developed from programmable array logic and general-purpose array logic. They emerged as a semi-custom circuit in the field of application-specific integrated circuits (ASICs), addressing the shortcomings of custom circuits while overcoming the limited gate count of traditional programmable devices. To improve the speed at which the CPU processes data, an FPGA-based graph convolutional neural network sparse matrix multiplication allocation system is needed.

[0003] However, existing graph convolutional neural networks (Graph Convolutional Neural Networks) use buffers within layers to determine which computational processes (PEs) have a heavier workload for sparse-dense matrix multiplication, thus achieving dynamic task allocation. However, this method can be simplified. Because sparse matrices are repeatedly used across layers in Graph Convolutional Neural Networks, the distribution of sparse matrices in each layer is identical. Therefore, a single task allocation for the sparse matrix, applied to each layer simultaneously, can effectively address the load imbalance problem in Graph Convolutional Neural Networks. Existing FPGA-based sparse matrix multiplication processes are similar: first compressing the matrix into the corresponding sparse matrix format, then allocating tasks row by row. The total computational load is randomly distributed across different computational units, making load imbalance difficult to avoid due to the random allocation method. Therefore, this invention proposes a novel FPGA-based sparse matrix multiplication allocation system for Graph Convolutional Neural Networks. Summary of the Invention

[0004] The main objective of this invention is to provide an FPGA-based graph convolutional neural network sparse matrix multiplication allocation system, which can effectively solve the problem of unbalanced load in existing image processing systems using graph convolutional neural networks.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: a graph convolutional neural network sparse matrix multiplication allocation system based on FPGA, comprising: an input device, a CPU, and an output device, wherein the CPU is provided with a memory, an arithmetic unit, and a controller, the arithmetic unit is provided with a sparse matrix preprocessing module, and the memory is provided with an FPGA module;

[0006] The input device is mainly used to input sparse matrices in CSC format and send the input sparse matrices in CSC format to the memory. The CSC format file compresses the sparse matrix into three arrays: a row index record array, a column index record array, and a non-zero value array.

[0007] The sparse matrix preprocessing module retrieves a data stream from memory, which includes an array of row indices, an array of column indices, and an array of non-zero values ​​of the sparse matrix. It then performs operations on these arrays, as follows: First, the row indices record the total number of non-zero elements from the first row to each subsequent row, and the column indices record the column number of each non-zero element. Based on the row index array, the number of non-zero elements in each row of the sparse matrix is ​​determined. Then, the matrix is ​​divided according to the number of non-zero elements in each row. During this division, the row number processed by each computational unit and the total number of non-zero elements in the corresponding row are saved. Based on the total number of non-zero elements, the rows of the sparse matrix are divided sequentially. Each time, the row read by the task allocation unit is assigned to the computational unit with the fewest non-zero elements. Finally, the sparse matrix is ​​allocated. Other computational modules in the CPU's arithmetic unit directly obtain the allocated sparse matrix and its specific data through the sparse matrix preprocessing module.

[0008] Preferably, the sparse matrix preprocessing module divides the CPU registers into blocks, resulting in register blocks. The size of each register block is the number of data reuses. Each thread reads the row index record array of the sparse matrix from global memory, ensuring that the row index record array of the sparse matrix is ​​reused in the registers. The difference in this algorithm lies in adjusting the reading order. The reading order of elements in the row index record array of the sparse matrix is ​​marked with numbers (1, 2, 3, ..., n). In the Warp function, the first thread calculates the first element of the k-th row, and then calculates the first element of the (k+1)-th row. These two elements reuse the row index record array of the sparse matrix. This new memory access order ensures the reuse of elements in the registers and the efficient use of memory. Then, using shared memory, the results calculated by each warp function thread are written to the shared memory. After synchronization, reduction is performed to reuse the data in the shared memory among different warp functions. Specifically, the number of times a thread reuses an element in X is determined by the number of rows calculated by a warp function. Here, the number of register blocks is 2, that is, one warp function calculates 2 rows. At the same time, from the perspective of reuse, the more rows calculated by a warp function, the more beneficial it is to reuse. However, a larger block will cause each thread to occupy more registers, which may reduce the number of threads and CPU utilization, improve computing efficiency, and ultimately improve the speed of CPU processing data and instructions, thereby improving the efficiency of computer data processing.

[0009] Preferably, the CPU, or Central Processing Unit, is the core of a computer's operation and control. Its main functions are to interpret computer instructions and process data in computer software. The CPU includes an arithmetic logic unit (ALU), a control unit, registers, a cache, and a data, control, and status bus that connects them. The CPU is mainly used to process data and arithmetic commands. In this embodiment, it is mainly used for sparse matrix allocation and operation. By performing sparse matrix allocation operations, the efficiency of the CPU in processing data and running the system is improved, and the operating speed is increased.

[0010] Preferably, the memory is divided into external storage and main memory, used to store data, storing data in binary format, an input device (a device for the user to give commands to the computer and input specific content into the CPU), and an output device (a device for the computer to report results to the user).

[0011] Preferably, during the process of dividing the registers in the CPU into blocks, the sparse matrix preprocessing module needs to select the size of the blocks. Different block sizes will have different effects on the performance of the program. A larger block will increase the number of times the row index record array of the sparse matrix is ​​reused, which is beneficial to the performance improvement. However, a larger block will cause each thread to occupy more registers, which may reduce the number of threads and the CPU utilization rate.

[0012] Preferably, the row indices of the three arrays in the CSC format file are used as the input of this multiplication allocation unit, the column index record array and the non-zero value array are entered into the FPGA memory, the calculation unit divides the sparse matrix by row, and the result of the division is stored in the FPGA memory in the form of four arrays.

[0013] Preferably, to reduce the resource consumption caused by on-chip data copying, the column index array and the non-zero element value array are also synchronously divided after being partitioned by row, and stored separately in different BRAMs (Block RAMs). This increases the number of BRAMs without consuming more resources, thereby increasing the system bandwidth and effectively reducing bandwidth requirements. FPGAs have two types of RAM (Random Access Memory) resources: BRAM and Distributed RAM. RAM is synthesized using synthesis tools and implemented through multi-level LUT lookup table resource cascading. It's named "distributed" because it's implemented by cascading LUTs that are geographically distant. Distributed RAM can utilize flexible LUT resources and can be flexibly configured according to usage, making it suitable for applications where RAM latency is not critical. In contrast, BRAM is a dedicated block RAM resource added to the FPGA outside of its logic resources. Compared to distributed RAM, BRAM blocks undergo special placement and routing within themselves and between them and logic resources, resulting in high operating speed and a defined latency period. However, the number of BRAMs on each FPGA is limited; once used, they are gone. Therefore, increasing the number of BRAMs on each block increases the bandwidth of the data buffer. In network communication and digital signal processing applications, BRAM is one of the most important resources for achieving high-speed data caching.

[0014] Preferably, each computing unit can directly read the four pre-divided arrays and the other two arrays in the CSC format except for the row index, thereby performing sparse matrix multiplication operations, which improves the rate of matrix operation and thus increases the speed of CPU execution of commands and data processing.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] In this invention, a sparse matrix preprocessing module is built on the CPU hardware. This module can partition the sparse matrix by row, ensuring that the number of non-zero elements in each part is as similar as possible. After this preprocessing step, other system modules can directly obtain the sparse matrix data from this module without needing to repartition it. Furthermore, because the number of non-zero elements is similar, the load on different computing units is more balanced, thereby improving computational efficiency and speeding up the computation process.

[0017] Since the sparse matrix preprocessing module only performs partitioning operations on the sparse matrix, it can accelerate various forms of sparse matrix multiplication operations to a certain extent.

[0018] Experiments have demonstrated that the sparse matrix preprocessing module effectively achieves load balancing, rapidly partitioning sparse matrices for data with varying structures, ultimately improving the CPU's processing speed for sparse matrix data instructions. According to experiments, for the Cora dataset, after allocation by this preprocessing module, the load difference across different PEs is less than 3%.

[0019] In this invention, the registers in the CPU are divided into blocks in the sparse matrix preprocessing module to obtain register blocks. The size of the register block is the number of times the data is reused. Each thread reads the row index record array of the sparse matrix from the global memory to ensure that the row index record array of the sparse matrix is ​​reused in the register, thereby improving the computational efficiency and ultimately improving the speed of CPU data and instruction processing, thus improving the efficiency of computer data processing. Attached Figure Description

[0020] Figure 1 This is a system block diagram of the sparse matrix multiplication allocation system based on FPGA for graph convolutional neural networks according to the present invention.

[0021] Figure 2 This is a system block diagram of the sparse matrix preprocessing module in the sparse matrix multiplication allocation system of the graph convolutional neural network based on FPGA of the present invention. Detailed Implementation

[0022] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0023] In the description of this invention, it should be noted that the terms "upper," "lower," "inner," "outer," "front end," "rear end," "both ends," "one end," and "the other end," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0024] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installed," "equipped with," "connected," etc., should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0025] Please refer to Figure 1-2 As shown, the present invention provides a graph convolutional neural network sparse matrix multiplication allocation system based on FPGA, including: an input device, a CPU and an output device, wherein the CPU is provided with a memory, an arithmetic unit and a controller, the arithmetic unit is provided with a sparse matrix preprocessing module and the memory is provided with an FPGA module.

[0026] The input device is mainly used to input sparse matrices in CSC format and send the input sparse matrices in CSC format to the memory. The CSC format file compresses the sparse matrix into three arrays: a row index record array, a column index record array, and a non-zero value array.

[0027] The sparse matrix preprocessing module retrieves a data stream from memory, which includes an array of row indices, an array of column indices, and an array of non-zero values ​​for the sparse matrix. It then performs operations on these arrays, as follows: First, the row indices record the total number of non-zero elements from the first row to each subsequent row, and the column indices record the column number of each non-zero element. Based on the row index array, the number of non-zero elements in each row of the sparse matrix is ​​determined. Then, the matrix is ​​divided according to the number of non-zero elements in each row. During this division, the row number processed by each computational unit and the total number of non-zero elements in that row are saved. Based on the total number of non-zero elements, the rows of the sparse matrix are divided sequentially. Each time, the row read by the task allocation unit is assigned to the computational unit with the fewest non-zero elements. Finally, the sparse matrix is ​​allocated. Other computational modules in the CPU's arithmetic unit directly obtain the allocated sparse matrix and its specific data through the sparse matrix preprocessing module.

[0028] The sparse matrix preprocessing module divides the CPU registers into blocks, with the size of each block representing the number of data reuses. Each thread reads the row index record array of the sparse matrix from global memory, ensuring that this array is reused in the registers. The difference in this algorithm lies in the adjusted reading order; the elements in the row index record array are marked with numbers (1, 2, 3, ..., n). In the warp function, the first thread calculates the first element of row k, then the first element of row k+1. These two elements reuse the row index record array. This new memory access order ensures element reuse in registers and merged memory access. Then, using shared memory, the results calculated by each warp function thread are written to the array, and synchronization is set. The reduction process allows data in shared memory to be reused across different warp functions. Specifically, the number of times a thread reuses an element in X is determined by the number of rows calculated by a warp function. Here, the register block size is 2, meaning a warp function calculates 2 rows. From a reuse perspective, the more rows a warp function calculates, the better for reuse. However, a larger block size will cause each thread to occupy more registers, which may reduce the number of threads and CPU utilization, improve computational efficiency, and ultimately increase the speed at which the CPU processes data and instructions, thus improving the efficiency of computer data processing. The warp function is mainly used to clip raster data based on the input vector, and can also perform raster resampling and reprojection operations.

[0029] The CPU, or Central Processing Unit, is the core of a computer's processing and control. Its main functions are to interpret computer instructions and process data in computer software. The CPU consists of an arithmetic logic unit (ALU), a control unit, registers, a cache, and a data, control, and status bus that connects them. The CPU is mainly used to process data and arithmetic commands. In this embodiment, it is mainly used for sparse matrix allocation and calculation. By performing sparse matrix allocation and calculation, the efficiency of the CPU in processing data and running the system is improved, thereby increasing the operating speed.

[0030] Among them, memory is divided into external storage and main memory. It is used to store data and stores data in binary format. Input devices are devices that allow users to give commands to the computer and input specific content into the CPU. Output devices are devices that allow the computer to report results to the user.

[0031] In the process of dividing the registers in the CPU into blocks, the sparse matrix preprocessing module needs to select the size of the blocks. Different block sizes will have different effects on the program's performance. A larger block will increase the number of times the row index record array of the sparse matrix is ​​reused, which is beneficial to performance improvement. However, a larger block will cause each thread to occupy more registers, which may reduce the number of threads and CPU utilization.

[0032] In this multiplication allocation unit, the row indices of the three arrays in the CSC format file are used as inputs, the column index record array and the non-zero value array are entered into the FPGA memory, the calculation unit divides the sparse matrix by row, and the result of the division is stored in the FPGA memory in the form of four arrays.

[0033] To reduce resource consumption caused by on-chip data copying, the column index array and non-zero element value array are simultaneously divided after row-based partitioning and distributed across different BRAMs. This increases the number of BRAMs without consuming more resources, thereby increasing system bandwidth and effectively reducing bandwidth requirements. FPGAs have two types of RAM resources: BRAM and Distributed RAM. Distributed RAM is synthesized using multi-level LUT lookup table cascading. Its name comes from the fact that it is implemented by cascading widely spaced LUTs. Distributed RAM can utilize flexible LUT resources and can be flexibly configured according to usage, making it suitable for applications with low RAM latency requirements. Block RAM, on the other hand... Compared to distributed RAM, BRAM is a dedicated block RAM resource added to the FPGA in addition to the logic resources. Compared to distributed RAM, the RAM block has a special layout and routing between itself and the logic resources, which makes BRAM have a high operating speed and a fixed latency period. However, the number of BRAMs on each FPGA is limited, and once they are used up, they are gone. Therefore, by increasing the number of BRAMs on each block, the bandwidth of data buffering can be increased. In network communication and digital signal processing applications, BRAM is one of the most important resources to achieve high-speed data caching.

[0034] Each computing unit can directly read the four pre-divided arrays and the two arrays in the CSC format other than the row index, thereby performing sparse matrix multiplication operations, which improves the speed of matrix operation and thus increases the speed of CPU execution of commands and data processing.

[0035] In practical use:

[0036] The input device is used to input a sparse matrix in CSC format and send the input CSC format sparse matrix to the memory. The CSC format file compresses the sparse matrix into three arrays: a row index record array, a column index record array, and a non-zero value array. For example, the input sparse matrix is: The sparse matrix is ​​compressed into three arrays using a CSC format file: row index record array [0, 3, 4, 6, 7, 8, 10, 11, 12], column index record array [0, 2, 5, 1, 0, 2, 3, 4, 0, 5, 6, 7], and non-zero value array [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1].

[0037] The sparse matrix preprocessing module retrieves a data stream from memory. This data stream includes arrays of row indices, column indices, and non-zero values ​​of the sparse matrix. It then performs operations on these arrays, as follows: First, the row indices record the total number of non-zero elements from the first row to each subsequent row. The column indices record the column number of each non-zero element. Based on the row index array, the number of non-zero elements in each row of the sparse matrix is ​​calculated. Then, the matrix is ​​partitioned based on the number of non-zero elements in each row. During partitioning, the row number processed by each computational unit and the total number of non-zero elements in that row are saved. In graph convolutional neural networks, sparse matrices are symmetric; therefore, the sparse matrices in the following formula are also symmetric matrices. For example, in the first step:

[0038]

[0039] Based on the total number of non-zero elements, the rows of the sparse matrix are divided sequentially, and each time the row read by the task allocation unit is assigned to the calculation unit with the fewest total number of non-zero elements.

[0040] Step Two:

[0041]

[0042] Repeat the above steps to allocate the rows read by the task allocation unit to the calculation unit with the fewest total number of non-zero elements.

[0043] Step 3:

[0044]

[0045] Repeat the above steps to allocate the rows read by the task allocation unit to the calculation unit with the fewest total number of non-zero elements.

[0046] Step 4:

[0047] ;

[0048] Finally, the sparse matrix allocation is completed. Other processing modules in the CPU's arithmetic unit directly obtain the allocated sparse matrix and its specific data through the sparse matrix preprocessing module.

[0049] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A graph convolutional neural network sparse matrix multiplication allocation system based on FPGA, comprising: The system includes an input device, a CPU, and an output device. The CPU contains a memory, an arithmetic logic unit (ALU), and a controller. The ALU contains a sparse matrix preprocessing module, and the memory contains an FPGA module. The input device is mainly used to input sparse matrices in CSC format and send the input sparse matrices in CSC format to the memory. The CSC format file compresses the sparse matrix into three arrays: a row index record array, a column index record array, and a non-zero value array. The row index records the total number of non-zero elements from the first row to each row, the column index records the column number of each non-zero element, and the non-zero value array records the value of each non-zero element. The sparse matrix preprocessing module retrieves a data stream from memory and performs operations on the row index record array, column index record array, and non-zero value array of the sparse matrix in the data stream. The operation process is as follows: First, the number of non-zero elements in each row of the sparse matrix is ​​obtained based on the row index record array. Then, the matrix is ​​divided according to the number of non-zero elements in each row. During the division, the row number processed by each FGPA computing unit and the total number of non-zero elements in the corresponding row are saved. Based on the total number of non-zero elements, the rows of the sparse matrix are divided in turn. Each time, the row read by the task allocation unit in the CPU is assigned to the computing unit with the fewest non-zero elements, and finally the allocation of the sparse matrix is ​​completed.

2. The FPGA-based graph convolutional neural network sparse matrix multiplication allocation system according to claim 1, characterized in that: The sparse matrix preprocessing module divides the registers in the CPU into blocks, resulting in register blocks. The size of each register block represents the number of times data is reused. Each thread reads the row index record array of the sparse matrix from global memory, ensuring that the row index record array of the sparse matrix is ​​reused in the registers. The reading order of the elements in the row index record array of the sparse matrix is ​​marked with numbers (1, 2, 3, ..., n). In the warp function, the first thread calculates the first element of the k-th row, and then calculates the first element of the (k+1)-th row. These two elements reuse the row index record array of the sparse matrix. Then, using shared memory, the results calculated by each warp function thread are written into the shared memory. After synchronization is set, reduction is performed to ensure that the data in the shared memory is reused among different warp functions.

3. The FPGA-based graph convolutional neural network sparse matrix multiplication allocation system according to claim 1, characterized in that: The row index record array from the three arrays in the CSC format file is used as the input to the multiplication allocation unit in the CPU. The column index record array and the non-zero value array are entered into the FPGA's memory. The calculation unit divides the sparse matrix by row, and the result of the division is stored in the FPGA's memory in the form of four arrays.

4. The FPGA-based graph convolutional neural network sparse matrix multiplication allocation system according to claim 1, characterized in that: To reduce the resource consumption caused by on-chip data copying, the column index array and the non-zero element value array are also divided synchronously after being divided by row, and stored in different BRAMs.

5. The FPGA-based graph convolutional neural network sparse matrix multiplication allocation system according to claim 1, characterized in that: Each computing unit directly reads the four pre-divided arrays, the column index record array in CSC format, and the non-zero value array to perform sparse matrix multiplication operations.

Images