Hardware acceleration device for processing dynamic graph convolutional neural network

Abstract

The invention discloses a hardware acceleration device for processing a dynamic graph convolutional neural network, and the device comprises an FPGA parallel acceleration system which comprises a plurality of acceleration modules used for processing different types of calculation tasks; the dynamic data flow control system is used for calling acceleration modules to carry out data calculation according to a data processing task of the dynamic graph convolutional neural network, and managing data transmission and states between the called acceleration modules; and the memory management and data scheduling system is used for distributing a memory space for the acceleration module and storing input data required when the data processing task is executed, an intermediate calculation result obtained when the data processing task is executed and output data obtained after the data processing task is executed. The invention provides a hardware acceleration scheme with low delay, high throughput and high energy efficiency ratio for the dynamic graph convolutional neural network.

Description

Technical Field

[0001] The present invention belongs to the technical field of hardware acceleration, and in particular relates to a hardware acceleration device for processing a dynamic graph convolutional neural network. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, deep learning has achieved remarkable results in computer vision, natural language processing, and data mining. Among them, convolutional neural networks (CNNs) have become a hot topic in research and application due to their outstanding performance in image recognition and processing. However, traditional CNNs are mainly suitable for regular grid data (such as images), but have certain limitations for irregular structured data (such as graphs, social networks, and molecular structures). Dynamic graph convolutional neural network (DGCNN), as a new type of neural network structure, is designed to process irregular structured data.

[0003] DGCNN can dynamically build and update the structure of the graph according to the characteristics of the input data, so as to better capture and learn the intrinsic relationship of the data. This makes DGCNN perform well in applications such as processing point cloud data, social network analysis, and molecular structure prediction. Although DGCNN has significant advantages in processing complex data structures, its computational complexity is high, and the training and inference processes require a lot of computing resources. Although traditional acceleration methods based on CPU or GPU can provide certain computing power, they still have shortcomings in energy consumption, latency, and parallel processing capabilities. FPGA (field programmable gate array), as a flexible and efficient hardware accelerator, has become an important choice for deep learning acceleration due to its low latency, high throughput, and high energy efficiency. In recent years, FPGA, as an efficient hardware accelerator, has gradually been applied to the field of deep learning. However, existing research has only achieved the acceleration of different types of convolutional neural networks (CNNs) and graph neural networks (GNNs) on FPGA. In other words, it is urgent to design a hardware acceleration solution for DGCNN with low latency, high throughput, and high energy efficiency. Summary of the invention

[0004] In order to solve the above problems existing in the prior art, the present invention provides a hardware acceleration device for processing dynamic graph convolutional neural networks.

[0005] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] The present invention provides a hardware acceleration device for processing a dynamic graph convolutional neural network, comprising: a dynamic data flow control system, a memory management and data scheduling system, an FPGA parallel acceleration system, and a local cache; wherein the FPGA parallel acceleration system is connected to the dynamic data flow control system and the memory management and data scheduling system respectively;

[0007] The FPGA parallel acceleration system includes a plurality of acceleration modules, each of which is used to execute a type of computing task in the dynamic graph convolutional neural network, and different types of acceleration modules execute different types of computing tasks;

[0008] The dynamic data flow control system is used to call the acceleration module to perform data calculation according to the data processing task of the dynamic graph convolutional neural network, and manage the data transmission and status between the called acceleration modules;

[0009] The memory management and data scheduling system is used to allocate memory space for the acceleration module, and store input data required for executing the data processing task, intermediate calculation results obtained when executing the data processing task, and output data obtained after executing the data processing task;

[0010] The local cache is used to store the intermediate calculation results of the acceleration module.

[0011] Compared with the prior art, the present invention has the following beneficial effects:

[0012] The present invention introduces dynamic data flow control and flexible memory management through a dynamic data flow control system and a memory management and data scheduling system, thereby improving the efficiency of data transmission and processing. At the same time, through an FPGA parallel acceleration system including a variety of acceleration modules, the computing parallelism and hardware utilization can be improved, thereby improving the computing efficiency and energy efficiency ratio, enhancing the stability and flexibility of the computing system, and realizing efficient and stable processing of the DGCNN model. It provides a hardware acceleration solution with low latency, high throughput and high energy efficiency ratio for DGCNN, and can provide more flexible and stable hardware acceleration support in various application scenarios such as point cloud data processing and network analysis.

[0013] The present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic diagram of a network structure of a dynamic graph convolutional neural network supported by a hardware acceleration device provided in an embodiment of the present invention;

[0015] Figure 2 It is a schematic diagram of the connection between the hardware acceleration device and the host provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0016] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0017] The present invention provides a hardware acceleration device for processing dynamic graph convolutional neural network, and the hardware acceleration device is designed based on FPGA. The hardware acceleration device comprises: a dynamic data flow control system, a memory management and data scheduling system, an FPGA parallel acceleration system, and a local cache; wherein the FPGA parallel acceleration system is respectively connected to the dynamic data flow control system and the memory management and data scheduling system.

[0018] The FPGA parallel acceleration system includes a variety of acceleration modules, each of which is used to execute a type of computing task in the dynamic graph convolutional neural network, and different types of acceleration modules execute different types of computing tasks. It should be noted that the acceleration modules included in the FPGA parallel acceleration system can execute computing tasks in parallel, and the types of acceleration modules included in the FPGA parallel acceleration system and the number of each acceleration module can be set according to actual needs, which is not limited by the present invention. Exemplarily, the types of computing tasks in the dynamic graph convolutional neural network include 1×1 convolution operations, matrix multiplication, basic operations including addition, subtraction, multiplication, and division operations, dimensionality reduction operations, sorting and TopK selection operations, Gather operations, and splicing operations.

[0019] The dynamic data flow control system is used to call the acceleration module to perform data calculation according to the data processing task of the dynamic graph convolutional neural network, and manage the data transmission and status between the called acceleration modules. It should be noted that the data processing task of the dynamic graph convolutional neural network can be a data processing task of the entire dynamic graph convolutional neural network, or it can be a data processing task of a partial network in a dynamic graph convolutional neural network. In other words, the present invention can provide hardware accelerated computing for a complete dynamic graph convolutional neural network, and can also provide hardware accelerated computing for partial network modules or network layers in a complete dynamic graph convolutional neural network. Exemplarily, Figure 1 This is a schematic diagram of the network structure of a dynamic graph convolutional neural network supported by the present invention. The dynamic graph convolutional neural network can process point cloud data, wherein n×3, n×64, n×128, and 1024 all represent data dimensions, and the network module Edge Conv MLP includes two parts: Edge Conv and MLP.

[0020] The memory management and data scheduling system is used to allocate memory space for the acceleration module, and to store the input data required when executing data processing tasks, the intermediate calculation results obtained when executing data processing tasks, and the output data obtained after executing data processing tasks.

[0021] The local cache is used to store the intermediate calculation results of the acceleration module. Here, using the local cache to store the intermediate calculation results can reduce the frequency of access to the external memory.

[0022] In some embodiments, the multiple acceleration modules may include four acceleration modules: a shared-weight multilayer perceptron (Share-MLP) acceleration module, a matrix multiplication (MatMul) acceleration module, a basic operation (BasicOps) acceleration module, and a dimensionality reduction (Reduction) acceleration module. That is to say, the FPGA parallel acceleration system includes four acceleration modules: a shared-weight multilayer perceptron (Share-MLP) acceleration module, a matrix multiplication (MatMul) acceleration module, a basic operation (BasicOps) acceleration module, and a dimensionality reduction (Reduction) acceleration module. The Share-MLP acceleration module is used to perform 1×1 convolution and matrix multiplication operations; the MatMul acceleration module is used to perform matrix multiplication operations; the BasicOps acceleration module is used to perform basic operations including addition, subtraction, multiplication, and division; the Reduction acceleration module is used to perform dimensionality reduction operations, for example, for performing dimensionality reduction operations such as ReduceSum and ReduceMax.

[0023] In some embodiments, in addition to the above four acceleration modules, the multiple acceleration modules also include a TopK acceleration module, which is used to determine the data in the Top K position from the data to be sorted, where the value of K can be set according to actual needs.

[0024] In some embodiments, in addition to the above four or five acceleration modules, the multiple acceleration modules also include two acceleration modules: a Gather acceleration module and a Concat acceleration module; the Gather acceleration module is used to perform a Gather operation, and the Concat acceleration module is used to perform a concatenation operation.

[0025] In some embodiments, each acceleration module includes: an input data buffer, an output data buffer, and multiple parallel computing units. The structure of each computing unit can be designed using GemmHLS, and each computing unit contains multiple parallel computing cores, each computing core contains a multiplier and an adder; the input data buffer and the output data buffer are both connected to the memory management and data scheduling system; the input data buffer and the output data buffer in each acceleration module are also connected to the multiple parallel computing units in the acceleration module through a local bus. It should be noted that the internal structures of the computing units contained in different acceleration modules are different, and the computing units contained in different acceleration modules and the number of computing cores contained in each computing unit can be the same or different. In addition, the computing units contained in each acceleration module, the number of computing cores contained in each computing unit, and the number of adders and multipliers contained in each computing core can be set according to actual needs, and the present invention does not limit this.

[0026] Specifically, the input data buffer in the Share-MLP acceleration module is used to store data that needs to be subjected to 1×1 convolution and matrix multiplication operations, the output data buffer in the Share-MLP acceleration module is used to store the results of 1×1 convolution and matrix multiplication operations, the multiple parallel computing units in the Share-MLP acceleration module are used to perform 1×1 convolution and matrix multiplication operations, and each computing unit supports pipeline operations. The input data buffer in the MatMul acceleration module is used to store data that needs to be subjected to matrix multiplication operations, the output data buffer in the MatMul acceleration module is used to store the results of matrix multiplication operations, and the multiple parallel computing units in the MatMul acceleration module are used to perform matrix multiplication operations. The input data buffer in the BasicOps acceleration module is used to store operands, the output data buffer in the BasicOps acceleration module is used to store the results of basic operations, the multiple parallel computing units in the BasicOps acceleration module are used to perform basic operations such as addition, subtraction, multiplication, and division, and each computing unit supports pipeline operations and parallel computing. The input data buffer in the Reduction acceleration module is used to store data that needs to be reduced in dimension, and the output data buffer in the Reduction acceleration module is used to store the results of the dimension reduction operation. The multiple parallel computing units in the Reduction acceleration module are used to perform dimension reduction operations, and each computing unit supports pipeline operations and parallel computing. The input data buffer in the TopK acceleration module is used to store data to be sorted, and the output data buffer in the TopK acceleration module is used to store the results of the Top K operation. The multiple parallel computing units in the Reduction acceleration module are used to perform the Top K sorting operation.

[0027] In some embodiments, the dynamic data flow control system includes: a task scheduler and a data transmission controller, wherein each acceleration module is respectively connected to the task scheduler and the data transmission controller; specifically, each acceleration module is respectively connected to the task scheduler and the data transmission controller via an AXI interface. The task scheduler is used to determine the dependencies and priorities between the computing tasks to be executed according to the data processing tasks of the dynamic graph convolutional neural network, and call the acceleration module to execute the computing tasks according to the dependencies and priorities between the computing tasks to be executed. It should be noted that the task scheduler determines which acceleration modules need to be called to execute which computing tasks respectively according to the dependencies and priorities between the computing tasks to be executed, and the task scheduler can call one or more acceleration modules each time. Exemplarily, the task scheduler uses the OpenCL event mechanism to track the data dependencies and computing priorities between different computing tasks to ensure that the computing tasks are executed in order to avoid resource conflicts. The task scheduler can use other scheduling algorithms, which are not limited by the present invention. The data transfer controller is used to manage the data transmission and status between the called acceleration modules using the OpenCL API. Specifically, data is transmitted between the called acceleration modules through the AXI interface. The data transfer controller dynamically adjusts the data flow by monitoring the status of the called acceleration modules. In this way, the real-time and consistency of the data can be ensured and the computing efficiency can be optimized. It should be noted that the data transfer controller uses the OpenCL API to monitor the execution status of each called acceleration module and control the data transmission between the called acceleration modules. Specifically, when the data that the acceleration module A needs to calculate is the calculation result of the acceleration module B, the data transfer controller can monitor the execution status of the acceleration module B in real time, and after monitoring the acceleration module B outputting the calculation result, control the calculation result of the acceleration module B to be transmitted to the input data buffer of the acceleration module A, so that the acceleration module A can perform subsequent calculations on the calculation result.

[0028] In some embodiments, the memory management and data scheduling system includes: an external DDR memory and a memory controller, wherein the input data buffer and the output data buffer in each acceleration module are connected to the external DDR memory and the memory controller through an AXI interface. The external DDR memory is used to store the input data required when executing the data processing task, the intermediate calculation results obtained when executing the data processing task, and the output data obtained after executing the data processing task. Here, the output data buffer is connected to the external DDR memory through the AXI interface to ensure efficient storage of the calculation results; the output data buffer is connected to the corresponding computing unit (CU) through the local bus, and the calculation results can be received in time, and the calculation results are written to the external DDR memory through the AXI interface. The memory controller is used to allocate the size of the input data buffer and the size of the output data buffer for each acceleration module, and control the read and write operations of the data in the external DDR memory, and ensure the consistency and validity of the data. Specifically, the memory controller can allocate the size of the input data buffer to the acceleration module according to the dimension and data type of the data that each called acceleration module needs to calculate, and allocate the size of the output data buffer to the acceleration module according to the dimension and data type of the calculation result output by the acceleration module.

[0029] For example, Figure 2 As shown, the hardware acceleration device provided by the present invention is connected to the host through a PCle interface, and is used to perform hardware accelerated calculation according to the host's instructions, and return the final calculation result to the host. Specifically, the task scheduler in the hardware acceleration device starts working according to the received host's instructions, the data transmission controller outputs the final calculation result to the external DDR memory, and the memory controller returns the final calculation result stored in the external DDR memory to the host.

[0030] Exemplarily, a workflow of a hardware acceleration device for processing a dynamic graph convolutional neural network provided by the present invention is as follows:

[0031] First, the external DDR reads input data (e.g., point cloud data set) from the outside, and the task scheduler receives the start instruction from the host. After that, the task scheduler uses the OpenCL event mechanism to assign computing tasks to the corresponding acceleration modules according to the dependencies and priorities of the computing tasks. After assigning the tasks, the memory controller configures the size of the input data buffer of the acceleration module that first executes the computing task according to the dimension and data type of the data to be stored in the input data buffer of the acceleration module that first executes the computing task. The data transfer controller stores the data in the input data buffer of the acceleration module that first executes the computing task, and then starts the specific calculation. During the specific calculation process, the task scheduler uses the OpenCL event mechanism to track the execution status of the calculation task to ensure that the calculation tasks are executed in sequence and avoid resource conflicts; at the same time, the memory controller continues to configure the size of the input data buffer of the acceleration module according to the dimension and data type of the data that needs to be stored in the acceleration module that needs to perform the calculation task, and configures the size of the output data buffer of the acceleration module according to the dimension and data type of the result calculated by the acceleration module that performs the calculation task; the data transfer controller transmits data between the input data buffer and output data buffer of the acceleration module that performs the calculation task, and the output data buffer of the acceleration module that performs the calculation task and the external DDR memory through the AXI interface to ensure the real-time and consistency of the data, and at the same time, store some intermediate calculation results in the external DDR memory. For example, when the input data is a point cloud data set containing 1024 points, and the dependencies and priorities of the computing tasks determined by the task scheduler are: 1×1 convolution and matrix multiplication, neighborhood feature extraction, basic operations and dimensionality reduction, and 1×1 convolution and matrix multiplication, neighborhood feature extraction, basic operations and dimensionality reduction are executed in sequence, the specific calculation process is: the Shared-MLP acceleration module reads the data in the input data buffer, performs 1×1 convolution operations and matrix multiplications. Specifically, for each point in the point cloud data set, the 1×1 convolution operation multiplies its coordinate value with the weight matrix to generate a new feature vector, where the size of the weight matrix is ​​3×64, indicating that the 3 coordinate values ​​of each point are converted into 64 eigenvalues. After that, the calculation results are stored in the output data buffer of the Shared-MLP acceleration module. Next, the topK acceleration module reads the output data of the previous step and performs neighborhood feature extraction. Specifically, for each point, the features of the 20 nearest points are extracted to generate a new feature matrix, where the nearest point is selected based on the Euclidean distance. The calculation results are then stored in the output data buffer of the topK acceleration module. Next, the BasicOps acceleration module further processes the extracted neighborhood features. Specifically, it performs operations such as addition, subtraction, multiplication, and division on the extracted neighborhood features. The calculation results are then stored in the output data buffer of the BasicOps acceleration module.Next, the Reduction accelerator module performs ReduceSum and ReduceMax operations on the calculation results of the BasicOps accelerator module to generate the final feature vector and stores the calculation results in the output data buffer of the Reduction accelerator module. After the specific calculation is completed, the data transfer controller outputs the final calculation results to the external DDR memory, and the memory controller returns the final calculation results stored in the external DDR memory to the host.

[0032] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0033] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

[0034] In the specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality of components. Certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good effects.

[0035] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A hardware acceleration device for processing dynamic graph convolutional neural networks, characterized in that: include: A dynamic data flow control system, a memory management and data scheduling system, an FPGA parallel acceleration system, and a local cache; wherein the FPGA parallel acceleration system is connected to the dynamic data flow control system and the memory management and data scheduling system respectively; The FPGA parallel acceleration system includes a plurality of acceleration modules, each of which is used to execute a type of computing task in the dynamic graph convolutional neural network, and different types of acceleration modules execute different types of computing tasks; The dynamic data flow control system is used to call the acceleration module to perform data calculation according to the data processing task of the dynamic graph convolutional neural network, and manage the data transmission and status between the called acceleration modules; The memory management and data scheduling system is used to allocate memory space for the acceleration module, and store input data required for executing the data processing task, intermediate calculation results obtained when executing the data processing task, and output data obtained after executing the data processing task; The local cache is used to store the intermediate calculation results of the acceleration module.

2. The hardware acceleration device for processing dynamic graph convolutional neural network according to claim 1, characterized in that: The data processing task of the dynamic graph convolutional neural network is a data processing task of the entire dynamic graph convolutional neural network, or the data processing task of the dynamic graph convolutional neural network is a data processing task of a partial network in a dynamic graph convolutional neural network.

3. The hardware acceleration device for processing dynamic graph convolutional neural network according to claim 1, characterized in that: The multiple acceleration modules include: a weight-sharing multi-layer perceptron acceleration module, a matrix multiplication acceleration module, a basic operation acceleration module, and a dimensionality reduction operation acceleration module; The shared-weight multilayer perceptron acceleration module is used to perform 1×1 convolution and matrix multiplication operations; The matrix multiplication acceleration module is used to perform matrix multiplication operations; The basic operation acceleration module is used to perform basic operations including addition, subtraction, multiplication and division; The dimension reduction operation acceleration module is used to perform the dimension reduction operation.

4. The hardware acceleration device for processing dynamic graph convolutional neural network according to claim 3, characterized in that: Each acceleration module includes: an input data buffer, an output data buffer and a plurality of parallel computing units, each computing unit includes a plurality of parallel computing cores, and each computing core includes a multiplier and an adder; The input data buffer and the output data buffer are both connected to the memory management and data scheduling system; The input data buffer and the output data buffer in each acceleration module are also connected to the multiple parallel computing units in the acceleration module through a local bus.

5. The hardware acceleration device for processing dynamic graph convolutional neural network according to claim 4, characterized in that: In the shared-weight multilayer perceptron acceleration module, the input data buffer is used to store data that needs to be subjected to 1×1 convolution and matrix multiplication operations, the output data buffer is used to store 1×1 convolution and matrix multiplication operation results, and the multiple parallel computing units are used to perform 1×1 convolution and matrix multiplication operations; In the matrix multiplication acceleration module, the input data buffer is used to store data that needs to be subjected to matrix multiplication operation, the output data buffer is used to store matrix multiplication operation results, and the multiple parallel computing units are used to perform matrix multiplication operation; In the basic operation acceleration module, the input data buffer is used to store operands, the output data buffer is used to store calculation results of basic operations, and the multiple parallel computing units are used to perform basic operation calculations; In the dimension reduction operation acceleration module, the input data buffer is used to store data that needs dimension reduction, the output data buffer is used to store dimension reduction operation results, and the multiple parallel computing units are used to perform dimension reduction operations.

6. The hardware acceleration device for processing dynamic graph convolutional neural network according to claim 4, characterized in that: The memory management and data scheduling system comprises: an external DDR memory and a memory controller, wherein the input data buffer and the output data buffer are both connected to the external DDR memory and the memory controller; The external DDR memory is used to store input data required when executing the data processing task, intermediate calculation results obtained when executing the data processing task, and output data obtained after executing the data processing task; The memory controller is used to allocate the size of the input data buffer and the size of the output data buffer for each called acceleration module, and to control the reading and writing operations of data in the external DDR memory.

7. The hardware acceleration device for processing dynamic graph convolutional neural network according to claim 1, characterized in that: The multiple acceleration modules further include: a TopK acceleration module; the TopK acceleration module is used to determine data in the Top K position from the data to be sorted.

8. The hardware acceleration device for processing dynamic graph convolutional neural network according to claim 1, characterized in that: The dynamic data flow control system comprises: a task scheduler and a data transmission controller, wherein each acceleration module is connected to the task scheduler and the data transmission controller respectively; The task scheduler is used to determine the dependencies and priorities between the computing tasks that need to be executed according to the data processing tasks of the dynamic graph convolutional neural network, and call the acceleration module to execute the computing tasks according to the dependencies and priorities between the computing tasks that need to be executed; The data transmission controller is used to manage the data transmission and status between the called acceleration modules.

9. The hardware acceleration device for processing dynamic graph convolutional neural network according to claim 1 or 7, characterized in that: The multiple acceleration modules also include: a Gather acceleration module and a Concat acceleration module; The Gather acceleration module is used to perform a Gather operation; The Concat acceleration module is used to perform concatenation operations.

10. The hardware acceleration device for processing dynamic graph convolutional neural network according to claim 1, characterized in that: The dynamic data flow control system and the memory management and data scheduling system are also connected to the host. The dynamic data flow control system is also used to receive instructions from the host, and the memory management and data scheduling system is also used to transmit the output data obtained after executing the data processing task back to the host.

Images