Systems and methods for accelerating feature extraction of graphics

Abstract

An apparatus is disclosed. The apparatus can include a multiplication module, an accumulation and aggregation (ACG) module, and a control unit. The multiplication module can perform multiplication based on node data or weight data of a graph. The ACG module can perform accumulation and aggregation operations based on the multiplication module or edge data of the graph. The control unit can manage the multiplication module and the ACG module.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefits of U.S. Provisional Patent Application No. 63 / 134585, filed January 6, 2021; U.S. Provisional Patent Application No. 63 / 248422, filed September 24, 2021; U.S. Provisional Patent Application No. 63 / 251581, filed October 1, 2021; and U.S. Patent Application No. 17 / 530452, filed November 18, 2021. All of these contents are incorporated herein by reference for all purposes. Technical Field

[0003] This invention generally relates to accelerators, and more specifically, to accelerators that support graphics network processing. Background Technology

[0004] Graphs, especially unstructured graphs, can be difficult for machines to process. For example, comparing two graphs to determine if they are similar can be difficult to perform programmatically. Nodes and edges may be labeled differently, presented in different orders, and assigned different weights (but proportionally), which can cause two similar graphs to look different.

[0005] Features still need to be extracted from the graph for later use. Summary of the Invention

[0006] Embodiments of the present invention include an accelerator for extracting features from graphs. Node data can be prefetched and followed by one or more Graph Convolutional Network (GCN) layers. The GCN layers may include multiplication modules and accumulation-aggregation (ACG) modules. Attached Figure Description

[0007] The accompanying drawings described below are examples of how embodiments of the invention are implemented and are not intended to limit the scope of the invention. Various embodiments of the invention may include elements not shown in certain drawings and / or elements shown in certain drawings may be omitted. The drawings are intended to provide illustration and may not be drawn to scale.

[0008] Figure 1 A machine including an accelerator for extracting features from graphics is shown according to an embodiment of the present invention.

[0009] Figure 2 An embodiment of the present invention is shown. Figure 1 Details of the machine.

[0010] Figure 3 A view illustrating the process of extracting features from an example graphic according to an embodiment of the present invention is shown.

[0011] Figure 4AThis illustrates a representation in a computer system according to an embodiment of the present invention. Figure 3 The first way to create a graphic.

[0012] Figure 4B This illustrates a representation in a computer system according to an embodiment of the present invention. Figure 3 The second way to depict a graphic.

[0013] Figure 4C This illustrates a representation in a computer system according to an embodiment of the present invention. Figure 3 The third way to depict a graphic.

[0014] Figure 5 An embodiment of the invention is shown that can be used to extract from Figure 3 Extracting features from images Figure 1 The accelerator.

[0015] Figure 6 An embodiment of the present invention is shown. Figure 5 Details of the multiplication module.

[0016] Figure 7 Various processing elements that can update the same position are shown according to embodiments of the present invention.

[0017] Figure 8 An embodiment of the present invention is shown. Figure 6 Details of the decision-making device.

[0018] Figure 9 An embodiment of the present invention is shown. Figure 5 Details of the Accumulation and Aggregation (ACG) module.

[0019] Figure 10 An embodiment of the invention is shown for use with Figure 1 Example flowchart of the preprocessor's operation.

[0020] Figure 11 An embodiment of the invention is shown for use with Figure 5 Prefetcher and / or Figure 9 Example flowchart of the operation of the trimmer.

[0021] Figure 12 An embodiment of the invention is shown for use with Figure 6 Example flowchart of the operation of the decision-maker.

[0022] Figure 13 An embodiment of the invention is shown for use with Figure 1 Accelerator determination Figure 3 A flowchart illustrating the characteristics of a graphic.

[0023] Figure 14 An embodiment of the invention is shown for use with Figure 1 Accelerator determination Figure 3 A flowchart of an alternative example process for the characteristics of a graphic.

[0024] Figure 15 An embodiment of the invention is shown for use with Figure 1 Preprocessor preprocessing Figure 3 The flowchart is an example of a graphical process.

[0025] Figure 16 An embodiment of the invention is shown for use with Figure 5 The prefetcher is used for Figure 3 A flowchart of an alternative example process for graphical data.

[0026] Figure 17 An embodiment of the present invention is shown. Figure 5 The multiplication module uses Figure 3 The flowchart illustrates an example process of performing multiplication operations on graphical data.

[0027] Figure 18 An embodiment of the present invention is shown. Figure 5 ACG module use Figure 3 The flowchart illustrates an example process of performing accumulation and aggregation operations on graphical data. Detailed Implementation

[0028] Reference will now be made in detail to embodiments of the invention, examples of which are illustrated in the accompanying drawings. Numerous specific details are set forth in the following detailed description in order to provide a thorough understanding of this disclosure. However, it should be understood that those skilled in the art can practice the invention without these specific details. In other instances, well-known methods, processes, components, circuits, and networks have not been described in detail to avoid unnecessarily obscuring aspects of the embodiments.

[0029] It should be understood that although the terms first, second, etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first module may be referred to as a second module without departing from the scope of the invention, and similarly, a second module may be referred to as a first module.

[0030] The terminology used in this description is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in the description of this invention and the appended claims, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” should also include the plural forms. It should also be understood that “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items. It will be further understood that these terms “comprising” and / or “including,” when used in this specification, specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Components and features in the accompanying drawings are not necessarily drawn to scale.

[0031] One method for extracting information from graphs is called Graph Convolutional Networks (GCNs). This method can be used to compare graphs or otherwise utilize them. GCNs involve extracting features from graphs by aggregating elements, performing feature transformations on the aggregated elements, and then performing activations to produce features. This process can be performed two or more times to extract the desired features.

[0032] Embodiments of the present invention can use an accelerator to implement GCN. The multiplication module can perform element-wise multiplication in parallel, and the accumulation and aggregation (ACG) module can perform feature extraction from the data provided by the multiplication module. Multiple layers can be implemented using multiple multiplication modules and ACG models as needed.

[0033] The multiplication module may include a discriminator for selecting the element to be processed and a Single Instruction Multiple Data Processing (SIMD-PE) element for performing the processing. The discriminator may determine whether a particular element is likely to attempt to read a value that has not yet been written (a phenomenon known as a data dependency) and may insert bubbles to delay the operation on that element.

[0034] The ACG module may include a SIMD PE to perform accumulation and aggregation operations based on data provided by the multiplication module. The ACG module may also include a Corrected Linear Unit (ReLU) for performing activation: that is, isolating only positive values ​​(and replacing negative values ​​with zeros). The ACG module may also include a pruner for removing any zero values ​​from the data (which can be returned from the accelerator or provided to the multiplication module in the next layer).

[0035] The accelerator may also include a prefetcher. The prefetcher can retrieve graphics data from memory and buffer the data in the accelerator. This process can reduce execution time by minimizing the number of memory accesses. The prefetcher can also prefetch data in a performance-enhancing manner. For example, when performing matrix multiplication A×B on matrices A and B, values ​​in the columns of matrix A can be multiplied in pairs with rows of matrix B, and then the pairs of products can be summed. For example, the prefetcher can retrieve data in a specific order (retrieve matrix A column-wise and matrix B row-wise).

[0036] Finally, the preprocessor can perform preprocessing on the data. This preprocessing may include removing any zeros (similar to a pruning operation). This preprocessing may also involve reordering the data to help resolve potential data dependencies that could affect the latency of processing elements by attempting to ensure that different operations involving the same elements are performed in different cycles.

[0037] Figure 1 A machine comprising an accelerator for extracting features from graphics is shown according to an embodiment of the present invention. Figure 1 The machine 105, also referred to as the host or system, may include a processor 110, a memory 115, and a storage device 120. The processor 110 can be any type of processor. (For ease of illustration, the processor 110 and other components discussed below are shown externally to the machine; embodiments of the invention may include these components internal to the machine.) Although Figure 1 A single processor 110 is shown, but machine 105 may include any number of processors, each of which may be a single-core or multi-core processor, each of which may implement a Reduced Instruction Set Computer (RISC) architecture or a Complex Instruction Set Computer (CISC) architecture (among other possibilities), and may be mixed in any desired combination.

[0038] Processor 110 may be coupled to memory 115. Memory 115 may be any type of memory, such as flash memory, dynamic random access memory (DRAM), static random access memory (SRAM), persistent random access memory, ferroelectric random access memory (FRAM), or non-volatile random access memory (NVRAM), such as magnetoresistive random access memory (MRAM). Memory 115 may also be any desired combination of different memory types and may be managed by memory controller 125. Memory 115 may be used to store data that can be referred to as "short-term": that is, data that is not expected to be stored for a long time. Examples of short-term data may include temporary files, data used locally by the application (which may have been copied from other storage locations), etc.

[0039] Processor 110 and memory 115 can also support various applications running on them. These applications can issue requests (also called commands) to read data from or write data to memory 115. When storage device 120 is used to support applications that read or write data through a certain file system, device driver 130 can be used to access storage device 120. Although Figure 1 A storage device 120 is shown, but any number (one or more) of storage devices may exist in machine 105.

[0040] although Figure 1 Using the generic term "storage device," embodiments of the present invention may include any storage device format that can benefit from the use of computing storage units, examples of which may include hard disk drives and solid-state drives (SSDs). The reference to SD below should be understood to include such other embodiments of the invention.

[0041] Machine 105 may also include accelerometer 135 (which may also be referred to as a device). As described below, accelerometer 135 may support graphics (which may be stored in memory 115 or storage device 120, or stored in another machine accessed via a network) Figure 1 (Not shown in the image) Features are extracted. Accelerator 135 is shown to communicate with memory 115, but if the graphics are stored somewhere other than memory 115, accelerator 135 may communicate with other storage locations (such as storage device 120).

[0042] Accelerator 135 can be implemented using any desired hardware. For example, accelerator 135 can be implemented using a field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), central processing unit (CPU), graphics processing unit (GPU), general-purpose GPU (GPGPU), data processing unit (DPU), or tensor processing unit (TPU), to name a few possibilities. Accelerator 135 can also be implemented using combinations of these components. Finally, accelerator 135 can be implemented as a computational storage unit that can be used to support operations on storage device 120 (which could be advantageous if graphics are stored on storage device 120 instead of memory 115).

[0043] Machine 105 may also include a preprocessor 140, which may be software executed, for example, by a component within processor 110 or accelerator 135. Preprocessor 140 may remove zeros from the data and reorder the data to remove data dependencies. More generally, preprocessor 140 may modify the data in any desired manner: for example, reordering the data, changing values ​​in the data, adding values ​​to the data, and / or removing values ​​from the data. (See below for reference.) Figure 5 Further discussion on preprocessor 140.

[0044] Figure 2 An embodiment of the present invention is shown. Figure 1 Details of machine 105. Figure 2 Typically, machine 105 includes one or more processors 110, which may include a memory controller 120 and a clock 205 for coordinating the operation of machine components. Processor 110 may also be coupled to memory 115, which may include random access memory (RAM), read-only memory (ROM), or other state-keeping media. Processor 110 may also be coupled to storage device 125 and network connector 210, which may be, for example, an Ethernet connector or a wireless connector. Processor 110 may also be connected to bus 215, where user interface 220 and input / output (I / O) interface ports, which can be managed using I / O engine 225, may be attached.

[0045] Figure 3 A view illustrating the process of extracting features from an example drawing according to an embodiment of the present invention is shown. Figure 3 Figure 305 is shown in the image. A graph similar to Figure 305 can be considered as a collection of nodes and edges. Each node can be identified in some way to distinguish two nodes in the graph. For example, Figure 305 is shown as including nodes identified using the numbers 1 to 5. Although... Figure 3 As not shown, nodes can have weights associated with them (to indicate the cost of the node itself): the nodes in Figure 305 are unweighted (they can be modeled as having zero weights). In general, weights can be identified as weighted data.

[0046] Edges can connect nodes in a graph. For example, graph 305 has edges connecting nodes 1 and 2, nodes 2 and 4, nodes 2 and 5, and nodes 3 and 4. Although not in Figure 3 As shown in the figure, edges can also have weights associated with them (to indicate the cost of traversing nodes): the edges in Figure 305 are unweighted (they can be modeled as having zero weights).

[0047] In graph 305, edges are undirected. This means edges can be traversed in either direction: for example, it doesn't matter whether the edge connecting nodes 1 and 2 travels from node 1 to node 2 or from node 2 to node 1. However, in some graphs, edges may be directed, meaning they can only be traversed in one direction, from the source to the sink (or destination). Directed edges can be likened to one-way streets, while undirected edges can be likened to two-way streets: one-way streets can only travel in one direction; you cannot travel backwards along a directed edge from the sink to the source.

[0048] In a city, not every pair of street intersections is connected by a street. Similarly, not every pair of nodes in a graph has an edge connecting them. For example, in graph 305, there is no edge connecting nodes 1 and 3. A path can be understood as a sequence of edges (directed or undirected, depending on the type of edges in the graph) that connects pairs of nodes in the sequence, enabling traversal from the source node to the sink node. For example, in graph 305, a path from node 1 to node 3 exists by traversing the edges between nodes 1 and 2, nodes 2 and 4, and nodes 3 and 4, even though there is no edge directly connecting nodes A and D.

[0049] An undirected graph is called a connected graph if every pair of nodes includes a path connecting them; otherwise, it is called a disconnected graph. Figure 305 is an undirected graph, and it can be seen that there exists a path connecting any two nodes in Figure 305: therefore, Figure 305 is a connected graph. For directed graphs, the terminology is slightly different: a directed graph is called weakly connected if there exists an undirected path connecting any two nodes (i.e., a sequence of edges, ignoring edge direction); it is called strongly connected if there exists a directed path connecting any two nodes (i.e., a sequence of edges considering edge direction); and it is called disconnected if there exists a pair of nodes that have no directed or undirected path connecting them. Note that any strongly connected directed graph is also weakly connected; the converse is not necessarily true. Also note that for a directed graph, the existence of a directed path from node A to node B does not necessarily mean that there is a directed path from node B to node A; this must be checked separately to determine if the directed graph is strongly connected.

[0050] In computer systems, graphs can be represented as lists of node identifiers (which, if nodes are weighted, can be paired with weights) and lists of edges that pair two nodes connected by an edge (which, if edges are weighted, can also be paired with weights). These lists can be represented using any desired data structure: for example, linked lists, arrays, tables, etc., are various data structures that can be used to store information representing graphs. For directed graphs, the order of nodes can indicate direction: for example, an edge represented using a pair (A, B) can be understood as an edge where node A is the source node and node B is the sink node. For undirected graphs, a single pair (A, B) can be used to represent an edge, or two pairs (A, B) and (B, A) can be used so that the order of nodes does not affect the search for edges. Note that edges can also be represented using a two-dimensional table, where 1 (or weight) indicates that an edge exists between associated nodes (identified by the row and column values), and 0 (or some other acceptable value, such as infinite (or very high) cost) can be used to indicate that an edge is missing between two associated nodes. When using such a table, for an undirected graph, the value indicating the existence of an edge can be stored twice to represent two pairs (A, B) and (B, A): in other words, the table can be symmetric about one of its diagonals. On the other hand, such a table representing edges in a directed graph may not be symmetric unless the graph includes every edge in every direction.

[0051] Figure 4A This illustrates a representation in a computer system according to an embodiment of the present invention. Figure 3 The first method of graphic 305. In Figure 4A middle, Figure 3 Graph 305 is shown as including a list 405 of nodes and a list 410 of edge pairs. Note that in list 410, Figure 3 Each edge of graph 305 is represented twice, once for each "direction" (because...). Figure 3 The graph 305 is undirected. This can be simplified by... Figure 3 The path to find the 305 error in the image: If Figure 3 If the search for graph 305 is currently at node 2 and wants to reach node 1, then list 410 can be searched to find the edge between nodes 2 and 1 without having to reverse their order. However, if you want to reduce storage space to represent the graph, you can use... Figure 4B List 415 (but in this case, Figure 3 A graph 305 search may involve searching for edges that include the current node as the source or convergence point of each edge.

[0052] Figure 4C This illustrates a representation in a computer system according to an embodiment of the present invention. Figure 3 The third way of graphic 305. In Figure 4C In the list of nodes 405 and Figure 4A and4B The same as in [the previous section]. However, Table 420 can be used instead of storing the edges as [the current section]. Figure 4A List 410 or Figure 4B List 415. Each row and column in Table 420 includes an identifier for each node in List 405. For each pair of nodes, one node identifier can be used to select a row in Table 420, and the other node identifier can be used to select a column in Table 420. At the intersection of the row and column, a value of 1 indicates that the nodes are connected by an edge; a value of zero indicates that the nodes are not connected. Note that, as mentioned above, Table 420 is symmetrical about the main diagonal of Table 420 (from row 1, column 5 to row 5, column 5): whatever value is found in row i, column j, can also be found in column j, row i.

[0053] Back Figure 3 Because nodes can be assigned different identifiers in different graphs, nodes can be identified in different orders in different graphs, edges can be presented in different orders in different graphs, edges can be assigned different weights in different graphs, and so on, it can be difficult to compare two graphs to determine whether they are similar. For example, two graphs with the same set of nodes and edges, but where the edges in one graph are assigned twice the weights of the edges in the other graph, may look dissimilar. However, multiplying the edge weights by a constant value does not change the minimum cost path between the two nodes: the minimum cost path P in one graph is the same as the minimum cost path in the other graph. Similarly, changing the node identifiers may make the graphs look different, but the minimum cost path remains unchanged.

[0054] Using Graph Convolutional Networks (GCNs), features of a graph can be extracted. These features can then be compared with features of other graphs to determine their similarity, even if they may use different identifiers, different information orders, different weights, etc. GCNs are also used in deep learning techniques, machine learning, artificial intelligence, and more. GCNs can be used to extract node embeddings from graphs, where each node embedding can contain information about its role in the graph. A GCN can consist of multiple layers in which node embeddings are propagated until rich information about the input graph is derived. In each layer, node embeddings are updated by collecting the embeddings of their neighbors (aggregation) and passing their weighted sum through filters (feature transformation). To introduce non-linearity into the model, activation functions of the form of Modified Linear Units (ReLU) can be used at the ends of each layer.

[0055] Figure 3The process is illustrated. Node embeddings 310-1 to 310-5 (collectively referred to as node embeddings 310) can be created for each node in graph 305. Node embeddings 310 can be considered as vectors of predetermined length. Node embeddings 310 can initially be assigned random values ​​as vector coordinates, allowing GCN to establish their final values, or more specific values ​​can be assigned to node embeddings 310. Although... Figure 3 The diagram shows that each node embedding includes four coordinates / values, but embodiments of the invention may include any number (one or more) of coordinates / values ​​in the node embedding.

[0056] In aggregation, node embeddings are updated based on the node embeddings of neighboring nodes (i.e., nodes connected to a given node via edges). Thus, for example, in graph 305, node 1 is a neighbor of node 2, node 2 is a neighbor of nodes 1, 4, and 5, node 3 is a neighbor of node 4, node 4 is a neighbor of nodes 2 and 3, and node 5 is a neighbor of node 2. Note that a node is also a neighbor of itself (even if there may be no edges representing this relationship in graph 305). Therefore, for example, node embedding 315-1 of node 1 is updated based on node embeddings 310-1 and 310-2.

[0057] In the feature transformation, the weighted sum of the node embeddings can be passed through a filter. This feature transformation can take into account the values ​​of all coordinates within the node embedding. Therefore, for example, node embedding 320-1 can be the result of a feature transformation of node embedding 315-1.

[0058] Finally, the node embeddings can be activated. Activation may involve the Corrected Linear Unit (ReLU), which could introduce nonlinearity into the model. Furthermore, activation may involve eliminating any non-positive values ​​in the node embeddings by replacing them with zero. Therefore, node embedding 325-1 could be the result of applying ReLU to node embedding 320-1 and eliminating any non-positive values ​​from node embedding 320-1.

[0059] Although Figure 3 The illustration shows the aggregation, feature transformation, and activation processes occurring only once, but embodiments of the invention can support any number of iterations (one or more) of this sequence: each such sequence can be referred to as a layer. Thus, for example, node embedding 325-1 can be used as input to node 1 in the activation operation of the second layer, and similarly, the activation results of other node embeddings can be used as input to the activation operation of the second layer. Note that if multiple layers are executed, embodiments of the invention can enable variations in parameters within each layer. For example, different weights can be used in different layers, and different aggregation, feature transformation, and / or activation operations can be used in different layers, etc.

[0060] Figure 5 An embodiment of the invention is shown that can be used from Figure 3305 feature extraction of graphics Figure 1 Accelerator 135. In Figure 5 In the diagram, accelerator 135 is shown as including control unit 505, prefetcher 510, and GCN layers 515-1 and 515-2 (which may be collectively referred to as GCN layer 515 or simply layer 515). Control unit 505 can signal prefetcher 510 and GCN layer 515 to perform certain operations. To distinguish between data flow and control flow, data flow is shown with solid arrows, and control flow is shown with dashed arrows.

[0061] As described above, the prefetcher 510 can retrieve data from memory 115 (or...) Figure 1 The data for graphics can be retrieved from storage device 120, or from some other component (and possibly even from some storage device within accelerator 135) if the data for graphics is stored in some other component. Prefetcher 510 can store data in one or more buffers within accelerator 135. Figure 5 (Not shown) This includes on-chip buffers or caches. In this way, by internally transferring data between components to accelerator 135, load or store requests to memory 115 can be minimized, potentially resulting in faster overall operation. Prefetcher 510 can load data from memory 115 in the most efficient manner for its operation. As described above, since matrix multiplication may involve multiplying the columns of one matrix by the rows of another, prefetcher 510 can load graph data from memory 115 in a manner that reads some data in column order (e.g., node embeddings) and other data in row order (e.g., weights) to accelerate matrix multiplication.

[0062] The prefetcher 510 can load data into a buffer that may include one or more first-in-first-out (FIFO) queues. Figure 5 (Not shown in the image). The prefetcher 510 can store data values ​​in FIFO queues in a round-robin fashion, where the first value is placed in the first FIFO queue, the second value is placed in the second FIFO queue, and so on. The FIFO queues will refer to the following... Figure 6 Further discussion is needed.

[0063] In some embodiments of the invention, the prefetcher 510 may load data into a buffer large enough to store all the data for the graph. In such embodiments, the graph may be relatively small. If the graph is large enough, the buffer in the accelerator 135 may not be large enough to store all the data for the graph. In such embodiments of the invention, the prefetcher 510 may prefetch as much data as possible for the graph (while optimizing the operation as much as possible), and may leave other data in the memory 115 for later retrieval: for example, when some data from the buffer in the accelerator 135 has been processed and is no longer needed. If the buffer in the accelerator 135 is not large enough to store all the data for the graph, the accelerator 135 may write some data back to the memory 115 to free up space in the buffer for other data, which may need to be read back from the memory 115 later.

[0064] GCN layer 515 may include two modules: multiplication modules 520-1 and 520-2 (collectively referred to as multiplication module 520) and accumulation and aggregation (ACG) modules 525-1 and 525-2 (collectively referred to as ACG module 525). Multiplication module 520 performs matrix multiplication, and ACG module 525 performs accumulation and aggregation operations. Together, multiplication module 520 and ACG module 525 can implement feature transformation. ACG module 525 can also perform activation, which may include nonlinear functions applied to node embedding.

[0065] Multiplication module 520 and ACG module 525 can begin computation once they have obtained sufficient starting data from their sources. For example, multiplication module 520-1 does not need to wait for prefetcher 510 to prefetch all the data of the graphics from memory 115: if enough data has been prefetched for multiplication module 520-1 to begin, multiplication module 520-1 can begin operation at any time. Similarly, ACG module 525-1 does not need to wait for multiplication module 520-1 to complete all its operations: all that is required is that multiplication module 525-1 has sufficient data to begin its operation. Likewise, as long as sufficient data is available, GCN layer 515-2 does not need to wait for ACG module 525-1 to complete its operation before it can begin. See below for reference. Figure 6-9 Let's discuss this concept further.

[0066] Figure 5 An accelerator 135 comprising two GCN layers 515 is shown. However, embodiments of the invention may include any number (one or more) of GCN layers 515, with three to four GCN layers being typical and eight GCN layers being a reasonable (but not limited to) upper limit. Once the last GCN layer 515 has completed its processing, the last GCN layer 515 may write its output back to memory 115 (or to a different memory location as needed) for later use in other processing. Furthermore, although Figure 5 It is suggested that each layer can be separated from other layers (and include its own multiplication module 520 and / or ACG module 525), but embodiments of the invention may include feeding back data output from a GCN layer 515 to itself for another iteration. Embodiments of the invention can reduce the hardware used in the accelerator 135 by reusing the implementation of the GCN layer 515.

[0067] In addition to memory 115, host 105 may also include a preprocessor 140. Preprocessor 140 can read data from the graph from memory 115 and can preprocess that data. This preprocessing may include removing zeros (or other values) from the data (to avoid unnecessary calculations), adding values ​​to the data, changing values ​​in the data, and / or reordering the data so that multiple data operations that may update the same value can be performed in sufficiently long intervals to avoid data dependencies, which can slow down the overall operation. Determining how far the data can be moved may be a function of the number of cycles required for the processing to complete its operation on a particular piece of data. For example, if it takes five cycles for processing an element to complete its operation (this number is arbitrarily chosen for the purposes of this example and can be replaced with other numbers as needed / appropriately), preprocessor 140 may reorder the data that may update the same value so that the data are at least five cycles apart in terms of processing. Preprocessor 140 may store the preprocessed data back to memory 115 or to another storage location: for example, Figure 1 The storage device 120, or the storage device within the accelerator 135 ( Figure 5 (Not shown in the image).

[0068] Although Figure 5 The preprocessor 140 and memory 115 are shown located within machine 105, but embodiments of the invention may position the preprocessor 140 and / or memory 115 in other locations. For example, the preprocessor 140 and / or memory 115 may be implemented within accelerator 135. That is, accelerator 135 may include components (such as a dedicated FPGA, ASIC, or other equivalent components) capable of implementing the operation of preprocessor 115. Alternatively, some other component of accelerator 135, such as a trimmer in ACG module 525, may be used to perform the operation of preprocessor 140. Alternatively, accelerator 135 may include a processor (e.g., as part of control unit 505) that can execute software to implement the operation of preprocessor 140. Figure 5The preprocessor 140 and memory 115 shown are part of machine 105 and are merely exemplary arrangements of the preprocessor 140 and memory 115. Furthermore, the components of accelerator 135 (control unit 505, prefetcher 510, multiplication module 520, and ACG module 525) can be implemented individually as software, hardware, or a combination of both. For example, control unit 505 can be implemented as an FPGA, prefetcher 510 can be implemented as an ASIC, and multiplication module 520 and ACG module 525 can be implemented using software executed by a processor.

[0069] Figure 6 An embodiment of the present invention is shown. Figure 5 Details of the multiplication module 520. Figure 6 In this module, the multiplication module 520 may include an optimizer 605, a weight buffer 610, and a two-dimensional (2D) single instruction multiple data processing element (SIMD-PE) 615.

[0070] The decision-maker 605 can select the value to be processed by the 2D SIMD PE 615 from FIFO queues 620-1 to 620-2 (collectively referred to as FIFO queue 620). The value can be selected from... Figure 5 The prefetcher 510 (if the multiplier module 520 is located in) Figure 1 Accelerator 135 Figure 5 In the first GCN layer 515) or by as Figure 5 The trimmer, a part of the ACG module 525, is placed in the FIFO queue 620. (See below for reference.) Figure 9 Further discussion on trimmers.

[0071] The reason for including FIFO queue 620 may not be obvious. After all, if Figure 1 Since the preprocessor 140 has correctly preprocessed the graph data, the graph data should not contain any zeros and should be reordered so that there are no data dependencies. Although in Figure 1 The fact that the preprocessor 140 has already preprocessed the data may be true, but it may not be true in subsequent GCN layers. That is, in... Figure 5 In GCN layer 515-2, multiplication module 520-2 may access data including zeros, or currently has data dependencies. See below for reference. Figure 9 The subject of discussion, as Figure 5 The trimmer, part of the ACG module 525, can be used for... Figure 5Zeros are eliminated from the graph data used in subsequent GCN layers 515, but data dependencies may still exist. By using FIFO queue 620, if the resolver 605 determines that a data dependency exists, the resolver 605 can leave the data in its FIFO queue for one or more cycles to attempt and resolve the data dependency. See below for reference. Figure 7 Further discussion on data dependencies.

[0072] The number of FIFO queues 620 can vary depending on the implementation. In some embodiments of the invention, the number of FIFO queues 620 can be equal to or greater than the number of PEs in the 2D SIMD PE 615.

[0073] The decision maker 605 can read data from the FIFO queue 620. If the data is non-zero (if...), it will... Figure 5 Preprocessor 140 and Figure 9 If the trimmer of the ACG module 525 has operated correctly (meaning the data should be non-zero), then the decision-maker 605 can determine whether a particular value will update data that has already been updated by another processing element. If so, the decision-maker 605 can insert a bubble (i.e., indicate the processing element that should have processed the value) to perform no operation (no operation). See below for reference. Figure 8 Further discussion on the operation of the decision-maker 605.

[0074] The weight buffer 610 can store information about the weights 625 (which may also be referred to as weight data) used in the feature transformation. The weights 625 can be... Figure 5 The prefetcher 510 prefetches so that it can be obtained from Figure 1 Access them from within the accelerator 135 (instead of from) Figure 1 (Read from memory 115). Although Figure 6 It is suggested that the weights 625 may simply be a vector (a one-dimensional set of weights), but embodiments of the present invention may include the weights 625 as two-dimensional data (the data is provided to the multiplication module 615 in some order).

[0075] 2D SIMD PE 615 can be a two-dimensional array of processing elements. As the name suggests, the same instructions can be applied to each processing element in 2D SIMD PE 615, but to different data. As shown in the enlarged figure, 2D SIMD PE 615 can include processing elements 630-1, 630-2, 630-3, and 630-4 (collectively referred to as processing element 630). Processing elements 630-1 to 630-2 can form rows of processing elements in 2D SIMD PE 615, with other rows formed by other processing elements (upwards up to the rows including processing elements 630-3 to 630-4). Furthermore, processing elements 630-1 to 630-3 can form columns of processing elements in 2D SIMD PE 615, with other columns formed by other processing elements (upwards up to the columns including processing elements 630-2 to 630-4).

[0076] Each processing element 630 in a 2D SIMD PE 615 can, for example, perform a multiplication as part of a global matrix multiplication. A 2D SIMD PE 615 is functionally equivalent to a set of one-dimensional row-oriented SIMD PEs (which may be formed by processing elements 630-1 to 630-2 or processing elements 630-3 to 630-4), a set of one-dimensional column-oriented SIMD PEs (which may be formed by processing elements 630-1 to 630-3 or processing elements 630-2 to 630-4), or even simply a set of SIMD PEs without any "dimensional organization." The term 2D SIMD PE should be understood to include such sets.

[0077] Once the adjudicator has assigned an operation to a processing element in 2D SIMD PE 615, 2D SIMD PE 615 can begin its operation, even if there may be other data waiting to be loaded and processed (i.e., values ​​are still in FIFO queue 620). This may happen even if not every processing element in 2D SIMD PE 615 has had a value loaded into it for processing. For example, processing element 630-1 may be able to perform its computation even if processing elements 630-2, 630-3, and 630-4 do not yet have their data. Ultimately, however, 2D SIMD PE 615 should output a value for each processing element (at least within each interval in which 2D SIMD PE 615 processes values: each GCN layer may involve multiple such intervals to complete all computations).

[0078] In some embodiments of the invention, data from the FIFO queue 620 and / or weight 625 can be provided to two or more processing elements 630 simultaneously. In such embodiments of the invention, processing elements 630 can begin their computations simultaneously. However, in some embodiments of the invention, one processing element completes providing data to processing element 630 at a time, and each processing element 630 receives its data at different intervals. In such embodiments of the present disclosure, different processing elements 630 can begin computations at different times.

[0079] Clarifying some terminology might be helpful in this regard. The term "cycle" refers to the period used to achieve... Figure 1 The accelerator 135 represents the time required for the hardware to execute a single instruction (similar to a CPU cycle). Note that while cycles may vary depending on the underlying hardware, a cycle is a unit of time. For example, a processor running at 200 MHz executes 200,000,000 cycles per second, meaning each cycle is 0.000000005 seconds (which can also be expressed as 5 x 10^15 seconds). - Seconds or 5 nanoseconds (ns). The term "interval" refers to the time required for a processor to complete a specific sequence of operations, each of which may require one or more cycles to complete. For example, the time required for processing element 630 to perform calculations on the input data may be five cycles: this number of cycles can be understood as the interval. (As mentioned above, the number 5 used is merely exemplary, and the number of cycles used by processing element 630 to perform calculations may be greater than or less than 5.) The term "iteration" refers to... Figure 5 Operations of GCN layer 515. Figure 5 A specific GCN layer 515 can perform its operation in a single iteration. Note that in some embodiments of the invention, it can be used multiple times. Figure 5 A single GCN layer 515 (e.g., if the operation is the same in two GCN layers, but for the data performing the operation): can be Figure 5 Each such use of the GCN layer 515 is considered an iteration. Therefore, an iteration can involve one or more intervals, and an interval can involve one or more cycles.

[0080] As mentioned above, processing can begin immediately after sufficient data is loaded, and data dependencies can exist. Figure 7 It shows how these situations occur.

[0081] Consider the case where outer product matrix multiplication is performed on matrices 705 and 710 to produce matrix 715. To determine the output value 720, values ​​725 and 730 are multiplied, then values ​​735 and 740 are multiplied, then values ​​745 and 750 are multiplied, and so on. Once all these individual multiplications have been performed, the results can be added together, resulting in the output value 720.

[0082] This processing is typically not considered when performing mathematical operations manually. However, when a computer performs calculations, many different calculations may be updating a specific location. More specifically, this involves updating the output value 620. Figure 6 The processing element 630 can be operated by multiplying each pair of input values ​​(such as values ​​725 and 730, values ​​735 and 740, and values ​​745 and 750) respectively, and then using the product to increase the output value 720. However, increasing the output value 720 may involve reading the output value 720 into... Figure 6 In element 630, addition is performed, and then the updated value is written back to the output value 720. If Figure 6 If two (or more) processing elements 630 attempt the update simultaneously, the result may be incomplete. For example, two processing elements may simultaneously read the output value 720 and independently write its value back. Therefore, whichever write operation completes the second time may miss the update from the first write operation, resulting in an incorrect value. Thus, a data dependency may exist between the two computations.

[0083] A solution to this data dependency could be to schedule the computation of updating the output value 720 at different intervals. That is, one interval could update the output value 720 based on the product of values ​​725 and 730, the next interval could be based on the product of values ​​735 and 740, the next interval could be based on the product of values ​​745 and 750, and so on. In this way, the data dependency can be resolved.

[0084] Using this information, one can understand Figure 1 The preprocessor 140 performs operations related to data reordering in graphics. Assume... Figure 6 The 2D SIMD PE 615 comprises n processing elements 630. If the same value can be updated using two different values, then these two values ​​have a data dependency. If two different values ​​are within each other in data order (n values), then the two values ​​may end up being processed within the same time interval. Since this situation can lead to computational errors (because one update to the same value occurs within the same time interval, another update may be missed), it is helpful that the different values ​​used to update the same value occur in different time intervals. The intervals between these values ​​can be large enough that the two processing elements can avoid updating the same value simultaneously: for example, this can be achieved by spacing these values ​​at least n x l values, where l is the number of cycles (i.e., the length of the interval, or the delay of the processing element) required to process the value.

[0085] Combinations of values ​​725, 735, and 745 with values ​​755, 760, and 765 can also be used to update output value 770; combinations of values ​​775, 780, and 785 with values ​​730, 740, and 750 can be used to update output value 790. These operations do not affect the computation of updating output value 720. Therefore, while updates to output value 720 can be computed at non-overlapping intervals to avoid data dependencies, there is no problem in performing operations in parallel on, for example, values ​​725 and 730, values ​​725 and 755, and values ​​775 and 730, because these operations update output values ​​720, 770, and 790 respectively (similar parallel operations can be used for other operations that do not update the same output value). In other words, one interval might involve operations on values ​​725 and 730, 725 and 755, and 775 and 730 to update values ​​720, 770, and 790 respectively; a later interval might involve operations on values ​​735 and 740, 735 and 760, and 780 and 740 to update values ​​720, 770, and 790 respectively; another later interval might involve operations on values ​​745 and 750, 745 and 765, and 785 and 750 to update values ​​720, 770, and 790 respectively, and so on. Note that these intervals may not overlap: for example, if each interval comprises five periods, the intervals where values ​​735 and 740, 735 and 760, and 780 and 740 update values ​​720, 770, and 790 respectively may begin five periods after the intervals where values ​​725 and 730, 725 and 755, and 775 and 730 update values ​​720, 770, and 790 respectively, and the intervals where values ​​745 and 750, 745 and 765, and 785 and 750 update values ​​720, 770, and 790 respectively. For example, data from FIFO queue 620-1 may be stored as value 725 in one processing element, data from FIFO queue 620-2 may be stored as value 750 in another processing element, and so on, to take advantage of available parallel computing.

[0086] Figure 8 An embodiment of the present invention is shown. Figure 6 Details of the 605 decision-maker. Figure 8 In this context, the optimizer 605 may include a next-round identifier 805, a previous-round identifier 810, a current-round identifier 815, and a dependency window 820. The next-round identifier 805 can identify... Figure 6 The next FIFO queue 620 from which the arbiter 605 can read data for processing.

[0087] It might be unclear why the adjudicator 605 would keep track of the next FIFO queue 620 from which data is read. After all, the adjudicator 605 could simply read data from each FIFO queue 620 sequentially until all data has been read. Alternatively, the adjudicator 605 could read data until... Figure 6 All processing elements 630 have data, then wait for the interval to complete, and then read more data to fill the gap. Figure 6 Similarly, for element 630, in this case, all decision-makers 605 will need to know... Figure 5 How many multiplication modules are there in 520? Figure 6 The processing element is 630. However, these conclusions contain some hidden assumptions, which may not be correct. In the first case, if the data volume is greater than 630, it will be suitable for... Figure 6 If the processing element 630 is not available, the sorting unit 605 may have to wait until it can read the data; or if the sorting unit 605 can process the data, the sorting unit 605 may have to wait until it can read the data. Figure 5 The prefetcher 510 reads data from the FIFO queue 620 faster (for...). Figure 5 The first GCN layer (515) or previous GCN layers Figure 5 ACG module 525 (for Figure 5 The subsequent GCN layer 515 can provide data. In this case, the sorter 605 may have to remember which FIFO queue 620 should be read next: the next round identifier 805 can provide the sorter 605 with the next time to read from the FIFO queue. Figure 6 The FIFO queue 620 reads data information. In the second case, for Figure 5 The assumption about the first GCN layer 515 might be reasonable; however, in Figure 5 After the first GCN layer 515, there may be new zeros in the graph data that can be skipped. If these zeros should be identified by the divider 605, then these zeros can be skipped, and the divider 605 can actually skip them. Figure 6 FIFO queue 620 access ratio Figure 6 The processing element 630 has more values. Therefore, it is assumed that in each cycle (or interval, depending on how many values ​​can be provided to the processing element simultaneously), the judge 605 can only read values ​​that exist. Figure 6 It is unreasonable to process elements with the same number of values ​​as 630.

[0088] Furthermore, the above discussion will Figure 6 Each processing element 630 is considered "locked" during its operation, and no other operation can be performed until its processing is completed. However, in some embodiments of the invention, Figure 6 The processing element 630 can be operated in a pipeline manner. That is to say, Figure 6The processing element 630 can be considered to include two or more stages, each stage producing partial or intermediate results, which are then used by the next stage of the pipeline. In such embodiments of the invention, when Figure 6 While other stages of processing element 630 are performing calculations, let Figure 6 It might be wasteful for a stage of processing element 630 to do nothing. For example, updating... Figure 7 The output value 720 may involve... Figure 7 Multiply the values ​​725 and 730, then read... Figure 7 The current value of the output value 720 is then used to determine the output value. Figure 7 The output value of 720 is increased by the current value. Figure 7 The product of the values ​​725 and 730: This sum can then be written as the result. Figure 7 The output value is 720. This sequence can be considered as three stages: Figure 7 Multiply the values ​​725 and 730; read Figure 7 The output value is the current value of 720, and it will be... Figure 7 The current value of the output value 720 is added to the product calculated in stage 1 (other methods may be used to divide these operations into stages: embodiments of the invention are intended to include all such variations). However, in Figure 7 After multiplying the values ​​725 and 730, Figure 6 The first stage of processing element 630 can be assigned to perform another multiplication of other values. Therefore, Figure 6 The processing element 630 can receive data in each cycle, even if Figure 6 The interval required for the processing element 630 to complete all its operations can exceed one cycle.

[0089] Data dependencies may exist if multiple operations that might involve updating a specific output value are processed simultaneously. For example, consider... Figure 7 The values ​​725 and 730 are input in one cycle. Figure 6 The specific processing element 630, and Figure 7 The 735 and 740 were input in the next cycle. Figure 6 The same case of processing element 630. In the first stage, Figure 7 While multiplying the values ​​735 and 740, the second stage can read... Figure 7 The output value is 720, so that... Figure 7 The output value of 720 increased Figure 7 The product of the values ​​725 and 730. However, in the next cycle, Figure 6 The second stage of the specific processing element 630 can attempt to read Figure 7 The output value is 720, while Figure 6The third stage of the specific processing element 630 can Figure 7 The output value of 720 increased Figure 7 The product of the values ​​725 and 730. Based on the "first" operation (although operations are intended to occur in parallel, they may occur in one order or another, which can be unpredictable), in Figure 6 The specific processing element 630 is read in the second stage. Figure 7 The output value of 720, representing the "current" value, may or may not be reflected in the output. Figure 6 The third phase of the specific processing element 630 is executed. Figure 7 The product of the values ​​725 and 730 increases. In other words, if Figure 6 The second phase of the specific processing element 630 accesses at an "incorrect" time. Figure 7 If the output value is 720, then... Figure 7 The product of the values ​​725 and 730 Figure 7 The increase of the output value of 720 may be lost. In this case, the value can be adjusted accordingly. Figure 7 The various updates to the output value of 720 are scheduled far enough apart (based on...). Figure 6 The processing of element 630 is delayed to avoid this data dependency. For example, Figure 6 Specific processing element 630 can be scheduled to update in one cycle. Figure 7 The output value is 720, and then it is updated in the next cycle. Figure 7 The output value is 790, and then it is updated in the third cycle. Figure 7 The output value is 770. At this time, Figure 6 Any data in the specific processing element 630 will not be updated. Figure 7 The output value is 720, therefore there will be no scheduling pair. Figure 7 The output value of 720 is another updated data dependency.

[0090] The previous period identifier 810 can be used to determine the last period for updating a specific feature. Therefore, the previous period identifier 810 can be a vector, rather than a single value, for storing... Figure 6 Information for each processed element 630. The current cycle identifier 815 can be used to determine... Figure 5 The current cycle of the multiplication module 520. Dependency window 820 can use... Figure 6 The number of cycles required to complete the calculation for processing element 630 (i.e., Figure 6The delay of processing element 630 is used to determine this. Comparing the previous period identifier 810 with the current period identifier 815 and the dependency window 820 determines if a data dependency exists: if the difference between the previous period identifier 810 and the current period identifier 815 for a particular feature is less than or equal to the dependency window 820, then updating that feature may result in a data dependency. The optimizer 605 can then insert a no-op into the processing element instead of removing the value from... Figure 6 FIFO queue 620 moved to Figure 6 In the processing element 630, problematic values ​​are tracked and awaiting processing. However, if the difference between the previous cycle identifier 810 and the current cycle identifier 815 is greater than the dependency window 820, the arbiter 605 can select the problematic value from the current cycle identifier 815. Figure 6 The value of the FIFO queue 620 is loaded into the processing element 630, and the previous cycle identifier 810 of that feature can be updated to be equal to the current cycle identifier 815. Note that if the arbiter 605 inserts a bubble... Figure 6 In the processing of element 630, this fact does not mean that from Figure 6 Data is removed from FIFO queue 620: Data can be retained in Figure 6 The data may be stored in the FIFO queue 620 or in a buffer within the resolver 605 until the data dependency is resolved.

[0091] The current period identifier 815 can be based on the period past time. Figure 2 The clock 205 is updated. This can be done from the resolver 605. Figure 6 When reading data from the FIFO queue 620, the identifier 810 of the previous cycle is updated.

[0092] While it may appear that the adjudicator 605 can operate on one value at a time, this assumption is incorrect. In fact, the adjudicator 605 can access values ​​from... Figure 6 A collection of values ​​in a FIFO queue 620, and these values ​​can be stored in parallel. Figure 6 In the processing element 630 (or insert the bubble) Figure 6 (In the processing element 630). The arbiter 605 can actually be derived from... Figure 6 The FIFO queue 620 reads enough values ​​to fill it. Figure 6 The 2D SIMD PE 615. Note that the value of this quantity can be less than... Figure 6 The number of elements 630 processed. For example, as described above, the judge 605 can track the number of elements not loaded into... Figure 6 This involves processing certain values ​​in element 630. (The last part, "already from," appears to be incomplete and lacks context. It's unclear what it refers to.) Figure 6 The elements read by FIFO queue 620 are waiting to be processed in the next iteration. Therefore, if Figure 6 The 2D SIMD PE 615 has Figure 3 If there are n processing elements 630, and w elements waiting to be processed (as described above), then the arbitrator 605 can read only the elements from... Figure 6 The FIFO queue 620 has n–w elements: between these read elements and w waiting elements, the judge 605 will have enough values ​​to fill the processing element 630 for another iteration.

[0093] Figure 9 An embodiment of the present invention is shown. Figure 5 Details of the ACG module 525. Figure 9 In the diagram, the ACG module 525 is shown as including two 2D SIMD PEs 905 and 910, two buffers 915 and 920, a ReLU 925, and a trimmer 930. The 2D SIMD PEs 905 and 910 are connected to... Figure 6 Similar to the 2D SIMD PE 615, but the 2D SIMD PE 905 and 910 can perform different operations: the 2D SIMD PE 905 can perform accumulation operations, while the 2D SIMD PE 910 can perform multiplication and accumulation operations (and operate on the edge data of the graph rather than the node data, as shown in the second data input to the 2D SIMD PE 910). Compared to... Figure 6 Similar to the 2D SIMD PE 615, the 2D SIMD PE 905 and 910 can begin working as soon as there is enough available data, rather than waiting for the 2D SIMD PE 905 and 910 to start working. Figure 6 All processing elements 630 have data.

[0094] Buffers 915 and 920 can be used to store the outputs of 2D SIMD PEs 905 and 910 within the ACG module 525 for further processing: such as Figure 9 As shown, buffer 915 can store intermediate feature calculations from 2D SIMD PE 905, which can in turn be used as input to 2D SIMD PE 910, and buffer 920 can store feature calculations from 2D SIMD PE 910, which can then be used as input to ReLU 925.

[0095] exist Figure 5 At the end of each GCN layer 515, an activation function may exist. The activation function may be in the form of a ReLU 925, which can be used to activate features from the buffer 920. The ReLU 925 may use a nonlinear function (such as, for example, a function that determines the maximum or zero value of the input) to perform activation.

[0096] The trimmer 930 can trim any zero from a feature processed by the ReLU 925. Note that the ReLU 925 can replace negative values ​​with zero; the trimmer 930 can then remove zero (or other values) from the graph data. The trimmer 930 can also modify the graph data: for example, adding or changing values. The trimmer 930 can also place the (non-zero) data of the graph into... Figure 6 In the FIFO queue 620, for use Figure 5 In the next GCN layer 515.

[0097] Note that in some embodiments of the present invention, in Figure 5 In the final GCN layer 515, the output of the ACG module 525 can be complete: that is, including zeros. By including zeros in the output of the ACG module 525, feature extraction can be more complete (because subsequent use of features may expect the presence of zeros). Therefore, in Figure 5 In the final GCN layer 515 of the ACG module 525, the trimmer 930 can be omitted (or not used). However, in other embodiments of the invention, even... Figure 5 In the final GCN layer 515 of the ACG module 930, the trimmer 930 can also work, provided that enough information is provided to reintroduce zeros when needed (e.g., by identifying which elements in the output matrix include the data output by the trimmer 930, or by identifying which elements in the output matrix will be zero).

[0098] Figure 10 An embodiment of the invention is shown for use with Figure 1 An example flowchart illustrating the operation of the preprocessor 140. Figure 10 In block 1005, Figure 1 The preprocessor 140 can be obtained from Figure 1 The element is read from memory 115. In block 1010, Figure 1 The preprocessor 140 can identify and remove any zero elements (or other values). Figure 1 The preprocessor 140 can also make other modifications to the data used for graphics, such as changing values ​​in the data or adding values ​​to the data. In block 1015, Figure 1 The preprocessor 140 can reorder the elements to avoid any data dependencies. Finally, in block 1020, Figure 1 The preprocessor 140 can write the preprocessed data of the graphics back Figure 1 The memory 115 is for use with Figure 5 The prefetcher 510 reads.

[0099] Figure 11 An embodiment of the invention is shown for use with Figure 5 The prefetcher 510 and / or Figure 9Example flowchart of the operation of trimmer 930. In block 1105, Figure 5 The prefetcher 510 and / or Figure 9 The trimmer 930 can read elements. Figure 5 The prefetcher 510 can be obtained from Figure 1 The element is retrieved from memory 115, and Figure 9 The trimmer 930 can be used from Figure 5 Elements are retrieved from the buffer or cache within the ACG module 525. The number of elements that can be retrieved can vary depending on the implementation: for example, Figure 5 The prefetcher 510 and / or Figure 9 The trimmer 930 can obtain at least with Figure 5 The next GCN layer 515 Figure 5 The multiplication module 520 Figure 6 The processing of element 630 is the same as the element. In block 1110, Figure 5 The prefetcher 510 and / or Figure 9 The trimmer 930 can check if an element is non-zero. Note that block 1110 can be executed in parallel on all prefetched elements if needed to take advantage of... Figure 1 The accelerator 135 supports parallelism. In block 1115, if the element is non-zero, the element can be written. Figure 6 The appropriate FIFO queue is 620. (Note that if...) Figure 5 The preprocessor 140 has already eliminated zeros from the data from the graphics. Figure 5 The prefetcher 510 can continue to block 1115 without performing the checks in block 1110, because the checks in block 1110 may always return true. Blocks 1110 and 1115 can also be generalized: Figure 1 Preprocessor 140 and / or Figure 9 The trimmer 930 can change values ​​in data, add values ​​to data, or remove values ​​other than zero from data.

[0100] Figure 12 An embodiment of the invention is shown for use with Figure 6 Example flowchart of the operation of the decision-maker 605. Figure 12 In block 1205, Figure 6 The 605 decision-maker can be used from Figure 6 The FIFO queue 620 retrieves elements. This operation may include retrieving elements previously retrieved via hold due to data dependencies. Therefore, from... Figure 6 The actual number of elements retrieved by the FIFO queue 620 can be... Figure 6 The difference between the number of elements 630 to be processed and the number of elements waiting to be processed from the previous cycle. In block 1210, Figure 6 The arbiter 605 can check whether data dependencies are found in the elements retrieved in block 1205. Note that, if needed, block 1215 can be executed in parallel on all retrieved elements to take advantage of... Figure 1 The accelerator 135 supports parallelism. If a data dependency is found, then in block 1210, Figure 6 The 605 analyzer can insert a bubble (no-op). Figure 6 2D SIMD PE 615; otherwise, in block 1220, Figure 6 The 605 sorting unit can insert elements. Figure 6 The 2DSIMD PE 615. (Note that if...) Figure 5 The preprocessor 140 has reordered the data from the graph to eliminate data dependencies. Figure 6 The arbiter 605 can continue to block 1220 without performing the checks in block 1215 or the operations in block 1210, because the checks in block 1215 may always return false results. Therefore, if Figure 6 The decision maker 605 is located in Figure 5 In the first GCN layer 515, then Figure 6 The decision-maker 605 can omit blocks 1215 and 1210. (See above for reference.) Figure 8 Discussion, if Figure 6 The 605 calibrator inserts the bubble. Figure 6 In the 2D SIMD PE 615, it may have already been inserted into Figure 6 Data in the 2D SIMD PE 615 can be retained. Figure 6 In the FIFO queue 620, or stored in Figure 6 The buffer in the resolver 605 is used for subsequent processing once the data dependency has been resolved.

[0101] Figure 13 An embodiment of the invention is shown for use with Figure 1 Accelerator 135 confirmed Figure 3 A flowchart illustrating the characteristics of a graphic. In Figure 13 In block 1305, Figure 1 The accelerator 135 can Figure 3 The node data of graph 305 is loaded into Figure 6 In the 2D SIMD PE 615. In block 1310, Figure 5 The 520 multiplication module can be loaded into Figure 6 2D SIMDPE 615 Figure 3 The node data of graph 305 are multiplied to produce a product.

[0102] In block 1315, Figure 1 The accelerator 135 can load the product into Figure 9 In the 2D SIMD PE 905. In block 1320, Figure 5 The ACG module 525 can... Figure 9 In the 2D SIMD PE 905, the product is accumulated to produce intermediate features, which can be stored... Figure 9 In the intermediate feature buffer 915.

[0103] In block 1325, Figure 1 The accelerator 135 can load intermediate features into Figure 9 In the 2D SIMD PE 910. In block 1330, Figure 1 The accelerator 135 can also... Figure 3 The edge data of graph 305 is loaded into Figure 9 In the 2D SIMD PE 910. In block 1335, Figure 5 The ACG module 525 can... Figure 9 In the 2D SIMD PE 910 Figure 3 The intermediate features and edge data of graph 305 are multiplied and accumulated to produce features, which can be stored in the graph. Figure 9 The feature buffer 920.

[0104] Finally, at block 1340, Figure 9 The pruner 930 can prune zeros from features to produce output data. More generally, Figure 9 The pruner 930 can modify data in any desired way: by adding values ​​to the data, changing values ​​in the data, or removing values ​​from the data (which can be non-zero). This output data can be written to... Figure 1 The memory 115, and can be used by the memory 115. Figure 1 Various applications run on the machine 105.

[0105] Figure 14 An embodiment of the invention is shown for use with Figure 1 Accelerator 135 confirmed Figure 3 A flowchart illustrating alternative examples of the graphical characteristics. Figure 14 In block 1405, Figure 1 The preprocessor 140 can preprocess Figure 3 The data in graph 305. In block 1410, Figure 5 The prefetcher 510 can prefetch Figure 3 The data in graph 305. This data may include, for example, node data, edge data, and weights. In block 1415, Figure 5 The multiplication module 520 can perform multiplication on... Figure 3Some data in graph 305 are multiplied. In block 1420, Figure 5 The ACG module 525 can... Figure 3 The data in graph 305 undergoes accumulation and aggregation operations. Finally, in block 1425, if there are more GCN layers to execute, control can return to block 1415 to process another layer; otherwise, the operation can complete.

[0106] Figure 15 An embodiment of the present invention is shown. Figure 1 Preprocessor 140 to Figure 3 A flowchart illustrating an example process for preprocessing graphics 305. Figure 15 In block 1505, Figure 1 The preprocessor 140 can be from, for example Figure 1 Memory 115 loaded Figure 3 The data in graphic 305. In block 1510, Figure 1 The preprocessor 140 can be obtained from Figure 3 Trim zeros from the data in graph 305 (or more generally, by removing zeros from...). Figure 3 To modify the data in graph 305, add, change, and / or remove values ​​that may be non-zero. Figure 3 (Data from graphic 305). In block 1515, Figure 1 The preprocessor 140 can reorder the data to remove data dependencies. Finally, in block 1520, Figure 1 Preprocessor 140 can Figure 3 The preprocessed data of the graphics 305 is stored back, for example. Figure 1 In memory 115.

[0107] Figure 16 An embodiment of the invention is shown for use with Figure 5 510 prefetcher Figure 3 A flowchart illustrating an alternative example process for the data in Figure 305. Figure 16 In block 1605, Figure 5 The prefetcher 510 can, for example, Figure 1 Memory 115 prefetch Figure 3 The data in graphic 305. In block 1610, Figure 5 The prefetcher 510 can Figure 3 The data of the graphic 305 is stored in Figure 1 In the buffer or cache of the accelerator 135.

[0108] Figure 17 An embodiment of the present invention is shown. Figure 5 The multiplication module 520 uses Figure 3The flowchart illustrates an example process of performing multiplication operations on data from graph 305. Figure 17 In block 1705, Figure 6 The 605 decision-maker can be used from Figure 6 FIFO queue 620 is loaded with elements. In block 1710, Figure 6 The adjudicator 605 can determine that the element being processed involves a data dependency: that is, the element is used to update a value that is also being updated by another element currently being processed. If so, then in block 1715, Figure 6 The 605 detector can insert a bubble. Figure 6 In 2D SIMD PE 615; otherwise, in block 1720, Figure 6 The 605 sorting unit can insert elements. Figure 6 In the 2D SIMD PE 615.

[0109] Regardless of the method, in block 1725, Figure 6 The 605 decision-maker can load weights into Figure 6 In the 2D SIMD PE615, and in block 1730, Figure 5 The multiplication module 520 can be used Figure 6 The 2D SIMD PE 615 performs multiplication operations.

[0110] Figure 18 An embodiment of the present invention is shown. Figure 5 The ACG module 525 uses Figure 3 Figure 1905 is a flowchart illustrating an example process of performing accumulation and aggregation operations on the data. Figure 18 In block 1805, Figure 5 The ACG module 525 can Figure 5 The output of the multiplication module 520 is loaded into Figure 9 In the 2D SIMD PE 905. In block 1810, Figure 5 The ACG module 525 can be used Figure 9 The 2D SIMD PE 905 performs an accumulation operation, which can be stored in the intermediate feature buffer 915.

[0111] In block 1815, Figure 5 The ACG module 525 can load features from the intermediate feature buffer 915 into Figure 9 In the 2D SIMD PE 910. In block 1820, Figure 5 The ACG module 525 can also... Figure 3 The edge data of graph 305 is loaded into Figure 9 In the 2D SIMD PE 910. In block 1825, Figure 5The ACG module 525 can be used Figure 9 The 2D SIMD PE 910 performs multiplication and accumulation operations, which can be stored in the feature buffer 920.

[0112] In block 1830, Figure 9 The ReLU 925 can perform activation on features in feature buffer 920. Finally, in block 1835, Figure 9 The trimmer 930 can trim any zero (or more generally,) from the active features. Figure 9 The trimmer 930 can be accessed from... Figure 3 To modify the data in graph 305, add, change, and / or remove values ​​that may be non-zero. Figure 3 (Graphic data 305).

[0113] exist Figures 10-18 Some embodiments of the invention are illustrated in the figures. However, those skilled in the art will recognize that other embodiments of the invention are possible by changing the order of blocks, omitting blocks, or by including links not shown in the figures. All such variations of the flowcharts, whether explicitly described or not, are considered embodiments of the invention.

[0114] Embodiments of the present invention include an accelerator for performing graph convolutional networks (GCNs). The accelerator can eliminate zero elements, thereby speeding up the overall operation. The accelerator can identify and eliminate data dependencies (as part of preprocessing) or prevent data dependencies from introducing errors (by inserting bubbles where data dependencies might occur). The accelerator can minimize accesses to graph data from memory, potentially speeding up the operation, since accesses to memory can be slower than accesses to buffers within the accelerator.

[0115] The following discussion is intended to provide a brief, general description of one or more suitable machines in which certain aspects of the invention may be implemented. One or more machines may be controlled at least in part by input from conventional input devices such as keyboards, mice, etc., and by instructions received from another machine, interaction with a virtual reality (VR) environment, biometric feedback, or other input signals. As used herein, the term "machine" is intended to broadly encompass a single machine, a virtual machine or communication-coupled machine system, a virtual machine, or devices operating together. Exemplary machines include computing devices such as personal computers, workstations, servers, portable computers, handheld devices, telephones, tablets, etc., and transportation devices such as private or public transportation vehicles, such as cars, trains, taxis, etc.

[0116] One or more machines may include embedded controllers, such as programmable or non-programmable logic devices or arrays, application-specific integrated circuits (ASICs), embedded computers, smart cards, etc. One or more machines may utilize one or more connections to one or more remote machines, such as via network interfaces, modems, or other communication coupling. Machines may be interconnected via physical and / or logical networks, such as intranets, the Internet, local area networks (LANs), wide area networks (WANs), etc. Those skilled in the art will understand that network communications can utilize a variety of wired and / or wireless short-range or long-range carriers and protocols, including radio frequency (RF), satellite, microwave, IEEE 802.11, Bluetooth, optical, infrared, cable, laser, etc.

[0117] Embodiments of the present invention can be described by referring to or in conjunction with relevant data, including functions, processes, data structures, applications, etc., which, when accessed by a machine, cause the machine to perform tasks or define abstract data types or low-level hardware contexts. The associated data can be stored, for example, in volatile and / or non-volatile memory (e.g., RAM, ROM, etc.), or in other storage devices and their associated storage media, including hard disk drives, floppy disks, optical storage, magnetic tape, flash memory, memory sticks, digital video disks, bio-memory disks, etc. The associated data can be transmitted over a transmission environment (including physical and / or logical networks) in the form of packets, serial data, parallel data, propagated signals, etc., and can be used in compressed or encrypted formats. The associated data can be used in a distributed environment and stored locally and / or remotely for machine access.

[0118] Embodiments of the present invention may include a tangible, non-transitory machine-readable medium, including instructions executable by one or more processors, including instructions for performing elements of the present invention as described herein.

[0119] The various operations described above can be performed by any suitable means capable of performing the operations, such as various hardware and / or software components, circuits and / or modules. Software may include an ordered list of executable instructions for implementing logical functions and may be embodied in any "processor-readable medium" for instruction execution systems, apparatuses, or devices, such as single-core or multi-core processors or processors containing systems, or related to them.

[0120] The blocks or steps of the methods or algorithms described in conjunction with the embodiments disclosed herein, as well as their functions, may be directly embodied in hardware, software modules executed by a processor, or a combination of both. If implemented in software, the functions may be stored as one or more instructions or code on or transmitted through a tangible, non-transitory computer-readable medium. The software modules may reside in random access memory (RAM), flash memory, read-only memory (ROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art.

[0121] Having described and illustrated the principles of the invention with reference to the illustrated embodiments, it will be appreciated that the illustrated embodiments may be modified in arrangement and detail without departing from these principles, and may be combined in any desired manner. However, while the foregoing discussion focuses on particular embodiments, other configurations are contemplated. In particular, even though expressions such as "according to an embodiment of this disclosure" are used herein, these phrases are intended to broadly refer to the possibilities of embodiments and are not intended to limit this disclosure to particular embodiment configurations. As used herein, these terms may refer to the same or different embodiments that can be combined into other embodiments.

[0122] The illustrative embodiments described above should not be construed as limiting the disclosure therein. Although some embodiments have been described, those skilled in the art will readily understand that many modifications can be made to these embodiments without substantially departing from the novel teachings and advantages of the invention. Therefore, all such modifications are intended to be included within the scope of this disclosure as defined in the claims.

[0123] Embodiments of the present invention may be extended to the following statements, but are not limited to:

[0124] Statement 1. Embodiments of the present invention include a device comprising:

[0125] The multiplication module is used to perform multiplication based at least on node data or weight data of the graph;

[0126] An accumulation and aggregation (ACG) module for performing accumulation and aggregation based at least in part on the edge data of the multiplication module or the graph; and

[0127] The control unit manages the multiplication module and the ACG module.

[0128] Statement 2. Embodiments of the present invention include the device according to Statement 1, wherein the device is implemented at least in part using at least one of a field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), central processing unit (CPU), graphics processing unit (GPU), general-purpose GPU (GPGPU), data processing unit (DPU), or tensor processing unit (TPU).

[0129] Statement 3. Embodiments of the present invention include the apparatus according to Statement 1, wherein:

[0130] The device also includes a prefetcher for retrieving node and weight data of the graph from memory; and

[0131] The control unit is configured to manage the prefetcher.

[0132] Statement 4. Embodiments of the invention include the apparatus according to Statement 3, wherein the prefetcher includes at least one first-in-first-out (FIFO) queue for storing values ​​of node data from the graph.

[0133] Statement 5. Embodiments of the present invention include the device according to Statement 4, wherein the multiplication module is configured to select a value from at least one FIFO queue.

[0134] Statement 6. Embodiments of the present invention include the apparatus according to Statement 1, wherein the multiplication module includes at least one single instruction, multiple data processing element (SIMD-PE) to generate a product based at least partially on graph-based node data.

[0135] Statement 7. Embodiments of the present invention include the device according to Statement 6, wherein the multiplication module includes at least two SIMDPEs.

[0136] Statement 8. Embodiments of the present invention include the device according to Statement 6, wherein the multiplication module comprises a two-dimensional (2D) SIMDPE.

[0137] Statement 9. Embodiments of the present invention include a device according to Statement 6, wherein the SIMD-PE is configured to perform a multiplication operation to produce a product.

[0138] Statement 10. Embodiments of the present invention include the device according to Statement 6, wherein the SIMD-PE is further configured to perform multiplication operations to produce a product based at least in part on graph-based node data and weight data.

[0139] Statement 11. Embodiments of the present invention include the apparatus according to Statement 10, wherein the multiplication module further includes a weight buffer for storing weight data.

[0140] Statement 12. Embodiments of the present invention include the apparatus according to Statement 6, wherein the multiplication module further includes an optimizer for selecting values ​​of node data of a graph for SIMD-PE processing.

[0141] Statement 13. Embodiments of the invention include a device according to Statement 1, wherein the ACG module includes at least one SIMDPE to generate features at least in part based on the multiplication module.

[0142] Statement 14. Embodiments of the present invention include a device according to Statement 13, wherein the ACG module includes at least two SIMD PEs.

[0143] Statement 15. Embodiments of the present invention include the device according to Statement 13, wherein the ACG module includes a 2D SIMD PE.

[0144] Statement 16. Embodiments of the invention include a device according to Statement 13, wherein the SIMD-PE is configured to perform an accumulation operation at least in part based on a multiplication module to generate features.

[0145] Statement 17. Embodiments of the present invention include the device according to Statement 13, wherein the ACG module further includes a feature buffer for storing features.

[0146] Statement 18. Embodiments of the invention include a device according to Statement 13, wherein the ACG module further includes a second SIMDPE to generate a second feature at least in part based on the feature.

[0147] Statement 19. One embodiment of the invention includes a device according to Statement 18, wherein a second SIMD-PE is configured to perform multiplication and accumulation operations to generate a second feature based at least in part on graph feature or edge data.

[0148] Statement 20. Embodiments of the present invention include the apparatus according to Statement 18, wherein the ACG module further includes a modified linear unit (ReLU) for performing an activation function to generate an activation feature at least in part based on a second feature.

[0149] Statement 21. Embodiments of the present invention include the device according to Statement 20, wherein the ACG module further includes a trimmer for modifying values ​​in the activation feature.

[0150] Statement 22. Embodiments of the invention include a device according to Statement 21, wherein the trimmer is configured to remove zeros from an active feature.

[0151] Statement 23. Embodiments of the invention include the device according to Statement 21, wherein the trimmer includes at least one FIFO queue for storing values ​​from data.

[0152] Statement 24. Embodiments of the present invention include the device according to Statement 23, wherein the arbiter is configured to select data from at least one FIFO queue.

[0153] Statement 25. Embodiments of the present invention include the apparatus according to Statement 1, wherein:

[0154] The multiplication module and the ACG module form a layer; and

[0155] The device also includes a second multiplication module and a second ACG module that form the second layer.

[0156] Statement 26. Embodiments of the present invention include the apparatus according to Statement 1, the apparatus further comprising a preprocessor for modifying values ​​in node data of a graph.

[0157] Statement 27. Embodiments of the invention include the apparatus according to Statement 26, wherein the preprocessor is configured to remove zeros from the node data of the graph.

[0158] Statement 28. Embodiments of the present invention include the apparatus according to Statement 26, wherein the preprocessor is configured to store node data and edge data of a graph in a memory.

[0159] Statement 29. Embodiments of the present invention include a device according to statement 28, wherein the device includes a memory.

[0160] Statement 30. Embodiments of the present invention include the device according to statement 28, wherein the memory includes host memory.

[0161] Statement 31. Embodiments of the present invention include the apparatus according to Statement 26, wherein a preprocessor is configured to reorder node data of a graph.

[0162] Statement 32. Embodiments of the invention include the apparatus according to statement 31, wherein the preprocessor is configured to reorder node data of a graph to remove data dependencies.

[0163] Statement 33. Embodiments of the present invention include the apparatus according to statement 26, wherein the preprocessor is executed at least partially on the host processor.

[0164] Statement 34. Embodiments of the present invention include a device according to statement 26, wherein the preprocessor is executed at least partially on components of the device.

[0165] Statement 35. Embodiments of the present invention include the device according to statement 34, wherein the components include at least one of an FPGA, an ASIC, a CPU, a GPU, a GPGPU, a DPU, or a TPU.

[0166] Statement 36. Embodiments of the present invention include a system comprising:

[0167] Host processor;

[0168] Host memory, coupled to the host processor; and

[0169] The equipment includes:

[0170] The multiplication module is used to perform multiplication based at least on node data or weight data of the graph;

[0171] An accumulation and aggregation (ACG) module for performing accumulation and aggregation based at least in part on the edge data of the multiplication module or the graph; and

[0172] The control unit manages the multiplication module and the ACG module.

[0173] Statement 37. Embodiments of the present invention include a system according to Statement 36, wherein the device is implemented at least in part using at least one of a field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), central processing unit (CPU), graphics processing unit (GPU), general-purpose GPU (GPGPU), data processing unit (DPU), or tensor processing unit (TPU).

[0174] Statement 38. Embodiments of the present invention include the system according to statement 36, wherein:

[0175] The device also includes a prefetcher for retrieving node and weight data of the graph from memory; and

[0176] The control unit is configured to manage the prefetcher.

[0177] Statement 39. Embodiments of the invention include a system according to Statement 38, wherein the prefetcher includes at least one first-in-first-out (FIFO) queue for storing values ​​of node data from the graph.

[0178] Statement 40. Embodiments of the present invention include a system according to statement 39, wherein the multiplication module is configured to select a value from at least one FIFO queue.

[0179] Statement 41. Embodiments of the present invention include a system according to Statement 36, wherein the multiplication module includes at least one single instruction, multiple data processing element (SIMD-PE) to generate a product based at least partially on graph-based node data.

[0180] Statement 42. Embodiments of the present invention include the system according to statement 41, wherein the multiplication module includes at least two SIMD PEs.

[0181] Statement 43. Embodiments of the present invention include a system according to statement 41, wherein the multiplication module comprises a two-dimensional (2D) SIMD PE.

[0182] Statement 44. Embodiments of the present invention include a system according to statement 41, wherein the SIMD-PE is configured to perform multiplication to produce a product.

[0183] Statement 45. Embodiments of the present invention include a system according to statement 41, wherein the SIMD-PE is further configured to perform multiplication operations to produce a product based at least in part on graph-based node data and weight data.

[0184] Statement 46. Embodiments of the present invention include the system according to statement 45, wherein the multiplication module further includes a weight buffer for storing weight data.

[0185] Statement 47. Embodiments of the present invention include the system according to statement 41, wherein the multiplication module further includes a discriminator for selecting values ​​of node data of a graph for SIMD-PE processing.

[0186] Statement 48. Embodiments of the present invention include a system according to statement 36, wherein the ACG module includes at least one SIMD PE to generate features at least in part based on the multiplication module.

[0187] Statement 49. Embodiments of the present invention include a system according to statement 48, wherein the ACG module includes at least two SIMD PEs.

[0188] Statement 50. Embodiments of the present invention include the system according to statement 48, wherein the ACG module includes a 2D SIMD PE.

[0189] Statement 51. Embodiments of the present invention include a system according to statement 48, wherein the SIMD-PE is configured to perform an accumulation operation at least in part based on the multiplication module to produce features.

[0190] Statement 52. Embodiments of the present invention include the system according to statement 48, wherein the ACG module further includes a feature buffer for storing features.

[0191] Statement 53. Embodiments of the present invention include the system according to statement 48, wherein the ACG module further includes a second SIMDPE to generate a second feature at least in part based on the feature.

[0192] Statement 54. Embodiments of the invention include a system according to Statement 53, wherein a second SIMD-PE is configured to perform multiplication and accumulation operations to generate a second feature based at least in part on the features or edge data of the graph.

[0193] Statement 55. Embodiments of the present invention include the system according to Statement 53, wherein the ACG module further includes a modified linear unit (ReLU) for performing an activation function to generate an activation feature at least in part based on a second feature.

[0194] Statement 56. Embodiments of the present invention include the system according to Statement 55, wherein the ACG module further includes a pruner that modifies values ​​in the activation feature.

[0195] Statement 57. Embodiments of the invention include a system according to statement 56, wherein a trimmer is configured to remove zeros from an active feature.

[0196] Statement 58. Embodiments of the present invention include a system according to Statement 56, wherein the trimmer includes at least one FIFO queue for storing values ​​from data.

[0197] Statement 59. Embodiments of the present invention include a system according to statement 58, wherein the arbiter is configured to select data from at least one FIFO queue.

[0198] Statement 60. Embodiments of the present invention include the system according to statement 36, wherein:

[0199] The multiplication module and the ACG module form a layer; and

[0200] The device also includes a second multiplication module and a second ACG module that form the second layer.

[0201] Statement 61. Embodiments of the present invention include the system according to Statement 36, the system further including a preprocessor for modifying values ​​in the node data of a graph.

[0202] Statement 62. Embodiments of the invention include a system according to statement 61, wherein a preprocessor is configured to remove zeros from the node data of a graph.

[0203] Statement 63. Embodiments of the present invention include a system according to statement 61, wherein a preprocessor is configured to store node data and edge data of a graph in memory.

[0204] Statement 64. Embodiments of the present invention include a system according to statement 63, wherein the device includes a memory.

[0205] Statement 65. Embodiments of the present invention include a system according to statement 63, wherein the memory includes host memory.

[0206] Statement 66. Embodiments of the invention include a system according to statement 61, wherein a preprocessor is configured to reorder node data of a graph.

[0207] Statement 67. Embodiments of the present invention include a system according to Statement 66, wherein the preprocessor is further configured to reorder node data of a graph to remove data dependencies.

[0208] Statement 68. Embodiments of the present invention include a system according to statement 61, wherein the preprocessor is executed at least partially on a host processor.

[0209] Statement 69. Embodiments of the present invention include a system according to statement 61, wherein the preprocessor is executed at least partially on a component of the device.

[0210] Statement 70. Embodiments of the present invention include the system according to statement 69, wherein the components include at least one of an FPGA, an ASIC, a CPU, a GPU, a GPGPU, a DPU, or a TPU.

[0211] Statement 71. An embodiment of the present invention includes a method comprising:

[0212] Load the node data of the graph into the device’s first Single Instruction Multiple Data Processing Element (SIMD-PE);

[0213] The first SIMD PE is used to perform multiplication on the node data used for the graph to produce a product;

[0214] The product is loaded into the second SIMD PE in the device;

[0215] The product is accumulated using a second SIMD-PE to produce the first feature;

[0216] Load the first feature into the device's third SIMD-PE;

[0217] Load the edge data of the graph into the device's third SIMD-PE;

[0218] The third SIMD PE is used to perform multiplication and accumulation operations on the first feature and the edge data of the graph to produce the second feature; and

[0219] Zeros are pruned from the second feature to produce output data.

[0220] Statement 72. Embodiments of the present invention include the method according to Statement 71, wherein the device is implemented at least in part using at least one of a field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), central processing unit (CPU), graphics processing unit (GPU), general-purpose GPU (GPGPU), data processing unit (DPU), or tensor processing unit (TPU).

[0221] Statement 73. Embodiments of the present invention include the method according to statement 71, and further include prefetching node data of a graph.

[0222] Statement 74. Embodiments of the present invention include the method according to statement 73, wherein prefetching node data of a graph includes prefetching node data of a graph from a memory.

[0223] Statement 75. Embodiments of the present invention include the method according to statement 74, wherein the memory includes host memory.

[0224] Statement 76. Embodiments of the present invention include the method according to statement 74, wherein the apparatus includes a memory.

[0225] Statement 77. Embodiments of the present invention include the method according to Statement 71, wherein prefetching node data of a graph includes storing the node data of the graph in a buffer of the device.

[0226] Statement 78. Embodiments of the present invention include the method according to Statement 71, further comprising preprocessing node data of a graph to generate preprocessed node data.

[0227] Statement 79. Embodiments of the present invention include the method according to statement 78, and further include storing preprocessed node data in a memory.

[0228] Statement 80. Embodiments of the invention include the method according to statement 78, wherein preprocessing node data of a graph to produce preprocessed node data includes trimming (1510) zeros from the node data used for the graph.

[0229] Statement 81. Embodiments of the present invention include the method according to statement 78, wherein preprocessing the node data of a graph includes reordering the node data of the graph.

[0230] Statement 82. Embodiments of the present invention include the method according to Statement 81, wherein reordering the node data of the preprocessed graph includes removing data dependencies.

[0231] Statement 83. Embodiments of the present invention include the method according to Statement 78, wherein preprocessing node data of a graph to generate preprocessed node data includes using a host processor to preprocess node data of a graph to generate preprocessed node data.

[0232] Statement 84. Embodiments of the present invention include the method according to statement 78, wherein preprocessing node data of a graph to generate preprocessed node data includes preprocessing node data of a graph to generate preprocessed node data using components of the device.

[0233] Statement 85. Embodiments of the present invention include the method according to statement 84, wherein the components include at least one of an FPGA, an ASIC, a CPU, a GPU, a GPGPU, a DPU, or a TPU.

[0234] Statement 86. Embodiments of the present invention include the method according to statement 71, wherein loading node data of a graph into a first SIMD-PE of the device includes loading weight data into the first SIMD-PE of the device.

[0235] Statement 87. Embodiments of the invention include the method according to Statement 71, wherein loading node data of a graph into a first SIMD PE of the device includes inserting values ​​of the node data from the graph into a first-in-first-out (FIFO) queue based at least in part on values ​​that are positive.

[0236] Statement 88. Embodiments of the present invention include the method according to Statement 87, wherein the FIFO queue is located in the prefetcher of the device.

[0237] Statement 89. Embodiments of the present invention include the method according to Statement 87, wherein the FIFO queue is located in the trimmer of the device.

[0238] Statement 90. Embodiments of the present invention include the method according to statement 87, wherein loading node data of a graph into a first SIMD-PE of the device further includes loading values ​​from a FIFO queue into the first SIMD-PE.

[0239] Statement 91. Embodiments of the present invention include the method according to Statement 90, wherein loading a value from a FIFO queue into a first SIMD PE includes loading the value from the FIFO queue into the first SIMD PE using an optimizer.

[0240] Statement 92. Embodiments of the invention include the method according to Statement 71, wherein loading node data of a graph into a first SIMD-PE of the device comprises loading values ​​into the first SIMD-PE at least in part based on values ​​independent of any pending computations.

[0241] Statement 93. Embodiments of the present invention include the method according to statement 71, wherein:

[0242] Loading node data of a graph into the first SIMD-PE of the device includes loading at least two elements of the node data of the graph into the first SIMD-PE of the device;

[0243] Loading the product into the second SIMD PE includes loading at least two products into the second SIMD PE of the device; and

[0244] Loading a first feature into the third SIMD PE of the device includes loading at least two first features into the third SIMD PE of the device.

[0245] Statement 94. Embodiments of the present invention include the method according to statement 71, wherein:

[0246] Loading the node data of the graph into the first SIMD-PE of the device includes loading the node data of the graph into the first two-dimensional (2D) SIMD-PE of the device;

[0247] Loading the product into a second SIMD PE in the device includes loading the product into a second 2D SIMD PE in the device; and

[0248] Loading the first feature into the third SIMD PE of the device includes loading the first feature into the third 2D SIMD PE of the device.

[0249] Statement 95. Embodiments of the present invention include the method according to statement 71, wherein:

[0250] Loading the node data of the graph into the first SIMD-PE of the device includes loading the node data of the graph from the first buffer of the device into the first SIMD-PE of the device;

[0251] Loading the product into the second SIMD-PE of the device includes loading the product from the second buffer in the device into the second 2D SIMD-PE of the device; and

[0252] Loading the first feature into the third SIMD PE of the device includes loading the first feature from the third buffer in the device into the third 2D SIMD PE of the device.

[0253] Statement 96. Embodiments of the present invention include an article comprising a non-transitory storage medium storing instructions that, when executed by a machine, result in the following:

[0254] Load the node data of the graph into the device’s first Single Instruction Multiple Data Processing Element (SIMD-PE);

[0255] The first SIMD PE is used to perform multiplication on the node data used for the graph to produce a product;

[0256] The product is loaded into the second SIMD PE in the device;

[0257] The product is accumulated using a second SIMD-PE to produce the first feature;

[0258] Load the first feature into the device's third SIMD-PE;

[0259] Load the edge data of the graph into the device's third SIMD-PE;

[0260] The third SIMD PE is used to perform multiplication and accumulation operations on the first feature and the edge data of the graph to produce the second feature; and

[0261] Zeros are pruned from the second feature to produce output data.

[0262] Statement 97. Embodiments of the present invention include articles according to Statement 96, wherein the device is implemented at least in part using at least one of a field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), central processing unit (CPU), graphics processing unit (GPU), general-purpose GPU (GPGPU), data processing unit (DPU), or tensor processing unit (TPU).

[0263] Statement 98. Embodiments of the invention include an article according to Statement 96, wherein the non-transitory storage medium stores further instructions thereon that, when executed by a machine, cause node data of a graph to be prefetched.

[0264] Statement 99. Embodiments of the present invention include an article according to statement 98, wherein the node data of the prefetched graphic includes node data of the prefetched graphic from a memory.

[0265] Statement 100. Embodiments of the present invention include an article according to statement 99, wherein the memory includes a host memory.

[0266] Statement 101. Embodiments of the present invention include an article according to statement 99, wherein the device includes a memory.

[0267] Statement 102. Embodiments of the present invention include an article according to statement 96, wherein prefetching node data of a graphic includes storing the node data of the graphic in a buffer in the device.

[0268] Statement 103. Embodiments of the invention include an article according to Statement 96, wherein the non-transitory storage medium stores further instructions thereon that, when executed by a machine, cause the node data of a preprocessed graph to produce preprocessed node data.

[0269] Statement 104. Embodiments of the invention include an article according to Statement 103, wherein the non-transitory storage medium stores further instructions thereon, which, when executed by a machine, cause pre-processed node data to be stored in memory.

[0270] Statement 105. Embodiments of the invention include an article according to statement 103, wherein preprocessing the node data of a graph to produce preprocessed node data includes trimming (1510) zeros from the node data of the graph.

[0271] Statement 106. Embodiments of the invention include an article according to statement 103, wherein preprocessing the node data of a graphic includes reordering the node data of the graphic.

[0272] Statement 107. Embodiments of the invention include an article according to Statement 106, wherein reordering the node data of a graph includes removing data dependencies.

[0273] Statement 108. Embodiments of the present invention include an article according to Statement 103, wherein preprocessing node data of a graph to generate preprocessed node data includes using a host processor to preprocess node data of a graph to generate preprocessed node data.

[0274] Statement 109. Embodiments of the present invention include an article according to Statement 103, wherein preprocessing node data of a graphic to generate preprocessed node data includes using components of a device to preprocess node data of a graphic to generate preprocessed node data.

[0275] Statement 110. Embodiments of the present invention include articles according to statement 109, wherein the components include at least one of an FPGA, an ASIC, a CPU, a GPU, a GPGPU, a DPU, or a TPU.

[0276] Statement 111. Embodiments of the present invention include an article according to statement 96, wherein loading node data of a graph into a first SIMD-PE of the device includes loading weight data into the first SIMD-PE of the device.

[0277] Statement 112. Embodiments of the invention include an article according to statement 96, wherein loading node data of a graph into a first SIMD PE of the device includes at least in part a value of inserting node data from the graph into a first-in-first-out (FIFO) queue based on a value that is a positive value.

[0278] Statement 113. Embodiments of the invention include an article according to statement 112, wherein a FIFO queue is located in the prefetcher of the device.

[0279] Statement 114. Embodiments of the invention include an article according to Statement 112, wherein the FIFO queue is located in the trimmer of the device.

[0280] Statement 115. Embodiments of the invention include the article according to statement 112, wherein loading node data of a graph into a first SIMD-PE of the device further includes loading values ​​from a FIFO queue into the first SIMD-PE.

[0281] Statement 116. Embodiments of the invention include the article according to statement 115, wherein loading values ​​from a FIFO queue into a first SIMD PE includes using an optimizer to load values ​​from a FIFO queue into the first SIMD PE.

[0282] Statement 117. Embodiments of the invention include an article according to statement 96, wherein loading node data of a graph into a first SIMD-PE of the device comprises loading values ​​into the first SIMD-PE at least in part based on values ​​independent of any pending computation.

[0283] Statement 118. Embodiments of the invention include the article according to statement 96, wherein:

[0284] Loading node data of a graph into the first SIMD-PE of the device includes loading at least two elements of the node data of the graph into the first SIMD-PE of the device;

[0285] Loading the product into the second SIMD PE of the device includes loading at least two products into the second SIMD PE; and

[0286] Loading a first feature into the third SIMD PE of the device includes loading at least two first features into the third SIMD PE of the device.

[0287] Statement 119. Embodiments of the invention include the article according to statement 96, wherein:

[0288] Loading the node data of the graph into the first SIMD-PE of the device includes loading the node data of the graph into the first two-dimensional (2D) SIMD-PE of the device;

[0289] Loading the product into the second SIMD PE of the device includes loading the product into the second 2D SIMD PE of the device; and

[0290] Loading the first feature into the third SIMD PE of the device includes loading the first feature into the third 2D SIMD PE of the device.

[0291] Statement 120. Embodiments of the invention include the article according to statement 96, wherein:

[0292] Loading the node data of the graph into the first SIMD-PE of the device includes loading the node data of the graph from the first buffer of the device into the first SIMD-PE of the device;

[0293] Loading the product into the device's second SIMD-PE includes loading the product from the device's second buffer into the device's second 2D SIMD-PE; and

[0294] Loading the first feature into the third SIMD PE of the device includes loading the first feature from the third buffer of the device into the third 2D SIMD PE in the device.

[0295] Therefore, given the various arrangements of the embodiments described herein, this detailed description and accompanying materials are intended to be illustrative only and should not be considered as limiting the scope of the invention. Thus, the disclosure sought is for all modifications that may be made within the scope and spirit of the following claims and their equivalents.

Claims

1. A system for accelerating image feature extraction, comprising: Host processor; Host memory, coupled to the host processor; Preprocessor; as well as Devices implemented in hardware include: The first graph convolutional network (GCN) layer includes: A first multiplication module is configured to perform multiplication based at least on node data or weight data of the graph stored in a buffer in the device; and A first accumulation and aggregation ACG module, used to perform accumulation and aggregation based at least in part on edge data of a first multiplication module or graph; and The second graph convolutional network (GCN) layer includes: A second multiplication module is used to perform multiplication based at least on the output of the first ACG module; and A second ACG module is used to perform accumulation and aggregation based at least in part on the edge data of the second multiplication module or the graph; and The control unit is implemented as a field-programmable gate array (FPGA) to manage the first multiplication module and the first ACG module in the first GCN layer, as well as the second multiplication module and the second ACG module in the second GCN layer. The preprocessor is configured to modify values ​​in the node data of the graph, and the preprocessor is executed at least partially on the host processor, wherein the preprocessor is configured to remove zeros from the node data of the graph. in: The device further includes a prefetcher for retrieving modified node data and weight data of the graph from the host memory; and The control unit is configured to manage the prefetcher, and The process of prefetching node data of the graphic includes storing the node data of the graphic in the buffer of the device.

2. The system according to claim 1, wherein, At least one of the first multiplication module or the second multiplication module includes at least one single instruction, multiple data processing element (SIMD PE) to generate a product based at least partially on graph-based node data.

3. The system according to claim 2, wherein, At least one of the first multiplication module or the second multiplication module further includes a discriminator for selecting the value for the node data of the graph for SIMD PE processing.

4. The system according to any one of claims 1-3, wherein, The first ACG module or the second ACG module includes at least one SIMD PE (Multi-Data Processing Element) to generate features at least in part based on the first multiplication module or the second multiplication module.

5. The system according to claim 4, wherein, The first ACG module or the second ACG module further includes a second SIMD PE to generate a second feature at least in part based on the feature.

6. The system according to claim 5, wherein, The first ACG module or the second ACG module further includes a modified linear unit (ReLU) for performing an activation function to generate an activation feature based at least in part on the second feature.

7. The system according to claim 6, wherein, The first ACG module or the second ACG module further includes a trimmer for modifying values ​​in the activation feature.

8. The system according to claim 1, wherein, The preprocessor is also configured to store the node data and edge data of the graph in a memory.

9. The system according to claim 8, wherein, The preprocessor is also configured to reorder the node data of the graph.

10. A method for accelerating graphic feature extraction, which uses the system according to any one of claims 1 to 9 to extract features from a graphic.

Images