Information processing device, information processing method

Abstract

To provide a technology for efficiently performing graph convolution operations. [Solution] The information processing device obtains the weights and destinations of the nodes in the graph from a list (a list containing the weights of the graph nodes and the destinations of those nodes) from which information representing the non-zero values ​​of the adjacency matrix representing the connection relationships of the nodes in the graph has been extracted. The information processing device then obtains data to be input to the arithmetic unit from the data to be processed, based on the destinations, and uses the arithmetic unit to perform calculations using the obtained data and the obtained node weights.

Description

Technical Field

[0001] The present invention relates to a convolutional operation technique based on a graph.

Background Art

[0002] With the development of deep learning technology, it has become possible to analyze the molecular structure of compounds, social networks, natural languages, etc. using data with a graph structure (hereinafter abbreviated as graph). For example, in Non-Patent Document 1, a technique for object detection is disclosed in which a graph is generated from point cloud data obtained by LiDAR, and a convolutional operation using coefficients obtained by deep learning technology is hierarchically executed on this graph. Also, as shown in Patent Document 1 and Non-Patent Document 2, techniques for extracting more appropriate information from a graph are disclosed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Non-Patent Documents

[0004]

Non-Patent Document 1

Non-Patent Document 2

[0005] Patent Document 1 discloses a technique for performing graph convolution operations by calculating the product of a normalized adjacency matrix with a self-loop, a feature matrix summarizing the feature data of each node constituting the graph, and a parameter matrix of a trained neural network. On the other hand, the adjacency matrix of a graph (a matrix representing the weights of edges connecting nodes) often has most elements with values ​​of 0 (values ​​indicating that nodes are not connected and therefore no edge weights exist). Consequently, when calculating the product of this adjacency matrix with other matrices, the elements with values ​​of 0 in the adjacency matrix are also multiplied unnecessarily, increasing power consumption and processing time. The present invention provides a technique for efficiently performing graph convolution operations. [Means for solving the problem]

[0006] One embodiment of the present invention is an information processing device that performs a graph convolution operation on a graph, comprising: a first acquisition means for acquiring the weights and destinations of nodes in the graph from a list in which information is extracted from the portion of the adjacency matrix representing the connection relationships of nodes in the graph in which the values ​​representing the connection relationships are non-zero, the list having the weights of nodes in the graph and the destinations of the nodes; a second acquisition means for acquiring data to be input to an arithmetic unit from the data to be calculated, based on the destinations; and an arithmetic means for performing calculations using the data acquired by the second acquisition means and the weights of nodes acquired by the first acquisition means, using the arithmetic unit. [Effects of the Invention]

[0007] According to the present invention, graph convolution operations can be performed efficiently. [Brief explanation of the drawing]

[0008] [Figure 1] A block diagram showing an example of the hardware configuration of an information processing device. [Figure 2] A block diagram showing an example of the hardware configuration of the arithmetic unit 101. [Figure 3] A flowchart illustrating the processes performed by an information processing device. [Figure 4] A diagram illustrating the difference in the number of multiplications depending on the order of matrix calculations. [Figure 5] A diagram illustrating an example of the process for generating an edge weight list. [Figure 6] A diagram illustrating the operation of the arithmetic unit 101. [Figure 7] A diagram illustrating feature maps and feature vectors. [Figure 8] A diagram illustrating the graph structuring of feature vectors. [Figure 9] A diagram showing the network structure of a graph convolution operation. [Figure 10] A diagram showing the computational complexity based on the order of matrix operations. [Figure 11] A flowchart illustrating the processes performed by an information processing device. [Figure 12] A diagram illustrating the operation of the arithmetic unit 101. [Figure 13] A conceptual diagram illustrating how to generate an edge weight list. [Figure 14] A diagram illustrating the operation of the arithmetic unit 101. [Modes for carrying out the invention]

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0010] [First Embodiment] In this embodiment, a case will be described in which an information processing apparatus executes a task of classifying papers using a Cora dataset in which the citation relationship of papers is represented by a graph.

[0011] The Cora dataset (reference homepage, https: / / relational.fit.cvut.cz / dataset / CORA) is composed of 2,708 scientific papers and is a graph composed of 5,429 edges indicating citation and cited relationships, with each paper being one node. Also, for one node (paper), the feature data is defined by values of 0 (absence of appearance) and 1 (presence of appearance) indicating the presence or absence of 1,433 words related to the determination of the paper's category. Using the graph representing this Cora dataset as input, each paper is classified into seven categories by a two-layer graph convolutional layer.

[0012] For the operation of the two-layer graph convolutional layer used in this embodiment, the convolutional operation described in Non-Patent Document 2 is used. An overview of the case of applying it to the Cora dataset will be described. The data used as input in this embodiment is a feature matrix X0 with a height of 2708 and a width of 1433, where the number of nodes (number of papers) is 2708 and each node has 1433 feature data. Also, the adjacency matrix A of the graph (a matrix representing edge weights) is a matrix with a height of 2708 and a width of 2708 to show the connection relationship between nodes, and it is a matrix with non-zero values only in the elements with a citation-cited relationship with each other. When processing the paper classification task, it is preferable to use an adjacency matrix including self-loops, and the non-zero values are normalized by the degree (the number of edges connected to each node) to suppress the large difference in the output range of the convolution result of each node.

[0013] In the first-layer graph convolutional operation, using the parameter matrix W1 learned by machine learning so that the input feature number is 1433 and the output feature number is 16, the operation of X1 = Relu(A·X0·W1) is performed. Here, Relu() is a function for non-linear transformation that clips values that become negative values to 0, and the process of applying this function to transform values is called Relu processing.

[0014] In the second-layer graph convolutional operation, using the parameter matrix W2 learned by machine learning so that the input feature number is 16 and the output feature number is 7, the operation of X2 = A·X1·W2 is performed.

[0015] In the task of this embodiment, the seven output features respectively indicate the likelihood of the category of the paper, and the category corresponding to the feature corresponding to the highest likelihood among these likelihoods is classified as the category of the paper (node). Therefore, in the second-layer graph convolutional operation, non-linear transformation such as Relu() is unnecessary.

[0016] An example of the hardware configuration of an information processing device that functions as a graph convolution device, having an arithmetic unit for performing the above calculations, will be explained using the block diagram in Figure 1. Note that the hardware configuration shown in Figure 1 is merely one example of a hardware configuration of an information processing device applicable to a graph convolution device, and can be modified as appropriate.

[0017] The calculation unit 101 performs a graph convolution operation on a graph, which is data having a graph structure, by executing operations on each layer in the graph convolution layer. In this embodiment, since two graph convolution layers are used, the calculation unit 101 performs operations on each layer in the two graph convolution layers.

[0018] The CPU (Central Processing Unit) 103 executes various processes using computer programs and data stored in the RAM (Random Access Memory) 105. In doing so, the CPU 103 controls the operation of the entire information processing unit and executes or controls the various processes described as those performed by the information processing unit.

[0019] In this embodiment, the CPU 103 performs a process to extract classification categories from the feature data after performing calculations on the two-layer graph convolution. In this embodiment, the calculations on the two-layer graph convolution calculate seven features (likelihoods) that indicate which category each node (paper) should be classified into. The CPU 103 classifies the paper (node) into the category corresponding to the feature with the highest likelihood among these seven features.

[0020] The ROM (Read Only Memory) 104 stores configuration data for the information processing device, computer programs and data related to the startup of the information processing device, computer programs and data related to the basic operation of the information processing device, and so on. The ROM 104 also stores computer programs and data for causing the CPU 103 and arithmetic unit 101 to execute or control various processes described as being performed by the information processing device.

[0021] RAM 105 can be configured with a large-capacity DRAM (Dynamic Access Memory), etc. RAM 105 has an area for storing computer programs and data loaded from ROM 104. Furthermore, RAM 105 has a work area used by the CPU 103 and arithmetic unit 101 when executing various processes. In this way, RAM 105 can provide various areas as appropriate.

[0022] For example, RAM 105 functions as a data memory that holds feature data that becomes input when a graph convolution operation is performed in the arithmetic unit 101, as well as intermediate feature data and output data generated in the graph convolution operation. RAM 105 also functions as a coefficient memory that holds the "parameter matrix calculated in advance through learning" used in the graph convolution operation.

[0023] The user interface unit 106 is a user interface such as a keyboard or mouse, which allows the user to input various instructions and information to the information processing device through user operation. The user interface unit 106 may also include a display device having an LCD screen or a touch panel screen. For example, the user interface unit 106 can display the results of the paper category classification, a GUI (Graphical User Interface) for selecting the processing to be executed by the calculation unit 101, and so on. The user can operate such a GUI by operating the user interface unit 106.

[0024] The arithmetic unit 101, CPU 103, ROM 104, RAM 105, and user interface unit 106 are all connected to the system bus 107. Information processing devices with this hardware configuration can be applied to computer devices such as PCs (Personal Computers), smartphones, and tablet devices.

[0025] Next, an example of the hardware configuration of the calculation unit 101 described above will be explained using the block diagram in Figure 2. The reading unit 201 reads multiple feature data from the feature matrix stored in the data memory of RAM 105 and reads the graph edge weight list. The reading unit 202 reads the parameter matrix, which has been pre-calculated by learning, from the coefficient memory of RAM 105.

[0026] The sum-of-products unit 203 performs a sum-of-products operation on the feature data read by the reading unit 201 and the coefficient data in the parameter matrix read by the reading unit 202.

[0027] The duplication unit 204 repeatedly outputs the result of the sum-of-accumulate operation performed by the sum-of-accumulate operation unit 203 (sum-of-accumulate operation result) to the post-processing unit 205. The post-processing unit 205 performs processing such as nonlinear transformation using the ReLU function described above, as part of the graph convolution operation. The write unit 206 writes the processing result from the post-processing unit 205 to the data memory in the RAM 105. The control unit 207 controls the operation of the entire calculation unit 101 and controls it so that data to be processed does not accumulate and degrade performance.

[0028] Next, we will explain the processes performed by the information processing device according to the flowchart in Figure 3. First, we will explain the preparatory processes before inference, according to the flowchart in Figure 3(a).

[0029] In step S300, CPU 103 generates an edge weight list from the adjacency matrix. An example of the process for generating an edge weight list from the adjacency matrix is ​​explained using Figure 5.

[0030] Generally, the adjacency matrix of a graph has many elements with a value of 0, and in Figure 5, only the elements shown by the black rectangles have non-zero edge weights. Also, in an adjacency matrix with a self-loop, non-zero elements appear in the diagonal elements. An edge weight list is generated from this adjacency matrix. The edge weight list has a structure in which elements corresponding to non-zero edge weights appear for each source node. Each element is a pair of a non-zero edge weight and a destination node. This makes it possible to sequentially access only destination nodes that have a non-zero edge weight (i.e., nodes with a connection relationship) from the source node. The higher the sparsity (proportion of zero elements) of the adjacency matrix, the more the data size can be compressed. In this embodiment, for the Cora dataset, the adjacency matrix has 2708 x 2708 elements, while the edge weight list has 13556 elements (5429 x 2 + 2708 edges).

[0031] Note that the process in step S300 only needs to be performed once if the adjacency matrix is ​​predetermined, and it can be executed by the CPU 103 or by another device. The generated edge weight list is stored in the data memory in RAM 105 before the inference is executed.

[0032] Next, the operation in a two-layer graph convolution layer (two-layer convolution operation) will be explained according to the flowchart in Figure 3(b). In the flowchart in Figure 3(b), the product is calculated sequentially starting from step S301. When performing a graph convolution operation, the product of three matrices—the adjacency matrix, the feature matrix, and the parameter matrix—is calculated, but there are two ways to perform this calculation. One method is to first calculate the product of the first two matrices and then calculate the product of the result of that calculation with the last matrix. The other method is to first calculate the product of the last two matrices and then calculate the product of the result of that calculation with the first matrix.

[0033] Figure 4 illustrates the difference in the number of multiplications depending on the order of matrix calculations when performing a first-layer graph convolution operation on the Cora dataset. Due to the difference between the number of input features in the feature matrix (1433 in the first layer) and the number of output features in the graph convolution operation (16 in the first layer), the procedure in Figure 4(b), which first calculates the product of the feature matrix and the parameter matrix and then the product of the adjacency matrix, is computationally more efficient than the procedure in Figure 4(a), which first calculates the product of the adjacency matrix and the feature matrix and then the product of the parameter matrix. Therefore, in this embodiment as well, the product calculation is performed according to the procedure in Figure 4(b).

[0034] Furthermore, in the second layer of graph convolution, since the number of input features is 16 and the number of output features is 7, performing the calculations in the same order as in Figure 4(b) is computationally more efficient. In addition, in this embodiment, the graph convolution is performed using the edge weight list generated in step S300 instead of the adjacency matrix. This does not change the order of multiplications due to the order of matrix calculations, and the same calculation results as when using the adjacency matrix are obtained.

[0035] In step S301, the sum-of-products unit 203 calculates the product of the feature matrix X0 and the parameter matrix W1. The calculation of the product of the feature matrix and the parameter matrix will be explained using Figure 6(a).

[0036] First, the reading unit 201 reads one feature data from the feature matrix X. Next, the reading unit 202 reads multiple coefficient data corresponding to the feature data from the parameter matrix W. The number of coefficient data to be read is the same as the number of parallel calculations performed by the sum-of-accumulate unit 203, and in this embodiment, an example of reading four at a time is described. Since this is a matrix calculation, in order to calculate the elements of the output intermediate matrix XW, the feature matrix advances one column horizontally, and the parameter matrix advances vertically while reading multiple sets of four consecutive coefficient data horizontally.

[0037] The multiply-accumulate unit 203 performs multiplication and accumulation (i.e., multiply-accumulate operation) using the four coefficient data and the feature data. When the feature data reaches the right end of the feature matrix and the coefficient data reaches the bottom end of the parameter matrix, the calculation is completed for the four elements of the intermediate matrix, and the multiply-accumulate unit 203 outputs the multiply-accumulate result ("each element of the intermediate matrix") to the replication unit 204.

[0038] The duplication unit 204 outputs the sum-of-products calculation result to the post-processing unit 205, repeating the process four times, corresponding to the number of parallel processes. The post-processing unit 205 sequentially selects each sum-of-products calculation result output from the duplication unit 204 and outputs the selected sum-of-products calculation result to the subsequent write unit 206.

[0039] The writing unit 206 completes the intermediate matrix XW by setting the sum-of-products operation results (elements of the intermediate matrix) output from the post-processing unit 205 to the corresponding elements in the intermediate matrix, and then stores the intermediate matrix XW in the data memory.

[0040] The multiply-accumulate unit 203 has a large number of parallel operations, but it takes the number of cycles equal to the width of the feature matrix (1433 in the case of the Cora dataset in the first layer) to calculate one matrix element. On the other hand, the post-processing unit 205 takes the same number of processing cycles as the multiply-accumulate unit 203, so in this embodiment it takes 4 cycles. The duplication unit 204 holds multiple multiply-accumulate operation results and repeatedly outputs these multiply-accumulate operation results, so that even if the number of parallel operations of the post-processing unit 205 and the multiply-accumulate unit 203 are different, the data flows smoothly and each unit behaves to achieve its maximum processing performance.

[0041] In step S302, the sum-of-products unit 203 calculates the product of the intermediate matrix X0W1 generated in step S301 and the edge weight list, and the post-processing unit 205 applies a ReLU process to each element (each feature) in the matrix obtained by the product.

[0042] The method for calculating the product of the intermediate matrix and the edge weight list will be explained using Figure 6(b). First, the reading unit 201 reads one element from the edge weight list, which is obtained by compressing the adjacency matrix. The elements of the edge weight list contain edge weight data and information indicating the destination node (destination node information). Of this information, the edge weight data is output to the sum-of-products unit 203. The destination node information is output to the reading unit 202.

[0043] The reading unit 202 uses the destination node information as the reading destination information to generate address information (access destination) for the intermediate matrix XW to be accessed. Based on this address information, multiple data points (feature data) of the intermediate matrix in the data memory can be read in a continuous horizontal direction. Therefore, although access to the intermediate matrix is ​​random, data points that are continuous horizontally can be acquired efficiently. The multiple data points read are output to the multiply-accumulate unit 203. From here on, multiplication and accumulation are performed between the corresponding intermediate matrix data and the edge weight data, as described in step S301. When the list for one source node in the edge weight list reaches the end, the multiply-accumulate operation is completed for a number of elements corresponding to the parallelism of the multiply-accumulate unit 203. If all elements cannot be generated in a single operation for one source node, the reading of the edge weight list, access to the intermediate matrix, and multiply-accumulate operation are repeated.

[0044] The replication unit 204 repeatedly outputs the sum-of-accumulate calculation results. Since the length of the edge weight list differs for each connected source node, the number of cycles required for processing in the sum-of-accumulate calculation unit 203 varies, but the replication unit 204 buffers the performance difference with the post-processing unit 205. When the sum-of-accumulate calculation results are output from the replication unit 204, the post-processing unit 205 sequentially executes the ReLU process.

[0045] Subsequently, the writing unit 206 completes the feature matrix X1 by setting the sum-of-products operation results (elements of the feature matrix X1 representing the results of the graph convolution operation of the first layer) output from the post-processing unit 205 to the corresponding elements in the feature matrix X1.

[0046] In step S303, the sum-of-products unit 203 calculates the product of the feature matrix X1 and the parameter matrix W2 in the same manner as in step S301. In step S304, the sum-of-products unit 203 calculates (generates) a feature matrix X2 representing the result of the second layer graph convolution operation by multiplying the intermediate matrix X1W2 generated in step S303 with the edge weight list, using the same process as in step S302. The write unit 206 then writes the feature matrix X2 to the data memory.

[0047] In step S305, as described above, CPU 103 classifies the categories corresponding to the features with the highest likelihood among the likelihoods of each element of the feature matrix X2 generated in step S304 as the categories of papers (nodes).

[0048] Thus, according to this embodiment, efficient graph convolution operations can be achieved by eliminating redundant calculations and performing data access based on the destination information of the edge weight list, without requiring a large-capacity adjacency matrix. Furthermore, by efficiently executing graph convolution operations in this way, the power consumption and processing time of the calculation unit that performs the graph convolution operations can be kept to the absolute minimum.

[0049] Furthermore, in this embodiment, we have described an example where the number of parallel operations for the sum-of-accumulate operation unit is set to 4, and the data is read accordingly. However, the processing time for the graph convolution operation can be shortened in proportion to the number of parallel operations for the sum-of-accumulate operation unit.

[0050] Furthermore, the replication unit can absorb the difference in processing time between the sum-of-accumulate unit and the post-processing unit, thereby maximizing the processing performance of each. In particular, the graph structure allows for efficient processing even when the length of the edge weight list differs.

[0051] [Second Embodiment] In this embodiment, the differences from the first embodiment will be explained, and unless otherwise specified below, it will be assumed to be the same as the first embodiment. In this embodiment, a case will be described in which graph convolution is performed using a graph generated by learning of these feature vectors in a CNN feature map where a feature vector is defined for each grid point, such as in image data.

[0052] While conventional CNNs perform convolution in both the spatial and feature directions on image data, research into graph convolution has led to efforts to reduce computational complexity while maintaining task accuracy by structuring both the spatial and feature directions into graphs. In this embodiment, we will explain the graph structuring in the feature direction and the graph convolution operation for one feature vector 702 (number of features = 8: ch = 0~7) from the feature map 701 shown in Figure 7.

[0053] In the graph convolution operation performed in this embodiment, each element (each feature data) in the feature vector 702 is treated as a single node (with 1 feature), and edges are constructed to connect the feature data, thereby defining it as a graph. The adjacency matrix composed of the feature data represents the influence between the feature data, but it is preferable to calculate the appropriate influence between the feature data through learning, according to the task to be performed.

[0054] Figure 8 shows the graph and adjacency matrix for the 702 feature vectors, with all feature data interconnected, and the pruned versions of these. The adjacency matrix on the left in Figure 8, which was trained with all feature data interconnected, may contain redundant information depending on the actual task, and pruning of the adjacency matrix is ​​performed, similar to pruning of CNN weights. Pruning can be achieved, for example, by setting the weight of edges with an absolute value smaller than a predetermined threshold to 0. Edges with an edge weight of 0 can be considered deleted edges, resulting in a graph with sparse connections, as shown in the graph on the right in Figure 8.

[0055] Figure 9 shows the network structure of the graph convolution operation in this embodiment. Such a network structure can be implemented by the CPU 103 and the arithmetic unit 101. The encoder 901 performs encoding (graph convolution) on a feature vector 702 with 8 features to obtain an 8x8 encoded result (feature matrix). The decoder 902 performs decoding (graph convolution) on the encoded result from the encoder 901 to obtain a feature vector with 8 features.

[0056] This two-layer graph convolution operation allows a convolution operation to be performed in the feature direction for each feature vector of the feature map 701. In this embodiment, the graph convolution operations are independent of each other between different feature vectors of the feature map 701, so they can be processed in parallel.

[0057] Figure 10 shows the computational complexity depending on the order of matrix operations. As shown in Figures 10(a) and (b), in encoding by encoder 901, it is more efficient to perform the operation (product) of the adjacency matrix and feature vector first. Similarly, as shown in Figures 10(c) and (d), in decoding by decoder 902, it is more efficient to perform the operation (product) of the feature matrix and parameter matrix first. The most efficient order of operations is determined by the relative numbers of input and output features.

[0058] Next, the processing performed by the information processing device will be explained according to the flowchart in Figure 11. First, the preparatory processing before inference will be explained according to the flowchart in Figure 11(a). In step S1000, the CPU 103 generates an edge weight list from the adjacency matrix in the same manner as in step S300 above.

[0059] Next, the operation of the encoder 901 and decoder 902 described above will be explained according to the flowchart in Figure 11(b).

[0060] In step S1001, the encoder 901 calculates the product of the edge weight list and the feature vector, and then calculates the product of the result of this product and the parameter matrix. The operation of the calculation unit 101 in this step will be explained using Figure 12(a). In encoding by the encoder 901, the number of output features is greater than the number of input features, so it is advantageous to calculate the product of the edge weight list and the feature vector first, and therefore this calculation is performed first.

[0061] The reading unit 201 reads elements one by one from the edge weight list and reads the edge weight and the destination channel, which will be the reading destination information for the reading unit 202, from each element. The destination channel is output to the reading unit 202, and the address to be read from the feature vector is determined. Feature data is simultaneously read from multiple feature vectors that are different in the spatial direction of the feature map 701. The number of feature data to be read simultaneously depends on the number of parallel operations of the multiply-accumulate unit 203. The multiply-accumulate unit 203 performs a multiply-accumulate operation between the edge weight and the corresponding feature data. For each source channel, when the end of the edge weight list is reached, the multiply-accumulate operation is completed and output to the duplication unit 204. The duplication unit 204 duplicates the same value as many times as there are channels in the parameter matrix (8 times in this embodiment) and outputs it to the post-processing unit 205. In this embodiment, the post-processing unit 205 is configured to multiply each sum-of-products operation result by one coefficient data of the parameter matrix, and is configured to calculate the product of the edge weight list and the feature vector with the parameter matrix. The encoding result calculated by the post-processing unit 205 is written to a desired address by the writing unit 206. In this embodiment, the calculation unit 101 processes the data by copying the intermediate data using the copying unit 204, making it possible to process the data without writing the intermediate data of the encoding by the encoder 901 to an external source. Alternatively, the parameter matrix may not be stored in the post-processing unit 205 but may be read from an external coefficient memory.

[0062] In step S1002, the decoder 902 calculates the product of the feature matrix and the parameter matrix to obtain an intermediate matrix for decoding. Since calculating the product of the feature matrix and the parameter matrix first is computationally advantageous for decoding by the decoder 902, the processing is carried out in that order. The operation of the arithmetic unit 101 in this step will be explained using Figure 12(b).

[0063] First, the reading unit 201 reads one parameter matrix from the coefficient memory. The reading unit 202 reads the corresponding feature data from multiple encoding results in space. The result calculated by the multiply-accumulate unit 203 is written to the desired intermediate matrix by the writing unit 206 via the replication unit 204 and the post-processing unit 205.

[0064] In step S1003, the product of the edge weight list and the intermediate matrix calculated in step S1002 is calculated for the second half of the decoding process by the decoder 902. As shown in Figure 12(c), the reading unit 202 reads the intermediate matrix based on the reading destination information on the edge weight list, and the result of the sum-of-products operation is written out by the writing unit 206. This makes it possible to obtain the decoding results by the decoder 902 in multiple feature vectors simultaneously.

[0065] In this way, it is possible to switch and execute the processing in an appropriate order depending on the relative sizes of the input and output features of the matrix used for graph convolution. The control unit 207 may switch the processing flow in a predetermined order based on the relative sizes of the input and output features, or it may automatically detect the number of input and output features and switch the processing flow.

[0066] Thus, according to this embodiment, efficient computation becomes possible by processing in an appropriate order according to the number of input and output features of the graph convolution operation. Furthermore, conventionally, data structures that perform random access, such as edge weight lists, require address decoding for each parallel read, making it difficult to increase the degree of parallelism. In contrast, when processing different feature graphs in space in parallel, as in this embodiment, if the feature data accessed at one time is arranged consecutively, efficient data transfer is possible, and the degree of parallelism can be increased.

[0067] [Third Embodiment] In this embodiment, the differences from the second embodiment will be described, and unless otherwise specified below, it will be assumed to be the same as the second embodiment. In this embodiment, an example will be described in which, when performing graph convolution operations hierarchically, the graph structure to be applied is generated from the feature matrix, which is the output result of the graph convolution operation of the previous layer.

[0068] Research is being conducted to reduce computational complexity by performing spatial and feature-direction graph structuring on structured data such as image data. Graph structuring can be pre-calculated using deep learning techniques, or it can be generated from intermediate feature maps where hierarchical graph convolution is performed. This embodiment demonstrates that an edge weight list representing the graph structure can be efficiently generated from such intermediate feature maps.

[0069] Figure 13 is a conceptual diagram illustrating the method for generating the edge weight list. The edge weight list is generated from the NxM feature matrix (N, M: natural numbers) which is the output result of the previous layer, and the MxN graph structure definition parameter matrix which is trained in advance. Here, the natural number N represents the number of nodes in the target graph structure, and in the case of image data, this corresponds to the number of pixels or the number of channels as shown in the second embodiment. The natural number M represents the dimensionality of the feature vector for each node. The edge weight list for the next layer is generated by operations on these matrices. The edge weights that make up the elements of the edge weight list are calculated by a convolution operation of the pre-trained parameter matrix and the feature matrix of the previous layer.

[0070] In this embodiment, the edge weight list is calculated by the calculation unit 101. The operation of the calculation unit 101 will be explained using Figure 14. The operation of this embodiment shown in Figure 14 is similar to the calculation method of the product of the feature matrix and the parameter matrix in the first embodiment described in Figure 6(a). The convolution operation reads out the data from the feature matrix one row at a time, reads out the data from the parameter matrix in the horizontal direction of four consecutive coefficient data to the bottom end, and performs cumulative addition. The duplication unit 204 repeatedly outputs the four cumulative sum values, and the post-processing unit 205 performs ReLU processing. Furthermore, the write unit 206 does not write if the result of the ReLU processing is 0 (if the sum-of-products calculation result is negative), and if the result of the ReLU processing is a non-zero value, it outputs the ReLU processing result along with the coordinate value indicating the position of the horizontal axis N on the parameter matrix that indicates the connected node of the graph structure. By repeating this operation, it is possible to generate an edge weight list. The edge weight list generated here will be used in the graph convolution operation of the next level, so it is desirable that each element of the edge weight list be written to a contiguous memory area. Therefore, after writing up to four elements, the feature matrix is ​​read again, one by one, and the parameter matrix is ​​read and processed, with four horizontally adjacent coefficient data points being read down to the bottom. This process is repeated. Once the parameter matrix coefficient data sets have been processed to the rightmost end, the row being read from the feature matrix is ​​moved down one level, and the above is repeated until all elements of the edge weight list are written to contiguous memory areas.

[0071] As described above, by defining nodes where the edge weights become negative due to the ReLU process as unconnected nodes in the graph structure, and by learning a parameter matrix for defining the graph structure, the graph structure can be generated from the feature matrix.

[0072] Furthermore, by writing only non-zero edge weights to the edge weight list, it may be possible to generate a graph structure with less memory capacity than an adjacency matrix, which always requires capacity equal to the number of nodes multiplied by the number of nodes. In addition, reducing the number of write operations can reduce power consumption.

[0073] In the embodiments described above, we explained a case in which all of the functional units in Figure 2 are implemented in hardware, but some of the functional units may be implemented in software. For example, the reading unit 201, reading unit 202, multiply-accumulate unit 203, duplication unit 204, post-processing unit 205, and writing unit 206 may be implemented in software (computer program). In this case, the CPU 103 and control unit 207 execute this computer program to realize the functions of the corresponding functional units.

[0074] The numerical values, processing timing, processing order, processing entity, data (information) structure / acquisition method / destination / source / storage location, etc., used in the above embodiment are given as examples for the purpose of providing a concrete explanation, and are not intended to limit the scope to such examples.

[0075] Furthermore, some or all of the embodiments described above may be used in appropriate combinations. Alternatively, some or all of the embodiments described above may be used selectively.

[0076] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0077] The inventions described herein include the following information processing devices, information processing methods, and computer programs. (Item 1) An information processing device that performs graph convolution operations on a graph, A first acquisition means for obtaining the weights and destinations of nodes in the graph from a list containing information of the portion where the values ​​representing the connection relationships are non-zero in an adjacency matrix representing the connection relationships of nodes in the graph, the list having the weights of the nodes in the graph and the destinations of those nodes. A second acquisition means that acquires data to be input to the arithmetic unit from the data to be calculated, based on the connection destination, A calculation means that uses the calculation unit to perform calculations using the data acquired by the second acquisition means and the node weights acquired by the first acquisition means. An information processing device characterized by comprising: (Item 2) The information processing apparatus according to item 1, characterized in that the second acquisition means generates address information of data to be input to the arithmetic unit from the data to be calculated based on the connection destination, and acquires data to be input to the arithmetic unit based on the address information. (Item 3) The information processing device according to item 1, characterized in that the data to be processed is the result of a sum-of-products operation between the feature data of the nodes in the graph and the learned parameters. (Item 4) If the number of channels in the feature data is greater than the number of channels in the calculation result of the calculation means, the data to be calculated is the result of a sum-of-products operation between the feature data of the graph nodes and the learned parameters. If the number of channels in the feature data is less than the number of channels in the calculation result of the calculation means, the data to be calculated is the feature data of the node in the graph. The information processing device described in item 3, characterized by the features described herein. (Item 5) The information processing apparatus according to item 1, characterized in that the calculation means applies a ReLU function to the result of a sum-of-products operation between the data acquired by the second acquisition means and the node weights acquired by the first acquisition means. (Item 6) The information processing device according to item 5, characterized in that the calculation means applies the ReLU function to the result of the sum-of-products operation on the first layer in a two-layer graph convolutional layer. (Item 7) moreover, The information processing apparatus according to item 1, characterized by comprising a classification means for performing categorical classification of the graph based on the results of calculations performed by the calculation means. (Item 8) The information processing apparatus according to item 7, characterized in that the classification means performs the category classification based on the result of calculations performed by the calculation means on the second layer of a two-layer graph convolutional layer. (Item 9) moreover, The information processing apparatus according to item 1, characterized by comprising a duplication means for storing the sum of products result from the calculation means and repeatedly outputting the same result. (Item 10) The information processing apparatus according to item 1, characterized in that the calculation means processes multiple feature vectors in different spaces of the feature map in parallel. (Item 11) The information processing device according to item 1, characterized in that the first acquisition means excludes elements in the graph structure that have no connections between nodes, and acquires a list structure in which elements are paired, consisting of the weight of a node in the graph that indicates the influence between nodes, and the destination of the data that convolves the weight from the source node to the destination node. (Item 12) The information processing device according to item 11, characterized in that the first acquisition means reads the list from a contiguous area in memory. (Item 13) The information processing apparatus according to item 1, characterized in that the calculation means performs a convolution operation from the data acquired by the second acquisition means and the node weights acquired by the first acquisition means, and outputs the weights of the non-zero graph and the data connection destinations corresponding to the weights. (Item 14) The information processing apparatus according to item 13, characterized in that the calculation means writes elements, each consisting of the weight of a node in the graph and the destination of the data corresponding to that weight, to a contiguous area in memory. (Item 15) An information processing method performed by an information processing device that performs graph convolution operations on a graph, The first acquisition means of the information processing device includes a first acquisition step of acquiring the weights and destinations of the nodes of the graph from a list in which information is extracted from the portion of the adjacency matrix representing the connection relationships of the nodes of the graph in which the values ​​representing the connection relationships are non-zero, the list having the weights of the nodes of the graph and the destinations of the nodes; The second acquisition means of the information processing device includes a second acquisition step of acquiring data to be input to the arithmetic unit from the data to be calculated, based on the connection destination, The calculation means of the information processing device performs a calculation using the calculation unit, with the calculation step being to perform calculations using the data acquired in the second acquisition step and the node weights acquired in the first acquisition step. An information processing method characterized by comprising: (Item 16) A computer program that causes a computer to function as one of the means of an information processing device described in any one of items 1 through 14.

[0078] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0079] 201: Read unit 202: Read unit 203: Multiply-accumulate unit 204: Copy unit 205: Post-processing unit 206: Write unit 207: Control unit

Claims

1. An information processing device that performs graph convolution operations on a graph, A first acquisition means for obtaining the weights and destinations of nodes in the graph from a list containing information of the portion where the values ​​representing the connection relationships are non-zero in an adjacency matrix representing the connection relationships of nodes in the graph, the list having the weights of the nodes in the graph and the destinations of those nodes. A second acquisition means for acquiring data to be input to the arithmetic unit from the data to be calculated, based on the connection destination, A calculation means that uses the calculation unit to perform calculations using the data acquired by the second acquisition means and the node weights acquired by the first acquisition means. An information processing device characterized by comprising:

2. The information processing apparatus according to claim 1, characterized in that the second acquisition means generates address information of data to be input to the arithmetic unit from the data to be calculated based on the connection destination, and acquires data to be input to the arithmetic unit based on the address information.

3. The information processing apparatus according to claim 1, characterized in that the data to be processed is the result of a sum-of-products operation between the feature data of the nodes of the graph and the learned parameters.

4. If the number of channels in the feature data is greater than the number of channels in the calculation result of the calculation means, the data to be calculated is the result of a sum-of-products operation between the feature data of the graph nodes and the learned parameters. If the number of channels in the feature data is less than the number of channels in the calculation result of the calculation means, the data to be calculated is the feature data of the node in the graph. The information processing apparatus according to claim 3.

5. The information processing apparatus according to claim 1, characterized in that the calculation means applies the Relu function to the result of a sum-of-products operation between the data acquired by the second acquisition means and the node weights acquired by the first acquisition means.

6. The information processing apparatus according to claim 5, characterized in that the calculation means applies the Relu function to the result of the sum-of-products operation on the first layer in the two-layer graph convolution layer.

7. moreover, The information processing apparatus according to claim 1, further comprising a classification means for performing categorical classification of the graph based on the results of calculations performed by the calculation means.

8. The information processing apparatus according to claim 7, characterized in that the classification means performs the category classification based on the result of calculations performed by the calculation means on the second layer of the two-layer graph convolutional layer.

9. moreover, The information processing apparatus according to claim 1, further comprising a duplication means for storing the result of the sum of products calculated by the calculation means and repeatedly outputting the same result.

10. The information processing apparatus according to claim 1, characterized in that the calculation means processes a plurality of feature vectors in different spaces of the feature map in parallel.

11. The information processing apparatus according to claim 1, characterized in that the first acquisition means excludes elements in the graph structure that have no connections between nodes, and acquires a list structure in which elements are set together, each containing the weight of a node in the graph that indicates the influence between nodes, and the destination of the data that convolves the weight from the source node to the destination node.

12. The information processing apparatus according to claim 11, characterized in that the first acquisition means reads the list from a contiguous area in memory.

13. The information processing apparatus according to claim 1, characterized in that the calculation means performs a convolution operation from the data acquired by the second acquisition means and the node weights acquired by the first acquisition means, and outputs the weights of the non-zero graph and the connection destination of the data corresponding to the weights.

14. The information processing apparatus according to claim 13, characterized in that the calculation means writes elements, each consisting of the weight of a node in the graph and the destination of the data corresponding to the weight, to a contiguous area in memory.

15. An information processing method performed by an information processing device that performs graph convolution operations on a graph, The first acquisition means of the information processing device includes a first acquisition step of acquiring the weights and destinations of the nodes of the graph from a list in which information is extracted from the portion of the adjacency matrix representing the connection relationships of the nodes of the graph in which the values ​​representing the connection relationships are non-zero, the list having the weights of the nodes of the graph and the destinations of the nodes. The second acquisition means of the information processing device includes a second acquisition step of acquiring data to be input to the arithmetic unit from the data to be calculated, based on the connection destination, The calculation means of the information processing device performs a calculation using the calculation unit, with the calculation step being to perform calculations using the data acquired in the second acquisition step and the node weights acquired in the first acquisition step. An information processing method characterized by comprising:

16. A computer program for causing a computer to function as one of the means of an information processing apparatus described in any one of claims 1 to 14.

Images