Learning device and learning method

Abstract

To increase the potential for reusing hardware resources. [Solution] A learning device comprising a control unit that performs learning using a C graph, which is a graph having a one-to-one correspondence between each node of a target network, which is a neural network to be pruned, and in which case the edge weights are 0 or 1, and updating an S graph, which is a graph having a one-to-one correspondence between each node of the target network, and in which case the edge weights are -1 or 1, and an M graph, which is a graph having a one-to-one correspondence between each node of the target network, and in which case the edge weights are positive integers, wherein the C graph is not updated during the learning process.

Description

[Technical Field]

[0001] The present invention relates to a learning device and a learning method. [Background technology]

[0002] Deep neural networks (DNNs) are known to be over-parameterized because a large portion of their weights can be pruned after training without affecting their accuracy. The sparsity introduced by pruning is commonly used by dedicated neural engines to compress model size through entropy coding and reduce computational cost through sparse computation. While the connection patterns in DNNs discovered through pruning were sufficient for efficient inference, it was generally believed that training them from scratch on their own was impossible.

[0003] However, in 2019, it was shown that an iterative process of size-based pruning and weight reinitialization could reveal efficient subnetworks that could be trained independently. More specifically, it was discovered that randomly initialized dense neural networks contain subnetworks (so-called "winning tickets") that, when trained independently, can achieve the same test accuracy as the original network. This discovery is known as the "Lottery Ticket Hypothesis" (LTH).

[0004] A further noteworthy discovery was made in 2021 (see Non-Patent Document 1). This discovery was that over-parameterized DNNs already contain highly efficient subnetworks, even when randomly initialized. More specifically, it was found that randomly initialized dense neural networks contain subnetworks (so-called "strong tickets") that can achieve high test accuracy without weight training.

[0005] Furthermore, it was shown that these subnetworks can be obtained simply by training a binary pruning mask, known as a supermask, without learning the weights. These findings regarding LTHs are known as the "Strong Lottery Ticket Hypothesis" (SLTH).

[0006] SLTH is not merely a novel concept; for example, neural engine designers have proposed efficient accelerator designs based on SLTH, as well as improved supermasks to enhance the inference accuracy, density, and model size of SLTH (see Non-Patent Documents 2-5). [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari, What's hidden in a randomly weighted neural network?, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2021. [Non-Patent Document 2] Nils Koster, Oliver Grothe, and Achim Rettinger, Signing the supermask: Keep, hide, invert, Proceedings of the International Conference on Learning Representations, 2022. [Non-Patent Document 3] Yasuyuki Okoshi, Angel Lopez Garcia-Arias, Kazutoshi Hirose, Kota Ando, ​​Kazushi Kawamura, Thiem Van Chu, Masato Motomura, and Jaehoon Yu, Multicoated supermasks enhance hidden networks, Proceedings of the International Conference on Machine Learning, 2022. [Non-Patent Document 4] Angel Lopez Garcia-Arias, Yasuyuki Okoshi, Masanori Hashimoto, Masato Motomura, and Jaehoon Yu, Recurrent residual networks contain stronger lottery tickets, IEEE Access, 2023. [Non-Patent Document 5] Jiale Yan, Hiroaki Ito, Angel Lopez Garcia-Arias, Yasuyuki Okoshi, Hikari Otsuka, Kazushi Kawamura, Thiem Van Chu, and Masato Motomura, Multicoated and folded graph neural networks with strong lottery tickets, Proceedings of the Learning on Graphs Conference, 2024. [Overview of the Initiative] [Problems that the invention aims to solve]

[0008] However, some of these improved supermasks completely sacrifice strong ticket randomness, generating models composed only of fully optimized parameters, thus losing the opportunity to leverage randomness. Furthermore, it is unclear how to leverage the optimal connectivity generated by SLTH in the various analog computing infrastructures being restored for DNN acceleration, as the hardware topology often determines connectivity at manufacturing time. Therefore, SLTH needs to be adapted to accommodate a wider range of randomness and looser connectivity constraints.

[0009] Let's explain the problem with a more concrete example. For simplicity, we will use an example where an optical system such as silicon photonics represents a neural network, such as an inference model. When a neural network is represented as a graph, as a result of designs based on SLTH proposed so far, there may be edges with weights of 0. An edge with a weight of 0 means that the edge is unnecessary. Since edges are realized as waveguides in optical systems, in some cases, the waveguide represented by an edge designed to have a weight of 0 is not implemented at all.

[0010] As a result, with optical systems obtained through such designs, even if the inference model for which the design was created becomes unnecessary and an attempt is made to implement a new inference model in that optical system, it may not be possible. In other words, it can be difficult to reuse an optical system that has been used to implement a neural network for the implementation of a new neural network.

[0011] This is because, according to previously proposed techniques based on SLTH, connectivity, i.e., the presence or absence of edges between each node (which is indicated by the supermask), is inferred through learning. Therefore, the more places where waveguides are not implemented, the less appropriate the supermask optimization becomes when learning is performed assuming implementation in that optical system. As a result, there are situations where it is difficult to use an optical system that has already been implemented for implementation in other inference models.

[0012] Furthermore, it is generally difficult to add waveguides to an optical system once it has been implemented, especially in areas where waveguides were not previously present. Once an optical system is implemented, the only thing that can be done is to change its parameters by applying heat or electromagnetic waves; adding structures such as waveguides is difficult.

[0013] Thus, once a neural network designed using SLTH technology is implemented using hardware such as optics, it can be difficult to reuse that hardware to implement other neural networks. This challenge is not limited to SLTH-based technologies but is common to pruning techniques in general.

[0014] In view of the above circumstances, the present invention aims to provide a technology that enhances the possibility of reusing hardware resources. [Means for solving the problem]

[0015] One aspect of the present invention is a learning method using a C graph, which is a graph having a one-to-one correspondence between each node of a target network, which is a neural network to be pruned, and the edge weights are 0 or 1. The learning method includes a control unit that updates an S graph, which is a graph having a one-to-one correspondence between each node of the target network, and the edge weights are -1 or 1, and an M graph, which is a graph having a one-to-one correspondence between each node of the target network, and the edge weights are positive integers. The learning method includes a network execution process, an influence update process, an SM graph update process, and a subset network acquisition process, and the network execution process involves the target network, the C graph, the S graph, and the M graph. The learning device is a process that executes a subset network, which is a neural network obtained based on the above, the influence update process is a process that updates edge influence, which is a value assigned to each edge of the target network and indicates the degree of influence that the assigned edge is estimated to have on the accuracy of inference by the subset network, based on the inference results of the subset network, the SM graph update process is a process that updates the S graph and the M graph based on each of the influences after the influence update process, the subset network acquisition process acquires a subset network based on the target network, the C graph, the S graph and the M graph, and the C graph is not updated during learning.

[0016] One aspect of the present invention is learning using a C-graph having nodes corresponding one-to-one to each node of a target neural network that is a neural network to be pruned, wherein the weight of an edge is 0 or 1, the learning being: learning to update an S-graph having nodes corresponding one-to-one to each node of the target neural network, wherein the weight of an edge is -1 or 1, and an M-graph having nodes corresponding one-to-one to each node of the target neural network, wherein the weight of an edge is a positive integer; a control unit that performs the learning; in the learning, network execution processing, influence degree update processing, SM-graph update processing, and partial network acquisition processing are executed; the network execution processing is processing for executing a partial network that is a neural network obtained based on the target neural network, the C-graph, the S-graph, and the M-graph; the influence degree update processing is processing for updating, based on the inference result of the partial network, an edge influence degree that is a value given to each edge of the target neural network and indicates the influence degree estimated to be given by the edge at the destination to the accuracy of inference by the partial network; the SM-graph update processing is processing for updating the S-graph and the M-graph based on each of the influence degrees updated by the influence degree update processing; in the partial network acquisition processing, a partial network is acquired based on the target neural network, the C-graph, the S-graph, and the M-graph; and in the learning, the C-graph is not updated; a learning method executed by a learning device, the learning method having a learning step in which the control unit performs the learning.

Effect of the Invention

[0017] According to the present invention, it is possible to increase the possibility of reusing hardware resources.

Brief Description of the Drawings

[0018] [Figure 1] An explanatory diagram for explaining the learning device of the embodiment. [Figure 2] A diagram showing an example of the hardware configuration of the learning device in the embodiment. [Figure 3] A flowchart showing an example of the processing flow performed by the learning device in the embodiment. [Figure 4] A diagram showing an example of the hardware configuration of the inference device in the embodiment. [Figure 5] A flowchart showing an example of the processing flow performed by the inference device in the embodiment. [Modes for carrying out the invention]

[0019] (Embodiment) <Learning device 1> Figure 1 is an explanatory diagram illustrating the learning device 1 of the embodiment. Incidentally, it is well known that the computation process of a neural network is generally represented using a graph consisting of nodes and edges (i.e., a graph in graph theory), where nodes represent neurons and edges represent signals. Therefore, in the following explanation, when we refer to a node of a neural network, we mean a node when the neural network is represented as a graph, and when we refer to an edge of a neural network, we mean an edge when the neural network is represented as a graph.

[0020] The learning device 1 includes a control unit 11 which has a processor 91 such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or NPU (Neural Network Processing Unit) connected by a bus, and memory 92, and executes a program.

[0021] The control unit 11, for example, executes a learning process. The learning process is a process that performs learning using a C graph, and in this learning process, a network execution process, an influence update process, an SM graph update process, and a partial network acquisition process are executed.

[0022] A C-graph is a graph in which each node of the target neural network (the neural network to be pruned) has a one-to-one correspondence with a node, and the edge weights are either 0 or 1. In graph theory, edges are called "eda" in Japanese, so the above pruning means that some of the edges included in the target neural network are removed. The target of pruning is the neural network that is subjected to such pruning. In the example in Figure 1, the neural network represented by image G101 is an example of a target network.

[0023] As defined above, a C-graph is a type of supermask. Based on the definition of a C-graph, it can be said that a C-graph is a graph that shows the connectivity of a neural network.

[0024] The network execution process is the process of executing a subset network. A subset network is a neural network obtained based on the target network, the C graph, the S graph, and the M graph, according to predetermined rules (hereinafter referred to as "subset network acquisition rules"). In the example in Figure 1, the neural network represented by image G103 is an example of a subset network.

[0025] An S-graph is a graph in which each node of the target network has a one-to-one correspondence with a node, and the edge weights are -1 or 1. An M-graph is a graph in which each node of the target network has a one-to-one correspondence with a node, and the edge weights are positive integers. Note that an S-graph is a type of supermask, for example, the difference between a Signed Supermask graph and a C-graph disclosed in Non-Patent Document 2. An M-graph is also a type of supermask, for example, the difference between a Multicoated Supermask graph and a C-graph disclosed in Non-Patent Document 3.

[0026] A subset network is, for example, a subnetwork included in the target network, which is obtained based on the C graph, S graph, and M graph. Such a subnetwork consists only of nodes and edges of the target network whose influence, indicated by, for example, the edge influence index, is equal to or greater than a predetermined influence index, and the nodes connected to those edges. Such a subnetwork may also consist only of nodes and edges of the target network that satisfy the condition, for example, that the influence level indicated by, for example, the edge influence index, is no more than or equal to the Pth-th-highest (where P is a predetermined constant of 1 or more), and the nodes connected to those edges.

[0027] Edge impact is a value assigned to each edge of the target network, indicating the estimated impact that the assigned edge has on the accuracy of inference by the subsystem. Edge impact may be defined such that a larger value indicates a greater impact, or a smaller value indicates a greater impact; in either case, it is a value that indicates the magnitude of the impact.

[0028] Let's explain edge impact using a concrete example. For instance, suppose there is an edge called edge E12 in the target network, and we focus on this edge E12. Let's assume that the extent to which edge E12 affects the accuracy of inference by the subset network is estimated using a certain technique. Furthermore, let's assume that the estimated value is "20". In this case, edge E12 is assigned the value "20". This assigned value is the edge impact for edge E12. Edge impact is not assigned only to edge E12. Edge impact is also assigned to other edges in the target network. Edge impact is a value like this.

[0029] Edge influence is a value that can be updated during the training process and is not necessarily a fixed value. The process of updating edge influence is called influence update processing. In the example in Figure 1, the graph for image G104 has the same nodes as the graph for image G101, and the edge weights represent the edge influence.

[0030] The influence update process updates the edge influence based on the results of the subsystem's inference. The update is performed according to a predetermined rule to improve the accuracy of the subsystem's inference. The predetermined rule can be any rule that improves the accuracy of the subsystem's inference, for example, a rule that updates the error obtained by a loss function called cross-entropy so that the error is reduced by a training method called backpropagation.

[0031] The SM graph update process updates the S graph and M graph based on the updated edge influence levels from the influence update process.

[0032] The partial network acquisition process is a process that obtains a network based on the target network, the C graph, the S graph, and the M graph, according to predetermined rules. Therefore, in the partial network acquisition process executed after the SM graph update process, the partial network is updated based on the target network, the C graph, and the S graph and M graph after the SM graph update process.

[0033] Thus, the learning process updates the S-graph and M-graph. On the other hand, the C-graph is not updated during the learning process. In fact, because the C-graph is not updated, the possibility of reusing hardware resources can be increased.

[0034] Furthermore, in the learning process, the weights of the target network may be values ​​that have been randomly set in advance before the learning process is executed. In other words, the neural network to be pruned may be a neural network whose weights are predetermined random values. Once the weight values ​​are set, they will not be changed from their initial values ​​during the learning process thereafter. Such weight setting (i.e., weight initialization) can be done using any technique that allows the value of each weight to be set randomly, for example, the KN (Kaiming Normal) initialization method or the SC (Signed Kaiming Constant) initialization method.

[0035] The above learning process will continue until a predetermined condition for terminating the learning process (hereinafter referred to as the "learning termination condition") is met. The learning termination condition may be, for example, that the change in the subset network obtained after updating the edge influence is smaller than a predetermined change compared to before the update. The learning termination condition may also be, for example, that the change in the inference accuracy of the subset network obtained after updating the edge influence is smaller than a predetermined change compared to before the update. The learning termination condition may also be, for example, that the edge influence has been updated a predetermined number of times.

[0036] In the above learning process, the edge influence is updated to improve the accuracy of the inference of the subset network. The accuracy of the inference of the subset network may be represented by a predetermined loss. For example, labeled data may be used in the learning process. In such a case, the data to be labeled in the labeled data is input to the subset network, and the subset network performs inference based on the input data. In the learning process, the edge influence may be updated to minimize the difference between the result of the inference of this subset network and the labels of the labeled data. In this case, the predetermined loss is the difference between the result of the inference of the subset network and the labels of the labeled data. This is so-called supervised learning, but it does not necessarily have to be supervised learning; unsupervised learning or semi-supervised learning may also be performed.

[0037] <Effects of the learning process> This section explains an example of the effects of the learning process. More specifically, it explains why the fact that the C-graph is not updated increases the possibility of reusing hardware resources. As mentioned above, the C-graph is a graph that shows the connectivity of a neural network. In other words, the C-graph is a graph that shows the presence or absence of edges between each node that makes up the neural network.

[0038] Therefore, when a neural network is implemented using an optical system, there should be no waveguides where the C-graph indicates there are no edges, and conversely, there must be waveguides where the C-graph indicates there are edges. Thus, the C-graph can be said to be information that restricts the arrangement of waveguides in the optical system that implements the neural network.

[0039] Therefore, in the case of techniques that obtain a desired neural network by optimizing the C-graph, once the C-graph is optimized and the neural network is realized in the optical system, it is not possible to re-optimize the C-graph when obtaining a new mathematical model. This is because, when the optical system is realized using photonic crystals, etc., the optical system is constructed using techniques such as etching, and in such cases, once the arrangement of waveguides is determined and the system is assembled, the waveguides cannot be rearranged. Since the waveguides cannot be rearranged, the C-graph, which contains information indicating the arrangement of waveguides, must not be changed. Consequently, new optimization is not possible.

[0040] On the other hand, in the learning process performed by learning device 1, the C graph is not updated. Only the S graph or M graph is updated, and a subsystem with higher inference accuracy can be obtained. Therefore, even if the C graph has been optimized using the techniques described in Non-Patent Documents 1 to 5 and an optical system has been assembled to realize it, a new subsystem can be obtained using the S graph and M graph while keeping the optimized C graph as is. And since the C graph has not been modified, the obtained subsystem can be realized using the already constructed optical system. Specifically, the obtained subsystem can be realized simply by applying heat, pressure, or an electromagnetic field to part or all of the optical system to change its physical properties.

[0041] Therefore, the learning process can increase the potential for reusing hardware resources.

[0042] <An example of a partial network acquisition rule explained using a mathematical formula> Subnetwork acquisition specifically refers to the process of obtaining the edge weights between each node in the subset. An edge weight of zero means that the edge does not exist.

[0043] The weight of each edge in a subset network can be obtained by a process represented, for example, by equation (1) below. The rule that the weight of each edge in a subset network can be obtained by equation (1) is an example of a subset network acquisition rule.

[0044]

number

[0045] Here, matrix W rand (l) T is a matrix representing the weights of each edge in the target network. (l) This is a tensor defined by the following equation (2). In the example in Figure 1, the graph of image G102 is this T (l) This is an example of a graph shown by [the program / platform].

[0046] [Mathematics]

[0047] Note that the symbol represented by the following formula (3) means the Hadamard product.

[0048] [Mathematics]

[0049] C (l) is a matrix indicating the weight of each edge of the C graph. M (l) is a matrix indicating the weight of each edge of the M graph. S (l) is a matrix indicating the weight of each edge of the S graph.

[0050] Next, a specific example of M (l) will be described using mathematical formulas. M (l) may be represented by, for example, the following formula (4). [[ID=3S]]

[0051] [Mathematics]

[0052] Here, K is a list indicating N super-mask densities. Note that N is a predetermined constant of 1 or more. The definition of the super-mask density is described in, for example, Non-Patent Document 3. k0 represents the first in the density list.

[0053] Note that the definition of the density list is a list containing N elements from k1 to kN. Here, kn represents the ratio at which the positive integer n is used in the M graph. Specifically, k1 represents the ratio of 1 in the M graph, k2 represents the ratio of 2 in the M graph, ···, and kN represents the ratio of N in the M graph.

[0054] The function H is defined by equation (5), and is a so-called supermask generating function. k represents the density, which means the proportion of non-zero elements in the matrix.

[0055]

number

[0056] Since function H is a supermask generating function, this is noted just in case, but h(z uv ,k) is a step function defined by the following equation (6): z uv This is matrix Z (l) This represents the value of the element in row u and column v of Z. (l) This is a matrix representing the edge impact of each edge in the target network.

[0057]

number

[0058] Here z t This is a predetermined threshold value.

[0059] Next S (l) Let's explain this using a mathematical formula. (l) This can be expressed, for example, by the following equation (7).

[0060]

number

[0061]

number

[0062] <Example of hardware configuration for learning device 1> Figure 2 shows an example of the hardware configuration of the learning device 1 in the embodiment. The learning device 1 includes a control unit 11 that executes a program, and functions as a device comprising the control unit 11, an interface unit 12, and a storage unit 13 through the execution of the program.

[0063] More specifically, the processor 91 reads the program stored in the memory unit 13 and stores the read program in the memory 92. By executing the program stored in the memory 92, the learning device 1 functions as a device comprising a control unit 11, an interface unit 12, and a memory unit 13.

[0064] The control unit 11 controls the operation of each functional unit of the learning device 1. As described above, the control unit 11 performs, for example, a learning process. The control unit 11 also retrieves, for example, information stored in the memory unit 13. Specifically, the process of retrieving information stored in the memory unit 13 is a read operation.

[0065] The interface unit 12 includes a communication interface for connecting the learning device 1 to an external device. The interface unit 12 communicates with the external device via wired or wireless means.

[0066] The external device may be, for example, a device that transmits data used for learning, such as the labeled data described above (hereinafter referred to as "training data"). In such a case, the learning device 1 may acquire the training data from the device that transmits the training data by communication via the interface unit 12. The external device may also be, for example, a device that transmits information indicating the target network. In such a case, the learning device 1 acquires information indicating the target network by communication via the interface unit 12. The information indicating the target network may be, for example, the matrix W described above. rand (l) That is the case.

[0067] The interface unit 12 may include input devices such as a mouse, keyboard, or touch panel. The interface unit 12 may also be configured as an interface for connecting these input devices to the learning device 1. In this way, the input devices of the interface unit 12 receive various types of information to the learning device 1 via wired or wireless connections. Note that the information does not necessarily have to be input to the communication interface of the interface unit 12; it may also be input to the input devices of the interface unit 12.

[0068] The interface unit 12 outputs various types of information, for example. The interface unit 12 includes, for example, a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display, as well as a speaker. The interface unit 12 may be configured as an interface for connecting these display devices or speakers to the learning device 1. Therefore, the interface unit 12 may output information input to its input device as an image or sound, for example.

[0069] The storage unit 13 is configured using a computer-readable storage medium (non-transitory computer-readable recording medium) such as a magnetic hard disk drive or a semiconductor memory device. The storage unit 13 stores various information related to the learning device 1. The storage unit 13 stores various information generated by the operation of the control unit 11, for example. The storage unit 13 may reside, for example, on the cloud.

[0070] Figure 3 is a flowchart showing an example of the processing flow performed by the learning device 1 in the embodiment. The control unit 11 initializes the weights of the target network (step S101). Next, the control unit 11 executes the learning process (step S102). The execution of the learning process yields the trained S-graph and M-graph.

[0071] <Inference device 2> Figure 4 shows an example of the hardware configuration of the inference device 2 in the embodiment. The inference device 2 includes a control unit 21 equipped with a processor 93 such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or NPU (Neural Network Processing Unit) connected by a bus, and memory 94, and executes a program. Through program execution, the inference device 2 functions as a device comprising an inference network 20, a control unit 21, an interface unit 22, and a storage unit 23.

[0072] More specifically, the processor 93 reads the program stored in the memory unit 23 and stores the read program in the memory 94. By executing the program stored in the memory 94, the inference device 2 functions as a device comprising an inference network 20, a control unit 21, an interface unit 22, and a memory unit 23.

[0073] The inference network 20 is a neural network obtained using a trained S-graph and a trained M-graph. A concrete example of a neural network obtained using a trained S-graph and a trained M-graph is the optical system that implements that neural network. Of course, the neural network does not necessarily have to be implemented as an optical system; it may also be implemented as an electronic or electrical circuit. A trained S-graph refers to the S-graph at the point when the training termination condition is met. A trained M-graph refers to the M-graph at the point when the training termination condition is met.

[0074] The control unit 21 controls the operation of each functional unit of the inference device 2. The control unit 21 executes the inference process. The inference process performs inference by controlling the operation of the inference network 20. The control unit 21, for example, retrieves information stored in the memory unit 23. Specifically, the process of retrieving information stored in the memory unit 23 is read.

[0075] The interface unit 22 includes a communication interface for connecting the inference device 2 to an external device. The interface unit 22 communicates with the external device via wired or wireless connection.

[0076] The external device is, for example, the source device of the data to be inferred in the inference process. The interface unit 22 acquires the data to be inferred in the inference process by communicating with such a source device of the data to be inferred.

[0077] The interface unit 22 may include input devices such as a mouse, keyboard, or touch panel. The interface unit 22 may also be configured as an interface connecting these input devices to the inference device 2. In this way, the input devices of the interface unit 22 receive various types of information to the inference device 2 via wired or wireless connections. Note that the information does not necessarily have to be input to the communication interface of the interface unit 22; it may also be input to the input devices of the interface unit 22.

[0078] The interface unit 22 outputs various types of information, for example. The interface unit 22 is comprised of a display device such as a CRT display, liquid crystal display, or organic EL display, and a speaker. The interface unit 22 may be configured as an interface for connecting these display devices or speakers to the inference device 2. Therefore, the interface unit 22 may output information input to its input device as an image or sound, for example.

[0079] The memory unit 23 is configured using a computer-readable recording medium such as a magnetic hard disk drive or a semiconductor memory device. The memory unit 23 stores various information related to the inference device 2. The memory unit 23 stores various information generated by the operation of the control unit 21, for example. The memory unit 23 may reside, for example, on the cloud.

[0080] Figure 5 is a flowchart showing an example of the processing flow performed by the inference device 2 in the embodiment. The control unit 21 acquires the inference target (step S201). The control unit 21 executes the inference process (step S202).

[0081] The learning device 1, configured in this way, performs the learning process. Therefore, as described in <Effects of the Learning Process>, the learning device 1 can increase the possibility of reusing hardware resources.

[0082] (modified version) Furthermore, the target network does not necessarily have to be a dense neural network; a sparse neural network is also acceptable. This is because the degree of over-parameterization is high, and even after removing (pruning) many parameters (weights), there are still excessive parameters. For example, it is known that the commonly used ResNet model can be trained using SLTH while pruning down to 1% of the original parameters. Therefore, even if the parameters are only 30% of the original from the beginning, 29% are still excessive. Thus, in such cases, there is room for training using learning device 1.

[0083] A dense neural network is one in which most of the weights are non-zero. "Most" can be any percentage above a certain threshold of 90%. Therefore, a dense neural network is, for example, one in which 90% or more of the weights are non-zero. Since the percentage of non-zero elements is the density, a dense neural network can also be described as a network with a density close to 1.

[0084] A sparse neural network is one in which most of the weights are zero. Therefore, a sparse neural network is, for example, one in which more than 80% of the weights are non-zero. The meaning of "most" in the definition of a dense neural network and the meaning of "most" in the definition of a sparse neural network may differ.

[0085] For example, after simultaneously training the C-graph, S-graph, and M-graph using a C-graph, S-graph, and M-graph, it is acceptable to update only the S-graph and M-graph without changing the results for the C-graph.

[0086] The learning device 1 may be implemented using multiple information processing devices connected to each other via a network. In this case, each process executed by the control unit 11 may be performed in a distributed manner by the multiple information processing devices.

[0087] The inference device 2 may be implemented using multiple information processing devices connected to each other via a network. In this case, each process executed by the control unit 21 may be performed in a distributed manner by multiple information processing devices.

[0088] Furthermore, the learning process and the inference process do not necessarily have to be performed by different devices. Therefore, for example, learning device 1 may perform not only the learning process but also the inference process, or inference device 2 may perform not only the inference process but also the learning process.

[0089] Furthermore, all or part of the functions of learning device 1 and inference device 2 may be implemented using hardware such as ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), or FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Computer-readable recording media include, for example, portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. The program may also be transmitted via a telecommunications line.

[0090] While embodiments of this invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and includes designs and the like that do not depart from the spirit of this invention. [Explanation of Symbols]

[0091] 1...Learning device, 2...Inference device, 11...Control unit, 12...Interface unit, 13...Storage unit, 21...Control unit, 22...Interface unit, 23...Storage unit, 91...Processor, 92...Memory, 93...Processor, 94...Memory

Claims

1. A control unit that performs learning using a C-graph, which is a graph in which each node of a target neural network to be pruned has a one-to-one correspondence with a node, and the edge weights are 0 or 1, and updates an S-graph, which is a graph in which each node of the target network has a one-to-one correspondence with a node, and the edge weights are -1 or 1, and an M-graph, which is a graph in which each node of the target network has a one-to-one correspondence with a node, and the edge weights are positive integers. Equipped with, In the aforementioned learning process, the following are executed: network execution process, impact update process, SM graph update process, and partial network acquisition process. The network execution process is the process of executing a subset network, which is a neural network obtained based on the target network, the C graph, the S graph, and the M graph. The aforementioned influence update process is a process that updates the edge influence, which is a value assigned to each edge of the target network and indicates the degree of influence that the assigned edge is estimated to have on the accuracy of inference by the subset network, based on the results of inference by the subset network. The SM graph update process is a process that updates the S graph and the M graph based on each of the influence levels after the influence level update process. In the aforementioned partial network acquisition process, the partial network is acquired based on the target network, the C graph, the S graph, and the M graph. In the aforementioned learning process, the C graph is not updated. Learning device.

2. The weight values ​​of the aforementioned target network are predetermined random values. The learning device according to claim 1.

3. The aforementioned target network is a sparse neural network. The learning device according to claim 1.

4. The neural network to be pruned is a target network, and the C graph is a graph in which each node of the target network has a one-to-one correspondence with a node, and the edge weights are 0 or 1. The C graph is a graph in which each node of the target network has a one-to-one correspondence with a node, and the edge weights are -1 or 1. The S graph is a graph in which each node of the target network has a one-to-one correspondence with a node, and the edge weights are positive integers. The M graph is a graph in which each node of the target network has a one-to-one correspondence with a node, and the edge weights are positive integers. The M graph is a control unit that performs learning, and the learning includes network execution processing, influence update processing, S and M graph update processing, and subset network acquisition processing. The network execution processing is obtained based on the target network, the C graph, the S graph, and the M graph. A learning method executed by a learning device, wherein the learning device executes a subset network which is a neural network, the influence update process updates the edge influence value, which is a value assigned to each edge of the target network and indicates the degree of influence that the assigned edge is estimated to have on the accuracy of inference by the subset network, based on the inference results of the subset network, the SM graph update process updates the S graph and the M graph based on each of the influence values ​​after the influence update process, the subset network acquisition process acquires the subset network based on the target network, the C graph, the S graph and the M graph, and the C graph is not updated during learning, The control unit performs the learning step, A learning method that has [something].

Images