Classification device, classification method, and classification program

Abstract

This technology provides the ability to improve classification performance for both previous and new tasks. [Solution] Classification device 1 is a feature extraction unit 20 that has performed learning including deleting or adding paths between nodes in adjacent layers of a neural network, and comprises a feature extraction unit 20 that extracts feature quantities from input data, and a classification unit 40 that holds classification weights for each class, takes the feature quantities as input, and classifies the input data based on the feature quantities and classification weights. Learning includes classifying multiple nodes of the neural network into stable nodes and plastic nodes with lower activity values ​​than the stable nodes, and connecting the stable node and the plastic node if there is a stable node in a predetermined layer of the neural network and a plastic node in the layer following that predetermined layer.

Description

Technical Field

[0001] The present invention relates to machine learning technology.

Background Art

[0002] Humans can learn new knowledge through long-term experience and maintain old knowledge without forgetting it. On the other hand, the knowledge of a Convolutional Neural Network (CNN) depends on the dataset used for learning, and in order to adapt to changes in the data distribution, it is necessary to relearn the parameters of the CNN for the entire dataset. In a CNN, as learning progresses for a new task, the estimation accuracy for old tasks decreases. Thus, in a CNN, when continuous learning is performed, catastrophic forgetting, where the learning results of old tasks are forgotten during the learning of new tasks, cannot be avoided.

[0003] Therefore, as one of the learning methods for neural networks different from the conventional ones, NISPA (Neuro-Inspired Stability-Plasticity Adaptation) has been proposed (see, for example, Non-Patent Document 1). NISPA is a method that mimics the memory mechanism of the human brain and deletes or adds paths between nodes in adjacent layers during learning. In NISPA, paths between nodes with high activation values (stable nodes) during learning are added, and paths including nodes with low activation values (plastic nodes) are added at a lower rate than those between stable nodes. Thereby, in NISPA, new knowledge can be acquired while maintaining the knowledge obtained in past learning.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

[0005] NISPA states that the density of connection paths (connection density) is kept constant without distinguishing between each layer from the input layer to the output layer. On the other hand, Non-Patent Literature 2 reports that if disconnections are made between predetermined layers, for example between the third and fourth layers, and between the fourth and fifth layers, the estimation accuracy may decrease significantly compared to when disconnections are made between other layers. It is thought that these predetermined layers are co-adapted to both previous and new tasks.

[0006] This invention has been made in view of these circumstances, and its purpose is to provide a technique for improving classification performance for previous and new tasks. [Means for solving the problem]

[0007] To solve the above problems, a classification device according to one aspect of the present invention is a feature extraction unit that has performed learning including deleting or adding paths between nodes in adjacent layers of a neural network, comprising: a feature extraction unit that extracts feature quantities from input data; and a classification unit that holds classification weights for each class, takes the feature quantities as input, and classifies the input data based on the feature quantities and the classification weights, wherein the learning includes classifying a plurality of nodes of the neural network into stable nodes and plastic nodes with lower activity values ​​than the stable nodes, and connecting the stable nodes and the plastic nodes if there are stable nodes in a predetermined layer of the neural network and plastic nodes in the layer following that predetermined layer.

[0008] Another aspect of the present invention is a classification method. This method includes a learning step of performing learning which involves deleting or adding paths between nodes in adjacent layers of a neural network; a feature extraction step of extracting features from input data; and a classification step of storing classification weights for each class, taking the features as input, and classifying the input data based on the features and the classification weights, wherein the learning step includes a classification step of classifying a plurality of nodes of the neural network into stable nodes and plastic nodes with lower activity values ​​than the stable nodes; and a connection step of connecting a stable node and a plastic node if a predetermined layer of the neural network has stable nodes and the next layer after that predetermined layer.

[0009] Furthermore, any combination of the above components, as well as conversions of the expression of the present invention between methods, apparatus, systems, recording media, computer programs, etc., are also valid embodiments of the present invention. [Effects of the Invention]

[0010] According to one aspect of the present invention, classification performance for both previous and new tasks can be improved. [Brief explanation of the drawing]

[0011] [Figure 1] This is a functional block diagram schematically showing the general configuration of the classification device according to this embodiment. [Figure 2] Figure 1 is a schematic functional block diagram showing the machine learning device that performs the learning of the feature extraction unit. [Figure 3] Figure 2 is a flowchart illustrating an example of the learning process using a machine learning device. [Figure 4] Figure 4(a) shows an example of the initialization process in the learning process by NISPA, and Figure 4(b) shows an example of the initialization process in the learning process according to this embodiment. [Figure 5] Figure 5(a) shows an example of the connection adjustment process during learning by NISPA, and Figure 5(b) shows an example of the connection adjustment process during learning according to this embodiment. [Modes for carrying out the invention]

[0012] Embodiments of the present invention will be described below with reference to the drawings. The specific numerical values ​​and other details shown in these embodiments are merely examples to facilitate understanding of the invention and do not limit the present invention unless otherwise specified. Elements not directly related to the present invention are omitted from the drawings.

[0013] Figure 1 is a schematic functional block diagram showing the general configuration of the classification device 1 according to this embodiment. As shown in Figure 1, the classification device 1 comprises an input unit 10, a feature extraction unit 20, a classification unit 40, and an output unit 50.

[0014] The input unit 10 receives input data to be classified by the classification device 1. The input data is, for example, image data of an object, and the object being photographed can be an animal, a vehicle, a person, etc.

[0015] The feature extraction unit 20 extracts the feature amounts of the input data received by the input unit 10. The feature extraction unit 20 is a learned neural network model. The feature extraction unit 20 has performed learning in advance including deleting or adding paths between nodes in adjacent layers in the neural network. The feature extraction unit 20 is learned by a machine learning device 22 (see FIG. 2) described later. The feature extraction unit 20 may have completed learning or may be updatable by further performing learning. The number of layers of the neural network model included in the feature extraction unit 20 is, as an example, 7 layers, but is not particularly limited as long as it is 4 layers or more.

[0016] The classification unit 40 classifies the input data received by the input unit 10. The classification unit 40 holds the classification weights for each class. The classification unit 40 takes the input data and the feature amounts extracted by the feature extraction unit 20 as inputs, and classifies the input data based on the feature amounts and the classification weights. The classification weights held by the classification unit 40 are, for example, the feature amounts (centroids) obtained by averaging the feature amounts output by the feature extraction unit 20 using big data for each class. The classification unit 40 compares the feature amounts with the classification weights, and uses the class with the closest classification weight as the classification result.

[0017] The output unit 50 outputs the classification result by the classification unit 40. That is, the output unit 50 outputs information indicating to which class the input data has been classified.

[0018] FIG. 2 is a functional block diagram schematically showing a machine learning device 22 that learns the feature extraction unit 20. As shown in FIG. 2, the machine learning device 22 includes a feature extraction unit 20, a learning unit 24, an initialization unit 26, and a connection adjustment unit 28.

[0019] The learning unit 24 receives the input of the dataset for each class and trains the feature extraction unit 20 using the dataset. Each class has one or more learning phases, and the dataset for each learning phase is used. Each dataset includes a large number of samples. An example of a sample is an image, but it is not limited to this. When the sample is an image, one class is, for example, the classification of dog images and cat images, and another class is related to the classification of bird images and rabbit images. After training the feature extraction unit 20 for a certain class, the learning unit 24 trains it for another class.

[0020] The initialization unit 26 initializes the path between the nodes of the feature extraction unit 20 before the feature extraction unit 20 learns a new class. That is, it deletes or adds a path between the nodes. Details of the initialization will be described later.

[0021] The connection adjustment unit 28 adjusts the connection state between the nodes of the feature extraction unit 20, that is, deletes or adds a path, at the timing when a certain learning phase ends while the feature extraction unit 20 is learning a predetermined class. Details of the connection adjustment will be described later.

[0022] FIG. 3 is a flowchart for explaining an example of the procedure of the learning process by the machine learning device 22. The learning process in the present embodiment is an improvement on the method by NISPA. Hereinafter, points particularly different from the method by NISPA will be described to that effect.

[0023] First, before starting the learning of a new class, the initialization unit 26 initializes the connection state between adjacent layers of nodes included in each layer of the feature extraction unit 20 (S10). Figure 4(a) is a diagram showing an example of the initialization process in the learning process by NISPA, and Figure 4(b) is a diagram showing an example of the initialization process in the learning process according to this embodiment, i.e., the process in step S10. Specifically, Figures 4(a) and 4(b) show the state in which the initialization process by the initialization unit 26 has been executed after the learning of one class has finished and before the learning of the next class has started. Also, Figures 4(a) and 4(b) schematically show each node and its connection state in the third to fifth layers of the feature extraction unit 20.

[0024] The common elements of Figures 4(a) and 4(b) are explained below. Nodes are classified into stable nodes 60 and plastic nodes 62. Details of the classification of stable nodes 60 and plastic nodes 62 will be described later, but they are classified based on the node's activation value. The node's activation value is determined based on the activation value of the parent node connected in the layer immediately preceding the node, i.e., the layer closer to the input layer, and the weight of the connection with that parent node. Among the paths between nodes in adjacent layers, solid lines indicate paths that were connected before the current initialization process, and dashed lines indicate paths that become connected in the current initialization process.

[0025] As shown in Figure 4(a), in NISPA initialization, the existence of paths connecting nodes in adjacent layers, other than paths connecting stable nodes 60 and paths connecting a stable node 60 to a plastic node 62 in the next layer (the layer on the right in Figure 4), is random. In other words, in NISPA initialization, paths between stable nodes 60 are connected, and paths between a stable node 60 and a plastic node 62 in the next layer are deleted. Furthermore, the existence of paths between a plastic node 62 and a stable node 60 in the next layer, and paths between plastic nodes 62, is random. At this time, NISPA deletes or adds paths between nodes to maintain the connection density before the learning of the previous class began. Figure 4(a) shows layers 3 to 5, but NISPA processes all other layers similarly.

[0026] This embodiment will explain the differences from NISPA. As shown in Figure 4(b), the initialization unit 26 connects the paths between a stable node 60 and a plastic node 62 in the next layer (closer to the output layer) of the feature extraction unit 20, from the first predetermined layer (third layer in the figure; also simply called the predetermined layer) to the second predetermined layer (fifth layer in the figure), if a stable node 60 is in a certain layer. The layers from the first predetermined layer to the second predetermined layer are the layers closest to the input layer after the lower-order layers from the input layer to the first predetermined layer, and it is thought that information common to the input data and less dependent on the class being learned is transmitted in such layers. Therefore, with the above configuration, the feature extraction unit 20 maintains the path from the stable node 60 to the plastic node 62 between the first predetermined layer and the second predetermined layer, so that it can retain more paths that may be used to transmit information in the learning of a new task than NISPA, while utilizing the memory path information obtained from learning a previous task.

[0027] Although not shown in Figure 4(b), in the input layer of the feature extraction unit 20, that is, from the first layer to the first predetermined layer, all nodes in adjacent layers are connected to each other. This connection state may remain unchanged even in the processing described later. Here, all nodes in adjacent layers being connected means that a node has a path to all nodes in the adjacent layer. Since the lower-order layers from the input layer to the first predetermined layer are considered to be layers that transmit basic information of the input data that is independent of the class being learned, such as the contour and color of image data, the classification performance can be improved by configuring the feature extraction unit 20 as described above.

[0028] The initialization unit 26 may perform the same processing as NISPA for the layers of the feature extraction unit 20 from the second predetermined layer onwards, i.e., layers close to the output layer. This may also be the case for the processing described later. Furthermore, in the processing of step S10, if there is no previous class learning and this is the first time learning for a class, the initialization unit 26 may set all nodes of the feature extraction unit 20 as plastic nodes.

[0029] Returning to the explanation of Figure 3, the learning unit 24 starts the first learning phase of the new task (S12). At this time, the learning unit 24 changes all plastic nodes of the feature extraction unit 20 to candidate stable nodes. The learning unit 24 receives the input of the dataset for the current learning phase and uses this dataset to train the feature extraction unit 20 (S14). The feature extraction unit 20 updates the activation values ​​of the nodes included in each layer through learning.

[0030] Next, the learning unit 24 changes candidate stable nodes with low activation values ​​in the feature extraction unit 20 into plastic nodes (S16). Specifically, the learning unit 24 sorts the candidate stable nodes in each layer of the feature extraction unit 20 in descending order of activation value, retains the candidate stable nodes that fall within a predetermined proportion as candidate stable nodes, and changes the candidate stable nodes that do not fall within this predetermined proportion into plastic nodes.

[0031] The connection adjustment unit 28 adjusts the connection state between nodes of the feature extraction unit 20 (S18). Figure 5(a) is a diagram showing an example of the connection adjustment process in learning by NISPA, and Figure 5(b) is a diagram showing an example of the connection adjustment process in learning according to this embodiment. Specifically, Figures 5(a) and 5(b) schematically show each node and its connection state in the third to fifth layers of the feature extraction unit 20 when a certain learning phase of a certain class has been completed. In Figures 5(a) and 5(b), among the paths between nodes in adjacent layers, solid lines indicate paths that were already connected before the current learning phase, dashed lines indicate paths that will be newly connected in this process, and dotted lines indicate paths that will be newly deleted in this process.

[0032] As shown in Figure 5(a), NISPA randomly deletes all paths from the plastic node 62 to the stable node 60, plastic node 62, or candidate stable node 64 of the next layer. Subsequently, the paths from plastic node 62 to the plastic node 62 of the next layer and from plastic node 62 to the candidate stable node 64 of the next layer are connected randomly. However, paths that were connected at the time of initialization are not reconnected. Also, the path from plastic node 62 to the stable node 60 of the next layer is not reconnected.

[0033] The paths from candidate stable node 64 to stable node 60 in the next layer, and the paths from candidate stable node 64 to candidate stable node 64 in the next layer are connected. On the other hand, the paths from candidate stable node 64 to plastic node 62 are randomly disconnected. Subsequently, paths from candidate stable node 64 to plastic node 62 are randomly reconnected. However, paths that were connected during initialization are not reconnected.

[0034] Furthermore, in NISPA, all paths from stable node 60 to the next layer's stable node 60, plastic node 62, and candidate stable node 64 maintain their previous connection state. That is, the connected state is maintained where there is a path, and the disconnected state is maintained where there is no path.

[0035] The example shown in Figure 5(b) illustrates the feature extraction unit 20 of this embodiment. The connection adjustment unit 28 deletes and adds paths in the first predetermined layer to the second predetermined layer of the feature extraction unit 20, in the same manner as NISPA, except for matters described below. In this embodiment, the connection adjustment unit 28 connects all paths from the stable node 60 to the plastic node 62 of the next layer, and all paths from the stable node 60 to the candidate stable node 64 of the next layer, in the first predetermined layer to the second predetermined layer of the feature extraction unit 20.

[0036] Furthermore, the paths from candidate stable node 64 to plastic node 62 are randomly disconnected. Then, the paths from candidate stable node 64 to plastic node 62 are randomly reconnected, but paths that were connected during initialization are not reconnected. Although the connection of the paths from candidate stable node 64 to the plastic node 62 of the next layer is random, the connection density is at least higher than that of NISPA.

[0037] As a result, the connection adjustment unit 28 of this embodiment leaves more inter-layer paths than the connection adjustment process by NISPA, thereby improving classification performance.

[0038] Returning to the explanation of Figure 3, the learning unit 24 determines whether the learning has converged (S20). That is, it determines whether the classification accuracy is higher after the end of the previous learning phase than after the end of the current learning phase. If the learning unit 24 determines that the learning has not converged (N in S20), it starts the next learning phase (S22) and returns to the process in step S14. In other words, the machine learning device 22 repeats the process in steps S14 to S18 for each new learning phase until the classification accuracy of the feature extraction unit 20 after the end of the current learning phase is lower than that of the feature extraction unit 20 after the end of the previous learning phase.

[0039] When the learning unit 24 determines that the learning has converged (Y in S20), it proceeds to the process in step S24. In other words, if the learning has converged, the learning in the current learning phase will be overfitting, so the machine learning device 22 does not perform any further learning and uses the feature extraction unit 20 which has finished the previous learning phase.

[0040] The learning unit 24 changes candidate stable nodes 64 to stable nodes 60 or plastic nodes 62 based on the end state of the previous learning phase (S24). For example, throughout each learning phase, the learning unit 24 may change all candidate stable nodes 64 that were not changed to plastic nodes 62 by the processing in step S18 to stable nodes 60. The connection adjustment unit 28 deletes the paths of nodes that have been changed to plastic nodes 62 (S24). That is, the machine learning device 22 may delete all paths between layers from the first predetermined layer to the last layer (output layer) that are not between stable nodes.

[0041] If there is a next class (Y in S26), the machine learning device 22 returns to the process in step S10. If there is no next class (N in S26), the machine learning device 22 terminates the process.

[0042] As described above, the classification device 1 according to this embodiment is a feature extraction unit 20 that has performed learning including deleting or adding paths between nodes in adjacent layers of a neural network, and comprises a feature extraction unit 20 that extracts feature quantities from input data, and a classification unit 40 that holds classification weights for each class, takes the feature quantities as input, and classifies the input data based on the feature quantities and classification weights. The learning includes classifying a plurality of nodes in the neural network into stable nodes and plastic nodes with lower activity values ​​than the stable nodes, and if there are stable nodes in a predetermined layer of the neural network and plastic nodes in the layer following that predetermined layer, connecting the stable nodes and the plastic nodes.

[0043] As a result, the classification device 1 maintains a path from stable nodes to plastic nodes between predetermined layers, allowing it to utilize the memory path information obtained from learning previous tasks while retaining more paths that could potentially be used for information transfer during learning new tasks than NISPA. Therefore, the classification device 1 can improve its classification performance for both previous and new tasks.

[0044] Furthermore, in the classification device 1 according to this embodiment, the plastic node may be changed during learning to become a stable node and a candidate stable node different from the plastic node, and the learning may include connecting the stable node and the candidate stable node if there is a stable node in a predetermined layer and a candidate stable node in the next layer of that predetermined layer. As a result, the classification device 1 has more inter-layer paths than NISPA, and more information is transmitted between layers than NISPA, so the classification performance can be improved.

[0045] Furthermore, the feature extraction unit of the classification device 1 according to this embodiment may connect all nodes between adjacent layers from the input layer to a predetermined layer. Since the lower-order layers from the input layer to the predetermined layer are considered to be layers that transmit basic information of the input data that is independent of the class being learned (for example, information such as the contour and color of image data), the above configuration can improve classification performance.

[0046] The various processes of the classification device 1 and machine learning device 22 described above can, of course, be implemented as devices using hardware such as a CPU and memory, but they can also be implemented by firmware stored in ROM (read-only memory) or flash memory, or by software on a computer. The firmware program and software program can be recorded on a recording medium readable by a computer and provided, transmitted and received with a server via a wired or wireless network, or transmitted and received as data broadcasting on terrestrial or satellite digital broadcasting.

[0047] The present invention has been described above based on embodiments. The embodiments are illustrative, and it will be understood by those skilled in the art that various modifications are possible in combinations of their components and processing processes, and that such modifications also fall within the scope of the present invention. [Explanation of symbols]

[0048] 1 Classification unit, 10 Input unit, 20 Feature extraction unit, 22 Machine learning unit, 24 Learning unit, 26 Initialization unit, 28 Connection adjustment unit, 40 Classification unit, 50 Output unit, 60 Stabilization node, 62 Plasticity node, 64 Candidate stabilization node.

Claims

1. A feature extraction unit that has performed learning including deleting or adding paths between nodes in adjacent layers of a neural network, the feature extraction unit that extracts feature quantities from the input data, A classification unit that stores classification weights for each class, takes the feature quantities as input, and classifies the input data based on the feature quantities and the classification weights, The learning process includes classifying multiple nodes of the neural network into stable nodes and plastic nodes with lower activity values ​​than the stable nodes, and connecting the stable nodes and plastic nodes if there are stable nodes in a predetermined layer of the neural network and plastic nodes in the layer following that predetermined layer. Classification device.

2. During the learning process, the plastic node is modified to become a candidate stable node different from the stable node and the plastic node. The learning process includes connecting a stable node to a candidate stable node if there is a stable node in a predetermined layer and a candidate stable node in the layer following that predetermined layer. The classification device according to claim 1.

3. The feature extraction unit connects all nodes between adjacent layers from the input layer to the predetermined layer. A classification device according to claim 1 or 2.

4. A learning step that includes removing or adding connections between nodes in adjacent layers of a neural network, A feature extraction step that extracts features from the input data, A classification step includes storing classification weights for each class, taking the aforementioned features as input, and classifying the input data based on the aforementioned features and the classification weights, The learning step includes a classification step of classifying a plurality of nodes of the neural network into stable nodes and plastic nodes with lower activity values ​​than the stable nodes, and a connection step of connecting the stable nodes and the plastic nodes if there are stable nodes in a predetermined layer of the neural network and plastic nodes in the layer following that predetermined layer. Classification method.

5. A learning step that includes removing or adding connections between nodes in adjacent layers of a neural network, A feature extraction step that extracts features from the input data, A classification step that maintains classification weights for each class, takes the aforementioned features as input, and classifies the input data based on the aforementioned features and the classification weights, A classification program to be executed by a computer, The learning step includes a classification step of classifying a plurality of nodes of the neural network into stable nodes and plastic nodes with lower activity values ​​than the stable nodes, and a connection step of connecting the stable nodes and the plastic nodes if there are stable nodes in a predetermined layer of the neural network and plastic nodes in the layer following that predetermined layer. A classification program.

Images