Re-learning method, information processing device and program

Abstract

This reduces the degradation of classification accuracy of existing classes due to retraining with the addition of new classes, and improves classification accuracy across both existing and new classes. [Solution] The re-learning method disclosed herein inputs training images of a new class into a training model, acquires a first feature of the training image (S110), generates a pseudo-image of each existing class based on a gradient that minimizes the distance between a second feature representing each existing class and the first feature (S120), and re-trains the training model using the training images as training data for the new class and the pseudo-images of each existing class as training data for that existing class (S140).

Description

[Technical Field]

[0001] The present disclosure relates to a relearning method, an information processing device, and a program. [Background technology]

[0002] Patent Document 1 discloses a teacher data generation device that generates teacher data for executing machine learning. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-13293 Summary of the Invention [Means for solving the problem]

[0004] A re-learning method according to one embodiment of the present disclosure includes a first step of inputting training images of a new class to be newly added into a training model that classifies input images into any of the existing classes and acquiring first features of the training images; a second step of calculating a gradient that minimizes the distance between a second feature representing each existing class and the first feature and generating a pseudo-image of the existing class based on the gradient; and a third step of re-learning the training model using the training images as training data for the new class and the pseudo-images of each existing class as training data for the existing class.

[0005] An information processing device according to one embodiment of the present disclosure is an information processing device having at least one processor, and the at least one processor executes a first feature acquisition process of inputting a training image of a new class to be newly added into a learning model that classifies an input image into one of the existing classes and acquiring a first feature of the training image; a pseudo image generation process of calculating a gradient that minimizes the distance between a second feature representing each existing class and the first feature and generating a pseudo image of the existing class based on the gradient; and a learning process of re-training the learning model using the training image as training data for the new class and the pseudo image of each existing class as training data for the existing class. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 10 is a diagram illustrating an example of relearning of a learning model. [Figure 2] FIG. 10 is a diagram illustrating an example of a feature space. [Figure 3] FIG. 1 is a block diagram illustrating a hardware configuration of an information processing device according to the present disclosure. [Figure 4] 1 is a flowchart illustrating an example of the flow of a relearning method according to the present disclosure. [Figure 5] 10A and 10B are diagrams illustrating experimental results of a relearning method according to the present disclosure. [Figure 6] 10A and 10B are diagrams illustrating experimental results of a relearning method according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0007] Hereinafter, an embodiment of the present disclosure will be described in detail with reference to the drawings. For ease of understanding, the background and problems of the present disclosure will be described first, and then the details of the present disclosure will be described.

[0008] <Conventional relearning methods and their challenges> Conventionally, there are known techniques for classifying input images into multiple classes using a learning model. The Replay method and the Replay-Free method are known as methods for adding new classes that have not yet been learned to such a learning model.

[0009] The replay method is a method in which the training data used to generate the learning model is retrained with new training data that includes images of a new class. The replay-free method is a method in which the model is retrained using only new training data that includes images of a new class.

[0010] The Replay-Free method does not require training data used in previous training, making it easier to implement than the Replay method, but it may result in a decrease in the classification accuracy of previously trained classes. On the other hand, the Replay method has little impact on the classification accuracy of previously trained classes, but it may be difficult to obtain the training data used in previous training. For example, when adding a new class to a training model purchased from another company, it is often difficult to obtain the training data used in previous training.

[0011] The re-learning method according to the present disclosure aims to reduce a decrease in classification accuracy of existing, trained classes and improve classification accuracy across existing classes and new classes when adding a new class to a learning model that classifies input images into multiple classes through re-learning.

[0012] <Overview of the relearning method> An overview of the re-learning method according to the present disclosure will be explained using FIG. 1. At production sites such as factories, parts used in manufacturing are sometimes classified using a learning model that classifies input images into multiple classes. For example, an image IMG of a part taken at the manufacturing site may be input to a learning model M that has been trained to be able to classify parts A and B, and the part image IMG may be classified as part A or part B. The learning model M is a neuron network that includes one or more hidden layers between the input layer and the output layer.

[0013] The number of parts to be classified may increase with daily product development. For example, if part C appears that has not previously been the subject of classification by learning model M, re-training is performed to enable further classification of part C. With the Replay-Free method described above, re-training is performed using only the training data for part C, but this may result in a decrease in the classification accuracy of parts A and B that have already been trained. When re-training with the Replay method, it may be impossible to obtain the training data used to train parts A and B.

[0014] In the re-learning method disclosed herein, a training image showing a new component C is prepared, and based on the training image, a pseudo image having characteristics similar to those of component A (hereinafter referred to as the pseudo image of component A) and a pseudo image having characteristics similar to those of component B (hereinafter referred to as the pseudo image of component B) are generated. Then, re-learning of the learning model M is performed using the training image showing the new component C as training data for the new class of component C, the pseudo image of component A as training data for the existing class of component A, and the pseudo image of component B as training data for the existing class of component B.

[0015] According to the above configuration, by generating pseudo images that are closer to the characteristics of the existing classes than the training images for the new classes and using them as training data for the trained existing classes, the decision boundary between the existing classes and the new classes is appropriately determined in the feature space, thereby reducing the deterioration in classification accuracy of the existing classes and improving the classification accuracy of the existing classes and the new classes.

[0016] A method for generating a pseudo image of part A and a pseudo image of part B will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of a feature space. Note that, although Fig. 2 shows the feature space in two dimensions, the feature space is not limited to being two-dimensional.

[0017] In the feature space shown in FIG. 2, a cluster XA of the existing class of part A and a cluster XB of the existing class of part B are shown. A decision boundary (not shown) between the existing class of part A and the existing class of part B exists between cluster XA and cluster XB. Outside cluster XA and cluster XB, the first feature FP of the learning image of part C is C1 is plotted. The first feature FP C1 is output from the intermediate layer of the learning model M by inputting the learning image of the part C to the input layer of the learning model M before re-learning. For example, the first feature FP C1 is output from the intermediate layer immediately before the output layer of the learning model M by inputting a learning image of part C to the input layer of the learning model M before relearning.

[0018] Inside cluster XA, there is a second feature RP that represents the existing class of part A. A Also, inside the cluster XB, the second feature RP representing the existing class of part B is plotted. B The second feature RP is plotted. A and the second feature RP B are located in cluster XA and cluster XB, respectively. A and the second feature RP B is determined based on the learned learning data. For example, the average of each feature of the learned learning data of part A is set as the second feature RP A The second feature of the existing class may be generated and stored in advance when the learning model M before re-learning is generated.

[0019] In the re-learning method of the present disclosure, the first feature FP C1 and the second feature RP A The gradient G that minimizes the distance between CA1 and the first feature FP C1 and the second feature point RP B The gradient G that minimizes the distance between CB1 Then, the gradient G CA1 and the gradient G CB1are backpropagated to the inputs of the neurons at each stage of the learning model M, which has the learning image input to its input layer. In this backpropagation, the weights of the neurons included in the learning model M are not adjusted, but the input values ​​are adjusted to match the output values ​​expected to be output from each neuron.

[0020] Gradient G CA1 By backpropagating the feature vectors, the input to the input layer of the learning model M is the second feature vector RP A The image is updated to the pseudo image of part A that is closer to the FP shown in Figure 2. CA1 indicates the feature amount (first feature amount after update) of the pseudo image of part A generated by the first update.

[0021] Next, the feature FP CA1 (Updated first feature) and second feature RP A The gradient G that minimizes the distance between CA2 Then, calculate the gradient G CA2 is backpropagated to the input of the neurons at each stage of the learning model M. As a result, the feature becomes even more accurate than the pseudo image of part A generated by the first update. A A pseudo image of part A approaching the FP shown in Figure 2 is generated. CA2 indicates the feature amount of the pseudo image of part A generated by the second update. Thereafter, the input of the input layer of the learning model M is repeatedly updated to generate a pseudo image of part A. The number of times the input of the input layer of the learning model M is updated may be a predetermined number of times, or the feature amount of the pseudo image and the second feature amount RP A The process may be terminated when the distance between the target and the target becomes equal to or less than a predetermined threshold.

[0022] For part B, the same process as for part A is performed. Gradient G CB1 By backpropagating the feature vectors, the input of the input layer of the learning model M is the second feature vector RP B The image is updated to the pseudo image of part B, which is closer to the FP shown in Figure 2. CB1indicates the feature of the pseudo image of part B generated by the first update (first feature after update). CB1 (Updated first feature) and second feature RP B The gradient G that minimizes the distance between CB2 Then, calculate the gradient G CB2 is backpropagated to the input of the neurons at each stage of the learning model M. As a result, the feature becomes even more accurate than the pseudo image of part B generated by the first update. B A pseudo image of part B approaching the FP shown in Figure 2 is generated. CB2 indicates the feature amount of the pseudo image of part B generated by the second update. Thereafter, the input of the input layer of the learning model M is repeatedly updated to generate a pseudo image of part B. The number of times the input of the input layer of the learning model M is updated may be a predetermined number of times, or the feature amount of the pseudo image and the second feature amount RP B The process may be terminated when the distance between the target and the target becomes equal to or less than a predetermined threshold.

[0023] According to the above configuration, pseudo images of an existing class are generated from training images of a new class, so that re-training is possible even when training data of an existing class that has already been trained is no longer available.

[0024] This type of re-learning method can be used in various fields, such as autonomous driving technology. For example, in a learning model that captures images of the external environment of a vehicle using an on-board camera and classifies road signs in the captured images, when new road signs are added, the method can be used to re-learn about the new road signs.

[0025] The information processing device 1 will be described below with reference to the drawings. FIG. 3 is a block diagram showing the hardware configuration of the information processing device 1 according to the present disclosure.

[0026] <Configuration of information processing device 1> The information processing device 1 shown in FIG. 3 includes at least one processor 11, a main memory device 12, an auxiliary memory device 13, and a communication IF 14. These are electrically connected to one another via a communication bus. The processor 11 is composed of an arithmetic unit, a register, and / or a peripheral circuit. The main memory device 12 temporarily stores programs and data processed by the programs, and is, for example, a volatile memory such as a DRAM (Dynamic Random Access Memory). The auxiliary memory device 13 saves data and programs, and is, for example, a flash memory, an HDD (Hard Disc Drive), or a magneto-optical disk. The communication IF 14 is an interface for inputting and outputting signals for communicating with other computers via a network using a wired or wireless communication standard.

[0027] The learning model M and the learning images of a new class to be newly added to the learning model M are stored, for example, in the auxiliary storage device 13 or another information processing device with which the information processing device 1 can communicate via the communication IF 14. In addition, the trained learning data used to generate the learning model M may also be stored in the auxiliary storage device 13 or another information processing device with which the information processing device 1 can communicate via the communication IF 14.

[0028] <Relearning method flow> Fig. 4 is a flowchart showing an example of the flow of the relearning method according to the present disclosure. Processor 11 of information processing device 1 according to the present disclosure executes a program stored in auxiliary storage device 13 to perform the relearning method shown in Fig. 4.

[0029] In S100, the processor 11 of the information processing device 1 calculates a second feature (for example, RP) for each of the learned existing classes. A , R.P. B The processor 11 of the information processing device 1 obtains, for example, the average of each feature of the learning data of the learned part A as the second feature RP A The average of each feature of the training data of the trained part B is obtained as the second feature RPB When the learning model M is generated, if each second feature has already been acquired and stored in the auxiliary storage device 13, the processor 11 of the information processing device 1 may acquire the second feature of each existing class from the auxiliary storage device 13.

[0030] <First feature acquisition process> In S110, the processor 11 of the information processing device 1 reads out the learning images of the new class from the auxiliary storage device 13, inputs the learning images to the input layer of the learning model M, and calculates the first feature (for example, FP C1 The calculation results of each neuron of the learning model M performed in S110 are used to generate a pseudo image in the subsequent steps.

[0031] <Pseudo image generation process> In S120, the processor 11 of the information processing device 1 calculates the second feature (for example, RP) acquired in S100 for each existing class. A ) and the first feature value (e.g., FP C1 ) and the gradient (e.g., G CA1 ) is backpropagated to the input of the input layer of the learning model M. The information processing device 1 acquires the input of the input layer of the learning model M updated by the backpropagation (for example, a learning image) as a pseudo image of the existing class.

[0032] In S130, the processor 11 of the information processing device 1 determines whether or not to end the generation of pseudo images. For example, the information processing device 1 may end the generation of pseudo images when a predetermined number of pseudo images have been generated for each existing class, or may end the generation of pseudo images when a predetermined number of pseudo images have been generated for each existing class. CA1 ) and the second feature (e.g., RP A) becomes equal to or smaller than a predetermined threshold. When the processor 11 of the information processing device 1 determines that generation of a pseudo image is to be ended (S130: YES), the process proceeds to the process of S140, and when there are classes remaining for generating a pseudo image (S130: NO), the processor 11 further acquires a pseudo image (S120).

[0033] <Learning process> In S140, the processor 11 of the information processing device 1 associates the training images with the new class and associates the pseudo image of each existing class with the existing class, and performs re-training of the training model. In other words, the processor 11 of the information processing device 1 re-trains the training model M using the training images of the new class as training data for the new class and the pseudo image of each existing class as training data for the existing class.

[0034] 5 and 6 are diagrams showing the results of an actual trial of the re-training method of the present disclosure. FIG. 5 shows the results of a process in which images included in the CIFAR-100 dataset were divided into 10 classes, and a total of 100 classes were classified through the initial 10 classes and nine re-training rounds. FIG. 6 shows the results of a process in which images included in the CIFAR-100 dataset were divided into 20 classes, and a total of 100 classes were classified through the initial 20 classes and four re-training rounds. The horizontal axis of FIGS. 5 and 6 represents the number of re-training rounds, and the vertical axis represents the accuracy rate. As shown in FIGS. 5 and 6, the re-training method of the present disclosure achieves a higher accuracy rate than existing methods such as LwF (Learning without Forgetting).

[0035] <Modification> In the above embodiment, for each existing class, the gradient between the second feature value and the first feature value is backpropagated to the input of the input layer of the learning model M, and the input of the input layer of the learning model M updated by the backpropagation is acquired as a pseudo image of the existing class (S120). However, the method for generating pseudo images of existing classes is not limited to this method. Pseudo images may be generated by other methods as long as the feature values ​​of the generated pseudo images of the existing classes are closer to the second feature values ​​of the existing classes than the training images of the new classes. For example, the training images of the new classes may be transformed using a morphing technique based on the gradient between the second feature value and the first feature value.

[0036] <Software implementation example> The functions of each device (hereinafter referred to as "device") that constitutes the information processing device 1 can be realized by a program that causes a computer to function as the device, and a program that causes a computer to function as each control block of the device.

[0037] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The control device and storage device execute the program, thereby realizing the functions described in each of the above embodiments.

[0038] The program may be non-transitory and may be recorded on one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be supplied to the device via any wired or wireless transmission medium.

[0039] In addition, some or all of the functions of each of the control blocks can be realized by logic circuits. For example, integrated circuits in which logic circuits that function as each of the control blocks are formed are also included in the scope of the present disclosure. In addition, the functions of each of the control blocks can also be realized by, for example, a quantum computer.

[0040] Furthermore, each process described in each of the above embodiments may be executed by AI (Artificial Intelligence). In this case, the AI ​​may run on the control device or on another device (for example, an edge computer or a cloud server).

[0041] <Summary> The present disclosure describes at least the following aspects.

[0042] A re-learning method according to aspect 1 of the present disclosure includes a first step of inputting training images of a new class to be newly added into a training model that classifies input images into any of the existing classes and acquiring first features of the training images; a second step of calculating a gradient that minimizes the distance between a second feature representing each existing class and the first feature and generating a pseudo-image of the existing class based on the gradient; and a third step of re-learning the training model using the training images as training data for the new class and the pseudo-images of each existing class as training data for the existing class. According to the above configuration, the re-training method according to aspect 1 of the present disclosure generates pseudo images that are closer to the characteristics of the existing classes than the training images for the new class, and uses these pseudo images as training data for the trained existing classes. Because the decision boundary between the existing classes and the new class is appropriately determined in the feature space, it is possible to mitigate the degradation of classification accuracy of the trained existing classes due to re-training with the addition of a new class, and improve classification accuracy across the existing classes and the new class. Such effects will also contribute to achieving Goal 9 of the United Nations' Sustainable Development Goals (SDGs), which is "Build resilient infrastructure, promote inclusive and sustainable industrialization, and promote innovation and resilience."

[0043] In the re-learning method according to aspect 2 of the present disclosure, in the second step of aspect 1, the gradient is calculated for each of the existing classes, the gradient is backpropagated to the input of the learning model to which the learning image has been input, and the input to the input layer of the learning model updated by the backpropagation is obtained as the pseudo-image. According to the above configuration, the re-learning method according to aspect 2 of the present disclosure generates pseudo-images of an existing class from training images of a new class, so re-learning is possible even when training data of the existing class that has already been trained is unavailable.

[0044] In the re-learning method according to aspect 3 of the present disclosure, in aspect 2, the first feature is updated by updating the input layer of the learning model through backpropagation, and in the second step, the gradient of each existing class is updated by updating the first feature, and a pseudo-image of each existing class is further generated based on the updated gradient of each existing class. According to the above configuration, the re-learning method according to the third aspect of the present disclosure can easily increase the number of pseudo images of existing classes, thereby improving the classification accuracy of the learning model after re-learning.

[0045] The re-learning method according to aspect 4 of the present disclosure is the same as in aspect 3, wherein the second step is repeated until the distance in the feature space between the second feature and the updated first feature becomes equal to or less than a predetermined threshold. According to the above configuration, the re-training method according to the third aspect of the present disclosure can easily generate pseudo images sufficient to determine an appropriate decision boundary in feature space between the first feature of the training image of the new class and the second feature representing the existing class.

[0046] An information processing device according to aspect 5 of the present disclosure is an information processing device having at least one processor, wherein the at least one processor executes a first feature acquisition process for inputting a training image of a new class to be newly added into a learning model that classifies an input image into one of the existing classes and acquiring a first feature of the training image, a pseudo image generation process for calculating a gradient that minimizes the distance between a second feature representing each existing class and the first feature and generating a pseudo image of the existing class based on the gradient, and a learning process for re-training the learning model using the training image as training data for the new class and the pseudo image of each existing class as training data for the existing class. According to the above configuration, the information processing device according to the fourth aspect of the present disclosure generates pseudo images that are closer to the characteristics of the existing classes than the training images of the new classes, and uses the pseudo images as training data for the trained existing classes. Since the decision boundary between the existing classes and the new classes is appropriately determined in the feature space, it is possible to reduce the decrease in classification accuracy of the trained existing classes due to re-training with the addition of the new class, and improve classification accuracy across the existing classes and the new classes. Such effects will also contribute to achieving Goal 9 of the United Nations' Sustainable Development Goals (SDGs), which is "Build resilient infrastructure, promote inclusive and sustainable industrialization, and promote innovation and resilience."

[0047] A program according to aspect 6 of the present disclosure is a program for operating the information processing device of aspect 5, which causes the at least one processor to execute the first feature acquisition process, the pseudo image generation process, and the learning process. According to the above configuration, the program according to the fifth aspect of the present disclosure generates pseudo images that are closer to the characteristics of the existing classes than the training images for the new class, and uses the pseudo images as training data for the trained existing classes. Since the decision boundary between the existing classes and the new class is appropriately determined in the feature space, it is possible to reduce the degradation of the classification accuracy of the trained existing classes due to retraining with the addition of the new class. Such effects will also contribute to achieving Goal 9 of the United Nations' Sustainable Development Goals (SDGs), which is "Build resilient infrastructure, promote inclusive and sustainable industrialization, and promote innovation and resilience."

[0048] (Additional notes) The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present disclosure. [Explanation of symbols]

[0049] 1. Information processing equipment 11 processors 12 Main storage 13 Auxiliary storage device FP C1 First feature RP A , R.P. B Second feature

Claims

1. a first step of inputting a training image of a new class to be newly added into a training model that classifies an input image into one of existing classes, and acquiring a first feature amount of the training image; a second step of calculating a gradient that minimizes the distance between a second feature amount representative of each existing class and the first feature amount, and generating a pseudo image of the existing class based on the gradient; a third step of re-training the learning model using the training images as training data for the new class and the pseudo-images of each existing class as training data for that existing class.

2. In the second step, the gradient is calculated for each of the existing classes, and the gradient is backpropagated to the input of the learning model to which the learning image is input; The re-learning method according to claim 1 , wherein an input to an input layer of the learning model updated by the backpropagation is acquired as the pseudo image.

3. the first feature amount is updated by updating the input layer of the learning model through the backpropagation; In the second step, The gradient of each existing class is updated by updating the first feature amount; The re-training method of claim 2 , further comprising generating a pseudo-image for each existing class based on the updated gradients of each existing class.

4. The re-learning method according to claim 3 , wherein the second step is repeated until a distance in the feature space between the second feature and the updated first feature becomes equal to or less than a predetermined threshold.

5. An information processing device including at least one processor, At least one processor a first feature amount acquisition process for inputting a learning image of a new class to be newly added into a learning model that classifies an input image into any of the existing classes, and acquiring a first feature amount of the learning image; a pseudo image generation process for calculating a gradient that minimizes the distance between a second feature amount representative of each existing class and the first feature amount, and generating a pseudo image of the existing class based on the gradient; and executing a learning process to re-learn the learning model using the learning images as learning data for the new class and the pseudo-images of each existing class as learning data for that existing class.

6. 6. A program for operating the information processing device according to claim 5, the program causing the at least one processor to execute the first feature amount acquisition process, the pseudo image generation process, and the learning process.

Images