Machine learning equipment, machine learning methods, and machine learning programs
The machine learning device and method address catastrophic forgetting by generating and optimizing semantic vectors as ground truth, allowing new classes to be learned without semantic information, thus maintaining base class knowledge and enhancing learning accuracy.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- JVC KENWOOD CORP
- Filing Date
- 2025-08-26
- Publication Date
- 2026-06-30
AI Technical Summary
Existing machine learning models, particularly convolutional neural networks (CNNs), suffer from catastrophic forgetting, where learning new tasks leads to forgetting old tasks, and methods like incremental learning and few-shot learning struggle when new classes lack semantic information.
A machine learning device and method that uses a feature extraction unit, semantic vector generation, mapping, and optimization to generate and utilize semantic vectors as ground truth, even when semantic information is absent, by minimizing the distance between estimated and correct semantic vectors.
Enables learning new classes without semantic information, maintaining knowledge of base classes, and facilitating continuous learning with improved accuracy.
Smart Images
Figure 0007882404000001 
Figure 0007882404000002 
Figure 0007882404000003
Abstract
Description
Technical Field
[0003] , , , , ,
[0004] , , , , , , , , , ,
[0001] The present invention relates to machine learning technology.
Background Art
[0002] Humans can learn new knowledge through long-term experience and can maintain the old knowledge without forgetting it. On the other hand, the knowledge of a convolutional neural network (CNN) depends on the dataset used for learning, and in order to adapt to changes in the data distribution, it is necessary to relearn the parameters of the CNN for the entire dataset. In CNN, as learning progresses for a new task, the estimation accuracy for the old task decreases. Thus, in CNN, when continuous learning is performed, catastrophic forgetting, in which the learning results of the old task are forgotten during the learning of the new task, cannot be avoided. As a method for avoiding catastrophic forgetting, incremental learning or continual learning has been proposed. Incremental learning or continual learning is a learning method in which, when a new task or new data occurs, instead of learning the model from the beginning, the currently learned model is improved and learned. On the other hand, since new tasks often have only a small number of sample data available, few-shot learning has been proposed as a method for efficiently learning with a small amount of teacher data. In few-shot learning, without relearning the parameters learned once,
[0003]
[0004]
[0005] Learn a new task using a small number of parameters.
[0005] Learn a new class without fatal forgetting the learning results of the base class. Continuous learning and small-shot learning, which involves learning a small number of new classes compared to the basic class. This combination is called Incremental Few Shot Learning (IFSL). A method has been proposed (Non-Patent Document 1). In continuous small-shot learning, the base class Therefore, we train from a large dataset, and for new classes, we use a small amount of sample data. You can learn from this. [Prior art documents] [Non-patent literature]
[0006] [Non-Patent Document 1] Cheraghian, A., Rahman, S., Fang, P., Roy, SK, Petersson, L., & Harandi, M. (2021). Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learning. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 2534-2543). [Overview of the project] [Problems that the invention aims to solve]
[0007] As a continuous small-shot learning method, SaKD (Semantic-aware Kno) is described in Non-Patent Document 1. There is a method called SaKD (Wledge Distillation). In continuous small-shot learning, each class Semantic information is used as ground truth (correct data) for image classification. It will be used in the school. Generally, during pre-training of the basic class, semantic information is attached. Image datasets can be used, but the images used for training new classes will have semantics. Sometimes the class information is not provided. In SaKD, to learn a new class Semantic information corresponding to the new class of images is needed as ground truth data, and sema A challenge was that the model could not learn from images that lacked tactile information.
[0008] This invention has been made in view of these circumstances, and its purpose is to provide semantic information The objective is to provide a machine learning technique that can learn from new classes that do not have a corresponding class. . [Means for solving the problem]
[0009] To solve the above problems, a machine learning device according to one aspect of the present invention uses features from input data. A feature extraction unit that extracts vectors, and a semantic information attached to the input data A semantic vector generation unit that generates semantic vectors and the input data A pre-metatrained semantic vector is generated from the aforementioned feature vector. A tick prediction unit and a semantic vector are generated from the feature vector of the input data. This consists of a mapping unit that has already learned the base class, and when learning a new class, the new class If semantic information is not attached to the input data, the semantic prediction The semantic vector generated by the unit is used as the correct semantic vector. The semantic vector generated by the mapping unit and the correct semantic Optimization for optimizing the parameters of the mapping unit so that the distance of the cube vector is minimized It includes a section.
[0010] Another aspect of the present invention is a machine learning method. This method includes a feature extraction step of extracting a feature vector from input data, a semantic vector generation step of generating a semantic vector from semantic information given to the input data, a semantic prediction step of generating a semantic vector from the feature vector of the input data using a pre-metalearned semantic prediction module, a mapping step of generating a semantic vector from the feature vector of the input data using a mapping module learned for a base class, and when semantic information is not given to the input data of the new class during learning of the new class, using the semantic vector generated by the semantic prediction step as the correct semantic vector, and an optimization step of optimizing the parameters of the mapping module so that the distance between the semantic vector generated by the mapping step and the correct semantic vector is minimized. and a semantic vector generation step of generating a semantic vector from the semantic information given to the input data, a semantic prediction step of generating a semantic vector from the feature vector of the input data using a pre-metalearned semantic prediction module, a mapping step of generating a semantic vector from the feature vector of the input data using a mapping module learned for a base class, and when semantic information is not given to the input data of the new class during learning of the new class, using the semantic vector generated by the semantic prediction step as the correct semantic vector, and an optimization step of optimizing the parameters of the mapping module so that the distance between the semantic vector generated by the mapping step and the correct semantic vector is minimized. and a semantic prediction step of generating a semantic vector from the feature vector of the input data using a pre-metalearned semantic prediction module, a mapping step of generating a semantic vector from the feature vector of the input data using a mapping module learned for a base class, and when semantic information is not given to the input data of the new class during learning of the new class, using the semantic vector generated by the semantic prediction step as the correct semantic vector, and an optimization step of optimizing the parameters of the mapping module so that the distance between the semantic vector generated by the mapping step and the correct semantic vector is minimized. and a semantic prediction step of generating a semantic vector from the feature vector of the input data using a pre-metalearned semantic prediction module, a mapping step of generating a semantic vector from the feature vector of the input data using a mapping module learned for a base class, and when semantic information is not given to the input data of the new class during learning of the new class, using the semantic vector generated by the semantic prediction step as the correct semantic vector, and an optimization step of optimizing the parameters of the mapping module so that the distance between the semantic vector generated by the mapping step and the correct semantic vector is minimized. and a semantic prediction step of generating a semantic vector from the feature vector of the input data using a pre-metalearned semantic prediction module, a mapping step of generating a semantic vector from the feature vector of the input data using a mapping module learned for a base class, and when semantic information is not given to the input data of the new class during learning of the new class, using the semantic vector generated by the semantic prediction step as the correct semantic vector, and an optimization step of optimizing the parameters of the mapping module so that the distance between the semantic vector generated by the mapping step and the correct semantic vector is minimized. and a mapping step of generating a semantic vector from the feature vector of the input data using a mapping module learned for a base class, and when semantic information is not given to the input data of the new class during learning of the new class, using the semantic vector generated by the semantic prediction step as the correct semantic vector, and an optimization step of optimizing the parameters of the mapping module so that the distance between the semantic vector generated by the mapping step and the correct semantic vector is minimized. and when semantic information is not given to the input data of the new class during learning of the new class, using the semantic vector generated by the semantic prediction step as the correct semantic vector, and an optimization step of optimizing the parameters of the mapping module so that the distance between the semantic vector generated by the mapping step and the correct semantic vector is minimized. and when semantic information is not given to the input data of the new class during learning of the new class, using the semantic vector generated by the semantic prediction step as the correct semantic vector, and an optimization step of optimizing the parameters of the mapping module so that the distance between the semantic vector generated by the mapping step and the correct semantic vector is minimized. and using the semantic vector generated by the semantic prediction step as the correct semantic vector, and an optimization step of optimizing the parameters of the mapping module so that the distance between the semantic vector generated by the mapping step and the correct semantic vector is minimized. and using the semantic vector generated by the semantic prediction step as the correct semantic vector, and an optimization step of optimizing the parameters of the mapping module so that the distance between the semantic vector generated by the mapping step and the correct semantic vector is minimized. and an optimization step of optimizing the parameters of the mapping module so that the distance between the semantic vector generated by the mapping step and the correct semantic vector is minimized. It includes an optimization step.
[0011] In addition, any combination of the above components, and those obtained by converting the expression of the present invention among a method, an apparatus, a system, a recording medium, a computer program, etc. are also effective as aspects of the present invention. and those obtained by converting the expression of the present invention among a method, an apparatus, a system, a recording medium, a computer program, etc. are also effective as aspects of the present invention. is valid.
Effect of the Invention
[0012] According to the present invention, it is possible to provide a machine learning technique capable of learning a new class without semantic information. It is possible to provide a machine learning technique capable of learning a new class without semantic information. [Brief explanation of the drawing]
[0013] [Figure 1] This is a diagram showing the configuration of a conventional machine learning system. [Figure 2A] This diagram illustrates the configuration and operation of the machine learning device of this embodiment during base class training. [Figure 2B] This figure illustrates the configuration and operation of the machine learning device of this embodiment during training of a pseudo-small number of shot classes. [Figure 2C] This diagram illustrates the configuration and operation of the machine learning apparatus of this embodiment during the training of a new class. [Figure 3] This is a flowchart illustrating the continuous small-shot learning procedure using the machine learning device of this embodiment. [Modes for carrying out the invention]
[0014] Figure 1 is a diagram of the configuration of a conventional machine learning device 100. The machine learning device 100 is a semantic A hack vector generation unit 110, a feature extraction unit 120, a mapping unit 130, and an optimization unit. Includes 140.
[0015] In SaKD, when training the base class and when training new classes, the system applies to the input image. It is assumed that mantic information is provided as the correct data. The metric information, for example, in the case of a cat image, includes the cat's color, gender, etc. This is text information.
[0016] During base class training, the base class images and their semantic information are used for machine learning. This is input to device 100.
[0017] The semantic vector generation unit 110 generates semantic information l from the base class image. Convert the data into a semantic vector s, and use the semantic vector s as the ground truth data. It is applied to the optimization unit 140.
[0018] The feature extraction unit 120 extracts feature vectors g from the base class image x and maps them. Give to part 130.
[0019] The mapping unit 130 converts the feature vector g of the base class image x into a semantic vector The function y is inferred, and the semantic vector y is provided to the optimization unit 140.
[0020] The optimization unit 140 uses the inferred semantic vector y of the base class and the ground truth semantic vector The distance of the s-thick vector in semantic space is calculated as the loss, and the loss is minimized. The parameters of the feature extraction unit 120 and the mapping unit 130 are optimized accordingly.
[0021] Similarly, when training a new class, the images of the new class and their semantic information are used by the machine This is input to the learning device 100.
[0022] The semantic vector generation unit 110 generates semantic information l of a new class of image Convert to a semantic vector s, and use the semantic vector s as the optimal ground truth data. Provide to chemical part 140.
[0023] The feature extraction unit 120 extracts feature vectors g from the new class image x, and the mapping unit Give it to 130.
[0024] The mapping unit 130 converts the feature vector g of the new class image x into a semantic vector The toll y is inferred, and the semantic vector y is provided to the optimization unit 140.
[0025] The optimization unit 140 compares the inferred semantic vector y of the new class with the ground truth semantic vector The distance of the tick vector s in semantic space is calculated as the loss, and the loss is minimized. The parameters of the feature extraction unit 120 and the mapping unit 130 are optimized accordingly.
[0026] Figures 2A to 2C are configuration diagrams of a machine learning device 200 according to an embodiment of the present invention. The machine learning device 200 includes a semantic vector generation unit 210, a feature extraction unit 220, and a mapper. It includes a processing unit 230, an optimization unit 240, and a semantic prediction unit 250.
[0027] Here, we will take images as an example of data input to the machine learning device 200, The input data can be any data, not just images.
[0028] Figure 2A is a diagram illustrating the configuration and operation of the machine learning device 200 during base class training. ru.
[0029] During base class training, the base class images and their semantic information are used for machine learning. The input is sent to device 200. The operation of the base class during training is the same as that of the conventional machine learning device 100. This is similar to learning the base class.
[0030] The semantic vector generation unit 210 generates semantic information l from the base class image. Convert the data into a semantic vector s, and use the semantic vector s as the ground truth data. Provide to the optimization unit 240.
[0031] The feature extraction unit 220 extracts feature vectors g from the base class image x and maps them. Provided to part 230.
[0032] The mapping unit 230 converts the feature vector g of the base class to a semantic vector y The system infers the value and provides the semantic vector y to the optimization unit 240.
[0033] The optimization unit 240 compares the estimated semantic vector y of the base class with the ground truth semantic vector The distance of the s-thick vector in semantic space is calculated as the loss, and the loss is minimized. The parameters of the feature extraction unit 220 and the mapping unit 230 are optimized to achieve this.
[0034] Figure 2B illustrates the configuration and operation of the machine learning device 200 during training of a pseudo-small number of shots. This is a diagram showing the parameters of the feature extraction unit 220 that have learned the base class from Figure 2A. The meter is used in a fixed position.
[0035] Generate images of a pseudo-small number of shots class from a base class. For example, randomly 5 Select each base class image individually and create an episodic version of the pseudo-small number of shots class image. The formula is sequentially input into the machine learning device 200 to meta-learn a pseudo-small number of shot classes.
[0036] During meta-training of the pseudo-small number of shot class, the images of the pseudo-small number of shot class and their semaphores The metric information is input to the machine learning device 200.
[0037] The semantic vector generation unit 210 generates semantics for images of the pseudo-small number of shots class. Convert the data l into a semantic vector s, and then convert the semantic vector s into the correct data. This is provided to the optimization unit 240.
[0038] The feature extraction unit 220 extracts feature vectors g from the pseudo-small number of shot class images x, This is provided to the semantic prediction unit 250.
[0039] The semantic prediction unit 250 is a module similar to the mapping unit 230, and semantics The initial parameters of the tic prediction unit 250 are set by the mapping unit 2, which has already learned the base class. Use 30 parameters.
[0040] The semantic prediction unit 250 uses the feature vector g of the pseudo-small number of shots to determine the semantics The tick vector y is inferred, and the semantic vector y is provided to the optimization unit 240.
[0041] The optimization unit 240 uses the estimated semantic vector y of the pseudo-small number of shots and The distance in semantic space of the correct semantic vector s is calculated as the loss, The parameters of the semantic prediction unit 250 are optimized to minimize errors. Feature extraction Section 220 fixes the parameters so as not to forget the base class knowledge, Here, the parameters are not optimized. As a result, the semantic prediction unit 250 is pseudo-minority It is pre-trained using shot classes.
[0042] The loss function during meta-learning is the semantic output from the semantic prediction unit 250. The estimated vector y and the semantic vector output from the semantic vector generation unit 210 We use the cosine distance of the correct vector s and minimize this cosine distance, that is, The learning process aims to make the semantic estimate vector y approximate the semantic ground truth vector s. Proceed.
[0043] Figure 2C is a diagram illustrating the configuration and operation of the machine learning device 200 during training of a new class. In Figure 2C, the parameters of the feature extraction unit 220, which has already learned the base class from Figure 2A, are fixed. To use as such.
[0044] Images of new classes may not have semantic information attached. Here, This section explains how to train images of a new class that does not have semantic information attached.
[0045] When training a new class, images of the new class are input to the machine learning device 200, as shown in Figure 2B. The pre-meta-trained semantic prediction unit 250 performs semantic analysis from images of a new class. It is used to predict data.
[0046] The feature extraction unit 220 extracts feature vectors g from the new class image x, and the mapping unit This is provided to 230 and the semantic prediction unit 250.
[0047] The semantic prediction unit 250 uses feature vectors g extracted from the image x of a new class. Then, we predict the semantic vector s and use the semantic vector s as the ground truth data. Provide to the optimization unit 240.
[0048] The mapping unit 230 converts the semantic vector y from the feature vector g of the new class. The system performs inference and provides the semantic vector y to the optimization unit 240.
[0049] The optimization unit 240 uses the estimated semantic vector y of the new class and semantic The semantic of the correct semantic vector s predicted by the prediction unit 250 The spatial distance is calculated as a loss, and the parameters of the mapping unit 230 are adjusted to minimize the loss. The data is optimized. The feature extraction unit 220 adjusts parameters so as not to forget the base class knowledge. Since the data is fixed, we do not optimize the parameters here. This allows for mapping Section 230 is fine-tuned using a new class.
[0050] If semantic information is attached to the image of the new class, see Figure 2A for details. Using this configuration, the semantic vector generation unit 210 generates semantic vectors for images of a new class. The correct semantic vector is generated from the data, and the same learning process as the base class is performed. This configuration allows for the presence or absence of semantic information corresponding to the new class, regardless of the type of semantic information. This enables learning and inference for new classes.
[0051] Figure 3 illustrates the continuous small-shot learning procedure using the machine learning device 200 of this embodiment. This is a flowchart.
[0052] Images of a new class are input to the machine learning device 200 (S10). The feature extraction unit 220 Feature vectors are extracted from the images of the new class (S20).
[0053] The mapping unit 230 estimates the semantic vector from the feature vectors of the new class of images. Generate Tor (S30).
[0054] If semantic information is attached to the image of the new class (Y in S40), semantic The tick vector generation unit 210 generates the correct semantic vector from the semantic information of the new class of images. Generate the tick vector (S50).
[0055] If the image of the new class does not have semantic information attached (N in S40), then the semantics The semantic prediction unit 250 calculates the correct semantic vector from the feature vectors of images of a new class. Predict Tor (S60).
[0056] The optimization unit 240 calculates the distance between the estimated semantic vector and the correct semantic vector. The parameters of the mapping unit 230 are optimized so that the value is minimized (S70).
[0057] The various processes of the machine learning device 200 described above utilize hardware such as the CPU and memory. Of course, it can be realized using ROM (Read-Only Memory). Firmware stored in flash memory, etc., and software on computers, etc. This can also be achieved through wearables. Specifically, through firmware programs and software. The program may also be provided by recording it on a recording medium that can be read by a computer, etc. Alternatively, data can be sent and received to and from the server via a wireless network, or via terrestrial or satellite radio. It is also possible to transmit and receive data broadcasts as digital broadcasting.
[0058] As described above, according to the machine learning device 200 of this embodiment, doubts arise from the base class. Generate a pseudo-minority shot class and extract semantics from the input images of the pseudo-minority shot class. The semantic prediction unit, which predicts information, is pre-trained using meta-training. A small number of new classes are trained. When doing so, the semantic prediction information generated by the meta-learned semantic prediction unit is used as the correct answer. It is used as data to continuously learn a new class. This allows it to acquire semantic information. This enables learning and inference for new classes.
[0059] The present invention has been described above based on embodiments. The embodiments are illustrative and their respective components The fact that various variations are possible in the combination of constituent elements and each processing process, and such variations Those skilled in the art will understand that this also falls within the scope of the present invention. [Explanation of Symbols]
[0060] 100 Machine learning device, 110 Semantic vector generation unit, 120 Feature extraction unit Output unit, 130 Mapping unit, 140 Optimization unit, 200 Machine learning device, 21 0 Semantic vector generation unit, 220 Feature extraction unit, 230 Mapping unit, 240 Optimization unit, 250 Semantic prediction unit.
Claims
1. A feature extraction unit that extracts feature vectors from the input data, A semantic vector is generated from the feature vector of the input data, in advance The learned semantic prediction unit, A base-based method for generating semantic vectors from the feature vectors of the input data. A mapping unit that has already learned the data, The semantic vector generated by the semantic prediction unit is used as the correct semantic vector. The semantic vector generated by the mapping unit is used as the tick vector. The distance between the `L` and the aforementioned correct semantic vector is minimized in the `parameters` of the mapping unit. A machine learning device characterized by including an optimization unit that optimizes a meter.
2. A feature extraction step that extracts feature vectors from the input data, Using a pre-metatrained semantic prediction module, the input data A semantic prediction step that generates a semantic vector from a feature vector, Using a mapping module that has already learned the base class, the characteristics of the input data A mapping step that generates a semantic vector from a vector, The semantic vector generated by the semantic prediction step is the correct answer. As semantic vectors, the semantic vector generated by the mapping step The mapper is set such that the distance between the ick vector and the correct semantic vector is minimized. A machine characterized by including an optimization step of optimizing the parameters of the ing module. Learning methods.
3. A feature extraction step that extracts feature vectors from the input data, Using a pre-metatrained semantic prediction module, the input data A semantic prediction step that generates a semantic vector from a feature vector, Using a mapping module that has already learned the base class, the characteristics of the input data A mapping step that generates a semantic vector from a vector, The semantic vector generated by the semantic prediction step is the correct answer. As semantic vectors, the semantic vector generated by the mapping step The mapper is set such that the distance between the ick vector and the correct semantic vector is minimized. The computer is made to perform an optimization step to optimize the parameters of the ng module. A machine learning program characterized by the following features.