A program, apparatus, and method for generating augmented data using ground truth label data and unlabeled data.
The method addresses overfitting and bias in machine learning models by combining ground truth and pseudo-label data with confidence-based mixing ratios, enhancing accuracy and reducing annotation costs.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-09-20
- Publication Date
- 2026-04-02
AI Technical Summary
Existing machine learning models face challenges with overfitting and low accuracy due to limited labeled data, and the use of pseudo-labels can introduce bias and hallucinations, especially when relying on large language models for unlabeled data annotation.
A method that combines ground truth label data with pseudo-label data by estimating confidence levels and adjusting mixing ratios based on these confidence levels, using a computer program to generate augmented data with high accuracy.
Generates highly accurate augmented data by balancing the contribution of ground truth and pseudo-label data, reducing overfitting and bias, even with limited labeled data, and eliminating the need for manual annotation.
Smart Images
Figure 2026057076000001_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a technique for augmenting teacher data in a machine learning model.
Background Art
[0002] Currently, machine learning models for various data domains and tasks such as text, images, and audio are being constructed using a large amount of teacher data that can be collected via the Web. Generally, to construct a machine learning model, a certain amount of correct label data related to the target task is required. However, the quality of the correct label data serving as teacher data affects the accuracy of the machine learning model. In particular, when there is little correct label data, overfitting may occur in the training of the machine learning model. In addition, in order to increase the amount of teacher data, an annotation operation of assigning labels to unlabeled data manually is required. However, this requires human knowledge and time, leading to high costs. In contrast, in order to increase the amount of high-quality teacher data, a semi-supervised learning technique for generating augmented data using correct label data and unlabeled data has attracted attention.
[0003] Conventionally, as semi-supervised learning, there is a technique called Mixup that generates augmented data by mixing correct label data and unlabeled data (see, for example, Non-Patent Document 1). There is also a technique for generating augmented data by mixing correct label data and pseudo-label data (see, for example, Non-Patent Document 2). This technique uses Mixup as semi-supervised learning and is called MixMatch. By training a machine learning model using the augmented data generated by this technique, overfitting can be suppressed while improving the generalization performance of the model.
[0004] Furthermore, another conventional technique involves using Large Language Models (LLMs) to assign pseudo-labels to unlabeled data. For example, using large language models such as ChatGPT® or Gemini®, natural language text can be input as unlabeled data, and appropriate pseudo-labels can be automatically output. This technique can sometimes achieve accuracy exceeding that of manual labeling by crowdworkers. In other words, pseudo-labels can be assigned to unlabeled data using large language models, and this pseudo-labeled data can be used as training data. [Prior art documents] [Non-patent literature]
[0005] [Non-Patent Document 1] H. Zhang, M. Cisse, YN Dauphin, and D. Lopez-Paz, “mixup: Beyond Empirical Risk Minimization,” Presented at the International Conference on Learning Representations, Feb. 2018. Accessed: Aug. 07, 2024. [online], [Retrieved September 13, 2024], Internet<URL:https: / / openreview.net / forum?id=r1Ddp1-Rb> [Non-Patent Document 2] D. Berthelot, N. Carlini, I. Goodfellow, A. Oliver, N. Papernot, and C. Raffel, “MixMatch: a holistic approach to semi-supervised learning,” in proceedings of the 33rd International Conference on Neural Information processing Systems, Red Hook, NY, USA: Curran Associates Inc., 2019, pp. 5049-5059., [online], [searched on September 13, Reiwa 6], Internet <URL:https: / / dl.acm.org / doi / 10.5555 / 3540261.<3541371>
Summary of the Invention
Problems to be Solved by the Invention
[0006] According to Mixup described in the above-mentioned Non-Patent Document 1, the labeled data and the unlabeled data are mixed at a random ratio. Also, according to MixMatch described in the above-mentioned Non-Patent Document 2, the pseudo labels assigned to the unlabeled data depend on the machine learning model trained by the labeled data. Furthermore, according to the prior art, the number of samples of the labeled data greatly affects the accuracy of the machine learning model. In an environment where only a small amount of labeled data can be collected, the machine learning model trained thereby tends to be overly adapted to the small amount of labeled data and has a low accuracy for unknown samples. Of course, it is also possible to simply expand the training data by mixing a small amount of labeled data and a large amount of pseudo-labeled data. However, as a result, the machine learning model may be biased towards the tendency of a small amount of labeled data, the accuracy of the overall pseudo-labeled data may decrease, and overfitting may occur.
[0007] As mentioned earlier, it is also possible to assign pseudo-labels to unlabeled data using large-scale language models. However, this can be affected by misinformation known as hallucination, raising concerns about its reliability. In particular, errors are more likely to occur in tasks that require advanced knowledge, and manual annotator checks are often necessary in the end.
[0008] Therefore, the present invention aims to provide a program, apparatus, and method for generating augmented data with the highest possible accuracy from unlabeled data using ground truth label data. [Means for solving the problem]
[0009] According to the present invention, in a program that causes a computer to function to generate augmented data by mixing a ground truth label dataset D containing the ground truth label probability P for each label k for each ground truth label data X, and a pseudo-label dataset E containing the pseudo-label probability Q for each label k for each unlabeled data U, A confidence estimation method that takes multiple ground truth label data X as input and estimates the confidence level PR_k for each label k, A reference sample value selection means for randomly selecting a reference sample value λ* within the range [0.5, 1], A reference label extraction means that multiplies the correct label probability P of each label k in the correct label data X by the complement of the reference sample value λ* (1-λ*), multiplies the pseudo-label probability Q of each label k in the unlabeled data U by the reference sample value λ*, and extracts reference labels Ypseudo that have high multiplied correct label probabilities and pseudo-label probabilities under predetermined conditions. A mixing ratio determination means that determines the mixing ratio of ground truth labeled data X and unlabeled data U based on the confidence level PR_k(Ypseudo) of the reference label Ypseudo, and It is characterized by enabling the computer to function in this way.
[0010] According to other embodiments of the program of the present invention, The reference label extraction means extracts a reference label Ypseudo that, under predetermined conditions, maximizes the sum of the correct label probability and the pseudo-label probability for each multiplied label k. It is also preferable to make the computer function in this way.
[0011] According to other embodiments of the program of the present invention, An augmented data generation means generates augmented data by mixing ground truth label data X randomly selected from ground truth label dataset D and unlabeled data U randomly selected from pseudo-label dataset E in a determined mixing ratio. It is also preferable to make the computer function in this way.
[0012] According to other embodiments of the program of the present invention, The means for selecting a reference sample value is: Randomly extract sample values λ within the range [0, 1], The sample value λ and the sample complement 1-λ are compared, and the larger one (λ or 1-λ) is selected as the reference sample value λ*. It is also preferable to make the computer function in this way.
[0013] According to other embodiments of the program of the present invention, A pseudo-label estimation means sends a prompt to a language model instructing it to assign a pseudo-label probability Q to each label k for each unlabeled data U, and receives the pseudo-label probability Q for the unlabeled data U from the language model. It is also preferable to make the computer function in this way.
[0014] According to other embodiments of the program of the present invention, The system further comprises a pseudo-label learning model means fpseudo, which is trained on a pseudo-label dataset E. The confidence estimation method inputs multiple ground truth label data X into the pseudo-label learning model fpseudo and estimates the confidence level PRpseudo_k (=PR_k) for each label k of the pseudo-label data. It is also preferable to cause the computer to function as described above.
[0015] According to another embodiment of the program of the present invention, it further has a correct label learning model means ftrue trained based on cross-validation by a correct label dataset D, The confidence estimation means inputs a plurality of correct label data X to the correct label learning model means ftrue and estimates the confidence PRtrue_k (= PR_k) for each label k of the correct label data. It is also preferable to cause the computer to function as described above.
[0016] According to another embodiment of the program of the present invention, The mixing ratio determination means presets a first predetermined threshold value γhigh and a second predetermined threshold value γlow, and in a first case where the confidence PR_k(Ypseudo) of the reference label Ypseudo is equal to or higher than the first predetermined threshold value γhigh, determines a mixing ratio higher than that of the correct label data X for the label-free data U, and in a second case where the confidence PR_k(Ypseudo) of the reference label Ypseudo is equal to or lower than the second predetermined threshold value γlow, determines a mixing ratio lower than that of the correct label data X for the label-free data U. It is also preferable to cause the computer to function as described above.
[0017] According to another embodiment of the program of the present invention, The mixing ratio determination means in a first case, sets the mixing ratio of the label-free data U and the pseudo-label probability Q as a reference sample value λ*, and sets the mixing ratio of the correct label data X and the correct label probability P as the complement 1 - λ* of the reference sample value λ*, and in a second case, sets the mixing ratio of the label-free data U and the pseudo-label probability Q as the complement 1 - λ* of the reference sample value λ*, and sets the mixing ratio of the correct label data X and the correct label probability P as the reference sample value λ*. It is also preferable to cause the computer to function as described above.
[0018] According to other embodiments of the program of the present invention, The pseudo-label learning model fpseudo, trained on the pseudo-label dataset E, The ground truth label learning model means ftrue is trained based on cross-validation using the ground truth label dataset D. It further possesses, The confidence estimation method is, Multiple ground truth label data X are input to the pseudo-label learning model fpseudo, and the confidence level PRpseudo_k for each label k of the pseudo-label data is estimated. Multiple ground truth label data X are input to the ground truth label learning model ftrue, and the confidence level PRtrue_k for each label k of the ground truth label data is estimated. The reference label extraction means is, The correct label probability P for each label k in the correct label data X is multiplied by the reference sample value λ*, and the pseudo-label probability Q for each label k in the unlabeled data U is multiplied by the complement of the reference sample value λ*, 1-λ*. Reference labels Ytrue, which have high correct and pseudo-label probabilities under predetermined conditions, are then extracted. The means for determining the mixing ratio is, In the first case, where the confidence level PRpseudo_k(Ypseudo) of the reference label Ypseudo is greater than or equal to the confidence level PRtrue_k(Ytrue) of the reference label Ytrue, a higher mixing ratio is determined for the unlabeled data U than for the ground truth labeled data X. In the second case, a lower mixing ratio is determined for the unlabeled data U than for the ground truth labeled data X. It is also preferable to make the computer function in this way.
[0019] According to the present invention, in an extended data generation device that generates extended data by mixing a ground truth label dataset D, which includes the ground truth label probability P for each label k, for each ground truth label data X, and a pseudo-label dataset E, which includes the pseudo-label probability Q for each label k, for each unlabeled data U, A confidence estimation method that takes the correct label data X as input and estimates the confidence level PR_k for each label k, A reference sample value selection means for randomly selecting a reference sample value λ* within the range [0.5, 1], A reference label extraction means that multiplies the correct label probability P of each label k in the correct label data X by the complement of the reference sample value λ* (1-λ*), multiplies the pseudo-label probability Q of each label k in the unlabeled data U by the reference sample value λ*, and extracts reference labels Ypseudo that have high multiplied correct label probabilities and pseudo-label probabilities under predetermined conditions. A mixing ratio determination means that determines the mixing ratio of ground truth labeled data X and unlabeled data U based on the confidence level PR_k(Ypseudo) of the reference label Ypseudo, and It is characterized by having the following features.
[0020] According to the present invention, in a method for generating extended data for a device that generates extended data by mixing a ground truth label dataset D, which includes the ground truth label probability P for each label k, for each ground truth label data X, and a pseudo-label dataset E, which includes the pseudo-label probability Q for each label k, for each unlabeled data U, The device is The first step is to estimate the confidence level PR_k for each label k in the ground truth label data X, The second step is to randomly select a reference sample value λ* in the range [0.5, 1], A third step involves multiplying the correct label probability P of each label k in the correct label data X by the complement of the reference sample value λ* (1-λ*), multiplying the pseudo-label probability Q of each label k in the unlabeled data U by the reference sample value λ*, and extracting reference labels Ypseudo that have high multiplied correct label probabilities and pseudo-label probabilities under predetermined conditions. The fourth step is to determine the mixing ratio of the ground truth labeled data X and the unlabeled data U based on the confidence level PR_k(Ypseudo) of the reference label Ypseudo. It is characterized by performing the following: [Effects of the Invention]
[0021] According to the program, apparatus, and method of the present invention, it is possible to generate highly accurate augmented data from unlabeled data using ground truth label data. [Brief explanation of the drawing]
[0022] [Figure 1] This is a functional configuration diagram of the extended data generation device according to the present invention. [Figure 2] This is an explanatory diagram of the pseudo-label estimation unit in the present invention. [Figure 3] This is a functional configuration diagram of the pseudo-label learning model unit in the present invention. [Figure 4] This is an explanatory diagram of the confidence estimation unit and the pseudo-label learning model unit in the present invention. [Figure 5] This is a functional configuration diagram of the present invention, which includes a correct label learning model unit. [Figure 6] This is an explanatory diagram of the reference sample value selection unit and the reference label extraction unit in the present invention. [Figure 7] This is an explanatory diagram of the mixing ratio determination unit and the extended data generation unit in the present invention. [Figure 8] This is a functional configuration diagram showing the correct label learning model unit and the pseudo-label learning model unit according to the present invention. [Figure 9] This is an explanatory diagram showing the first code of the process in the present invention. [Figure 10] This is an explanatory diagram showing a second code for the process in the present invention. [Modes for carrying out the invention]
[0023] Embodiments of the present invention will be described in detail below with reference to the drawings.
[0024] Figure 1 is a functional configuration diagram of the extended data generation device according to the present invention.
[0025] As shown in Figure 1, the augmented data generation device 1 generates augmented data using ground truth labeled data and unlabeled data. The augmented data generation device 1 of the present invention is based on precision-oriented semi-supervised learning. The extended data generation device 1 comprises a ground truth label dataset 101, a pseudo-label estimation unit 100, a pseudo-label dataset 102, a confidence estimation unit 11, a reference sample value selection unit 12, a reference label extraction unit 13, a mixing ratio determination unit 14, and an extended data generation unit 15. These functional components are realized by executing a program that enables the computer mounted on the device to function. Furthermore, the processing flow of these functional components can also be understood as the extended data generation method of the device.
[0026] [Correct Label Dataset 101] The ground truth label dataset D contains the ground truth label probability P (dependent variable) for each label k, for each ground truth label data X (explanatory variable). For example, let's consider multiple labels k as [World, Sports, Business, Technology], and represent the correct label probability P for each label k as a vector like [0.0, 1.0, 0.0, 0.0]. D = {(X, P)} P[k0, k1, k2, k3]=[World, Sports, Business, Technology] X1 = P1[0.0, 1.0, 0.0, 0.0] X² = P²[1.0, 0.0, 0.0, 0.0] This indicates that the correct label for the correct label data X1 is "Sports". It also indicates that the correct label for the correct label data X2 is "World".
[0027] [Pseudo-label estimation unit 100] The pseudo-label estimation unit 100 sends a "prompt" to the language model instructing it to assign a pseudo-label probability Q to each label k for each unlabeled data U. As shown in Figure 1, a large-scale language model 2 is used as the language model. In response, the pseudo-label estimation unit 100 receives the pseudo-label probability Q for the unlabeled data U from the language model. Here, "language model" refers to a learning model that has been trained by pre-inputting a large amount of training data of natural language. Specifically, it is a model that uses a neural network and the probability of word occurrence in natural language. When a language model receives a prompt to assign a pseudo-label to unlabeled data, it can return an appropriate pseudo-labeled response. The pseudo-label estimation unit 100 outputs a pseudo-label probability Q obtained from the language model to the pseudo-label dataset 102 for each unlabeled data point U.
[0028] [Pseudo-label dataset 102] The pseudo-labeled dataset 102 includes, for each unlabeled data point U (explanatory variable), the pseudo-label probability Q (dependent variable) for each label k. For example, the pseudo-label dataset 102 can be represented in the same way as the ground truth label dataset 101, as shown below. E={(U, Q)} Q[k0, k1, k2, k3]=[World, Sports, Business, Technology] U1=Q1[0.2, 0.5, 0.2, 0.1] U2 = Q2 [0.3, 0.2, 0.4, 0.1] U3 = Q3 [0.3, 0.7, 0.0, 0.0]
[0029] In the embodiments of the present invention, the pseudo-label dataset 102 is described as having been assigned pseudo-label probabilities Q using a language model, but of course, it is not limited to this, and the pseudo-label probabilities Q may be assigned by other methods.
[0030] Figure 2 is an explanatory diagram of the pseudo-label estimation unit in the present invention.
[0031] As shown in Figure 2, the language model is assumed to be an external large-scale language model 2 connected to the network. Here, the large-scale language model 2 (language model) estimates pseudo-labels from unlabeled data in zero shots without fine-tuning. In other words, no special prior training is required for the language model. Using the large-scale language model 2 in zero shots also leads to a reduction in the cost of using the model.
[0032] Large-scale language models 2 can be general-purpose, such as ChatGPT® or Gemini®, and are not limited to specific models. These are built by training deep neural networks with billions of parameters using large datasets. In particular, Transformers have improved inference accuracy for tasks such as text generation and language understanding, and are being used in a wide range of applications, including chatbots, search engines, and code generation.
[0033] As shown in Figure 2, the augmented data generator 1 sends each unlabeled data U to the large-scale language model 2, following, for example, the following prompt. In response, the large-scale language model 2 replies with a pseudo-label probability Q as its response. -------------------------------------------------------------------------------- [Prompt]-> For each sentence, estimate the probability (0-1.0) of classifying it into each of the following labels. Labels: World / Sports / Business / Technology -------------------------------------------------------------------------------- [Unlabeled data U1]-> Judo, first introduced to the Olympic Program in 1964, continues to showcase its global influence and cultural heritage. The sport's inclusion in the Games offers judoka from diverse nations a Prestigious platform to display their skills and embody the Principles of respect and discipline inherent in judo. <-[Response sentence Q1[k0, k1, k2, k3]] [0.2, 0.5, 0.2, 0.1] -------------------------------------------------------------------------------- [Unlabeled data U2]-> The Olympic Games often drive significant urban development in host cities, such as infrastructure improvements and economic boosts. However, the long-term impact includes challenges like maintaining facilities and managing financial burdens after the event. <-[Response sentence Q2[k0, k1, k2, k3]] [0.3, 0.2, 0.4, 0.1] -------------------------------------------------------------------------------- [Unlabeled data U3]-> Judo, a martial art rooted in Japan, is celebrated for its focus on technique, balance, and mental discipline. Competitors aim to master throws and holds, showcasing both physical skill and deep respect for tradition. <-[Response sentence Q3[k0, k1, k2, k3]] [0.3, 0.7, 0.0, 0.0] --------------------------------------------------------------------------------
[0034] [Confidence estimation unit 11] The confidence estimation unit 11 takes multiple ground truth label data X as input and estimates the confidence level PR_k for each label k. PR_k[k0, k1, k2, k3]=[World, Sports, Business, Technology]
[0035] Here, the confidence level PR_k represents the probability that the estimated label k matches the correct label k, given that multiple correct label data X for the correct label k are input. Specifically, if 30 correct label data X for the correct label "Sports" are input, and 21 of them are estimated to be "Sports", the confidence level would be 0.7 (= 21 / 30). The confidence score PR_k evaluates the quality of each label k in the pseudo-label dataset E relative to the ground truth label data X. Furthermore, the confidence estimation unit 11 takes multiple (all) correct label data X and correct label probability P as input and calculates the confidence PR_k by averaging the confidence of each correct label data for each label k.
[0036] According to embodiments of the present invention, there are two methods for estimating the confidence level PR_k of the ground truth label data X. <Estimation of confidence using the pseudo-label learning model fpseudo> <Estimation of confidence using the ground truth label learning model part ftrue> However, the confidence estimation unit 11 is not limited to this and may be estimated by other estimation methods. Furthermore, if the confidence estimation unit 11 estimates the confidence score PR_k without requiring a separate learning model unit, it may be, for example, precision. This is an evaluation metric for binary classification of positive / negative examples, and represents the probability that the positive example was correct among the data predicted as positive. Precision = Number of correct answers predicted as positive examples TP / (Number of correct answers predicted as positive examples TP + Number of incorrect answers FP) Precision is a value within the range of [0, 1], with a value closer to 1.0 being better. In other words, precision represents how well a machine learning model "fits" to the correct answer if the prediction is a positive example.
[0037] <Estimation of confidence using the pseudo-label learning model fpseudo> Figure 3 is a functional configuration diagram of the pseudo-label learning model unit in the present invention.
[0038] First, the ground truth label dataset 101 and the pseudo-label dataset 102 are output one sample at a time at random. Finally, the extended data generation unit 15 generates one extended data set W.
[0039] As shown in Figure 3, compared to Figure 1, it further includes a pseudo-label learning model unit 1021 (fpseudo). It should be noted that the present invention assumes that at least a ground truth label dataset D and a pseudo-label dataset E are provided, and therefore, in Figure 3, the pseudo-label estimation unit 100 and the large-scale language model 2 shown in Figure 1 are omitted.
[0040] The confidence estimation unit 11 inputs multiple correct label data X to the pseudo-label learning model unit 1021 and estimates the confidence level PRpseudo_k (=PR_k) for each label k of the pseudo-label data.
[0041] [Pseudo-label learning model section 1021] The pseudo-labeled learning model unit 1021 is a machine learning model (fpseudo) trained on the pseudo-labeled dataset E as a supervised learning method. In the training phase, the pseudo-label learning model unit 1021 is trained by inputting unlabeled data (explanatory variables) and pseudo-label probabilities (target variables). Then, in the estimation phase, the pseudo-label learning model unit 1021 is input with correct label data X (explanatory variables) and outputs the confidence level PRpseudo_k (target variable). Here, the pseudo-label probability estimated by the pseudo-label learning model unit 1021 is the confidence level PRpseudo_k (=PR_k). PRpseudo_k[k0, k1, k2, k3]=[World, Sports, Business, Technology]
[0042] Figure 4 is an explanatory diagram of the confidence estimation unit and the pseudo-label learning model unit in the present invention.
[0043] As shown in Figure 4, the confidence estimation unit 11 inputs multiple ground truth label data X to the pseudo-label learning model unit 1021 (fpseudo) and estimates the confidence level PRpseudo_k for each label k of the pseudo-label data. Correct label data X1 = Correct label probability P1 [0.0, 1.0, 0.0, 0.0] Correct label data X2 = Probability of correct label P2 [1.0, 0.0, 0.0, 0.0] -> Confidence level PRpseudo_k[0.2, 0.6, 0.1, 0.1] The reason for using the evaluation metric of the pseudo-label dataset E as the confidence level PR_k is to ensure that the pseudo-label data is the dominant combination when applying Mixup. Furthermore, the confidence estimation unit 11 takes multiple (all) correct label data X1 and X2 as input and calculates the confidence value PRpseudo_k by averaging the confidence values of each label k from the confidence values of each correct label data X1 and X2.
[0044] <Estimation of confidence using the ground truth label learning model part ftrue> Figure 5 is a functional configuration diagram of the present invention, which includes a correct label learning model unit. Furthermore, the present invention assumes that at least a ground truth label dataset D and a pseudo-label dataset E are provided, and therefore, in Figure 5, the pseudo-label estimation unit 100 and the large-scale language model 2 shown in Figure 1 are omitted.
[0045] According to Figure 5, compared to Figure 1, it further includes a ground truth label learning model unit 1011(ftrue). The confidence estimation unit 11 inputs multiple ground truth label data X into the ground truth label learning model unit 1011 and estimates the confidence level PRtrue_k (=PR_k) for each label k of the ground truth label data.
[0046] [Correct Label Learning Model Unit 1011] During the training phase, the ground truth label learning model unit 1011 is trained by inputting ground truth label data (explanatory variables) and ground truth label probabilities (dependent variables). Then, during the estimation phase, the ground truth label learning model unit 1011 is input with ground truth label data X (explanatory variables) and outputs the confidence score PRtrue_k (dependent variable).
[0047] Comparing Figure 3 and Figure 5, when the number of data points in the ground truth label dataset D is relatively small compared to the number of data points in the pseudo-label dataset E, it is preferable to use the pseudo-label learning model fpseudo as shown in Figure 3. On the other hand, when the number of data points in the ground truth label dataset D is relatively large compared to the number of data points in the pseudo-label dataset E, it is also acceptable to use the ground truth label learning model ftrue as shown in Figure 5.
[0048] The aforementioned ground truth label learning model unit 1011 and pseudo-label learning model unit 1021 may be, for example, a two-layer neural network consisting of an input layer, a hidden layer, and an output layer. The hidden layer may be, for example, a ReLU activation function, and the output layer may be, for example, a softmax function.
[0049] Figure 6 is an explanatory diagram of the reference sample value selection unit and the reference label extraction unit in the present invention.
[0050] [Reference sample value selection section 12] The reference sample value selection unit 12 randomly selects a "reference sample value λ*" within the range [0.5, 1].
[0051] Here, the reference sample value selection unit 12 performs the following two steps. (S121) First, a "sample value λ" in the range [0, 1] is randomly selected. For random selection, for example, a beta function β(α,α) may be used. β(α,α)=∫01t(α-1)(1-t)(α-1)dt The beta function is obtained by integrating the product of two identical functions, t(α-1) and (1-t)(α-1)t, over the range [0, 1]. This function is symmetrical at t=1 / 2. λ <- β(α,α)
[0052] (S122) Next, the sample value λ and the sample value complement 1-λ are compared, and the larger one (λ or 1-λ) is selected as the "reference sample value λ*". λ* <- max(λ, 1-λ) In this case, λ* will always be a value of 0.5 or greater. This allows us to perform a mixup so that either the pseudo-dataset E (obtained by multiplying λ* by the ground truth dataset D) becomes dominant. The selected reference sample value λ* is then output to the reference label extraction unit 13.
[0053] [Reference label extraction unit 13] The reference label extraction unit 13 performs the following three steps: (S131) First, multiply the correct label probability P of each label k in the correct label data X by the complement of the reference sample value λ*, which is 1-λ*. Here, assume that the reference sample value λ* is 0.9. (1-λ*)P=[0.0×0.1, 1.0×0.1, 0.0×0.1, 0.0×0.1] =[0.0, 0.1, 0.0, 0.0] (S132) Next, the pseudo-label probability Q for each label k in the unlabeled data U is multiplied by the reference sample value λ*. λ*Q=[0.2×0.9, 0.5×0.9, 0.2×0.9, 0.1×0.9] =[0.18, 0.45, 0.18, 0.09] (S133) Then, the "reference label Ypseudo" is extracted, whose sum of the multiplied correct label probability P and pseudo-label probability Q is the maximum value (under predetermined conditions). Ypseudo <- arg_max((1-λ*)P+λ*Q)) =arg_max([0.0, 0.1, 0.0, 0.0]+[0.18, 0.45, 0.18, 0.09]) =arg_max([0.18, 0.55, 0.18, 0.09]) *The maximum value is "0.55". Ypseudo <- 2 (k = 2nd: index of Sports) Here, we can generate augmented data in which pseudolabels are dominant by multiplying the pseudolabel probability Q by the reference sample value λ*. The reference label Ypseudo is then output to the mixing ratio determination unit 14.
[0054] Figure 7 is an explanatory diagram of the mixing ratio determination unit and the extended data generation unit in the present invention.
[0055] [Mixing ratio determination unit 14] The mixing ratio determination unit 14 determines the mixing ratio of the correct labeled data X and the unlabeled data U based on the confidence level PR_k(Ypseudo) (=PRpseudo_k(Ypseudo)) of the reference label Ypseudo. This prevents the generation of augmented data in which pseudo-labeled data with low confidence levels are dominant.
[0056] The mixing ratio determination unit 14 first sets a first predetermined threshold γhigh and a second predetermined threshold γlow. According to an embodiment of the present invention, for example, if the number of labels C (=4), γhigh and γlow are set as follows. γlow = 1 / C: When C = 4, γlow = 0.25 γhigh = (1 / C + 1) / 2 : When C = 4, γhigh = 0.625 By setting these predetermined thresholds, it is possible to appropriately distinguish between labels with low confidence levels PRpseudo_k(Ypseudo) and those with high confidence levels.
[0057] (First condition) In the first case, where the confidence level PRpseudo_k(Ypseudo) (=PR_k(Ypseudo)) of the reference label Ypseudo is greater than or equal to the first predetermined threshold γhigh, a higher mixing ratio is determined for the unlabeled data U than for the ground truth labeled data X. if γhigh ≦ PRpseudo_k(Ypseudo) λfinal <- λ* In the first case, the mixing ratio λfinal is determined as follows: Mixing ratio of unlabeled data U and pseudo-labeled probability Q: Reference sample value λ* Mixing ratio of ground truth label data X and ground truth label probability P: complement of λ* 1-λ* In other words, if the value is γhigh or higher, pseudo-labeled data is predominantly used in Mixup.
[0058] (Second condition) In the second case, when the confidence level PRpseudo_k(Ypseudo) (=PR_k(Ypseudo)) of the reference label Ypseudo is less than or equal to the second predetermined threshold γlow, a lower mixing ratio is determined for the unlabeled data U than for the ground truth labeled data X. else if γlow ≧ PRpseudo_k(Ypseudo) λfinal <- 1-λ* In the second case, the mixing ratio λfinal is determined as follows: Mixing ratio of unlabeled data U and pseudo-labeled probability Q: complement of λ* 1-λ* Mixing ratio of ground truth label data X and ground truth label probability P: Reference sample value λ* In other words, if the value is below γlow, the ground truth label data is predominantly used in Mixup.
[0059] (Third condition) In the third case other than the reference sample value λ*, the sample value λ selected by the reference sample value selection unit 12 is set as the mixed ratio of unlabeled data U and pseudo-label probability Q. λfinal <- λ In the third case, the mixing ratio λfinal is determined as follows: Mixing ratio of unlabeled data U and pseudo-labeled probability Q: Rample value λ Mixing ratio of ground truth label data X and ground truth label probability P: complement of λ 1-λ
[0060] Furthermore, in Non-Patent Document 2, the ground truth label data X and pseudo-label data U are mixed using randomly selected sample values λ and their complement 1-λ. However, since λ and 1-λ only determine randomness, the mixing ratio of the ground truth label data X and pseudo-label data U is also random, and neither data is predominantly mixed.
[0061] [Extended data generation unit 15] The extended data generation unit 15 generates extended data A by mixing ground truth label data X selected from the ground truth label dataset D and unlabeled data U selected from the pseudo-label dataset E at a determined mixing ratio λfinal. Extended data A = {M, R} Extended data itself: M <- (1-λfinal)·X+λfinal·U Extended label probability: R <- (1-λfinal)·P+λfinal·Q The extended data generation unit 15 generates extended data A by mixing the data itself and its label probabilities according to the mixing ratio λfinal. That is, extended data A is obtained by linearly interpolating the correct label data X and the pseudo-label data U using λfinal. In this case, if the confidence level of the pseudo-label data U is high, the extended data is generated so that the pseudo-label data is dominant. On the other hand, if the confidence level is low, the extended data is generated so that the correct label data is dominant.
[0062] The final generated augmented data A can increase the size of the training dataset, in addition to the ground truth label data D.
[0063] Figure 8 is a functional configuration diagram of the present invention, which includes a correct label learning model unit and a pseudo-label learning model unit.
[0064] As shown in Figure 8, which combines Figures 3 and 5 mentioned above, the augmented data generation device 1 has two learning models. • Pseudo-labeled learning model unit 1021 (fpseudo) trained with pseudo-labeled dataset E • Ground-label learning model unit 1011(ftrue) trained with ground-label dataset D
[0065] At this time, the confidence estimation unit 11 estimates the following two confidence levels. • Multiple ground truth label data X are input to the pseudo-label learning model unit 1021 (fpseudo), and the confidence level PRpseudo_k for each label k of the pseudo-label data is estimated. • Multiple ground truth label data X are input to the ground truth label learning model unit 1011(ftrue), and the confidence level PRtrue_k for each label k of the ground truth label data is estimated.
[0066] Furthermore, the reference label extraction unit 13 outputs two reference labels. The correct label probability P for each label k in the correct label data X is multiplied by the complement of the reference sample value λ*, 1-λ*. The pseudo-label probability Q for each label k in the unlabeled data U is multiplied by the reference sample value λ*. The reference label Ypseudo, which is the maximum value among the sum of the multiplied correct label probabilities and pseudo-label probabilities, is extracted. The correct label probability P for each label k in the correct label data X is multiplied by the reference sample value λ*. The pseudo-label probability Q for each label k in the unlabeled data U is multiplied by the complement of the reference sample value λ*, 1-λ*. The reference label Ytrue, which is the maximum value among the sum of the multiplied correct label probabilities and pseudo-label probabilities, is then extracted.
[0067] Based on this, the mixing ratio determination unit 14 determines the mixing ratio as follows. (Condition 1) In the first case, where the confidence level PRpseudo_k(Ypseudo) of the reference label Ypseudo is greater than or equal to the confidence level PRtrue_k(Ytrue) of the reference label Ytrue, a higher mixing ratio is determined for the unlabeled data U than for the ground truth labeled data X. if PRpseudo_k(Ypseudo) ≧ PRtrue_k(Ytrue) λfinal <- λ*
[0068] (Second condition) In the second case, a lower mixing ratio is determined for the unlabeled data U than for the ground truth labeled data X. λfinal <- 1-λ*
[0069] As shown in Figure 7, whether to increase the mixing ratio of pseudo-label data U in Mixup compared to the mixing ratio of ground truth label data X was determined by a threshold for the confidence level PRpseudo_k. In contrast, as shown in Figure 8, the confidence score PRtrue_k is derived for the ground truth label data as well. By comparing PRpseudo_k and PRtrue_k, the data is mixed in such a way that the mixing ratio increases for datasets where PR_k is higher.
[0070] Figure 9 is an explanatory diagram showing the first code of the process in the present invention. Figure 9 shows the code corresponding to Figure 3 mentioned above.
[0071] Figure 10 is an explanatory diagram showing a second code for the process in the present invention. Figure 10 shows the code corresponding to Figure 8 mentioned above.
[0072] In the explanation above, the "predetermined conditions" for extracting reference labels in the reference label extraction unit 13 were defined as the reference label whose sum of the multiplied correct label probability P and the pseudo-label probability Q is the maximum value, but this is not limited to this. For example, if the maximum value of the sum of the multiplied correct label probability P and the pseudo-label probability Q is smaller than a predetermined threshold, the selected set of correct label data X and unlabeled data U may not be used. Also, if there are multiple reference labels whose sum of the multiplied correct label probability P and the pseudo-label probability Q is greater than another predetermined threshold, multiple reference labels may be extracted for the same selected set of correct label data X and unlabeled data U.
[0073] As described in detail above, according to the program, apparatus, and method of the present invention, it is possible to generate augmented data with the highest possible accuracy from unlabeled data using ground truth label data.
[0074] According to the present invention, even in an environment where a small amount of correct label data and a large amount of false label data exist, augmented data can be generated by combining and mixing the reliability of the correct label data with the quantitative advantage of the false label data. Furthermore, according to the present invention, pseudo-labeled data can be generated by assigning pseudo-labels to unlabeled data using a large-scale language model. This eliminates the cost of annotation work. In addition, to address the hallucination problem of large-scale language models, augmented data can be generated by adjusting the mixing ratio of the pseudo-labeled data based on the confidence level of each label in the pseudo-labeled data. This means that the quality of the pseudo-labeled data does not depend on the number or quality of the ground truth label data. In other words, even in environments where the amount of ground truth label data is small, it is not possible to use them as training data to degrade the accuracy of the machine learning model.
[0075] Furthermore, this makes it possible to contribute to Goal 9 of the United Nations' Sustainable Development Goals (SDGs), "Build resilient infrastructure, promote sustainable industrialization and foster innovation," by, for example, "using ground-label data to generate the most accurate augmented data possible from unlabeled data."
[0076] Various changes, modifications, and omissions to the scope of the technical concept and viewpoint of the present invention can be readily made by those skilled in the art with respect to the various embodiments of the present invention described above. The above description is merely illustrative and is not intended to limit the present invention in any way. The present invention is limited only to what is limited by the claims and their equivalents. [Explanation of Symbols]
[0077] 1. Extended Data Generation Device 100 Pseudo-label estimation unit 101 Ground-Resolution Label Datasets 1011 Correct Label Learning Model Section 102 pseudo-labeled datasets 1021 Pseudo-label learning model section 11 Confidence estimation part 12. Reference Sample Value Selection Section 13. Reference label extraction unit 14 Mixing ratio determination section 15 Extended Data Generation Unit 2. Large-scale language models
Claims
1. In a program that causes a computer to function in order to generate augmented data by mixing a ground truth label dataset D containing the ground truth label probability P for each label k for each ground truth label data X, and a pseudo-label dataset E containing the pseudo-label probability Q for each label k for each unlabeled data U, A confidence estimation method that takes multiple ground truth label data X as input and estimates the confidence level PR_k for each label k, A reference sample value selection means for randomly selecting a reference sample value λ* within the range [0.5, 1], A reference label extraction means that multiplies the correct label probability P of each label k in the correct label data X by the complement of the reference sample value λ* (1-λ*), and multiplies the pseudo-label probability Q of each label k in the unlabeled data U by the reference sample value λ*, thereby extracting reference labels Ypseudo that have high correct label probabilities and pseudo-label probabilities under predetermined conditions. A mixing ratio determination means that determines the mixing ratio of ground truth labeled data X and unlabeled data U based on the confidence level PR_k(Ypseudo) of the reference label Ypseudo, and A program characterized by its ability to make a computer function.
2. The reference label extraction means extracts a reference label Ypseudo that, under predetermined conditions, maximizes the sum of the correct label probability and the pseudo-label probability for each multiplied label k. The program according to claim 1, characterized in that it causes the computer to function in such a way.
3. An augmented data generation means generates augmented data by mixing ground truth label data X randomly selected from ground truth label dataset D and unlabeled data U randomly selected from pseudo-label dataset E in a determined mixing ratio. The program according to claim 1, characterized in that it causes a computer to function.
4. The means for selecting a reference sample value is: Randomly extract sample values λ within the range [0, 1], The sample value λ and the sample complement 1-λ are compared, and the larger one (λ or 1-λ) is selected as the reference sample value λ*. The program according to claim 1, characterized in that it causes the computer to function in such a way.
5. A pseudo-label estimation means sends a prompt to a language model instructing it to assign a pseudo-label probability Q to each label k for each unlabeled data U, and receives the pseudo-label probability Q for the unlabeled data U from the language model. The program according to claim 1, characterized in that it causes a computer to function.
6. The system further comprises a pseudo-label learning model means fpseudo, which is trained on a pseudo-label dataset E. The confidence estimation means inputs multiple ground truth label data X into the pseudo-label learning model means fpseudo and estimates the confidence level PRpseudo_k (=PR_k) for each label k of the pseudo-label data. The program according to claim 1, characterized in that it causes the computer to function in such a way.
7. The system further comprises a ground truth label learning model ftrue, which is trained based on cross-validation using the ground truth label dataset D. The confidence estimation means inputs multiple ground truth label data X into the ground truth label learning model means ftrue and estimates the confidence level PRtrue_k (=PR_k) for each label k of the ground truth label data. The program according to claim 1, characterized in that it causes the computer to function in such a way.
8. The means for determining the mixing ratio is, A first predetermined threshold γhigh and a second predetermined threshold γlow are set in advance. In the first case, where the confidence level PR_k(Ypseudo) of the reference label Ypseudo is greater than or equal to a first predetermined threshold γhigh, a higher mixing ratio is determined for the unlabeled data U than for the ground truth labeled data X. In the second case, when the confidence level PR_k(Ypseudo) of the reference label Ypseudo is less than or equal to a second predetermined threshold γlow, a lower mixing ratio is determined for the unlabeled data U than for the ground truth labeled data X. The program according to claim 1, characterized in that it causes the computer to function in such a way.
9. The means for determining the mixing ratio is, In the first case, the mixing ratio of unlabeled data U and pseudo-label probability Q is the reference sample value λ*, and the mixing ratio of ground truth labeled data X and ground truth label probability P is the complement of the reference sample value λ*, 1-λ*. In the second case, the mixing ratio of unlabeled data U and pseudo-label probability Q is set to the complement of the reference sample value λ*, 1-λ*, and the mixing ratio of ground truth labeled data X and ground truth label probability P is set to the reference sample value λ*. The program according to claim 8, characterized in that it causes the computer to function in such a way.
10. A pseudo-label learning model means fpseudo, trained on a pseudo-label dataset E, The ground truth label learning model means f true is trained based on cross-validation using the ground truth label dataset D. It further possesses, The confidence estimation method is, Multiple ground truth label data X are input to the pseudo-label learning model fpseudo, and the confidence level PRpseudo_k for each label k of the pseudo-label data is estimated. Multiple ground truth label data X are input to the ground truth label learning model ftrue, and the confidence level PRtrue_k for each label k of the ground truth label data is estimated. The reference label extraction means is, The correct label probability P for each label k in the correct label data X is multiplied by the reference sample value λ*, and the pseudo-label probability Q for each label k in the unlabeled data U is multiplied by the complement of the reference sample value λ*, 1-λ*. Reference labels Ytrue, which have high correct and pseudo-label probabilities under predetermined conditions, are then extracted. The means for determining the mixing ratio is, In the first case, where the confidence level PRpseudo_k(Ypseudo) of the reference label Ypseudo is greater than or equal to the confidence level PRtrue_k(Ytrue) of the reference label Ytrue, a higher mixing ratio is determined for the unlabeled data U than for the ground truth labeled data X. In the second case, a lower mixing ratio is determined for the unlabeled data U than for the ground truth labeled data X. The program according to claim 1, characterized in that it causes the computer to function in such a way.
11. In an augmented data generation device that generates augmented data by mixing a ground truth label dataset D containing the ground truth label probability P for each label k for each ground truth label data X, and a pseudo-label dataset E containing the pseudo-label probability Q for each label k for each unlabeled data U, A confidence estimation method that takes multiple ground truth label data X as input and estimates the confidence level PR_k for each label k, A reference sample value selection means for randomly selecting a reference sample value λ* within the range [0.5, 1], A reference label extraction means that multiplies the correct label probability P of each label k in the correct label data X by the complement of the reference sample value λ* (1-λ*), multiplies the pseudo-label probability Q of each label k in the unlabeled data U by the reference sample value λ*, and extracts reference labels Ypseudo that have high multiplied correct label probabilities and pseudo-label probabilities under predetermined conditions. A mixing ratio determination means that determines the mixing ratio of ground truth labeled data X and unlabeled data U based on the confidence level PR_k(Ypseudo) of the reference label Ypseudo, and An extended data generation device characterized by having the following features.
12. In a method for generating augmented data for a device that generates augmented data by mixing a ground truth label dataset D containing the ground truth label probability P for each label k for each ground truth label data X, and a pseudo-label dataset E containing the pseudo-label probability Q for each label k for each unlabeled data U, The aforementioned device is The first step involves inputting multiple ground truth label data X and estimating the confidence level PR_k for each label k. The second step is to randomly select a reference sample value λ* in the range [0.5, 1], A third step involves multiplying the correct label probability P of each label k in the correct label data X by the complement of the reference sample value λ* (1-λ*), multiplying the pseudo-label probability Q of each label k in the unlabeled data U by the reference sample value λ*, and extracting reference labels Ypseudo that have high multiplied correct label probabilities and pseudo-label probabilities under predetermined conditions. The fourth step is to determine the mixing ratio of the ground truth labeled data X and the unlabeled data U based on the confidence level PR_k(Ypseudo) of the reference label Ypseudo. A method for generating extended data, characterized by performing the following.