Data generation system, data generation method and program

Abstract

The data generation system includes a control unit. The control unit, as a data acquisition unit (P1), acquires first data (136). The control unit, as a feature extraction unit (P2), extracts predetermined features from the first data (136) using a first machine learning model (131) that has been trained in advance using data different from the first data (136). The control unit, as an additional information generation unit (P3), generates additional information based on features extracted from a plurality of pieces of first data (136) and distance information between the features. The control unit, as a data generation unit (P5), generates second data (137) having the same format as the first data (136) based on the additional information.

Description

[Technical Field]

[0001] The present disclosure relates to a data generation system, a data generation method, and a program. [Background technology]

[0002] To obtain a machine learning model that can accurately classify and group objects to be identified, it is important to use a large amount of training data for training. However, depending on the field of the object to be trained, it may be difficult to prepare a large amount of training data. To address this issue, there is a technique for increasing the amount of training data by generating pseudo data similar to the object. Patent Document 1 discloses a technique for extracting multiple feature quantities from an object image of a detection target, and prioritizing invariant features that change little between object images to generate pseudo data for training. Furthermore, it is known that data imbalances between or within classes can lead to a decrease in the accuracy of the machine learning model. In response to this issue, Non-Patent Document 1 discloses a technique for increasing the amount of data while maintaining balance within a class. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2024-97598 [Non-patent literature]

[0004] [Non-Patent Document 1] Hase, N. et al., “Data augmentation for intra-class imbalance with generative adversarial network”, Proc. SPIE 11172, Fourteenth International Conference on Quality Control by Artificial Vision, July 16, 2019, 1117206 Summary of the Invention [Problem to be solved by the invention]

[0005] However, there are cases where there is little training data or gaps in the distribution of training data, particularly near the boundaries between groups of classification targets. With such training data, it is difficult to improve classification accuracy near the boundaries.

[0006] An object of the present disclosure is to provide a data generation system, a data generation method, and a program that can obtain training data for obtaining a machine learning model with more stable and improved classification accuracy. [Means for solving the problem]

[0007] In order to achieve the above object, the present disclosure provides: a data acquisition unit that acquires first data; a feature extraction unit that extracts predetermined features from the first data using a first machine learning model that has been trained in advance using data different from the first data; an additional information generating unit that generates additional information based on the feature amounts extracted from the plurality of first data and distance information between the feature amounts; a data generation unit that generates second data having the same format as the first data based on the additional information; A data generation system comprising: [Effects of the Invention]

[0008] According to the present disclosure, it is possible to obtain learning data for obtaining a machine learning model with more stable and improved classification accuracy. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 2 is a block diagram showing the functional configuration of a data processing device. [Figure 2A] FIG. 10 is a diagram schematically illustrating the distribution of learning data in a feature space. [Figure 2B] FIG. 10 is a diagram schematically illustrating the distribution of learning data in a feature space. [Figure 3A] FIG. 10 is a diagram schematically illustrating the distribution of learning data in a feature space. [Figure 3B] FIG. 10 is a diagram schematically illustrating the distribution of learning data in a feature space. [Figure 4] FIG. 1 is a diagram showing a flow of learning a machine learning model for object detection. [Figure 5] 10 is a flowchart showing a control procedure for additional data generation processing. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the functional configuration of a data processing device 1 according to this embodiment. The data processing device 1 serving as the data generation system of this embodiment may be an electronic calculator, i.e., a computer, a server device, etc. The data processing device 1 may include a control unit 11, a RAM 12 (Random Access Memory), a storage unit 13, a communication unit 14, a display unit 15, an operation reception unit 16, etc.

[0011] The control unit 11 has a hardware processor that controls the overall operation of the data processing device 1. The hardware processor may include, for example, a CPU 111 (Central Processing Unit) and a GPU 112 (Graphical Processing Unit). The CPU 111 and the GPU 112 perform arithmetic processing. The GPU 112 mainly performs processing for analyzing and generating image data.

[0012] The RAM 12 provides a working memory space for the control unit 11 and stores temporary data. The RAM 12 may be a DRAM or the like. The temporary data may include data to be processed, such as image data.

[0013] The storage unit 13 is a non-volatile memory and can continuously store data regardless of whether the data processing device 1 is operating. The storage unit 13 may be, for example, a flash memory or an HDD (Hard Disk Drive). The storage unit 13 does not have to be built into the data processing device 1. The storage unit 13 may be a peripheral device externally attached to the main body of the data processing device 1 including the control unit 11 via a connection line such as a USB (Universal Serial Bus) or a SATA (Serial ATA). Alternatively, the storage unit 13 may be another database server located on a LAN (Local Area Network) or the Internet, a network drive, cloud storage, or the like. The storage unit 13 stores a program 130, setting data, and the like.

[0014] The communication unit 14 controls data transmission and reception with external devices via a LAN or the Internet. The communication unit 14 includes a network card or the like, and enables data transmission and reception with external devices via wired and / or wireless communication. Wireless communication may include wireless LAN and / or Bluetooth (registered trademark).

[0015] The display unit 15 has a digital display screen, and displays various information on the digital display screen under the control of the control unit 11. The digital display screen may be, for example, a liquid crystal display or an organic EL (Electro-Luminescent) display.

[0016] The operation reception unit 16 receives an external input operation from a user or the like, and outputs an operation signal to the control unit 11. The operation reception unit 16 may include, for example, a keyboard, a switch, a pointing device such as a mouse or a touch panel, and the like. The display unit 15 and the operation reception unit 16 may be peripheral devices connected to the main body of the data processing device 1 via a USB, the communication unit 14, or the like. Alternatively, the data processing device 1 may not have the operation reception unit 16 and the display unit 15 except for a minimum of a power switch, a reset switch, and the like. In this case, an input operation received by an external terminal device may be acquired by the control unit 11 via the communication unit 14. The control unit 11 may output display control information for the display screen to the external terminal device and cause the display to be performed on the display screen of the terminal device.

[0017] Next, the functional operation of the data processing device 1 will be described. The data processing device 1 identifies a range of missing training data 135 based on existing first data 136 used for training a machine learning model related to target image recognition and detection. The data processing device 1 artificially generates second data 137 used for training the range. The format of the generated second data 137 is the same as that of the first data 136.

[0018] The second data 137 may be generated using, as additional information, features calculated by a first machine learning model 131 that has been trained in advance using data of the same format as the first data 136 but different from the first data 136. The features may be multidimensional vector data that are orthogonal to each other and may be standardized. That is, various norms indicating distance information between features in a feature space, such as Euclidean distance or Manhattan distance, may be used as an index of similarity. Furthermore, cosine similarity or the like may be used to evaluate similarity. Deep metric learning may be used in the first machine learning model 131 for calculating such features. In other words, in deep metric learning, an embedding space is defined in which the higher the similarity, the closer the features are located.

[0019] Correct labels of classification classes, i.e., classification type information, are associated with and set to a portion of the pre-stored training data 135, i.e., the first data 136. The proportion of the training data 135 to which correct labels are assigned may be small compared to the entire training data 135. For example, the proportion may be less than half of the total, or even less than 10%. As described above, multidimensional features, i.e., feature vectors, are calculated for each training data 135 by the first machine learning model 131. As an example, the training data 135 may be photographed images of an object such as a product for inspection, and the correct labels, i.e., the classes to be classified, may be the presence or absence of scratches, distortions, dents, protrusions, etc. on the part.

[0020] 2A, 2B, 3A, and 3B are diagrams schematically illustrating the distribution of training data 135 in a feature space. Here, the feature distribution is shown in a two-dimensional space; however, the feature space, i.e., the embedding space, may be three or more dimensions. Training data 135 assigned the same class by labeling are similar in at least some of the feature components. Therefore, training data 135 of the same class are located close together in a dimension-reduced space where all but the at least some components have disappeared or been reduced. For example, as shown in FIG. 2A, training data for classes A and B are located together at different positions in the feature space. Hereinafter, features projected into the dimension-reduced feature space will also be simply referred to as features. Representative values ​​of these features, such as the mean and variance, are calculated for each training data 135 of the same class to identify representative positions PA and PB for multiple classes. The mean value may be a simple average or a weighted average that takes into account variance, etc.

[0021] The features in this feature space can be used to classify the evaluation target using a machine learning model for image recognition. For example, convolutional neural networks can be used for mechanical classification. Data with features typical for each class can be easily classified with few errors.

[0022] On the other hand, there may be images whose feature values ​​fall between the representative positions of multiple classes. By using many images with feature values ​​of intermediate values ​​between the representative values ​​obtained for each of these classes as training data for training, the boundary between classes A and B can be more clearly defined. However, the number of training data 135 near such boundary positions is often insufficient. In one embodiment, the training data 135 is supplemented by additionally generating artificial data, i.e., second data 137, that includes such intermediate feature values ​​as additional information. For example, the second data 137 may be generated using the feature value at the midpoint PC of the line segment L connecting the representative positions PA and PB as additional information. Furthermore, multiple second data 137 may be generated at appropriate intervals along the line segment L, not just at the midpoint PC. Alternatively, as shown in FIG. 2B , multiple points may be dispersed around a specific point such as the midpoint PC using a random number or the like, and second data 137 may be generated for each of these points.

[0023] A machine learning model may be used to generate the second data 137, for example, to generate a photographed image of an object. The second machine learning model 132 for generating artificial data is given the above feature amounts as additional information, and is input with appropriate variance as necessary, thereby generating and outputting the second data 137. For example, an algorithm related to a GAN (Generative Adversarial Network) may be used for the second machine learning model 132. The first data 136 is used for training the second machine learning model 132. Unlabeled training data may also be used for training.

[0024] A label indicating a class may be attached to the generated second data 137. An expert or a person in charge who can identify classes may attach a label to a part of the obtained second data 137, which makes it easier to clearly define the boundaries of class classification.

[0025] On the other hand, as shown in FIG. 3A , a feature may be determined for a single first data 136 of class A so that second data 137 is generated in gaps in the distribution or around the first data 136. For example, the second data 137 may be generated by using, as additional information, feature values ​​at positions around the representative position PA where any of the feature values ​​of the first data 136 is farther away than a reference distance. The distance reference may be determined as an absolute value, or may be determined relatively based on a variance or standard deviation related to the variation of class A. One or more second data 137 may be generated by further varying the feature values ​​of this additional information using random numbers.

[0026] By expanding the learning data 135 from the specified range of class A to the periphery, it becomes possible to more clearly define the outer edge of the range of class A. Furthermore, if the number of data for class A is insufficient, the amount of learning data 135 for class A is increased, thereby improving the classification accuracy of class A.

[0027] 3B, second data 137 may be additionally generated in the feature space at a position PE within a range where the data distribution density is low, away from any of the first data 136, regardless of the class. The criteria for determining whether the distribution density is high or low may be determined from a relatively low distribution density based on the distribution density of each range obtained by dividing the embedding space into a mesh, or may be determined by an appropriate absolute value. Since it is impossible to determine which class a data will be classified into if there is no data, filling in the gaps in this way can more clearly define the class boundaries.

[0028] FIG. 4 is a diagram showing a flow of learning a machine learning model for object detection. First data 136 is prepared as the maximum number of training data possible, and some of the data is labeled (P1). The first data 136 is input to a first machine learning model 131, and each feature is extracted (P2).

[0029] Based on the feature amount of the obtained first data 136, additional information is set as a feature amount for second data 137 to be generated (P3). Meanwhile, the second machine learning model 132 is trained using the first data 136 to obtain a trained model for generating second data 137 (P4). The additional information is appropriately varied using random numbers or the like and input to the trained second machine learning model 132, thereby obtaining second data 137 (P5).

[0030] A machine learning model for detecting the detection target is trained using the first data 136 and the second data 137, and a trained model is obtained (P6).

[0031] FIG. 5 is a flowchart showing a control procedure by the control unit 11 in the additional data generation process. This additional data generation process is one embodiment of a data generation method, and may be called and started from the program 130 in response to a predetermined input operation by the user of the data processing device 1 to the operation reception unit 16, etc.

[0032] The control unit 11 functions as a data acquisition unit and acquires first data 136 (S1; data acquisition means). The first data 136 may be stored in advance in the storage unit 13. A part or all of the first data 136 may be acquired from an external device via the communication unit 14.

[0033] The control unit 11 sets a correct label to a part of the first data 136 (S2). For example, the control unit 11 may classify the first data 136, extract some typical data, and present the extracted data to an expert or the like who can set a correct label. In accordance with the content input by the expert or the like via the operation reception unit 16 or the like, the control unit 11 sets a label associated with the presented data. It should be noted that if the processing of step S2 has been performed in advance and the first data 136 with the label set has been stored in the storage unit 13, the processing content of step S2 may be omitted.

[0034] Control unit 11, as a feature extraction unit, inputs each of first data 136 into first machine learning model 131 to acquire features (S3; feature extraction means). As described above, first machine learning model 131 may be trained in advance using training data different from first data 136 acquired in step S1 so that features can be calculated.

[0035] The control unit 11 calculates a representative value of the feature amount of each class based on the feature amount of the first data 136 to which the correct label is set (S4). As described above, the representative value may be a simple average, a variance, or the like.

[0036] The control unit 11 trains the second machine learning model 132 for generating additional data using the first data 136 and the feature amount (S5). In this case, the first data 136 is used for training regardless of whether or not a label is set.

[0037] The control unit 11, as an additional information generating unit, sets the feature of the second data 137 to be added in the feature space as additional information based on the feature distribution of the stored first data 136 (S6; additional information generating means). As described above, the control unit 11 may determine the midpoint of the representative positions of multiple classes as the position of the second data 137. Alternatively, the control unit 11 may detect an area in the feature distribution where there is no first data 136 or where the distribution density of data is low, and determine the position of the second data 137. The position set in step S6 may be an approximate position.

[0038] The control unit 11, as a data generating unit, generates second data 137 (S7; data generating means). The control unit 11 generates second data 137 by the second machine learning model 132 using the feature amounts determined as described above or feature amounts that have been further varied by adding random numbers or the like. The feature amounts that include the variation are the final generation position of the second data 137.

[0039] The control unit 11 may set a label for a portion of the generated second data 137 (S8). The control unit 11 displays the generated second data 137 on the display unit 15, and determines a class by receiving an input operation from an expert or the like via the operation receiving unit 16. The control unit 11 associates the determined class with the additional data as a label. The control unit 11 adds the obtained second data 137 and the label associated with a portion of the second data 137 to the learning data 135 (S9). Then, the control unit 11 ends the additional data generation process.

[0040] By training a machine learning model for classification using the training data 135 obtained in this way, classification using the machine learning model, in this case classification of test results, becomes possible.

[0041] As described above, the data processing device 1 according to the data generation system of this embodiment includes a control unit 11. The control unit 11 functions as a data acquisition unit, acquiring first data 136. The control unit 11 functions as a feature extraction unit, extracting predetermined features from the first data 136 using a first machine learning model 131 previously trained using data different from the first data 136. The control unit 11 functions as an additional information generation unit, generating additional information based on features extracted from a plurality of first data 136 and distance information between the features. The control unit 11 functions as a data generation unit, generating second data 137 having the same format as the first data 136 based on the additional information. In this manner, the data processing device 1 generates the second data 137 at an appropriate position in the feature space using features, particularly features obtained by the machine learning model. Therefore, when machine learning data is insufficient, data can be efficiently supplemented to fill in the missing positions of the training data. This allows the data processing device 1 to obtain training data for obtaining a machine learning model with consistently improved classification accuracy.

[0042] Furthermore, the control unit 11 as a data generating unit may generate the second data 137 using the second machine learning model 132. Since the feature amount obtained by the first machine learning model 131 is used to generate the second data 137, using the machine learning model to generate the second data 137 also makes it easier to obtain data in which the features are appropriately reflected.

[0043] Furthermore, a class related to a predetermined classification may be associated with each part of the first data 136. The control unit 11 as the additional information generating unit may calculate a representative value of the feature amount related to the additional information for each piece of data belonging to the same class of the first data 136, and generate additional information located between the representative values. By generating additional data having intermediate features between the classes, the boundary between the feature amounts of the classes becomes easier to define more clearly.

[0044] Furthermore, the control unit 11 as the additional information generator may determine additional information located between representative values ​​based on a weighted average of representative values ​​of feature quantities related to multiple classes. By setting appropriate weights, data is appropriately added between data of multiple classes. Therefore, by including the additional data based on the additional information in the training data, classification and identification between the multiple classes can be more clearly determined.

[0045] Furthermore, the control unit 11 as the additional information generating unit may determine additional information whose feature amount is located a standard distance or more from the feature amount related to the first data 136. That is, by appropriately generating additional data for a portion where the first data 136 is lacking, it is possible to more efficiently and accurately improve the learning accuracy of the machine learning model.

[0046] Furthermore, the control unit 11 as the additional information generating unit may calculate, as a representative value, an average value of the feature amounts for each class in a portion of the first data 136. By determining the feature amounts of the additional data based on the average position in the feature amount space of each class, quantitatively suitable feature amounts of the additional data can be easily determined.

[0047] Furthermore, the control unit 11 as the additional information generating unit may determine additional information that is located in a range of distribution density below a standard in the distribution of feature values ​​within the first data 136 of the same class. In other words, by adding the second data 137 to a portion of the class where there is little data, it is possible to improve the reliability of the class range, especially the outer edge.

[0048] Furthermore, first machine learning model 131 may utilize deep metric learning. In deep metric learning, the distance between features directly serves as a quantitative index of similarity, and therefore first machine learning model 131 can obtain features suitable for determining the addition position of second data 137 to be added.

[0049] The additional information may also include some components of the features. Not all of the multidimensional features are necessary for class identification. Therefore, features that are unnecessary or of low importance for class identification may not be included in the additional information.

[0050] Furthermore, the control unit 11 as the additional information generator may compress the dimensions of the features and use them as additional information. That is, the control unit 11 may appropriately project multiple features, omit data of some dimensions, or reduce the weight and combine it with data of other dimensions. This makes it possible to more effectively use the features and generate additional second data 137 in areas where learning data is insufficient.

[0051] The data generation method by the control unit 11 of this embodiment includes the following steps: (1) acquiring first data 136; (2) extracting predetermined features from the first data 136 using a first machine learning model 131 previously trained using data different from the first data 136; (3) generating additional information based on the features extracted from the multiple first data 136 and distance information between the features; and (4) generating second data 137 in the same format as the first data 136 based on the additional information. This data generation method generates the second data 137 at an appropriate position in the feature space using features, particularly features obtained by the machine learning model. Therefore, even when there is little collectible data for machine learning, additional data can be generated to efficiently fill in positions where the training data is lacking, thereby compensating for the shortage. Therefore, this data generation method can obtain training data for obtaining a machine learning model with more consistently improved classification accuracy.

[0052] Furthermore, the program 130 relating to the data generation method may be installed and executed on a computer, which makes it possible to easily obtain data effective for training a machine learning model for classification without requiring a special hardware configuration.

[0053] The present disclosure is not limited to the above-described embodiment, and various modifications are possible. For example, in the above, the average position and variance are set as representative positions for each class, and additional information related to the second data 137 is set between these representative positions, but this is not limited to this. Additional information may be set simply based on the average position of all of a plurality of classes, or these may be combined. Furthermore, instead of the average position, a combination of medians for each component may be set as the representative position. Furthermore, the average position of representative values ​​for three or more classes may be used as the basis for setting additional information.

[0054] Furthermore, although random numbers are used to determine the variations in the second data 137 in the above example, random numbers do not have to be used. The multiple positions of the second data 137 may be set at equal intervals, for example.

[0055] Furthermore, in the above, multidimensional feature quantities are appropriately projected and used as additional information, but some or all of the components of the feature quantities may be used as additional information without being projected.

[0056] Although the above description has been given with reference to an example of generating data to clarify boundaries by expanding the main distribution range of each class, this is not limiting. If there is already some data near the current boundary, data near the boundary may be generated to further clarify the boundary.

[0057] In the above description, each piece of data is exclusively classified into one of the classes, but this is not limiting. Data may exist that is classified to belong to both of two adjacent groups near the boundary.

[0058] In the above description, deep metric learning is used for first machine learning model 131, but this is not limiting. Any algorithm capable of calculating a feature amount that allows similarity to be quantitatively evaluated may be used.

[0059] In the above description, only the first machine learning model 131 is used to extract the features, but this is not limiting. In addition to the features output by the first machine learning model 131, analytical and statistical features may also be used in combination.

[0060] In the above description, each process from calculating the feature amount from the first data 136 to generating the second data 137 is performed by a single data processing device 1, but some of the processes may be executed by other computers. That is, the processes may be distributed among multiple computers, or the same process may be performed in parallel by multiple computers, one for each part of the data.

[0061] Although the above description has been given using examples of scratches and dents on products or parts as detection targets, the detection targets are not limited to these. For example, the present invention may be used for detection targets in a wide range of fields, such as structural abnormalities in cells, blood vessels, and the digestive tract in medical tests, detection of human facial expressions, detection of traffic violations, and detection and classification of vehicle abnormalities. Furthermore, the detection source and learning data may be other than image data. For example, specific abnormal sounds or the cries of specific animals may be detected from sound data.

[0062] In the above description, the storage unit 13, which is composed of a nonvolatile memory such as an HDD or flash memory, has been used as an example of a computer-readable medium for storing the program 130 related to data generation of the present disclosure, but is not limited to this. Other computer-readable media may include other nonvolatile memories such as MRAM, and portable storage media such as CD-ROMs and DVD discs. Furthermore, a carrier wave may also be used as a medium for providing program data related to the present disclosure via a communication line. In addition, the specific configurations, contents and procedures of the processing operations, etc. shown in the above embodiments can be modified as appropriate without departing from the spirit of the present disclosure. The scope of the present invention includes the scope of the invention described in the claims and its equivalents. [Industrial Applicability]

[0063] The present disclosure can be used for a data generation system, a data generation method, and a program. [Explanation of symbols]

[0064] 1 Data processing device 11 Control section 111 CPU 112 GPU 12 RAM 13 Storage section 130 Programs 131 First Machine Learning Model 132 Second Machine Learning Model 135 training data 136 First Data 137 Second Data 14 Communications Department 15 Display section 16 Operation reception section

Claims

1. a data acquisition unit that acquires first data; a feature extraction unit that extracts predetermined features from the first data using a first machine learning model that has been trained in advance using data different from the first data; an additional information generating unit that generates additional information based on the feature amounts extracted from the plurality of first data and distance information between the feature amounts; a data generating unit that generates second data having the same format as the first data based on the additional information; A data generation system comprising:

2. The data generation system according to claim 1 , wherein the data generation unit generates the second data using a second machine learning model.

3. a classification type relating to a predetermined classification is associated with each part of the first data; the additional information generation unit calculates a representative value of a feature amount related to the additional information for each piece of data belonging to the same classification type of the first data, and generates the additional information located between the representative values. The data generation system of claim 1 .

4. The data generation system according to claim 3 , wherein the additional information generation unit determines the additional information located between the representative values ​​based on a weighted average of the representative values ​​of the feature amounts relating to a plurality of the classification types.

5. The data generation system according to claim 3 , wherein the additional information generation unit determines the additional information whose feature amount is located at a distance equal to or greater than a reference distance from the feature amount related to the first data.

6. The data generation system according to claim 3 , wherein the additional information generation unit calculates an average value of the feature amounts for each classification type in the portion of first data as the representative value.

7. The data generation system according to claim 4 , wherein the additional information generation unit determines the additional information located in a range of distribution density below a reference value in the distribution of the feature amount within the first data of the same classification type.

8. The data generation system of claim 1 , wherein the first machine learning model utilizes deep metric learning.

9. The data generation system according to claim 1 , wherein the additional information includes some components of the feature quantities.

10. The data generation system according to claim 1 , wherein the additional information generation unit compresses the dimensions of the feature quantities and uses the compressed dimensions as the additional information.

11. Acquire first data; extracting predetermined features from the first data using a first machine learning model that has been trained in advance using data different from the first data; generating additional information based on the feature amounts extracted from the plurality of first data and distance information between the feature amounts; generating second data in the same format as the first data based on the additional information; Data generation method.

12. Computer a data acquisition means for acquiring first data; a feature extraction means for extracting predetermined features from the first data using a first machine learning model that has been trained in advance using data different from the first data; additional information generating means for generating additional information based on the feature amounts extracted from the plurality of first data and distance information of the feature amounts between the plurality of first data; a data generating means for generating second data in the same format as the first data based on the additional information; A program that functions as a

Images