Data extension device, training system, data extension method, program, and machine learning model

Abstract

To provide a data extension technique corresponding to a dataset being an identification object.SOLUTION: One aspect of the present disclosure is related to a data extension device which includes: a distance calculation unit which calculates a first distance between first data of a first class in a dataset and first extension data extended from the first data and a second distance between the first extension data and second extension data extended from second data of a second class in the dataset; and a data extension unit which executes data extension on the dataset using a data extension parameter determined on the basis of the first distance and the second distance.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to a data augmentation device, a training system, a data augmentation method, a program, and a machine learning model. [Background technology]

[0002] With the recent advances in deep learning technology, machine learning models are being used in a wide range of technical fields. In supervised learning, machine learning models are trained using a dataset consisting of labeled training data. In order to properly train a machine learning model, it is necessary to prepare a dataset with a certain amount of data, and collecting such a dataset can be very costly.

[0003] In order to increase the number of data in a dataset, data augmentation may be performed on the collected dataset. For example, if the dataset is composed of image data, the image data of the dataset may be translated, rotated, blurred, or illuminated to generate augmented image data. By training a machine learning model using augmented data from the data of a given dataset in addition to the data of the dataset, the generalization performance of the machine learning model may be improved.

[0004] The augmented space defined by the data augmentation may be defined by data augmentation parameters, which may typically define at least one of a data augmentation type indicating a type of data augmentation such as translation, rotation, blurring, or brightness conversion, a data augmentation strength indicating the strength of a certain type of data augmentation, and a type application probability indicating the frequency of applying a certain type of data augmentation.

[0005] The importance of AutoML (Machine Learning) technology has been highlighted in order to automatically optimize such data augmentation parameters. Conventionally, data augmentation parameters have been mainly designed manually based on knowledge of the task and / or the dataset. AutoAugment has been used as a tool for automatically optimizing the data augmentation parameters, and various tools derived from AutoAugment (e.g., AdaAug, Faster AA, etc.) have been proposed. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Raphael Gontijo-Lopes, Sylvia Smullin, Ekin Dogus Cubuk, and Ethan Dyer. “Tradeoffs in data augmentation: An empirical study”. In International Conference on Learning Representations (ICLR), 2021. 3 Summary of the Invention [Problem to be solved by the invention]

[0007] However, conventional data augmentation optimization techniques such as AutoAugment are intended for general object recognition, such as classifying an object into a class such as an airplane, a car, or a flower, based on CIFAR (Canadian Institute for Advanced Research) or ImageNet, and are not intended for fine-grained classification such as classifying airplane models, car models, or flower types.

[0008] In fact, when the data set to be identified is different, the optimal augmentation space, i.e., the range of possible values ​​for the data augmentation parameters, may also be different, and it is preferable that the data augmentation parameters be optimized to correspond to the data set to be identified.

[0009] In view of the above problems, one object of the present disclosure is to provide a data augmentation technique that corresponds to a data set to be identified. [Means for solving the problem]

[0010] One aspect of the present disclosure relates to a data extension device having a distance calculation unit that calculates a first distance between first data of a first class in a dataset and first extended data extended from the first data, and a second distance between the first extended data and second extended data extended from second data of a second class in the dataset, and a data extension unit that performs data extension on the dataset using data extension parameters determined based on the first distance and the second distance. Effect of the Invention

[0011] According to the present disclosure, it is possible to provide a data augmentation technique that corresponds to a data set to be identified. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram illustrating a training system according to one embodiment of the present disclosure. [Diagram 2] 2A and 2B are diagrams illustrating an example of data expansion. [Diagram 3] 3A and 3B are diagrams illustrating an example of data augmentation according to one embodiment of the present disclosure. [Figure 4] FIG. 4 is a block diagram illustrating a hardware configuration of a data expansion device according to an embodiment of the present disclosure. [Diagram 5] FIG. 5 is a block diagram illustrating a functional configuration of a data expansion device according to an embodiment of the present disclosure. [Figure 6] FIG. 6 is a schematic diagram illustrating an extended space by data extension according to one embodiment of the present disclosure. [Figure 7] FIG. 7 is a diagram illustrating a pseudocode showing an algorithm for data augmentation processing according to an embodiment of the present disclosure. [Figure 8] FIG. 8 is a flow chart illustrating a training process according to one embodiment of the present disclosure. [Figure 9] FIG. 9 is a schematic diagram illustrating an application example of face recognition technology according to an embodiment of the present disclosure. [Figure 10] FIG. 10 is a schematic diagram showing a face recognition device according to an embodiment of the present disclosure. [Figure 11] FIG. 11 is a block diagram illustrating a functional configuration of a face recognition device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0014] In the following embodiment, a data augmentation apparatus is disclosed for performing data augmentation on a data set according to data augmentation parameters.

[0015] [Training System] In the training system 10 as shown in Fig. 1, the training device 200 trains the machine learning model 20 to be trained by using a dataset acquired from a training data database (DB) 50 and an extended dataset extended from the dataset by the data extension device 100. Note that in the illustrated embodiment, the data extension device 100 is shown as a device physically independent from the training device 200, but the data extension device 100 according to the present disclosure is not limited thereto and may be built into the training device 200, for example.

[0016] The data expansion device 100 generates expanded data from the data of the dataset according to data expansion parameters including at least one of a data expansion type, a data expansion strength, and a type application probability. Here, the data expansion type may indicate a type of data expansion, such as, for example, movement, rotation, blur, or brightness conversion. The data expansion strength may indicate the strength of a certain type of data expansion, and the type application probability may indicate the frequency of applying a certain type of data expansion. Expanded data can be generated from the data of the dataset according to these data expansion parameters.

[0017] In the following embodiment, the data expansion device 100 expands data x of class A in a data set. a And data x a Extended data x extended from a Distance between

number

number

[0018] In the embodiment described below, in order to achieve a good trade-off between intra-class variation and inter-class discrimination, the data augmentation device 100 augments data x of class A as shown in FIG. 3A. a and data x of class N n As shown in FIG. 3B, the distance d a ´ and d n ´ and the data augmentation strength θ k This allows for data augmentation that allows for intra-class variation while maintaining inter-class discriminability.

[0019] Here, the data extension device 100 is realized by a computing device such as a server, and may have a hardware configuration as shown in Fig. 4. That is, the data extension device 100 has a drive device 101, a storage device 102, a memory device 103, a processor 104, a user interface (UI) device 105, and a communication device 106, which are interconnected via a bus B.

[0020] A program or instruction for implementing various functions and processes in the data extension device 100, which will be described later, may be stored in a removable storage medium such as a CD-ROM (Compact Disk-Read Only Memory) or a flash memory. When the storage medium is set in the drive device 101, the program or instruction is installed from the storage medium to the storage device 102 or the memory device 103 via the drive device 101. However, the program or instruction does not necessarily have to be installed from the storage medium, and may be downloaded from any external device via a network or the like. The storage device 102 is realized by a hard disk drive or the like, and stores files, data, etc. used to execute the program or instruction together with the installed program or instruction. The memory device 103 is realized by a random access memory, a static memory, etc., and when the program or instruction is started, reads out the program, instruction, data, etc. from the storage device 102 and stores them. The storage device 102, the memory device 103, and the removable storage medium may be collectively referred to as a non-transitory storage medium.

[0021] The processor 104 may be realized by one or more central processing units (CPUs), graphics processing units (GPUs), processing circuitry, etc., which may be composed of one or more processor cores, and executes various functions and processes of the data extension device 100 described later according to programs, instructions, data such as parameters necessary to execute the programs or instructions, etc. stored in the memory device 103. The user interface (UI) device 105 may be composed of input devices such as a keyboard, mouse, camera, microphone, etc., output devices such as a display, speaker, headset, printer, etc., and input / output devices such as a touch panel, and realizes an interface between a user and the data extension device 100. For example, a user operates a GUI (Graphical User Interface) displayed on a display or a touch panel using a keyboard, mouse, etc. to operate the data extension device 100. The communication device 106 is realized by various communication circuits that execute communication processes with external devices, the Internet, communication networks such as a LAN (Local Area Network), etc.

[0022] However, the above hardware configuration is merely an example, and the data augmentation device 100 according to the present disclosure may be realized by any other suitable hardware configuration. In addition, the training device 200 may also be realized as a server having a similar hardware configuration.

[0023] [Data expansion device] Next, a data expansion device 100 according to an embodiment of the present disclosure will be described with reference to Fig. 5. In this embodiment, the data expansion device 100 acquires a dataset for training the training target model 20 from the training data DB 50, and generates expanded data from data in the dataset according to data expansion parameters capable of maintaining the discriminability of data of different classes in the dataset. Specifically, as described with reference to Fig. 3, the data expansion device 100 expands data x a ,x nFor data x a and its extended data x a ´ and the distance d a ´ and extended data x a ´ and Data x n Extended Data x n ´ and the distance d n ´ and calculate the calculated distance d a ´ and distance d n Then, the data augmentation device 100 performs data augmentation on the dataset according to the determined data augmentation parameters, and passes the generated augmented dataset to the training device 200.

[0024] Fig. 5 is a block diagram showing a functional configuration of a data extension device 100 according to an embodiment of the present disclosure. As shown in Fig. 5, the data extension device 100 includes a distance calculation unit 110 and a data extension unit 120. For example, one or more functional units of the distance calculation unit 110 and the data extension unit 120 may be realized by one or more processors 104 executing one or more programs or instructions.

[0025] The distance calculation unit 110 calculates the distance between the data x a And data x a Extended data x extended from a Distance between

number

number

[0026] The data extension unit 120 determines the distance d a ´,d n The data set is subjected to data augmentation according to data augmentation parameters determined based on the data augmentation parameter. Typically, the data augmentation is performed according to the data augmentation parameters, and the data augmentation parameters may include a data augmentation type, a data augmentation strength, and / or a type application probability. That is, the data augmentation type may indicate a type of data augmentation, such as, for example, movement, rotation, blur, or brightness conversion. The data augmentation strength may indicate the strength of a certain type of data augmentation, and for example, a range of rotation angles of 0 to 30 degrees may be set as the data augmentation strength. The type application probability may indicate the frequency of applying a certain type of data augmentation, and for example, a range of application probability of 0 to 0.3 may be set as the type application probability for rotation.

[0027] For example, the distance d a ´,d n When the data extension unit 120 obtains the distance d a ´,d n In one embodiment, the data augmentation unit 120 determines the data augmentation parameters based on the two distances d a ´,d n The data augmentation strength for which d' is equal may be determined as the data augmentation strength for the data set. a ´=d nFor example, as shown in FIG. 6, when there is data of class A and data of class N, it is preferable to determine the data augmentation strength so that the augmented space of class A, indicated by the dashed line, can be separated from SEMI-HARD data of class N, which is similar to class A. Here, HARD data of class N is inherently inseparable from data of class A due to labeling errors, etc., and EASY data of class N may still be separable from data of class A even after data augmentation is applied. The two distances d a ´,d n The purpose of determining the data augmentation strength at which ' is equal as the data augmentation parameter is to focus on the separability of SEMI-HARD data, not on the discrimination between HARD data and EASY data of different classes N. Note that in the illustrated example, SEMI-HRAD data and HARD data are determined for specific anchor data, but preferably, SEMI-HRAD data and HARD data may be determined globally for each class of the data set.

[0028] Thus, the two distances d a ´,d n We adopt a data augmentation strength that makes the augmented data x ´ equal to the class A,N, while maintaining the intra-class variation due to data augmentation. a ´,x n To make the vector x′ separable, the data augmentation unit 120 may specifically solve the following constrained optimization problem:

number

number

number

[0029] The objective function in equation (1) is for one data item in one class of the data set and one data item in the other class (which may be called a negative sample), but more generally, when the objective function is applied to the whole class, the distance d a ´ and d n The terms corresponding to ´ can be formulated as follows:

number

number

[0030] For example, the data augmentation unit 120 may determine a data augmentation strength θ for each data augmentation type k for the image data set according to an Augmentation Space Optimization (ASO) algorithm as shown in FIG. k The distance dist is replaced by the similarity h. The data augmentation strength is quantized with a predetermined granularity. The algorithm shown in the figure is as follows: for each data augmentation type k, data x a and extended data x a ´ and the distance d a ´ average d a ´ and extended data x a ´ and class N data x n Extended Data xn ´ and the distance d n ´ average d n ', i.e., the average d a ´ and average μd n ´ and the data augmentation strength θ k is determined.

[0031] [Data Augmentation Processing] Next, a data augmentation process according to an embodiment of the present disclosure will be described with reference to Fig. 8. Fig. 8 is a flowchart showing the data augmentation process according to an embodiment of the present disclosure. The data augmentation process is executed by the above-mentioned training system 10, and more specifically, may be realized by one or more processors 104 of the data augmentation device 100 and / or the training device 200 executing one or more programs or instructions stored in one or more memory devices 103. For example, the data augmentation process may be started by starting a training process for the training target model 20 in the training system 10.

[0032] As shown in FIG. 8, in step S101, the training device 200 uses a data set to train the trained model 20. Specifically, the training device 200 may train the trained model 20 without data augmentation to determine the distance between images, and use the trained model 20 as a baseline model. For example, when the trained model 20 is realized by a neural network for image processing, the training device 200 acquires training data consisting of a pair of image data and a label from the training data DB 50, and inputs the image data to the input layer of the trained model 20. Then, when the processing result is acquired from the output layer, the training device 200 compares the processing result with the label, and updates the parameters of the trained model 20 according to the error between the processing result and the label according to the backpropagation method. For example, the parameters of the trained baseline model are represented as φ in FIG. 7, and can be used to calculate the similarity between data.

[0033] In step S102, the data expansion device 100 expands the class A data x aand extended data x a Distance between

number

[0034] In step S103, the data extension device 100 outputs the extended data x a ´ and extended data for class N x n Distance between

number

[0035] In step S104, the data expansion device 100 calculates the distance d a ´ and d n Specifically, the data expansion device 100 determines the data expansion parameters based on the distance d a ´ and d n For example, the data augmentation device 100 may determine the data augmentation strength so that, for each data of each class in the data set, the data augmentation strength is set to be equal to two distances d a ´ and d n ´ and the determined distance d a ´ average d a ´ and distance d n ´ average d n Then, the data expansion device 100 calculates d a ´=d n Then, we determine the data augmentation strength θ such that

[0036] For example, for each data augmentation operation, such as translation, rotation, blurring, or brightness conversion, an objective function may be used to determine the data augmentation strength. The maximum and minimum values ​​of the range of data augmentation strength before optimization may be provided, and the range may be divided into L steps. In experiments, the range may be specified from the specifications of an augmentation library such as PIL. Using the trained baseline model, the training data may be augmented at the data augmentation strength and extracted features for each step. In FIG. 7,

number

number

number

[0037] In step S105, the data expansion device 100 performs data expansion according to the data expansion parameters. Specifically, the data expansion device 100 performs data expansion on the data set according to the determined data expansion strength θ to generate expanded data.

[0038] In step S106, the training device 200 trains the training target model 20 with the dataset including the extended data. For example, when the training target model 20 is realized by a neural network for image processing, the training device 200 acquires training data consisting of pairs of image data and labels in the dataset and the extended dataset, and inputs the image data to the input layer of the training target model 20. Then, when the processing result is acquired from the output layer, the training device 200 compares the processing result with the label, and updates the parameters of the training target model 20 according to the error between the processing result and the label according to the backpropagation method. When a predetermined termination condition is satisfied, the training device 200 terminates the training of the training target model 20.

[0039] [Facial recognition technology] The data augmentation process according to the above-mentioned embodiment can be applied to any machine learning model. For example, a machine learning model trained using the augmented data generated by the data augmentation process according to the above-mentioned embodiment may be realized as a neural network for image processing. The neural network trained using the augmented data generated by the data augmentation process according to the above-mentioned embodiment may be used for image recognition, object detection, face recognition, etc. For example, the neural network for face recognition may be applied to various solutions such as entrance and exit management for buildings and rooms, immigration procedures at airports, reception processing, store monitoring, and introduction to products and services, as shown in FIG. 9, without being limited thereto.

[0040] The trained model 30 trained by the training system 10 using the above-mentioned data augmentation process may be used by a face recognition device 300, for example, as shown in Fig. 10. As described above, the face recognition device 300 may be used for entrance / exit management to buildings and rooms, immigration procedures at airports, reception processing, store monitoring, introduction to products and services, etc. Here, the face recognition device 300 may be realized as a server having the hardware configuration as described with reference to Fig. 4.

[0041] 11 is a block diagram showing a functional configuration of a face recognition device 300 according to an embodiment of the present disclosure. As shown in FIG. 11, the face recognition device 300 includes a data acquisition unit 310 and a face recognition unit 320.

[0042] The data acquisition unit 310 acquires image data showing the face of the authentication target. Specifically, the data acquisition unit 310 acquires image data capturing an image of the face of the authentication target from a camera communicatively connected to the face authentication device 300, and passes the acquired image data to the face authentication unit 320.

[0043] The face recognition unit 320 recognizes faces in image data using a trained model 30. The trained model 30 here is a trained model for class A data x a And data x a Extended data x extended from a Distance between

number

number

[0044] For example, the face authentication unit 320 inputs image data showing a face to be authenticated to the trained model 30 and obtains an authentication result of the captured face from the trained model 30. For example, in entrance / exit management using face authentication technology, the face authentication unit 320 compares registered face data and / or its feature amount with the authentication result and determines whether the person captured in the image data is a registered person. If it is determined that the person captured in the image data is a registered person, the face authentication unit 320, for example, allows the person to enter the room, and if it is determined that the person captured in the image data is not a registered person, the face authentication unit 320, for example, does not allow the person to enter the room.

[0045] According to the above-mentioned embodiment, the data expansion device 100 can realize data expansion capable of maintaining inter-class discrimination while enabling changes within a class due to data expansion. In the case of face recognition, the change tendency of an image is different from that of ImageNet, etc., and accuracy improvement cannot be obtained by using a conventional data expansion space. Although manual tuning is also possible, the data expansion space has many parameters, and manual tuning may waste time and computational resources. According to the above-mentioned embodiment, it is possible to achieve the effect of improving accuracy while suppressing such costs.

[0046] (Appendix 1) a distance calculation unit that calculates a first distance between first data of a first class in a dataset and first extended data extended from the first data, and a second distance between the first extended data and second extended data extended from second data of a second class in the dataset; a data augmentation unit that performs data augmentation on the data set according to data augmentation parameters determined based on the first distance and the second distance; A data expansion device having the following: (Appendix 2) The data expansion device according to claim 1, wherein the data expansion unit performs data expansion on the data set with a data expansion strength that makes the first distance equal to the second distance. (Appendix 3) The data expansion device according to claim 1 or 2, wherein the data expansion parameters include at least one of a data expansion type, a data expansion strength, and a type application probability. (Appendix 4) 4. The data extension device according to claim 1, wherein the distance calculation unit calculates the first distance and the second distance based on a cosine similarity. (Appendix 5) a data augmentation device for performing data augmentation on a data set; A training device that trains a training target model using the dataset and the augmented data generated by the data augmentation; having The data expansion device includes: a distance calculation unit that calculates a first distance between first data of a first class in the data set and first extended data extended from the first data, and a second distance between the first extended data and second extended data extended from second data of a second class in the data set; a data augmentation unit that performs data augmentation on the data set according to data augmentation parameters determined based on the first distance and the second distance; A training system having the following features: (Appendix 6) Calculating a first distance between a first data of a first class in a dataset and a first extended data extended from the first data, and a second distance between the first extended data and a second extended data extended from a second data of a second class in the dataset; performing data augmentation on the data set with data augmentation parameters determined based on the first distance and the second distance; 16. A computer implemented method for data augmentation comprising: (Appendix 7) Calculating a first distance between a first data of a first class in a dataset and a first extended data extended from the first data, and a second distance between the first extended data and a second extended data extended from a second data of a second class in the dataset; performing data augmentation on the data set with data augmentation parameters determined based on the first distance and the second distance; A program that causes a computer to execute the following. (Appendix 8) Accepting the data, outputting a processing result for the data; A machine learning model that performs The machine learning model is Calculating a first distance between a first data of a first class in a dataset and a first extended data extended from the first data, and a second distance between the first extended data and a second extended data extended from a second data of a second class in the dataset; performing data augmentation on the data set with data augmentation parameters determined based on the first distance and the second distance; a machine learning model that is trained utilizing the dataset that includes augmented data generated by running

[0047] Although the examples of the present disclosure have been described in detail above, the present disclosure is not limited to the specific embodiments described above, and various modifications and variations are possible within the scope of the gist of the present disclosure described in the claims. [Industrial Applicability]

[0048] The present disclosure is useful for a data augmentation apparatus and method for data augmentation performed when training a machine learning model. [Explanation of symbols]

[0049] 10 Training System 20 Models to be trained 30 Pre-trained Models 50 Training Data DB 100 Data expansion device 110 Distance calculation unit 120 Data Extension 200 Training equipment 300 Face Recognition Device 310 Data Acquisition Department 320 Face Recognition Department

Claims

1. a distance calculation unit that calculates a first distance between first data of a first class in a dataset and first extended data extended from the first data, and a second distance between the first extended data and second extended data extended from second data of a second class in the dataset; a data augmentation unit that performs data augmentation on the data set according to data augmentation parameters that are determined based on the first distance and the second distance; A data expansion device having the following:

2. The data expansion device according to claim 1 , wherein the data expansion unit performs data expansion on the data set with a data expansion strength that makes the first distance equal to the second distance.

3. The data expansion device according to claim 1 , wherein the data expansion parameters include at least one of a data expansion type, a data expansion strength, and a type application probability.

4. The data expansion device according to claim 1 , wherein the distance calculation unit calculates the first distance and the second distance based on a cosine similarity.

5. a data augmentation device for performing data augmentation on a data set; A training device that trains a training target model using the dataset and the augmented data generated by the data augmentation; having The data expansion device includes: a distance calculation unit that calculates a first distance between first data of a first class in the data set and first extended data extended from the first data, and a second distance between the first extended data and second extended data extended from second data of a second class in the data set; a data augmentation unit that performs data augmentation on the data set according to data augmentation parameters that are determined based on the first distance and the second distance; A training system having:

6. Calculating a first distance between a first data of a first class in a dataset and a first extended data extended from the first data, and a second distance between the first extended data and a second extended data extended from a second data of a second class in the dataset; performing data augmentation on the data set with data augmentation parameters determined based on the first distance and the second distance; 16. A computer implemented method for data augmentation comprising:

7. Calculating a first distance between a first data of a first class in a dataset and a first extended data extended from the first data, and a second distance between the first extended data and a second extended data extended from a second data of a second class in the dataset; performing data augmentation on the data set with data augmentation parameters determined based on the first distance and the second distance; A program that causes a computer to execute the following.

8. Accepting the data, outputting a processing result for the data; A machine learning model that performs The machine learning model is Calculating a first distance between a first data of a first class in a dataset and a first extended data extended from the first data, and a second distance between the first extended data and a second extended data extended from a second data of a second class in the dataset; performing data augmentation on the data set with data augmentation parameters determined based on the first distance and the second distance; a machine learning model that is trained utilizing the dataset that includes augmented data generated by running

Images