Data augmentation method, data augmentation apparatus, and computer program product

Abstract

To automatically generate a data generation condition without manual definition or labeling, and generate output data from input data based on the automatically generated data generation condition, thereby enabling data expansion.SOLUTION: A data expansion method includes the steps of: performing sampling on a predetermined distribution curve to obtain sampling points; generating, by a first neural network model capable of reversible transformation, a first conditional feature vector on the basis of the sampling points; and generating, by a second neural network model, augmented data using input data and the first conditional feature vector as input.EFFECT: Complexity is reduced and data diversity is improved.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to data processing, and more particularly to a data expansion method, a data expansion apparatus and a computer program product. [Background technology]

[0002] Currently, deep learning models have achieved results in many fields, but training them relies on large amounts of labeled data, and a lack of labeled data or an imbalance in the distribution of data often leads to problems with poor performance.Data augmentation is a technology that automatically expands training data, which can improve the performance of deep learning models by reducing the cost of acquiring labeled data. Summary of the Invention [Means for solving the problem]

[0003] The present disclosure relates to a data augmentation method, a data augmentation device, a computer program product, and a non-transitory computer-readable storage medium that enable data augmentation without significant human involvement by automatically generating data generation conditions and enabling output data to be generated from input data based on the automatically generated data generation conditions.

[0004] According to one aspect of the present disclosure, there is provided a data augmentation method, the method including: sampling a predetermined distribution curve to obtain sampling points; generating a first conditional feature vector based on the sampling points using a first neural network model capable of performing reversible transformation; and generating augmented data using a second neural network model that receives input data and the first conditional feature vector as inputs.

[0005] According to one embodiment of the present disclosure, the second neural network model is trained to generate second data from the first data and the second conditional feature vector when the third neural network model generates the second conditional feature vector based on the first data and the second data.

[0006] According to one embodiment of the present disclosure, the first neural network model is trained such that when the third neural network model generates a second conditional feature vector based on the first data and the second data, the first neural network model can map the second conditional feature vector to a predetermined distribution curve.

[0007] According to one embodiment of the present disclosure, the first condition feature vector or the second condition feature vector indicates a type of offset of the data.

[0008] According to one embodiment of the present disclosure, the predetermined distribution curve is a Gaussian distribution curve.

[0009] According to one embodiment of the present disclosure, the first neural network model includes one or more coupling layer models.

[0010] According to one embodiment of the present disclosure, the input data is text data, and the extended data is offset text data that is offset from the text data.

[0011] According to one embodiment of the present disclosure, the input data is image data, and the dilated data is offset image data that is offset from the image data.

[0012] According to one embodiment of the present disclosure, the second neural network model includes an autoregressive model.

[0013] According to one embodiment of the present disclosure, the third neural network model includes a feature extractor.

[0014] According to another aspect of the present disclosure, there is provided a data expansion apparatus, the apparatus including a processor and a memory having stored thereon one or more computer programs that, when executed by the processor, cause the processor to perform any of the methods described above.

[0015] According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, causes the processor to perform any of the methods described above.

[0016] According to another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause the processor to perform any of the methods described above. [Effects of the Invention]

[0017] According to an embodiment of the present disclosure, data generation conditions can be automatically generated without manual definition or labeling, and output data can be generated from input data based on the automatically generated data generation conditions, thereby enabling data expansion, reducing complexity and improving data diversity.

[0018] The forms, features and advantages of the present disclosure will be more clearly and readily understood from the following description of examples taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 shows a schematic diagram of the overall technical concept of the present disclosure. [Figure 2] FIG. 2 shows a flowchart of a data augmentation method according to an embodiment of the present disclosure. [Figure 3] FIG. 3 shows a schematic diagram of a data expansion device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0020] Hereinafter, embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings. Although several embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure may be implemented in various forms and should not be construed as being limited to the embodiments described herein, but rather, these embodiments are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the accompanying drawings and embodiments are merely for illustrative purposes and are not intended to limit the scope of protection of the present disclosure.

[0021] It should be understood that the steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel, and that the method embodiments may include other steps and / or omit some steps.

[0022] As used herein, the term "comprises" and variations thereof are open-ended, i.e., "including, but not limited to." The term "based on" means "based at least in part on." The term "in one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one other embodiment," and "some embodiments" means "at least some embodiments." Definitions of other terms are provided in the description below.

[0023] It should be understood that the concepts of "first", "second", etc. referred to in this disclosure are merely intended to distinguish different devices, modules or units, and are not intended to limit the order of functions performed by these devices, modules or units or their dependencies on each other.

[0024] It should be noted that the modifications "one" and "multiple" referred to in this disclosure are exemplary rather than limiting, and it should be understood by those skilled in the art that these should be understood as "one or more" unless the context clearly dictates otherwise.

[0025] The data referred to in this disclosure may include various types of data, such as text data and image data. Many scenarios require data augmentation, but existing data augmentation techniques either require a large amount of human intervention or produce low diversity data. For example, in the field of automatic speech recognition (ASR) for text data, existing ASR models can receive a speech signal as input and convert it into text to output. However, the ASR conversion results often contain errors that are particularly obvious to humans, necessitating an error correction model for the ASR model to correct the recognized erroneous text. Such an error correction model may be obtained by training using a neural network model. Therefore, text pairs such as {correct text, incorrect text} are required as training data. However, building a dataset of {correct text, incorrect text} text pairs requires a lot of labeling work, such as labeling error types (homonyms, similar sounds, and variations of verb rewriting and noun rewriting) and the corresponding erroneous text. Also, for any different task, such as different types of text or different text datasets, it may be necessary to manually relabel or define the condition elements (e.g., the error types mentioned above), resulting in high consumption of human resources and ultimately a relatively small amount and variety of training data.

[0026] The present disclosure has been made in view of the above-mentioned problems. A data augmentation method, a data augmentation device, a computer program product, and a non-transitory computer-readable storage medium according to embodiments of the present disclosure can be used to automatically generate data generation conditions without manual definition or labeling, and can augment data by generating output data from input data based on the automatically generated data generation conditions. This reduces complexity and improves data diversity.

[0027] The idea of the present disclosure is to reduce complexity and improve data diversity in the data augmentation process by changing the data generation and augmentation conditions from being manually defined to automatically generated by a neural network model. Figure 1 schematically illustrates the overall technical idea of the present disclosure. The proposal of the present disclosure relies on artificial intelligence, specifically, various neural network models. The neural network model includes both a training phase and a data generation phase. Figure 1(a) shows the training phase, and Figure 1(b) shows the data generation phase.

[0028] As shown in FIG. 1(a), in the training phase, a neural network model is trained using multiple known or labeled {first data, second data} data pairs. For example, the first data may correspond to original data or correct data, and the second data may correspond to offset data or erroneous data corresponding to the first data. Three neural network models are involved in the training phase. As shown in FIG. 1(a), the third neural network model is used to extract a conditional feature vector based on each {first data, second data} data pair. The conditional feature vector may indicate the type of generation condition or deviation of the second data relative to the first data. For example, the conditional feature vector may be the difference between a vector representing the second data and a vector representing the first data, or may indicate the correlation or offset between the second data and the first data in other ways. Semantically, the type of data offset may correspond to a semantic offset, a syntactic offset, or the like, and may not have an actual semantic meaning. The third neural network model may include any of various existing feature extractors, such as a convolutional neural network (CNN) model or a recurrent neural network (RNN) model. The third neural network model may obtain a plurality of condition feature vectors corresponding to a plurality of data pairs {first data, second data}. By training the first neural network model using the condition feature vectors as inputs, the first neural network model can map the condition feature vectors to a predetermined distribution curve.In the present disclosure, a model capable of performing reversible transformation is used as the first neural network model to establish a bidirectional mapping relationship between the condition feature vector and a predetermined distribution curve, so that in a subsequent data generation phase, the first neural network model can perform an inverse mapping process, i.e., obtain the condition feature vector based on the predetermined distribution curve. The first neural network model may include, for example, a flow-based model, a variational autoencoder (VAE), a diffusion model, etc. Also, as shown in FIG. 1(a), the plurality of condition feature vectors output by the third neural network model are input to a second neural network model in addition to the first neural network model, and are used to train the second neural network model together with the plurality of first data, so that the second neural network model can output second data corresponding to the first data based on the first data and the condition feature vector corresponding to the first data. The second neural network model may be implemented using various existing generative models, including, for example, autoregressive models and non-autoregressive models (e.g., transformers). The second neural network model and the third neural network model may be trained simultaneously. That is, a data pair {first data, second data} is input to the third neural network to generate a condition feature vector, and the condition feature vector is input to the second neural network model together with the first data. At the same time, the second neural network model is trained using the second data as the output of the second neural network model. Thus, after training the second neural network model and the third neural network model, the second neural network model can output the second data when the input is the first data.

[0029] As shown in FIG. 1(b), in the data generation phase, a first neural network model and a second neural network model are used to generate augmented data based on the input data and a predetermined distribution curve. As described above, the first neural network model can perform a reversible transformation and obtain a conditional feature vector based on the predetermined distribution curve. For example, sampling is performed on the predetermined distribution curve to obtain sampling points, and the first neural network model can reversely map the sampling points to a conditional feature vector, i.e., automatically generate a conditional feature vector corresponding to the type of data offset. The second neural network model may then generate augmented data offset from the input data based on the input data and the conditional feature vector generated by the first neural network. By sampling the predetermined distribution curve multiple times to obtain multiple different sampling points, multiple different augmented data can be generated based on the same input data, thereby achieving data augmentation and augmented data diversity. Furthermore, since there is no need to previously store a large number of different manually labeled data generation conditions in memory, storage space consumption is reduced.

[0030] 2 shows a flowchart of a data augmentation method 200 according to an embodiment of the present disclosure. The method 200 may correspond to the data generation phase described above. As shown in FIG. 2, the method 200 according to an embodiment of the present disclosure includes steps S210 to S230.

[0031] In step S210, sampling is performed on a predetermined distribution curve to obtain sampling points. Here, the predetermined distribution curve may be any of various distribution curves whose distribution functions are known. Points on the distribution curve may correspond to conditional feature vectors, and thus, in the data augmentation process, various conditional feature vectors may be obtained based on the predetermined distribution curve. In one embodiment, the predetermined distribution curve may be a Gaussian distribution curve, such as a standard Gaussian distribution curve. In reality, many random variables tend to approximately follow a Gaussian distribution. Therefore, adopting a Gaussian distribution curve as the predetermined distribution curve may make the subsequent generation of conditional feature vectors more meaningful. Alternatively, the predetermined distribution curve may be another curve, such as an F distribution curve or a Poisson distribution curve.

[0032] In step S220, the first neural network model generates a first conditional feature vector based on the sampling points obtained in step S210. As described above, the first neural network model may perform a reversible transformation, and may include, for example, a flow-based model, a VAE, a diffusion model, etc. In one embodiment, the first neural network model may include one or more coupling layer models. The first neural network model may perform a reverse mapping process, i.e., obtain a conditional feature vector based on a predetermined distribution curve. Specifically, the first neural network model may establish a mapping between the conditional feature vector and the predetermined distribution curve in a training phase, and then map each sampling point on the predetermined distribution curve to a conditional feature vector in a data generation phase. The first conditional feature vector generated in step S220 may indicate a data offset type, i.e., a type of offset of the generated augmented data relative to the input data, as described above. It may be understood that the data offset type does not necessarily have an actual semantic meaning, as described above.

[0033] In step S230, the input data and the first conditional feature vector generated in step S220 are input to a second neural network model to generate augmented data. As described above, the second neural network model can be implemented using various existing generative models, including, for example, an autoregressive model or a non-autoregressive model. Before generating data using the second neural network model, the second neural network model is trained to receive the data and the conditional feature vector as input and generate augmented data as output. Since the first conditional feature vector may indicate a type of data offset, the data generated by the second neural network model is offset data resulting from the shift of the input data, thereby achieving data augmentation.

[0034] As will be understood by those skilled in the art, as described above, the neural network model may be pre-trained before generating data. During the training phase, a plurality of {first data, second data} data pairs may be used to train the second neural network model and the third neural network model. The first data may correspond to the original data or correct data, and the second data may correspond to offset data or erroneous data corresponding to the first data. In one embodiment, the second neural network model may be trained to generate second data based on the first data and the second conditional feature vector when the third neural network model generates a second conditional feature vector based on the first data and the second data. That is, the plurality of {first data, second data} data pairs and the corresponding plurality of second conditional feature vectors serve as training data for the second neural network model. The second neural network model receives the first data and the corresponding second conditional feature vector as input and outputs second data corresponding to the first data and the second conditional feature vector. The third neural network model may include any of various existing feature extractors, such as a convolutional neural network (CNN) model or a recurrent neural network (RNN) model. As described above, the third neural network model may extract the second conditional feature vector based on each (first data, second data) data pair. The second conditional feature vector may indicate the type of generation condition or offset of the second data relative to the first data. For example, the conditional feature vector may be the difference between a vector representing the second data and a vector representing the first data.Here, the term "second conditional feature vector" is used to refer to the conditional feature vector generated by the third neural network model based on the plurality of {first data, second data} data pairs in the data generation phase, in order to distinguish it from the "first conditional feature vector" generated by the first neural network model in the data generation phase. It should be understood that the terms "first" and "second" herein do not imply any preferred mode, priority, or dependency on each other.

[0035] Also, during the training phase, the first neural network model may be trained using the plurality of second condition feature vectors generated by the third neural network model based on the plurality of {first data, second data} data pairs as described above. In one embodiment, the first neural network model may be trained such that when the third neural network model generates the second condition feature vector based on the first data and the second data, the first neural network model can map the second condition feature vector to a predetermined distribution curve. As described above, the first neural network model is capable of performing reversible transformations. If the first neural network model is enabled to map the second condition feature vector to a predetermined distribution curve during the training phase, the first neural network model can perform the reverse mapping during the data generation phase, i.e., generate the corresponding condition feature vector, i.e., the first condition feature vector in step S220 described above, based on sampling points on the predetermined distribution curve.

[0036] In the data augmentation method according to the embodiment of the present disclosure, data generation conditions (i.e., the first condition feature vector described above) can be automatically generated without manual definition and labeling. This enables data augmentation by generating output data from input data based on the automatically generated data generation conditions. This reduces complexity and improves data diversity.

[0037] According to an embodiment of the present disclosure, a data expansion device is also provided. Figure 3 shows a schematic diagram of a data expansion device 300 according to an embodiment of the present disclosure.

[0038] 3, a data expansion device 300 according to an embodiment of the present disclosure includes a processor 310 and a memory 320. The memory 320 stores one or more computer programs.

[0039] The processor 310 is a program control device such as a microprocessor, and operates according to a program installed in, for example, the memory 320. The memory 320 is a storage element such as a ROM or RAM. The program executed by the processor 310 is stored in the memory 320. The data extension device 300 shown in FIG. 3 may be used to implement the data extension method disclosed in the present application.

[0040] In the device according to the embodiment of the present disclosure, data generation conditions can be automatically generated without manual definition or labeling. Therefore, by generating output data from input data based on the automatically generated data generation conditions, data expansion is possible, reducing the amount of work required and increasing the diversity of data.

[0041] According to an embodiment of the present disclosure, a computer program product is provided, which includes a computer program that, when executed by a processor, can implement the data augmentation method disclosed herein.

[0042] According to an embodiment of the present disclosure, there is provided a non-transitory computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause the processor to perform the data augmentation method disclosed herein.

[0043] The data extension method, data extension device, computer program product, and non-transitory computer-readable storage medium according to the embodiments of the present disclosure may be applied to various data processing fields, such as text data processing and image data processing. When applied to the text data processing field, the input data and the extended data in step S230 may be text data and offset text data offset from the text data, respectively. The offset text data may be, for example, text data whose semantics, syntax, structure, etc. have changed from the input text data. For example, the offset text data may include characters that are homonyms of the characters in the input text data, or different words that have the same or similar pronunciation as words in the input text data. When applied to the image data processing field, the input data and the extended data in step S230 may be image data and offset image data offset from the image data, respectively. The offset image data may be, for example, image data whose color, pixel position, etc. have changed from the input image data. For example, the offset image data may include pixels of a different color from the pixels in the input image data, pixels in a different position from the pixels in the input image data (e.g., pixels that have been translated or rotated), etc.

[0044] The data augmentation method, data augmentation device, computer program product, and non-transitory computer-readable storage medium according to the embodiments of the present disclosure are particularly suitable for scenarios in which multiple outputs are generated based on one input. For example, the data augmentation method, data augmentation device, computer program product, and non-transitory computer-readable storage medium according to the embodiments of the present disclosure may be applied to scenarios in which multiple incorrect texts are generated based on one correct text, multiple rewritten sentences are generated based on one sentence, multiple summary results are generated based on one text content, multiple translation results in multiple other languages are generated based on text in one language, multiple images with different colors or images with different pixel positions within an image are generated based on one image, etc.

[0045] The hardware computing devices described in this disclosure, in whole or in part, may be implemented by various suitable hardware means, including, but not limited to, FPGA, ASIC, SoC, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The devices, apparatus, methods, and systems described herein are not limited to any particular hardware architecture or configuration. Components in the disclosed devices, apparatus, and systems may be separate or integrated, combined in different ways, and / or replaced or supplemented by other components. It should be understood that the teachings of this disclosure may be embodied in various forms of hardware, software, firmware, special purpose processors, or combinations thereof.

[0046] Block diagrams of devices, apparatus, methods, and systems according to the present disclosure are merely exemplary and are not intended to require or imply that they must be connected, arranged, or configured in the manner illustrated in the block diagrams. Those skilled in the art will recognize that these circuits, apparatus, devices, apparatus, and systems may be connected, arranged, or configured in any manner that achieves a desired purpose.

[0047] In the above description, the present invention has been described based on the embodiments. It should be understood by those skilled in the art that these embodiments are merely illustrative and that various modifications can be made to the combination of components and processes in the embodiments, and that such modifications are also within the scope of the present invention.

Claims

1. Sampling a predetermined distribution curve to obtain sampling points; a first neural network model capable of performing a reversible transformation generating a first conditional feature vector based on the sampling points; a step of generating augmented data using a second neural network model that receives input data and the first conditional feature vector as input.

2. The second neural network model is 2. The data augmentation method of claim 1, wherein the second neural network model is trained to generate the second data based on the first data and the second conditional feature vector when a third neural network model generates the second conditional feature vector based on the first data and the second data.

3. The first neural network model is 2. The data augmentation method of claim 1, wherein when a third neural network model generates a second conditional feature vector based on the first data and the second data, the first neural network model is trained to map the second conditional feature vector to a predetermined distribution curve.

4. The data expansion method according to claim 1 , wherein the first conditional feature vector or the second conditional feature vector indicates a type of offset of data.

5. The data augmentation method according to claim 1 , wherein the predetermined distribution curve is a Gaussian distribution curve.

6. The data augmentation method of claim 1 , wherein the first neural network model includes one or more coupling layer models.

7. the input data is text data, The data extension method according to claim 1 , wherein the extension data is offset text data that is offset from the text data.

8. the input data is image data, The data extension method according to claim 1 , wherein the extension data is offset image data that is offset from the image data.

9. The data augmentation method of claim 1 , wherein the second neural network model comprises an autoregressive model.

10. The data augmentation method of claim 2 , wherein the third neural network model includes a feature extractor.

11. a processor; a memory having one or more computer programs stored therein; A data expansion device, wherein when the one or more computer programs are executed by the processor, the method according to any one of claims 1 to 10 is performed by the processor.

12. A computer program product comprising a computer program, A computer program product, wherein the computer program, when executed by a processor, performs the method of any one of claims 1 to 10.

Images