Method, apparatus and computer program product for data augmentation

Abstract

The present disclosure relates to a method, an apparatus and a computer program product for data augmentation. The method for data augmentation comprises the following steps: sampling a predetermined distribution curve to obtain sampling points; generating a first condition feature vector based on the sampling points through a first neural network model, wherein the first neural network model can realize reversible transformation; and inputting the input data and the first condition feature vector into a second neural network model to generate augmented data. According to the invention, the data generation condition can be automatically generated without manual definition or annotation, so that the output data can be generated from the input data based on the automatically generated data generation condition, and then data augmentation is realized. The manpower consumption is reduced; and the data diversity is improved.

Description

Technical Field

[0001] The present disclosure relates to data processing, and more particularly, to a method, apparatus, and computer program product for data augmentation. Background Art

[0002] Currently, deep learning models have achieved success in multiple fields. However, deep learning models rely on a large amount of labeled data for training, and often suffer from poor performance of deep learning models due to lack of labeled data and unbalanced data distribution. Data augmentation is a technique for automatically expanding training data, which can reduce the cost of obtaining labeled data and thus improve the performance of deep learning models. Summary of the Invention

[0003] The present disclosure relates to a method, apparatus, computer program product, and non-transitory computer-readable storage medium for data augmentation, which can automatically generate data generation conditions, so that output data can be generated from input data based on the automatically generated data generation conditions, thereby realizing data augmentation without a large amount of manual participation.

[0004] According to one aspect of the present disclosure, there is provided a method for data augmentation. The method includes: sampling a predetermined distribution curve to obtain sampling points; generating a first conditional feature vector based on the sampling points by a first neural network model, the first neural network model being capable of performing a reversible transformation; and inputting the input data and the first conditional feature vector into a second neural network model to generate augmented data.

[0005] According to some embodiments of the present disclosure, the second neural network model is trained such that: in the case where a second conditional feature vector is generated according to first data and second data by a third neural network model, the second neural network model can generate the second data according to the first data and the second conditional feature vector.

[0006] According to some embodiments of the present disclosure, the first neural network model is trained such that: in the case where a second conditional feature vector is generated according to first data and second data by a third neural network model, the first neural network model can map the second conditional feature vector onto the predetermined distribution curve.

[0007] According to some embodiments of the present disclosure, the first conditional feature vector or the second conditional feature vector indicates a data deviation type.

[0008] According to some embodiments of the present disclosure, the predetermined distribution curve is a Gaussian distribution curve.

[0009] According to some embodiments of the present disclosure, the first neural network model includes one or more coupling layer models.

[0010] According to some embodiments of the present disclosure, the input data is text data, and the augmented data is deviated text data that deviates from the text data.

[0011] According to some embodiments of the present disclosure, the input data is image data, and the augmented data is deviated image data that deviates from the image data.

[0012] According to some embodiments of the present disclosure, the second neural network model includes an autoregressive model.

[0013] According to some embodiments of the present disclosure, the third neural network model includes a feature extractor.

[0014] According to another aspect of the present disclosure, there is provided a device for data augmentation. The device includes: a processor; and a memory storing one or more computer programs; wherein, when the one or more computer programs are run by the processor, the processor is caused to execute any of the above methods.

[0015] According to still another aspect of the present disclosure, there is provided a computer program product including a computer program which, when executed by a processor, implements any of the above methods.

[0016] According to yet another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having computer program instructions stored thereon. The computer program instructions, when executed by a processor, cause the processor to execute any of the above methods.

[0017] According to the embodiments of the present disclosure, data generation conditions can be automatically generated without manual definition or annotation, so that output data can be generated from the input data based on the automatically generated data generation conditions, thereby achieving data augmentation. Manpower consumption is reduced and data diversity is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Aspects, features, and advantages of the present disclosure will become clearer and easier to understand through the following description of the embodiments in conjunction with the drawings, where:

[0019] Figure 1 Schematically shows the overall technical concept of the present disclosure;

[0020] Figure 2 Shows a flowchart of a method for data augmentation according to an embodiment of the present disclosure;

[0021] Figure 3 Shows a schematic diagram of a device for data augmentation according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not used to limit the protection scope of the present disclosure.

[0023] It should be understood that the various steps recited in the method embodiments of the present disclosure can be executed in a different order and / or in parallel. In addition, method embodiments may include other steps and / or omit certain steps.

[0024] As used herein, the term "including" and its variations are open-ended, that is, "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0025] It should be understood that the concepts such as "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependent relationships.

[0026] It should be noted that the modifications of "one" and "plural" mentioned in the present disclosure are illustrative rather than restrictive. Those skilled in the art should understand that, unless clearly specified otherwise in the context, it should be understood as "one or more".

[0027] The data mentioned in this disclosure may include various types of data such as text data and image data. There are many scenarios where data augmentation is required. However, current data augmentation techniques all require a large amount of manual participation or have poor diversity in the generated data. Taking the field of automatic speech recognition (ASR) involving text data as an example, existing ASR models can receive speech signals as input and convert them into text for output. However, in the ASR conversion results, there are still some errors that are very obvious to humans. Therefore, an error correction model for the ASR model is needed to correct the recognized incorrect text. This error correction model can be obtained through training using a neural network model. Therefore, text pairs such as {correct text, incorrect text} are required as training data. However, constructing a dataset of text pairs of {correct text, incorrect text} requires a large amount of manpower for annotation, such as annotating the error types (rewriting changes such as homophones, similar sounds, verb rewriting, noun rewriting, etc.) and the incorrect text corresponding to the error type. In addition, for any different tasks, such as different text types or different text datasets, manual re-annotation or definition of conditional factors (such as the above-mentioned error types) may be required. This results in a very high consumption of human resources, so the quantity and variety of the finally obtained data for training are relatively small.

[0028] This disclosure is made in view of the above problems. The method, device, computer program product, and non-transitory computer-readable storage medium for data augmentation according to the embodiments of this disclosure can be used to automatically generate data generation conditions without manual definition or annotation, so that output data can be generated from input data based on the automatically generated data generation conditions, thereby achieving data augmentation. It reduces human consumption and improves data diversity.

[0029] The concept of this disclosure is that in the process of data augmentation, changing the manual definition of data generation conditions or data augmentation conditions to the automatic generation of data generation conditions or data augmentation conditions by a neural network model, thereby reducing human consumption and improving data diversity. Figure 1 Schematically shows the overall technical concept of this disclosure. The solution of this disclosure relies on artificial intelligence, specifically, on various neural network models. The neural network model involves two stages: training and data generation. Figure 1 Part (a) of shows the training stage, Figure 1 Part (b) of shows the data generation stage.

[0030] As Figure 1As shown in part (a), in the training phase, a neural network model is trained using multiple data pairs of known or labeled {first data, second data}. For example, the first data may correspond to the original data or the correct data, and the second data may correspond to the deviated data or the incorrect data corresponding to the first data. The training phase involves three neural network models. As Figure 1 As shown in part (a), the third neural network model is used to extract conditional feature vectors based on each data pair of (first data, second data). The conditional feature vector can represent the generation condition or the deviation type of the second data relative to the first data. For example, the conditional feature vector can be the difference between the vector representing the second data and the vector representing the first data, or otherwise indicate the correlation or deviation between the second data and the first data. Semantically, the data deviation type may correspond to semantic deviation, syntactic deviation, etc., or may have no actual semantic meaning. The third neural network model can include any one of various existing feature extractors, such as a convolutional neural network (CNN) model, a recurrent neural network (RNN) model, etc. Using the third neural network model, multiple conditional feature vectors corresponding to multiple data pairs of {first data, second data} can be obtained. Using these multiple conditional feature vectors as input to train the first neural network model, so that the first neural network model can map these multiple conditional feature vectors to a predetermined distribution curve. In the present disclosure, a model capable of implementing a reversible transformation is used as the first neural network model to establish a bi-directional mapping relationship between the conditional feature vector and the predetermined distribution curve, so that in the subsequent data generation phase, the first neural network model can implement the reverse mapping process, that is, obtain the conditional feature vector based on the predetermined distribution curve. For example, the first neural network model can include a flow-based model, a variational autoencoder (VAE), a diffusion model, etc. In addition, as Figure 1As shown in part (a), multiple conditional feature vectors output by the third neural network model are input not only to the first network model but also to the second neural network model, and are used to train the second neural network model together with multiple first data, so that the second neural network model can output second data corresponding to the first data based on the first data and the conditional feature vectors corresponding to the first data. The second neural network model can be implemented using various existing generative models, including, for example, autoregressive models and non-autoregressive models (such as transformers). Regarding the second neural network model and the third neural network model, the two can be trained together, that is, inputting a data pair of (first data, second data) into the third neural network to generate conditional feature vectors, and inputting the conditional feature vectors and the first data into the second neural network model together, while using the second data as the output of the second neural network model to train the second neural network model. Thus, after training the second neural network model and the third neural network model, the second neural network model can output the second data under the foregoing input conditions.

[0031] As Figure 1 shown in part (b), in the data generation stage, the first neural network model and the second neural network model are used to generate augmented data based on input data and a predetermined distribution curve. As described above, the first neural network model can implement a reversible transformation, and it can obtain conditional feature vectors based on the predetermined distribution curve. For example, the predetermined distribution curve can be sampled to obtain sampling points, and the first neural network model can inverse-map the sampling points into conditional feature vectors, that is, automatically generate conditional feature vectors corresponding to the data deviation type. Then, the second neural network model can generate augmented data that deviates from the input data based on the input data and the conditional feature vectors generated by the first neural network. By sampling the predetermined distribution curve multiple times to obtain multiple different sampling points, multiple different augmented data can be generated based on the same input data, thereby realizing data augmentation and the diversity of augmented data. In addition, since there is no need to pre-store a large amount of various data generation conditions with manual annotations in the memory, the occupation of storage space is also reduced.

[0032] Figure 2 FIG. shows a flowchart of a method 200 for data augmentation according to an embodiment of the present disclosure. The method 200 may correspond to the above data generation stage. As Figure 2 shown, the method 200 according to an embodiment of the present disclosure includes steps S210 to S230.

[0033] In step S210, sample a predetermined distribution curve to obtain sampling points. The predetermined distribution curve here can be various distribution curves with known distribution functions. The points on the distribution curve can correspond to conditional feature vectors. Therefore, in the data augmentation process, various conditional feature vectors can be obtained based on the predetermined distribution curve. In one embodiment, the predetermined distribution curve can be a Gaussian distribution curve, such as a standard Gaussian distribution curve. In reality, many random variables often approximately follow a Gaussian distribution. Therefore, using a Gaussian distribution curve as the predetermined distribution curve can make the subsequent generation of conditional feature vectors more meaningful. In addition, the predetermined distribution curve can also be other curves, such as an F-distribution curve, a Poisson distribution curve, etc.

[0034] In step S220, generate a first conditional feature vector based on the sampling points obtained in step S210 through a first neural network model. As described above, the first neural network model can implement an invertible transformation, which can include, for example, a flow-based model, a VAE, a diffusion model, etc. In one embodiment, the first neural network model can include one or more coupling layer models. The first neural network model can implement an inverse mapping process, that is, obtain conditional feature vectors based on the predetermined distribution curve. Specifically, the first neural network model establishes a mapping between conditional feature vectors and the predetermined distribution curve during the training phase, and can map each sampling point on the predetermined distribution curve to a conditional feature vector during the data generation phase. The first conditional feature vector generated in step S220 can indicate the data deviation type as described above, that is, the deviation type of the augmented data to be generated relative to the input data. As described above, it can be understood that this data deviation type does not necessarily have an actual semantic meaning.

[0035] In step S230, input the input data and the first conditional feature vector generated in step S220 into a second neural network model to generate augmented data. As described above, the second neural network model can be implemented using various existing generation models, including, for example, an autoregressive model and a non-autoregressive model, etc. Before using the second neural network model for data generation, it has been trained so that it can take data and conditional feature vectors as inputs and produce augmented data as outputs. Since the first conditional feature vector can indicate the data deviation type, the data generated by the second neural network model is the deviated data generated after offsetting the input data, thereby realizing data augmentation.

[0036] As understood by those skilled in the art and as described above, before using a neural network model for data generation, it can be trained first. In the training stage, multiple data pairs of {first data, second data} can be used to train the second neural network model and the third neural network model. The first data can correspond to the original data or correct data, and the second data can correspond to the deviated data or incorrect data corresponding to the first data. In one embodiment, the second neural network model can be trained such that: in the case where the third neural network model generates a second conditional feature vector according to the first data and the second data, the second neural network model can generate the second data according to the first data and the second conditional feature vector. That is to say, multiple data pairs of {first data, second data} and the corresponding multiple second conditional feature vectors can be used as the training data of the second neural network model, so that the second neural network model can take the first data and the corresponding second conditional feature vector as inputs and output the second data corresponding to the first data and the second conditional feature vector. The above-mentioned third neural network model can include any one of various existing feature extractors, such as a convolutional neural network (CNN) model, a recurrent neural network (RNN) model, etc. As described above, the third neural network model can extract the above-mentioned second conditional feature vector based on each data pair of (first data, second data). The second conditional feature vector can represent the generation condition or deviation type of the second data relative to the first data. For example, the second conditional feature vector can be the difference between the vector representing the second data and the vector representing the first data. It can be understood that the term "second conditional feature vector" is used here to describe the conditional feature vector generated by the third neural network model based on the multiple data pairs of {first data, second data} in the training stage, in order to distinguish it from the "first conditional feature vector" generated by the first neural network model in the data generation stage. Here, "first" and "second" do not intend to represent any preference, order, or interdependence relationship.

[0037] In addition, in the training stage, the above-mentioned multiple second conditional feature vectors generated by the third neural network model based on the multiple data pairs of {first data, second data} can also be used to train the first neural network model. In one embodiment, the first neural network model can be trained such that: in the case where the third neural network model generates a second conditional feature vector according to the first data and the second data, the first neural network model can map the second conditional feature vector to a predetermined distribution curve. As described above, the first neural network model can achieve a reversible transformation. In the case where the first neural network model can achieve the mapping of the second conditional feature vector to the predetermined distribution curve in the training stage, in the data generation stage, the first neural network model will be able to achieve an inverse mapping, that is, based on the sampling points on the predetermined distribution curve, generate the corresponding conditional feature vector, that is, the first conditional feature vector in the above step S220.

[0038] In the method for data augmentation according to an embodiment of the present disclosure, data generation conditions (i.e., the above-mentioned first conditional feature vectors) can be automatically generated without manual definition or annotation, so that output data can be generated from input data based on the automatically generated data generation conditions, thereby realizing data augmentation. This reduces human consumption and improves data diversity.

[0039] Embodiments of the present disclosure also provide a device for data augmentation. Figure 3 A schematic diagram of a device 300 for data augmentation according to an embodiment of the present disclosure is shown.

[0040] As Figure 3 shown, the device 300 for data augmentation according to an embodiment of the present disclosure includes a processor 310 and a memory 320. The memory 320 stores one or more computer programs.

[0041] The processor 310 is a program control device such as a microprocessor. For example, it operates according to the program installed in the memory 320. The memory 320 is a storage element such as ROM or RAM, etc. The program to be executed by the processor 310 etc. is stored in the memory 320. Figure 3 The shown device 300 for data augmentation can be used to implement the method for data augmentation disclosed in the present application.

[0042] In the device according to an embodiment of the present disclosure, data generation conditions can be automatically generated without manual definition or annotation, so that output data can be generated from input data based on the automatically generated data generation conditions, thereby realizing data augmentation. This reduces human consumption and improves data diversity.

[0043] Embodiments of the present disclosure also provide a computer program product. The computer program product includes a computer program which, when executed by a processor, can implement the method for data augmentation disclosed in the present application.

[0044] Embodiments of the present disclosure also provide a non-transitory computer-readable storage medium. Computer program instructions are stored on the computer-readable storage medium. When the computer program instructions are executed by a processor, the processor is caused to execute the method for data augmentation disclosed in the present application.

[0045] The method, device, computer program product, and non-transitory computer-readable storage medium for data augmentation according to embodiments of the present disclosure can be applied to various data processing fields, such as the text data processing field or the image data processing field. In the case of being applied to the text data processing field, the input data and the augmented data involved in step S230 above can be text data and deviated text data deviating from the text data, respectively. The deviated text data can be, for example, text data that varies from the input text data in terms of semantics, grammar, structure, etc. For example, the deviated text data can include words that are homophonic with the words in the input text data, different words that have the same or similar pronunciations as the words in the input text data, and so on. In the case of being applied to the image data processing field, the input data and the augmented data involved in step S230 above can be image data and deviated image data deviating from the image data, respectively. The deviated image data can be, for example, image data that varies from the input image data in terms of color, element position, etc. For example, the deviated image data can include elements with different colors from the elements in the input image data, elements with different positions from the elements in the input image data (such as elements obtained after translation or rotation), and so on.

[0046] The method, device, computer program product, and non-transitory computer-readable storage medium for data augmentation according to embodiments of the present disclosure are particularly applicable to scenarios where multiple outputs are generated based on one input. For example, the method, device, computer program product, and non-transitory computer-readable storage medium for data augmentation according to embodiments of the present disclosure can be applicable to scenarios where multiple incorrect texts are generated based on one correct text, scenarios where multiple sentence rewriting results are generated based on one sentence, scenarios where multiple summary results are generated based on one text content, scenarios where multiple translation results in another language are generated based on a text in one language, scenarios where multiple pictures with different colors or pictures with changed positions of elements in the picture are generated based on one picture, and so on.

[0047] The whole or its components of the hardware computing device described in the present disclosure can be implemented by various suitable hardware means, including but not limited to FPGA, ASIC, SoC, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The devices, equipment, methods, and systems involved in the present disclosure are not limited to any specific hardware architecture or configuration. The components in the disclosed devices, equipment, and systems can be separated or integrated, and can be combined in different ways and / or replaced or supplemented by other components. It should be understood that the teachings of the present disclosure can be implemented in various forms of hardware, software, firmware, dedicated processors, or combinations thereof.

[0048] The block diagrams of the devices, equipment, methods, and systems involved in this disclosure are merely exemplary and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these circuits, devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any way as long as the desired purpose can be achieved.

[0049] In the above description, the present invention has been described based on embodiments. These embodiments are merely illustrative, and those skilled in the art should understand that the combination of the constituent elements and processes of these embodiments can be modified in various ways, and such modifications are also within the scope of the present invention.

Claims

1. A method for data augmentation, comprising: Sampling a predetermined distribution curve to obtain sampling points; Generating a first conditional feature vector based on the sampling points by a first neural network model, the first neural network model being capable of implementing a reversible transformation; And Inputting the input data and the first conditional feature vector into a second neural network model to generate augmented data.

2. The method according to claim 1, wherein, The second neural network model is trained such that: In the case where a second conditional feature vector is generated by a third neural network model according to first data and second data, the second neural network model can generate the second data according to the first data and the second conditional feature vector.

3. The method according to claim 1, wherein The first neural network model is trained such that: In the case where a second conditional feature vector is generated by a third neural network model according to first data and second data, the first neural network model can map the second conditional feature vector onto the predetermined distribution curve.

4. The method according to any one of claims 1-3, wherein the first conditional feature vector or the second conditional feature vector indicates a data deviation type.

5. The method according to any one of claims 1-3, wherein the predetermined distribution curve is a Gaussian distribution curve.

6. The method according to any one of claims 1-3, wherein the first neural network model includes one or more coupling layer models.

7. The method according to any one of claims 1-3, wherein the input data is text data, and the augmented data is deviated text data deviating from the text data.

8. The method according to any one of claims 1-3, wherein the input data is image data, and the augmented data is deviated image data deviating from the image data.

9. The method according to any one of claims 1-3, wherein the second neural network model includes an autoregressive model.

10. The method according to any one of claims 2-3, wherein the third neural network model includes a feature extractor.

11. A device for data augmentation, comprising: A processor; And A memory storing one or more computer programs; Wherein, when the one or more computer programs are run by the processor, the processor executes the method according to any one of claims 1-10.

12. A computer program product, comprising a computer program, which implements the method according to any one of claims 1-10 when executed by a processor.

Images