Data processing method, device and computer program product

Abstract

The data processing method comprises the steps of generating a first number of first hidden vectors based on input data through a first neural network model, wherein the first number is greater than 1; respectively mapping the first number of first hidden vectors to the first number of predetermined distribution curves, wherein each of the first number of first hidden vectors respectively corresponds to a numerical position on a predetermined distribution curve; sampling each of the first number of predetermined distribution curves to obtain a first numerical value group, the first numerical value group comprising the first number of numerical values, and each numerical value corresponding to a numerical value position on a predetermined distribution curve; and output data is generated based on the first numerical group through a second neural network model, each of the first number of first hidden vectors respectively expresses a part of information of the input data, and the information expressed by the first hidden vectors is not completely overlapped.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular, to a data processing method, device, and computer program product, which can be used for data augmentation. Background Art

[0002] Currently, deep learning models have achieved success in multiple fields. However, deep learning models rely on a large amount of labeled data for training, and there are often problems such as poor performance of deep learning models due to lack of labeled data and unbalanced data distribution. Data augmentation is a technique for automatically augmenting training data, which can reduce the cost of obtaining labeled data and thus improve the performance of deep learning models. Summary of the Invention

[0003] The purpose of the present application is to provide a data processing method, device, computer program product, and non-transitory computer-readable storage medium, which can generate different output data corresponding to the input data based on the input data, and improve the quality of the output data when the length of the input data is long while ensuring the diversity of the output data.

[0004] According to an aspect of the present disclosure, a data processing method is provided, including: generating a first number of first hidden vectors based on input data through a first neural network model, where the first number is greater than 1; mapping each of the first number of first hidden vectors to a first number of predetermined distribution curves, and each of the first number of first hidden vectors corresponds to a numerical position on one of the first number of predetermined distribution curves; sampling each of the first number of predetermined distribution curves to obtain a first numerical group, where the first numerical group includes the first number of numerical values, and each numerical value corresponds to a numerical position on one of the first number of predetermined distribution curves; and generating output data based on the first numerical group through a second neural network model, where each of the first number of first hidden vectors respectively represents a part of the information of the input data, and the information represented by each of the first hidden vectors does not completely overlap.

[0005] In some embodiments, the first neural network model includes: an encoding unit that encodes the input data to generate a second hidden vector; and the first number of information extraction units, and each information extraction unit extracts a part of the information of the second hidden vector to respectively generate each first hidden vector.

[0006] In some embodiments, the information extraction unit uses an attention mechanism to generate the first hidden vector.

[0007] In some embodiments, the first quantity of first hidden vectors are respectively mapped to the first quantity of predetermined distribution curves based on a reversible generative model.

[0008] In some embodiments, the reversible generative model is a flow-based model or a diffusion-based model.

[0009] In some embodiments, the first quantity of first hidden vectors are respectively mapped to the first quantity of predetermined distribution curves based on a variational autoencoder.

[0010] In some embodiments, the input data is text data.

[0011] In some embodiments, the first numerical group is different from the numerical group corresponding to the numerical position of the input data in the first quantity of predetermined distribution curves.

[0012] According to one aspect of the present disclosure, there is provided an apparatus for performing a data processing method, including: a processor; and a memory storing one or more computer programs; wherein, when the one or more computer programs are run by the processor, the processor is caused to execute the data processing method as described above.

[0013] According to one aspect of the present disclosure, there is provided a computer program product including a computer program which, when executed by a processor, implements the data processing method as described above.

[0014] According to one aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having computer program instructions stored thereon. The computer program instructions, when executed by a processor, cause the processor to execute the above data processing method.

[0015] According to the data processing method, device, computer program product, and non-transitory computer-readable storage medium provided by the present disclosure, during the data processing, different information in the input data is expressed by a plurality of first hidden vectors, more information of the input data can be retained, the quality of the output data when the length of the input data is long can be improved, and furthermore, the plurality of first hidden vectors are respectively mapped to different predetermined distribution curves, and by sampling on the predetermined distribution curves, the diversity of the data processing results can be ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a schematic diagram of an example system for implementing the data processing method according to an embodiment of the present disclosure.

[0017] Figure 2 is a flowchart of the data processing method according to an embodiment of the present disclosure.

[0018] Figure 3It is a flowchart showing the generation of a first hidden vector by a first neural network model according to an embodiment of the present disclosure.

[0019] Figure 4 It is a flowchart showing that a second neural network model generates output data based on a first numerical group according to an embodiment of the present disclosure.

[0020] Figure 5 It is a flowchart showing the training process of a model applied in a data processing method related to the present disclosure.

[0021] Figure 6 It is a schematic diagram of a computing device according to an embodiment of the present disclosure.

[0022] Specific implementation

[0023] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not used to limit the protection scope of the present disclosure.

[0024] The term "including" and its variants used herein are open-ended, that is, "including but not limited to". The term "based on" is "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0025] It should be understood that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules, units, models, data, etc., and are not used to limit the order of functions executed by these devices, modules, units, models, data, the order of generation, or the interdependent relationship.

[0026] It should be noted that the modification of "one" and "multiple" mentioned in the present disclosure is illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly specified in the context, it should be understood as "one or more".

[0027] In existing data augmentation methods, there are problems such as poor diversity of the generated augmented data and poor quality of the augmented data when the length of the input data is long. For example, using existing data augmentation methods, only one augmented data may be generated from one input data, resulting in poor data diversity. In addition, when the input data is text and the length of the text is long, the text obtained by using existing data augmentation methods may be a sequence of characters whose meaning cannot be understood, that is, the quality of the augmented data is poor and it cannot be used as valid data.

[0028] The present disclosure is made in view of the above problems. Through the data processing method, device, computer program product, and non-transitory computer-readable storage medium of the embodiments of the present disclosure, the diversity of augmented data can be achieved, and the quality of the augmented data can be improved.

[0029] The concept of the present disclosure is to extract different information of the input data by setting multiple hidden vectors, map the multiple hidden vectors to a predetermined distribution curve, sample on the predetermined distribution curve, and decode the sampling result to obtain the output data.

[0030] Figure 1 It is a schematic diagram of an example system for implementing the data processing method according to the embodiments of the present disclosure.

[0031] As shown in Figure 1 the figure, the system for implementing the data processing method of the embodiments of the present disclosure may include a first neural network model 110 and a second neural network model 120. The first neural network model 110 processes the input information and generates multiple first hidden vectors. For simplicity, the number of multiple first hidden vectors is illustrated as 2 below. Correspondingly, Figure 1 schematically shows 2 first hidden vectors h 112 , h 113 . As an example, the first neural network model 110 may include an encoding unit 111 and 2 information extraction units 112, 113 to perform the above operations. The 2 first hidden vectors can be respectively mapped to 2 predetermined distribution curves 130, 131, and each first hidden vector corresponds to a numerical position on the predetermined distribution curve. For example, the second neural network model 120 may include hidden vector mapping units 122, 123 to map the first hidden vectors h 112 and h 113 to a numerical position on the predetermined distribution curves 130, 131 respectively (at Figure 1in which points indicated by small hollow circles on the predetermined distribution curves 130 and 131 are shown). Thus, a combination of a numerical point on the predetermined distribution curve 130 and a numerical point on the predetermined distribution curve 131 can represent a set of two first hidden vectors, and the set of two first hidden vectors can correspond to an input data (e.g., a text). Subsequently, the two predetermined distribution curves 130 and 131 can be sampled to obtain a first numerical set (in Figure 1 in which points indicated by solid dots on the predetermined distribution curves 130 and 131 are shown), the first numerical set includes two numerical values, and each numerical value corresponds to a numerical position on a predetermined distribution curve respectively. The second neural network model 120 can process based on the first numerical set to generate output data. As an example, the second neural network model 120 can include a decoding unit 121 and the above two hidden vector mapping units 122 and 123 to generate output data based on the sampled numerical set. As an example, the hidden vector mapping units 122 and 123 can be invertible generative models, that is, they can map the first hidden vector to a numerical point on the predetermined distribution curve and can also generate the first hidden vector based on the numerical point on the predetermined distribution curve. Thus, the hidden vector mapping units 122 and 123 can convert the numerical values in the adopted first numerical set into the first hidden vector. Subsequently, the decoding unit 121 can decode the converted first hidden vector into output data, and the decoding unit 121 can be any suitable neural network model, such as a decoder. In the embodiments of the present disclosure, the numerical values in the first numerical set can be any points on the predetermined distribution curves 130 and 131. In some embodiments, in order to achieve the correlation between the output data and the input data, the relationship between the numerical positions in the first numerical set and the numerical positions mapped by the input data on the predetermined distribution curve can be set, such as setting a predetermined offset.

[0032] It should be noted that Figure 1 is only an example of the system structure for implementing the embodiments of the present disclosure, and those skilled in the art clearly know that other system structures can be adopted to implement the embodiments of the present disclosure. For example, the hidden vector mapping units 122 and 123 do not have to be invertible generative models, as long as they can achieve the mapping from the first hidden vector to the predetermined distribution curve and can also achieve the mapping from the sampling on the predetermined distribution curve to the first hidden vector. The hidden vector mapping units 122 and 123 can also not belong to a part of the second neural network model 120, and the second neural network model 120 can directly generate output data according to the predetermined distribution curve without first obtaining the first hidden vector through the hidden vector mapping units 122 and 123. Additionally, it can be understood that the first neural network model 110 does not necessarily adopt the form including an encoding unit and multiple information extraction units, as long as it can process the input information and generate multiple first hidden vectors.

[0033] In addition, the neural network model according to the present disclosure can adopt various known models and be trained using common rules.

[0034] According to an embodiment of the present disclosure, multiple first hidden vectors can be generated based on input data, and the multiple first hidden vectors can be mapped to multiple predetermined distribution curves. Different information in the input data can be expressed through the multiple distribution curves, which can retain more information of the input data, thereby improving the quality of the output data when the length of the input data is long; sampling can be performed as needed on different predetermined distribution curves, which can achieve the diversity of data processing results. In addition, since various data can be generated quickly and diversely with high quality, it is not necessary to pre-store a large amount of data, but the required data can be generated at any time as needed, thereby reducing the occupation of the storage space of the computer.

[0035] Figure 2 It is a flowchart of a data processing method 200 according to an embodiment of the present disclosure. The data processing method 200 includes steps S210 to S240. Similar to Figure 1 , for simplicity, the number of multiple hidden vectors is taken as 2 for description below.

[0036] As Figure 2 shown, in step S210 of the data processing method of the present disclosure, through the first neural network model 110, a first number (2) of first hidden vectors h 112 , h 113 are generated based on the input data. Then, in step S220, the 2 first hidden vectors are respectively mapped to 2 predetermined distribution curves 130, 131, and each of the first number of first hidden vectors corresponds to a numerical position on the predetermined distribution curve. Then, in step S230, sampling is performed on the 2 predetermined distribution curves 130, 131 to obtain a first numerical group, the first numerical group includes 2 numerical values, and each numerical value corresponds to a numerical position on a predetermined distribution curve respectively. Then, in step S240, the second neural network model 120 generates output data based on the first numerical group. The above 2 first hidden vectors h 112 , h 113Each of them respectively represents a part of the information of the input data, and the information represented by each first hidden vector does not completely overlap. Thus, by mapping two first hidden vectors to two corresponding predetermined distribution curves, different information in the input data can be represented by the two predetermined distribution curves, so that more information of the input data can be retained. Thus, more information can be utilized to improve the quality of the generated data when generating the output data, which is particularly advantageous when the length of the input data is long. For example, when the input data is long text data, the information of the input data can include, for example, nouns, verbs, subject-predicate relationships, etc. in the text. Since multiple first hidden vectors as described above are adopted and the multiple first hidden vectors are mapped to multiple predetermined distribution curves, these information can be retained more, so that the generated output text can be a sequence of words whose meaning can be better understood. In addition, since sampling can be performed as needed on different predetermined distribution curves, the diversity of the data processing results can be achieved.

[0037] In one embodiment, the input data may be text data. More specifically, the input data may be a sentence. Here, the sentence is just an example. The input data may be words shorter than a sentence or paragraphs longer than a sentence, etc. In this case, before performing method 200, the input data may be preprocessed first. When using a computer to execute the data processing method, since the computer cannot directly process a sentence in the form of natural language, the sentence needs to be converted into a vector form first. For example, each word in the sentence can be respectively converted into a vector of a specific dimension through an Embedding operation. Or, the sentence can be tokenized first, and then the multiple words after tokenization can be respectively converted into vectors of a specific dimension. When applying the data processing method involved in the present disclosure to process a sentence, the process of converting a sentence in the form of natural language into a vector form can be regarded as preprocessing, and existing methods can be used for the Embedding operation, which will not be elaborated here. Hereinafter, in order to more specifically describe the data processing method involved in the present disclosure, the input sentence will no longer be distinguished as being in the form of natural language or vector form. Those skilled in the art can convert the sentence into a vector form in any possible way according to needs, and then use the data processing method involved in the present disclosure for processing.

[0038] When the input data is text data, the data processing method 200 can, for example, take a sentence as the input data, and in step S210, generate a first number of first hidden vectors h 112 、h 113 based on the input data through the first neural network model 110. The following specifically describes step S210 Figure 2 therein.

[0039] As described above, for simplicity in the present disclosure, the number of the first hidden vectors (i.e., the first number) is described as 2. However, the first hidden vectors are not limited to 2, and the number of the first hidden vectors can be any integer greater than 1. The first number can be set during training. In some embodiments, the first number can be used as a hyperparameter and set or adjusted according to the length of the data in the training dataset.

[0040] As described above, in order to improve the quality of the output data, the data processing method of the present disclosure seeks to extract information from the input data through multiple first hidden vectors for generating the output data. For this purpose, it should be ensured that the information of the input data extracted by each first hidden vector does not completely overlap.

[0041] In some embodiments, the input data can be divided into different parts, and the first neural network model 110 can include an encoding unit 111 to encode different parts of the input data to obtain different encoded vectors as different first hidden vectors, so as to enable each first hidden vector to extract different information from the input information.

[0042] In some other embodiments, in addition to the encoding unit 111, the first neural network model 110 can further include a first number (e.g., 2) of information extraction units 112 and 113. The information extraction units in the first neural network model can be set such that the information extraction units 112 and 113 respectively extract different parts of the encoded vector output by the encoding unit 111, or extract different parts of the encoded vector output by the encoding unit 111 using different weights, thereby generating different first hidden vectors. For example, the information extraction units 112 and 113 can be associated with different weight vectors, and by using different weight vectors to perform operations on the encoded vector of the input data, different first hidden vectors can be generated, such that the information of the input data extracted by each first hidden vector does not completely overlap.

[0043] For example, different labels can be set for the information extraction units 112 and 113, and the self-attention mechanism between the vectors of the labels is used to ensure that the vectors of each label are different. Subsequently, the cross-attention mechanism can be used to perform operations on the vectors of the labels and the encoded vector of the input data to generate the first hidden vectors, such that the information of the input data extracted by each first hidden vector does not completely overlap. It should be noted that the above embodiments are only examples, and those skilled in the art can adopt various ways to ensure that the information of the input data extracted by each first hidden vector does not completely overlap.

[0044] Figure 3 FIG. shows an example flowchart of generating the first hidden vector by the first neural network model according to an embodiment of the present disclosure. As Figure 3As shown, the above step S210 may include sub-steps S211 to S213. In step S211, the label vector with self-attention weights generated during the training of the first neural network model may be read.

[0045] The process of generating the label vector with self-attention weights during the training of the first neural network model will be described below.

[0046] Specifically, each information extraction unit is preset with a label during training, and the labels of each information extraction unit are different from each other. The label can be, for example, a number, a letter, or other forms. For example, the information extraction unit 112 may be preset with the label "1" during training, and the information extraction unit 113 is preset with the label "2". The labels "1" of the information extraction unit 112 and "2" of the information extraction unit 113 can be processed respectively to convert the labels of each information extraction unit into different vectors. For example, the same method as the Embedding operation used in the preprocessing of the input data can be adopted to process the labels of the information extraction units 112 and 113 to generate the label vectors l 112 、l 113 . Among them, the dimensions of the label vectors l 112 、l 113 can be set flexibly. For example, in the case of using "1" and "2" as labels, the dimensions of the label vectors l 112 、l 113 can be set to 1*256, etc.

[0047] Then, the two label vectors l 112 and l 113 are operated based on the self-attention mechanism to obtain the label vectors sa 112 and sa 113 with self-attention weights. Specifically, the label vector with self-attention weights can be calculated, for example, by the following formula (1).

[0048]

[0049] Taking the above label vectors l 112 、l 113 as the first row and the second row of matrix I respectively. Since the self-attention mechanism is adopted here, the matrices Q, K, and V in the above formula (1) are respectively the inner product operations of matrix I with matrices W Q 、W K 、W V , and can be recorded as follows:

[0050] Q = IW q ;

[0051] K = IW k ;

[0052] V = IW v 。

[0053] Wherein, W q 、W k 、W v are parameter matrices obtained through pre-training. The dimensions of each parameter matrix are related to the dimensions of the label vectors obtained after performing an Embedding operation on the labels of the information extraction units. More specifically, the number of rows of each parameter matrix is equal to the number of columns of the label vector. For example, when the dimension of the label vector l 112 is set to 1*256, the dimensions of W q 、W k 、W v can be set to 256*x, where x is any suitable integer that can be set. In addition, "K T " in the above formula (1) represents the transpose of K, Softmax() is the normalized exponential function, and d k represents the size of the dimension (number of columns) of K.

[0054] At this time, the Attention(Q, K, V) obtained by calculating the above formula (1) is a matrix with a dimension of 2*x. The first row of this matrix is the label vector sa 112 with self-attention weights, and the second row is the label vector sa 113 with self-attention weights.

[0055] It should be noted that the first quantity and the labels of each information extraction unit are determined during the training phase. When using the data processing method of the present disclosure to generate output data, the first quantity and the labels of each information extraction unit will not change. Further, the parameter matrices used when calculating the self-attention weights are determined at the end of the training. When using the data processing method of the present disclosure to generate data, the parameter matrices used when calculating the self-attention weights will not change. Therefore, the label vectors with self-attention weights can be saved at the end of the training, and when applying the data processing method to generate output data, the label vectors with self-attention weights can be directly read and used for subsequent steps.

[0056] Returning to Figure 3 , after reading the label vectors sa 112 and sa 113 with self-attention weights in step S211, in step S212, the encoding unit 111 encodes the input data to generate a second hidden vector h 111 .

[0057] In some embodiments, the encoding unit 111 may include a pre-trained neural network that encodes the input data to generate a second hidden vector h 111 . The second hidden vector h 111 may be related to the length of the input data. For example, when the input data is a sentence with 7 characters, the dimension of the second hidden vector may be 7 * 256. Here, the number of columns of the second hidden vector is determined during the Embedding operation on the input data and is equal to the number of columns of the vector obtained after the Embedding operation on the labels of the information extraction units 112 and 113 as described above.

[0058] The encoding unit 111 may include any neural network capable of encoding input data, such as but not limited to a recurrent neural network (RNN), a bidirectional recurrent neural network (BiRNN), a long short-term memory network (LSTM), a bidirectional long short-term memory network (BiLSTM), etc. The present disclosure does not particularly limit the type or structure of the neural network of the encoding unit.

[0059] Here, the step of generating the second hidden vector h 111 , i.e., step S212, may also be executed before the step of reading the label vectors sa 112 and sa 113 with self-attention weights, i.e., step S211. In addition, step S211 and step S212 may also be executed in parallel.

[0060] After that, in step S213, the information extraction units 112 and 113 calculate the cross-attention weights of the label vectors sa 112 and sa 113 with self-attention weights read in the above step S211, and the second hidden vector h 111 generated in the above step S212, respectively, based on the cross-attention mechanism, to generate the first hidden vectors h 112 and h 113 with cross-attention weights, i.e., the first hidden vector h 111 and h 112 and h 113 .

[0061] Specifically, taking the label vectors sa 112 and sa 113 with self-attention weights as the first row and the second row of the matrix J respectively, calculate the cross-attention weights of the label vectors sa 112 and sa 113 (constituting the matrix J) and the above second hidden vector h 111The cross-attention, at this time, Q in the above formula (1) is the second hidden vector h 111 obtained by taking the inner product operation with the vector W Q ', K and V are respectively obtained by taking the inner product operation of the matrix J with W k ' and W v ', and can be denoted as follows:

[0062] Q = h 111 W q ';

[0063] K = JW k ';

[0064] V = JW v '.

[0065] Among them, W q ', W k ', W v ' are parameter matrices obtained through pre-training. The dimensions of each parameter matrix can be related to the dimensions of the vectors obtained after performing Embedding operations on the labels of the information extraction unit and on the input data. More specifically, the number of rows of each parameter matrix is equal to the number of columns of the label vector and the second hidden vector. For example, when the dimension of the second hidden vector is 7*256, the dimensions of W q , W k , W v can be set to 256*x, where x is any suitable integer that can be set.

[0066] At this time, the Attention(Q, K, V) obtained by calculating the above formula (1) is a matrix with a dimension of 2*x. The first row of this matrix is the first hidden vector h 112 , and the second row is the first hidden vector h 113 .

[0067] It should be noted that the above formula (1) for calculating self-attention and cross-attention is merely an example, and other suitable formulas can be used to calculate the label vectors sa 112 , sa 113 and / or the first hidden vectors h 112 , h 113 . In addition, the dimensions of the above matrices and vectors are also merely an example, and the matrices and vectors can adopt other suitable dimensions.

[0068] In some embodiments, each of the first quantity of first hidden vectors h 112 , h 113 respectively represents a part of the information of the input data. The first hidden vector h 112 and h113 The expressed information does not completely overlap.

[0069] Specifically, due to the inertia characteristics of the training process of the neural network, it will not repeat learning the features that have been mastered. During training, by assigning different labels to each information extraction unit, in step S211, the label vector with self-attention weights is read, and in step S213, the label vector with self-attention weights is used to perform cross-attention operations with the second hidden vector generated by the encoding unit, so that the generated first hidden vector h 112 will no longer learn the first hidden vector h 113 the knowledge that has been mastered, and vice versa. Such a design can ensure that the first hidden vector h 112 、h 113 generated by the first neural network model 110 are different, have their own unique features, and can express different information in the input data. When the appropriate first quantity is set according to the data length of the data set, compared with other data processing methods, it can improve the expression ability of the input data and improve the quality of the output data when the length of the input data is long.

[0070] Return Figure 2 , in step S210, the first neural network model 110 generates the first quantity of first hidden vectors h 112 、h 113 After that, in step S220, each first hidden vector h 112 、h 113 is respectively mapped to the first quantity of predetermined distribution curves, and each of the first quantity of first hidden vectors h 112 、h 113 corresponds to a numerical position on a predetermined distribution curve respectively. The second neural network model 120 may include the first quantity (for example, 2) of hidden vector mapping units ( Figure 1 122, 123 in

[0071] In some embodiments, the predetermined distribution curve may be a Gaussian distribution curve, but is not limited to the Gaussian distribution curve, and may be other known distribution curves.

[0072] In some embodiments, the first quantity of first hidden vectors h 112 、h 113Are respectively mapped to a first number of predetermined distribution curves. In other words, the hidden vector mapping units 122 and 123 can be respectively an invertible generative model. In this embodiment, the invertible generative model and the variational autoencoder described later are both examples of the hidden vector mapping unit. As described above, in some embodiments, the second neural network model 120 can include a first number (2) of such hidden vector mapping units ( Figure 1 122 and 123 in

[0073] In some embodiments, the invertible generative model can be a Flow-based model or a Diffusion-based model.

[0074] In other embodiments, a first number of first hidden vectors h 112 and h 113 Can be respectively mapped to a first number of predetermined distribution curves based on a Variational AutoEncoder (VAE). In other words, the hidden vector mapping units 122 and 123 can be respectively a VAE.

[0075] Here, the flow-based model, the diffusion-based model, and the VAE are all used to map the first hidden vectors h 112 and h 113 Output by the first neural network model 110 to a predetermined distribution curve. In this embodiment, other models capable of achieving the above purposes can also be used, and multiple variants of the flow-based model, the diffusion-based model, and the VAE can be used respectively.

[0076] Hereinafter, the case where the hidden vector mapping units 122 and 123 are a Flow-based model and the predetermined distribution curve is a Gaussian distribution curve will be taken as an example for explanation.

[0077] In Figure 1 In the example of, in the second neural network model 120, the first hidden vectors h 112 and h 113 Can be respectively mapped to the Gaussian distribution curves 130 and 131 by the flow-based models 122 and 123. Among them, the flow-based models 122 and 123 are pre-trained. At this time, the flow-based model can receive a first hidden vector as input and output a numerical position on the Gaussian distribution curve. Each first hidden vector corresponds to a numerical position. When the first neural network model 110 receives input data, the numerical positions on the respective Gaussian distribution curves to which the first hidden vectors generated based on the input data are mapped are different from the numerical positions on the same Gaussian distribution curve to which the first hidden vectors generated based on other input data are mapped.

[0078] For example, assume that the two first hidden vectors generated based on the input data a are denoted as h 112,a and h 113,a . The flow-based model 122 maps h 112,a to a numerical position on the Gaussian distribution curve 130 denoted as g 130,a . The flow-based model 123 maps h 113,a to a numerical position on the Gaussian distribution curve 131 denoted as g 131,a . In addition, assume that the first hidden vectors generated based on the input data b different from the input data a are denoted as h 112,b and h 113,b . The flow-based model 122 maps h 112,b to a numerical position on the Gaussian distribution curve 130 denoted as g 130,b . The flow-based model 123 maps h 113,b to a numerical position on the Gaussian distribution curve 131 denoted as g 131,b . At this time, g 130,a is different from g 130,b and / or g 131,a is different from g 131,b .

[0079] Return Figure 2 . In step S220, after mapping the first quantity of first hidden vectors h 112 , h 113 to the first quantity of predetermined distribution curves respectively, in step S230, each of the above first quantity of predetermined distribution curves is sampled to obtain a first numerical group. The first numerical group includes the first quantity of numerical values, and each numerical value corresponds to a numerical position on one of the predetermined distribution curves. Hereinafter, step S230 will be described by taking the predetermined distribution curve as a Gaussian distribution curve.

[0080] The first numerical group may be different from the numerical group corresponding to the numerical positions of the input data on the first quantity of Gaussian distribution curves. In other words, on each Gaussian distribution curve, sampling can be performed at numerical positions different from the numerical positions to which the multiple first hidden vectors corresponding to the current input data are mapped on each Gaussian distribution curve.

[0081] Specifically, for example, assume that the current input data is a. The flow-based model 122 maps h 112,a to a numerical position g 130,a on the Gaussian distribution curve 130. The flow-based model 123 maps h 113,a to a numerical position g 131,a on the Gaussian distribution curve 131. When sampling, a numerical position g 130,a different from the numerical position g130,a ’ is sampled, and sampling can be performed on a numerical position different from the numerical position g on the Gaussian distribution curve 131 131,a at a different numerical position g 131,a ’. At this time, the numerical value group corresponding to the numerical position of the input data a in the Gaussian distribution curves 130 and 131 can be denoted as [g 130,a , g 131,a , and the first numerical value group obtained by sampling can be denoted as [g 130,a ’, g 131,a ’].

[0082] In some embodiments, the sampling may be sampling of a numerical position offset based on a random number.

[0083] For example, by generating a set of non-zero random numbers, the elements obtained by adding, subtracting, or multiplying each element in the numerical value group [g 130,a , g 131,a by the random number can be used as the elements of the first numerical value group. Specifically, assuming the numerical value group is [g 130,a , g 131,a , and the first quantity of random numbers generated is [r1, r2], then g 130,a ’ = g 130,a ⊙ r1, g 131,a ’ = g 131,a ⊙ r2. Wherein, "⊙" can be any operation such as addition, subtraction, multiplication, or other operations.

[0084] In addition, the sampling may be sampling of a numerical position close to each element in the numerical value group.

[0085] In this case, for example, the value range of each element in the above random numbers [r1, r2] can be limited. For example, when "⊙" is addition or subtraction, r1 and / or r2 can be made to take values close to 0, and when "⊙" is multiplication, r1 and / or r2 can be made to take values close to 1. In this case, the difference between the output data finally generated by the data processing method and the input data can be made smaller than the case where the value range of the random number is not limited.

[0086] In addition, instead of using random numbers, the magnitude of the offset at the numerical position in the numerical value group during sampling can be specified.

[0087] Which sampling method to use can be flexibly selected according to the purpose of applying the data processing method or the size of the data set, etc.

[0088] Return Figure 2 , after obtaining the first numerical value group in step S230, in step S240, through the second neural network model 120, output data is generated based on the first numerical value group. The following refers to Figure 4Description is given to step S240. Figure 4 It is a flowchart showing that the second neural network model generates output data based on the first numerical group. As Figure 4 shown, step S240 may specifically include sub-steps S241 to S242.

[0089] In step S241, first, through the flow-based models 122 and 123, the first numerical group representing the numerical positions on the Gaussian distribution curve obtained by sampling is converted into the first hidden vectors h 112 、h 113 . Here, sampling restoration refers to the process in which the hidden vector mapping unit generates the first hidden vector based on the numerical positions on the Gaussian distribution curve. In the above example, after obtaining the first numerical group [g 130,a ’, g 131,a ’] by sampling, the flow-based model 122 generates the first hidden vector h 130,a after sampling restoration based on g 112 . In addition, the flow-based model 122 generates the first hidden vector h 131,a after sampling restoration based on g 113 .

[0090] Then, in step S242, based on the first hidden vectors h 112 、h 113 after sampling restoration, output data is generated. Here, the second neural network model 120 may further include a decoding unit 121.

[0091] The decoding unit 121 may include any pre-trained neural network capable of decoding the first hidden vectors h 112 、h 113 after sampling restoration, such as but not limited to a recurrent neural network (RNN), a bidirectional recurrent neural network (BiRNN), a long short-term memory network (LSTM), a bidirectional long short-term memory network (BiLSTM), etc. The present disclosure does not make a particular limitation on the type or structure of the neural network of the decoding unit 121.

[0092] Since in the above step S230, sampling is performed on the numerical positions on the predetermined distribution curve that are different from the numerical positions corresponding to the input data, the output data obtained by the decoding unit 121 can be different from the input data of the data processing method, so as to achieve the purpose of data augmentation. And since any position on the predetermined distribution curve can be sampled, and the output data is different according to different sampling positions, the data processing method of the present disclosure can achieve the diversity of the output data.

[0093] The data processing method involved in the present disclosure has been described above in conjunction with the accompanying drawings. Among them, the encoding unit 111, information extraction units 112, 113 included in the first neural network model 110 used in the above method, and the hidden vector mapping units 122, 123, decoding unit 121 included in the second neural network model 120, etc. are all pre-trained. The following refers to Figure 5 to describe the training process 500 of the model applied in the data processing method involved in the present disclosure. For the steps that are the same as those already described above, they will not be specifically described hereinafter.

[0094] As Figure 5 shown, first, in step S510, a first quantity is set, and labels are set for the first quantity of information extraction units 112, 113, and the labels of each information extraction unit are different from each other. Among them, the first quantity may be related to the length of the input data. For example, when the input data is text data, the longer the text length in the training data set, the larger the first quantity can be set.

[0095] Then, in step S520, an Embedding operation is performed on the labels of each information extraction unit 112, 113 to generate label vectors.

[0096] Then, for the label vectors obtained in the above step S520, in step S530, each information extraction unit calculates based on the self-attention mechanism for each label vector to generate a label vector with self-attention weights. The calculation process here is the same as that described above. Among them, in the process of calculating the self-attention weights, W q 、W k 、W v are parameter matrices to be trained. The values of the parameter matrices can be randomly set initially and are continuously updated during the training process. The dimensions of each parameter matrix are related to the dimensions of the label vectors.

[0097] After that, in step S540, the training input data is encoded by the encoding unit 111 in the first neural network model 110 to generate a second hidden vector during training. Among them, the parameters of the encoding unit 111 can be the preset initial parameters and are continuously updated during the training process.

[0098] The steps of generating a label vector with self-attention weights and generating a second hidden vector can be executed simultaneously or in sequence. That is, step S540 can also be executed before steps S520 and S530, or can be executed simultaneously with steps S520 and S530. After that, in step S550, each information extraction unit performs an operation on the label vector with self-attention weights and the second hidden vector based on the cross-attention mechanism to generate a first hidden vector h 112 、h113 Among them, during the process of calculating the self-attention weights, W q ’, W k ’, W v ’ are parameter matrices to be trained. The values of the parameter matrices can be randomly set initially and are continuously updated during the training process. The dimensions of each parameter matrix are related to the dimension of the label vector.

[0099] Next, in step S560, the second neural network model 120 (specifically, the hidden vector mapping units 122, 123) maps each first hidden vector h 112 、h 113 during training to a first number of predetermined distribution curves respectively. Each of the first number of first hidden vectors h 112 、h 113 corresponds to a numerical position on one of the predetermined distribution curves.

[0100] Then, in step S570, sampling is performed on the first number of such numerical positions, that is, without offsetting the numerical positions, so that the hidden vector mapping units 122, 123 generate the first hidden vectors h 112 、h 113 after sampling and restoration. The parameters in the hidden vector mapping units 122, 123 can be preset initial parameters and are continuously updated during the training process.

[0101] Here, the first loss function loss1 is calculated. In an ideal state, after the hidden vector mapping units 122, 123 determine the numerical positions on the Gaussian distribution curve based on the first hidden vectors h 112 、h 113 , if sampling is still performed on this numerical position, the output vector (the first hidden vector after sampling and restoration) obtained at this time should be the same as the original first hidden vector. In fact, there may be information loss when the hidden vector mapping units 122, 123 convert the first hidden vector into the numerical position on the Gaussian distribution curve and when generating the first hidden vector based on the sampled numerical position on the Gaussian distribution curve. The first loss function loss1 represents the difference between the first hidden vector obtained after sampling and restoration by the hidden vector mapping units 122, 123 and the original first hidden vector. The first loss function loss1 can use existing loss functions according to the type of neural network model adopted in the hidden vector mapping unit.

[0102] Next, in step S580, the first hidden vector during training (i.e., the first hidden vector that has not been processed by the hidden vector mapping units 122 and 123) can be directly input to the decoding unit 121. The decoding unit 121 decodes the first hidden vectors during training in a first quantity to generate the output data during training. Among them, the parameters in the decoding unit 121 can be preset initial parameters and are continuously updated during the training process. Here, the second loss function loss2 is calculated. The output data during training can be compared with the input data, and the difference between the output data and the input data during training is represented by the second loss function loss2. The second loss function loss2 can use an existing loss function according to the types of neural network models adopted in the encoding unit and the decoding unit.

[0103] As described above, when training the model used in the data processing method, the overall loss function can be composed of two parts. The goal of training is to minimize these two loss functions, namely the first loss function loss1 and the second loss function loss2, and in step S590, the parameters in the above-mentioned respective units are updated based on backpropagation.

[0104] In some embodiments, the first loss function loss1 and the second loss function loss2 can be optimized separately.

[0105] In other embodiments, these two loss functions can also be combined. For example, the sum of the first loss function loss1 and the second loss function loss2 can be used as the optimization goal of the overall model. Alternatively, the weighted sum of the first loss function loss1 and the second loss function loss2 can be used as the optimization goal of the overall model. In this case, the weights of the first loss function loss1 and the second loss function loss2 can be adjusted based on the convergence speed of each model during training.

[0106] The above steps S530 to S590 can be repeatedly executed until the loss function is reduced to a preset training target threshold or a preset number of iterations has been executed.

[0107] The data processing method related to one aspect of the present disclosure has been described in detail above.

[0108] According to another aspect of the present disclosure, a data processing device is also provided. Figure 6 A schematic diagram of a data processing device 600 according to an embodiment of the present disclosure is shown.

[0109] As Figure 6 shown, the data processing device 600 according to this embodiment includes a processor 610 and a memory 620. The memory 620 stores one or more computer programs.

[0110] The processor 610 is a program control device such as a microprocessor, for example, which operates according to a program installed in the memory 620. The memory 620 is, for example, a storage element such as ROM or RAM. The program to be executed by the processor 610 and the like is stored in the memory 620. Figure 6 The data processing device 600 shown can be used to implement the data processing method of the present disclosure.

[0111] In the data processing device according to an embodiment of the present disclosure, different information in the input data can be expressed based on a first number of first hidden vectors. When a suitable first number is set according to the data length of the data set, the expression ability for the input data can be improved, and the quality of the output data can be improved when the length of the input data is long. In addition, by sampling numerical positions different from the numerical positions corresponding to the input data on a predetermined distribution curve, the output data obtained by the decoding unit can be made different from the input data of the data processing method, so as to achieve the purpose of data augmentation. And since the sampling positions can be different, the output data can also be different, and the diversity of the output data can be achieved.

[0112] According to another aspect of the present disclosure, there is provided a computer program product, which includes a computer program that can implement the data processing method of the present disclosure when executed by a processor.

[0113] According to another aspect of the present disclosure, there is also provided a non-transitory computer-readable storage medium. Computer program instructions are stored on the computer-readable storage medium. When the computer program instructions are executed by a processor, the processor is caused to execute the data processing method disclosed in the present application.

[0114] In the above embodiments, the word vectors of each character of the sentence are used as the input and subsequent processing is performed. However, it is also possible to first convert each character of the sentence into phonemes, and then use the word vectors of the phonemes of each character as the input. A phoneme is the smallest speech unit divided according to the natural attributes of speech, and a sentence can be converted into a phoneme sequence through various methods. For example, in the case of converting a sentence into a phoneme sequence based on Python, for a Chinese sentence, the conversion of the sentence to a pinyin sequence can be achieved through the "pypinyin" library; for an English sentence, the conversion of the sentence to a phonetic symbol sequence can be achieved through the "nltk" library; for a Japanese sentence, the conversion of the sentence to a romanized sequence can be achieved through the "romkan" library. The above methods for converting a sentence into phoneme form are only examples, and those skilled in the art can use any possible way to convert a sentence into phoneme form, and the present disclosure does not make special limitations. In this case, the augmented sentence generated by the data processing method may be different from the input sentence at the phoneme level, and a more similar augmented result to the input sentence can be obtained. This method of converting each character of the sentence into phonemes and then performing processing can be used, for example, in scenarios for improving the performance of a speech recognition system (ASR). Specifically, when an existing speech recognition system receives a speech signal as the input and outputs text as the speech recognition result, there may be a problem of incorrect text recognition, and a post-processing module for the ASR is required to correct the recognized incorrect text. Among them, the post-processing module is obtained through training and requires text pairs such as <correct text, incorrect text> as training data. However, constructing a dataset of text pairs <correct text, incorrect text> requires a large amount of computing power and manpower. Therefore, based on the correct text, the incorrect text corresponding to the correct text can be automatically synthesized through the data processing method of the present disclosure to form a text pair <correct text, incorrect text>, so as to reduce the cost of constructing the training dataset of the post-processing module, thereby improving the ability of the post-processing module to correct incorrect text. The augmented sentence generated by the data processing method is different from the input sentence at the phoneme level, making the augmented sentence more suitable for the speech recognition system.

[0115] In addition, in the above embodiments, text data is used as an example for illustration. However, it should be understood that the data processing method of the present disclosure is not limited to text data and can also be applied to other types of data such as image data. When the input data is image data, applying the data processing method of the present disclosure can generate an output image different from the input image.

[0116] In addition, in the above embodiments, steps S210 to S220 are executed to generate augmented data corresponding to the input data. In other cases, the data processing method according to the present disclosure may omit the above steps S210 to S220, that is, only use a first number of pre-trained hidden vector mapping units and decoding units to sample each predetermined distribution curve, input a numerical group including a first number of numerical positions to each hidden vector mapping unit to obtain a first number of first hidden vectors after sampling and restoration, and decode these first hidden vectors after sampling and restoration through the decoding unit to generate output data. In this case, the output data generated by the data processing method according to the present disclosure does not have corresponding input data, and can be used, for example, in cases where the dataset is small or the data distribution of different classes in the dataset is unbalanced and a certain type of data needs to be augmented.

[0117] The whole or its components of the hardware computing device described in the present disclosure can be implemented by various suitable hardware means, including but not limited to FPGA, ASIC, SoC, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The devices, equipment, methods, and systems involved in the present disclosure are not limited to any specific hardware architecture or configuration. The components in the disclosed devices, equipment, and systems can be separated or integrated, and can be combined in different ways and / or replaced or supplemented by other components. It should be understood that the teachings of the present disclosure can be implemented in various forms of hardware, software, firmware, dedicated processors, or combinations thereof.

[0118] The block diagrams of the devices, equipment, methods, and systems involved in the present disclosure are only exemplary and do not intend to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these circuits, devices, equipment, and systems can be connected, arranged, and configured in any way as long as the desired purpose can be achieved.

[0119] In the above description, the present invention has been described based on embodiments. These embodiments are merely illustrative, and those skilled in the art should understand that the combinations of the constituent elements and processes of these embodiments can be modified in various ways, and such modifications are also within the scope of the present invention.

Claims

1. A data processing method, comprising: generating, by a first neural network model, a first number of first hidden vectors based on input data, where the first number is greater than 1; mapping the first number of first hidden vectors to the first number of predetermined distribution curves respectively, and each of the first number of first hidden vectors corresponds to a numerical position on one of the first number of predetermined distribution curves; sampling each of the first number of predetermined distribution curves to obtain a first numerical group, the first numerical group including the first number of numerical values, and each numerical value corresponds to a numerical position on one of the first number of predetermined distribution curves; and generating output data by a second neural network model based on the first numerical group, wherein each of the first number of first hidden vectors expresses a part of the information of the input data, and the information expressed by each first hidden vector does not completely overlap.

2. The data processing method according to claim 1, wherein The first neural network model includes: an encoding unit configured to encode the input data to generate a second hidden vector; and the first number of information extraction units, and each information extraction unit extracts a part of the information of the second hidden vector to generate each first hidden vector respectively.

3. The data processing method according to claim 2, wherein the information extraction unit uses an attention mechanism to generate the first hidden vector.

4. The data processing method according to claim 1, wherein the first number of first hidden vectors are respectively mapped to the first number of predetermined distribution curves based on a reversible generation model.

5. The data processing method according to claim 4, wherein the reversible generation model is a flow-based model or a diffusion-based model.

6. The data processing method according to claim 1, wherein the first number of first hidden vectors are respectively mapped to the first number of predetermined distribution curves based on a variational autoencoder.

7. The data processing method according to any one of claims 1 to 6, wherein the input data is text data.

8. The data processing method according to any one of claims 1 to 6, wherein the first numerical group is different from the numerical group corresponding to the numerical positions of the input data on the first number of predetermined distribution curves.

9. An apparatus for executing the data processing method, comprising: a processor; and a memory storing one or more computer programs; wherein when the one or more computer programs are run by the processor, the processor is caused to execute the data processing method according to any one of claims 1 to 8.

10. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images