Artificial intelligence-based text data augmentation method and related device
By training text generation and text classification models, similar texts are generated and latent vectors are updated, solving the problem of enhancing data diversity and keeping labels unchanged in existing technologies, thus improving the performance of deep learning models.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- PING AN TECH (SHENZHEN) CO LTD
- Filing Date
- 2023-02-10
- Publication Date
- 2026-06-26
AI Technical Summary
Existing text data augmentation methods struggle to maintain the diversity of augmented data while preserving its text labels, leading to excessive reliance on labels by deep learning models and consuming significant human and time resources.
By training a text generation model and a text classification model, similar text is generated using the text generation model and the latent vectors are updated using the text classification model to ensure that the text labels of the augmented characters remain unchanged. The gradient descent method is used to optimize the model parameters and generate diverse augmented text data.
While ensuring the diversity of enhanced text data, the invariance of text labels was maintained, reducing the dependence on labels and improving the performance of deep learning models.
Smart Images
Figure CN116127041B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device and storage medium for text data enhancement based on artificial intelligence. Background Technology
[0002] Intelligent voice customer service is widely used in various scenarios such as insurance, banking, and e-commerce systems. It first converts user questions into text data, analyzes the text data to understand and identify the user's intent, and finally communicates with the user in a human-like manner, providing information and other related services. The core of intelligent voice customer service is understanding and identifying user intent, and then providing targeted answers after clarifying that intent.
[0003] Currently, deep learning models are commonly used to analyze text data to understand and identify user intent. However, the effectiveness of deep learning models is highly dependent on large amounts of labeled text data, requiring significant manpower and time. Data augmentation can effectively utilize limited labeled text data to improve the performance of deep learning models and reduce reliance on labels.
[0004] Currently, text data augmentation often employs back-translation, which involves translating the text into another language and then back into the original language. Other methods include randomly deleting, inserting, or replacing words with synonyms in the original text. However, these text data augmentation methods struggle to maintain the text labels of the augmented data while ensuring the diversity of the augmented data. Summary of the Invention
[0005] In view of the above, it is necessary to propose an artificial intelligence-based text data augmentation method and related equipment to solve the technical problem of how to keep the text labels of the augmented data unchanged while ensuring the diversity of the augmented data. The related equipment includes an artificial intelligence-based text data augmentation device, electronic equipment and storage medium.
[0006] This application provides a text data augmentation method based on artificial intelligence, the method comprising:
[0007] S10, Train a text generation model. The output of the text generation model is similar text to the input text. The text generation model includes an input layer, an encoding layer, and an output layer.
[0008] S11, the text to be enhanced is used as the input text of the input layer to obtain the latent vector output by the encoding layer. The text to be enhanced includes text labels, and the latent vector is the feature vector of the input text.
[0009] S12, update the latent vector based on the text label and text classification model, and input the updated latent vector into the output layer to obtain the enhanced character;
[0010] S13, If the enhanced character is not a terminating character, use the enhanced character and the text to be enhanced as the input text of the input layer to obtain a new hidden vector, and repeat step S12 to obtain a new enhanced character until the new enhanced character is a terminating character, then execute step S14.
[0011] S14, arrange all the enhanced characters in the order they were acquired to obtain the enhanced text data of the text to be enhanced.
[0012] In some embodiments, the trained text generation model includes:
[0013] For text data with text labels, any two text data with the same text label are considered as a similar pair;
[0014] For text data without text tags, calculate the semantic similarity between any two text data. If the semantic similarity is greater than a preset threshold, then the two text data are considered as a similar pair.
[0015] Store all similar pairs as the generated training set;
[0016] Randomly select a similar pair from the generated training set as a training pair, take any one text data in the training pair as the input text, and take the other text data as the target text;
[0017] The input text is input into the text generation model to obtain the output result. Based on the output result and the target text, the value of the preset loss function is calculated, and the text generation model is updated based on the gradient descent method.
[0018] The text generation model is continuously updated by randomly selecting training pairs from the generated training set until the value of the preset loss function no longer changes, at which point the update stops and the training of the text generation model is completed.
[0019] In some embodiments, calculating the semantic similarity between any two text data includes:
[0020] Extract semantic vectors from any two text data based on the text feature extraction model;
[0021] The semantic similarity between any two text data is calculated based on the semantic vector, and the semantic similarity satisfies the following relation:
[0022] Sim(i,j)=exp(-D(h i ,h j))
[0023] Where Sim(i,j) represents the semantic similarity between text data i and text data j, h i ,h j The semantic vectors corresponding to text data i and text data j are D(h) i ,h j ) indicates the calculation of h i ,h j The distance between them.
[0024] In some embodiments, the method further includes: training a text classification model, specifically including:
[0025] The text classification model includes a feature extraction layer and a classification layer;
[0026] Collect multiple text data sets with the aforementioned text labels as a classification training set;
[0027] Randomly select a text data from the classification training set as the training text;
[0028] The training text is input into the feature extraction layer to output semantic features, and the semantic features are input into the classification layer to output classification results;
[0029] Based on the classification results and the text labels of the training text, a cross-entropy loss function is constructed, and the text classification model is updated based on the gradient descent method.
[0030] The text classification model is continuously updated by randomly selecting training texts from the classification training set until the value of the cross-entropy loss function no longer changes, thus completing the training of the text classification model. The input of the text classification model is text data, and the output is the classification result of the text data.
[0031] In some embodiments, updating the latent vector based on the text label and text classification model includes:
[0032] A1, input the latent vector into the classification layer of the text classification model to obtain the latent vector classification result;
[0033] A2, Construct a cross-entropy loss function based on the latent vector classification results and the text labels;
[0034] A3, update the latent vector based on the cross-entropy loss function to obtain the updated latent vector;
[0035] A4. After repeating steps A1 to A3 a preset number of times, stop updating the hidden vector.
[0036] In some embodiments, updating the latent vector based on the cross-entropy loss function to obtain the updated latent vector includes:
[0037] The backpropagation gradient of each parameter is obtained by taking the partial derivative of the cross-entropy loss function with respect to each parameter in the latent vector.
[0038] The change in each parameter is calculated based on the backpropagation gradient, and the change satisfies the following relationship:
[0039] Δθ=-γG()
[0040] Where G(θ) is the backpropagation gradient of parameter θ, γ is the preset learning rate, and Δθ is the change in parameter θ;
[0041] Add the change amount corresponding to each parameter in the latent vector to obtain the updated latent vector.
[0042] In some embodiments, using the enhanced character and the text to be enhanced as input text for the input layer to obtain a new latent vector includes:
[0043] The enhanced character is added to the end of the text to be enhanced to create a new text to be enhanced.
[0044] Use the new text to be enhanced as the input text for the input layer in the text generation model;
[0045] The encoding layer in the text generation model extracts features from the input text to obtain new latent vectors.
[0046] This application also provides an artificial intelligence-based text data enhancement device, the device comprising:
[0047] An acquisition unit is used to train a text generation model, the output of which is similar text to the input text. The text generation model includes an input layer, an encoding layer, and an output layer.
[0048] An input unit is used to take the text to be enhanced as the input text of the input layer and obtain the latent vector output by the coding layer. The text to be enhanced includes text labels, and the latent vector is the feature vector of the input text.
[0049] The output unit is used to update the latent vector based on the text label and the text classification model, and input the updated latent vector into the output layer to obtain the enhanced character;
[0050] The repeating unit is used to obtain a new latent vector by taking the enhanced character and the text to be enhanced as the input text of the input layer if the enhanced character is not a terminating character, and inputting the new latent vector into the output unit to obtain a new enhanced character, until the new enhanced character is a terminating character.
[0051] The arrangement unit is used to arrange all the enhanced characters in the order they were acquired to obtain the enhanced text data of the text to be enhanced.
[0052] This application embodiment also provides an electronic device, the electronic device comprising:
[0053] Memory, storing at least one instruction;
[0054] The processor executes the instructions stored in the memory to implement the AI-based text data augmentation method.
[0055] This application also provides a computer-readable storage medium storing at least one instruction, which is executed by a processor in an electronic device to implement the artificial intelligence-based text data enhancement method described above.
[0056] In summary, this application uses similar texts obtained from the input text through a text generation model to augment text data, ensuring the diversity of augmented text data. At the same time, during the generation of similar texts, the latent vectors in the text generation model are updated through the backpropagation gradient of the text classification model, thus maintaining the text labels of the augmented data while ensuring the diversity of the augmented data. Attached Figure Description
[0057] Figure 1 This is a flowchart of a preferred embodiment of the AI-based text data augmentation method involved in this application.
[0058] Figure 2 This is a schematic diagram of the process for obtaining enhanced characters involved in this application.
[0059] Figure 3 This is a functional block diagram of a preferred embodiment of the artificial intelligence-based text data enhancement device involved in this application.
[0060] Figure 4 This is a schematic diagram of the structure of an electronic device that is a preferred embodiment of the artificial intelligence-based text data enhancement method involved in this application. Detailed Implementation
[0061] To better understand the purpose, features, and advantages of this application, a detailed description of the application is provided below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of this application can be combined with each other. Numerous specific details are set forth in the following description to provide a thorough understanding of this application; the described embodiments are only a part of the embodiments of this application, and not all of them.
[0062] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.
[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.
[0064] This application provides an artificial intelligence-based text data enhancement method that can be applied to one or more electronic devices. An electronic device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.
[0065] Electronic devices can be any electronic product that allows human-computer interaction with a customer, such as personal computers, tablets, smartphones, personal digital assistants (PDAs), game consoles, interactive network television (IPTV), smart wearable devices, etc.
[0066] Electronic devices may also include network devices and / or client devices. The network devices include, but are not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.
[0067] The networks in which electronic devices are located include, but are not limited to, the Internet, wide area networks, metropolitan area networks, local area networks, and virtual private networks (VPNs).
[0068] like Figure 1 The diagram shown is a flowchart of a preferred embodiment of the text data augmentation method based on artificial intelligence according to this application. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements.
[0069] The AI-based text data augmentation method provided in this application can be applied to any scenario that requires text data augmentation. Therefore, this method can be applied to products in these scenarios, such as intelligent customer service systems in scenarios like electronic trading, securities banking, and insurance.
[0070] S10, Train a text generation model. The output of the text generation model is similar text to the input text. The text generation model includes an input layer, an encoding layer, and an output layer.
[0071] In an optional embodiment, before training the text generation model, the method further includes: building the text generation model, specifically including:
[0072] The text generation model includes: an input layer, an encoding layer, and an output layer;
[0073] The input layer is used to receive input text and send the input text into the encoding layer;
[0074] The encoding layer is used to extract features from the input text to obtain the latent vector of the input text;
[0075] The output layer is used to classify the hidden vectors to obtain output characters;
[0076] The output character and the input text are used as new input text and input into the text generation model to obtain new output characters.
[0077] When the output character is a terminating character, all output characters are arranged in the order they were acquired to obtain the output result of the text generation model. The output result is similar text to the input text.
[0078] The text generation model can be an existing sequence-to-sequence (Seq2seq) model such as Transformer, BERT, GPT, UniLM, or LSTM. The terminating character is a special character indicating the end of text data, located at the end of a text data entry; for example, the terminating character can be [SEP]. Preferably, the text generation model uses the UniLM model.
[0079] In this optional embodiment, the output result is similar text to the input text. Similar text to the input text refers to text data that has the same text tag as the input text or whose semantic similarity is greater than a preset threshold. The preset threshold is 0.95, and having the same text tag indicates that the text belongs to the same category as the input text in the application scenario. For example, in an intelligent customer service system in a financial scenario, the text tag is the intent category corresponding to the input text. Therefore, text data with the same intent category as the input text and text data with a semantic similarity greater than the preset threshold are all considered similar text to the input text.
[0080] In this optional embodiment, after the text generation model is built, in order to ensure that the text generation model can generate accurate output results, it is necessary to train the text generation model. The trained text generation model can obtain accurate output results.
[0081] In an optional embodiment, the trained text generation model includes:
[0082] For text data with text labels, any two text data with the same text label are considered as a similar pair;
[0083] For text data without text tags, calculate the semantic similarity between any two text data. If the semantic similarity is greater than a preset threshold, then the two text data are considered as a similar pair.
[0084] Store all similar pairs as the generated training set;
[0085] Randomly select a similar pair from the generated training set as a training pair, take any one text data in the training pair as the input text, and take the other text data as the target text;
[0086] The input text is input into the text generation model to obtain the output result. Based on the output result and the target text, the value of the preset loss function is calculated, and the text generation model is updated based on the gradient descent method.
[0087] The text generation model is continuously updated by randomly selecting training pairs from the generated training set until the value of the preset loss function no longer changes, at which point the update stops and the training of the text generation model is completed.
[0088] The preset loss function is related to the structure of the text generation model. If the text generation model adopts the UniLM model, then the preset loss function is the damage function of the UniLM model. The text label can be the category information of the text data in the application scenario. The preset threshold value is 0.95.
[0089] In an optional embodiment, calculating the semantic similarity between any two text data includes:
[0090] Extract semantic vectors from any two text data based on the text feature extraction model;
[0091] The semantic similarity between any two text data is calculated based on the semantic vector, and the semantic similarity satisfies the following relation:
[0092] Sim(i,j)=exp(-D(h i ,h j ))
[0093] Where Sim(i,j) represents the semantic similarity between text data i and text data j, h i ,h j The semantic vectors corresponding to text data i and text data j are D(h) i ,h j ) indicates the calculation of h i ,h j The distance between them.
[0094] The distance can be cosine distance, Euclidean distance, Hamming distance, etc., and this application does not impose any restrictions. The text feature extraction model can be the SimCSE model.
[0095] Thus, the training of the text generation model is completed. The text generation model can output similar texts to the input text, providing a model foundation for text data augmentation. At the same time, text data augmentation is achieved through similar text generation methods, thereby improving the diversity of augmented text data.
[0096] S11, the text to be enhanced is used as the input text of the input layer to obtain the latent vector output by the encoding layer. The text to be enhanced includes text labels, and the latent vector is the feature vector of the input text.
[0097] In an optional embodiment, the text to be enhanced is any text data with text labels. The text to be enhanced is used as input text and input into the input layer of the text generation model. The input layer sends the input text to the encoding layer, which extracts features from the input text to obtain the latent vector of the text to be enhanced. The latent vector is the feature vector of the input text, and thus can reflect the semantic features of the text to be enhanced.
[0098] Thus, the latent vector is obtained by using the text to be enhanced as the input text of the input layer, and the latent vector reflects the semantic features of the text to be enhanced.
[0099] S12, update the latent vector based on the text label and text classification model, and input the updated latent vector into the output layer to obtain enhanced characters.
[0100] In an optional embodiment, before updating the latent vector based on the text label and the text classification model, the method further includes: training the text classification model, specifically including:
[0101] A text classification model is constructed, which includes a feature extraction layer and a classification layer;
[0102] Collect multiple text data sets with the aforementioned text labels as a classification training set;
[0103] Randomly select a text data from the classification training set as the training text;
[0104] The training text is input into the feature extraction layer to output semantic features, and the semantic features are input into the classification layer to output classification results;
[0105] Based on the classification results and the text labels of the training text, a cross-entropy loss function is constructed, and the text classification model is updated based on the gradient descent method.
[0106] The text classification model is continuously updated by randomly selecting training texts from the classification training set until the value of the cross-entropy loss function no longer changes, thus completing the training of the text classification model. The input of the text classification model is text data, and the output is the classification result of the text data.
[0107] The text tags can be category information of text data in the application scenario. For example, in an intelligent customer service system, the text tags can be the intent category corresponding to the text data, and the classification result output by the text classification model is the intent classification result; in insurance and banking scenarios, the text tags can be category tags indicating whether it is fraud, and the classification result output by the text classification model is the fraud identification result.
[0108] It should be noted that the semantic features output by the feature extraction layer in the text classification model have the same size as the latent vectors output by the encoding layer in the text generation model.
[0109] In an optional embodiment, updating the latent vector based on the text label and text classification model includes:
[0110] A1, input the latent vector into the classification layer of the text classification model to obtain the latent vector classification result;
[0111] A2, Construct a cross-entropy loss function based on the latent vector classification results and the text labels;
[0112] A3, update the latent vector based on the cross-entropy loss function to obtain the updated latent vector;
[0113] A4. After repeating steps A1 to A3 a preset number of times, stop updating the hidden vector.
[0114] Preferably, the preset number of times is any one of 3, 4, or 5.
[0115] In an optional embodiment, updating the latent vector based on the cross-entropy loss function to obtain the updated latent vector includes:
[0116] The backpropagation gradient of each parameter is obtained by taking the partial derivative of the cross-entropy loss function with respect to each parameter in the latent vector.
[0117] The change in each parameter is calculated based on the backpropagation gradient, and the change satisfies the following relationship:
[0118] Δθ=-γG()
[0119] Where G(θ) is the backpropagation gradient of parameter θ, γ is the preset learning rate, and Δθ is the change in parameter θ;
[0120] Add the change amount corresponding to each parameter in the latent vector to obtain the updated latent vector.
[0121] The parameters include all the values in the latent vector. For example, if the size information of the latent vector is M rows and N columns, then the latent vector contains a total of M×N parameters.
[0122] In this optional embodiment, the updated latent vector is input into the output layer of the text generation model to obtain the output character, and the output character is used as the enhanced character corresponding to the latent vector.
[0123] Please see Figure 2This is a schematic diagram of the process for obtaining enhanced characters provided in an embodiment of this application. Assuming the text to be enhanced includes n characters X1, X2, X3…Xn, the text to be enhanced is input into the input layer of a text generation model. The encoding layer extracts features from the text to be enhanced to obtain latent vectors. These latent vectors are then input into the classification layer of a text classification model to obtain latent vector classification results. Further, a cross-entropy loss function is constructed based on the latent vector classification results and the text labels of the text to be enhanced. The latent vectors are updated using backpropagation gradients, and the updated latent vectors are input into the output layer of the text generation model to obtain the enhanced characters.
[0124] In this way, the latent vectors of the generative model are updated by using the text classification model, so that the updated latent vectors learn the relevant information of the labels. The updated latent vectors are then input into the output layer of the text generation model to obtain enhanced characters related to the label information.
[0125] S13, if the enhanced character is not a terminating character, use the enhanced character and the text to be enhanced as the input text of the input layer to obtain a new hidden vector, and repeat step S12 to obtain a new enhanced character until the new enhanced character is a terminating character, then execute step S14.
[0126] In an optional embodiment, the terminating character is a special character that indicates the end of text data and is located at the end of a text data.
[0127] In this optional embodiment, after obtaining an enhanced character, it is determined whether the enhanced character is a terminating character. If the enhanced character is not a terminating character, it means that there are still unoutputted characters after the enhanced character. In this case, the enhanced character and the text to be enhanced need to be used as the input text of the input layer to obtain a new latent vector.
[0128] In an optional embodiment, the step of using the enhanced character and the text to be enhanced as input text for the input layer to obtain a new latent vector includes:
[0129] The enhanced character is added to the end of the text to be enhanced to create a new text to be enhanced.
[0130] Use the new text to be enhanced as the input text for the input layer in the text generation network;
[0131] The encoding layer in the text generation network extracts features from the input text to obtain new latent vectors.
[0132] In this optional embodiment, after obtaining the new hidden vector, step S12 is repeated to obtain new enhanced characters until the new enhanced character is a terminating character, indicating that all enhanced characters have been obtained, and then step S14 is executed.
[0133] Thus, the enhanced characters are added to the end of the text to be enhanced to form new text to be enhanced. New enhanced characters are continuously obtained based on the new text to be enhanced, and all enhanced characters are obtained.
[0134] S14, arrange all the enhanced characters in the order they were acquired to obtain the enhanced text data of the text to be enhanced.
[0135] In an optional embodiment, all enhancement characters are arranged in the order they are acquired to form the enhancement text data of the text to be enhanced. The enhancement text data includes multiple enhancement characters and ends with a terminating character. All enhancement characters constitute the enhancement text data of the text to be enhanced.
[0136] For example, if the text to be enhanced is "renew insurance", and all the enhanced characters are "I", "want", "renew", "insurance", and "[SEP]" in the order they are obtained, where "[SEP]" is the termination character, then the enhanced text data of the text to be enhanced is "I want to renew insurance", and the text tags corresponding to "renew insurance" and "I want to renew insurance" are the same.
[0137] Thus, the enhanced text data of the text to be enhanced is obtained.
[0138] As can be seen from the above technical solutions, this application obtains similar texts of the input text through a text generation model, and then performs text data augmentation based on the similar texts, ensuring the diversity of the augmented text data. At the same time, during the process of generating similar texts, the latent vectors in the text generation model are updated through the backpropagation gradient of the text classification model, so as to keep the text labels of the augmented data unchanged while ensuring the diversity of the augmented data.
[0139] Please see Figure 3 , Figure 3 This is a functional block diagram of a preferred embodiment of the AI-based text data enhancement device of this application. The AI-based text data enhancement device 11 includes an acquisition unit 110, an input unit 111, an output unit 112, a repetition unit 113, and an arrangement unit 114. The module / unit referred to in this application refers to a series of computer-readable instruction segments that can be executed by the processor 13 and perform a fixed function, and are stored in the memory 12. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.
[0140] In an optional embodiment, the acquisition unit 110 is used to acquire a trained text generation model, the output of which is similar text to the input text, and the text generation model includes an input layer, an encoding layer, and an output layer.
[0141] In an optional embodiment, before training the text generation model, the method further includes: building the text generation model, specifically including:
[0142] The text generation model includes: an input layer, an encoding layer, and an output layer;
[0143] The input layer is used to receive input text and send the input text into the encoding layer;
[0144] The encoding layer is used to extract features from the input text to obtain the latent vector of the input text;
[0145] The output layer is used to classify the hidden vectors to obtain output characters;
[0146] The output character and the input text are used as new input text and input into the text generation model to obtain new output characters.
[0147] When the output character is a terminating character, all output characters are arranged in the order they were acquired to obtain the output result of the text generation model. The output result is similar text to the input text.
[0148] The text generation model can be an existing sequence-to-sequence (Seq2seq) model such as Transformer, BERT, GPT, UniLM, or LSTM. The terminating character is a special character indicating the end of text data, located at the end of a text data entry; for example, the terminating character can be [SEP]. Preferably, the text generation model uses the UniLM model.
[0149] In this optional embodiment, the output result is similar text to the input text. Similar text to the input text refers to text data that has the same text tag as the input text or whose semantic similarity is greater than a preset threshold. The preset threshold is 0.95, and having the same text tag indicates that the text belongs to the same category as the input text in the application scenario. For example, in an intelligent customer service system in a financial scenario, the text tag is the intent category corresponding to the input text. Therefore, text data with the same intent category as the input text and text data with a semantic similarity greater than the preset threshold are all considered similar text to the input text.
[0150] In this optional embodiment, after the text generation model is built, in order to ensure that the text generation model can generate accurate output results, it is necessary to train the text generation model. The trained text generation model can obtain accurate output results.
[0151] In an optional embodiment, the trained text generation model includes:
[0152] For text data with text labels, any two text data with the same text label are considered as a similar pair;
[0153] For text data without text tags, calculate the semantic similarity between any two text data. If the semantic similarity is greater than a preset threshold, then the two text data are considered as a similar pair.
[0154] Store all similar pairs as the generated training set;
[0155] Randomly select a similar pair from the generated training set as a training pair, take any one text data in the training pair as the input text, and take the other text data as the target text;
[0156] The input text is input into the text generation model to obtain the output result. Based on the output result and the target text, the value of the preset loss function is calculated, and the text generation model is updated based on the gradient descent method.
[0157] The text generation model is continuously updated by randomly selecting training pairs from the generated training set until the value of the preset loss function no longer changes, at which point the update stops and the training of the text generation model is completed.
[0158] The preset loss function is related to the structure of the text generation model. If the text generation model adopts the UniLM model, then the preset loss function is the damage function of the UniLM model. The text label can be the category information of the text data in the application scenario. The preset threshold value is 0.95.
[0159] In an optional embodiment, calculating the semantic similarity between any two text data includes:
[0160] Extract semantic vectors from any two text data based on the text feature extraction model;
[0161] The semantic similarity between any two text data is calculated based on the semantic vector, and the semantic similarity satisfies the following relation:
[0162] Sim(i,j)=exp(-D(h i ,h j ))
[0163] Where Sim(i,j) represents the semantic similarity between text data i and text data j, h i ,h j The semantic vectors corresponding to text data i and text data j are D(h)i ,h j ) indicates the calculation of h i ,h j The distance between them.
[0164] The distance can be cosine distance, Euclidean distance, Hamming distance, etc., and this application does not impose any restrictions. The text feature extraction model can be the SimCSE model.
[0165] In an optional embodiment, the input unit 111 is used to take the text to be enhanced as the input text of the input layer to obtain the latent vector output by the encoding layer, wherein the text to be enhanced includes text labels and the latent vector is the feature vector of the input text.
[0166] In an optional embodiment, the text to be enhanced is any text data with text labels. The text to be enhanced is used as input text and input into the input layer of the text generation model. The input layer sends the input text to the encoding layer, which extracts features from the input text to obtain the latent vector of the text to be enhanced. The latent vector is the feature vector of the input text, and thus can reflect the semantic features of the text to be enhanced.
[0167] In an optional embodiment, the output unit 112 is used to update the latent vector based on the text label and the text classification model, and input the updated latent vector into the output layer to obtain enhanced characters.
[0168] In an optional embodiment, before updating the latent vector based on the text label and the text classification model, the method further includes: training the text classification model, specifically including:
[0169] A text classification model is constructed, which includes a feature extraction layer and a classification layer;
[0170] Collect multiple text data sets with the aforementioned text labels as a classification training set;
[0171] Randomly select a text data from the classification training set as the training text;
[0172] The training text is input into the feature extraction layer to output semantic features, and the semantic features are input into the classification layer to output classification results;
[0173] Based on the classification results and the text labels of the training text, a cross-entropy loss function is constructed, and the text classification model is updated based on the gradient descent method.
[0174] The text classification model is continuously updated by randomly selecting training texts from the classification training set until the value of the cross-entropy loss function no longer changes, thus completing the training of the text classification model. The input of the text classification model is text data, and the output is the classification result of the text data.
[0175] The text tags can be category information of text data in the application scenario. For example, in an intelligent customer service system, the text tags can be the intent category corresponding to the text data, and the classification result output by the text classification model is the intent classification result; in insurance and banking scenarios, the text tags can be category tags indicating whether it is fraud, and the classification result output by the text classification model is the fraud identification result.
[0176] It should be noted that the semantic features output by the feature extraction layer in the text classification model have the same size as the latent vectors output by the encoding layer in the text generation model.
[0177] In an optional embodiment, updating the latent vector based on the text label and text classification model includes:
[0178] A1, input the latent vector into the classification layer of the text classification model to obtain the latent vector classification result;
[0179] A2, Construct a cross-entropy loss function based on the latent vector classification results and the text labels;
[0180] A3, update the latent vector based on the cross-entropy loss function to obtain the updated latent vector;
[0181] A4. After repeating steps A1 to A3 a preset number of times, stop updating the hidden vector.
[0182] Preferably, the preset number of times is any one of 3, 4, or 5.
[0183] In an optional embodiment, updating the latent vector based on the cross-entropy loss function to obtain the updated latent vector includes:
[0184] The backpropagation gradient of each parameter is obtained by taking the partial derivative of the cross-entropy loss function with respect to each parameter in the latent vector.
[0185] The change in each parameter is calculated based on the backpropagation gradient, and the change satisfies the following relationship:
[0186] Δθ=-γG()
[0187] Where G(θ) is the backpropagation gradient of parameter θ, γ is the preset learning rate, and Δθ is the change in parameter θ;
[0188] Add the change amount corresponding to each parameter in the latent vector to obtain the updated latent vector.
[0189] The parameters include all the values in the latent vector. For example, if the size information of the latent vector is M rows and N columns, then the latent vector contains a total of M×N parameters.
[0190] In this optional embodiment, the updated latent vector is input into the output layer of the text generation model to obtain the output character, and the output character is used as the enhanced character corresponding to the latent vector.
[0191] In an optional embodiment, the repeating unit 113 is configured to, if the enhanced character is not a terminating character, use the enhanced character and the text to be enhanced as the input text of the input layer to obtain a new latent vector, and input the new latent vector into the output unit to obtain a new enhanced character, until the new enhanced character is a terminating character.
[0192] In an optional embodiment, the terminating character is a special character that indicates the end of text data and is located at the end of a text data.
[0193] In this optional embodiment, after obtaining an enhanced character, it is determined whether the enhanced character is a terminating character. If the enhanced character is not a terminating character, it means that there are still unoutputted characters after the enhanced character. In this case, the enhanced character and the text to be enhanced need to be used as the input text of the input layer to obtain a new latent vector.
[0194] In an optional embodiment, the step of using the enhanced character and the text to be enhanced as input text for the input layer to obtain a new latent vector includes:
[0195] The enhanced character is added to the end of the text to be enhanced to create a new text to be enhanced.
[0196] Use the new text to be enhanced as the input text for the input layer in the text generation network;
[0197] The encoding layer in the text generation network extracts features from the input text to obtain new latent vectors.
[0198] In this optional embodiment, after obtaining the new latent vector, the new latent vector is input into the output unit to obtain a new enhanced character, and the process stops when the new enhanced character is a terminating character.
[0199] In an optional embodiment, the arranging unit 114 is used to arrange all the enhanced characters in the order they were acquired to obtain the enhanced text data of the text to be enhanced.
[0200] The enhanced text data includes multiple enhanced characters, and the end of the enhanced text data is a terminating character.
[0201] For example, if the text to be enhanced is "renew insurance", and all the enhanced characters are "I", "want", "renew", "insurance", and "[SEP]" in the order they are obtained, where "[SEP]" is the termination character, then the enhanced text data of the text to be enhanced is "I want to renew insurance", and the text tags corresponding to "renew insurance" and "I want to renew insurance" are the same.
[0202] As can be seen from the above technical solutions, this application obtains similar texts of the input text through a text generation model, and then performs text data augmentation based on the similar texts, ensuring the diversity of the augmented text data. At the same time, during the process of generating similar texts, the latent vectors in the text generation model are updated through the backpropagation gradient of the text classification model, so as to keep the text labels of the augmented data unchanged while ensuring the diversity of the augmented data.
[0203] Please see Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 1 includes a memory 12 and a processor 13. The memory 12 is used to store computer-readable instructions, and the processor 13 executes the computer-readable instructions stored in the memory to implement the artificial intelligence-based text data enhancement method described in any of the above embodiments.
[0204] In an alternative embodiment, the electronic device 1 further includes a bus and a computer program stored in the memory 12 and executable on the processor 13, such as an AI-based text data enhancement program.
[0205] Figure 4 Only electronic device 1 with memory 12 and processor 13 is shown. It will be understood by those skilled in the art that... Figure 4 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.
[0206] Combination Figure 1 The memory 12 in the electronic device 1 stores a plurality of computer-readable instructions to implement an artificial intelligence-based text data augmentation method, and the processor 13 can execute the plurality of instructions to achieve:
[0207] S10, Obtain the trained text generation model. The output of the text generation model is similar text to the input text. The text generation model includes an input layer, an encoding layer, and an output layer.
[0208] S11, the text to be enhanced is used as the input text of the input layer to obtain the latent vector output by the encoding layer, wherein the text to be enhanced includes text labels;
[0209] S12, update the latent vector based on the text label and the trained text classification model, and input the updated latent vector into the output layer to obtain the enhanced character;
[0210] S13, If the enhanced character is not a terminating character, use the enhanced character and the text to be enhanced as the input text of the input layer to obtain a new hidden vector, and repeat step S12 to obtain a new enhanced character until the new enhanced character is a terminating character, then execute step S14.
[0211] S14, arrange all the enhanced characters in the order they were acquired to obtain the enhanced text data of the text to be enhanced.
[0212] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.
[0213] Those skilled in the art will understand that the schematic diagram is merely an example of electronic device 1 and does not constitute a limitation on electronic device 1. Electronic device 1 can be a bus-type structure or a star-type structure. Electronic device 1 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, electronic device 1 may also include input / output devices, network access devices, etc.
[0214] It should be noted that electronic device 1 is only an example. Other existing or future electronic products that are suitable for this application should also be included within the scope of protection of this application and are incorporated herein by reference.
[0215] The memory 12 includes at least one type of readable storage medium, which can be non-volatile or volatile. The readable storage medium includes flash memory, portable hard drives, multimedia cards, card-type memory (e.g., SD or DX memory), magnetic storage, magnetic disks, optical disks, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 1, such as a portable hard drive of the electronic device 1. In other embodiments, the memory 12 can also be an external storage device of the electronic device 1, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 1. The memory 12 can be used not only to store application software and various types of data installed on the electronic device 1, such as the code of a text data augmentation program based on artificial intelligence, but also to temporarily store data that has been output or will be output.
[0216] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the electronic device 1, connecting various components of the electronic device 1 via various interfaces and lines. It executes programs or modules stored in the memory 12 (e.g., executing AI-based text data enhancement programs) and calls data stored in the memory 12 to perform various functions and process data in the electronic device 1.
[0217] The processor 13 executes the operating system of the electronic device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the various embodiments of the artificial intelligence-based text data augmentation method described above, for example... Figure 1 The steps are shown.
[0218] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete this application. The one or more modules / units may be a series of computer-readable instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device 1. For example, the computer program may be divided into an acquisition unit 110, an input unit 111, an output unit 112, a repetition unit 113, and an arrangement unit 114.
[0219] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute portions of the artificial intelligence-based text data enhancement methods described in the various embodiments of this application.
[0220] If the modules / units integrated in electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware devices. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above.
[0221] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, and other memory.
[0222] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of blockchain nodes, etc.
[0223] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.
[0224] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, in... Figure 4 The symbol is represented by only one arrow, but this does not indicate that there is only one bus or one type of bus. The bus is configured to enable communication between the memory 12 and at least one processor 13, etc.
[0225] This application also provides a computer-readable storage medium (not shown) storing computer-readable instructions, which are executed by a processor in an electronic device to implement the artificial intelligence-based text data enhancement method described in any of the above embodiments.
[0226] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.
[0227] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0228] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.
[0229] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in the specification may also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.
[0230] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.
Claims
1. A text data augmentation method based on artificial intelligence, characterized in that, The method includes: S10, Train a text generation model. The output of the text generation model is similar text to the input text. The text generation model includes an input layer, an encoding layer, and an output layer. S11, the text to be enhanced is used as the input text of the input layer to obtain the latent vector output by the encoding layer. The text to be enhanced includes text labels, and the latent vector is the feature vector of the input text. S12, update the latent vector based on the text label and text classification model, and input the updated latent vector into the output layer to obtain enhanced characters. The step of updating the latent vector based on the text label and text classification model includes: A1, inputting the latent vector into the classification layer of the text classification model to obtain the latent vector classification result; A2, constructing a cross-entropy loss function based on the latent vector classification result and the text label; A3, updating the latent vector based on the cross-entropy loss function to obtain the updated latent vector; A4, repeating steps A1 to A3 a preset number of times, and then stopping the updating of the latent vector. S13, If the enhanced character is not a terminating character, use the enhanced character and the text to be enhanced as the input text of the input layer to obtain a new hidden vector, and repeat step S12 to obtain a new enhanced character until the new enhanced character is a terminating character, then execute step S14. S14, arrange all the enhanced characters in the order they were acquired to obtain the enhanced text data of the text to be enhanced.
2. The text data augmentation method based on artificial intelligence as described in claim 1, characterized in that, The trained text generation model includes: For text data with text labels, any two text data with the same text label are considered as a similar pair; For text data without text tags, calculate the semantic similarity between any two text data. If the semantic similarity is greater than a preset threshold, then the two text data are considered as a similar pair. Store all similar pairs as the generated training set; Randomly select a similar pair from the generated training set as a training pair, take any one text data in the training pair as the input text, and take the other text data as the target text; The input text is input into the text generation model to obtain the output result. Based on the output result and the target text, the value of the preset loss function is calculated, and the text generation model is updated based on the gradient descent method. The text generation model is continuously updated by randomly selecting training pairs from the generated training set until the value of the preset loss function no longer changes, at which point the update stops and the training of the text generation model is completed.
3. The text data augmentation method based on artificial intelligence as described in claim 2, characterized in that, The calculation of the semantic similarity between any two text data includes: Extract semantic vectors from any two text data based on the text feature extraction model; The semantic similarity between any two text data is calculated based on the semantic vector, and the semantic similarity satisfies the following relation: in, This represents the semantic similarity between text data i and text data j. These are the semantic vectors corresponding to text data i and text data j, respectively. Indicates calculation The distance between them.
4. The text data augmentation method based on artificial intelligence as described in claim 1, characterized in that, The method further includes: training a text classification model, specifically including: The text classification model includes a feature extraction layer and a classification layer; Collect multiple text data sets with the aforementioned text labels as a classification training set; Randomly select a text data from the classification training set as the training text; The training text is input into the feature extraction layer to output semantic features, and the semantic features are input into the classification layer to output classification results; Based on the classification results and the text labels of the training text, a cross-entropy loss function is constructed, and the text classification model is updated based on the gradient descent method. The text classification model is continuously updated by randomly selecting training texts from the classification training set until the value of the cross-entropy loss function no longer changes, thus completing the training of the text classification model. The input of the text classification model is text data, and the output is the classification result of the text data.
5. The text data augmentation method based on artificial intelligence as described in claim 1, characterized in that, The step of updating the latent vector based on the cross-entropy loss function to obtain the updated latent vector includes: The backpropagation gradient of each parameter is obtained by taking the partial derivative of the cross-entropy loss function with respect to each parameter in the latent vector. The change in each parameter is calculated based on the backpropagation gradient, and the change satisfies the following relationship: in, For parameters The backpropagation gradient, To preset the learning rate, For parameters The change in; Add the change amount corresponding to each parameter in the latent vector to obtain the updated latent vector.
6. The text data augmentation method based on artificial intelligence as described in claim 1, characterized in that, The step of using the enhanced character and the text to be enhanced as input text for the input layer to obtain a new latent vector includes: The enhanced character is added to the end of the text to be enhanced to create a new text to be enhanced. Use the new text to be enhanced as the input text for the input layer in the text generation model; The encoding layer in the text generation model extracts features from the input text to obtain new latent vectors.
7. A text data enhancement device based on artificial intelligence, characterized in that, The device includes: An acquisition unit is used to train a text generation model, the output of which is similar text to the input text. The text generation model includes an input layer, an encoding layer, and an output layer. An input unit is used to take the text to be enhanced as the input text of the input layer and obtain the latent vector output by the coding layer. The text to be enhanced includes text labels, and the latent vector is the feature vector of the input text. The output unit is used to update the latent vector based on the text label and the text classification model, and input the updated latent vector into the output layer to obtain enhanced characters. The step of updating the latent vector based on the text label and the text classification model includes: A1, inputting the latent vector into the classification layer of the text classification model to obtain the latent vector classification result; A2, constructing a cross-entropy loss function based on the latent vector classification result and the text label; A3, updating the latent vector based on the cross-entropy loss function to obtain the updated latent vector; A4, repeating steps A1 to A3 a preset number of times, and then stopping the updating of the latent vector. The repeating unit is used to obtain a new latent vector by taking the enhanced character and the text to be enhanced as the input text of the input layer if the enhanced character is not a terminating character, and inputting the new latent vector into the output unit to obtain a new enhanced character, until the new enhanced character is a terminating character. The arrangement unit is used to arrange all the enhanced characters in the order they were acquired to obtain the enhanced text data of the text to be enhanced.
8. An electronic device, characterized in that, The electronic device includes: Memory, which stores computer-readable instructions; and The processor executes computer-readable instructions stored in the memory to implement the AI-based text data enhancement method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the artificial intelligence-based text data enhancement method as described in any one of claims 1 to 6.