Text processing method and device, electronic equipment and storage medium

Abstract

The application discloses a text processing method and device, electronic equipment and a storage medium, wherein the text processing method comprises: inputting a first text vector of a first text into a set encoder to obtain a first feature vector output by the set encoder; determining a first key vector corresponding to each transformation matrix in at least one transformation matrix for the first feature vector; inputting the first feature vector and the corresponding at least one first key vector into a set decoder to obtain a second text vector output by the set decoder; the set decoder is used for calculating the input first feature vector and the corresponding at least one first key vector to determine the second text vector; based on the second text vector, a second text corresponding to a first label is generated; the first label represents the label of the first text.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a text processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, in order to train Natural Language Processing (NLP) models, especially text classification models, a large amount of manually labeled text is required as training samples, which results in high costs for obtaining model training samples. Summary of the Invention

[0003] In view of this, embodiments of this application provide a text processing method, apparatus, electronic device, and storage medium to at least solve the problem of high cost of obtaining model training samples in related technologies.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a text processing method, the method comprising:

[0006] Input the first text vector of the first text into the set encoder to obtain the first feature vector output by the set encoder;

[0007] Determine that the first feature vector corresponds to the first key vector of each transformation matrix in at least one transformation matrix;

[0008] The first feature vector and at least one corresponding first key vector are input into a set decoder to obtain a second text vector output by the set decoder; the set decoder is used to calculate the second text vector by inputting the first feature vector and at least one corresponding first key vector.

[0009] Based on the second text vector, a second text with a corresponding first annotation is generated; the first annotation represents the annotation of the first text.

[0010] In the above scheme, after inputting the first feature vector and at least one corresponding first key vector into the setting decoder, and before the setting decoder outputs the second text vector, the method includes:

[0011] In the defined decoder, the similarity between the i-th second feature vector and each of the at least one first key vectors is calculated to obtain the weight corresponding to each first key vector;

[0012] Based on each first key vector and its corresponding weight in the at least one first key vector, the (i+1)th second feature vector is obtained; where,

[0013] i is a positive integer; when i = 1, the i-th second feature vector is the first feature vector; the (i+1)-th second feature vector is used to determine the i-th character in the second text.

[0014] In the above scheme, determining that the first feature vector corresponds to the first key vector of each transformation matrix in at least one transformation matrix includes:

[0015] The first feature vector is processed through a first hidden layer; the first hidden layer represents a set hidden layer that discards at least one neuron;

[0016] Based on each of the at least one transformation matrix, determine the first key vector corresponding to the processed first feature vector.

[0017] In the above scheme, before inputting the first feature vector and at least one corresponding first key vector into the decoder, the method further includes:

[0018] The decoder is trained and configured based on the third feature vector and at least one corresponding second key vector; wherein...

[0019] The third feature vector represents the feature vector output by the encoder based on the third text vector corresponding to the training sample; the at least one second key vector represents the key vector of each transformation matrix in the at least one transformation matrix corresponding to the third feature vector.

[0020] In the above scheme, before training and setting the decoder based on the third feature vector and at least one corresponding second key vector, the method further includes:

[0021] Replace at least one element in the third text vector with a set value.

[0022] In the above scheme, the encoder is set to the BERT model; and / or the decoder is set to the GPT model.

[0023] The method in the above scheme further includes:

[0024] A natural language processing (NLP) model is trained based on the second text and the first annotation.

[0025] This application also provides a text processing apparatus, including:

[0026] The first input unit is used to input the first text vector of the first text into the set encoder to obtain the first feature vector output by the set encoder;

[0027] A first processing unit is configured to determine that the first feature vector corresponds to a first key vector of each transformation matrix in at least one transformation matrix;

[0028] The second input unit is used to input the first feature vector and at least one corresponding first key vector into the setting decoder to obtain the second text vector output by the setting decoder; the setting decoder is used to calculate the input first feature vector and at least one corresponding first key vector to determine the second text vector.

[0029] The generation unit is used to generate a second text with a first annotation based on the second text vector; the first annotation represents the annotation of the first text.

[0030] This application also provides an electronic device, including: a processor and a memory for storing a computer program capable of running on the processor.

[0031] The processor is used to execute the steps of the above-described text processing method when running the computer program.

[0032] This application also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described text processing method.

[0033] This application provides a text processing method, apparatus, electronic device, and storage medium. A first text vector of a first text is input into a designated encoder to obtain a first feature vector output by the designated encoder. The first feature vector is then determined to correspond to a first key vector in each of at least one transformation matrix. The first feature vector and the corresponding first key vector are input into a designated decoder to obtain a second text vector output by the designated decoder. Based on the second text vector, a second text with corresponding first annotations is generated. The designated decoder is used to calculate the second text vector by processing the input first feature vector and the corresponding first key vector. The first annotation represents the annotation of the first text. Encoding the first text vector of the first text to obtain the first feature vector, inputting the first feature vector and the corresponding first key vector into the designated decoder to decode and obtain the second text vector, and generating second text with the same annotations as the first text based on the second text vector—this method uses the second text and the first annotations to train the model, eliminating the need for extensive manual annotation to obtain training samples, thus reducing the cost of acquiring model training samples. Attached Figure Description

[0034] Figure 1 This is a schematic diagram illustrating the implementation flow of the text processing method provided in the embodiments of this application;

[0035] Figure 2A schematic diagram illustrating the text processing method provided in an embodiment of this application;

[0036] Figure 3 A schematic diagram illustrating the implementation flow of the text processing method provided in the application embodiments of this application;

[0037] Figure 4 This is a schematic diagram of the structure of a text processing device provided in an embodiment of this application;

[0038] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0039] Currently, to train NLP models, taking text classification as an example, the common approach is to fine-tune the model based on a pre-trained model. Each time a new task scenario is handled, data annotation is required for that specific scenario. This is especially true when there are many categories and the task is complex, necessitating a large amount of manually annotated text as training samples, leading to high costs in obtaining training samples. Furthermore, due to differences in individual understanding, the correspondence between training samples and annotations is not accurate, resulting in models trained using manually annotated results that often perform poorly.

[0040] Based on this, in various embodiments of this application, a first text vector of the first text is input into a set encoder to obtain a first feature vector output by the set encoder, and the first feature vector is determined to correspond to a first key vector of each transformation matrix in at least one transformation matrix; the first feature vector and the corresponding at least one first key vector are input into a set decoder to obtain a second text vector output by the set decoder, and a second text with a corresponding first annotation is generated based on the second text vector; wherein, the set decoder is used to calculate the input first feature vector and the corresponding at least one first key vector to determine the second text vector, and the first annotation represents the annotation of the first text. Encoding the first text vector of the first text to obtain the first feature vector, inputting the first feature vector and the corresponding at least one first key vector into the set decoder to decode to obtain the second text vector, and generating a second text with the same annotation as the first text based on the second text vector, in this way, training the model using the second text and the first annotation does not require obtaining training samples through a large amount of manual annotation, thus reducing the cost of obtaining model training samples.

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0042] Figure 1This is a schematic diagram illustrating the implementation flow of the text processing method provided in this application embodiment. This application embodiment provides a text processing method applied to an electronic device, wherein the electronic device includes, but is not limited to, servers, terminals, and other electronic devices. It includes:

[0043] Step 101: Input the first text vector of the first text into the set encoder to obtain the first feature vector output by the set encoder.

[0044] The first text vector X = [x1, x2, ..., x...] n The first text is obtained by vectorizing it, and the elements x1, x2, ..., x3 in the first text vector are... n The first text vector represents the characters in the first text, where n is the length of the first text. The first text vector is input into the set encoder to obtain the encoded result output by the set encoder, which is the first feature vector.

[0045] Here, the first text represents the annotated text, and the annotation of the first text is the first annotation. Furthermore, the first text can consist of characters from one or more languages, such as Chinese and English.

[0046] Step 102: Determine that the first feature vector corresponds to the first key vector of each transformation matrix in at least one transformation matrix.

[0047] Based on each transformation matrix, a first key vector corresponding to the first feature vector is obtained. Thus, the first feature vector yields at least one corresponding first key vector. This can be achieved by inputting the first feature vector into the second hidden layer to transform the corresponding transformation matrix, thereby obtaining the corresponding first key vector. The transformation matrix can be a learnable parameter obtained through training the second hidden layer.

[0048] Here, the number of transformation matrices is the same as the number of corresponding first key vectors, and the number can be set based on the number of the third hidden layer of the decoder.

[0049] Taking the first feature vector with a dimension of 1×n as an example, using equation (1), based on each transformation matrix w in the m transformation matrices n×n, a corresponding first key vector 1×n is obtained:

[0050] k i =Z·w i (1)

[0051] Where, k i Z represents the i-th first key vector; Z represents the first feature vector; w i Represents the i-th transformation matrix.

[0052] Here, m is the number of the third hidden layer of the encoder, and n is the dimension of the encoder output vector.

[0053] In this way, we obtain m first key vectors, where m and n are both positive integers.

[0054] Step 103: Input the first feature vector and at least one corresponding first key vector into the set decoder to obtain the second text vector output by the set decoder.

[0055] The decoder is used to calculate the second text vector by analyzing the input first feature vector and at least one corresponding first key vector.

[0056] The first feature vector and at least one corresponding first key vector are input into the setting decoder. The setting decoder calculates the input first feature vector and at least one corresponding first key vector to determine the second text vector, and outputs the determined second text vector.

[0057] Here, the set encoder and set decoder can be an autoencoder (AE) or a variational autoencoder (VAE), meaning that the functions of the set encoder and set decoder in this embodiment can be implemented using an autoencoder's encoder and decoder. The set encoder and set decoder can be transformer-based models. Preferably, the set decoder is an attention mechanism decoder.

[0058] In one embodiment, the encoder is set to a BERT model; and / or, the decoder is set to a GPT model.

[0059] Here, the encoder is set to the BERT model, or the decoder is set to the GPT model, or the encoder is set to the BERT model and the decoder is set to the GPT model.

[0060] Step 104: Based on the second text vector, generate the second text corresponding to the first annotation.

[0061] Wherein, the first annotation represents the annotation of the first text.

[0062] Based on the output second text vector, a corresponding second text is generated, which corresponds to the first annotation of the first text. The way the second text corresponds to the first annotation can be that the second text carries the first annotation, or it can be any other way that can represent the correspondence between the second text and the first annotation, which is not limited here.

[0063] Here, the beam-search algorithm can be used during text generation. Compared to the greedy algorithm, which is prone to getting stuck in local optima, the text generated by the beam-search algorithm is more semantically accurate and fluent, thus improving the quality of the generated second text.

[0064] In this embodiment, the first text vector of the first text is encoded to obtain a first feature vector. The first feature vector and at least one corresponding first key vector are input into a decoder to obtain a second text vector. Based on the second text vector, a second text with the same annotation as the first text is generated. In this way, the model is trained using the second text and the first annotation, without the need to obtain training samples through a large number of manual annotations, thus reducing the cost of obtaining model training samples.

[0065] Furthermore, by processing the first feature vector into at least one first key vector, these first key vectors are used when generating each character in the second text. This means that each generated character is directly related to the first feature vector. Thus, even with increased text length, the association between the second and first texts is maintained. In other words, this overcomes the problem of reduced dependency of encoded information due to increased text length, improving the correlation between the generated second and first texts. The model trained using the second text and the first annotation achieves higher recognition accuracy and better model performance.

[0066] In one embodiment, the method further includes:

[0067] A natural language processing (NLP) model is trained based on the second text and the first annotation.

[0068] After generating the second text, the NLP model is trained based on the second text and the corresponding annotations of the first text (i.e., the second text and the first annotation). Here, NLP models, especially text classification models, require a large amount of manually annotated text as training samples when there are many categories and the task is complex, leading to high costs in obtaining model training samples. This approach eliminates the need for extensive manual annotation to obtain training samples, reducing the cost of acquiring model training samples.

[0069] In one embodiment, after inputting the first feature vector and at least one corresponding first key vector into a setting decoder, and before the setting decoder outputs a second text vector, the method includes:

[0070] In the defined decoder, the similarity between the i-th second feature vector and each of the at least one first key vectors is calculated to obtain the weight corresponding to each first key vector;

[0071] Based on each first key vector and its corresponding weight in the at least one first key vector, the (i+1)th second feature vector is obtained; where,

[0072] i is a positive integer; when i = 1, the i-th second feature vector is the first feature vector; the (i+1)-th second feature vector is used to determine the i-th character in the second text.

[0073] After inputting the first feature vector and at least one corresponding first key vector into the set decoder, the first feature vector is used as the first second feature vector. Similarity calculation is performed between the first second feature vector and each of the at least one first key vectors to obtain the weight corresponding to each first key vector. Based on each of the at least one first key vectors and its corresponding weight, a second second feature vector is calculated. The first character of the text to be output is determined based on the second second feature vector, and similarity calculation is performed between the second second feature vector and each of the at least one first key vectors. This loop is repeated until the loop termination condition of the set decoder is met, resulting in all characters of the second text.

[0074] The loop termination condition can be set according to the actual scenario and the characteristics of the decoder.

[0075] Here, the similarity calculation between the i-th second feature vector and each of the first key vectors in at least one first key vector can be achieved by setting at least one third hidden layer in the decoder. Specifically, as follows: Figure 2 The illustrated text processing method involves inputting a first key vector into the corresponding third hidden layer. The third hidden layer processes the current input feature vector based on the corresponding first key vector and outputs the processed feature vector.

[0076] Combination Figure 2 When generating each character in the second text, a first key vector is obtained based on the second feature vector corresponding to the previous character and at least one first key vector corresponding to the first feature vector. In other words, the second feature vector of each predicted character is used as input when generating the next character; that is, input Z predicts character g1, input g1 generates predicted character g2, and so on. In this way, the current character is associated with previous characters during the generation of the second text, improving the quality of the generated second text.

[0077] In one embodiment, determining that the first feature vector corresponds to the first key vector of each transformation matrix in at least one transformation matrix includes:

[0078] The first feature vector is processed through a first hidden layer; the first hidden layer represents a set hidden layer that discards at least one neuron;

[0079] Based on each of the at least one transformation matrix, determine the first key vector corresponding to the processed first feature vector.

[0080] Randomly discard some neurons in the first hidden layer, input the first feature vector into the first hidden layer, and the first hidden layer discards information in the first feature vector with a set probability. Based on the processed first feature vector, the corresponding first key vector is obtained.

[0081] Since the neurons discarded by the first hidden layer are not exactly the same each time, even if the same first feature vector is input, the first feature vector obtained after each processing will be different. By inputting the first feature vector into the first hidden layer multiple times, multiple different processed first feature vectors are obtained. In this way, the multiple second texts obtained based on the feature vectors are similar but not exactly the same. Therefore, by setting the first hidden layer, a large number of different second texts can be obtained, reducing the cost of obtaining model training samples.

[0082] In one embodiment, before inputting the first feature vector and at least one corresponding first key vector into the set decoder, the method further includes:

[0083] The decoder is trained and configured based on the third feature vector and at least one corresponding second key vector; wherein...

[0084] The third feature vector represents the feature vector output by the encoder based on the third text vector corresponding to the training sample; the at least one second key vector represents the key vector of each transformation matrix in the at least one transformation matrix corresponding to the third feature vector.

[0085] The third text vector of the training sample is input into the set encoder to obtain the third feature vector output by the set encoder. Based on each of the at least one transformation matrix, a second key vector corresponding to each transformation matrix and the third feature vector is obtained. The set decoder is trained based on the third feature vector and the corresponding at least one second key vector to obtain the trained set decoder.

[0086] In one embodiment, before training and setting the decoder based on the third feature vector and at least one corresponding second key vector, the method further includes:

[0087] Replace at least one element in the third text vector with a set value.

[0088] When training the predefined decoder, at least one random element in the third text vector of the training samples is replaced with a predefined value to obtain a processed third text vector. This processed third text vector is then input into the predefined encoder to obtain the third feature vector output by the encoder. By randomly processing the third text vector and using the corresponding third feature vector, the predefined decoder is trained. This allows the decoder to learn inference about the input feature vectors through training. This avoids outputting second text identical to the first text, ensuring the diversity of generated text and reducing the cost of obtaining training samples for the model.

[0089] In one embodiment, the method further includes:

[0090] The third feature vector is processed through a fourth hidden layer; the fourth hidden layer represents a set hidden layer that discards at least one neuron;

[0091] Based on each of the at least one transformation matrix, determine the second key vector corresponding to the processed third feature vector.

[0092] Randomly discard some neurons in the fourth hidden layer, input the third feature vector into the fourth hidden layer, and discard the information in the third feature vector at a set probability in the fourth hidden layer. Based on the processed third feature vector, obtain the corresponding second key vector.

[0093] Since the neurons discarded by the fourth hidden layer are not exactly the same each time, even if the same third feature vector is input, the third feature vector obtained after each processing will be different. Inputting the third feature vector into the fourth hidden layer multiple times will result in multiple different third feature vectors after processing.

[0094] The present application will be further described in detail below with reference to application examples.

[0095] Figure 3 This diagram illustrates a text processing method provided in one application embodiment of this application. The autoencoder's encoder and decoder are BERT and GPT, respectively, both models based on a self-attention mechanism. BERT excels at encoding text vectors to obtain corresponding feature vectors, while GPT excels at generation tasks. The input to the autoencoder is a first text vector of a first text. First, the first text vector is encoded by the encoder to obtain a first feature vector. Then, the first feature vector of the encoded result is fed to the decoder, allowing the decoder to process the first feature vector to obtain a second text vector, thereby generating a second text similar to the first text.

[0096] Taking the prediction process (i.e., the usage process) as an example, the differences between the prediction process and the training process will be explained in the corresponding section:

[0097] The input text vector of the autoencoder is X = [x1, x2, ..., x...]. n The text vector is obtained by vectorizing the original text. The elements x1, x2, ..., x3 in the text vector are... n The characters in the text are represented by n, where n is the length of the original text. Here, the original text is typically a sentence. And according to BERT's rules, two tokens, [CLS] and [SEP], are added to the beginning and end of each sentence, respectively. During training, different original texts than those used in the prediction process can be used as training samples, such as unlabeled text.

[0098] After obtaining the text vector, the training process involves step 1, where characters in the text are masked with a certain probability p, i.e., the characters in the original text are replaced with a set value [MASK], resulting in a new text vector X' = [x1, [MASK], ..., x n For the prediction process, step 1 is skipped, i.e., X' = X, to avoid losing key information in the text due to masking.

[0099] In step 2, the text vector X' is input into the encoder BERT. BERT encodes the input text vector and outputs a feature vector. Here, the dimension of the output feature vector is smaller than that of the text vector. The text vector is a discrete and sparse high-dimensional vector, while the feature vector is a continuous vector in a low-dimensional space. In this way, the semantic information represented by the feature vector can better represent the semantic information of the text and is easier for the model to understand.

[0100] In step 3, the vector corresponding to the BERT starting position [CLS] in the output feature vector is taken as the vector representation of the entire text, i.e. Figure 3 Z in the middle.

[0101] In step 4, the BERT output [CLS] (i.e., vector Z) first passes through a hidden layer. By randomly discarding a portion of neurons, some information in the feature vector is discarded with a certain probability. This improves the diversity of the generated text. Specifically, during both the training and prediction processes, information from the feature vector can be discarded with a certain probability through the hidden layer.

[0102] The processed feature vector [CLS] is passed through m single-layer hidden layers, with each hidden layer yielding a key vector, thus obtaining k1, k2, ..., k. m This performs a linear transformation on the feature vectors output by BERT. This will yield k1, k2, ..., k m Input the corresponding hidden layer of GPT, where m is the number of the third hidden layer of the GPT model.

[0103] Taking the BERT hidden layer vector dimension and the GPT hidden layer vector dimension as examples, both with a dimension of 768, the feature vector Z has a dimension of 1×768. To obtain each k... i With a dimension of 1×768, it is transformed by a learnable transformation matrix w. i The dimension is 768×768, and a corresponding key vector 1×n is obtained through equation (1).

[0104] To enable GPT to understand BERT encoding information, in the application embodiments of this application, in addition to [CLS] as input, k1,k2,…,k are obtained by linear transformation of the BERT output encoding. m As a key vector, input it into the model.

[0105] For the GPT model with its attention mechanism, the output of each position in the GPT hidden layer is the query. Attention is calculated between the key and value, and the model then focuses on the regions with higher attention values. Here, the value can be set equal to the key value. In each subsequent prediction, due to GPT's attention mechanism, the currently generated query interacts with the previously obtained key and value, thus effectively utilizing the information from the encoder's feature vectors.

[0106] exist Figure 3 In the middle, g1,…,g n To predict a given word, the feature vector of each predicted character is used as input to generate the next character. That is, input Z predicts the feature vector of character g1, input the feature vector of g1 to generate the predicted character g2, and so on. Here, a character can be an English word, a Chinese character, etc.

[0107] In this application embodiment, an autoencoder generative model is used to determine similar text to the labeled text. Data augmentation and amplification are achieved with a small amount of labeled text, eliminating the need for extensive manual annotation to obtain training samples. This increases the quantity of training data, expands the training sample set, and reduces the cost of acquiring model training samples. Furthermore, the model is improved by introducing at least one key vector related to the feature vector, enhancing the model's semantic understanding ability and thus improving the quality of the generated text. Simultaneously, beam-search is used for text generation to prevent simple greedy algorithms from getting stuck in local optima, which could lead to incoherent sentences.

[0108] Furthermore, during training and prediction, different feature vectors are obtained based on the original text by randomly discarding neurons in the hidden layer between the encoder and decoder. The decoder then generates different texts based on these different input feature vectors, thereby increasing the diversity of generated texts. Moreover, during encoding, some characters in the text are masked, and elements in the text vector are randomly replaced with predetermined values. This allows the decoder to learn to reason about the input data, preventing the output text from being identical to the input text. In this way, the problem that the output of an autoencoder is usually a repetition of the input information is overcome, further improving the diversity of generated texts.

[0109] To implement the method of the embodiments of this application, the embodiments of this application also provide a text processing device, such as... Figure 4 As shown, the device includes:

[0110] The first input unit 401 is used to input the first text vector of the first text into the set encoder to obtain the first feature vector output by the set encoder.

[0111] The first processing unit 402 is configured to determine that the first feature vector corresponds to the first key vector of each transformation matrix in at least one transformation matrix;

[0112] The second input unit 403 is used to input the first feature vector and at least one corresponding first key vector into the setting decoder to obtain the second text vector output by the setting decoder; the setting decoder is used to calculate the input first feature vector and at least one corresponding first key vector to determine the second text vector.

[0113] The generation unit 404 is used to generate a second text with a first annotation based on the second text vector; the first annotation represents the annotation of the first text.

[0114] In one embodiment, the apparatus further includes a second processing unit for:

[0115] In the defined decoder, the similarity between the i-th second feature vector and each of the at least one first key vectors is calculated to obtain the weight corresponding to each first key vector;

[0116] Based on each first key vector and its corresponding weight in the at least one first key vector, the (i+1)th second feature vector is obtained; where,

[0117] i is a positive integer; when i = 1, the i-th second feature vector is the first feature vector; the (i+1)-th second feature vector is used to determine the i-th character in the second text.

[0118] In one embodiment, the first processing unit 402 is configured to:

[0119] Determining that the first feature vector corresponds to the first key vector of each transformation matrix in at least one transformation matrix includes:

[0120] The first feature vector is processed through a first hidden layer; the first hidden layer represents a set hidden layer that discards at least one neuron;

[0121] Based on each of the at least one transformation matrix, determine the first key vector corresponding to the processed first feature vector.

[0122] In one embodiment, the apparatus further includes a first training unit for:

[0123] The decoder is trained and configured based on the third feature vector and at least one corresponding second key vector; wherein...

[0124] The third feature vector represents the feature vector output by the encoder based on the third text vector corresponding to the training sample; the at least one second key vector represents the key vector of each transformation matrix in the at least one transformation matrix corresponding to the third feature vector.

[0125] In one embodiment, the apparatus further includes a third processing unit for:

[0126] Replace at least one element in the third text vector with a set value.

[0127] In one embodiment, the encoder is set to a BERT model; and / or the decoder is set to a GPT model.

[0128] In one embodiment, the apparatus further includes a second training unit for:

[0129] A natural language processing (NLP) model is trained based on the second text and the first annotation.

[0130] In practical applications, the first input unit 401, the first processing unit 402, the second input unit 403, the generation unit 404, the second processing unit, the first training unit, the third processing unit, and the second training unit can be implemented by processors in text processing devices, such as central processing units (CPUs), digital signal processors (DSPs), microcontroller units (MCUs), or field-programmable gate arrays (FPGAs).

[0131] It should be noted that the text processing device provided in the above embodiments is only illustrated by the division of the above program modules. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the text processing device and the text processing method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0132] Based on the hardware implementation of the above program modules, and in order to implement the text processing method of the embodiments of this application, the embodiments of this application also provide an electronic device. Figure 5 This is a schematic diagram of the hardware structure of the electronic device according to an embodiment of this application, as shown below. Figure 5 As shown, the electronic device includes:

[0133] Communication interface 1 enables information exchange with other devices, such as network devices;

[0134] Processor 2 is connected to communication interface 1 to enable information interaction with other devices and to execute the methods provided by one or more of the above-mentioned technical solutions when running computer programs. The computer program is stored in memory 3.

[0135] Of course, in practical applications, the various components in an electronic device are coupled together through bus system 4. It can be understood that bus system 4 is used to achieve communication and connection between these components. In addition to the data bus, bus system 4 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 5 The general will label all buses as Bus System 4.

[0136] The memory 3 in this embodiment is used to store various types of data to support the operation of the electronic device. Examples of such data include any computer program used to operate on the electronic device.

[0137] It is understood that memory 3 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 2 described in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0138] The methods disclosed in the embodiments of this application can be applied to processor 2, or implemented by processor 2. Processor 2 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 2 or by instructions in the form of software. The processor 2 may be a general-purpose processor, DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 2 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory 3. Processor 2 reads the program in memory 3 and completes the steps of the aforementioned method in combination with its hardware.

[0139] When processor 2 executes the program, it implements the corresponding processes in the various methods of the embodiments of this application. For the sake of brevity, these will not be described in detail here.

[0140] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory 3 that stores a computer program, which can be executed by a processor 2 to complete the steps described in the aforementioned method. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0141] In the several embodiments provided in this application, it should be understood that the disclosed apparatus, electronic devices, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0142] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0143] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0144] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0145] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0146] It should be noted that the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict. Unless otherwise stated and limited, the term "connection" should be interpreted broadly. For example, it can refer to an electrical connection, or the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above term according to the specific circumstances.

[0147] Furthermore, in the examples of this application, "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the objects distinguished by "first," "second," and "third" can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than those illustrated or described herein.

[0148] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Additionally, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0149] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0150] The specific technical features described in the various embodiments in the detailed implementation can be combined in various ways without contradiction. For example, different implementation methods can be formed by combining different specific technical features. In order to avoid unnecessary repetition, the various possible combinations of the specific technical features in this application will not be described separately.

Claims

1. A text processing method, characterized in that, The method includes: Input the first text vector of the first text into the set encoder to obtain the first feature vector output by the set encoder; Determine that the first feature vector corresponds to the first key vector of each transformation matrix in at least one transformation matrix; The first feature vector and at least one corresponding first key vector are input into a set decoder to obtain a second text vector output by the set decoder; the set decoder is used to calculate the second text vector by inputting the first feature vector and at least one corresponding first key vector; when the number of transformation matrices is greater than 1, the same first feature vector corresponds to multiple different second text vectors. Based on the second text vector, a second text with a corresponding first annotation is generated; the first annotation represents the annotation of the first text, and the second text is used as sample data to train the text classification model. Determining that the first feature vector corresponds to the first key vector of each transformation matrix in at least one transformation matrix includes: The first feature vector is processed through a first hidden layer; the first hidden layer represents a set hidden layer that discards at least one neuron; Based on each of the at least one transformation matrix, determine the first key vector corresponding to the processed first feature vector; The number of transformation matrices is the same as the number of corresponding first key vectors, and the number is the number of the third hidden layer of the decoder, wherein the number of the third hidden layer is greater than or equal to one.

2. The method according to claim 1, characterized in that, After inputting the first feature vector and at least one corresponding first key vector into the setting decoder and before the setting decoder outputs the second text vector, the method includes: In the defined decoder, for the first i The similarity between each second feature vector and each of the at least one first key vectors is calculated to obtain the weight corresponding to each first key vector. Based on each first key vector and its corresponding weight in the at least one first key vector, the first... i +1 second feature vectors; where, i It is a positive integer; i When =1, the first i The second eigenvector is the first eigenvector; the... i +1 second feature vector is used to determine the first one in the second text. i One character.

3. The method according to claim 1, characterized in that, Before inputting the first feature vector and at least one corresponding first key vector into the decoder, the method further includes: The decoder is trained and configured based on the third feature vector and at least one corresponding second key vector; wherein... The third feature vector represents the feature vector output by the encoder based on the third text vector corresponding to the training sample; the at least one second key vector represents the key vector of each transformation matrix in the at least one transformation matrix corresponding to the third feature vector.

4. The method according to claim 3, characterized in that, Before training and setting the decoder based on the third feature vector and at least one corresponding second key vector, the method further includes: Replace at least one element in the third text vector with a set value.

5. The method according to claim 1, characterized in that, The encoder is set to the BERT model; and / or the decoder is set to the GPT model.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: A natural language processing (NLP) model is trained based on the second text and the first annotation.

7. A text processing device, characterized in that, include: The first input unit is used to input the first text vector of the first text into the set encoder to obtain the first feature vector output by the set encoder; A first processing unit is configured to determine that the first feature vector corresponds to a first key vector of each transformation matrix in at least one transformation matrix; The second input unit is used to input the first feature vector and at least one corresponding first key vector into the setting decoder to obtain the second text vector output by the setting decoder; the setting decoder is used to calculate the input first feature vector and at least one corresponding first key vector to determine the second text vector; when the number of transformation matrices is greater than 1, the same first feature vector corresponds to multiple different second text vectors; The generation unit is used to generate second text with the first annotation based on the second text vector; The first annotation represents the annotation of the first text, and the second text is used as sample data to train the text classification model; Determining that the first feature vector corresponds to the first key vector of each transformation matrix in at least one transformation matrix includes: The first feature vector is processed through a first hidden layer; the first hidden layer represents a set hidden layer that discards at least one neuron; Based on each of the at least one transformation matrix, determine the first key vector corresponding to the processed first feature vector; The number of transformation matrices is the same as the number of corresponding first key vectors, and the number is the number of the third hidden layer of the decoder, wherein the number of the third hidden layer is greater than or equal to one.

8. An electronic device, characterized in that, include: The processor and the memory used to store computer programs that can run on the processor. When the processor is used to run the computer program, it performs the steps of the method according to any one of claims 1 to 6.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images