A small sample text classification method and model

Abstract

The present invention discloses a small sample text classification method and model, comprising the following steps: step a. performing word vector conversion on the vocabulary in the input corpus information to obtain a word vector matrix; step b. processing the word vector matrix through a bidirectional long short-term memory layer to obtain a hidden layer state sequence corresponding to the word vector matrix; and extracting a self-attention feature vector e based on the hidden layer state sequence to obtain a self-attention feature vector matrix E; step c. characterizing a category vector c through a capsule network based on the self-attention feature vector matrix E to obtain category vectors c of all known categories; step d. performing a relationship score on the input corpus information based on the category vectors c of all known categories to obtain the category vector c with the highest similarity to the input corpus information, and the category corresponding to the category vector c is the classification result of the input corpus information. The present invention can learn an inductive network model of text spatial features and perform more effective induction on the class representation of text semantics.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a small sample text classification method and model. Background Art

[0002] Current artificial intelligence is essentially an intelligence that is achieved to a certain extent by using the powerful computing power of computers to find the potential biological or mathematical meaning of massive amounts of data.

[0003] However, data acquisition and annotation are very expensive, so how to learn a usable model with a small number of samples has become a hot topic. In existing technologies, twin networks, prototype networks, or relational networks are generally used to achieve this.

[0004] In the existing technology, the algorithms used are all the means or distance metrics used to calculate class representations. They measure samples in the same dimension, but lose the spatial information of the sample distribution. For example, "I danced with joy because I got good grades" and "My dancing has nothing to do with good grades." The key parts of the two sentences, "dancing with joy" and "good grades," are the same, but because of different spatial information (different order relationships), they express different emotions and belong to different categories. In this case, the algorithms in the existing technology cannot effectively summarize the class representation of text semantics and cannot reflect the deeper spatial characteristics of the text. Summary of the Invention

[0005] The main purpose of the present invention is to provide a small sample text classification method and model, which can learn text space features, more effectively summarize the class representation of text semantics, and reflect deeper text space features.

[0006] To achieve the above-mentioned purpose, the present invention provides a small sample text classification method, which includes the following steps: step a. performing word vector conversion on the vocabulary in the input corpus information to obtain a word vector matrix; step b. processing the word vector matrix through a bidirectional long short-term memory layer to obtain a hidden state sequence corresponding to the word vector matrix; and extracting a self-attention feature vector e based on the hidden state sequence to obtain a self-attention feature vector matrix E; step c. based on the self-attention feature vector matrix E, representing the category vector c through a capsule network to obtain the category vectors c of all known categories; step d. performing relationship scoring on the input corpus information based on the category vectors c of all known categories, wherein the relationship scoring refers to judging the similarity between the input corpus information and the category vector c to obtain the category vector c with the highest similarity to the input corpus information, and the category corresponding to the category vector c is the classification result of the input corpus information.

[0007] Optionally, before step a, a data reading step is also included: samples of the support set and the query set are randomly selected from the original corpus information in a C-way K-shot manner to construct a training set; and samples of the support set and the query set are randomly selected from the original corpus information in a C-way Q-shot manner to construct a validation set; wherein C represents the number of categories, K represents the number of training set samples, and Q represents the number of validation set samples.

[0008] Optionally, the input corpus information is obtained after data processing of the training set; the data processing includes the following steps: Step 1: pre-processing the training samples in the training set and constructing a character table, which is a list of all Chinese characters contained in the training samples; digitizing each character in the training samples based on the character table to obtain a digitized training sample; Step 2: classifying the sentence text of the digitized training sample to obtain the intent category of each sentence text, reading the sentence text and its corresponding category in batches; and segmenting the sentence text to obtain annotated labels; Step 3: splicing the annotated labels with the sentence text length and text mask to form the input corpus information.

[0009] Optionally, step a is implemented by using one of a BERT pre-training model, a skip-gram model or a cbow model.

[0010] Optionally, in step a, the maximum length of the sentences in the input corpus information is set to n, the word vector dimension is set to k, and the word vector matrix is an n*k matrix; the maximum length of the sentences n is set to 32 or 64, and the word vector dimension k is set to 128 or 256.

[0011] Optionally, step b includes the following steps: b1. processing the word vector matrix through a bidirectional long short-term memory layer to obtain a hidden state sequence Q1 corresponding to the word vector matrix; b2. performing a dot multiplication operation on the hidden state sequence Q1 and K1 to obtain the correlation of the word vector matrix, where K1 represents the first weight sequence vector; b3. summing the correlations and taking the average value to obtain the self-attention feature vector matrix E.

[0012] Optionally, performing similarity judgment between the input corpus information and the category vector c refers to calculating the similarity between the self-attention feature vector e and the category vector c; this is achieved by calculating one of the inner product, the cosine similarity, or the Pearson correlation coefficient.

[0013] Optionally, the step c includes the following steps: c1. linearly mapping the self-attention feature vector matrix E to obtain the mapping attention feature vector matrix e', and then compressing the mapping attention feature vector matrix e' into a unit vector of capsule size; c2. initializing the variable b = 0, summing and normalizing the variable b to obtain the vector d, wherein the vector d is the weight of the routing index; c3. performing a dot multiplication operation on the vector d and the mapping attention feature vector matrix e', and then performing a dot multiplication operation on the result of the operation and the mapping attention feature vector matrix e' to obtain the variable b'; c4. updating the vector b with the variable b', and repeating steps c1 and c2, iterating m times, until the mapping of each category sample to the category summary vector is obtained; c5. normalizing the mapping of each category sample to the category summary vector to obtain the representation probability of the corresponding category.

[0014] Optionally, the step d includes the following steps: d1. performing an inner product operation on the category vector c and the self-attention feature vector e to obtain a relationship matrix between the category vector c and the self-attention feature vector e; d2. performing neural network modeling on the relationship matrix and then activating it using an activation function to compress the calculation result of the relationship matrix to the range of (0, 1); d3. performing one-hot encoding on the compressed relationship matrix to obtain the classification result of the input corpus information.

[0015] In addition, corresponding to the small sample text classification method, a small sample text classification model includes: a text representation layer: used to convert the vocabulary in the input corpus information into word vectors to obtain a word vector matrix; a BiLSTM layer: used to process the word vector matrix to obtain a hidden state sequence corresponding to the word vector matrix; a self-attention mechanism layer: extracting the attention feature vector based on the hidden state sequence to obtain an attention feature vector matrix e; a dynamic routing layer: based on the attention feature vector matrix e, the category vector c is represented by a capsule network to obtain the category vectors c of all known categories; a relationship layer: based on the category vectors c of all known categories, the input corpus information is scored for the relationship, and the relationship scoring refers to judging the similarity between the input corpus information and the category vector c to obtain the category vector c with the highest similarity to the input corpus information. The category corresponding to the category vector c is the classification result of the input corpus information.

[0016] The beneficial effects of the present invention are:

[0017] (1) This paper proposes a small sample text classification method that can learn text space features through text semantic encoding, text class representation induction, and text relationship evaluation. It can more effectively summarize the class representation of text semantics and reflect deeper text space features.

[0018] (2) Train the model through the training set to determine the basic parameters of the model, and adjust the basic parameters through the validation set to assist in model construction;

[0019] (3) Data processing of the original corpus information to obtain input corpus information can effectively improve the efficiency of subsequent text intent classification;

[0020] (4) By setting the maximum sentence length n and word vector dimension k, the problem of low efficiency of batch input caused by inconsistent sentence length can be avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0022] Figure 1 This is a simplified flowchart of the small sample text classification method of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0024] like Figure 1 As shown, a small sample text classification method of the present invention includes the following steps: step a. converting the vocabulary in the input corpus information into word vectors to obtain a word vector matrix; step b. processing the word vector matrix through a bidirectional long short-term memory layer to obtain a hidden state sequence corresponding to the word vector matrix; and extracting the self-attention feature vector e based on the hidden state sequence to obtain a self-attention feature vector matrix E; step c. based on the self-attention feature vector matrix E, representing the category vector c through a capsule network to obtain the category vectors c of all known categories; step d. performing relationship scoring on the input corpus information based on the category vectors c of all known categories, the relationship scoring refers to judging the similarity between the input corpus information and the category vector c, and obtaining the category vector c with the highest similarity to the input corpus information, and the category corresponding to the category vector c is the classification result of the input corpus information.

[0025] The main purpose of the present invention is to provide a small sample text classification method and model, which can learn text space features, more effectively summarize the class representation of text semantics, and reflect deeper text space features.

[0026] In this embodiment, a data reading step is also included before step a: samples of the support set and the query set are randomly selected from the original corpus information in a C-way K-shot manner to construct a training set; and samples of the support set and the query set are randomly selected from the original corpus information in a C-way Q-shot manner to construct a validation set; where C represents the number of categories, K represents the number of training set samples, and Q represents the number of validation set samples.

[0027] In this embodiment, the training set is used to train the model and determine its basic parameters, while the validation set is used to adjust these parameters during the training process. Training is terminated when the model parameters are finalized. In other words, the training set is used to train the model and determine its basic parameters, while the validation set is used to adjust these parameters and assist in model construction.

[0028] In this embodiment, step a is implemented by using one of the BERT pre-training model, the Skip-Gram model, or the CBOW model.

[0029] In this embodiment, in step a, the maximum sentence length in the input corpus is set to n, the word vector dimension is set to k, and the word vector matrix is an n*k matrix; the maximum sentence length n is set to 32 or 64, and the word vector dimension k is set to 128 or 256. By setting the maximum sentence length n and word vector dimension k, the problem of inefficient batch input caused by inconsistent sentence lengths can be avoided.

[0030] In this embodiment, the input corpus information is obtained after data processing of the training set; the data processing includes the following steps: Step 1: Preprocess the training samples in the training set and construct a character table, which is a list of all Chinese characters contained in the training samples; digitize each character in the training samples based on the character table to obtain a digitized training sample. Step 2: Perform text intent classification on the sentence text of the digitized training sample to obtain the intent category to which each sentence text belongs, read the sentence text and its corresponding category in batches; and segment the sentence text to obtain annotated labels. Step 3: Concatenate the annotated labels with the sentence text length and text mask to form the input corpus information.

[0031] Preferably, the preprocessing in step 1 further includes removing special characters, stop words and emoticons.

[0032] In this embodiment, the digitization and vectorization of the input corpus information are illustrated by taking the medical beauty industry corpus as an example. Please refer to the digitization and vectorization processing results of the input corpus information in Table 1.

[0033]

[0034] Table 1

[0035] The present invention obtains input corpus information by performing data processing on original corpus information, which can effectively improve the efficiency of subsequent text intent classification.

[0036] It should be noted that in the medical field, the present invention can achieve an accuracy of 0.76 in the text intent classification task when only a small number of samples are included in each class, which is not much different from the accuracy of about 0.8 obtained with massive data.

[0037] In this embodiment, step b includes the following steps: b1. Processing the word vector matrix through the bidirectional long short-term memory layer to obtain the hidden state sequence Q1 corresponding to the word vector matrix; b2. Performing a dot multiplication operation on the hidden state sequence Q1 and K1 to obtain the correlation of the word vector matrix, where K1 represents the first weight sequence vector; b3. Sum the correlations and take the average value to obtain the self-attention feature vector matrix E.

[0038] In this embodiment, performing similarity judgment between the input corpus information and the category vector c refers to calculating the similarity between the self-attention feature vector e and the category vector c; this is achieved by calculating one of the inner product, the cosine similarity, or the Pearson correlation coefficient.

[0039] In this embodiment, step c includes the following steps: c1. linearly mapping the self-attention feature vector matrix E to obtain the mapping attention feature vector matrix e', and then compressing the mapping attention feature vector matrix e' into a unit vector of capsule size; c2. initializing the variable b = 0, summing and normalizing the variable b to obtain the vector d, where the vector d is the weight of the routing index; c3. performing a dot product operation on the vector d and the mapping attention feature vector matrix e', and then performing a dot product operation on the result of the operation with the mapping attention feature vector matrix e' to obtain the variable b'; c4. updating the vector b with the variable b', and repeating steps c1 and c2, iterating m times, until the mapping of each category sample to the category summary vector is obtained; c5. normalizing the mapping of each category sample to the category summary vector to obtain the representation probability of the corresponding category.

[0040] It should be noted that m is a hyperparameter and can be set as needed. The iteration part is time-consuming, so the value of m is generally set to a relatively small value, preferably m=3, or m=4, or m=5.

[0041] In this embodiment, step d includes the following steps: d1. performing an inner product operation on the category vector c and the self-attention feature vector e to obtain a relationship matrix between the category vector c and the self-attention feature vector e; d2. performing neural network modeling on the relationship matrix and then activating it using an activation function to compress the calculated result of the relationship matrix to the range of (0, 1); d3. performing one-hot encoding on the compressed relationship matrix to obtain the classification result of the input corpus information.

[0042] In this embodiment, a small sample text classification model includes: a text representation layer: used to convert words in the input corpus information into word vectors to obtain a word vector matrix; a BiLSTM layer: used to process the word vector matrix to obtain a hidden state sequence corresponding to the word vector matrix; a self-attention mechanism layer: extracting attention feature vectors based on the hidden state sequence to obtain an attention feature vector matrix e; a dynamic routing layer: based on the attention feature vector matrix e, representing the category vector c through a capsule network to obtain the category vectors c of all known categories; a relationship layer: based on the category vectors c of all known categories, performing relationship scoring on the input corpus information. Relationship scoring refers to judging the similarity between the input corpus information and the category vector c to obtain the category vector c with the highest similarity to the input corpus information. The category corresponding to the category vector c is the classification result of the input corpus information.

[0043] As used herein, the terms "comprises," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0044] While the foregoing description shows and describes preferred embodiments of the present invention, it should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. Rather, the present invention can be used in various other combinations, modifications, and environments, and can be modified within the scope of the present invention by the teachings herein or by techniques or knowledge in the relevant art. Modifications and variations made by those skilled in the art without departing from the spirit and scope of the present invention are intended to be within the scope of the appended claims.

Claims

1. A small sample text classification method, characterized in that: The following steps are involved: Step a. Convert the words in the input corpus information into word vectors to obtain a word vector matrix; Step b. Processing the word vector matrix through a bidirectional long short-term memory layer to obtain a hidden layer state sequence corresponding to the word vector matrix; and extracting a self-attention feature vector e based on the hidden layer state sequence to obtain a self-attention feature vector matrix E; Step c. Based on the self-attention feature vector matrix E, the category vector c is represented by the capsule network to obtain the category vectors c of all known categories; Step d. performing a relationship score on the input corpus information based on the category vectors c of all known categories, wherein the relationship score refers to performing a similarity judgment between the input corpus information and the category vector c, and obtaining the category vector c with the highest similarity to the input corpus information. The category corresponding to the category vector c is the classification result of the input corpus information; The step c comprises the following steps: c1. Perform a linear mapping on the self-attention feature vector matrix E to obtain the mapped attention feature vector matrix e', and then compress the mapped attention feature vector matrix e' into a unit vector of the capsule size; c2 initialize the variable b = 0, the variable b is summed and normalized to obtain a vector d, the vector d is the weight of the routing index; c3. Perform a dot product operation on vector d and the mapping attention feature vector matrix e', and then perform a dot product operation on the result of the operation with the mapping attention feature vector matrix e' to obtain the variable b'; c4. Update vector b with variable b' and repeat steps c1 and c2 for m times until the mapping of each category sample to the category summary vector is obtained; c5. Normalize the mapping of each category sample to the category summary vector to obtain the representation probability of the corresponding category; Before step a, a data reading step is also included: The samples of the support set and query set are randomly selected from the original corpus information using the C-way K-shot method to construct the training set; the samples of the support set and query set are randomly selected from the original corpus information using the C-way Q-shot method to construct the validation set; where C represents the number of categories, K represents the number of training set samples, and Q represents the number of validation set samples; The input corpus information is obtained after data processing of the training set; The data processing includes the following steps: Step 1: Preprocess the training samples in the training set and construct a character table, which is a list of all Chinese characters contained in the training samples; digitize each character in the training samples based on the character table to obtain a digitized training sample; Step 2: Perform text intent classification on the sentence texts of the digitized training samples to obtain the intent category of each sentence text, read the sentence texts and their corresponding categories in batches; and segment the sentence texts to obtain annotation labels; Step 3: Concatenate the annotation label with the sentence text length and text mask to form the input corpus information; Described step b comprises the following steps: b1. Process the word vector matrix through a bidirectional long short-term memory layer to obtain a hidden layer state sequence Q1 corresponding to the word vector matrix; b2. Perform a dot product operation on the hidden state sequence Q1 and K1 to obtain the correlation of the word vector matrix, where K1 represents the first weight sequence vector; b3. Sum and average the correlations to obtain the self-attention feature vector matrix E.

2. The small sample text classification method according to claim 1, characterized in that: The step a is implemented by using one of the BERT pre-training model, the Skip-Gram model or the CBOW model.

3. The small sample text classification method according to claim 1, characterized in that: In step a, the maximum length of the sentences in the input corpus information is set to n, the word vector dimension is set to k, and the word vector matrix is an n*k matrix; The maximum sentence length n is set to 32 or 64, and the word vector dimension k is set to 128 or 256.

4. The small sample text classification method according to claim 1, characterized in that: Performing similarity judgment on the input corpus information and the category vector c refers to calculating the similarity between the self-attention feature vector e and the category vector c; This is achieved by calculating the inner product, the cosine similarity, or the Pearson correlation coefficient.

5. The small sample text classification method according to claim 1, characterized in that: The step d comprises the following steps: d1. Perform an inner product operation on the category vector c and the self-attention feature vector e to obtain a relationship matrix between the category vector c and the self-attention feature vector e; d2. After performing neural network modeling on the relationship matrix, activation is performed using an activation function to compress the calculated result of the relationship matrix into the range of (0, 1); d3. Perform one-hot encoding on the compressed relationship matrix to obtain the classification results of the input corpus information.

6. A small sample text classification system, characterized by: include: Text representation layer: It is used to randomly select samples of the support set and query set from the original corpus information in the C-way K-shot method to construct the training set; and randomly select samples of the support set and query set from the original corpus information in the C-way K-shot method to construct the training set; The Q-shot method randomly selects samples from the original corpus information to construct a verification set; wherein C represents the number of categories, K represents the number of training set samples, and Q represents the number of verification set samples; the vocabulary in the input corpus information is converted into a word vector to obtain a word vector matrix; the input corpus information is obtained after data processing of the training set; the data processing includes the following steps: Step 1: preprocessing the training samples in the training set and constructing a character table, which is a list of all Chinese characters contained in the training sample; digitizing each character in the training sample based on the character table to obtain a digitized training sample; Step 2: classifying the sentence text of the digitized training sample into a text intent category to obtain the intent category of each sentence text, reading the sentence text and its corresponding category in batches; and segmenting the sentence text to obtain an annotation label; Step 3: splicing the annotation label with the sentence text length and text mask to form the input corpus information; BiLSTM layer: used to process the word vector matrix to obtain the hidden state sequence corresponding to the word vector matrix; Self-attention mechanism layer: Process the word vector matrix through a bidirectional long short-term memory layer to obtain a hidden state sequence corresponding to the word vector matrix; and extract the attention feature vector based on the hidden state sequence to obtain a self-attention feature vector matrix E; at least including the following steps: b1. Process the word vector matrix through a bidirectional long short-term memory layer to obtain a hidden state sequence Q1 corresponding to the word vector matrix; b2. Perform a dot product operation on the hidden state sequence Q1 and K1 to obtain the correlation of the word vector matrix, where K1 represents the first weight sequence vector; b3. Sum and average the correlations to obtain the self-attention feature vector matrix E; Dynamic routing layer: Based on the self-attention feature vector matrix E, the category vector c is represented by the capsule network to obtain the category vectors c of all known categories; specifically, the following steps are included: c1. Perform a linear mapping on the self-attention feature vector matrix E to obtain the mapping attention feature vector matrix e', and then compress the mapping attention feature vector matrix e' into a unit vector of the capsule size; c2. Initialize the variable b=0, sum and normalize the variable b to obtain the vector d, and the vector d is the weight of the routing index; c3. Perform a dot product operation on the vector d and the mapping attention feature vector matrix e', and then perform a dot product operation on the result with the mapping attention feature vector matrix e' to obtain the variable b'; c4. Update the vector b with the variable b', and repeat steps c1 and c2 for m times until the mapping of each category sample to the category summary vector is obtained; c5. Normalize the mapping of each category sample to the category summary vector to obtain the representation probability of the corresponding category; Relationship layer: The input corpus information is scored based on the category vectors c of all known categories. The relationship scoring refers to judging the similarity between the input corpus information and the category vector c, and obtaining the category vector c with the highest similarity to the input corpus information. The category corresponding to the category vector c is the classification result of the input corpus information.

Images