Text classification method and apparatus, terminal device, and storage medium

Abstract

The application discloses a text classification method and device, terminal equipment and a storage medium. The text classification method comprises the following steps: obtaining a text to be processed; inputting the text to be processed into a pre-constructed text classification model for processing to obtain classified text data, wherein the text classification model comprises an encoder, a plurality of adaptive adapters and a preset template, and the text classification model is obtained through model prediction and integrated learning training based on the encoder, the plurality of adaptive adapters, the preset template and an optimizer. The application solves the problem of requiring a large amount of labeled data, improves the stability of model training, and reduces the required calculation and storage resources for training.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of natural language processing, and in particular to a text classification method and device, a terminal device, and a storage medium. BACKGROUND

[0002] At present, the financial audit text data is generally complex and difficult to label. For the scene of financial audit intent classification, the main method is to convert the text into token id of the corresponding dictionary by word segmentation, input the token id into the pre-trained model of different nlp fields, and directly output different classification results after average pooling and full connection layer splicing of the output token features.

[0003] However, the above-mentioned scheme still has many technical defects and deficiencies. The problem of the intent classification method based on the pre-trained model is that the method depends on a large amount of labeled data, and the training data needs to be preprocessed, and the entire pre-trained model weight is updated in the model fine-tuning stage, which leads to poor training stability of the model and consumes a large amount of training resources. SUMMARY

[0004] The main purpose of the present application is to provide a text classification method, device, terminal device and storage medium, which aims to solve the technical problems of unstable model training and large amount of calculation and storage resources caused by the need for a large amount of labeled data in text classification.

[0005] To achieve the above-mentioned purpose, the embodiment of the present application provides a text classification method, which comprises:

[0006] acquiring a text to be processed;

[0007] inputting the text to be processed into a pre-constructed text classification model for processing to obtain classified text data, wherein the text classification model comprises an encoder, a plurality of adaptive adapters, and a preset template, and the text classification model is obtained by model prediction and integrated learning training based on the encoder, the plurality of adaptive adapters, the preset template, and an optimizer.

[0008] Optionally, the step of inputting the text to be processed into the pre-constructed text classification model for processing to obtain classified text data further comprises:

[0009] obtaining the text classification model by model prediction and integrated learning training based on a preset model, an encoder, a plurality of adaptive adapters, a MASK template, and an optimizer.

[0010] Optionally, the step of obtaining the text classification model by model prediction and integrated learning training based on an encoder, a plurality of adaptive adapters, a MASK template, and an optimizer comprises:

[0011] obtaining pre-processed text sample data;

[0012] inputting the text sample data, word segmentation and mapping into word granularity id into an encoder to extract local features;

[0013] constructing an initial classification model based on the preset model, local features, a plurality of adaptive adapters and a MASK template;

[0014] training the initial classification model through the optimizer and the loss function to minimize the loss function, obtain model parameter values, and obtain a trained classification model;

[0015] performing sample prediction through the trained classification model to obtain model classification results;

[0016] based on the model classification results of a plurality of trained classification models, selecting a final text classification model through ensemble learning and a preset voting strategy.

[0017] Optionally, the step of inputting the text sample data, word segmentation and mapping into word granularity id into an encoder to extract local features comprises:

[0018] using a preset pre-training module to perform feature output on the text sample data, and the encoder performs full connection and convolution operations on input word vectors to extract local features.

[0019] Optionally, the step of constructing an initial classification model based on the local features, a plurality of adaptive adapters and a MASK template comprises:

[0020] adding the plurality of adaptive adapters to the preset model to generate an adapter layer;

[0021] generating a MASK template based on the preset model, the adapter layer and the text sample data;

[0022] obtaining an initial classification model according to the MASK template.

[0023] Optionally, the step of training the initial classification model through the optimizer and the loss function to minimize the loss function, obtain model parameter values, and obtain a trained classification model comprises:

[0024] According to the classification results calculated by the initial classification model, the loss function calculates the difference between the scores of a plurality of hidden tokens token spliced after the text sample data and the classification results obtained by a plurality of initial models and the true results, to obtain an optimized loss function.

[0025] The optimizer combines the optimized loss function to perform gradient descent method iteration to obtain minimized loss function and model parameter value, and obtain the trained classification model.

[0026] Optionally, the step of obtaining the model classification result by the trained classification model includes:

[0027] Obtaining training samples;

[0028] Inputting the training samples into the trained classification model for prediction;

[0029] According to the preset label vector ， Calculating the vector value of each hidden word token in the training sample and the average vector value of all hidden words tokens in the current label position in any label in the training sample;

[0030] According to the similarity between the vector value of each hidden word token and the average vector value of the hidden word token in the current label position, the current classification result position is measured;

[0031] Fusing each classification result position to obtain the final model classification result.

[0032] Optionally, the step of selecting the final text classification model based on the model classification results of the plurality of trained classification models through ensemble learning and a preset voting strategy includes:

[0033] Based on the model classification results of the plurality of trained classification models, a plurality of text classification models are trained using the strategy of ensemble learning, and the optimal text classification model is selected as the final text classification model through a preset voting strategy.

[0034] Optionally, the step of adding the plurality of adaptive adapters to the preset model to obtain an adapter layer includes:

[0035] The plurality of adaptive adapters are added to the preset model, and the local feature parameters are obtained after dimension reduction by processing the local feature through the adaptive adapter;

[0036] The dimension of the local feature parameters after dimension reduction is restored through the neural network in the preset model to obtain the adapter layer.

[0037] Optionally, the step of generating a Mask template based on the preset model, the adapter layer and the text sample data includes:

[0038] Learning the implicit description of each text sample data through the adapter layer in the preset model to generate an adaptive task template;

[0039] concatenate several hidden tokens behind the text sample data;

[0040] process the several hidden tokens through an encoder to obtain a hidden state representation of each hidden token;

[0041] obtain a MASK template according to the adaptive task template and the hidden state representation;

[0042] The step of obtaining an initial classification model according to the MASK template comprises:

[0043] According to a pre-set label vector, calculate the inner product of each hidden state representation and the label vector at the corresponding position through the MASK template to obtain an initial model classification result.

[0044] The embodiment of the present application also provides a text classification device, which comprises:

[0045] a text acquisition module, configured to acquire a text to be processed;

[0046] a text classification module, configured to input the text to be processed into a pre-created text classification model for classification to obtain classified text data, wherein the text classification model comprises an encoder, several adaptive adapters, a preset template, an optimizer, model prediction, and is obtained based on the encoder, the several adaptive adapters, the preset template, the optimizer, the model prediction and ensemble learning training.

[0047] The embodiment of the present application also provides a terminal device, which comprises a memory, a processor and a text classification program stored on the memory and executable on the processor, and when the text classification program is executed by the processor, the steps of the text classification method are realized.

[0048] The embodiment of the present application also provides a computer readable storage medium, which stores a text classification program, and when the text classification program is executed by a processor, the steps of the text classification method are realized.

[0049] This invention proposes a text classification method, apparatus, terminal device, and storage medium. The method involves acquiring text to be processed and inputting it into a pre-constructed text classification model for processing, resulting in classified text data. The text classification model includes an encoder, several adaptive adapters, and a preset template. The model is trained through model prediction and ensemble learning based on the encoder, adaptive adapters, preset template, and optimizer. Classifying text using the trained text classification model solves the problem of requiring a large amount of standardized data and avoids the waste of significant training resources. Based on this invention, starting from the complexity and difficulty of labeling financial audit text data, a text classification model is constructed. The effectiveness of the proposed text classification method is verified on text sample data. Finally, the text classification model obtained through this invention reduces the number of parameters that need to be updated and saved during model fine-tuning, improves model training stability, and reduces the computational and storage resources required for training. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the functional modules of the terminal device to which the text classification device of the present invention belongs;

[0051] Figure 2 This is a flowchart illustrating an exemplary embodiment of the text classification method of the present invention.

[0052] Figure 3 This is a flowchart illustrating another exemplary embodiment of the text classification method of the present invention;

[0053] Figure 4 This is a schematic diagram illustrating the process of constructing an initial classification model in the text classification method embodiment of the present invention, which involves model prediction and ensemble learning training based on an encoder, several adaptive adapters, a mask template, and an optimizer to obtain the text classification model.

[0054] Figure 5 This is a schematic diagram of the overall process for obtaining a text classification model in an embodiment of the text classification method of the present invention.

[0055] Figure 6 This is a schematic diagram illustrating the Mask template principle involved in an embodiment of the text classification method of the present invention;

[0056] Figure 7 This is a schematic diagram illustrating the process of obtaining the text classification model based on an encoder, several adaptive adapters, a mask template, and an optimizer through model prediction and ensemble learning training in an embodiment of the text classification method of the present invention.

[0057] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0058] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0059] The main solution of this invention is as follows: Pre-processed text sample data is obtained by configuring an encoder, adaptive adapter, mask template, and optimizer in a text classification model; the text sample data is segmented and mapped into word-level IDs, which are then input into the encoder to extract local features; an initial classification model is constructed based on the preset model, local features, several adaptive adapters, and mask templates; the initial classification model is trained using the optimizer and a loss function, minimizing the loss function to obtain model parameter values, resulting in a trained classification model; sample prediction is performed using the trained classification model to obtain the model classification result; based on the model classification results of several trained classification models, the final text classification model is selected through ensemble learning and a preset voting strategy. When classifying text to be classified using the trained text classification model, the text to be classified is input into the pre-created text classification model for classification, resulting in classified text. This solves the problem of requiring a large amount of labeled data, improves the stability of model training, and reduces the computational and storage resources required for training. Based on the present invention, starting from the complexity and difficulty of annotation of financial audit text data, a text classification model is constructed, and the effectiveness of the text classification method proposed in this invention is verified on text sample data. Finally, the text classification model of this invention, by combining the lightweight transfer learning method of the adaptive adapter idea, reduces the number of parameters that need to be updated and saved in the model fine-tuning stage, improves the training stability of the model, and reduces the computational and storage resources required for training.

[0060] Technical terms involved in the embodiments of this invention:

[0061] Token: A token can be a word, a letter, or even "half a word";

[0062] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for effective communication between humans and computers using natural language; it's a science that integrates linguistics, computer science, and mathematics. Its aim is to extract information from textual data. The goal is to enable computers to process or "understand" natural language to perform tasks such as automatic translation, text classification, and sentiment analysis. Natural Language Processing is one of the most challenging problems in artificial intelligence.

[0063] Model fine-tuning: refers to making minor adjustments to the model.

[0064] Loss function: (Focal Loss) Each sample will be processed by the model to obtain a predicted value, and the difference between the predicted value and the true value is called the loss.

[0065] Adam optimizer: (Adaptive Moment Estimation) can adapt to sparse gradients and alleviate the problem of gradient oscillations;

[0066] BERT, short for Bidirectional Encoder Representations from Transformer, is a pre-trained language representation model based on Transformer bidirectional encoder representations. It emphasizes a departure from traditional unidirectional language models or shallow concatenation of two unidirectional language models for pre-training. Instead, it employs a novel masked language model (MLM) to generate deep bidirectional language representations. The goal of BERT is to train on large-scale unlabeled corpora to obtain representations of text containing rich semantic information—that is, semantic representations of the text—and then fine-tune these semantic representations for specific NLP tasks before applying them to those tasks.

[0067] [Mask]: In order to use contextual information without revealing label information during the training of the language model, a portion of the tokens in the randomly selected sequence are marked with [Mask], where [Mask] is represented by a hidden word;

[0068] Label embedding refers to the process of grouping samples under a label into a vector. For a new prediction sample, after encoding, the encoded sample and this vector are used to calculate a score, and finally the prediction result of the sample is obtained.

[0069] Layer norm: refers to the sample itself;

[0070] Feedforward: refers to a network in which there is no feedback and the signal propagates unidirectionally from the input layer to the output layer. It can be represented by a directed acyclic graph.

[0071] Embedding layer: When the input to a neural network contains symbolic classification features, each possible value is typically associated with a vector. The mapping from symbolic features to vectors is achieved through the embedding layer (lookup layer).

[0072] Multi-head attention is an attention mechanism used in natural language processing models that allows the model to consider information from different locations simultaneously. By using multiple different "heads" to perform attention calculations, the model can understand text from multiple different perspectives. This helps improve the model's accuracy and efficiency.

[0073] The embodiments of the present invention take into account that the related technologies rely on a large amount of labeled data, and at the same time require preprocessing of training data and updating of parameters of the entire pre-trained model weights during the model fine-tuning stage, resulting in poor training stability of the model and a large amount of training resources being consumed.

[0074] Therefore, the embodiments of the present invention, starting from the practical problem of text intent classification, design a text classification model based on an encoder, an adaptive adapter, and a preset template, which solves the problem of relying on a large amount of labeled data in the model fine-tuning stage, improves the training stability of the model, and reduces the computing and storage resources required for training.

[0075] Specifically, refer to Figure 1 , Figure 1 This is a functional template diagram of the terminal device to which the text classification device of the present invention belongs. The text classification device can be an independent device capable of text classification and text classification model training, and can be implemented on the terminal device in hardware or software form. The terminal device can be a smart mobile terminal with data processing capabilities, such as a mobile phone or tablet computer, or a fixed terminal device or server with data processing capabilities.

[0076] In this embodiment, the terminal device to which the text classification device belongs includes at least an output module 110, a processor 120, a memory 130, and a communication module 140.

[0077] The memory 130 stores the operating system and the text classification program. The text classification device can process the acquired text to be processed through the text classification model to obtain the classified text, as well as the pre-processed text sample data; the text sample data is segmented and mapped into word granularity IDs and input into the encoder to extract local features; an initial classification model is constructed based on the local features, several adaptive adapters, and preset templates; the initial classification model is trained using the optimizer and loss function based on the classification results, the loss function is minimized, the model parameter values ​​are obtained, and the trained classification model is obtained; the trained classification model is used to predict samples, and the model classification results and other information are stored in the memory 130.

[0078] When the text classification program in memory 130 is executed by the processor, it performs the following steps:

[0079] Get the text to be processed;

[0080] The text to be processed is input into a pre-built text classification model for processing to obtain classified text data. The text classification model includes a preset model, an encoder, several adaptive adapters, and a mask template. The text classification model is obtained by model prediction and ensemble learning training based on the preset model, encoder, several adaptive adapters, mask template, and optimizer.

[0081] Furthermore, when the text classification program in memory 130 is executed by the processor, it also performs the following steps:

[0082] The text classification model is obtained by performing model prediction and ensemble learning training based on a preset model, encoder, several adaptive adapters, mask templates, and optimizer.

[0083] Furthermore, when the text classification program in memory 130 is executed by the processor, it also performs the following steps:

[0084] Obtain pre-processed text sample data;

[0085] The text sample data is segmented and mapped into word granular IDs, which are then input into the encoder to extract local features;

[0086] An initial classification model is constructed based on the preset model, local features, several adaptive adapters, and the MASK template.

[0087] The initial classification model is trained using the optimizer and loss function, the loss function is minimized, the model parameter values ​​are obtained, and the trained classification model is obtained.

[0088] The trained classification model is used to predict the sample classification result.

[0089] Based on the classification results of several trained classification models, the final text classification model is selected through ensemble learning and a pre-set voting strategy.

[0090] Furthermore, when the text classification program in memory 130 is executed by the processor, it also performs the following steps:

[0091] The text sample data is used to output features using a preset pre-training module. The encoder performs fully connected and convolutional operations on the input word vector encoding to extract local features.

[0092] Furthermore, when the text classification program in memory 130 is executed by the processor, it also performs the following steps:

[0093] The aforementioned adaptive adapters are added to a preset model to generate an adapter layer;

[0094] Based on the preset model, a MASK template is generated using the adapter layer and text sample data;

[0095] An initial classification model is obtained based on the MASK template.

[0096] Furthermore, when the text classification program in memory 130 is executed by the processor, it also performs the following steps:

[0097] Based on the classification results calculated by the initial classification model, the difference between the scores of several hidden word tokens concatenated after the text sample data and the classification results obtained by several initial models and the true results is calculated by using a loss function, and an optimized loss function is obtained.

[0098] By combining the optimized loss function with an optimizer, gradient descent is used to iteratively solve the problem, resulting in the minimized loss function and model parameter values, thus obtaining the trained classification model.

[0099] Furthermore, when the text classification program in memory 130 is executed by the processor, it also performs the following steps:

[0100] Obtain training samples;

[0101] The training samples are input into the trained classification model for prediction.

[0102] Based on the preset label vector ， Calculate the vector value generated by each hidden word token in the training sample and the average vector value of all hidden word tokens at the current label position in any label of the training sample;

[0103] The position of the current classification result is measured based on the similarity between the vector value generated by each hidden word token and the average vector value of the hidden word tokens at the current label position.

[0104] The final model classification result is obtained by merging the positions of each classification result.

[0105] Furthermore, when the text classification program in memory 130 is executed by the processor, it also performs the following steps:

[0106] Based on the classification results of the several trained classification models, several text classification models are trained using an ensemble learning strategy. The text classification model with the best performance is selected as the final text classification model through a preset voting strategy.

[0107] Furthermore, when the text classification program in memory 130 is executed by the processor, it also performs the following steps:

[0108] The aforementioned adaptive adapters are added to the preset model, and the local features are processed through the adaptive adapters to obtain the number of local feature parameters after dimensionality reduction.

[0109] The adapter layer is obtained by restoring the dimensionality of the local feature parameters after dimensionality reduction through the neural network inside the preset model.

[0110] Furthermore, when the text classification program in memory 130 is executed by the processor, it also performs the following steps:

[0111] By using the adapter layer in the preset model, the implicit description of each text sample data is learned, and an adaptive task template is generated.

[0112] Several hidden word tokens are appended to the end of the text sample data;

[0113] The encoder processes the several hidden word tokens to obtain the hidden state representation of each hidden word token;

[0114] Based on the adaptive task template and the hidden state representation, obtain the MASK template;

[0115] The step of obtaining the initial classification model based on the MASK template includes:

[0116] Based on the pre-defined label vectors, the inner product of each hidden state representation and the corresponding label vector is calculated using the MASK template to obtain the initial model classification result.

[0117] This embodiment, through the above-described scheme, specifically obtains the text to be processed; inputs the text to be processed into a pre-constructed text classification model for processing, obtaining classified text data. The text classification model includes a preset model, an encoder, several adaptive adapters, and a mask template. The text classification model is trained through model prediction and ensemble learning based on the preset model, encoder, several adaptive adapters, mask template, and optimizer. Classifying text using the trained text classification model solves the problem of poor model training stability caused by the need for large amounts of data annotation, avoiding the consumption of large amounts of training resources. Based on the scheme of this invention, starting from the complexity and difficulty of annotation of financial audit text data, a text classification model is constructed, and the effectiveness of the text classification method proposed in this invention is verified on text sample data. Finally, the text classification model of this invention reduces the number of parameters that need to be updated and saved during the model fine-tuning stage, improves the training stability of the model, and reduces the computational and storage resources required for training.

[0118] Based on, but not limited to, the terminal device architecture described above, embodiments of the method of the present invention are proposed.

[0119] Reference Figure 2 , Figure 2 This is a flowchart illustrating an exemplary embodiment of the text classification method of the present invention. The text repair method includes:

[0120] Step S101: Obtain the text to be processed;

[0121] The execution subject of the method in this embodiment can be a text classification device, a text classification terminal device, or a server. This embodiment takes a text classification device as an example, which can be integrated into a terminal device with data processing function.

[0122] This embodiment mainly achieves text classification, especially financial audit text, improves the training stability of the text classification model, and reduces the consumption of training resources.

[0123] The text can be internal financial audit documents or other texts that need to be categorized.

[0124] Step S102: The text to be processed is input into a pre-built text classification model and processed to obtain classified text data. The text classification model includes an encoder, several adaptive adapters, and a preset template. The text classification model is obtained by model prediction and ensemble learning training based on the encoder, several adaptive adapters, the preset template, and the optimizer.

[0125] This embodiment starts from the practical problem of text intent classification and designs a text classification model based on BERT model, encoder, adaptive adapter, MASK template, and optimizer for model prediction and ensemble learning training.

[0126] This embodiment uses a text classification model to classify text. The framework of this text classification model includes: a BERT model, an encoder, an adaptive adapter, and a mask template. Wherein:

[0127] The encoder is used to perform operations such as fully connected layers and convolutions on preprocessed text data and extract local features;

[0128] The adaptive adapter is used to fully connect local features to achieve dimensionality reduction and further reduce the number of parameters. After passing through the neural network inside the original BERT model, the dimensionality is restored again to obtain the adapter layer.

[0129] The Mask template is used to enable the adapter layer of each text classification task to learn the implicit description of each text classification task. Based on the set label embedding, several hidden word tokens are concatenated to the input text sample data. After passing through the encoder, each hidden word token obtains a hidden state representation. The inner product of each hidden state representation and the corresponding label embedding is calculated to obtain the model classification result.

[0130] In this embodiment, the text classification model design takes into account the advantages of adaptive adapters, which allow for fine-tuning of the model with fewer parameters based on text data, and combines this with the ability of mask templates to make the adjusted parameters more accurate. Therefore, by using the trained text classification model to classify text, the problem of requiring a large amount of labeled data can be solved, the stability of model training can be improved, and the computational and storage resources required for training can be reduced.

[0131] Reference Figure 3 , Figure 3 This is a flowchart illustrating another exemplary embodiment of the text classification method of the present invention.

[0132] Based on the above Figure 2 In the embodiment shown, before the step of inputting the text to be processed into a pre-created text classification model for classification to obtain the classified text, the text classification method further includes:

[0133] Step S100 involves performing model prediction and ensemble learning training based on a preset model, encoder, several adaptive adapters, mask templates, and optimizer to obtain the text classification model. In this embodiment, step S100 is implemented before step S101. In other embodiments, step S100 can also be implemented between step S101 and step S102.

[0134] Compared to the above Figure 2 The embodiment shown also includes a scheme for obtaining a text classification model.

[0135] Specifically, this embodiment involves a BERT model equipped with a pre-trained transformer encoder model. First, pre-processed text sample data is acquired; the acquired text sample data is segmented and mapped into word granularity IDs, which are then input into the pre-trained transformer encoder model to extract local features.

[0136] Then, several adaptive adapters are added to the original BERT model. The text sample data is reduced in dimensionality by the adaptive adapters, which further reduces the number of parameters in the text sample data. The reduced number of parameters in the text sample data is then increased in dimensionality by the neural network inside the BERT model to obtain the adapter layer.

[0137] An adapter layer is used to generate a mask template, allowing each text classification task's unique adapter layer to automatically learn the implicit descriptions for that task. Several hidden word tokens are appended to the input text sample data. After passing through the encoder, each hidden word token yields a hidden state representation. Based on the pre-defined label vectors, the generated mask template is used to calculate the model's classification result.

[0138] The difference between the model classification results and the true labels of the text is calculated using a loss function. The result of the loss function calculation is then solved using the Adam optimizer to obtain the minimized loss function and model parameter values.

[0139] For a sample that produces several hidden word tokens, calculate the similarity between the vector produced by each hidden word token and the average vector of the hidden word tokens of all training samples at this position for a certain label. Use this similarity as a measure of the distance between the current sample at the current position and the label. Combine the results from each position to obtain the final label.

[0140] By using the concept of ensemble learning, several weak learners are trained, and through a certain voting strategy, a strong learner is finally formed, resulting in the text classification model with the best performance.

[0141] Then, the intent of the text can be classified using the trained text classification model.

[0142] This embodiment, through the above-described scheme, specifically obtains the text classification model by performing model prediction and ensemble learning training based on a preset model, encoder, several adaptive adapters, mask templates, and optimizer. Classifying input text data using the trained text classification model solves the problem of requiring large amounts of data annotation, improves the stability of model training, and avoids the waste of training resources.

[0143] Furthermore, referring to Figure 4 , Figure 4 This is a schematic diagram illustrating the process of constructing an initial classification model in the text classification method embodiment of the present invention, which involves model prediction and ensemble learning training based on an encoder, several adaptive adapters, a mask template, and an optimizer to obtain the text classification model.

[0144] Based on the above Figure 3The embodiment described herein further includes model construction, which may include:

[0145] Step S1001: Obtain pre-processed text sample data;

[0146] Step S1002 involves segmenting the text sample data into words and mapping them to word granularity IDs, then inputting these IDs into the encoder to extract local features.

[0147] Step S1003: Construct an initial classification model based on the preset model, local features, several adaptive adapters, and the MASK template;

[0148] Compared to the above Figure 3 The embodiment described herein also includes a scheme for constructing a text classification model.

[0149] Specifically, refer to Figure 4 As shown, step S1001, obtaining pre-processed text sample data may include:

[0150] Upon receiving financial audit sample data to be processed, interference and noise in the data are removed through filtering and screening operations, and more training data that meets the conditions is constructed through data augmentation.

[0151] Furthermore, referring to Figure 4 As shown, step S1002 involves inputting the text sample data, which has been segmented and mapped to word granularity IDs, into the encoder. Extracting local features may include:

[0152] The text sample data is used to output features using a preset pre-trained model. The encoder performs fully connected and convolutional operations on the input word vector encoding to extract local features.

[0153] Specifically, the preprocessed text information is segmented and mapped into word-granular IDs, and then input into a pre-trained transformer encoder model. For Chinese corpus scenarios, different pre-trained models such as Nezha or Electra are used for feature output. The encoder performs non-linear operations such as fully connected and convolution on the input word vector encoding to extract local features.

[0154] Furthermore, referring to Figure 4 As shown, step S1003, constructing an initial classification model based on the preset template, local features, several adaptive adapters, and the MASK template, may include:

[0155] The aforementioned adaptive adapters are added to a preset model to generate an adapter layer;

[0156] Based on the preset model, a MASK template is generated using the adapter layer and text sample data;

[0157] An initial classification model is obtained based on the MASK template.

[0158] The process of adding the plurality of adaptive adapters to a preset model to generate an adapter layer may include: adding the plurality of adaptive adapters to the preset model, processing the text sample data through the adaptive adapters to obtain the dimensionality-reduced text sample data parameter quantity; and restoring the dimensionality of the dimensionality-reduced text sample data parameter quantity through a neural network within the preset model to obtain the adapter layer.

[0159] Specifically, such as Figure 5 As shown, several adaptive adapters are added to the original BERT model. These adapters reduce the dimensionality of the input text sample data through fully connected layers, further decreasing the number of parameters. The data is then processed by the neural network within the BERT model to restore the dimensionality, forming a bottleneck structure—the adapter layer. During model fine-tuning, the parameters of the original pre-trained BERT model are fixed, and only the adapter parameters are updated, significantly reducing the number of parameters that need to be updated and saved during model fine-tuning.

[0160] The process of generating a MASK template based on a preset model, using the adapter layer and text sample data, and obtaining an initial classification model based on the MASK template includes: learning the implicit description of each text sample data through the adapter layer in the preset model to generate an adaptive task template; concatenating several hidden word tokens after the text sample data; processing the several hidden word tokens through an encoder to obtain the hidden state representation of each hidden word token; and obtaining the MASK template based on the adaptive template and the hidden state representations. The step of obtaining the initial classification model based on the MASK template includes: calculating the inner product of each hidden state representation and the corresponding label vector using the MASK template based on a pre-defined label vector to obtain the initial model classification result.

[0161] Specifically, the generation of MASK templates mainly includes adaptive task template generation and hidden state representation of the Mask token.

[0162] First, we directly utilize the adapter layer unique to each text classification task, allowing it to automatically learn the implicit descriptions for each task and generate adaptive task templates. This approach reduces reliance on manual design and is more stable. During training, other parameters of the language model are fixed, and only the adaptive task template part is fine-tuned, resulting in lower resource consumption.

[0163] Subsequently, the mask token hidden state representation refers to appending multiple hidden words to each input, and performing a multi-class classification task for each hidden word. Specifically, the input text is appended with the hidden word token, and after passing through a transformer, each hidden word token yields a hidden state representation. Here, the label embedding is a K*M*H tensor, where K represents the number of classes, M represents the number of tokens used for each class, and H represents the dimension of each label embedding. The hidden state representation of each hidden word token and the corresponding label embedding are multiplied by an inner product, such as... Figure 6 As shown, in the embedding layer, where each input is followed by multiple hidden word tokens, the adaptive adapter added to the BERT model is used. Through the multi-head attention mechanism, the parameters in the adaptive adapter and the parameters of the standard layer are analyzed to obtain the K-classification result calculated for each token.

[0164] Finally, through a feed-forward approach, the hidden state representation is shown as the classification result of the current model.

[0165] This embodiment, through the above-described scheme, specifically generates an adapter layer by adding several adaptive adapters to a preset model; based on the preset model, a mask template is generated using the adapter layer and text sample data; and an initial classification model is obtained based on the mask template. Compared to existing manual design methods, adding adaptive adapters to the BERT model is more sensitive and stable than manual design. During training, other parameters of the language model are fixed, and only the adapter part is fine-tuned, resulting in lower resource consumption. Adding the mask template makes the classification results more accurate.

[0166] Reference Figure 7 , Figure 7 This is a schematic diagram illustrating the process of obtaining the text classification model based on an encoder, several adaptive adapters, a mask template, and an optimizer through model prediction and ensemble learning training in an embodiment of the text classification method of the present invention.

[0167] Based on the above Figure 4 The embodiment described herein further includes, after model construction, training the model, which may include:

[0168] Step S1004: Train the initial classification model using the optimizer and loss function, minimize the loss function, obtain the model parameter values, and obtain the trained classification model.

[0169] Step S1005: Predict samples using the trained classification model to obtain the model classification result;

[0170] Step S1006: Based on the classification results of several trained classification models, the final text classification model is selected through ensemble learning and a preset voting strategy.

[0171] Compared to the above Figure 4 The embodiment described herein also includes a training scheme for the text classification model.

[0172] Reference Figure 7 Step S1004, which involves training the initial classification model using the optimizer and loss function, minimizing the loss function, obtaining model parameter values, and acquiring the trained classification model, may include:

[0173] Based on the classification results calculated by the initial classification model, the difference between the scores of several hidden word tokens concatenated after the text sample data and the classification results obtained by several initial models and the true results is calculated by using a loss function, and an optimized loss function is obtained.

[0174] By combining the optimized loss function with an optimizer, gradient descent is used to iteratively solve the problem, resulting in the minimized loss function and model parameter values, thus obtaining the trained classification model.

[0175] Specifically, such as Figure 5 The classification results calculated using the MASK template are shown below. Figure 6 As shown, a loss function is used to calculate the difference between the score of each token at each position and the true label. Focal Loss, as a loss function, reduces the weight of easily classified samples and increases the weight of difficult-to-classify samples. Its scaling factor dynamically adjusts the weight of easy samples during training and allows the model to quickly focus on difficult samples. When a sample is misclassified, the predicted probability pt of the label tends to 0, and the modulation factor tends to 1, making the loss function almost unaffected. On the other hand, if the sample is correctly classified, pt will tend to 1, and the modulation factor will tend to 0, making the loss very close to 0, thus reducing the weight of that particular sample.

[0176] Furthermore, such as Figure 5 As shown, the model is trained using the Adam optimizer combined with an optimized loss function and gradient descent. To minimize the loss function, gradient descent is used iteratively step-by-step to obtain the minimized loss function and model parameter values.

[0177] Furthermore, referring to Figure 7Step S1005, which involves predicting samples using the trained classification model to obtain the model classification result, may include:

[0178] Obtain training samples;

[0179] The training samples are input into the trained classification model for prediction.

[0180] Based on the preset label vector ， Calculate the vector value generated by each hidden word token in the training sample and the average vector value of all hidden word tokens at the current label position in any label of the training sample;

[0181] The position of the current classification result is measured based on the similarity between the vector value generated by each hidden word token and the average vector value of the hidden word tokens at the current label position.

[0182] The final model classification result is obtained by merging the positions of each classification result.

[0183] Specifically, such as Figure 5 As shown, a small sample of data is obtained as training data to predict the model. The method compares the representation of the current training sample with the average representation of all labeled samples in the training data, fusing hidden word tokens from multiple positions to obtain the final classification result. For each training sample, based on the previously predetermined label vector, the model generates M hidden word tokens. The similarity between the vector generated by each hidden word token and the average vector of hidden word tokens from all training samples at that position for a given label is calculated. This similarity serves as a measure of the distance between the current sample and the label at that position. Finally, the results from each position are fused to obtain the final label, which is the model's classification result.

[0184] Furthermore, referring to Figure 7 Step S1006, based on the classification results of several trained classification models, and through ensemble learning and a preset voting strategy, selecting the final text classification model may include:

[0185] Based on the classification results of the several trained classification models, several text classification models are trained using an ensemble learning strategy. The text classification model with the best performance is selected as the final text classification model through a preset voting strategy.

[0186] Specifically, such as Figure 5 As shown, based on the predicted model, several weak learners are trained simultaneously using the idea of ​​ensemble learning. Through a certain voting strategy, a strong learner can be formed in the end, and finally the best-performing strong learning model is used to output the result.

[0187] This embodiment, through the above-described scheme, specifically trains the initial classification model using the optimizer and loss function, minimizes the loss function to obtain model parameter values, and obtains the trained classification model; uses the trained classification model to predict samples and obtain the model classification result; based on the model classification results of several trained classification models, the final text classification model is selected through ensemble learning and a preset voting strategy. By training the text classification model using the loss function, Adam optimizer, and ensemble learning strategy to obtain the best-performing text classification model, and then using the text classification model to output the results, a more accurate text classification result can be obtained.

[0188] Furthermore, embodiments of the present invention also propose a text classification device, the text classification device comprising:

[0189] The text acquisition module is used to acquire the text to be processed.

[0190] The text classification module is used to classify the text to be processed into a pre-created text classification model to obtain classified text data. The text classification model includes an encoder, several adaptive adapters, a preset template, an optimizer, and a model prediction. The text classification model is trained based on the encoder, several adaptive adapters, the preset template, the optimizer, the model prediction, and ensemble learning.

[0191] Furthermore, the text classification device also includes:

[0192] The model training module is used to perform model prediction and ensemble learning training based on the encoder, several adaptive adapters, preset templates, and optimizers to obtain the text classification model.

[0193] The principle and implementation process of text classification in this embodiment are explained in the above embodiments and will not be repeated here.

[0194] Furthermore, this embodiment of the invention also proposes a terminal device, which includes a memory, a processor, and a text classification program stored in the memory and executable on the processor. When the text classification program is executed by the processor, it implements the steps of the text classification method as described above.

[0195] Since this text classification program employs all the technical solutions of all the foregoing embodiments when executed by the processor, it has at least all the beneficial effects brought about by all the technical solutions of all the foregoing embodiments, which will not be repeated here.

[0196] Furthermore, embodiments of the present invention also propose a computer-readable storage medium storing a text classification program, which, when executed by a processor, implements the steps of the text classification method as described above.

[0197] Since this text classification program employs all the technical solutions of all the foregoing embodiments when executed by the processor, it has at least all the beneficial effects brought about by all the technical solutions of all the foregoing embodiments, which will not be repeated here.

[0198] Compared to existing technologies, the text classification method, apparatus, terminal device, and storage medium proposed in this invention acquire the text to be processed; input the text to be processed into a pre-constructed text classification model for processing to obtain classified text data. The text classification model includes an encoder, several adaptive adapters, and a mask template. The text classification model is trained through model prediction and ensemble learning based on the encoder, several adaptive adapters, the mask template, and an optimizer. Classifying text using the trained text classification model solves the problem of requiring a large amount of labeled data during text classification, improves the model's training stability, and reduces training resource consumption. Based on this invention, starting from the complexity and difficulty of labeling financial audit text data, a text classification model is constructed, and the effectiveness of the proposed text classification method is verified on text sample data. Finally, the text classification model obtained through this invention reduces the number of parameters that need to be updated and saved during the model fine-tuning stage, improves the model's training stability, and reduces the computational and storage resources required for training.

[0199] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0200] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0201] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, controlled terminal, or network device, etc.) to execute the methods of each embodiment of the present invention.

[0202] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A text classification method, characterized in that, The text classification method includes the following steps: Get the text to be processed; The text to be processed is input into a pre-built text classification model for processing to obtain classified text data. The text classification model includes a preset model, an encoder, several adaptive adapters, and a mask template. The text classification model is obtained through model prediction and ensemble learning training based on the preset model, encoder, several adaptive adapters, mask template, and optimizer. Specifically, an initial classification model is constructed based on the preset model, encoder, several adaptive adapters, and mask template. The text classification model is then obtained through model prediction and ensemble learning training based on the initial classification model and optimizer. The construction of the initial classification model based on the preset model, encoder, several adaptive adapters, and mask template includes: The aforementioned adaptive adapters are added to a preset model to generate an adapter layer; By using the adapter layer in the preset model, the implicit description of each pre-processed text sample data is learned, and an adaptive task template is generated. Several hidden word tokens are appended to the end of the text sample data; The encoder processes the several hidden word tokens to obtain the hidden state representation of each hidden word token; Based on the adaptive task template and the hidden state representation, obtain the MASK template; Based on the pre-defined label vectors, the inner product of each hidden state representation and the corresponding label vector is calculated using the MASK template to obtain the initial classification model.

2. The text classification method according to claim 1, characterized in that, Before the step of processing the text to be processed by a pre-built text classification model to obtain classified text data, the method further includes: The text classification model is obtained by performing model prediction and ensemble learning training based on a preset model, encoder, several adaptive adapters, mask templates, and optimizer.

3. The text classification method according to claim 2, characterized in that, The steps for obtaining the text classification model through model prediction and ensemble learning training based on a preset model, encoder, several adaptive adapters, mask templates, and optimizer include: Obtain pre-processed text sample data; The text sample data is segmented and mapped into word granular IDs, which are then input into the encoder to extract local features; An initial classification model is constructed based on the preset model, local features, several adaptive adapters, and the MASK template. The initial classification model is trained using the optimizer and loss function, the loss function is minimized, the model parameter values ​​are obtained, and the trained classification model is obtained. The trained classification model is used to predict the sample classification result. Based on the classification results of several trained classification models, the final text classification model is selected through ensemble learning and a pre-set voting strategy.

4. The text classification method according to claim 3, characterized in that, The step of segmenting and mapping the text sample data into word granular IDs and inputting them into the encoder to extract local features includes: The text sample data is used to output features using a preset pre-training module. The encoder performs fully connected and convolutional operations on the input word vector encoding to extract local features.

5. The text classification method according to claim 3, characterized in that, The steps of training the initial classification model using the optimizer and loss function, minimizing the loss function, obtaining model parameter values, and obtaining the trained classification model include: Based on the classification results calculated by the initial classification model, the difference between the scores of several hidden word tokens concatenated after the text sample data and the classification results obtained by several initial models and the true results is calculated by using a loss function, and an optimized loss function is obtained. By combining the optimized loss function with an optimizer, gradient descent is used to iteratively solve the problem, resulting in the minimized loss function and model parameter values, thus obtaining the trained classification model.

6. The text classification method according to claim 3, characterized in that, The steps of predicting samples using the trained classification model and obtaining the model classification result include: Obtain training samples; The training samples are input into the trained classification model for prediction. Based on the preset label vector, calculate the vector value generated by each hidden word token in the training sample and the average vector value of all hidden word tokens at the current label position in any label of the training sample; The position of the current classification result is measured based on the similarity between the vector value generated by each hidden word token and the average vector value of the hidden word tokens at the current label position. The final model classification result is obtained by merging the positions of each classification result.

7. The text classification method according to claim 3, characterized in that, The steps for selecting the final text classification model based on the classification results of several trained classification models, through ensemble learning and a preset voting strategy, include: Based on the classification results of the several trained classification models, several text classification models are trained using an ensemble learning strategy. The text classification model with the best performance is selected as the final text classification model through a preset voting strategy.

8. The text classification method according to claim 3, characterized in that, The step of adding the plurality of adaptive adapters to the preset model to obtain the adapter layer includes: The aforementioned adaptive adapters are added to the preset model, and the local features are processed through the adaptive adapters to obtain the number of local feature parameters after dimensionality reduction. The adapter layer is obtained by restoring the dimensionality of the local feature parameters after dimensionality reduction through the neural network inside the preset model.

9. A text classification device, characterized in that, The text classification device includes: The text acquisition module is used to acquire the text to be processed. The text classification module is used to classify the text to be processed into a pre-created text classification model to obtain classified text data. The text classification model includes an encoder, several adaptive adapters, a preset model, an optimizer, and a model prediction. The text classification model is obtained by model prediction and ensemble learning training based on the encoder, several adaptive adapters, the preset model, and the optimizer. The text classification module is further configured to construct an initial classification model based on the preset model, encoder, several adaptive adapters, and MASK template, and to perform model prediction and ensemble learning training based on the initial classification model and optimizer to obtain a text classification model; wherein, constructing the initial classification model based on the preset model, encoder, several adaptive adapters, and MASK template includes: The aforementioned adaptive adapters are added to a preset model to generate an adapter layer; By using the adapter layer in the preset model, the implicit description of each pre-processed text sample data is learned, and an adaptive task template is generated. Several hidden word tokens are appended to the end of the text sample data; The encoder processes the several hidden word tokens to obtain the hidden state representation of each hidden word token; Based on the adaptive task template and the hidden state representation, obtain the MASK template; Based on the pre-defined label vectors, the inner product of each hidden state representation and the corresponding label vector is calculated using the MASK template to obtain the initial classification model.

10. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a text classification program stored in the memory and executable on the processor. When the text classification program is executed by the processor, it implements the steps of the text classification method as described in any one of claims 1-8.

11. A calculator-readable storage medium, characterized in that, The computer-readable storage medium stores a text classification program, which, when executed by a processor, implements the steps of the text classification method as described in any one of claims 1-8.

Images