Text classification model training method, recognition method and system

Abstract

The invention provides a text classification model training method and system and a text classification model recognition method and system.The training method comprises the steps that a marked text data set is obtained, and the text data set is preprocessed to obtain a text training set; performing structural feature extraction on the text training set to obtain a text structural feature set; respectively inputting the text training sets into different large language frameworks to obtain text abstract feature sets and text content feature sets output by the different large language frameworks; according to the text content feature set, performing multiple rounds of attention fusion on the text abstract feature set to obtain multiple rounds of fusion feature sets; and performing parameter updating on the initialized text classification model based on the text label, the text structure feature set and the multi-round fusion feature set in the text data set to obtain a trained text classification model. According to the text classification model obtained by training through the training method, the accuracy of text data classification can be effectively improved. The invention relates to the technical field of natural language processing.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a training method, a recognition method and a system for a text classification model. Background Art

[0002] With the rapid development of information technology, the scale of text data generated by platforms such as the Internet, social media and online education is becoming increasingly large, and text data classification has become one of the key technical contents that people are paying increasing attention to.

[0003] At present, traditional text data classification methods usually classify text data based on a single text representation or feature. However, due to the complexity and diversity of text data, this method often cannot comprehensively capture the diverse and complex feature representations of text data, and the accuracy of text data classification is unsatisfactory.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0005] The purpose of the present invention is to solve one of the technical problems existing in the related art to at least a certain extent.

[0006] The main purpose of the embodiments of the present application is to propose a training method for a text classification model. The text classification model trained by the training method can effectively improve the accuracy of text data classification.

[0007] To achieve the above purpose, an embodiment of the present application provides a training method for a text classification model, the training method comprising:

[0008] Obtaining a labeled text data set, and preprocessing the text data set to obtain a text training set;

[0009] Extracting structural features from the text training set to obtain a text structural feature set;

[0010] Inputting the text training set into the first large language framework for training, obtaining the first large language framework trained by local fine-tuning, and a text summary feature set output by the first large language framework trained by local fine-tuning;

[0011] Inputting the text training set into the second largest language framework for training, obtaining the second largest language framework trained by local fine-tuning, and a text content feature set output by the second largest language framework trained by local fine-tuning;

[0012] According to the text content feature set, performing multiple rounds of attention fusion on the text summary feature set to obtain a multiple round fusion feature set;

[0013] Based on the text labels in the text data set, the text structure feature set and the multi-round fusion feature set, the parameters of the initialized text classification model are updated to obtain a trained text classification model.

[0014] In some embodiments, the preprocessing of the text data set to obtain a text training set includes:

[0015] Performing a data integrity check on the text data set to obtain a first intermediate data set;

[0016] Deduplication of the first intermediate data set is performed to obtain a second intermediate data set;

[0017] The second intermediate data set is subjected to text content purification processing to obtain the text training set.

[0018] In some embodiments, the text training set is input into a large language framework to obtain a large language framework trained by local fine-tuning, including:

[0019] Acquire a target data set, wherein the target data set is a text summary set or a text content set corresponding to the text training set;

[0020] Inputting the target data set into a pre-trained language model to perform text segmentation to obtain basic text units;

[0021] Inputting the text basic unit into the quantized low-rank adapter and the pre-trained language model for training to obtain feature data;

[0022] Determining a quantized low-rank loss value according to a text label corresponding to the target data set and the feature data;

[0023] According to the quantized low-rank loss value, updating the parameters of the quantized low-rank adapter to obtain a trained quantized low-rank adapter;

[0024] According to the pre-trained language model and the trained quantized low-rank adapter, the locally fine-tuned trained large language framework is obtained.

[0025] In some embodiments, performing multiple rounds of attention fusion on the text summary feature set according to the text content feature set to obtain a multiple round fusion feature set includes:

[0026] Adding and performing a first multi-head attention process on the text content feature set and the text summary feature set to obtain a first fusion weight set;

[0027] According to the first fusion weight set, weighted fusion is performed on the text content feature set and the text summary feature set to obtain an intermediate fusion feature set;

[0028] According to the text content feature set and the text summary feature set, the intermediate fusion feature set is subjected to attention-aware fusion to obtain the multi-round fusion feature set.

[0029] In some embodiments, performing attention-aware fusion on the intermediate fusion feature set according to the text content feature set and the text summary feature set to obtain the multi-round fusion feature set includes:

[0030] Performing a second multi-head attention process on the intermediate fusion feature set to obtain a second fusion weight set;

[0031] According to the second fusion weight set, the text content feature set and the text summary feature set are completely perceptually fused to obtain the multi-round fusion feature set.

[0032] In some embodiments, the updating of parameters of the initialized text classification model based on the text labels in the text data set, the text structure feature set and the multi-round fusion feature set to obtain a trained text classification model includes:

[0033] According to the text structure feature set, feature concatenation is performed on the multiple rounds of fusion feature sets to obtain a target feature set;

[0034] Performing classification prediction on the target feature set to obtain a classification prediction result;

[0035] Determining a target loss value based on the classification prediction result and the text labels in the text dataset;

[0036] Based on the target loss value, the initialized text classification model is trained and updated to obtain the trained text classification model.

[0037] To achieve the above purpose, another aspect of the embodiment of the present application proposes a method for identifying a text classification model, including:

[0038] Get the text data to be classified;

[0039] The text data is input into the trained text classification model to obtain the text classification result.

[0040] To achieve the above purpose, another aspect of the embodiment of the present application provides a training system for a text classification model, including:

[0041] A first processing unit is used to obtain a labeled text data set and preprocess the text data set to obtain a text training set;

[0042] A second processing unit is used to extract structural features from the text training set to obtain a text structural feature set;

[0043] A third processing unit is used to input the text training set into the first large language framework for training, to obtain the first large language framework trained by local fine-tuning, and a text summary feature set output by the first large language framework trained by local fine-tuning;

[0044] A fourth processing unit is used to input the text training set into the second largest language framework for training, to obtain the second largest language framework trained by local fine-tuning, and a text content feature set output by the second largest language framework trained by local fine-tuning;

[0045] A fifth processing unit, configured to perform multiple rounds of attention fusion on the text summary feature set according to the text content feature set to obtain a multiple rounds of fusion feature set;

[0046] The sixth processing unit is used to update the parameters of the initialized text classification model based on the text labels in the text data set, the text structure feature set and the multi-round fusion feature set to obtain a trained text classification model.

[0047] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0048] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0049] The embodiments of the present application include at least the following beneficial effects:

[0050] The present application provides a training method, a recognition method and a system for a text classification model, wherein the training method obtains a labeled text data set, and pre-processes the text data set to obtain a text training set; extracts structural features from the text training set to obtain a text structural feature set; inputs the text training set into a first large language framework for training to obtain the first large language framework trained by local fine-tuning, and a text summary feature set output by the first large language framework trained by local fine-tuning; inputs the text training set into a second large language framework for training to obtain the second large language framework trained by local fine-tuning, and a text content feature set output by the second large language framework trained by local fine-tuning; performs multi-round attention fusion on the text summary feature set according to the text content feature set to obtain a multi-round fusion feature set; and updates parameters of an initialized text classification model based on text labels in the text data set, the text structural feature set and the multi-round fusion feature set to obtain a trained text classification model. This training method extracts the content features and summary features of text data respectively through different large language frameworks, which can achieve in-depth analysis of the text content and more comprehensively identify and understand the semantic information, grammatical structure and style characteristics in the text data. In addition, this training method also analyzes the text data from multiple perspectives (such as the text summary perspective and the text content perspective) based on multi-round attention fusion, which can enhance the weights of key features in the text data and suppress noise interference, effectively improving the accuracy and generalization ability of the text classification model in classifying and predicting text data. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a flowchart of a training method for a text classification model provided in an embodiment of the present application;

[0052] Figure 2 is a detailed flow chart of step S110 provided in an embodiment of the present application;

[0053] Figure 3 It is a detailed flow chart about a large language framework provided by an embodiment of the present application;

[0054] Figure 4 is a detailed flow chart of step S150 provided in an embodiment of the present application;

[0055] Figure 5 is a logic diagram of step S150 provided in an embodiment of the present application;

[0056] Figure 6 is a detailed flow chart of step C3 provided in an embodiment of the present application;

[0057] Figure 7is a detailed flow chart of step S160 provided in an embodiment of the present application;

[0058] Figure 8 It is a flowchart of a text classification model recognition method provided in an embodiment of the present application;

[0059] Fig. 9 It is a schematic diagram of a framework of a text classification model training system provided in an embodiment of the present application;

[0060] Fig.10 It is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices / equipment and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0062] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0063] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0065] At present, traditional text data classification methods usually classify text data based on a single text representation or feature. However, due to the complexity and diversity of text data, and the fact that simply superimposing the features of text data may cause the omission of important information and amplify noise, this method often cannot comprehensively capture the diverse and complex feature representations of text data, and the accuracy of text data classification is unsatisfactory.

[0066] In view of this, the embodiments of the present application provide a training method, a recognition method and a system for a text classification model. The training method extracts content features and summary features of text data respectively through different large language frameworks, which can achieve in-depth analysis of the text content and can more comprehensively identify and understand the semantic information, grammatical structure and style characteristics in the text data. In addition, the training method also analyzes the text data based on multi-round attention fusion, and can analyze the text data from multiple angles and levels, so that the weights of key features in the text data are enhanced and noise interference is suppressed, and parallel learning of global features and local features of text data is achieved, effectively improving the accuracy and generalization ability of the text classification model in classifying and predicting text data.

[0067] The training method and recognition method of a text classification model provided in the embodiment of the present application can be applied to text classification application scenarios. In the text classification application scenario, the text classification service provider can train the text classification model through the training method provided in the embodiment of the present application, and classify the text data provided by the user based on the trained text classification model, so as to obtain a text data category or text score with higher accuracy.

[0068] A training method and recognition method for a text classification model provided in an embodiment of the present application can be applied to a terminal, a server, or software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the method, etc., but is not limited to the above forms.

[0069] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0070] Reference Figure 1 , Figure 1 is an optional flow chart of a text classification model training method provided in an embodiment of the present application. Figure 1 The training method may include but is not limited to steps S110 to S160.

[0071] Step S110, obtaining a labeled text data set, and preprocessing the text data set to obtain a text training set;

[0072] In an embodiment of the present application, the annotated text data set can be a text classification data set publicly available on the Internet, which includes a number of text data, each of which includes key fields such as text ID, text content, and text label, wherein the text ID is the unique identifier of the text data, the text content is the text to be classified and evaluated, and the text label is the score or category of the text content.

[0073] Reference Figure 2 In some embodiments, the step S110 of preprocessing the text data set to obtain a text training set includes:

[0074] A1. Performing a data integrity check on the text data set to obtain a first intermediate data set;

[0075] A2. Deduplication of the first intermediate data set is performed to obtain a second intermediate data set;

[0076] A3. Perform text content purification on the second intermediate data set to obtain the text training set.

[0077] It is understandable that the data integrity check in step A1 can be to check whether each text data in the text data set contains missing values, and delete the samples (i.e., text data) containing missing values, so as to obtain the first intermediate data set. Then, step A2 can be based on the text ID of each text data, and perform deduplication processing on the text data set, and remove duplicate samples, so as to obtain the second intermediate data set, and the second intermediate data set can ensure that each text ID appears only once in the data set, thereby avoiding data redundancy and deviation of the text classification model during the training process.

[0078] It should be noted that the text content purification process in step A3 can be to purify the text content of each text data of the second intermediate data set, which can specifically be to remove hypertext tags, special characters and irrelevant format information in the text content to reduce potential interference with the performance of the text classification model. At the same time, the label column composed of the text labels of all text data is verified and adjusted to ensure that the text label of each text data belongs to a predefined text type or text rating level, to ensure the standardization of the text labels, and to improve the accuracy and generalization ability of the model.

[0079] Step S120, extracting structural features from the text training set to obtain a text structural feature set;

[0080] In the embodiment of the present application, the structural feature extraction in step S120 may be to extract the organizational structural features of each text data in the text training set. Specifically, for a certain text data. Structural feature extraction can specifically include paragraph feature extraction, which evaluates the organization and fluency of a text by counting the number of natural paragraphs in the text; word count, which reflects the length and amount of information in a text by counting the total number of words in the text; stop word analysis, which determines the common words that contribute less to the meaning of the text by counting the number of stop words in the text; lexical diversity analysis, which measures the lexical richness of a text by counting the number of different words after removing stop words; spelling check, which identifies the number of misspelled words that may exist in the text; sentence feature extraction, which evaluates the complexity of a text and the level of detail of expression by counting the number of sentences in the text and the average length of sentences; punctuation analysis, which reflects the grammatical structure and diversity of expression of a text by counting the number of punctuation marks in the text; part-of-speech distribution analysis, which analyzes the grammatical structure and style characteristics of a text by counting the number of nouns, articles, prepositions, adjectives, adverbs, verbs and conjunctions respectively; and transition word statistics, which identifies and calculates the number of transition words in a text, which are usually used to express a change or emphasis in a point of view.

[0081] It is understandable that the rest of the text data is similar and can be simply deduced by referring to the above content. In addition, based on the structural feature extraction of each text data in the text training set, the embodiment of the present application can provide a more comprehensive text feature for the text classification model to improve the classification and evaluation ability of the text classification model for text structure and content instructions.

[0082] Exemplarily, in the embodiment of the present application, the organizational structure characteristics of the text can be specifically extracted by quantitative statistical methods, specifically by regular expressions, to count basic features, including calculating the number of natural paragraphs, the total number of words, the number of stop words, the average length of words and other basic indicators of the text, so as to obtain a preliminary understanding of the scale and composition of the text; then, the vocabulary diversity analysis can be performed by using Python's collection dependency library, and the number of different words can be counted on the basis of removing stop words to evaluate the vocabulary richness of the text; in addition, the natural language processing toolkit pyspellchecker is used to identify the number of misspelled words to evaluate the quality of the text; and for sentence feature extraction, it can use regular expressions to count the number of sentences in the text and calculate the average length of the sentences to evaluate the complexity of the sentence structure; for punctuation analysis, it can use the string library to implement punctuation statistics, and reflect the grammatical details and expression diversity of the text by calculating the number of punctuation marks; for grammatical component statistics, the natural language processing toolkit nltk can be used to count grammatical components such as nouns, articles, prepositions, adjectives, adverbs, verbs, conjunctions and transition words in the text to deeply analyze the grammatical structure of the text.

[0083] Step S130: input the text training set into the first large language framework for training, to obtain the first large language framework trained by local fine-tuning, and a text summary feature set output by the first large language framework trained by local fine-tuning;

[0084] Reference Figure 3 In some embodiments, the text training set is input into a large language framework to obtain a large language framework trained by local fine-tuning, including:

[0085] B1. Obtain a target data set, where the target data set is a text summary set or a text content set corresponding to the text training set;

[0086] B2. Inputting the target data set into a pre-trained language model to perform text segmentation to obtain basic text units;

[0087] B3, inputting the text basic unit into the quantized low-rank adapter and the pre-trained language model for training to obtain feature data;

[0088] B4. Determine a quantized low-rank loss value according to the text label corresponding to the target data set and the feature data;

[0089] B5. According to the quantized low-rank loss value, update the parameters of the quantized low-rank adapter to obtain a trained quantized low-rank adapter;

[0090] B6. According to the pre-trained language model and the trained quantized low-rank adapter, the locally fine-tuned trained large language framework is obtained.

[0091] In the embodiment of the present application, the pre-trained language model can be any large language model (LLM), and the present application does not limit the specific large language model used. In addition, if the large language framework outputs a text summary feature set, the large language framework is the first large language framework, which is used to capture the core information of each text data in the text training set and provide a refined representation of the key content of the text for the text classification model.

[0092] It can be understood that if the target data set of step B1 is a text summary set, the text summary set can specifically extract the core information of the text data of the text training set based on the Bart-large-CNN open source large language model, and then match and combine the extracted text summary with the corresponding text label. The corresponding text label can be determined by the text ID of the text data, so as to obtain the text summary set; step B2 can be to input the text summary set into a pre-trained language model, and output the text basic unit corresponding to the text summary set through the word segmentation function provided by the pre-trained language model. The text basic unit can be any one of the digital IDs, special symbols, etc. corresponding to the words in the text summary set in the pre-trained language model dictionary, and this application is not limited here.

[0093] It should be noted that step B3 can be inputting the basic text units corresponding to the text summary set into the quantized low-rank adapter and the pre-trained language model. Specifically, the quantized low-rank adapter and the pre-trained language model can be trained according to the embedding matrix corresponding to the basic text unit, so as to obtain the feature data output by the quantized low-rank adapter and the pre-trained language model. Specifically, the quantized address adapter can be a weight matrix parallel to the pre-trained language model parameter matrix, and the weight matrix can be further split into a dimensionality reduction matrix and a dimensionality increase matrix based on the low rank. In addition, when the text summary set is first input into the pre-trained language model and the quantized low-rank adapter, the dimensionality reduction matrix can be obtained by initializing the Gaussian function, and the dimensionality increase matrix can be an all-zero matrix.

[0094] It is worth mentioning that after obtaining the feature data, step B4 can calculate the quantized low-rank loss value based on the feature data and the text label, and then step B5 can update the parameters of the quantized low-rank adapter based on the calculated quantized low-rank loss value, while the original parameters of the pre-trained language model remain unchanged, thereby obtaining a trained quantized low-rank adapter.

[0095] Specifically, step B5 can update the relevant parameters of the quantized low-rank adapter through the back-propagation algorithm, thereby improving the performance of the quantized low-rank adapter. In an embodiment of the present application, the training method can be performed in a cyclic iterative manner, that is, the parameters of the quantized low-rank adapter can be updated round by round. Exemplarily, for example, multiple text summary sets can be obtained, and then these text summary sets are divided into different training data groups. First, the data of one training data group is used to calculate the corresponding quantized low-rank loss value, and the parameters of one round of quantized low-rank adapter are updated. Then, based on the pre-trained language model and the updated quantized low-rank adapter, the data of another training data group is used to calculate the corresponding quantized low-rank loss value, and the parameters of one round of quantized low-rank adapter are continued to be updated. In this way, a trained quantized low-rank adapter is obtained by iterating in a cycle. Here, the conditions for determining the end of training can be set according to actual needs. For example, in some embodiments, the target round of training iterations can be set, and the iteration rounds can be recorded during the training process. The parameters of the quantized low-rank adapter are updated once each time, and the iteration rounds can be increased by 1. If the iteration round reaches the target round after the parameters of the quantized low-rank adapter are updated, the training can be considered to be completed and a trained quantized low-rank adapter is obtained; if the iteration round does not reach the target round after the parameters of the quantized low-rank adapter are updated, iterative training can continue. In some embodiments, a test data set can also be obtained, and the prediction accuracy of the first large language framework is detected by the test data set. The parameters of the quantized low-rank adapter are updated once each time, and the test data set is used for detection once. If the prediction accuracy obtained by the test meets the pre-set index requirements after the parameters of the quantized low-rank adapter are updated, the training can be considered to be completed and a trained quantized low-rank adapter is obtained; if the prediction accuracy obtained by the test does not meet the pre-set index requirements after the parameters of the quantized low-rank adapter are updated, iterative training can continue.

[0096] Step B6 can be to combine the parameter matrix of the pre-trained language model, and the low-dimensional matrix and high-dimensional matrix of the quantized low-rank adapter to obtain the first language framework that has been locally fine-tuned and trained. There are many specific combination methods, which will not be elaborated in this application.

[0097] It is easy to understand that the embodiment of the present application achieves fine-tuning training of the pre-trained language model by training and updating the quantized low-rank adapter, and then obtains the first large language framework that has been locally fine-tuned and trained. It is not limited to adjusting the output parameters of the last layer of the pre-trained language model, but can adjust more model parameters according to the input data set, and has better adaptability. Compared with the fine-tuning of all parameters of the pre-trained language model, it can effectively reduce the required computing resources, and the fine-tuning training efficiency is higher.

[0098] Step S140: input the text training set into the second largest language framework for training, to obtain the second largest language framework trained by local fine-tuning, and the text content feature set output by the second largest language framework trained by local fine-tuning;

[0099] In the embodiment of the present application, the content of step S140 is similar to the content of the aforementioned step S130, and can be simply deduced. Specifically, if the target data set of step B1 is a text content set, the large language framework is the second largest language framework, and the second largest language framework is used to extract deep features of the text content in the text content set, and its output is a text content feature set, which is not elaborated in this application.

[0100] Reference Figure 4 and Figure 5 , step S150, performing multiple rounds of attention fusion on the text summary feature set according to the text content feature set to obtain a multiple round fusion feature set;

[0101] In some embodiments, the step S150, performing multiple rounds of attention fusion on the text summary feature set according to the text content feature set to obtain a multiple rounds of fusion feature set, includes:

[0102] C1, adding the text content feature set and the text summary feature set and performing a first multi-head attention process to obtain a first fusion weight set;

[0103] C2. According to the first fusion weight set, weighted fusion is performed on the text content feature set and the text summary feature set to obtain an intermediate fusion feature set.

[0104] In the embodiment of the present application, step C1 may firstly perform a sum operation on the text content feature set and the text summary feature set to obtain an intermediate sum feature set; then, the intermediate sum feature set is input into a multi-head attention mechanism module (Multi-Head Attention, MHA) for calculation, and the calculated result is input into an activation module, specifically, the weight is output through a Sigmoid activation function to obtain a first fusion weight set; then, the first fusion weight set is respectively weighted averaged with the original input text content feature set and the text summary feature set to obtain an intermediate fusion feature set, and the equivalent expression of the intermediate fusion feature set can be:

[0105]

[0106] Among them, F text1 is the intermediate fusion feature set; F content is the text content feature set; Z is the intermediate sum feature set; sigmoid(Z) is the first fusion weight set; F summary is the text summary feature set.

[0107] C3. According to the text content feature set and the text summary feature set, the intermediate fusion feature set is subjected to attention-aware fusion to obtain the multi-round fusion feature set.

[0108] Reference Figure 6 Further, the step C3, performing attention-aware fusion on the intermediate fusion feature set according to the text content feature set and the text summary feature set to obtain the multi-round fusion feature set, includes:

[0109] C31, performing a second multi-head attention process on the intermediate fusion feature set to obtain a second fusion weight set;

[0110] C32. Perform complete perceptual fusion on the text content feature set and the text summary feature set according to the second fusion weight set to obtain the multi-round fusion feature set.

[0111] In an embodiment of the present application, the second multi-head attention processing of step C31 can be to input the intermediate fusion feature set into another multi-head attention mechanism module (Multi-Head Attention, MHA) for calculation, and input the calculated result into the activation module, specifically output the weight through the Sigmoid activation function to obtain the second fusion weight set. Step C32 can be based on the second fusion weight set, and perform a weighted average operation with the original input text content feature set text summary feature set, so as to achieve complete perception of the input features and obtain a multi-round fusion feature set.

[0112] Exemplarily, the equivalent expression of the multi-round fusion feature set can be

[0113]

[0114] Among them, Ft ext2 is the multi-round fusion feature set; Ft ext1 is the intermediate fusion feature set; Fc ontent is the text content feature set; sigmoid(F text1 ) is the second fusion weight set; F summary is the text summary feature set.

[0115] It is worth adding that the intermediate fusion feature set obtained through steps C1 and C2 is only an ordinary addition and superposition of the two input features, which may result in the omission of important information and the amplification of noise, and often fails to perceive the input features more comprehensively; and the embodiment of the present application is based on steps C31 and C32, and uses a multi-head attention mechanism to perform attention fusion on the intermediate fusion feature set, which can more comprehensively capture the diverse and complex feature representations of text data and suppress noise interference.

[0116] Step S160: Based on the text labels in the text data set, the text structure feature set and the multi-round fusion feature set, update the parameters of the initialized text classification model to obtain a trained text classification model.

[0117] Reference Figure 7 In some embodiments, the step S160, based on the text labels in the text data set, the text structure feature set and the multi-round fusion feature set, updates the parameters of the initialized text classification model to obtain a trained text classification model, includes:

[0118] D1. According to the text structure feature set, the multi-round fusion feature sets are subjected to feature splicing to obtain a target feature set;

[0119] D2. Classify and predict the target feature set to obtain a classification prediction result;

[0120] D3, determining a target loss value based on the classification prediction result and the text labels in the text data set;

[0121] D4. Based on the target loss value, the initialized text classification model is trained and updated to obtain the trained text classification model.

[0122] In an embodiment of the present application, step D1 may be to concatenate the obtained text structure feature set and the multi-round fusion feature set to obtain a target feature set representing multiple dimensions of text structure, text content and text summary, and the target feature in the target feature set may be a high-dimensional feature vector for representing various aspects of text data. Step D2 may be to input the target feature set into a classifier for classification prediction, and the classifier may be a multi-layer perceptron (MLP) to map the target features in the target feature set to predefined scoring categories, thereby obtaining classification prediction results.

[0123] It is understandable that the classification prediction accuracy of the text classification model can be measured by the consistency between the classification prediction result and the text label. The closer the classification prediction result is to the corresponding text label, the more accurate the classification prediction result provided by the text classification model is, and the better the reliability is; conversely, the closer the classification prediction result is to the corresponding text label, the less accurate the classification prediction result provided by the text classification model is, and the worse the reliability is. Therefore, in an embodiment of the present application, a loss value can be determined based on the classification prediction result and the text label, and the obtained loss value can be recorded as the target loss value. For the loss function used to determine the target loss value, any one of a 0-1 loss function, a square loss function, an absolute loss function, a logarithmic loss function, a cross entropy loss function, etc. can be used.

[0124] It should be noted that the training update in step D4 can be implemented by using the Adam back-propagation algorithm to update the parameters of the text classification model. Specifically, it can be implemented by updating the parameters of the multi-round attention fusion module that implements the multi-round attention fusion function and the classification layer of the text classification function in the text classification model. A trained text classification model can be obtained after several rounds of iterations. The specific number of iterations can be set in advance, or the training is considered to be completed when the test set meets the accuracy requirements.

[0125] Reference Figure 8 , Figure 8 is an optional flow chart of a text classification model recognition method provided in an embodiment of the present application. Figure 8 The identification method may include but is not limited to steps S170 to S180.

[0126] Step S170, obtaining text data to be classified;

[0127] Step S180: input the text data into the trained text classification model as described above to obtain a text classification result.

[0128] In an embodiment of the present application, the text data to be classified can be input into a trained text classification model, and the text classification model can be used to predict the text category to which the text data belongs or the text score of the text data to obtain a text classification result corresponding to the text data.

[0129] See also Fig. 9 The present application also provides a text classification model training system, the training system comprising:

[0130] A first processing unit 810 is used to obtain a labeled text data set and pre-process the text data set to obtain a text training set;

[0131] The second processing unit 820 is used to extract structural features from the text training set to obtain a text structural feature set;

[0132] The third processing unit 830 is used to input the text training set into the first large language framework for training, to obtain the first large language framework trained by local fine-tuning, and the text summary feature set output by the first large language framework trained by local fine-tuning;

[0133] The fourth processing unit 840 is used to input the text training set into the second largest language framework for training, to obtain the second largest language framework trained by local fine-tuning, and the text content feature set output by the second largest language framework trained by local fine-tuning;

[0134] A fifth processing unit 850 is used to perform multi-round attention fusion on the text summary feature set according to the text content feature set to obtain a multi-round fusion feature set;

[0135] The sixth processing unit 860 is used to update the parameters of the initialized text classification model based on the text labels in the text data set, the text structure feature set and the multi-round fusion feature set to obtain a trained text classification model.

[0136] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0137] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0138] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0139] See also Fig.10 , Fig.10 The hardware structure of an electronic device of an embodiment is illustrated, and the electronic device includes:

[0140] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0141] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 902, and the processor 901 calls and executes the methods of the embodiments of this application;

[0142] Input / output interface 903, used to implement information input and output;

[0143] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0144] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0145] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0146] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program implements the above method when executed by a processor.

[0147] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0148] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0149] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0150] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0151] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0152] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0153] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0154] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0155] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0156] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0157] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0158] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0159] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A training method for a text classification model, It is characterized in that include: Obtaining a labeled text data set, and preprocessing the text data set to obtain a text training set; Extracting structural features from the text training set to obtain a text structural feature set; Inputting the text training set into the first large language framework for training, obtaining the first large language framework trained by local fine-tuning, and a text summary feature set output by the first large language framework trained by local fine-tuning; Inputting the text training set into the second largest language framework for training, obtaining the second largest language framework trained by local fine-tuning, and a text content feature set output by the second largest language framework trained by local fine-tuning; According to the text content feature set, performing multiple rounds of attention fusion on the text summary feature set to obtain a multiple round fusion feature set; Based on the text labels in the text data set, the text structure feature set and the multi-round fusion feature set, the parameters of the initialized text classification model are updated to obtain a trained text classification model.

2. The training method according to claim 1, It is characterized in that The preprocessing of the text data set to obtain a text training set includes: Performing a data integrity check on the text data set to obtain a first intermediate data set; Deduplication of the first intermediate data set is performed to obtain a second intermediate data set; The second intermediate data set is subjected to text content purification processing to obtain the text training set.

3. The training method according to claim 1, It is characterized in that The text training set is input into the large language framework for training, and a large language framework trained by local fine-tuning is obtained, including: Acquire a target data set, wherein the target data set is a text summary set or a text content set corresponding to the text training set; Inputting the target data set into a pre-trained language model to perform text segmentation to obtain basic text units; Inputting the text basic unit into the quantized low-rank adapter and the pre-trained language model for training to obtain feature data; Determining a quantized low-rank loss value according to a text label corresponding to the target data set and the feature data; According to the quantized low-rank loss value, updating the parameters of the quantized low-rank adapter to obtain a trained quantized low-rank adapter; According to the pre-trained language model and the trained quantized low-rank adapter, the locally fine-tuned trained large language framework is obtained.

4. The training method according to claim 1, It is characterized in that The step of performing multiple rounds of attention fusion on the text summary feature set according to the text content feature set to obtain a multiple round fusion feature set includes: Adding and performing a first multi-head attention process on the text content feature set and the text summary feature set to obtain a first fusion weight set; According to the first fusion weight set, weighted fusion is performed on the text content feature set and the text summary feature set to obtain an intermediate fusion feature set; According to the text content feature set and the text summary feature set, the intermediate fusion feature set is subjected to attention-aware fusion to obtain the multi-round fusion feature set.

5. The training method according to claim 4, It is characterized in that The step of performing attention-aware fusion on the intermediate fusion feature set according to the text content feature set and the text summary feature set to obtain the multi-round fusion feature set includes: Performing a second multi-head attention process on the intermediate fusion feature set to obtain a second fusion weight set; According to the second fusion weight set, the text content feature set and the text summary feature set are completely perceptually fused to obtain the multi-round fusion feature set.

6. The training method according to claim 1, It is characterized in that The method of updating parameters of the initialized text classification model based on the text labels in the text data set, the text structure feature set and the multi-round fusion feature set to obtain a trained text classification model includes: According to the text structure feature set, feature concatenation is performed on the multiple rounds of fusion feature sets to obtain a target feature set; Performing classification prediction on the target feature set to obtain a classification prediction result; Determining a target loss value based on the classification prediction result and the text labels in the text dataset; Based on the target loss value, the initialized text classification model is trained and updated to obtain the trained text classification model.

7. A recognition method for a text classification model, It is characterized in that include: Get the text data to be classified; The text data is input into the trained text classification model as described in any one of claims 1-6 to obtain a text classification result.

8. A training system for a text classification model, It is characterized in that include: A first processing unit is used to obtain a labeled text data set and preprocess the text data set to obtain a text training set; A second processing unit is used to extract structural features from the text training set to obtain a text structural feature set; A third processing unit is used to input the text training set into the first large language framework for training, to obtain the first large language framework trained by local fine-tuning, and a text summary feature set output by the first large language framework trained by local fine-tuning; A fourth processing unit is used to input the text training set into the second largest language framework for training, to obtain the second largest language framework trained by local fine-tuning, and a text content feature set output by the second largest language framework trained by local fine-tuning; A fifth processing unit, configured to perform multiple rounds of attention fusion on the text summary feature set according to the text content feature set to obtain a multiple rounds of fusion feature set; The sixth processing unit is used to update the parameters of the initialized text classification model based on the text labels in the text data set, the text structure feature set and the multi-round fusion feature set to obtain a trained text classification model.

9. An electronic device, It is characterized in that include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images