A text classification model training method, identification method and system

Abstract

The application provides a text classification model training method, a recognition method and a system. The training method obtains a labeled text dataset, pre-processes the text dataset to obtain a text training set, extracts structural features of the text training set to obtain a text structural feature set, inputs the text training set into different large language frameworks respectively to obtain a text summary feature set and a text content feature set output by the different large language frameworks, performs multi-round attention fusion on the text summary feature set according to the text content feature set to obtain a multi-round fusion feature set, and updates parameters of an initialized text classification model based on a text label in the text dataset, the text structural feature set and the multi-round fusion feature set to obtain a trained text classification model. The text classification model trained by the training method can effectively improve the accuracy of text data classification. The application relates to the technical field of natural language processing.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a training method, recognition method and system for a text classification model. Background Technology

[0002] With the rapid development of information technology, the scale of text data generated by platforms such as the Internet, social media, and online education is becoming increasingly massive, and text data classification has become one of the key technical issues that people are paying more and more attention to.

[0003] Currently, traditional text data classification methods are usually based on a single text representation or feature to classify text data. However, due to the complexity and diversity of text data, this method often fails to comprehensively capture the diverse and complex feature representations of text data, resulting in unsatisfactory accuracy in text data classification.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0005] The purpose of this invention is to at least partially solve one of the technical problems existing in the related art.

[0006] The main objective of this application is to propose a training method for a text classification model, which can effectively improve the accuracy of text data classification.

[0007] To achieve the above objectives, one aspect of this application proposes a method for training a text classification model, the method comprising:

[0008] Obtain the labeled text dataset and preprocess the text dataset to obtain the text training set;

[0009] Structural features are extracted from the text training set to obtain a text structural feature set;

[0010] The text training set is input into the first major language framework for training, resulting in a locally fine-tuned first major language framework and a text summary feature set output by the locally fine-tuned first major language framework.

[0011] The text training set is input into the second major language framework for training, resulting in a locally fine-tuned second major language framework and the text content feature set output by the locally fine-tuned second major language framework.

[0012] Based on the text content feature set, the text summary feature set is subjected to multi-round attention fusion to obtain a multi-round fused feature set;

[0013] Based on the text labels in the text dataset, the text structure feature set, and the multi-round fusion feature set, the parameters of the initialized text classification model are updated to obtain a trained text classification model.

[0014] In some embodiments, preprocessing the text dataset to obtain a text training set includes:

[0015] Perform a data integrity check on the text dataset to obtain a first intermediate dataset;

[0016] The first intermediate dataset is deduplicated to obtain the second intermediate dataset;

[0017] The second intermediate dataset is subjected to text content cleansing to obtain the text training set.

[0018] In some embodiments, the text training set is input into a large language framework to obtain a locally fine-tuned large language framework, including:

[0019] Obtain the target dataset, which is a text summary set or text content set corresponding to the text training set;

[0020] The target dataset is input into a pre-trained language model for text segmentation to obtain basic text units.

[0021] The basic text units are input into a quantized low-rank adapter and the pre-trained language model for training to obtain feature data.

[0022] The quantization low-rank loss value is determined based on the text labels corresponding to the target dataset and the feature data.

[0023] Based on the quantized low-rank loss value, the parameters of the quantized low-rank adapter are updated to obtain the trained quantized low-rank adapter.

[0024] Based on the pre-trained language model and the trained quantized low-rank adapter, the locally fine-tuned large language framework is obtained.

[0025] In some embodiments, the step of performing multi-round attention fusion on the text summary feature set based on the text content feature set to obtain a multi-round fused feature set includes:

[0026] The text content feature set and the text summary feature set are summed and subjected to a first multi-head attention process to obtain a first fusion weight set;

[0027] Based on the first fusion weight set, the text content feature set and the text summary feature set are weighted and fused to obtain an intermediate fusion feature set;

[0028] Based on the text content feature set and the text summary feature set, attention-aware fusion is performed on the intermediate fusion feature set to obtain the multi-round fusion feature set.

[0029] In some embodiments, the step of performing attention-aware fusion on the intermediate fusion feature set based on the text content feature set and the text summary feature set to obtain the multi-round fusion feature set includes:

[0030] The intermediate fusion feature set is subjected to a second multi-head attention process to obtain a second fusion weight set;

[0031] Based on the second fusion weight set, the text content feature set and the text summary feature set are subjected to complete perceptual fusion to obtain the multi-round fusion feature set.

[0032] In some embodiments, updating the parameters of the initialized text classification model based on the text labels in the text dataset, the text structure feature set, and the multi-round fusion feature set to obtain a trained text classification model includes:

[0033] Based on the text structure feature set, the multi-round fusion feature set is concatenated to obtain the target feature set;

[0034] The target feature set is classified and predicted to obtain the classification prediction result;

[0035] Based on the classification prediction results and the text labels in the text dataset, determine the target loss value;

[0036] Based on the target loss value, the initialized text classification model is trained and updated to obtain the trained text classification model.

[0037] To achieve the above objectives, another aspect of this application proposes a text classification model recognition method, including:

[0038] Obtain the text data to be classified;

[0039] The text data is input into the trained text classification model to obtain the text classification result.

[0040] To achieve the above objectives, another aspect of this application proposes a training system for a text classification model, comprising:

[0041] The first processing unit is used to acquire the labeled text dataset and preprocess the text dataset to obtain the text training set;

[0042] The second processing unit is used to extract structural features from the text training set to obtain a text structural feature set.

[0043] The third processing unit is used to input the text training set into the first large language framework for training, to obtain the locally fine-tuned first large language framework, and the text summary feature set output by the locally fine-tuned first large language framework.

[0044] The fourth processing unit is used to input the text training set into the second major language framework for training, to obtain the locally fine-tuned second major language framework, and the text content feature set output by the locally fine-tuned second major language framework.

[0045] The fifth processing unit is used to perform multi-round attention fusion on the text summary feature set based on the text content feature set to obtain a multi-round fused feature set;

[0046] The sixth processing unit is used to update the parameters of the initialized text classification model based on the text labels in the text dataset, the text structure feature set, and the multi-round fusion feature set, so as to obtain a trained text classification model.

[0047] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0048] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0049] The embodiments of this application include at least the following beneficial effects:

[0050] This application provides a training method, recognition method, and system for a text classification model. The training method involves: acquiring a labeled text dataset and preprocessing the dataset to obtain a text training set; extracting structural features from the training set to obtain a text structure feature set; inputting the training set into a first major language framework for training to obtain a locally fine-tuned first major language framework and a text summary feature set output by the fine-tuned first major language framework; inputting the training set into a second major language framework for training to obtain a locally fine-tuned second major language framework and a text content feature set output by the fine-tuned second major language framework; performing multi-round attention fusion on the text summary feature set based on the text content feature set to obtain a multi-round fused feature set; and updating the parameters of an initialized text classification model based on the text labels in the text dataset, the text structure feature set, and the multi-round fused feature set to obtain a trained text classification model. This training method extracts content and summary features from text data using different large language frameworks, enabling in-depth analysis of text content and a more comprehensive identification and understanding of semantic information, grammatical structure, and stylistic features. Furthermore, based on multi-round attention fusion, it analyzes text data from multiple perspectives (such as text summary and text content), enhancing the weight of key features and suppressing noise interference. This effectively improves the accuracy and generalization ability of the text classification model in predicting text data. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating a training method for a text classification model provided in an embodiment of this application;

[0052] Figure 2 This is a detailed flowchart of step S110 provided in an embodiment of this application;

[0053] Figure 3 This is a detailed flowchart of a large language framework provided in an embodiment of this application;

[0054] Figure 4 This is a detailed flowchart of step S150 provided in an embodiment of this application;

[0055] Figure 5 This is a logical diagram of step S150 provided in an embodiment of this application;

[0056] Figure 6 This is a detailed flowchart of step C3 provided in an embodiment of this application;

[0057] Figure 7This is a detailed flowchart of step S160 provided in an embodiment of this application;

[0058] Figure 8 This is a flowchart illustrating a text classification model recognition method provided in an embodiment of this application;

[0059] Figure 9 This is a schematic diagram of the framework of a training system for a text classification model provided in an embodiment of this application;

[0060] Figure 10 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses / devices and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0062] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0063] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0065] Currently, traditional text data classification methods are usually based on a single text representation or feature to classify text data. However, due to the complexity and diversity of text data, and the fact that simply superimposing the features of text data may cause the omission of important information and amplification of noise, such methods often fail to capture the diverse and complex feature representations of text data in a comprehensive manner, resulting in unsatisfactory accuracy in text data classification.

[0066] In view of this, embodiments of this application provide a training method, recognition method, and system for a text classification model. This training method extracts content features and summary features of text data through different large language frameworks, enabling in-depth analysis of text content and a more comprehensive identification and understanding of semantic information, grammatical structure, and stylistic features in the text data. In addition, this training method also analyzes text data based on multi-round attention fusion, allowing analysis from multiple angles and levels. This enhances the weight of key features in the text data and suppresses noise interference, enabling parallel learning of global and local features of the text data. This effectively improves the accuracy and generalization ability of the text classification model in classifying and predicting text data.

[0067] This application provides a training method and recognition method for a text classification model, which can be applied to text classification application scenarios. In text classification application scenarios, text classification service providers can train the text classification model using the training method provided in this application embodiment, and classify the text data provided by the user based on the trained text classification model, thereby obtaining text data categories or text scores with high accuracy.

[0068] This application provides a text classification model training and recognition method, which can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited thereto; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application implementing the method, but is not limited to the above forms.

[0069] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0070] Reference Figure 1 , Figure 1 This is a schematic diagram of an optional flowchart of a text classification model training method provided in an embodiment of this application. Figure 1 The training methods may include, but are not limited to, steps S110 to S160.

[0071] Step S110: Obtain the labeled text dataset and preprocess the text dataset to obtain the text training set;

[0072] In this embodiment of the application, the labeled text dataset can be a publicly available text classification dataset on the Internet. The text dataset includes several text data, each of which includes key fields such as text ID, text content, and text tag. The text ID is a unique identifier for the text data, the text content is the text to be classified and evaluated, and the text tag is the score or category of the text content.

[0073] Reference Figure 2 In some embodiments, step S110, preprocessing the text dataset to obtain a text training set, includes:

[0074] A1. Perform a data integrity check on the text dataset to obtain the first intermediate dataset;

[0075] A2. Remove duplicate data from the first intermediate dataset to obtain the second intermediate dataset;

[0076] A3. Perform text content purification processing on the second intermediate dataset to obtain the text training set.

[0077] Understandably, the data integrity check in step A1 could involve checking whether each text data in the text dataset contains missing values, deleting samples (i.e., text data) with missing values, thus obtaining the first intermediate dataset. Next, step A2 could involve deduplicating the text dataset based on the text ID of each text data, removing duplicate samples, thus obtaining the second intermediate dataset. This second intermediate dataset ensures that each text ID appears only once in the dataset, thereby avoiding data redundancy and bias in the text classification model during training.

[0078] It should be noted that the text content purification process in step A3 can be applied to the text content of each text data point in the second intermediate dataset. Specifically, this involves removing hypertext tags, special characters, and irrelevant formatting information from the text content to reduce potential interference with the performance of the text classification model. Simultaneously, the label columns formed by the text labels of all text data points are validated and adjusted to ensure that the text labels of each text data point belong to a predefined text type or text rating level, thereby guaranteeing the standardization of text labels and improving the accuracy and generalization ability of the model.

[0079] Step S120: Extract structural features from the text training set to obtain a text structural feature set;

[0080] In this embodiment of the application, the structural feature extraction in step S120 may be the extraction of the organizational structure features of each text data in the text training set. Specifically, for a certain text data... Structural feature extraction can specifically include paragraph feature extraction, which assesses the organization and fluency of a text by counting the number of natural paragraphs; word count, which reflects the length and information content of a text by counting the total number of words; stop word analysis, which identifies common words with minor semantic contributions by counting the number of stop words; lexical diversity analysis, which measures the lexical richness of a text by counting the number of distinct words after removing stop words; spell checking, which identifies the number of potentially misspelled words in a text; sentence feature extraction, which assesses the complexity and level of detail of a text by counting the number of sentences and their average length; punctuation analysis, which reflects the grammatical structure and expressive diversity of a text by counting the number of punctuation marks; part-of-speech distribution analysis, which analyzes the grammatical structure and stylistic features of a text by counting the number of nouns, articles, prepositions, adjectives, adverbs, verbs, and conjunctions; and transition word statistics, which identifies and counts the number of transition words in a text, which are typically used to express shifts in viewpoint or emphasis.

[0081] It is understandable that the same principle applies to the remaining text data, and the same reasoning can be applied to derive the same conclusions from the foregoing. Furthermore, the embodiments of this application, based on the extraction of structural features from each text data point in the text training set, can provide the text classification model with more comprehensive text features, thereby improving the text classification model's ability to classify and evaluate text structure and content instructions.

[0082] For example, in this embodiment, the organizational structure features of the text can be extracted using quantitative statistical methods. Specifically, regular expressions can be used to count basic features, including calculating the number of paragraphs, total number of characters, number of stop words, average word length, and other basic indicators to gain a preliminary understanding of the text's scale and composition. Then, lexical diversity analysis can be performed using Python's collection library. After removing stop words, the number of different words can be counted to assess the text's lexical richness. Additionally, the number of misspelled words can be identified using the natural language processing toolkit pyspellchecker to assess the text's quality. For sentence feature extraction, regular expressions can be used to count the number of sentences in the text and calculate the average sentence length to assess the complexity of the sentence structure. For punctuation analysis, the string library can be used to implement punctuation statistics, and the number of punctuation marks can reflect the text's grammatical details and expressive diversity. For grammatical component statistics, the natural language processing toolkit nltk can be used to count grammatical components in the text, such as nouns, articles, prepositions, adjectives, adverbs, verbs, conjunctions, and transition words, to analyze the text's grammatical structure in depth.

[0083] Step S130: Input the text training set into the first large language framework for training to obtain the locally fine-tuned first large language framework and the text summary feature set output by the locally fine-tuned first large language framework.

[0084] Reference Figure 3 In some embodiments, the text training set is input into a large language framework to obtain a locally fine-tuned large language framework, including:

[0085] B1. Obtain the target dataset, which is a text summary set or text content set corresponding to the text training set;

[0086] B2. Input the target dataset into a pre-trained language model to perform text segmentation and obtain basic text units;

[0087] B3. Input the basic text units into the quantized low-rank adapter and the pre-trained language model for training to obtain feature data;

[0088] B4. Determine the quantized low-rank loss value based on the text labels corresponding to the target dataset and the feature data;

[0089] B5. Update the parameters of the quantized low-rank adapter according to the quantized low-rank loss value to obtain the trained quantized low-rank adapter.

[0090] B6. Based on the pre-trained language model and the trained quantized low-rank adapter, the locally fine-tuned large language framework is obtained.

[0091] In the embodiments of this application, the pre-trained language model can be any large language model (LLM), and this application does not impose any restrictions on the specific large language model used. In addition, if the output of the large language framework is a text summarization feature set, then the large language framework is the first large language framework. The first large language framework is used to capture the core information of each text data in the text training set, and to provide a refined representation of the key content of the text for the text classification model.

[0092] It is understandable that if the target dataset of step B1 is a text summary set, the text summary set can be specifically extracted based on the Bart-large-CNN open-source large language model to extract the core information of the text training set text data, and then the extracted text summary is matched and combined with the corresponding text label. The corresponding text label can be determined by the text ID of the text data, thereby obtaining the text summary set; step B2 can be to input the text summary set into a pre-trained language model, and output the basic text unit corresponding to the text summary set through the word segmentation function provided by the pre-trained language model. The basic text unit can be any one of the numeric ID, special symbol, etc. corresponding to the words in the text summary set in the dictionary of the pre-trained language model, and this application does not impose any restrictions here.

[0093] It should be noted that step B3 can involve inputting the basic text units corresponding to the text summary set into the quantization low-rank adapter and the pre-trained language model. Specifically, this can be done by training the quantization low-rank adapter and the pre-trained language model based on the embedding matrix corresponding to the basic text units, thereby obtaining the feature data output by the quantization low-rank adapter and the pre-trained language model. Specifically, the quantization address adapter can be a weight matrix parallel to the parameter matrix of the pre-trained language model. This weight matrix can be further decomposed into a dimension-reduced matrix and a dimension-increasing matrix based on its low-rank property. Furthermore, when the text summary set is input into the pre-trained language model and the quantization low-rank adapter for the first time, the dimension-reduced matrix can be obtained by initializing it with a Gaussian function, while the dimension-increasing matrix can be an all-zero matrix.

[0094] It is worth mentioning that after obtaining the feature data, step B4 can calculate the quantized low-rank loss value based on the feature data and text labels. Then, step B5 can update the parameters of the quantized low-rank adapter based on the calculated quantized low-rank loss value, while the original parameters of the pre-trained language model remain unchanged, thus obtaining the trained quantized low-rank adapter.

[0095] Specifically, step B5 can update the relevant parameters of the quantized low-rank adapter using the backpropagation algorithm, thereby improving the performance of the quantized low-rank adapter. In this embodiment, the training method can be performed in a cyclic iterative manner, that is, the parameters of the quantized low-rank adapter can be updated round by round. For example, multiple text summary sets can be obtained, and these text summary sets can be divided into different training data groups. First, the corresponding quantized low-rank loss value is calculated using the data from one training data group, and the parameters of the quantized low-rank adapter are updated in one round. Then, based on the pre-trained language model and the updated quantized low-rank adapter, the corresponding quantized low-rank loss value is calculated using the data from another training data group, and the parameters of the quantized low-rank adapter are updated in another round. This process is repeated cyclically to obtain a trained quantized low-rank adapter. Here, the condition for determining the end of training can be set according to actual needs. For example, in some embodiments, a target number of training iterations can be set, and the iteration number can be recorded during training. The parameters of the quantized low-rank adapter are updated once each time, and the iteration number can be incremented by 1. If the number of iterations reaches the target number after updating the parameters of the quantized low-rank adapter, training can be considered complete, and a trained quantized low-rank adapter is obtained. If the number of iterations does not reach the target number after updating the parameters of the quantized low-rank adapter, iterative training can continue. In some embodiments, a test dataset can also be obtained, and the prediction accuracy of the first major language framework can be tested using the test dataset. Each time the parameters of the quantized low-rank adapter are updated, the test dataset is used for testing. If the prediction accuracy obtained after updating the parameters of the quantized low-rank adapter meets the preset metric, training can be considered complete, and a trained quantized low-rank adapter is obtained. If the prediction accuracy obtained after updating the parameters of the quantized low-rank adapter does not meet the preset metric, iterative training can continue.

[0096] Step B6 can be to combine the parameter matrix of the pre-trained language model with the low-dimensional and high-dimensional matrices of the quantized low-rank adapter to obtain the first large language framework that has been fine-tuned locally. There are many ways to combine them, which will not be elaborated here.

[0097] It is easy to understand that the embodiments of this application achieve fine-tuning training of the pre-trained language model by training and updating the quantized low-rank adapter, thereby obtaining the first major language framework with local fine-tuning training. It is not limited to adjusting the output parameters of the last layer of the pre-trained language model, but can adjust more model parameters according to the input dataset, with better adaptability. Moreover, compared with the full parameter fine-tuning of the pre-trained language model, it can effectively reduce the required computing resources and has higher fine-tuning training efficiency.

[0098] Step S140: Input the text training set into the second large language framework for training to obtain the locally fine-tuned second large language framework and the text content feature set output by the locally fine-tuned second large language framework.

[0099] In this embodiment of the application, the content of step S140 is similar to that of the aforementioned step S130, and can be simply deduced by analogy. Specifically, if the target dataset of step B1 is a text content set, then the large language framework is the second large language framework. The second large language framework is used to extract deep features of the text content set, and its output is a text content feature set, which will not be elaborated here.

[0100] Reference Figure 4 and Figure 5 Step S150: Based on the text content feature set, perform multi-round attention fusion on the text summary feature set to obtain a multi-round fused feature set;

[0101] In some embodiments, step S150, performing multi-round attention fusion on the text summary feature set according to the text content feature set to obtain a multi-round fused feature set, includes:

[0102] C1. The text content feature set and the text summary feature set are summed and subjected to a first multi-head attention process to obtain a first fusion weight set;

[0103] C2. Based on the first fusion weight set, perform weighted fusion on the text content feature set and the text summary feature set to obtain an intermediate fusion feature set.

[0104] In this embodiment, step C1 can first involve summing the text content feature set and the text summary feature set to obtain an intermediate summed feature set. Then, the intermediate summed feature set is input into the Multi-Head Attention (MHA) module for calculation, and the calculated result is input into the activation module. Specifically, the weights are output through the Sigmoid activation function to obtain a first fusion weight set. Next, the first fusion weight set is weighted and averaged with the original input text content feature set and text summary feature set to obtain an intermediate fusion feature set. The equivalent expression for this intermediate fusion feature set can be:

[0105]

[0106] Among them, F text1 For intermediate fusion feature set; F content Z is the text content feature set; Z is the intermediate summation feature set; sigmoid(Z) is the first fusion weight set; F summary This is a feature set for text summarization.

[0107] C3. Based on the text content feature set and the text summary feature set, perform attention-aware fusion on the intermediate fusion feature set to obtain the multi-round fusion feature set.

[0108] Reference Figure 6 Further, step C3, performing attention-aware fusion on the intermediate fusion feature set based on the text content feature set and the text summary feature set to obtain the multi-round fusion feature set, includes:

[0109] C31. Perform a second multi-head attention process on the intermediate fusion feature set to obtain a second fusion weight set;

[0110] C32. Based on the second fusion weight set, perform complete perceptual fusion on the text content feature set and the text summary feature set to obtain the multi-round fusion feature set.

[0111] In this embodiment, step C31, the second multi-head attention processing, can involve inputting the intermediate fused feature set into another multi-head attention mechanism module (MHA) for calculation, and inputting the calculated result into the activation module. Specifically, the weights are output through the Sigmoid activation function to obtain the second fused weight set. Step C32 can be based on the second fused weight set, performing a weighted average operation with the original input text content feature set and text summary feature set respectively, thereby achieving complete perception of the input features and obtaining a multi-round fused feature set.

[0112] For example, the equivalent expression for this multi-round fusion feature set can be:

[0113]

[0114] Among them, Ft ext2 For multi-round fusion feature sets; Ft ext1 For intermediate fusion feature set; Fc ontent For text content feature set; sigmoid(F text1 F is the second fusion weight set; summary This is a feature set for text summarization.

[0115] It is worth noting that the intermediate fusion feature set obtained through steps C1 and C2 is merely a simple addition and superposition of the two input features, which may result in the omission of important information and the amplification of noise, and often fails to provide a comprehensive perception of the input features. In contrast, the embodiments of this application, based on steps C31 and C32, use a multi-head attention mechanism to perform attention fusion on the intermediate fusion feature set, which can more comprehensively capture the diverse and complex feature representations of text data and suppress noise interference.

[0116] Step S160: Based on the text labels in the text dataset, the text structure feature set, and the multi-round fusion feature set, update the parameters of the initialized text classification model to obtain the trained text classification model.

[0117] Reference Figure 7 In some embodiments, step S160, updating the parameters of the initialized text classification model based on the text labels in the text dataset, the text structure feature set, and the multi-round fusion feature set to obtain a trained text classification model, includes:

[0118] D1. Based on the text structure feature set, perform feature concatenation on the multi-round fusion feature set to obtain the target feature set;

[0119] D2. Perform classification prediction on the target feature set to obtain the classification prediction result;

[0120] D3. Based on the classification prediction results and the text labels in the text dataset, determine the target loss value;

[0121] D4. Based on the target loss value, train and update the initialized text classification model to obtain the trained text classification model.

[0122] In this embodiment, step D1 may involve concatenating the obtained text structure feature set and the multi-round fusion feature set to obtain a target feature set representing multiple dimensions of text structure, text content, and text summary. Specifically, the target feature in this target feature set can be a high-dimensional feature vector used to represent various aspects of the text data. Step D2 may involve inputting the target feature set into a classifier for classification prediction. This classifier may specifically be a multilayer perceptron (MLP) to map the target features in the target feature set to predefined scoring categories, thereby obtaining the classification prediction result.

[0123] Understandably, the classification prediction accuracy of a text classification model can be measured by the consistency between the classification prediction result and the text label. The closer the classification prediction result is to the corresponding text label, the more accurate and reliable the classification prediction result provided by the text classification model is; conversely, the less accurate and reliable the classification prediction result is to the corresponding text label, the less accurate and reliable the classification prediction result provided by the text classification model is. Therefore, in this embodiment, a loss value can be determined based on the classification prediction result and the text label, and the obtained loss value is recorded as the target loss value. The loss function used to determine the target loss value can be any one of the following: 0-1 loss function, squared loss function, absolute loss function, logarithmic loss function, cross-entropy loss function, etc.

[0124] It should be noted that the training update in step D4 can be implemented by using the Adam backpropagation algorithm to update the parameters of the text classification model. Specifically, it can be to update the parameters of the multi-round attention fusion module that implements the multi-round attention fusion function and the classification layer that implements the text classification function in the text classification model. After several iterations, the trained text classification model can be obtained. The specific number of iterations can be preset, or the training can be considered complete when the accuracy requirement is met on the test set.

[0125] Reference Figure 8 , Figure 8 This is an optional flowchart illustrating a text classification model recognition method provided in an embodiment of this application. Figure 8 The identification method may include, but is not limited to, steps S170 to S180.

[0126] Step S170: Obtain the text data to be classified;

[0127] Step S180: Input the text data into the trained text classification model as described above to obtain the text classification result.

[0128] In this embodiment of the application, the text data to be classified can be input into a trained text classification model, and the text classification model can predict the text category to which the text data belongs or the text score of the text data to obtain the text classification result corresponding to the text data.

[0129] Please see Figure 9 This application also provides a training system for a text classification model, the training system comprising:

[0130] The first processing unit 810 is used to acquire the labeled text dataset and preprocess the text dataset to obtain a text training set.

[0131] The second processing unit 820 is used to extract structural features from the text training set to obtain a text structural feature set.

[0132] The third processing unit 830 is used to input the text training set into the first large language framework for training, to obtain the locally fine-tuned first large language framework, and the text summary feature set output by the locally fine-tuned first large language framework.

[0133] The fourth processing unit 840 is used to input the text training set into the second large language framework for training, to obtain the locally fine-tuned second large language framework, and the text content feature set output by the locally fine-tuned second large language framework;

[0134] The fifth processing unit 850 is used to perform multi-round attention fusion on the text summary feature set according to the text content feature set to obtain a multi-round fused feature set;

[0135] The sixth processing unit 860 is used to update the parameters of the initialized text classification model based on the text labels in the text dataset, the text structure feature set, and the multi-round fusion feature set, so as to obtain a trained text classification model.

[0136] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0137] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0138] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0139] Please see Figure 10 , Figure 10 This illustration depicts the hardware structure of an electronic device according to one embodiment. The electronic device includes:

[0140] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0141] The memory 902 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901.

[0142] The input / output interface 903 is used to implement information input and output;

[0143] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0144] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0145] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0146] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0147] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0148] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0149] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0150] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0151] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0152] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0153] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0154] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0155] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0156] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0157] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0158] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0159] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A training method for a text classification model, characterized in that, include: Obtain the labeled text dataset and preprocess the text dataset to obtain the text training set; Structural features are extracted from the text training set to obtain a text structural feature set; The text training set is input into the first major language framework for training, resulting in a locally fine-tuned first major language framework and a text summary feature set output by the locally fine-tuned first major language framework. The text training set is input into the second major language framework for training, resulting in a locally fine-tuned second major language framework and the text content feature set output by the locally fine-tuned second major language framework. Based on the text content feature set, the text summary feature set is subjected to multi-round attention fusion to obtain a multi-round fused feature set; Based on the text labels in the text dataset, the text structure feature set, and the multi-round fusion feature set, the parameters of the initialized text classification model are updated to obtain a trained text classification model.

2. The training method according to claim 1, characterized in that, The preprocessing of the text dataset to obtain the text training set includes: Perform a data integrity check on the text dataset to obtain a first intermediate dataset; The first intermediate dataset is deduplicated to obtain the second intermediate dataset; The second intermediate dataset is subjected to text content cleansing to obtain the text training set.

3. The training method according to claim 1, characterized in that, The text training set is input into the large language framework for training, resulting in a locally fine-tuned large language framework, including: Obtain the target dataset, which is a text summary set or text content set corresponding to the text training set; The target dataset is input into a pre-trained language model for text segmentation to obtain basic text units. The basic text units are input into a quantized low-rank adapter and the pre-trained language model for training to obtain feature data. The quantization low-rank loss value is determined based on the text labels corresponding to the target dataset and the feature data. Based on the quantized low-rank loss value, the parameters of the quantized low-rank adapter are updated to obtain the trained quantized low-rank adapter. Based on the pre-trained language model and the trained quantized low-rank adapter, the locally fine-tuned large language framework is obtained.

4. The training method according to claim 1, characterized in that, The step of performing multi-round attention fusion on the text summary feature set based on the text content feature set to obtain a multi-round fused feature set includes: The text content feature set and the text summary feature set are summed and subjected to a first multi-head attention process to obtain a first fusion weight set; Based on the first fusion weight set, the text content feature set and the text summary feature set are weighted and fused to obtain an intermediate fusion feature set; Based on the text content feature set and the text summary feature set, attention-aware fusion is performed on the intermediate fusion feature set to obtain the multi-round fusion feature set.

5. The training method according to claim 4, characterized in that, The step of performing attention-aware fusion on the intermediate fusion feature set based on the text content feature set and the text summary feature set to obtain the multi-round fusion feature set includes: The intermediate fusion feature set is subjected to a second multi-head attention process to obtain a second fusion weight set; Based on the second fusion weight set, the text content feature set and the text summary feature set are subjected to complete perceptual fusion to obtain the multi-round fusion feature set.

6. The training method according to claim 1, characterized in that, The process of updating the parameters of the initialized text classification model based on the text labels in the text dataset, the text structure feature set, and the multi-round fusion feature set to obtain a trained text classification model includes: Based on the text structure feature set, the multi-round fusion feature set is concatenated to obtain the target feature set; The target feature set is classified and predicted to obtain the classification prediction result; Based on the classification prediction results and the text labels in the text dataset, determine the target loss value; Based on the target loss value, the initialized text classification model is trained and updated to obtain the trained text classification model.

7. A recognition method for a text classification model, characterized in that, include: Obtain the text data to be classified; The text data is input into the trained text classification model as described in any one of claims 1-6 to obtain the text classification result.

8. A training system for a text classification model, characterized in that, include: The first processing unit is used to acquire the labeled text dataset and preprocess the text dataset to obtain the text training set; The second processing unit is used to extract structural features from the text training set to obtain a text structural feature set. The third processing unit is used to input the text training set into the first large language framework for training, to obtain the locally fine-tuned first large language framework, and the text summary feature set output by the locally fine-tuned first large language framework. The fourth processing unit is used to input the text training set into the second major language framework for training, to obtain the locally fine-tuned second major language framework, and the text content feature set output by the locally fine-tuned second major language framework. The fifth processing unit is used to perform multi-round attention fusion on the text summary feature set based on the text content feature set to obtain a multi-round fused feature set; The sixth processing unit is used to update the parameters of the initialized text classification model based on the text labels in the text dataset, the text structure feature set, and the multi-round fusion feature set, so as to obtain a trained text classification model.

9. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method as described in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

Images