Text classification model training method, text generation method, and related device

Abstract

The application discloses a text classification model training method, a text generation method and related equipment, the text classification model training method comprises the following steps: converting a question text into an embedding sequence through an embedding layer, effectively extracting coding features in the text through an encoding layer, capturing text features of different granularities by using a multi-scale convolution, enhancing the understanding ability of long text, and introducing a self-attention mechanism into the weighted processing of convolution features and pooling features, effectively improving the weight of important features, reducing the influence of redundant features, optimizing the accuracy of intent classification, finally generating a classification result through an output layer, and realizing model training according to the classification result to obtain a trained text classification model, effectively improving the accuracy of text classification, and the intelligent customer service system can take the classification result output by the trained text classification model as the basis for reply, thereby effectively improving the reply efficiency and reply accuracy of the intelligent customer service system under the condition of limited resources.

Description

Technical Field

[0001] This invention relates to the field of data processing, and in particular to a training method for a text classification model, a text generation method, and related equipment. Background Technology

[0002] With the leapfrog development of Natural Language Processing (NLP) and Artificial Intelligence (AI) technologies, intelligent customer service question-answering systems have been widely adopted by various enterprises. Traditional human and intelligent customer service systems, based on limited knowledge bases, can only answer questions with fixed templates, making it difficult to meet the diverse needs of a massive number of customers. With the rapid development of neural network models and large-scale models, intelligent customer service systems have found new solutions.

[0003] Existing intelligent customer service systems utilize large-scale models to more accurately identify user needs and answer user questions. However, with the increase in users and a large number of customers simultaneously making requests to the intelligent customer service system, limited resources necessitate that the system prioritize user questions based on the intensity of user emotions, determining which questions require urgent attention. The depth and accuracy of existing intelligent customer service systems in classifying user intent need improvement, and the diverse questioning styles of users also pose significant challenges.

[0004] Therefore, how to effectively improve the accuracy of intent classification in intelligent customer service systems has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] Therefore, it is necessary to provide a training method for a text classification model, a text generation method, and related equipment to address the aforementioned technical problems and solve the issue of low accuracy in intent classification in traditional intelligent customer service systems.

[0006] A method for training a text classification model, the method comprising:

[0007] Obtain the text classification model to be trained and the text dataset. The text dataset includes sample question texts and classification labels of the sample question texts. The text classification model to be trained includes an embedding layer, an encoding layer, a character convolutional layer, a pooling layer and an output layer.

[0008] The sample question text is converted into an embedding sequence through the embedding layer, and the embedding sequence is encoded through the encoding layer to obtain encoded features;

[0009] The encoded features are input into the character convolutional layer to perform multi-scale character-level convolution to obtain convolutional features. The attention weights of each feature unit in the convolutional features are calculated based on the self-attention mechanism and then weighted to obtain weighted features.

[0010] The weighted features are input into the pooling layer for max pooling to obtain pooled features. The attention weights of each feature unit in the pooled features are adjusted based on the self-attention mechanism to obtain the target weighted features.

[0011] The target weighted features are input into the output layer to obtain the classification result of the sample question text;

[0012] Based on the classification results and the classification labels, the parameters of the text classification model to be trained are iterated to obtain a trained text classification model.

[0013] Optionally, the encoding layer includes multiple sub-encoding layers connected in sequence. Each sub-encoding layer includes an attention sub-layer, a residual connection sub-layer, and a normalization layer. The attention sub-layer is either a simplified attention layer or a full attention layer. The simplified attention layer consists of feedforward computation units, and the full attention layer consists of multi-head attention units. The encoding process of the embedded sequence through the encoding layer to obtain encoded features includes:

[0014] For each sub-coding layer, if the sub-coding layer is the first sub-coding layer, the input feature is the embedding sequence; if the sub-coding layer is not the first sub-coding layer, the input feature is the feature vector output by the previous sub-coding layer.

[0015] When the attention sublayer of the sub-coding layer is a simplified attention layer, the feedforward features are obtained by performing position-by-position linear transformation and nonlinear activation on the input features through the feedforward computation unit.

[0016] When the attention sublayer of the sub-coding layer is a complete attention layer, the attention weights between each position in the input features are calculated by a multi-head attention unit, and the weights are applied to the input features to obtain weighted features.

[0017] The feedforward features or the weighted features are superimposed on the input features through the residual connection sub-layer, and then scaled through the normalization layer to output the feature vector of the current sub-encoding layer.

[0018] The feature vector output from the last sub-coding layer is used as the encoded feature.

[0019] Optionally, the step of inputting the encoded features into the character convolutional layer and performing multi-scale character-level convolution to obtain convolutional features includes:

[0020] The encoded features are divided into feature matrices at the character-level granularity;

[0021] Based on the size of the convolution kernel, the sliding stride or receptive field range of the feature matrix is ​​matched and adjusted to obtain the adjusted feature matrix;

[0022] Window-by-window convolution calculations are performed on the adjusted feature matrix using multiple convolution kernels, and the convolution response results corresponding to each convolution kernel are extracted.

[0023] The convolutional response results of different convolutional kernels are fused to obtain the multi-scale convolutional features.

[0024] Optionally, the step of inputting the weighted features into the pooling layer for max pooling to obtain pooled features includes:

[0025] The weighted features are divided into sliding sections according to a preset window size, and the feature unit with the largest convolution output value is selected within each window range as the representative feature of the corresponding window.

[0026] The representative features of each window are combined sequentially to form the pooled features.

[0027] Optionally, the step of inputting the target weighted features into the output layer to obtain the classification result of the sample question text includes:

[0028] The target weighted features are linearly combined with preset model parameters through the output layer to obtain the category probability of the sample question text;

[0029] The classification result is determined based on the category probability.

[0030] A text generation method, the method comprising:

[0031] Obtain a question text set, which includes question texts from multiple users awaiting responses;

[0032] The question text is input into a trained text classification model so that the trained text classification model can classify the question text and obtain the classification result of the question text. The trained text classification model is trained using a text classification model training method.

[0033] Based on the classification results of each question text, the response priority for each question text is determined;

[0034] Based on the response priority, determine the response strategy for the question text;

[0035] According to the response priority from high to low, the corresponding response texts are generated sequentially based on the response strategy, and the response texts are sent back to the corresponding users who need a response.

[0036] A training device for a text classification model, the device comprising:

[0037] The first acquisition module is used to acquire the text classification model to be trained and the text dataset. The text dataset includes sample question texts and classification labels of the sample question texts. The text classification model to be trained includes an embedding layer, an encoding layer, a character convolutional layer, a pooling layer and an output layer.

[0038] The first encoding module is used to convert the sample question text into an embedding sequence through the embedding layer, and to encode the embedding sequence through the encoding layer to obtain encoded features;

[0039] The first convolution module is used to input the encoded features into the character convolution layer, perform multi-scale character-level convolution to obtain convolution features, and calculate the attention weights of each feature unit in the convolution features based on the self-attention mechanism and perform weighted processing to obtain weighted features;

[0040] The first pooling module is used to input the weighted features into the pooling layer for max pooling to obtain pooled features, and adjust the attention weights of each feature unit in the pooled features based on the self-attention mechanism to obtain the target weighted features.

[0041] The first output module is used to input the target weighted features into the output layer to obtain the classification result of the sample question text;

[0042] The first iteration module is used to perform parameter iteration on the text classification model to be trained based on the classification result and the classification label, so as to obtain a trained text classification model.

[0043] A text generation apparatus, the apparatus comprising:

[0044] The second acquisition module is used to acquire a question text set, which includes question texts from multiple users who need to be answered.

[0045] The first input module is used to input the question text into a trained text classification model, so that the trained text classification model can classify the question text and obtain the classification result of the question text. The trained text classification model is trained by a text classification model training method.

[0046] The first determining module is used to determine the response priority for each question text based on the classification result of each question text;

[0047] The second determining module is used to determine the response strategy for the question text based on the response priority;

[0048] The first generation module generates corresponding response texts for the question texts according to the response priority from high to low, based on the response strategy, and sends the response texts back to the corresponding users who need to be responded to.

[0049] A computer device includes a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor, when executing the computer-readable instructions, implements the above-described text classification model training method or the above-described text generation method.

[0050] A readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the training method of the above-described text classification model or the above-described text generation method.

[0051] The training method for the above text classification model involves obtaining a text classification model to be trained and a text dataset. The text dataset includes sample question texts and their classification labels. The text classification model to be trained includes an embedding layer, an encoding layer, a character convolutional layer, a pooling layer, and an output layer. The embedding layer converts the sample question texts into embedding sequences, and the encoding layer encodes these sequences to obtain encoded features. These encoded features are then input into the character convolutional layer for multi-scale character-level convolution to obtain convolutional features. Attention weights for each feature unit in the convolutional features are calculated based on a self-attention mechanism and weighted to obtain weighted features. These weighted features are then input into the pooling layer for max pooling to obtain pooled features. The attention weights for each feature unit in the pooled features are adjusted based on a self-attention mechanism to obtain target weighted features. These target weighted features are then input into the output layer to obtain the classification result of the sample question text. Based on the classification result and the classification labels, the parameters of the text classification model to be trained are iterated to obtain a trained text classification model. By transforming the question text into an embedding sequence through an embedding layer, effectively extracting encoded features from the text through an encoding layer, and capturing text features of different granularities using multi-scale convolution, the understanding of long texts is enhanced. Furthermore, a self-attention mechanism is introduced into the weighted processing of convolutional and pooling features, effectively increasing the weight of important features, reducing the impact of redundant features, and optimizing the accuracy of intent classification. Finally, the classification results generated by the output layer are used to train the model, resulting in a well-trained text classification model that effectively improves the accuracy of text classification. Moreover, the intelligent customer service system can use the classification results output by the trained text classification model as the basis for responses, thereby effectively improving the response efficiency and accuracy of the intelligent customer service system under limited resource conditions. Attached Figure Description

[0052] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a flowchart illustrating the training method of a text classification model in one embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of the structure of a text classification model in one embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram of the coding layer structure in one embodiment of the present invention;

[0056] Figure 4 This is a flowchart illustrating a text generation method in one embodiment of the present invention;

[0057] Figure 5 This is a schematic diagram of the structure of a training device for a text classification model in one embodiment of the present invention;

[0058] Figure 6 This is a schematic diagram of the structure of a text generation device in one embodiment of the present invention;

[0059] Figure 7 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] In one embodiment, such as Figure 1 As shown, a method for training a text classification model is provided, including the following steps:

[0062] 101. Obtain the text classification model to be trained and the text dataset.

[0063] In this embodiment of the invention, the aforementioned text dataset includes sample question texts and classification labels for the sample question texts. The sample question texts can be various questions entered by users in an intelligent customer service system, such as user feedback on products, service evaluations, or other consultative questions. The user-input questions are preprocessed, for example, by text segmentation and cleaning up stop words, special symbols, and meaningless content, ultimately resulting in the sample question texts. The classification labels can be the emotion category or intent category corresponding to the sample question texts. Classification labels include, but are not limited to, the following types: negative emotions (such as anger, disappointment), positive emotions (such as happiness, surprise), and neutral emotions (such as inquiry, worry). Each question text is associated with a classification label, which indicates the emotion or intent expressed by the question, enabling the model to learn the characteristics of different emotions or intents. Specifically, intent analysis can be performed on the sample question texts to obtain the corresponding classification labels.

[0064] The above text dataset can be further illustrated in the following table:

[0065] Table 1

[0066]

[0067] As shown in Table 1, the user-input text refers to the questions entered by users through the intelligent customer service system. After segmenting the text of these questions and removing stop words, special symbols, and meaningless content, the sample question text (i.e., the filtered content in Table 1) is obtained. It can be seen that the classification labels include negative (anger), positive (happiness), neutral / slightly negative (disappointment), neutral (inquiry), negative (anger), positive (surprise), neutral / slightly anxious (worry), and positive (excitement).

[0068] The text classification model to be trained includes an embedding layer, an encoding layer, a character convolutional layer, a pooling layer, and an output layer.

[0069] The aforementioned embedding layer can be an embedding structure used to map discrete text symbols (such as characters, words, or sub-words) into continuous vector representations. Specifically, it can employ a combination of word embedding, sentence embedding, and positional embedding, enabling the model to not only understand the semantic relationships between words but also recognize the positional order of words within a sentence. For example, for the input sentence "customer service attitude is extremely poor," the embedding layer can transform it into a vector sequence representing semantic and syntactic information, providing input for subsequent semantic modeling.

[0070] The aforementioned encoding layer can be a hierarchical structure composed of multiple sub-encoders, used to extract local and global semantic features from vector sequences. Each sub-encoder layer can include a simplified attention layer and a full attention layer, capturing local features through a feedforward network and establishing global dependencies through a multi-head attention mechanism. This structure can effectively reduce computational cost while maintaining sensitivity to semantic context when processing long text sequences.

[0071] The aforementioned character convolutional layer can be a multi-scale convolutional structure that performs convolution operations based on character-level input. This layer can extract features of different granularities at different kernel sizes; for example, small kernels are suitable for capturing word-level features, while large kernels are suitable for capturing syntactic structures, thus obtaining richer semantic expressions. The output features of this convolutional layer will be further used to distinguish between emotion intensity or intent.

[0072] The pooling layer mentioned above can be a feature compression layer using max pooling, used to reduce redundant information while preserving the main features. By windowing the convolution output, each window retains the feature unit with the highest response value, thereby focusing on the feature regions that contribute most to classification, improving the model's generalization ability and computational efficiency.

[0073] The output layer described above can be a classification layer composed of a fully connected network, used to map attention-weighted and pooled features to a specific classification label space. The output layer calculates the probability values ​​of each category through logistic regression or the Softmax function, achieving the final classification of the input question text, such as "angry," "happy," and "neutral."

[0074] Specifically, the above text classification model can be implemented through methods such as... Figure 2 The schematic diagram of a text classification model shown further illustrates this. Figure 2 As can be seen, the encoding layer can be constructed using ModernBERT modules for deep semantic modeling of the input user question text. A ModernBERT module includes an embedding layer, a Flash Attention unit (i.e., an attention sublayer), residual connections, and a normalization layer, and may also include a safe softmax layer. The embedding layer transforms textual information into high-dimensional semantic vectors through a combination of word embeddings, sentence embeddings, and positional embeddings, enabling the model to simultaneously understand word meaning and contextual positional relationships. Flash Attention accelerates the attention calculation process, reduces redundant computation, and improves computational efficiency under long sequence text inputs. The combination of residual connections and normalization layers ensures the stability of the deep network, avoiding problems such as gradient vanishing or exploding, while safe softmax improves the robustness and stability of the model during numerical computation.

[0075] The output of the ModernBERT module is fed into the CharCNN module for further local feature extraction and fine-grained classification. The CharCNN module consists of convolutional layers, self-attention layers, pooling layers, and fully connected layers. The convolutional layers perform sliding convolutions on the features output by ModernBERT using multi-scale convolutional kernels to extract local features at different granularities. The self-attention layers calculate the correlation weights between different feature units based on the convolutional output, thereby highlighting key semantic features and suppressing redundant information. The pooling layers aggregate features from the convolutional results, using max pooling to retain the most salient features and reduce information dimensionality. After the features from the pooling layers undergo a second self-attention processing step, they are fed into the fully connected layers for feature fusion and final classification, outputting the classification result for the user's question text.

[0076] By combining the ModernBERT and CharCNN modules, the model achieves synergistic optimization of global semantic understanding and local detail mining, which can significantly improve the accuracy of text classification while ensuring computational efficiency, thus providing a reliable basis for responses to intelligent customer service systems.

[0077] 102. The sample question text is converted into an embedding sequence through an embedding layer, and the embedding sequence is encoded through an encoding layer to obtain encoded features.

[0078] In this embodiment of the invention, the embedding sequence can be a high-dimensional vector sequence generated by the embedding layer to represent the semantic and syntactic information of the text. This embedding sequence is generated within the embedding layer through a combination of word embedding, sentence embedding, and positional embedding. It is used to map discrete text symbols (such as characters or words) into continuous numerical representations, enabling the model to capture the semantic relationships between different words and their positional relationships within a sentence. For example, for the text "customer service attitude is extremely poor," the embedding layer will convert each word into a corresponding vector, thereby obtaining an embedding sequence representing the semantic structure of the entire sentence.

[0079] Encoded features can be deep feature representations obtained by performing multi-layer semantic modeling on the embedded sequence through encoding layers. These features, while preserving the original semantics, incorporate contextual dependencies and global attention information, reflecting the semantic relationships and sentiment tendencies of the input text at different levels. Encoded features can be viewed as the model's internal semantic representation of the input text after understanding it, serving as the foundational input for subsequent convolutional and attention-weighted layers for classification.

[0080] Specifically, the encoding layer can include multiple sequentially connected sub-encoding layers. Each sub-encoding layer consists of an attention sub-layer, a residual connection sub-layer, and a normalization layer. The attention sub-layer can adopt different structural forms at different levels. Some sub-layers use a simplified attention structure, which performs position-by-position linear transformations and non-linear activations on the embedded sequence through feedforward computation units to extract local contextual features. Other sub-layers use a complete multi-head attention structure, which captures global dependency information by calculating the relevance weights between positions in the sequence, thereby enhancing the model's understanding of semantic relationships in long texts.

[0081] As the embedded sequence passes through each sub-encoding layer sequentially, the model progressively abstracts text features at different depths, ensuring that the output of each layer not only contains the original semantic information but also incorporates the emotional connections and syntactic structure between contexts. After residual connections and normalization operations, the model can maintain feature stability and prevent gradient decay. The final output encoded features are a high-dimensional semantic representation resulting from the combined effect of multi-layer attention mechanisms and feedforward networks.

[0082] For example, given the input text "Express delivery is super fast and the packaging is also exquisite", after processing by the encoding layer, the encoded features can reflect the semantic association between the two positive emotional expressions "fast delivery" and "exquisite packaging", providing a basis for the subsequent convolutional layer to extract feature strength and the attention layer to perform emotion weighting.

[0083] 103. Input the encoded features into the character convolutional layer, perform multi-scale character-level convolution to obtain convolutional features, and calculate the attention weights of each feature unit in the convolutional features based on the self-attention mechanism and perform weighted processing to obtain weighted features.

[0084] In this embodiment of the invention, the character convolutional layer is used to perform multi-scale character-level convolution processing on the encoded features to extract local feature information at different granularities. Specifically, the character convolutional layer includes multiple convolutional channels with different kernel sizes, used to scan the input feature matrix within different feature receptive fields, thereby capturing local semantic differences and detailed patterns. Smaller convolutional kernels can identify fine-grained emotional features at the word or phrase level, while larger convolutional kernels can extract structural features at the sentence level or with a longer contextual span. Through multi-scale convolution, the model can take into account word-level, phrase-level, and semantic-level features when processing diverse user expressions, thereby achieving more comprehensive sentiment analysis.

[0085] Based on the feature maps obtained from multi-scale convolution, a self-attention mechanism is introduced to weight the convolutional features. The self-attention mechanism determines the importance of each feature unit by calculating the correlation between them. Features with higher weights typically correspond to parts with strong emotional expression or prominent semantics, while features with lower weights may be redundant or noisy information. After normalizing these weights, the model performs a weighted summation of the features to form a weighted feature vector.

[0086] For example, when the input text is "The customer service attitude is extremely bad, I will never shop here again", the convolutional layer will extract local emotional features such as "bad attitude" and "will never come again". The self-attention mechanism further identifies the importance of the above features and gives them higher weights, so that the final weighted features can better reflect the user's strong negative emotions and provide a reliable basis for subsequent classification.

[0087] The calculation of the attention weights mentioned above can be further explained using the following formula:

[0088]

[0089] in, The weight matrix represents the attention weights of each feature unit. The input feature matrix (which can be the output of a convolutional or pooling layer) is represented by a feature unit, with each row or column representing a feature unit. It is a learnable weight matrix used for linear transformation of input features; For bias terms; Introducing nonlinear activation enhances the model's ability to express complex feature relationships; It is a trainable vector used to map the activated result to a scalar space; Normalize the results into a probability distribution so that the sum of all weights is 1.

[0090] 104. Input the weighted features into the pooling layer for max pooling to obtain pooled features, and adjust the attention weights of each feature unit in the pooled features based on the self-attention mechanism to obtain the target weighted features.

[0091] In this embodiment of the invention, the pooling layer is used to further filter key information from the convolutional and weighted features to compress feature dimensions and enhance feature representativeness. Specifically, the pooling layer employs a max pooling strategy, sliding the weighted features into a window of preset size, and selecting the feature unit with the largest convolutional output value within each window as the representative feature of that window. Through this method, the most significant feature responses within a specific semantic range can be effectively extracted, thereby retaining the parts with the strongest emotional expression or the greatest semantic contribution.

[0092] Subsequently, a self-attention mechanism is used to redistribute the weights of each feature unit in the pooled features to further enhance the influence of semantic key points and reduce the interference of irrelevant features. The attention weights can also be calculated using the same formula as above, specifically by using the pooled features as input features. Then, substituting these values ​​into the attention weight calculation formula above, we can calculate the attention weights (i.e., the attention weight matrix) of each feature unit in the pooled feature. ).

[0093] Finally, the weighted output is the target weighted feature, which retains the most discriminative semantic information in the input text, providing a high-confidence input basis for subsequent output layer classification.

[0094] 105. Input the target weighted features into the output layer to obtain the classification results of the sample question text.

[0095] In this embodiment of the invention, the output layer is used to generate the final text classification result based on the target weighted features after attention weighting and pooling processing. The output layer may include one or more fully connected layers and non-linear activation units, used to map high-dimensional semantic features to corresponding classification labels.

[0096] During training, the output layer receives target weighted features from the pooling layer, integrates semantic information from different dimensions through fully connected computation, and outputs the predicted probabilities for each category, thereby determining the emotion or intent category to which the text belongs. For example, in a user emotion classification analysis task, the output layer can map the feature representation output by the model to labels such as "angry," "happy," "neutral," and "worried," which can be used for subsequent priority ranking and processing decisions.

[0097] The design of this output layer enables the model to accurately map from abstract features to specific classification results while maintaining high semantic expressiveness, thereby improving the overall discriminative performance of the model in text intent recognition and emotion classification tasks.

[0098] 106. Based on the classification results and classification labels, perform parameter iteration on the text classification model to be trained to obtain a trained text classification model.

[0099] In this embodiment of the invention, parameter iteration is used to continuously optimize the weight parameters of the text classification model during the training phase, so that the classification result output by the model gradually approaches the true classification label. Specifically, in each training iteration, the model inputs the sample question text into the embedding layer, encoding layer, character convolutional layer, pooling layer, and output layer to generate a predicted classification result; then, the predicted result is compared with the corresponding classification label, and the loss value is calculated to measure the difference between the model prediction and the true label.

[0100] Based on the calculated loss value, the backpropagation algorithm is used to update the parameters of each layer of the model using gradients, adjusting the embedding layer weights, attention weights, convolutional kernel parameters, and fully connected layer parameters to minimize the overall loss. Through multiple rounds of iterative training, the model can gradually learn the mapping relationship between text features and emotion labels, thus accurately identifying the emotion category or intent category when faced with new and unseen text input.

[0101] Through this parameter iteration process, the finally trained text classification model has strong generalization ability and robustness, and can maintain stable classification performance under diverse and complex user input scenarios.

[0102] In this embodiment of the invention, a text classification model to be trained and a text dataset are obtained. The text dataset includes sample question texts and classification labels for the sample question texts. The text classification model to be trained includes an embedding layer, an encoding layer, a character convolutional layer, a pooling layer, and an output layer. The sample question texts are converted into embedding sequences through the embedding layer, and the embedding sequences are encoded through the encoding layer to obtain encoded features. The encoded features are input into the character convolutional layer for multi-scale character-level convolution to obtain convolutional features. The attention weights of each feature unit in the convolutional features are calculated based on a self-attention mechanism and weighted to obtain weighted features. The weighted features are input into the pooling layer for max pooling to obtain pooled features. The attention weights of each feature unit in the pooled features are adjusted based on a self-attention mechanism to obtain target weighted features. The target weighted features are input into the output layer to obtain the classification result of the sample question text. Based on the classification result and the classification labels, the parameters of the text classification model to be trained are iterated to obtain a trained text classification model. By transforming the question text into an embedding sequence through an embedding layer, effectively extracting encoded features from the text through an encoding layer, and capturing text features of different granularities using multi-scale convolution, the understanding of long texts is enhanced. Furthermore, a self-attention mechanism is introduced into the weighted processing of convolutional and pooling features, effectively increasing the weight of important features, reducing the impact of redundant features, and optimizing the accuracy of intent classification. Finally, the classification results generated by the output layer are used to train the model, resulting in a well-trained text classification model that effectively improves the accuracy of text classification. Moreover, the intelligent customer service system can use the classification results output by the trained text classification model as the basis for responses, thereby effectively improving the response efficiency and accuracy of the intelligent customer service system under limited resource conditions.

[0103] It is understood that in the specific implementation of this application, data such as text datasets and question text sets are involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required. Furthermore, the collection, use and processing of related data, as well as the construction, training and use of large language models and text classification models, must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0104] Optionally, the encoding layer includes multiple sequentially connected sub-encoding layers. Each sub-encoding layer includes an attention sub-layer, a residual connection sub-layer, and a normalization layer. The attention sub-layer can be a simplified attention layer or a full attention layer. The simplified attention layer consists of feedforward computation units, while the full attention layer consists of multi-head attention units. In the step of encoding the embedded sequence through the encoding layer to obtain the encoded features, for each sub-encoding layer, if it is the first sub-encoding layer, the input feature is the embedded sequence; if it is not the first sub-encoding layer, the input feature is the feature vector output by the previous sub-encoding layer. When the attention sublayer of the coding layer is a simplified attention layer, the input features are obtained by performing position-by-position linear transformation and non-linear activation through the feedforward computation unit. When the attention sublayer of the sub-coding layer is a complete attention layer, the attention weights between each position in the input features are calculated through the multi-head attention unit, and the weights are applied to the input features to obtain weighted features. The feedforward features or weighted features are superimposed with the input features through the residual connection sublayer, and then scaled through the normalization layer to output the feature vector of the current sub-coding layer. The feature vector output by the last sub-coding layer is used as the encoded features.

[0105] In this embodiment of the invention, the coding layer adopts the ModernBERT structure, which includes multiple sub-coding layers connected in sequence. Each sub-coding layer contains an attention sub-layer, a residual connection sub-layer, and a normalization layer, which are used to perform hierarchical coding modeling of the embedded sequence to generate a deep semantic feature representation.

[0106] Specifically, the attention sublayer can be a simplified attention layer or a full attention layer. The simplified attention layer mainly consists of feedforward computation units, which achieve efficient extraction of semantic features through position-by-position linear transformation and nonlinear activation. In this simplified layer, the model removes the Q, K, and V calculations from the self-attention mechanism and directly replaces them with a feedforward network, achieving a lightweight and efficient computational structure. Its core computation process can be represented as follows:

[0107]

[0108]

[0109] in, , These are the weight matrices of the feedforward network. This is represented by the matrix dimension corresponding to the weight matrix. and For the corresponding bias term, This is a Gaussian error linear unit activation function used to enhance the model's expressive power in nonlinear feature extraction. This structure, through point-to-point feature transformation and layer-by-layer residual connections, significantly reduces redundant computation while maintaining semantic integrity.

[0110] In the full attention layer, the model employs multi-head attention units to perform global attention computation, enabling it to simultaneously capture global dependencies between different positions in the input sequence. By calculating the attention weight matrix between features at each position and applying these weights to the input features to achieve weighted combination, a feature representation incorporating contextual information is generated.

[0111] The entire encoding layer can be stacked in an alternating manner between the two types of sub-layers mentioned above, i.e., the structure FFAFFA…, where F represents a simplified feedforward layer (i.e., a simplified attention layer) and A represents a full attention layer. Specifically, layers 1, 2, 4, 5, 7, and 8 are simplified layers, and layers 3, 6, and 9 are full attention layers. Through this alternating structure, the model significantly improves computational efficiency and memory utilization while maintaining its ability to model the attention mechanism.

[0112] The entire forward propagation process of the ModernBERT encoder can be formally described as follows:

[0113] Let L be the total number of floors, then:

[0114]

[0115]

[0116]

[0117]

[0118] in, Represents the input text sequence; The input is an embedded sequence; Indicates the first The output feature vector of the layer contains the features passed through the previous layer. The contextual semantic information processed by the sub-coding layer; L represents the total number of coding layers; Used to identify the first The type of layer, when equal When this is the case, the layer is a complete attention layer; when... equal At this time, this layer is a simplified attention layer, which only performs feedforward computation; This represents the computational process of the multi-head attention mechanism, which is used to calculate the dependencies between features at different locations globally, thereby extracting global semantic features of the input sequence. This represents the mapping operation of the feedforward network, which is used to perform linear transformations and nonlinear activations on the feature vectors at each location, thereby enhancing the feature representation capability of the model. The representation layer normalization operation is used to standardize the data distribution in the output of each layer, improving training stability and accelerating convergence. This represents the intermediate feature representation obtained by superimposing it with the input residual after multi-head attention or feedforward computation; Residual Connection is used to directly superimpose the input features of the previous layer into the output of the current layer, thereby alleviating the gradient vanishing problem in deep networks and maintaining semantic continuity; The operational form represents feature stacking after feedforward mapping, enabling the model to capture nonlinear features while preserving the original semantics. This represents the final standardized output result, which is the output feature vector of the current sub-coding layer.

[0119] Through the above structure, the ModernBERT encoding layer implements a computational mechanism of "alternating distribution of simplified and complete layers", which reduces the amount of computation while maintaining the global modeling capability of long texts, making the model more efficient and stable in user emotion recognition and classification tasks.

[0120] Furthermore, the aforementioned coding layer can also be achieved through, for example... Figure 3 The schematic diagram of one type of coding layer further illustrates this. Figure 3 In the structural diagram shown, the encoding layer alternates between simplified attention layers and full attention layers to capture text features at different semantic levels.

[0121] Specifically, the "full attention layer" at the bottom of the diagram indicates that the model performs global dependency modeling on all positions of the input sequence in this layer. Each feature unit can pay attention to the contextual information at any position in the sequence. For example, the word "sad" in the user's emotions may have a semantic connection with words such as "anxious" and "angry". In the "simplified layer", the model only performs local calculations on features within the adjacent window range to capture short-range dependencies, thereby reducing computational complexity.

[0122] The attention arrows in the diagram represent the distribution range of attention across different layers: in the full attention layer, the arrow covers the entire sequence, indicating global feature interaction; in the simplified layer, the arrow only covers a local window area, indicating local feature modeling. By alternating these structures, the model significantly improves computational efficiency while preserving global information.

[0123] In addition, from Figure 3 It can also be seen that self-attention computation is removed in certain layers, and only the feedforward computation module is retained to perform linear transformations and nonlinear activation operations. This design allows the model to reduce the computation of Q, K, and V matrices without sacrificing expressive power, thereby reducing memory consumption and computational overhead.

[0124] Overall, Figure 3 The structure shown embodies the hierarchical encoding strategy adopted in this invention, which forms an efficient Transformer structure that balances global understanding and local modeling by alternately stacking full attention layers and simplified feedforward layers. This ensures that when processing user text sentiment classification, it balances the accuracy of long sequence understanding and the efficiency of model inference.

[0125] Optionally, in the step of inputting the encoded features into a character convolutional layer and performing multi-scale character-level convolution to obtain convolutional features, the encoded features can be divided into feature matrices at the character-level granularity; the sliding stride or receptive field range of the feature matrix can be matched and adjusted according to the size of the convolution kernel to obtain the adjusted feature matrix; window-by-window convolution calculations can be performed on the adjusted feature matrix by multiple convolution kernels to extract the convolution response results corresponding to each convolution kernel; and the convolution response results of different convolution kernels can be fused to obtain multi-scale convolutional features.

[0126] In this embodiment of the invention, character convolutional layers are used to extract text features at different scales to capture multi-level feature representations of local character combinations and phrase patterns in the input text. Specifically, the encoded features obtained from the encoding layer can be divided into feature matrices at the character-level granularity to refine the semantic units down to the letter, pinyin, or Chinese character level. For different convolutional kernel sizes, the stride or receptive field range is dynamically adjusted to match the feature extraction granularity with the input text structure, thereby achieving multi-scale feature capture.

[0127] Subsequently, window-by-window convolution computation is performed on the adjusted feature matrix using multiple convolutional kernels to identify local patterns of character combinations within different feature receptive regions. During the sliding computation, the convolutional kernels extract the convolutional response values ​​of the input text vector within a specific window, forming a set of multi-scale convolutional features. For example, small-sized convolutional kernels can capture local features of phrases or sentiment words, while large-sized convolutional kernels can identify the overall trend of sentence structure or tone changes.

[0128] Finally, the convolutional response results from different convolutional kernels are fused to integrate feature information at various scales, forming multi-scale convolutional features with rich semantic expressive power, providing high-dimensional feature input for subsequent attention weighting and pooling layers.

[0129] Specifically, the calculation of convolution can be expressed as:

[0130]

[0131] in, Represents the input vector sequence; Represents the convolution kernel; This is the sliding step size; For bias terms, ; This represents the output of the convolution operation at the y-th window position.

[0132] In convolution operations, the receptive field of the convolution can be flexibly changed by adjusting the kernel size j and stride h, enabling the model to model contextual features at different granularities. For example, small convolution kernels are used to identify local features such as emotional words (e.g., "angry" or "satisfied"), while large convolution kernels can capture global features of the emotional structure and tone changes of the entire sentence.

[0133] Finally, the convolutional responses from different kernels are fused to obtain a convolutional feature matrix that integrates multi-scale features, providing input for subsequent attention weighting and pooling operations. Through multi-scale convolution and granularity adaptation, the model can balance fine-grained character features with overall syntactic information, thereby significantly improving its ability to recognize and classify complex emotional texts.

[0134] Optionally, in the step of inputting the weighted features into the pooling layer for max pooling to obtain the pooled features, the weighted features can be further divided into sliding segments according to a preset window size, and the feature unit with the largest convolution output value within each window is selected as the representative feature of the corresponding window; the representative features of each window are then combined sequentially to form the pooled features.

[0135] In this embodiment of the invention, the pooling layer is used to further filter out the most representative emotion features from the multi-scale features extracted by convolution, so as to reduce feature redundancy and enhance the model's discriminative ability. Specifically, the weighted features are divided into sliding segments according to a preset window size, with each window corresponding to a feature interval in the input sequence.

[0136] Within each window, the output response value of each convolutional unit in that window is calculated, and the feature unit with the largest convolutional output value is selected as the representative feature of that window, thus forming a set of locally optimal feature representations. This method allows for the filtering out of low-weight or noisy information while preserving salient features, enabling subsequent classification layers to focus on key patterns related to emotion.

[0137] The calculation of max pooling can be expressed as:

[0138]

[0139] in, Represents the input vector sequence; This represents the output sequence of the pooling layer; The width of the pooled window; This is the sliding step size; This is a bias term.

[0140] Using the above formula, the model retains only the features corresponding to the maximum response value within each sliding window, thus obtaining pooled features that are sparser but semantically stronger globally.

[0141] Max pooling effectively captures the most salient emotional segments in sentences, such as high-response words like "very satisfied," making the model's classification of user emotions more accurate and stable. Simultaneously, this method improves the model's generalization performance while reducing computational cost, providing more representative input features for subsequent attention-weighted and fully connected classification.

[0142] Optionally, in the step of inputting the target weighted features into the output layer to obtain the classification result of the sample question text, the target weighted features can also be linearly combined with the preset model parameters through the output layer to obtain the class probability of the sample question text; and the classification result can be determined based on the class probability.

[0143] In this embodiment of the invention, the output layer is used to perform final classification on the target weighted features after pooling and attention weighting, in order to determine the emotion category corresponding to the input text. Specifically, the output layer can adopt a fully connected layer structure, which linearly combines the target weighted features with preset model parameters to achieve a mapping from a high-dimensional semantic feature space to a classification space.

[0144] In this process, we first assume that the target weighted feature vector output by the pooling layer is... The model parameters include the weight vector. With bias terms. By linear combination and processing via a non-linear activation function (such as the sigmoid function), the probability that the text belongs to a specific category can be calculated. The calculation formula is as follows:

[0145]

[0146] in, This represents the class probability output by the model; Indicates the model weight parameters; This represents the input feature vector; The probability is an exponential function. The above category probabilities reflect the confidence level of the input text in each emotion category. According to the principle of maximizing probability, the category with the highest probability value can be taken as the final classification result of the text, thereby realizing the emotion classification determination of user question text.

[0147] To further optimize classification performance, a minimum cost function can be used to constrain the model parameters during the model training phase, allowing the model to gradually converge to its optimal state over multiple training rounds. The cost function (i.e., the loss function) can be expressed as:

[0148]

[0149]

[0150]

[0151] in, This is the log-likelihood loss term when the model predicts the true class, used to measure how well the predicted result fits the true class. This is the loss term for when the model predicts a non-true class, used to measure the model's ability to distinguish the wrong class; The overall cost function is used to measure the overall predictive performance of the model by averaging it over all samples. The total number of training samples; Let represent the text feature vector of the i-th sample; This represents the classification label corresponding to the i-th sample.

[0152] In one embodiment, such as Figure 4 As shown, a text generation method is provided, including the following steps:

[0153] 401. Obtain the problem text set;

[0154] 402. Input the question text into the trained text classification model so that the trained text classification model can classify the question text and obtain the classification result of the question text;

[0155] 403. Based on the classification results of each question text, determine the response priority for each question text;

[0156] 404. Determine the response strategy for the question text based on response priority;

[0157] 405. Based on the response priority from high to low, generate the corresponding response text for the question text according to the response strategy, and send the response text back to the corresponding user who needs a response.

[0158] In this embodiment of the invention, the question text set includes multiple question texts from users awaiting responses, and the trained text classification model is obtained through the text classification model training method. The above text generation method can be applied to an intelligent customer service system.

[0159] Specifically, the system first obtains multiple user question texts to be processed, forming a question text set, such as "Why hasn't my order been shipped yet?", "The product works well, I will buy it again next time", "The customer service attitude is terrible", etc.

[0160] Subsequently, the text of each question is input into the trained text classification model. This model, trained using the aforementioned text classification model training method, is able to identify and classify the input question text based on text semantics, sentence structure, sentiment, and expression intensity, obtaining corresponding classification results such as "negative complaint," "neutral consultation," or "positive feedback."

[0161] The system determines the response priority for each question text based on the classification results, so as to prioritize urgent or emotionally charged questions under limited resources. For example, when text with strong negative emotions is detected, the system can automatically mark it as a high-priority task.

[0162] Next, the system matches corresponding response strategies based on different priority question categories. For neutral inquiry questions, a standardized response based on a knowledge base can be automatically generated; for positive feedback questions, thank-you and interactive statements can be generated; and for negative complaint questions, human intervention or a reassuring automatic response can be generated.

[0163] Finally, the system generates response texts for each question in descending order of priority and sends the generated responses back to the corresponding users, thus achieving a fast, accurate, and emotionally-aware automated response process. This method can significantly improve the response quality and user satisfaction of intelligent customer service systems while ensuring service efficiency. The response text is not limited to text format; it can also be converted to voice or other formats and fed back to the user.

[0164] For example, the order of priority for the above responses can be further illustrated by the following table:

[0165] Table 2

[0166]

[0167] As shown in Table 2, it can automatically identify and output corresponding classification labels and priority ranking results based on the semantic and emotional characteristics of different user input texts.

[0168] For example, when someone enters "This product is absolutely terrible! It's unusable, I'm so angry!!###", the model recognizes that it contains the words "terrible" and "angry", which are strong negative emotions. Therefore, it determines that the product is categorized as "negative (angry)" and ranks it as the highest priority 1, so that the system can trigger human customer service intervention first.

[0169] When the input is "Wow, the express delivery was super fast, and the packaging was also very beautiful. I am very satisfied!", the model recognizes the positive emotion words "super fast", "beautiful", and "satisfied" and determines its category label as "positive (happy)", which has a lower ranking priority.

[0170] Through this sorting mechanism, the system can prioritize tasks based on category tags under limited resources, ensuring that user requests with strong emotions and urgent needs are processed immediately, while positive or neutral feedback is processed later. This result not only improves the resource utilization efficiency of the customer service system but also significantly enhances user experience and satisfaction, demonstrating the intelligent scheduling and response capabilities of this invention in a multi-emotion, multi-task concurrent environment.

[0171] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0172] In one embodiment, a training apparatus for a text classification model is provided, which corresponds one-to-one with the training method for the text classification model in the above embodiments. For example... Figure 5 As shown, the training device for this text classification model includes a first acquisition module 501, a first encoding module 502, a first convolution module 503, a first pooling module 504, a first output module 505, and a first iteration module 506. Detailed descriptions of each functional module are as follows:

[0173] The first acquisition module 501 is used to acquire the text classification model to be trained and the text dataset. The text dataset includes sample question texts and classification labels of the sample question texts. The text classification model to be trained includes an embedding layer, an encoding layer, a character convolutional layer, a pooling layer and an output layer.

[0174] The first encoding module 502 is used to convert the sample question text into an embedding sequence through the embedding layer, and to encode the embedding sequence through the encoding layer to obtain encoded features;

[0175] The first convolution module 503 is used to input the encoded features into the character convolution layer, perform multi-scale character-level convolution to obtain convolution features, and calculate the attention weights of each feature unit in the convolution features based on the self-attention mechanism and perform weighted processing to obtain weighted features.

[0176] The first pooling module 504 is used to input the weighted features into the pooling layer for max pooling to obtain pooled features, and adjust the attention weights of each feature unit in the pooled features based on the self-attention mechanism to obtain the target weighted features.

[0177] The first output module 505 is used to input the target weighted features into the output layer to obtain the classification result of the sample question text;

[0178] The first iteration module 506 is used to perform parameter iteration on the text classification model to be trained based on the classification result and the classification label to obtain a trained text classification model.

[0179] Optionally, the coding layer includes multiple sub-coding layers connected in sequence. Each sub-coding layer includes an attention sub-layer, a residual connection sub-layer, and a normalization layer. The attention sub-layer is either a simplified attention layer or a full attention layer. The simplified attention layer is composed of feedforward computation units, and the full attention layer is composed of multi-head attention units. The first coding module 502 is further configured to:

[0180] For each sub-coding layer, if the sub-coding layer is the first sub-coding layer, the input feature is the embedding sequence; if the sub-coding layer is not the first sub-coding layer, the input feature is the feature vector output by the previous sub-coding layer.

[0181] When the attention sublayer of the sub-coding layer is a simplified attention layer, the feedforward features are obtained by performing position-by-position linear transformation and nonlinear activation on the input features through the feedforward computation unit.

[0182] When the attention sublayer of the sub-coding layer is a complete attention layer, the attention weights between each position in the input features are calculated by a multi-head attention unit, and the weights are applied to the input features to obtain weighted features.

[0183] The feedforward features or the weighted features are superimposed on the input features through the residual connection sub-layer, and then scaled through the normalization layer to output the feature vector of the current sub-encoding layer.

[0184] The feature vector output from the last sub-coding layer is used as the encoded feature.

[0185] Optionally, the first convolution module 503 is further configured to:

[0186] The encoded features are divided into feature matrices at the character-level granularity;

[0187] Based on the size of the convolution kernel, the sliding stride or receptive field range of the feature matrix is ​​matched and adjusted to obtain the adjusted feature matrix;

[0188] Window-by-window convolution calculations are performed on the adjusted feature matrix using multiple convolution kernels, and the convolution response results corresponding to each convolution kernel are extracted.

[0189] The convolutional response results of different convolutional kernels are fused to obtain the multi-scale convolutional features.

[0190] Optionally, the first pooling module 504 is further configured to:

[0191] The weighted features are divided into sliding sections according to a preset window size, and the feature unit with the largest convolution output value is selected within each window range as the representative feature of the corresponding window.

[0192] The representative features of each window are combined sequentially to form the pooled features.

[0193] Optionally, the first output module 505 is further configured to:

[0194] The target weighted features are linearly combined with preset model parameters through the output layer to obtain the category probability of the sample question text;

[0195] The classification result is determined based on the category probability.

[0196] Specific limitations regarding the training device for the text classification model can be found in the above section on the limitations of the training method for the text classification model, and will not be repeated here. Each module in the aforementioned training device for the text classification model can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0197] In one embodiment, a text generation apparatus is provided, which corresponds one-to-one with the text generation methods described in the above embodiments. For example... Figure 6 As shown, the text generation device includes a second acquisition module 601, a first input module 602, a first determination module 603, a second determination module 604, and a first generation module 605. Detailed descriptions of each functional module are as follows:

[0198] The second acquisition module 601 is used to acquire a question text set, which includes multiple question texts from users to be answered.

[0199] The first input module 602 is used to input the question text into a trained text classification model so that the trained text classification model can classify the question text and obtain the classification result of the question text. The trained text classification model is trained by a text classification model training method.

[0200] The first determining module 603 is used to determine the response priority for each question text based on the classification result of each question text;

[0201] The second determining module 604 is used to determine the response strategy for the question text based on the response priority;

[0202] The first generation module 605 generates corresponding response texts for the question texts according to the response priority from high to low, based on the response strategy, and sends the response texts back to the corresponding users who need to be responded to.

[0203] For specific limitations regarding the text generation device, please refer to the limitations on the text generation method above, which will not be repeated here. Each module in the aforementioned text generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0204] In one embodiment, a computer device is provided, which may be a terminal device, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, and network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes a readable storage medium storing computer-readable instructions. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer-readable instructions implement a text classification model training method or a text generation method. The readable storage medium provided in this embodiment includes both non-volatile and volatile readable storage media.

[0205] In this application embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor. When the processor executes the computer-readable instructions, it implements the steps of the training method of the text classification model or the text generation method described above.

[0206] In one embodiment of the application, a readable storage medium is provided, which stores computer-readable instructions. When the computer-readable instructions are executed by a processor, they implement the steps of the training method for the text classification model or the text generation method described above.

[0207] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When executed, these computer-readable instructions can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0208] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0209] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A training method for a text classification model, characterized in that, The method includes: Obtain the text classification model to be trained and the text dataset. The text dataset includes sample question texts and classification labels of the sample question texts. The text classification model to be trained includes an embedding layer, an encoding layer, a character convolutional layer, a pooling layer and an output layer. The sample question text is converted into an embedding sequence through the embedding layer, and the embedding sequence is encoded through the encoding layer to obtain encoded features; The encoded features are input into the character convolutional layer to perform multi-scale character-level convolution to obtain convolutional features. The attention weights of each feature unit in the convolutional features are calculated based on the self-attention mechanism and then weighted to obtain weighted features. The weighted features are input into the pooling layer for max pooling to obtain pooled features. The attention weights of each feature unit in the pooled features are adjusted based on the self-attention mechanism to obtain the target weighted features. The target weighted features are input into the output layer to obtain the classification result of the sample question text; Based on the classification results and the classification labels, the parameters of the text classification model to be trained are iterated to obtain the trained text classification model. The encoding layer comprises multiple sequentially connected sub-encoding layers. Each sub-encoding layer includes an attention sub-layer, a residual connection sub-layer, and a normalization layer. The attention sub-layer is either a simplified attention layer or a full attention layer. The simplified attention layer consists of feedforward computation units, and the full attention layer consists of multi-head attention units. The encoding process performed on the embedded sequence through the encoding layer to obtain encoded features includes: For each sub-coding layer, if the sub-coding layer is the first sub-coding layer, the input feature is the embedding sequence; if the sub-coding layer is not the first sub-coding layer, the input feature is the feature vector output by the previous sub-coding layer. When the attention sublayer of the sub-encoding layer is a simplified attention layer, the input features are obtained by performing position-wise linear transformation and non-linear activation through the feedforward computation unit; the simplified attention layer is composed of feedforward computation units, and semantic features are extracted through position-wise linear transformation and non-linear activation; in the simplified attention layer, the model removes the Q, K, V calculations in the self-attention mechanism and replaces them with a feedforward network. When the attention sublayer of the sub-encoding layer is a full attention layer, attention weights between positions in the input features are calculated using multi-head attention units, and these weights are applied to the input features to obtain weighted features. In the full attention layer, the model uses multi-head attention units to perform global attention calculations, which can simultaneously capture global dependencies between different positions in the input sequence. By calculating the attention weight matrix between features at each position and applying the weights to the input features, a weighted combination is achieved, thereby generating a feature representation that incorporates contextual information. The feedforward features or the weighted features are superimposed on the input features through the residual connection sub-layer, and then scaled through the normalization layer to output the feature vector of the current sub-encoding layer; the entire encoding layer stacks two types of sub-layers in an alternating manner, i.e., the structure of FFAFFA..., where F represents a simplified attention layer and A represents a complete attention layer; The feature vector output from the last sub-coding layer is used as the encoded feature.

2. The training method for the text classification model as described in claim 1, characterized in that, The step of inputting the encoded features into the character convolutional layer and performing multi-scale character-level convolution to obtain convolutional features includes: The encoded features are divided into feature matrices at the character-level granularity; Based on the size of the convolution kernel, the sliding stride or receptive field range of the feature matrix is ​​matched and adjusted to obtain the adjusted feature matrix; Window-by-window convolution calculations are performed on the adjusted feature matrix using multiple convolution kernels, and the convolution response results corresponding to each convolution kernel are extracted. The convolutional response results of different convolutional kernels are fused to obtain the convolutional features.

3. The training method for the text classification model as described in claim 1, characterized in that, The step of inputting the weighted features into the pooling layer for max pooling to obtain pooled features includes: The weighted features are divided into sliding sections according to a preset window size, and the feature unit with the largest convolution output value is selected within each window range as the representative feature of the corresponding window. The representative features of each window are combined sequentially to form the pooled features.

4. The training method for the text classification model as described in claim 1, characterized in that, The step of inputting the target weighted features into the output layer to obtain the classification result of the sample question text includes: The target weighted features are linearly combined with preset model parameters through the output layer to obtain the category probability of the sample question text; The classification result is determined based on the category probability.

5. A text generation method, characterized in that, The method includes: Obtain a question text set, which includes question texts from multiple users awaiting responses; The question text is input into a trained text classification model so that the trained text classification model can classify the question text to obtain the classification result of the question text. The trained text classification model is trained by the training method of the text classification model as described in any one of claims 1 to 4. Based on the classification results of each question text, the response priority for each question text is determined; Based on the response priority, determine the response strategy for the question text; According to the response priority from high to low, the corresponding response texts are generated sequentially based on the response strategy, and the response texts are sent back to the corresponding users who need a response.

6. A training device for a text classification model, characterized in that, The device includes: The first acquisition module is used to acquire the text classification model to be trained and the text dataset. The text dataset includes sample question texts and classification labels of the sample question texts. The text classification model to be trained includes an embedding layer, an encoding layer, a character convolutional layer, a pooling layer and an output layer. The first encoding module is used to convert the sample question text into an embedding sequence through the embedding layer, and to encode the embedding sequence through the encoding layer to obtain encoded features; The first convolution module is used to input the encoded features into the character convolution layer, perform multi-scale character-level convolution to obtain convolution features, and calculate the attention weights of each feature unit in the convolution features based on the self-attention mechanism and perform weighted processing to obtain weighted features; The first pooling module is used to input the weighted features into the pooling layer for max pooling to obtain pooled features, and adjust the attention weights of each feature unit in the pooled features based on the self-attention mechanism to obtain the target weighted features. The first output module is used to input the target weighted features into the output layer to obtain the classification result of the sample question text; The first iteration module is used to perform parameter iteration on the text classification model to be trained based on the classification result and the classification label to obtain the trained text classification model. The encoding layer includes multiple sub-encoding layers connected in sequence. Each sub-encoding layer includes an attention sub-layer, a residual connection sub-layer, and a normalization layer. The attention sub-layer is either a simplified attention layer or a full attention layer. The simplified attention layer is composed of feedforward computation units, and the full attention layer is composed of multi-head attention units. The first encoding module is further configured to: For each sub-coding layer, if the sub-coding layer is the first sub-coding layer, the input feature is the embedding sequence; if the sub-coding layer is not the first sub-coding layer, the input feature is the feature vector output by the previous sub-coding layer. When the attention sublayer of the sub-encoding layer is a simplified attention layer, the input features are obtained by performing position-wise linear transformation and non-linear activation through the feedforward computation unit; the simplified attention layer is composed of feedforward computation units, and semantic features are extracted through position-wise linear transformation and non-linear activation; in the simplified attention layer, the model removes the Q, K, V calculations in the self-attention mechanism and replaces them with a feedforward network. When the attention sublayer of the sub-encoding layer is a full attention layer, attention weights between positions in the input features are calculated using multi-head attention units, and these weights are applied to the input features to obtain weighted features. In the full attention layer, the model uses multi-head attention units to perform global attention calculations, which can simultaneously capture global dependencies between different positions in the input sequence. By calculating the attention weight matrix between features at each position and applying the weights to the input features, a weighted combination is achieved, thereby generating a feature representation that incorporates contextual information. The feedforward features or the weighted features are superimposed on the input features through the residual connection sub-layer, and then scaled through the normalization layer to output the feature vector of the current sub-encoding layer; the entire encoding layer stacks two types of sub-layers in an alternating manner, i.e., the structure of FFAFFA..., where F represents a simplified attention layer and A represents a complete attention layer; The feature vector output from the last sub-coding layer is used as the encoded feature.

7. A text generation device, characterized in that, The device includes: The second acquisition module is used to acquire a question text set, which includes question texts from multiple users who need to be answered. The first input module is used to input the question text into a trained text classification model, so that the trained text classification model can classify the question text and obtain the classification result of the question text. The trained text classification model is trained by the training method of the text classification model as described in any one of claims 1 to 4. The first determining module is used to determine the response priority for each question text based on the classification result of each question text; The second determining module is used to determine the response strategy for the question text based on the response priority; The first generation module generates corresponding response texts for the question texts according to the response priority from high to low, based on the response strategy, and sends the response texts back to the corresponding users who need to be responded to.

8. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and running on the processor, characterized in that, When the processor executes the computer-readable instructions, it implements the training method of the text classification model as described in any one of claims 1 to 4 or the text generation method as described in claim 5.

9. A readable storage medium having computer-readable instructions stored thereon, characterized in that, When the computer-readable instructions are executed by the processor, they implement the training method of the text classification model as described in any one of claims 1 to 4 or the text generation method as described in claim 5.

Images