Multi-head attention text classification mechanism and method

Abstract

The invention provides a multi-head attention text classification mechanism and method, and belongs to the technical field of computers, and the method comprises the following steps: 1, data preprocessing; 2, text vectorization: constructing a RoBERTa model according to a self-attention mechanism, multi-head self-attention and a feedforward neural network, training the RoBERTa model, designing a vector dimension of the RoBERTa model, and finally performing forward propagation; 3, applying a multi-head attention mechanism; 4, constructing and training a classifier; step 5, performing hyper-parameter optimization; and 6, deploying and monitoring the model. The accuracy and robustness of text classification are improved by fusing multiple advanced technologies. The comprehensive method is particularly suitable for complex classification tasks needing to accurately understand text semantics and context relations, and the texts of the type can be more effectively recognized and classified.

Description

Technical Field

[0001] The present invention mainly relates to the field of computer technology, and specifically to a multi-head attention text classification mechanism and method. Background Art

[0002] 1. Current state of the art

[0003] Pre-trained language models: Pre-trained language models such as BERT and RoBERTa are widely used in the field of NLP. They handle various text classification tasks through two stages: pre-training and fine-tuning. The BERT model is composed of a stack of Transformer encoders and can learn rich language features for text. RoBERTa is optimized based on BERT and improves its performance on multiple tasks.

[0004] Multi-head attention mechanism: The multi-head attention mechanism is a key part of the Transformer model, which can learn information from different subspaces in parallel and effectively encode long-distance dependencies. This mechanism performs well in processing complex text sequences, especially in understanding local and global dependencies in text.

[0005] Capsule Network: Capsule Network (CapsuleNet) is a neural network structure that aims to address the shortcomings of traditional convolutional neural networks (CNNs) in processing spatial hierarchical information. It better maintains spatial hierarchical relationships and can more effectively recognize complex patterns by using vector neural units and dynamic routing update mechanisms.

[0006] 2. Disadvantages of existing technology

[0007] Pre-trained language models: Although pre-trained language models such as RoBERTa have strong language representation capabilities, in specific tasks such as text classification, relying solely on these models may not be able to fully capture the spatial hierarchy and multi-dimensional dependencies of the text.

[0008] Multi-head attention mechanism: Although the multi-head attention mechanism can effectively encode dependencies within text, its effect still needs to be improved when processing text with semantic transitions or complex spatial structures. This mechanism needs to be combined with other technologies to further improve classification accuracy.

[0009] Capsule Network: Although the traditional capsule network can process spatial hierarchical information well, it cannot selectively focus on the important parts of the text and lacks the ability to encode long-distance dependencies, which has great limitations in identifying text with semantic transitions.

[0010] 3. Causes of shortcomings

[0011] Model structure limitations: The model structure limitations in existing technologies lead to limited effectiveness in handling specific text classification tasks. For example, although capsule networks have advantages in processing spatial hierarchical structures, they lack the ability to selectively focus on important parts of the text.

[0012] Insufficient encoding of long-range dependencies: Many existing techniques fail to effectively encode long-range dependencies in text, resulting in poor classification results when faced with text with complex semantic structures.

[0013] Limitations of a single model: A single model often finds it difficult to take into account multiple aspects such as language representation, spatial structure processing, and multi-dimensional dependency encoding. Therefore, there is still room for improvement in its performance in complex text classification tasks.

[0014] In summary, a multi-head attention text classification mechanism and method aims to overcome the shortcomings of existing technologies and improve the accuracy and robustness of text classification by integrating multiple advanced technologies. This comprehensive method is particularly suitable for complex classification tasks that require precise understanding of text semantics and contextual relationships, and provides a new solution to complex natural language processing problems. Summary of the invention

[0015] The present invention mainly provides a multi-head attention text classification mechanism and method to solve the technical problems raised in the above background technology.

[0016] The technical solution adopted by the present invention to solve the above technical problems is:

[0017] A multi-head attention text classification mechanism and method, comprising the following steps:

[0018] Step 1: Data preprocessing, used to remove irrelevant characters and perform word segmentation;

[0019] Step 2: Text vectorization: construct the RoBERTa model based on the self-attention mechanism, multi-head self-attention, and feedforward neural network, train the RoBERTa model, design the vector dimension of the RoBERTa model, and finally perform forward propagation;

[0020] Step 3: Apply the multi-head attention mechanism, and use the multi-head attention mechanism and multi-head attention mechanism parameters to output the multi-head attention mechanism for subsequent machine learning tasks;

[0021] Step 4: Classifier construction and training, output and train the multi-layer perceptron, optimize the algorithm while training the multi-layer perceptron, and finally evaluate and adjust;

[0022] Step 5: Hyperparameter optimization, define the parameter grid and parameter space, search for the optimal parameters, and obtain the optimized model;

[0023] Step 6: Model deployment and monitoring: Deploy the platform, containerize and service the platform, and monitor various indicators and data of the platform.

[0024] Furthermore, in the present invention, step 1 of performing data preprocessing specifically includes the following steps:

[0025] (1) Remove irrelevant characters:

[0026] Remove HTML tags, remove non-alphanumeric characters, remove extra spaces, and remove stop words;

[0027] (2) Word segmentation: Set the text string to T. The goal of word segmentation is to split T into a list. Word segmentation is a sequence labeling problem, where each character c i Assigned a label l i , indicating whether the character is the beginning, middle, end or non-character;

[0028] Label set: B: start; I: middle; E: end; S: single character; O: non-character; word segmentation can be represented as a function f, which maps a text string T to a label sequence L = {l1,l2,...,l m}:

[0029] f(T)=L

[0030] Where m is the length of the text T in characters, then the list W is constructed based on the label sequence L to construct W by the following steps:

[0031] First, initialize an empty list W, and then traverse the label sequence L.

[0032] Furthermore, in the present invention, the self-attention mechanism in step 2 is:

[0033]

[0034] Where Q, K, and V represent query, key, and value, respectively. They are matrices obtained from the input sequence through linear transformation. k is the dimension of the key vector, used to scale the attention scores to avoid gradient vanishing;

[0035] The multi-head self-attention in step 2 is:

[0036] MultiHead(Q,K,V)=Concat(head1,head2,...,head h )W O

[0037]

[0038] where h is the number of heads, is the linear transformation matrix of queries, keys, and values ​​of different headers, W O is the output transformation matrix;

[0039] The feedforward neural network in step 2 is:

[0040] FFN(x)=max(0,xW1+b1)W2+b2

[0041] Where W1 and W2 are weight matrices, b1 and b2 are bias vectors, and max(0, ) represents the ReLU activation function.

[0042] Furthermore, in the present invention, the specific steps of training the RoBERTa model in step 2 are:

[0043] Step 1: Input representation, convert the input text sequence into an embedding vector and add position encoding;

[0044] Step 2: Transformer layers are stacked to forward propagate the embedding vector through multiple layers of Transformer layers.

[0045] Step 3: Masked language model. In the pre-training stage, the model learns the deep representation of language by predicting randomly masked tokens in the input sequence.

[0046] Step 4: Loss calculation, calculate the cross entropy loss between the predicted token and the actual token;

[0047] Step 5: Back propagation, using optimization algorithms such as gradient descent to update the model parameters;

[0048] Step 6: Repeat Step 2 to Step 5 until the model achieves satisfactory performance on the pre-training task.

[0049] Furthermore, in the present invention, the specific steps of performing text vectorization in step 2 include:

[0050] S1: Text segmentation, function g converts the text T into the input representation X of the model, and divides the text T into or sub-word units;

[0051] S2: Map to vocabulary, map the segmented or sub-word units to the vocabulary of the RoBERTa model to obtain the corresponding token IDs;

[0052] S3: sequence construction, constructing the input sequence, including adding special tokens and sequence padding;

[0053] S4: Embedding representation, converting the token IDs sequence into an embedding vector and adding position encoding: X = g(T);

[0054] Where X is the input representation and g is the input representation function.

[0055] Furthermore, in the present invention, the specific steps of performing forward propagation in step 2 include:

[0056] S1: Output extraction, H = f RoBERTa (X), where H is the hidden state, f RoBERTa It is the forward propagation function of the RoBERTa model, and the output extraction function h extracts the vector representation V of the text from the hidden state H;

[0057] S2: Select the output of a specific layer, select the output of the last layer of the model, or aggregate multiple layers of output;

[0058] S3: Pooling operation, which performs a pooling operation on the hidden state of the sequence to obtain a vector representation of a fixed dimension. This process can be described by a mathematical formula:

[0059] V=h(H)

[0060] Where V is the vector representation of the text, h is the output extraction function; the final vector V is a d-dimensional numerical vector that can be used for subsequent machine learning tasks.

[0061] Furthermore, in the present invention, the output of the multi-head attention mechanism in step 3 is specifically:

[0062] The output is a multi-head attention weighted vector representation V′, which is a d-dimensional numerical vector. This process can be described by a mathematical formula:

[0063]

[0064]

[0065] A i =similarity(Q i ,K i )

[0066] W i =softmax(A i )

[0067] V′ i =W i ·V i

[0068]

[0069] Where similarity represents the similarity measurement function, softmax represents the softmax function, and h represents the number of attention heads.

[0070] Furthermore, in the present invention, the optimization algorithm used in step 4 is:

[0071]

[0072] where w t is the current weight, η is the learning rate, is the gradient of the loss function with respect to the weights;

[0073] The training process can be described by the following mathematical formula:

[0074] Initialize weights: w←init()

[0075] For each training cycle:

[0076] Iterate over the entire dataset:

[0077] For each sample or batch of samples:

[0078] Calculate the prediction results: ← model(x;w)

[0079] Calculate the value of the loss function: L←loss function

[0080] Compute the gradient:

[0081] Update weight: w←optimizationalgorithm(w,g)

[0082] Where x is the input vector and y is the true label. is the prediction result, L is the value of the loss function, w is the weight, and g is the gradient.

[0083] Furthermore, in the present invention, the specific steps of hyperparameter optimization in step 5 are:

[0084] S1: Define parameter grid:

[0085] θ={θ1,θ2,…,θ n}

[0086] θ i = {v i1 ,v i2 ,…,v im}

[0087] where Θ is the parameter space, θ i is the set of values ​​of the ith hyperparameter, v ijis the jth value of the ith hyperparameter;

[0088] S2: Traverse the parameter grid:

[0089] Models = {Model(θ)|θ∈Θ}

[0090] Where Models is the set of all trained models, and Model(θ) is the model trained using the parameter combination θ;

[0091] S3: Evaluate model performance:

[0092] Performance={Metric(Model(θ),Validation Set)|θ∈Θ}

[0093] Performance is the set of performance indicators of all models, and Metric is the calculation function of the performance indicators;

[0094] S4: Select the best parameter combination:

[0095]

[0096] where θ * is the best parameter combination;

[0097] S5: Define parameter space:

[0098] θ={θ1,θ2,…,θ n}

[0099] Select the optimization method:

[0100] Optimizer∈{Grid Search,Random Search,Bayesian Optimization}

[0101] Search for optimal parameters:

[0102] θ * =Optimizer(Θ,Model,Validation Set)

[0103] Model retraining:

[0104] Model * =Retrain(Model,θ * )

[0105] where θ * are the optimized model parameters, Model * is the retrained model.

[0106] Furthermore, in the present invention, in step six, monitoring of various indicators and data of the platform specifically includes: performance indicator monitoring, inference delay monitoring, data drift monitoring, and anomaly detection.

[0107] Compared with the prior art, the present invention has the following beneficial effects:

[0108] 1. Improved the accuracy and robustness of text classification, especially when dealing with complex text classification tasks, it can more accurately understand the text semantics and contextual relationships.

[0109] 2. It overcomes the shortcomings of existing technologies, such as model structure limitations and insufficient encoding of long-distance dependencies. By integrating the advantages of RoBERTa and CapsuleNet, it enhances the model's ability to handle complex semantic structures and multi-dimensional dependencies.

[0110] 3. It provides a comprehensive text classification solution that combines the advantages of pre-trained language models, multi-head attention mechanisms, and capsule networks, and is suitable for complex classification tasks that require precise understanding of text.

[0111] 4. Innovations in text classification mechanisms and methods have been achieved through steps such as data preprocessing, text vectorization, application of multi-head attention mechanism, classifier construction and training, hyperparameter optimization, model deployment and monitoring.

[0112] 5. The model is more effective in processing texts with semantic transitions or complex spatial structures, and can more effectively identify and classify such texts.

[0113] 6. Through hyperparameter optimization and model deployment and monitoring, the performance and stability of the model in the application environment are guaranteed.

[0114] The present invention will be explained in detail below with reference to the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0115] Figure 1 This is a mind map of the multi-head attention text classification mechanism and method based on RoBERTa and CapsuleNet of the present invention;

[0116] Figure 2 Schematic diagram of the multi-head attention mechanism of the present invention;

[0117] Figure 3 It is a schematic diagram of the Transformer model structure of the present invention;

[0118] Figure 4 It is a schematic diagram of the RoBERTa structure of the present invention. DETAILED DESCRIPTION

[0119] To facilitate understanding of the present invention, the present invention will be described more comprehensively below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings, but the present invention can be implemented in different forms and is not limited to the embodiments described in the text. On the contrary, these embodiments are provided to make the content disclosed in the present invention more thorough and comprehensive.

[0120] It should be noted that when an element is referred to as being "fixed on" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used in this article are for illustrative purposes only.

[0121] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which the present invention belongs. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0122] For examples, please refer to the attached Figure 1-4 , a multi-head attention text classification mechanism and method, comprising the following steps:

[0123] Step 1: Data preprocessing, used to remove irrelevant characters and perform word segmentation;

[0124] Step 2: Text vectorization: construct the RoBERTa model based on the self-attention mechanism, multi-head self-attention, and feedforward neural network, train the RoBERTa model, design the vector dimension of the RoBERTa model, and finally perform forward propagation;

[0125] Step 3: Apply the multi-head attention mechanism, and use the multi-head attention mechanism and multi-head attention mechanism parameters to output the multi-head attention mechanism for subsequent machine learning tasks;

[0126] Step 4: Classifier construction and training, output and train the multi-layer perceptron, optimize the algorithm while training the multi-layer perceptron, and finally evaluate and adjust;

[0127] Step 5: Hyperparameter optimization, define the parameter grid and parameter space, search for the optimal parameters, and obtain the optimized model;

[0128] Step 6: Model deployment and monitoring: Deploy the platform, containerize and service the platform, and monitor various indicators and data of the platform.

[0129] Specifically, step 1 of data preprocessing includes the following steps:

[0130] (1) Remove irrelevant characters:

[0131] Remove HTML tags, remove non-alphanumeric characters, remove extra spaces, and remove stop words;

[0132] (2) Word segmentation: Set the text string to T. The goal of word segmentation is to split T into a list. Word segmentation is a sequence labeling problem, where each character c i Assigned a label l i , indicating whether the character is the beginning, middle, end or non-character;

[0133] Label set: B: start; I: middle; E: end; S: single character; O: non-character; word segmentation can be represented as a function f, which maps a text string T to a label sequence L = {l1,l2,...,l m}:

[0134] f(T)=L

[0135] Where m is the length of the text T in characters, then the list W is constructed based on the label sequence L to construct W by the following steps:

[0136] First initialize an empty list W, and then traverse the label sequence L;

[0137] It should be noted that, in this embodiment, for each label l i :If l i is B or S, start a new one; if l i Is I or E, continue with the current state; if l i Is S or E, add the current to the list W; if l i Is O, ignore the current character.

[0138] Specifically, the self-attention mechanism in step 2 is:

[0139]

[0140] Where Q, K, and V represent query, key, and value, respectively. They are matrices obtained from the input sequence through linear transformation. k is the dimension of the key vector, used to scale the attention scores to avoid gradient vanishing;

[0141] The multi-head self-attention in step 2 is:

[0142] MultiHead(Q,K,V)=Concat(head1,head2,...,headh )W O

[0143]

[0144] where h is the number of heads, is the linear transformation matrix of queries, keys, and values ​​of different headers, W O is the output transformation matrix;

[0145] The feedforward neural network in step 2 is:

[0146] FFN(x)=max(0,xW1+b1)W2+b2

[0147] Where W1 and W2 are weight matrices, b1 and b2 are bias vectors, and max(0, ) represents the ReLU activation function.

[0148] Specifically, the specific steps for training the RoBERTa model in step 2 are:

[0149] Step 1: Input representation, convert the input text sequence into an embedding vector and add position encoding;

[0150] Step 2: Transformer layers are stacked to forward propagate the embedding vector through multiple layers of Transformer layers.

[0151] Step 3: Masked language model. In the pre-training stage, the model learns the deep representation of language by predicting randomly masked tokens in the input sequence.

[0152] Step 4: Loss calculation, calculate the cross entropy loss between the predicted token and the actual token;

[0153] Step 5: Back propagation, using optimization algorithms such as gradient descent to update the model parameters;

[0154] Step 6: Repeat Step 2 to Step 5 until the model achieves satisfactory performance on the pre-training task;

[0155] It should be noted that, in this embodiment, the pre-trained RoBERTa model can be expressed as a function f RoBERTa , which maps the input text T to a high-dimensional vector V:

[0156] V=f RoBERTa (T)

[0157] where f RoBERTa It is obtained by pre-training on a large-scale text corpus.

[0158] Implementation of pre-trained models:

[0159] f RoBERTa (θ)=Load(D,θ)

[0160] Where Load is a function that represents the process of loading the weights of the pre-trained model, D represents the pre-trained corpus, and θ represents the parameters of the pre-trained model.

[0161] The vector dimensions of the RoBERTa model are:

[0162] In the RoBERTa model, the vector dimension d is usually the same as the hidden layer dimension of the model. The two configurations of the RoBERTa model are:

[0163] RoBERTa-base: The hidden layer dimension is 768.

[0164] RoBERTa-large: The hidden layer dimension is 1024.

[0165] These dimensions are determined during the pre-training process and are fixed when the model is loaded. When choosing to use a pre-trained RoBERTa model, the vector dimension d is usually fixed.

[0166] Implementation of setting vector dimensions:

[0167] V d =f linear (V base )

[0168] Where V base is the original output vector of the pre-trained model, f linear is a linear transformation function used to transform the vector dimension from d base Adjust to d, V d is the adjusted output vector.

[0169] Specifically, the specific steps for text vectorization in step 2 include:

[0170] S1: Text segmentation, function g converts the text T into the input representation X of the model, and divides the text T into or sub-word units;

[0171] S2: Map to vocabulary, map the segmented or sub-word units to the vocabulary of the RoBERTa model to obtain the corresponding token IDs;

[0172] S3: sequence construction, constructing the input sequence, including adding special tokens and sequence padding;

[0173] S4: Embedding representation, converting the token IDs sequence into an embedding vector and adding position encoding: X = g(T);

[0174] Where X is the input representation and g is the input representation function.

[0175] Specifically, the specific steps of forward propagation in step 2 include:

[0176] S1: Output extraction, H = f RoBERTa (X), where H is the hidden state, f RoBERTa It is the forward propagation function of the RoBERTa model, and the output extraction function h extracts the vector representation V of the text from the hidden state H;

[0177] S2: Select the output of a specific layer, select the output of the last layer of the model, or aggregate multiple layers of output;

[0178] S3: Pooling operation, which performs a pooling operation on the hidden state of the sequence to obtain a vector representation of a fixed dimension. This process can be described by a mathematical formula:

[0179] V=h(H)

[0180] Where V is the vector representation of the text, h is the output extraction function; the final vector V is a d-dimensional numerical vector that can be used for subsequent machine learning tasks.

[0181] Specifically, the output of the multi-head attention mechanism in step 3 is:

[0182] The output is a multi-head attention weighted vector representation V′, which is a d-dimensional numerical vector. This process can be described by a mathematical formula:

[0183]

[0184] A i =similarity(Q i ,K i )

[0185] W i =softmax(A i )

[0186] V′ i =W i ·V i

[0187]

[0188] Where similarity represents the similarity measurement function, softmax represents the softmax function, and h represents the number of attention heads;

[0189] It should be noted that in this embodiment, the multi-head attention mechanism:

[0190] Divide the attention heads: Divide the input vector V into h attention heads, each head focuses on a subspace of the input vector. The dimension of each head is d / h.

[0191] Linear transformation: Perform a linear transformation on the input vector of each attention head to obtain the transformed query vector Q i , key vector K i Sum value vector V i , linear transformations can use learnable weight matrices and to achieve.

[0192] Calculate attention score: For each attention head, calculate the attention score A i , the attention score is the query vector Q i and the key vector K i The similarity between them can be calculated using the dot product or other similarity metrics.

[0193] Apply soft max function: Apply soft max function to the attention score of each attention head to get the attention weight W i , which ensures that the sum of the weights of each attention head is 1.

[0194] Weighted sum: sum the value vector V of each attention head i And the corresponding attention weight W i Multiply them, and then add the results of all heads to get the final weighted vector representation V′.

[0195] Multi-head attention mechanism parameters:

[0196] Number of attention heads h: Set the number of attention heads to control the number of different subspaces of the input vector that the model focuses on.

[0197] Weight adjustment: Adjust the weight matrix in the attention mechanism and To control the influence of each attention head on the input vector subspace.

[0198] Specifically, the optimization algorithm used in step 4 is:

[0199]

[0200] where w t is the current weight, η is the learning rate, is the gradient of the loss function with respect to the weights;

[0201] The training process can be described by the following mathematical formula:

[0202] Initialize weights: w←init()

[0203] For each training cycle:

[0204] Iterate over the entire dataset:

[0205] For each sample or batch of samples:

[0206] Calculate the prediction results: ← model(x;w)

[0207] Calculate the value of the loss function: L←loss function

[0208] Compute the gradient:

[0209] Update weight: w←optimizationalgorithm(w,g)

[0210] Where x is the input vector and y is the true label. is the prediction result, L is the value of the loss function, w is the weight, and g is the gradient.

[0211] It should be noted that, in this embodiment, the input is: the vector representation V′ after multi-head attention weighting, which is a d-dimensional numerical vector.

[0212] Multilayer Perceptron: The input layer is the first layer of the MLP and it receives the input vector V′. The number of neurons in the input layer is the same as the dimension of the input vector, i.e. d.

[0213] Hidden layer: The layer between the input layer and the output layer in an MLP.

[0214] Number of neurons: Each layer has a different number of neurons.

[0215] Activation function:

[0216] ReLU (Rectified Linear Unit): ReLU(x)=max(0,x)

[0217] Sigmoid:

[0218]

[0219] Tanh:

[0220]

[0221] Output layer:

[0222]

[0223] where x i is the output of the ith neuron, and n is the number of neurons.

[0224] Multilayer Perceptron output:

[0225] MLP(V′)=Softmax(W n Activation(W n-1 ·…·Activation(W1·V′

[0226] +b1)+…)+b n )

[0227] Where W i and b i They represent the weight matrix and bias vector of the i-th layer respectively, Activation represents the activation function, Softmax represents the Softmax activation function, and n represents the number of layers of MLP.

[0228] Multilayer Perceptron training:

[0229]

[0230] Where L is the loss value, y i is the true label of the i-th sample, is the predicted probability of the ith sample, and n is the number of samples.

[0231] Specifically, the specific steps of hyperparameter optimization in step 5 are:

[0232] S1: Define parameter grid:

[0233] θ={θ1,θ2,…,θ n}

[0234] θ i = {v i1 ,v i2 ,…,v im}

[0235] where Θ is the parameter space, θ i is the set of values ​​of the ith hyperparameter, v ij is the jth value of the ith hyperparameter;

[0236] S2: Traverse the parameter grid:

[0237] Models = {Model(θ)|θ∈Θ}

[0238] Where Models is the set of all trained models, and Model(θ) is the model trained using the parameter combination θ;

[0239] S3: Evaluate model performance:

[0240] Performance={Metric(Model(θ),Validation Set)|θ∈Θ}

[0241] Performance is the set of performance indicators of all models, and Metric is the calculation function of the performance indicators;

[0242] S4: Select the best parameter combination:

[0243]

[0244] where θ * is the best parameter combination;

[0245] S5: Define parameter space:

[0246] θ={θ1,θ2,…,θ n}

[0247] Select the optimization method:

[0248] Optimizer∈{Grid Search,Random Search,Bayesian Optimization}

[0249] Search for optimal parameters:

[0250] θ * =Optimizer(Θ,Model,Validation Set)

[0251] Model retraining:

[0252] Model * =Retrain(Model,θ * )

[0253] where θ * are the optimized model parameters, Model * is the retrained model;

[0254] It should be noted that, in this embodiment, the subsequent evaluation and adjustment:

[0255] Accuracy:

[0256]

[0257] Recall:

[0258]

[0259] F1 score:

[0260]

[0261] Among them, TP (True Positive) represents the number of samples correctly predicted as positive, FP (False Positive) represents the number of samples incorrectly predicted as positive, and FN (False Negative) represents the number of samples incorrectly predicted as negative.

[0262] Specifically, in step six, various indicators and data of the platform are monitored, including: performance indicator monitoring, inference delay monitoring, data drift monitoring, and anomaly detection;

[0263] It should be noted that, in this embodiment,

[0264] Deployment Platform:

[0265]

[0266] Among them, Platform is the selected deployment platform, Utility is the utility function of the platform, and Requirements are the requirements.

[0267] Containerization and service-oriented:

[0268] Service = Containerize (Model)

[0269] Deploy Service=Orchestrate(Service)

[0270] Service is a containerized service, and Deploy Service is a deployed service.

[0271] API interface:

[0272] API=Define(Request,Response)

[0273] ImplementAPI=Develop(API,Framework)

[0274] Among them, API is the defined API and ImplementAPI is the implemented API.

[0275] Model loading and inference:

[0276] Load Model = Load(Model)

[0277] Inference = Model (Input Data)

[0278] Load Model is the loaded model, and Inference is the inference result.

[0279] Further, performance indicator monitoring:

[0280] Performance Metrics={Accuracy,Recall,F1 Score,…}

[0281] Monitor Metrics=Measure(Performance Metrics)

[0282] Analyze Metrics=Analyze(Monitor Metrics)

[0283] Performance Metrics is a collection of performance metrics, Monitor Metrics is the monitored performance metrics, and Analyze Metrics is the result of analyzing performance metrics.

[0284] Inference latency monitoring:

[0285] Inference Latency=Time(Model Inference)

[0286] Monitor Latency=Measure(Inference Latency)

[0287] Analyze Latency=Analyze(Monitor Latency)

[0288] Inference Latency is the inference latency, Monitor Latency is the monitored inference latency, and Analyze Latency is the result of analyzing the inference latency.

[0289] Data drift monitoring:

[0290] Data Drift Metrics={Distribution Distance,Statistical Tests,…}

[0291] Monitor Drift=Measure(Data Drift Metrics)

[0292] Analyze Drift=Analyze(Monitor Drift)

[0293] Data Drift Metrics is a collection of data drift metrics, Monitor Drift is the monitored data drift metric, and Analyze Drift is the result of analyzing data drift.

[0294] Anomaly Detection:

[0295] Anomaly Detection Metrics={Statistical Tests,Machine LearningScores,…}

[0296] MonitorAnomalies=Measure(Anomaly Detection Metrics)

[0297] Analyze Anomalies=Analyze(MonitorAnomalies)

[0298] Anomaly Detection Metrics is a collection of anomaly detection metrics, MonitorAnomalies is the monitored anomaly detection metrics, and Analyze Anomalies is the analysis of anomaly detection results.

[0299] The present invention has the following advantages:

[0300] (1) Model structure innovation: This patent proposes a text classification mechanism that combines the advantages of RoBERTa and CapsuleNet. Through the multi-head attention mechanism and the spatial hierarchical structure processing capability of the capsule network, it overcomes the limitations of a single model in processing complex text classification tasks;

[0301] (2) Data preprocessing technology: The patent describes detailed data preprocessing steps, including removing irrelevant characters and word segmentation, which are crucial to improving the accuracy and robustness of text classification;

[0302] (3) Text vectorization method: The patent describes in detail how to use the RoBERTa model for text vectorization, including the self-attention mechanism and output extraction function. These technologies ensure the full representation of text information.

[0303] (4) Application of multi-head attention mechanism: The patent proposes the application of multi-head attention mechanism. By dividing the attention heads and applying the attention mechanism, it can more effectively process long-distance dependencies in the text;

[0304] (5) Classifier construction and training: The patent describes how to build and train a classifier, including steps such as hyperparameter optimization and model retraining, which are critical to improving model performance and adapting to specific application scenarios;

[0305] (6) Model deployment and monitoring: The patent mentions the importance of model deployment and monitoring, including inference latency monitoring, performance indicator monitoring, anomaly detection indicator monitoring, etc. These measures ensure the stability and maintainability of the model in practical applications;

[0306] (7) Performance evaluation and tuning: The patent describes how to use indicators such as accuracy, recall, and F1 score to evaluate model performance, and optimize model performance by adjusting parameters and model structure.

[0307] The above is an exemplary description of the present invention in combination with the accompanying drawings. It is obvious that the specific implementation of the present invention is not limited to the above-mentioned method. As long as such non-substantial improvements are made by adopting the method concept and technical solution of the present invention, or the concept and technical solution of the present invention are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.

Claims

1. A multi-head attention text classification mechanism and method, characterized in that: The following steps are involved: Step 1: Data preprocessing, used to remove irrelevant characters and perform word segmentation; Step 2: Text vectorization: construct the RoBERTa model based on the self-attention mechanism, multi-head self-attention, and feedforward neural network, train the RoBERTa model, design the vector dimension of the RoBERTa model, and finally perform forward propagation; Step 3: Apply the multi-head attention mechanism, and use the multi-head attention mechanism and multi-head attention mechanism parameters to output the multi-head attention mechanism for subsequent machine learning tasks; Step 4: Classifier construction and training, output and train the multi-layer perceptron, optimize the algorithm while training the multi-layer perceptron, and finally evaluate and adjust; Step 5: Hyperparameter optimization, define the parameter grid and parameter space, search for the optimal parameters, and obtain the optimized model; Step 6: Model deployment and monitoring: Deploy the platform, containerize and service the platform, and monitor various indicators and data of the platform.

2. A multi-head attention text classification mechanism and method according to claim 1, characterized in that: Step 1: Data preprocessing includes the following steps: (1) Remove irrelevant characters: Remove HTML tags, remove non-alphanumeric characters, remove extra spaces, and remove stop words; (2) Word segmentation: Set the text string to T. The goal of word segmentation is to split T into a list. Word segmentation is a sequence labeling problem, where each character c i Assigned a label l i , indicating whether the character is the beginning, middle, end or non-character; label set: B: beginning; I: middle; E: end; S: single character; O: non-character; word segmentation can be represented as a function f, which maps the text string T to a label sequence L = {l1,l2,...,l m }: f(T)=L Where m is the length of the text T in characters, then the list W is constructed based on the label sequence L to construct W by the following steps: First, initialize an empty list W, and then traverse the label sequence L.

3. A multi-head attention text classification mechanism and method according to claim 1, characterized in that: The self-attention mechanism in step 2 is: Where Q, K, and V represent query, key, and value, respectively. They are matrices obtained from the input sequence through linear transformation. k is the dimension of the key vector, used to scale the attention scores to avoid gradient vanishing; The multi-head self-attention in step 2 is: MultiHead(Q,K,V)=Concat(head1,head2,...,head h )W O where h is the number of heads, is the linear transformation matrix of queries, keys, and values ​​of different headers, W O is the output transformation matrix; The feedforward neural network in step 2 is: FFN(x)=max(0,xW1+b1)W2+b2 Where W1 and W2 are weight matrices, b1 and b2 are bias vectors, and max(0, ) represents the ReLU activation function.

4. A multi-head attention text classification mechanism and method according to claim 3, characterized in that: The specific steps for training the RoBERTa model in step 2 are: Step 1: Input representation, convert the input text sequence into an embedding vector and add position encoding; Step 2: Transformer layers are stacked to forward propagate the embedding vector through multiple layers of Transformer layers. Step 3: Masked language model. In the pre-training stage, the model learns the deep representation of language by predicting randomly masked tokens in the input sequence. Step 4: Loss calculation, calculate the cross entropy loss between the predicted token and the actual token; Step 5: Back propagation, using optimization algorithms such as gradient descent to update the model parameters; Step 6: Repeat Step 2 to Step 5 until the model achieves satisfactory performance on the pre-training task.

5. A multi-head attention text classification mechanism and method according to claim 1, characterized in that: The specific steps for text vectorization in step 2 include: S1: Text segmentation, function g converts the text T into the input representation X of the model, and divides the text T into or sub-word units; S2: Map to vocabulary, map the segmented or sub-word units to the vocabulary of the RoBERTa model to obtain the corresponding token IDs; S3: sequence construction, constructing the input sequence, including adding special tokens and sequence padding; S4: Embedding representation, converting the token IDs sequence into an embedding vector and adding position encoding: X = g(T); Where X is the input representation and g is the input representation function.

6. A multi-head attention text classification mechanism and method according to claim 1, characterized in that: The specific steps of forward propagation in step 2 include: S1: Output extraction, H = f RoBERTa (X), where H is the hidden state, f RoBERTa It is the forward propagation function of the RoBERTa model, and the output extraction function h extracts the vector representation V of the text from the hidden state H; S2: Select the output of a specific layer, select the output of the last layer of the model, or aggregate multiple layers of output; S3: Pooling operation, which performs a pooling operation on the hidden state of the sequence to obtain a vector representation of a fixed dimension. This process can be described by a mathematical formula: V=h(H) Where V is the vector representation of the text, h is the output extraction function; the final vector V is a d-dimensional numerical vector that can be used for subsequent machine learning tasks.

7. A multi-head attention text classification mechanism and method according to claim 1, characterized in that: The output of the multi-head attention mechanism in step 3 is: The output is a vector representation V weighted by multi-head attention ′ , which is a d-dimensional numerical vector, this process can be described by a mathematical formula: A i =similarity(Q i ,K i ) W i =softmax(A i ) V′ i =W i ·V i Where similarity represents the similarity measurement function, softmax represents the softmax function, and h represents the number of attention heads.

8. A multi-head attention text classification mechanism and method according to claim 1, characterized in that: The optimization algorithm used in step 4 is: where w t is the current weight, η is the learning rate, is the gradient of the loss function with respect to the weights; The training process can be described by the following mathematical formula: Initialize weights: w←init() For each training cycle: Iterate over the entire dataset: For each sample or batch of samples: Calculate the prediction results: ← model(x;w) Calculate the value of the loss function: L←loss function Compute the gradient: Update weight: w←optimizationalgorithm(w,g) Where x is the input vector and y is the true label. is the prediction result, L is the value of the loss function, w is the weight, and g is the gradient.

9. A multi-head attention text classification mechanism and method according to claim 1, characterized in that: The specific steps of hyperparameter optimization in step 5 are: S1: Define parameter grid: Θ={θ1,θ2,…,θ n } θ i ={v i1 ,v i2 ,…,v im } where Θ is the parameter space, θ i is the set of values ​​of the ith hyperparameter, v ij is the jth value of the ith hyperparameter; S2: Traverse the parameter grid: Models = {Model(θ)|θ∈Θ} Where Models is the set of all trained models, and Model(θ) is the model trained using the parameter combination θ; S3: Evaluate model performance: Performance={Metric(Model(θ),Validation Set)|θ∈Θ} Performance is the set of performance indicators of all models, and Metric is the calculation function of the performance indicators; S4: Select the best parameter combination: where θ * is the best parameter combination; S5: Define parameter space: Θ={θ1,θ2,…,θ n } Select the optimization method: Optimizer∈{Grid Search,Random Search,Bayesian Optimization} Search for optimal parameters: θ * =Optimizer(Θ,Model,Validation Set) Model retraining: Model * =Retrain(Model,θ * ) where θ * are the optimized model parameters, Model * is the retrained model.

10. A multi-head attention text classification mechanism and method according to claim 1, characterized in that: In step six, various indicators and data of the platform are monitored, including: Performance indicator monitoring, inference latency monitoring, data drift monitoring, and anomaly detection.

Images