Emotion detection model training method and system based on Transform architecture and storage medium

Abstract

The invention relates to the technical field of emotion analysis, in particular to an emotion detection model training method and system based on Transform architecture and a storage medium. The emotion detection model training method based on the Transform architecture provided by the invention has an excellent self-attention mechanism, and can process complex language features and emotion expressions. According to the method, at least two data enhancement operations are defined, independent application probability and adjustment parameters are set for each operation, and the enhancement operations are dynamically selected and applied in the training process, so that the diversity of training data can be increased in a more targeted manner, and the model is more accurate and better in robustness when facing different emotion expression modes. By periodically calculating emotion prediction results and confidence difference values on a verification set and adjusting initial hyper-parameters and updating an enhanced data set according to the difference values, the mechanism enables the model to be adaptively optimized and continuously improved, and high efficiency of the training process is ensured.

Description

Technical Field

[0001] This invention relates to the field of sentiment analysis technology, and in particular to a method, system and storage medium for training a sentiment detection model based on the Transformer architecture. Background Technology

[0002] Sentiment analysis is a crucial task in the field of natural language processing, aiming to extract expressions of human emotions or feelings, such as positive, negative, or neutral sentiments, from text. With the proliferation of text data from social media and online comments, sentiment analysis is playing an increasingly important role in various fields, such as public opinion monitoring, market analysis, and customer service.

[0003] Sentiment analysis has made significant progress, driven by NLP (Natural Language Processing) and machine learning. Traditional methods typically rely on rule-based systems or simple machine learning algorithms, but these methods have limitations in understanding context and handling complex emotional expressions. With the development of deep learning technology, especially the emergence of advanced models such as BERT (Bidirectional Encoder Representations from Transformers), the analysis of text data has become more precise, better able to capture contextual relationships and subtle emotional cues. However, in practical applications, the distribution of sentiment labels is often unbalanced. For example, the number of texts with positive sentiment may be far greater than those with negative or neutral sentiment. This data imbalance may cause models to favor predicting more frequently occurring sentiment types, thus affecting their generalization ability and accuracy. Furthermore, deep learning models, especially the Transformer architecture, are prone to overfitting due to their large number of parameters and complex structure when there are few samples or the data quality is poor. Therefore, how to improve the accuracy of models across diverse sentiment categories remains a pressing challenge. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a method, system, and storage medium for training an emotion detection model based on the Transformer architecture.

[0005] To address the aforementioned technical problems, this invention provides the following technical solution: a method for training an emotion detection model based on the Transformer architecture, comprising the following steps: Build an initial model based on the Transformer architecture and set the initial hyperparameters; Obtain a standardized dataset with sentiment annotations and confidence scores, and divide the standardized dataset into a training set and a validation set according to a preset ratio; Configure at least two data augmentation operations, each associated with an independent application probability and adjustment parameter. Dynamically select a combination of data augmentation operations based on the application probability, and perform augmentation operations on the text in the training set according to the adjustment parameter of the selected combination to generate an augmented dataset. Iteratively execute the following steps until the validation set metrics reach the preset standard, and output the final model: The initial model is trained using the current reinforcement dataset, and sentiment prediction results are generated through a self-attention mechanism. The difference between the sentiment prediction results and the corresponding confidence levels of the validation set is calculated periodically. When the difference exceeds a preset threshold, the initial hyperparameters are updated based on the difference gradient. The application probability and adjustment parameters are optimized according to the text type and sentiment distribution to generate a new generation of reinforcement dataset.

[0006] Preferably, updating the initial hyperparameters based on the difference value gradient includes the following steps: The Softmax activation function is used to transform the sentiment prediction results into a prediction probability distribution for each sentiment category; The difference between the predicted probability distribution and the confidence level is calculated using the cross-entropy loss function, and the loss value is output. The gradient of the loss value with respect to each initial hyperparameter is calculated using the backpropagation algorithm, and the initial hyperparameters are updated using the gradient descent algorithm.

[0007] Preferably, the data augmentation operations include: randomly replacing words or phrases, randomly inserting words or phrases, randomly swapping the positions of words or phrases, randomly deleting words or phrases, randomly shuffling sentence order, and data back-translation on the text in the training set.

[0008] Preferably, the initial model constructed based on the Transformer architecture is any one of the following: a pre-trained BERT model, a multilingual BERT model, a lightweight BERT model, a RoBERTa model, or an XLNet model.

[0009] Preferably, the standardized dataset after annotation is divided into a training set, a validation set, and a test set in a ratio of 7:2:1, and the final model is applied to the test set to test its generalization ability.

[0010] Preferably, the initial hyperparameters include the learning rate. The first N rounds of the initial model training execute a Warmup phase, gradually increasing the learning rate from the initial value until a preset maximum learning rate is reached. After the Warmup phase is completed, the learning rate is gradually decreased according to a linear decay strategy until training ends, with the decrease process following a set decay path.

[0011] Preferably, training the initial model using the current reinforcement dataset includes the following steps: The enhanced dataset is divided into multiple subsets according to a preset number; The text is segmented into units by batch, and a unique index tag is assigned to each unit. At the same time, the position information of the index tag in the standardized dataset is recorded. The index tags are mapped to embedding vectors, and word embeddings, sentence identifier embeddings, and position embeddings are superimposed. The embedding vectors are then used as input to the initial model.

[0012] To solve the above-mentioned technical problems, the present invention provides another technical solution as follows: a sentiment detection system based on the Transformer architecture, including a labeling and partitioning module, which acquires a standardized dataset with sentiment labels and confidence scores, and divides the standardized dataset into a training set and a validation set according to a preset ratio; The data augmentation module is configured with at least two data augmentation operations, each associated with an independent application probability and adjustment parameters. It dynamically selects a combination of data augmentation operations based on the application probabilities and performs augmentation operations on the text in the training set according to the adjustment parameters of the selected combination to generate an augmented dataset. The model building and training module is used to build an initial model based on the Transformer architecture, set initial hyperparameters, train the initial model using the current reinforcement dataset, and generate sentiment prediction results through a self-attention mechanism. The verification module is used to periodically calculate the difference between the sentiment prediction results and the corresponding confidence levels of the verification set. When the difference exceeds a preset threshold, the initial hyperparameters are updated based on the difference gradient. The application probability and adjustment parameters are optimized according to the text type and sentiment distribution to generate a new generation of reinforcement dataset. The output module is used to output the final model when the validation set metrics reach the preset standard.

[0013] Preferably, the emotion detection system also includes a trend analysis module for incorporating time series analysis, and on this basis, combining hybrid expert adaptive matching optimization and Bayesian statistical principles.

[0014] To solve the above-mentioned technical problems, the present invention provides another technical solution as follows: a computer-readable storage medium storing a computer program thereon, wherein when the computer program is executed by a processor, it implements the above-mentioned training method for the emotion detection model based on the Transformer architecture.

[0015] Compared with existing technologies, the emotion detection model training method, system, and storage medium based on the Transformer architecture provided by this invention have the following beneficial effects: 1. This invention presents a training method for a sentiment detection model based on the Transformer architecture. The initial model is constructed using the Transformer architecture, which inherently possesses an excellent self-attention mechanism, enabling it to handle complex linguistic features and emotional expressions, thereby improving the model's understanding of text sentiment analysis. Standardized datasets reduce data quality uncertainty, making the model more stable during training. Dataset partitioning ensures the model's generalization ability, allowing it to perform well not only on training data but also maintain high predictive performance on unseen data. Setting a validation set allows for periodic evaluation of model performance and avoids overfitting. Defining at least two data augmentation operations and assigning independent application probabilities and adjustment parameters for each operation, dynamically selecting and applying augmentation operations during training, can more effectively increase the diversity of training data, making the model more accurate and robust when facing different emotional expressions. Periodically calculating the sentiment prediction results and confidence differences on the validation set, and adjusting the initial hyperparameters and updating the augmentation dataset based on these differences, allows the model to adaptively optimize and continuously improve, ensuring the efficiency and accuracy of the training process. The method's dynamic adjustment of the training process can reduce the impact of noise and bias in the data on the model, thereby improving the accuracy and stability of emotion detection.

[0016] 2. The emotion detection model training method based on the Transformer architecture of this invention effectively transforms the model output into probability distributions for various emotion types by employing the Softmax activation function, thereby improving the model's ability to distinguish between different emotion types. Using the cross-entropy loss function to calculate the difference between predictions and confidence scores accurately reflects the degree of error in model predictions, thus enabling effective optimization. The backpropagation algorithm is used to calculate gradients and update training parameters, ensuring that the model training process conforms to the optimization objective, thereby achieving accurate identification of emotion detection tasks.

[0017] 3. The emotion detection model training method based on the Transformer architecture of this invention employs at least two data augmentation operations on the text in the training set, including random word or phrase replacement, random word or phrase insertion, random word or phrase swapping, random word or phrase deletion, random sentence shuffling, and data back-translation, to generate an augmented dataset. These data augmentation operations significantly enhance the diversity of the training dataset, improving the model's robustness and generalization ability. These operations effectively simulate the variability of language in reality, improving the model's accuracy and flexibility in handling different language expressions, and reducing performance degradation caused by insufficient training data or overfitting. Simultaneously, data augmentation methods can help improve model performance in low-resource tasks, especially in situations where data is scarce, by augmenting the dataset to fill knowledge gaps.

[0018] 4. The emotion detection model training method based on the Transformer architecture of this invention constructs an initial model based on the Transformer architecture, which can be any one of the following: a pre-trained BERT model, a multilingual BERT model, a lightweight BERT model, a RoBERTa model, or an XLNet model. The BERT model is suitable for natural language processing tasks, such as text classification and question answering systems; the lightweight BERT model is suitable for resource-constrained environments; RoBERTa is an improved version of BERT, better adapting to large-scale text data and exhibiting better generalization ability on large-scale training sets; the XLNet model combines the advantages of autoregressive and autoencoder models, possessing outstanding text generation and understanding capabilities. This diversity of models provides flexible choices for different application scenarios.

[0019] 5. The emotion detection model training method based on the Transformer architecture of this invention divides the standardized dataset after labeling into training, validation, and test sets in a 7:2:1 ratio, and applies the final model to the test set to test its generalization ability. The 70% training set, 20% validation set, and 10% test set ratio ensures a balanced learning, optimization, and evaluation of the initial model at different stages. 70% of the training set is used to optimize model parameters, enabling it to fully learn data features and improve predictive ability; 20% of the validation set is used for initial hyperparameter tuning and early stopping to prevent overfitting and ensure the model doesn't lose generalization ability by only memorizing training data; and 10% of the test set is used for independent evaluation to measure the model's performance on completely unseen data, avoiding over-reliance on the validation set and affecting performance in real-world environments. This division balances the needs of model training, optimization, and evaluation, improving model stability and ensuring good performance in real-world applications, rather than achieving high accuracy on training data but poor performance on new data.

[0020] 6. The emotion detection model training method based on the Transformer architecture of this invention includes an initial hyperparameter, the learning rate. During the first N rounds of initial model training, a Warmup phase is executed, gradually increasing the learning rate from the initial value until a preset maximum learning rate is reached. After the Warmup phase, the learning rate is gradually decreased according to a linear decay strategy until training ends, with the decrease following a predetermined decay path. In the early stages of training, gradually increasing the learning rate through the Warmup mechanism avoids gradient explosion or training instability caused by an excessively high initial learning rate, helping the model smoothly enter the training state and reducing instability during training. In the later stages of training, the linear decay strategy gradually decreases the learning rate, helping the model finely adjust parameters as it approaches the global optimum, avoiding early trapping in local optima and improving the model's global optimization capability. Combining the Warmup and linear decay strategies allows the model to transition more smoothly between different learning rate phases during training, which not only helps stabilize the training process but also results in more refined and stable model performance.

[0021] 7. The Transformer-based sentiment detection model training method of this invention divides the dataset into multiple subsets for batch word segmentation, which improves the efficiency of the word segmentation process and avoids memory overload or slow processing speed by batch processing. Furthermore, batch processing makes the model more stable during training, improves training efficiency, and facilitates parallel processing, further enhancing the overall performance of the system. Word segmentation transforms text into unit forms that the model can understand, and by assigning index tags to each unit, the model can process based on position, preserving sentence structure information. Overlaying word embeddings, sentence tag embeddings, and positional embeddings enhances the model's ability to understand the context of the text, enabling the model to better capture semantic information in the text, thereby improving the accuracy of sentiment classification.

[0022] 8. The present invention also provides an emotion detection system based on the Transformer architecture, which has the same beneficial effects as the emotion detection method based on the Transformer architecture described above, and will not be elaborated here.

[0023] 9. This invention also provides a sentiment detection system based on the Transformer architecture. Based on time series analysis, it effectively improves the real-time performance and accuracy of sentiment detection by combining hybrid expert adaptive matching optimization and Bayesian statistical principles. It avoids the limitations of single-model prediction in traditional methods and further enhances the intelligence and adaptability of the text sentiment detection system.

[0024] 10. The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the above-described emotion detection method based on the Transformer architecture. It has the same beneficial effects as the above-described emotion detection method based on the Transformer architecture, and will not be elaborated upon here. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of steps S1 to S5 of the emotion detection method based on the Transformer architecture provided in the first embodiment of the present invention.

[0027] Figure 2 This is a flowchart of steps S51 to S52 of the emotion detection method based on the Transformer architecture provided in the first embodiment of the present invention.

[0028] Figure 3 This is a flowchart of steps S11 to S12 of the emotion detection method based on the Transformer architecture provided in the first embodiment of the present invention.

[0029] Figure 4 This is a flowchart of steps S521 to S523 of the emotion detection method based on the Transformer architecture provided in the first embodiment of the present invention.

[0030] Figure 5 The flowchart of the emotion detection method based on the Transformer architecture provided in the first embodiment of the present invention is a step S3 and step S5 replaced with steps S30 and step S5.

[0031] Figure 6 The first embodiment of the present invention provides a flowchart of the emotion detection method based on the Transformer architecture, including step S6.

[0032] Figure 7 This is a flowchart of steps S511 to S513 of the emotion detection method based on the Transformer architecture provided in the first embodiment of the present invention.

[0033] Figure 8This is a schematic diagram of the framework of the emotion detection system based on the Transformer architecture provided in the second embodiment of the present invention. Figure 1 .

[0034] Figure 9 This is a schematic diagram of the framework of the emotion detection system based on the Transformer architecture provided in the second embodiment of the present invention. Figure 2 .

[0035] Figure 10 This is a schematic diagram of the framework of the emotion detection system based on the Transformer architecture provided in the third embodiment of the present invention. Figure 1 .

[0036] Figure 11 This is a schematic diagram of the framework of the emotion detection system based on the Transformer architecture provided in the third embodiment of the present invention. Figure 2 .

[0037] Figure 12 This is a schematic diagram of the framework of a computer-readable storage medium provided in the fourth embodiment of the present invention.

[0038] Explanation of reference numerals in the attached diagram: 1. Emotion detection system; 2. Emotion detection system; 3. Storage medium; 10. Data Input Module; 11. Labeling and Segmentation Module; 12. Data Augmentation Module; 13. Model Building and Training Module; 14. Validation Module; 15. Output Module; 16. Testing Module; 20. Trend Analysis Module; 30. Computer Program. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0040] Please see Figure 1 and Figure 2 The first embodiment of the present invention provides a method for training an emotion detection model based on the Transformer architecture, comprising the following steps: Step S1: Build an initial model based on the Transformer architecture and set the initial hyperparameters; Step S2: Obtain the original dataset and perform data cleaning and standardization on the original dataset to obtain a standardized dataset; Step S3: Perform sentiment annotation on the standardized dataset, assign sentiment category labels and corresponding confidence scores to each sample, and divide the annotated standardized dataset into training and validation sets according to a preset ratio; Step S4: Configure at least two data augmentation operations, each associated with an independent application probability and adjustment parameter. Dynamically select the combination of data augmentation operations based on the application probability, and perform augmentation operations on the text in the training set according to the adjustment parameter of the selected combination to generate an augmented dataset. Step S5: Perform iterative training on the initial model until the validation set metrics reach the preset standard, and output the final model. The iterative training steps include: Step S51: Train the initial model using the current reinforcement dataset and generate sentiment prediction results through a self-attention mechanism; Step S52: Periodically calculate the difference between the sentiment prediction results and the corresponding confidence levels of the validation set. When the difference exceeds a preset threshold, update the initial hyperparameters based on the difference gradient, optimize the application probability and adjustment parameters according to the text type and sentiment distribution, and generate a new generation of reinforcement dataset.

[0041] Understandably, by periodically calculating the difference between sentiment predictions and confidence scores on the validation set, and adjusting the initial hyperparameters and updating the reinforcement dataset based on these differences, this mechanism enables the model to adaptively optimize and continuously improve, ensuring the efficiency and accuracy of the training process. This method's dynamic adjustment of the training process reduces the impact of noise and bias in the data on the model, thereby improving the accuracy and stability of sentiment detection.

[0042] In step S1, the initial model constructed based on the Transformer architecture has an excellent self-attention mechanism, which can handle complex language features and emotional expressions, thereby improving the model's ability to understand text sentiment analysis.

[0043] Understandably, obtaining the raw dataset involves collecting text data from multiple data sources, such as social media platforms, online customer service records, product review sections, and news websites, ensuring the data sources are broad and representative. The raw dataset can be collected via API download or web scraping techniques.

[0044] In step S2, data cleaning and standardization during the data preprocessing stage ensure data quality and consistency, avoiding the impact of data noise on model training. Data cleaning is used to eliminate noise, errors, and inconsistencies in the original dataset, ensuring data quality. Standardization is used to transform the data into a uniform scale, so that data with different dimensions can have an equivalent impact on the model.

[0045] Specifically, in this embodiment, data cleaning and standardization include: removing HTML tags, removing non-ASCII characters, correcting common spelling errors, converting all uppercase letters to lowercase, and removing noise information such as extra spaces.

[0046] Alternatively, data cleaning and standardization can be performed in batches using regular expressions or preprocessing tools in programming languages ​​such as Python, thereby ensuring data consistency and applicability.

[0047] In step S3, during the sentiment labeling stage, not only are sentiment type labels labeled, but each sample is also assigned a corresponding confidence level, which improves the model's ability to recognize multiple sentiments.

[0048] Understandably, when performing sentiment annotation, a professional annotation team should be organized and detailed sentiment annotation guidelines should be developed to ensure the accuracy and reliability of the annotation results. First, it is necessary to clearly define sentiment categories, such as positive, negative, neutral, joy, anger, and sadness, and provide clear definitions and examples for each category. Second, detailed annotation standards need to be developed, including specific annotation processes and requirements, while determining the granularity of annotation—whether it is at the sentence, paragraph, or document level. Furthermore, the boundaries between different sentiment categories should be clearly defined, the differences between similar sentiments should be explained, and typical cases and counterexamples should be provided to reduce the subjectivity of annotation. Further, to ensure annotation quality, a quality control mechanism can be established, including cross-validation and regular reviews, collecting annotation results and calculating consistency indicators, such as the Kappa coefficient, to evaluate annotation quality. Multiple rounds of review and voting should be conducted for controversial data. Finally, a feedback mechanism should be established to collect suggestions for improvement during the sentiment annotation process and update it regularly. Updates should include further refinement of sentiment tags, the introduction of new sentiment categories, and adjustments to annotation standards to adapt to constantly changing annotation needs and data characteristics. These steps collectively ensure the systematic and consistent nature of sentiment annotation work.

[0049] Understandably, the sentiment annotation stage in step S3 can be carried out through manual annotation to ensure accurate classification of complex semantics, or it can be further combined with automatic annotation, using pre-trained models, such as sentiment dictionaries or basic classifiers, to perform preliminary classification of a large number of samples, and then to verify and correct them.

[0050] Understandably, confidence is an important indicator for measuring the credibility of sentiment annotation, and it can be assessed through annotation consistency and accuracy. If multiple annotators classify a single text with high consistency, the confidence of the annotation is high; conversely, if multiple annotators do not, the confidence is low, and further review or re-annotation is necessary. The level of consistency among different annotators can be quantified using evaluation methods such as Cohen's Kappa and Fleiss's Kappa coefficients.

[0051] Furthermore, in step S3, when configuring confidence levels, confidence thresholds and weights can be set and adjusted according to actual needs. A confidence threshold sets a boundary to distinguish between high-quality and unreliable data. Setting a reasonable confidence threshold based on actual needs can filter out more accurate training data, avoid interference from noisy data, improve the generalization ability of the emotion detection model, and make it more robust in real-world applications. By assigning higher confidence weights to some samples, the model will pay more attention to data samples considered high-quality during learning, thereby enhancing the influence of these data.

[0052] In step S4, data augmentation operations are used to generate an enhanced dataset, which can increase the diversity of training data and make the model more accurate and robust when faced with different emotional expressions.

[0053] Optionally, in step S4, the data augmentation operations include: randomly replacing words or phrases, randomly inserting words or phrases, randomly swapping the positions of words or phrases, randomly deleting words or phrases, randomly shuffling the sentence order, and data back-translation, etc.

[0054] Understandingly, random word or phrase replacement refers to synonymous substitution, such as replacing "happy" with "joyful". Random word or phrase insertion refers to randomly inserting one or more words or phrases at certain positions in a sentence, such as inserting the word "very" into the sentence "I like programming" to change the sentence to "I very much like programming". Random word or phrase swapping refers to randomly selecting two words or phrases in a sentence and swapping their positions, such as changing "He likes to eat apples" to "He likes to eat apples". Random word or phrase deletion includes removing words or phrases that do not affect semantics, such as simplifying "The weather is very good today" to "The weather is very good". Randomly shuffling sentence order is suitable for text-level data augmentation, usually used for paragraphs or articles, to enhance the model's understanding and flexibility of context by randomly shuffling sentence order. Data back-translation refers to translating the original text into another language and then translating it back into the original language.

[0055] Understandably, these data augmentation operations can significantly enhance the diversity of training datasets, thereby improving the robustness and generalization ability of the model. These operations effectively simulate the variability of language in reality, improving the model's accuracy and flexibility in handling different linguistic expressions, and reducing performance degradation caused by insufficient training data or overfitting. Simultaneously, data augmentation methods can help improve model performance in low-resource tasks, especially in situations where data is scarce, by augmenting the dataset to fill knowledge gaps.

[0056] In step S52, the probability and parameters of the augmentation operation are dynamically adjusted according to different text types and sentiment characteristics. The best data augmentation scheme can then be determined by comparing the training results.

[0057] Understandably, the criteria for dynamic adjustment can be based on multiple aspects. First, different combinations of data augmentation operations are selected based on the type of text, such as news, social media, or comments. Second, the sentiment distribution of the text, such as positive, negative, or neutral, is considered, and the combination of data augmentation operations is adjusted to maintain sentiment consistency. Finally, the performance of the current model on the validation set is monitored in real time, and the initial hyperparameters are dynamically adjusted based on metrics such as the accuracy and loss value of the sentiment prediction results.

[0058] Understandably, in the process of data augmentation, the application probability refers to the probability that each data augmentation operation will be applied. For example, the application probability of randomly replacing words or phrases as a data augmentation operation is set to 70%, meaning that random word or phrase replacement will occur in 70% of cases.

[0059] Understandably, the adjustment parameters are the specific adjustment parameters for each data augmentation operation. For example, for a data augmentation operation that randomly inserts words or phrases, the number of words or phrases to be inserted can be set to 1 to 3, or the range of randomness of the insertion position can be selected.

[0060] Furthermore, methods for determining the optimal combination of data augmentation operations include: conducting multiple sets of experiments, applying different combinations of data augmentation operations and recording the performance metrics for each set; using cross-validation to ensure the reliability and generalization ability of the final output model; and performing statistical analysis on the training results to select the augmentation combination that performs best on the validation set.

[0061] Furthermore, the initial hyperparameters in step S1 include at least one or more of the following: number of training epochs, batch size, optimizer type, regularization parameter, learning rate, weight decay coefficient, gradient pruning threshold, etc.

[0062] Understandably, each iteration of steps S51 to S52 represents one round of training for the initial model. The number of training rounds refers to the total number of training iterations, affecting the sufficiency of learning and the risk of overfitting. Too few training rounds, such as less than 5 rounds, may result in the model not fully learning the data features, leading to underfitting and low accuracy. Too many training rounds, such as more than 50 rounds, may lead to overfitting, causing the model to perform well on the training set but with decreased generalization ability. To balance training effectiveness, an early stopping strategy can be used, stopping training when the loss value on the validation set no longer decreases, thus preventing overfitting.

[0063] It is understandable that the performance of the current model is evaluated periodically on the validation set. For example, after a certain number of training rounds, the learning curve can be plotted to observe the convergence of the current model, so as to adjust the training strategy based on the evaluation results, such as terminating training early to prevent overfitting and adjusting the learning rate to optimize the training process.

[0064] Understandably, batch size controls the number of samples used in each gradient update, affecting computational efficiency and training stability. A batch size that is too small, such as 8-16 samples, may lead to frequent gradient updates, higher noise levels, and reduced convergence stability. A batch size that is too large, such as 256 or more samples, while making the gradient smoother, may increase memory usage and lead to insufficient computational resources. In this embodiment, a batch size between 32 and 128 samples is chosen, and gradient accumulation can be combined to simulate larger batch training, achieving a balance between memory usage and convergence stability.

[0065] Optionally, the batch size can be set to have 16, 32, 64, and 128 samples per batch, etc.

[0066] Understandably, the optimizer controls how model parameters are updated, directly affecting the convergence speed of training and the quality of optimization. Optimizers that can be selected include: Adam optimizer, AdamW optimizer, SGD optimizer, AdaGrad optimizer, and RMSprop optimizer, etc.

[0067] Preferably, the Adam optimizer can be used in combination with learning rate decay, or optimization techniques such as Lookahead and RAdam can be employed to improve convergence and model performance.

[0068] Understandably, regularization parameters typically control the degree of influence of regularization terms on the model, preventing overfitting and improving generalization ability, including L1 regularization and L2 regularization. Preferably, a smaller L2 regularization is used, such as an L2 regularization parameter λ ranging from 0.01 to 0.001, to achieve a balance between model complexity and generalization ability.

[0069] Understandably, the learning rate determines the gradient update magnitude, directly affecting the model's convergence speed and stability. If the learning rate is too large, it may cause gradient oscillations, preventing the model from converging and even damaging the pre-trained weights; if the learning rate is too small, the training speed will be too slow, potentially leading to local optima and affecting the model's learning performance.

[0070] Optionally, the learning rate can be set to a range of 100%. Preferably, the learning rate is set within a range of 100%. .

[0071] Understandably, it can also be used in conjunction with a learning rate scheduler. For example, Hugging Face's get_scheduler can set a linear or cosine decay strategy based on the number of training steps or epochs, so that the model converges faster in the early stage of training and gradually reduces the learning rate to stabilize convergence in the later stage.

[0072] Furthermore, when iteratively training a sentiment detection model, LSTM and RNN can be used to capture the temporal dependencies of the input text data. Natural language is sequential, meaning the meaning of the current word often depends on the context. RNN, through recurrent connections, can progressively process the input sequence and remember past information, making it suitable for sequence-sensitive NLP tasks such as sentences and documents. Standard RNN suffers from gradient vanishing and long-distance dependencies, making it difficult to capture the dependencies of long texts. LSTM, as an improved version of RNN, introduces memory units and gating mechanisms, effectively preserving long-term information and preventing gradient vanishing, enabling it to handle the temporal dependencies of longer texts. In addition, in tasks such as sentiment analysis, text generation, and machine translation, LSTM can understand sentence structure, capture the connections between keywords, improve text modeling capabilities, and enable the model to more accurately understand and generate context-appropriate language expressions.

[0073] Further, please refer to Figure 3 In this embodiment, the learning rate adopts a segmented dynamic adjustment strategy, which specifically includes the following steps: Step S11: The first N rounds of initial model training are performed in the Warmup phase, starting with an initial learning rate that is gradually increased until the preset maximum learning rate value is reached; Step S12: After the Warmup phase is completed, the learning rate is gradually reduced according to the linear decay strategy until training ends. The reduction process is carried out along the set decay path.

[0074] Understandably, the Warmup phase, by gradually increasing the learning rate, allows the model to quickly enter a higher learning rate stage, improving initial learning efficiency and accelerating the overall convergence process. After the Warmup phase, the trained model is relatively stable, and the linear decay strategy gradually reduces the learning rate. This helps the model fine-tune parameters as it approaches the global optimum, avoiding early entrapment in local optima and improving the model's global optimization ability. Through a suitable Warmup mechanism and linear decay strategy, the model can not only train quickly but also perform fine-tuning of parameters with small steps in the final stage, thereby improving the final generalization performance and accuracy.

[0075] Furthermore, the validation set metrics in step S5 include at least one or more of the following: accuracy, precision, recall, and F1 score.

[0076] Understandably, by considering multiple evaluation metrics simultaneously, a model's performance can be comprehensively assessed. These metrics help measure the model's classification ability, retrieval ability, and overall performance, avoiding the misleading effects of a single metric and thus providing more accurate guidance for model optimization.

[0077] Accuracy is the proportion of correctly predicted samples out of the total number of samples, and it can be calculated using Formula 1, as follows:

[0078] Precision refers to the proportion of samples that are actually predicted as positive out of all samples predicted as positive. It can be calculated using Formula 2, as follows:

[0079] Recall rate refers to the proportion of samples that are actually positive that are correctly predicted as positive. It can be calculated using Formula 3, as follows:

[0080] The F1 score is the harmonic mean of precision and recall, used to comprehensively consider these two metrics. It can be calculated using Formula 4, as follows:

[0081] Where Accuracy is the model's accuracy, Precision is the model's precision, Recall is the model's recall rate, and F1 is the model's F1 score; TP represents true positives, indicating the number of samples correctly predicted as positive by the model; TN represents true negatives, indicating the number of samples correctly predicted as negative by the model; FP represents false positives, indicating the number of negative samples incorrectly predicted as positive by the model; and FN represents false negatives, indicating the number of positive samples incorrectly predicted as negative by the model.

[0082] Understandably, different validation set metrics focus on different aspects, and overemphasizing one metric may negatively impact others. The choice of metrics should be based on a trade-off between different business needs. For example, increasing recall may lead to a decrease in precision because the model may over-label positive classes, thus increasing false positives; conversely, increasing precision may decrease recall because the model may more rigorously screen, selecting only the most certain positive class samples, resulting in an increase in false negatives.

[0083] As specific examples, chatbots or voice assistants require high recall to ensure they capture user emotions and avoid missing negative feedback. However, precision cannot be too low, otherwise, normal emotions may be misjudged as negative, causing user confusion. Therefore, the F1 score needs to be optimized to balance comprehensiveness and accuracy. In customer complaint analysis systems, recall is crucial; negative emotions should be detected as much as possible to respond to customer dissatisfaction promptly. Even if precision decreases, increasing false positives, it's essential to avoid missing important complaints. Therefore, improving recall should be prioritized. For sentiment analysis market research, such as social media sentiment analysis, precision is even more important to prevent ordinary comments from being misjudged as strongly emotional comments, affecting the accuracy of brand sentiment analysis. Therefore, precision optimization should be prioritized to ensure data reliability.

[0084] Further, see Figure 4 Step S52, updating the initial hyperparameters based on the difference value gradient, includes the following steps: Step S521: Use the Softmax activation function to transform the sentiment prediction results into the prediction probability distribution of each sentiment category; Step S522: Use the cross-entropy loss function to calculate the difference between the predicted probability distribution and the confidence level, and output the loss value; Step S523: Calculate the gradient of the loss value with respect to each initial hyperparameter using the backpropagation algorithm, and update the initial hyperparameters using the gradient descent algorithm.

[0085] Understandably, by employing the Softmax activation function, the model output can be effectively transformed into a probability distribution for each sentiment type, thereby improving the model's ability to distinguish between different sentiment types. Using the cross-entropy loss function to calculate the difference between predictions and confidence scores accurately reflects the degree of error in the model's predictions, thus enabling effective optimization. The backpropagation algorithm is used to calculate gradients and update training parameters, ensuring that the model's training process conforms to the optimization objective, thereby achieving accurate identification of sentiment detection tasks.

[0086] In step S521, a classification head with a Softmax activation function is used to output the predicted probability distribution of the sentiment prediction results. The Softmax function converts the original output of the current model into a normalized probability vector, so that the predicted probability of each sentiment category falls within the range of 0 to 1, and the sum of the predicted probabilities of all sentiment categories is 1, thereby ensuring that the output conforms to the definition of the probability distribution. This process allows the score of each sentiment category to be interpreted as the predicted probability of that sentiment category.

[0087] In step S522, the cross-entropy loss function is used to measure the quality of the current model's prediction, making the probability of the correct class as close to 1 as possible. This can be expressed by Formula 5, as follows:

[0088] Where Loss represents the loss value, j represents the index of the sample, and there are a total of N samples. i represents the index of the class, and there are a total of C classes. It is the true label of sample j in category i. This is the probability that sample j belongs to class i, as predicted by the model. The predicted probability distribution output in step S521 is... .

[0089] In step S523, backpropagation is used to update the initial hyperparameters, gradually minimizing the loss value and improving classification accuracy, enabling the model to better predict sentiment of different categories. Optimization algorithms such as gradient descent are used to calculate the gradient of the loss value with respect to the current initial hyperparameters, and the next generation of initial hyperparameters is adjusted to gradually reduce the loss value.

[0090] Understandably, when setting initial hyperparameters, grid search, random search, or automatic hyperparameter tuning algorithms can be used to find the optimal combination of hyperparameters.

[0091] Understandably, to further optimize the validation set metrics, data augmentation can be used to increase negative sentiment samples, improve the model's ability to identify low-frequency categories, and thus optimize recall; class weights can be adjusted to optimize the loss function, reduce false positives, and thus improve precision; and the classification threshold can be adjusted to optimize the Softmax threshold, balancing precision and recall to improve the F1 score.

[0092] Furthermore, in step S1, the initial model constructed based on the Transformer architecture is any one of the following: a pre-trained BERT model, a multilingual BERT model, a lightweight BERT model, a RoBERTa model, or an XLNet model.

[0093] Understandably, the BERT model is suitable for natural language processing tasks, such as text classification and question answering systems; the multilingual BERT model is suitable for low-resource languages, cross-lingual tasks, and multilingual processing. The lightweight BERT model is suitable for resource-constrained environments; RoBERTa is an improved version of BERT, better adapting to large-scale text data and exhibiting better generalization ability on large training sets; the XLNet model combines the advantages of autoregressive and autoencoder models, possessing outstanding text generation and understanding capabilities. This diversity of models provides flexible choices for different application scenarios.

[0094] Further, please refer to Figure 5 In step S3, the standardized dataset after annotation can be further divided into training, validation, and test sets according to a preset ratio, and the final model can be applied to the test set to test its generalization ability. That is, step S3 can be replaced with: Step S30: Perform sentiment annotation on the standardized dataset, assign sentiment category labels and corresponding confidence scores to each sample, and divide the annotated standardized dataset into training set, validation set and test set according to a preset ratio.

[0095] Furthermore, step S5 is replaced with: Step S50: Perform iterative steps on the initial model until the validation set metrics reach the preset standard, output the final model, and apply the final model to the test set to test its generalization ability.

[0096] Understandably, the training set is used to optimize model parameters, enabling it to fully learn data features and improve predictive ability. The validation set is used for initial hyperparameter tuning and early stopping, preventing overfitting and ensuring the model doesn't lose its generalization ability by simply memorizing training data. The test set is used for independent evaluation, measuring the model's performance on completely unseen data, avoiding over-reliance on the validation set and affecting its performance in real-world environments.

[0097] It is understandable that when dividing a standardized dataset, it should be done randomly to ensure that the distribution of different sentiment categories in each subset is similar to that in the overall dataset, so as to avoid the impact of data skew on the model training effect.

[0098] Optionally, the set can be divided into three parts: 80% training set, 10% validation set, and 10% test set; or 60% training set, 20% validation set, and 20% test set; or 70% training set, 30% validation set, and 0% test set.

[0099] Preferably, the dataset is divided into 70% training set, 20% validation set, and 10% test set. This division balances the needs of model training, optimization, and evaluation, improves model stability, and ensures good performance in real-world applications, rather than achieving high accuracy on training data but performing poorly on new data.

[0100] Understandably, when testing the generalization ability of the final model using the test set, metrics such as accuracy, precision, recall, and F1 score can be used to quantify the current final model's prediction performance for each sentiment category. If the dataset's category distribution is severely imbalanced, methods such as macro-averaging or weighted average F1 scores can also be considered to measure overall performance. If the prediction performance does not meet expectations, steps S51 to S52 can be iterated to generate a new generation of the final model.

[0101] Understandably, before evaluating the final model using the test set, the model should be saved. Specifically, this requires fully storing the sentiment enhancement self-attention weights, multi-granularity semantic fusion module parameters, and the state of the contrastive learning optimizer. It also includes saving the domain-adapted word segmenter, dynamic vocabulary, and sentiment tag mapping table. Encrypted serialization, such as PyTorch's .pt or ONNX format, should be used to ensure model security and cross-platform deployment capabilities. Training hyperparameters and data preprocessing procedures should be recorded, supporting version rollback and incremental updates to ensure consistency and robustness of the model during inference, transfer learning, and feedback training.

[0102] Further, please refer to Figure 6 The training method for emotion detection models based on the Transformer architecture also includes the following steps: Step S6: Use a confusion matrix to analyze misclassifications in the sentiment prediction results.

[0103] Understandably, confusion matrices can help analyze a model's classification results across various emotion categories, particularly identifying misclassifications. By identifying which emotion types are prone to misclassification, the initial hyperparameters of the next-generation final model can be adjusted or appropriate methods for generating reinforcement datasets can be selected to optimize the model's ability to recognize different emotions and improve its performance in complex emotion classification tasks.

[0104] Understandably, once the confusion matrix identifies potential sources of error, the initial hyperparameters can be further tuned. For example, grid search or random search can be performed on key initial hyperparameters; Bayesian optimization methods can also be considered. Cross-validation or repeated testing on independent validation sets can also be used to evaluate the model performance under different combinations of initial hyperparameters, and the configuration that achieves the best results on the validation set can be selected as the final solution.

[0105] Further, please refer to Figure 7 Step S51 involves training an initial model using the current reinforcement dataset, including the following steps: Step S511: Divide the augmentation dataset into multiple subsets according to a preset number; Step S512: Perform word segmentation on different subsets in batches, decompose the text into units, assign a unique index tag to each unit, and record the position information of the index tag in the standardized dataset. Step S513: Map the index tags to embedding vectors, and overlay word embeddings, sentence identifier embeddings, and position embeddings, then use the embedding vectors as input to the initial model.

[0106] In step S511, dividing the dataset into multiple subsets for batch word segmentation can improve the efficiency of the word segmentation process and avoid memory overload or slow processing speed by batch processing.

[0107] For example, if the enhanced dataset contains 10,000 data entries and each batch contains 128 data entries, the enhanced dataset will be divided into 79 batches for word segmentation processing.

[0108] In step S512, batch processing makes the model more stable during training, improves training efficiency, and facilitates parallel processing, further enhancing the overall performance of the system. Through word segmentation, text can be converted into unit forms that the model can understand, and by assigning index tags to each unit, the model can process text based on its position, preserving sentence structure information.

[0109] Understandably, step S512 decomposes the text into units, which include words or subwords, representing the smallest input unit that the model can process. A word refers to a complete lexical unit, such as "like" and "weather"; while a subword unit decomposes a word into smaller subword fragments. This method is particularly suitable for handling out-of-vocabulary words or word inflections, such as decomposing "unhappiness" into the subword fragments "un" and "happiness". This decomposition method helps improve the model's ability to understand and process language.

[0110] Understandably, in terms of usage scenarios, sub-word units are suitable for handling low-frequency words and new words, while words are more suitable for semantically complete contextual analysis.

[0111] Understandably, assigning a unique index label to each unit facilitates mapping it to an embedding vector within the model, thereby improving computational and storage efficiency. Text data is inherently unstructured, while deep learning models can only process numerical inputs; therefore, it is necessary to convert the text into numerical indices and then map them to embedding vectors for computation.

[0112] As a concrete example, in the initial model built by Transformer, the sentence "I am happy" may become ["I", "very", "happy", "heart"] after word segmentation, and then mapped to unique index tags [2769, 3221, 2193, 5523]. These index tags are then passed as input to the embedding layer, which maps them to the corresponding embedding vectors.

[0113] Understandably, this indexing method improves storage efficiency, avoids storing the entire text, and reduces the overhead of string matching during computation. Furthermore, after indexing, the initial model based on Transformer components can perform attention computation more efficiently, ensuring accurate understanding of contextual semantics.

[0114] In step S513, word embeddings are responsible for representing word meanings and avoiding ambiguity; sentence identifier embeddings are used to distinguish different sentences and support tasks such as next sentence prediction; and positional embeddings ensure the preservation of word order information. Overlaying word embeddings, sentence identifier embeddings, and positional embeddings enhances the model's ability to understand the context of text, enabling it to better capture semantic information and thus improve the accuracy of sentiment classification.

[0115] As a concrete example, step S1 constructs an initial model based on the Transformer architecture, which is a pre-trained BERT model. Step S511 divides the augmented dataset into M subsets. Step S512 decomposes the text data in the subsets in batches using the WordPiece tokenizer, breaking it down into units containing words and subwords to ensure better adaptation to out-of-vocabulary words, and assigns an index tag to each unit so that it can be processed by the BERT model. In step S513, the BERT model maps the index tags to embedding vectors and sequentially stacks word embeddings, sentence identifier embeddings, and position embeddings, enabling the model to fully understand the semantics, structure, and word order of the text. The final input of the BERT model can be represented as the sum of word embeddings, sentence identifier embeddings, and position embeddings. This can be expressed by Equation Six, as follows:

[0116] in, It is the final input representation; It is the word embedding of each word, represented as the vector of that word in the vocabulary; It is the corresponding sentence identifier embedding, used to distinguish different sentences; It is a location embedding, which encodes the location information of words; This is the total number of words in the input text.

[0117] Understandably, during word segmentation and feature extraction, a word segmenter and feature engineering method suitable for the current model should be selected. Features should be standardized and normalized according to the input requirements of the current model to ensure that the data format meets the training needs of the current model. Standardization adjusts feature values ​​to a distribution with a mean of 0 and a standard deviation of 1. This process helps eliminate dimensional differences between features, making the model easier to train. Normalization scales feature values ​​to a specific range, such as [0, 1]. Normalization can speed up model training, especially for distance-based algorithms, because it ensures that all features are compared on the same scale. Through standardization and normalization, the model's performance and training efficiency can be improved.

[0118] Please see Figure 8 The second embodiment of the present invention provides an emotion detection system 1 based on the Transformer architecture, comprising: Data input module 10 acquires the raw dataset and performs data cleaning and standardization on the raw dataset to obtain a standardized dataset; The annotation and partitioning module 11 is used to perform sentiment annotation on the standardized dataset, assign sentiment category labels and corresponding confidence scores to each sample, and divide the standardized dataset after annotation into training set and validation set according to a preset ratio. The data augmentation module 12 is configured with at least two data augmentation operations. Each operation is associated with an independent application probability and adjustment parameter. The combination of data augmentation operations is dynamically selected according to the application probability, and the augmentation operation is performed on the text in the training set according to the adjustment parameter of the selected combination to generate an augmented dataset. The model building and training module 13 is used to build an initial model based on the Transformer architecture, set initial hyperparameters, train the initial model using the current reinforcement dataset, and generate sentiment prediction results through a self-attention mechanism. Validation module 14 is used to periodically calculate the difference between the sentiment prediction results and the corresponding confidence levels of the validation set. When the difference exceeds a preset threshold, the initial hyperparameters are updated based on the difference gradient. The application probability and adjustment parameters are optimized according to the text type and sentiment distribution to generate a new generation of reinforcement dataset. Output module 15 is used to output the final model when the validation set metrics reach the preset standard.

[0119] Further, please refer to Figure 9 The annotation and partitioning module 11 can also be used to: perform sentiment annotation on the standardized dataset, assign sentiment category labels and corresponding confidence levels to each sample, and divide the labeled standardized dataset into training set, validation set and test set according to a preset ratio.

[0120] The emotion detection system 1 is equipped with a test module 16: iterative steps are performed on the initial model until the validation set index reaches the preset standard, the final model is output, and the final model is applied to the test set to test its generalization ability.

[0121] Understandably, the emotion detection system based on the Transformer architecture in this embodiment has the same beneficial effects as the emotion detection method based on the Transformer architecture described above, and will not be elaborated upon here.

[0122] Please see Figure 10 and Figure 11 The third embodiment of the present invention provides an emotion detection system 2 based on the Transformer architecture. The difference between the emotion detection system 2 in this embodiment and the emotion detection system 1 described above is as follows: The emotion detection system 2 in this embodiment also includes a trend analysis module 20: used to incorporate time series analysis and, based on this, combine it with a hybrid expert adaptive matching optimization method. This design significantly improves the accuracy and dynamic response capability of emotion detection. This optimization method is based on the dynamic changes of multiple emotion categories, and the emotion detection system 2 can automatically select an appropriate time trend model to construct a dynamic trend network. This dynamic trend network avoids the single prediction method based on static input in traditional emotion prediction models, and flexibly adjusts the prediction results by analyzing the changing trends of the current emotion state in real time.

[0123] Furthermore, the trend analysis module 20 incorporates Bayesian statistical principles to dynamically adjust model predictions based on changes in real-time sentiment data. The Bayesian update mechanism can promptly correct previous predictions based on newly observed data, ensuring that the sentiment detection system can adaptively make the most appropriate sentiment judgments at different times and in different contexts.

[0124] Understandably, the emotion detection system 2 in this embodiment is based on time series analysis. By combining hybrid expert adaptive matching optimization and Bayesian statistical principles, it effectively improves the real-time performance and accuracy of emotion detection, avoids the limitations of single-model prediction in traditional methods, and further enhances the intelligence and adaptability of the text emotion detection system.

[0125] Please see Figure 12 The fourth embodiment of the present invention provides a computer-readable storage medium 3 on which a computer program 30 is stored. When the computer program 30 is executed by a processor, it implements the above-described emotion detection method based on the Transformer architecture.

[0126] Understandably, the storage medium 3 in this embodiment has the same beneficial effects as the emotion detection method based on the Transformer architecture described above, and will not be elaborated here.

[0127] In the embodiments provided by this invention, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0128] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Those skilled in the art should also recognize that the embodiments described in the specification are optional embodiments, and the actions and modules involved are not necessarily essential to the invention.

[0129] In various embodiments of the present invention, it should be understood that the sequence number of each process does not necessarily imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0130] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It is particularly important to note that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0131] Compared with existing technologies, the emotion detection model training method, system, and storage medium based on the Transformer architecture provided by this invention have the following beneficial effects: 1. This invention presents a training method for a sentiment detection model based on the Transformer architecture. The initial model is constructed using the Transformer architecture, which inherently possesses an excellent self-attention mechanism, enabling it to handle complex linguistic features and emotional expressions, thereby improving the model's understanding of text sentiment analysis. Standardized datasets reduce data quality uncertainty, making the model more stable during training. Data partitioning ensures the model's generalization ability, allowing it to perform well not only on training data but also maintain high predictive performance on unseen data. Setting a validation set allows for periodic evaluation of model performance and avoids overfitting. Defining at least two data augmentation operations and assigning independent application probabilities and adjustment parameters to each operation, dynamically selecting and applying augmentation operations during training, can more effectively increase the diversity of training data, making the model more accurate and robust when facing different emotional expressions. Periodically calculating the sentiment prediction results and confidence differences on the validation set, and adjusting the initial hyperparameters and updating the augmentation dataset based on these differences, allows the model to adaptively optimize and continuously improve, ensuring the efficiency and accuracy of the training process. The method's dynamic adjustment of the training process can reduce the impact of noise and bias in the data on the model, thereby improving the accuracy and stability of emotion detection.

[0132] 2. The emotion detection model training method based on the Transformer architecture of this invention effectively transforms the model output into probability distributions for various emotion types by employing the Softmax activation function, thereby improving the model's ability to distinguish between different emotion types. Using the cross-entropy loss function to calculate the difference between predictions and confidence scores accurately reflects the degree of error in model predictions, thus enabling effective optimization. The backpropagation algorithm is used to calculate gradients and update training parameters, ensuring that the model training process conforms to the optimization objective, thereby achieving accurate identification of emotion detection tasks.

[0133] 3. The emotion detection model training method based on the Transformer architecture of this invention employs at least two data augmentation operations on the text in the training set, including random word or phrase replacement, random word or phrase insertion, random word or phrase swapping, random word or phrase deletion, random sentence shuffling, and data back-translation, to generate an augmented dataset. These data augmentation operations significantly enhance the diversity of the training dataset, improving the model's robustness and generalization ability. These operations effectively simulate the variability of language in reality, improving the model's accuracy and flexibility in handling different language expressions, and reducing performance degradation caused by insufficient training data or overfitting. Simultaneously, data augmentation methods can help improve model performance in low-resource tasks, especially in situations where data is scarce, by augmenting the dataset to fill knowledge gaps.

[0134] 4. The emotion detection model training method based on the Transformer architecture of this invention constructs an initial model based on the Transformer architecture, which can be any one of the following: a pre-trained BERT model, a multilingual BERT model, a lightweight BERT model, a RoBERTa model, or an XLNet model. The BERT model is suitable for natural language processing tasks, such as text classification and question answering systems; the lightweight BERT model is suitable for resource-constrained environments; RoBERTa is an improved version of BERT, better adapting to large-scale text data and exhibiting better generalization ability on large-scale training sets; the XLNet model combines the advantages of autoregressive and autoencoder models, possessing outstanding text generation and understanding capabilities. This diversity of models provides flexible choices for different application scenarios.

[0135] 5. The emotion detection model training method based on the Transformer architecture of this invention divides the standardized dataset after labeling into training, validation, and test sets in a 7:2:1 ratio. The model is then applied to the test set to test its generalization ability. This 70% training set, 20% validation set, and 10% test set ratio ensures a balanced learning, optimization, and evaluation process at different stages. 70% of the training set is used to optimize model parameters, enabling it to fully learn data features and improve predictive ability; 20% of the validation set is used for initial hyperparameter tuning and early stopping to prevent overfitting and ensure the model doesn't lose generalization ability by merely memorizing training data; and 10% of the test set is used for independent evaluation to measure the model's performance on completely unseen data, avoiding over-reliance on the validation set and affecting performance in real-world environments. This division balances the needs of model training, optimization, and evaluation, improving model stability and ensuring good performance in real-world applications, rather than achieving high accuracy on training data but poor performance on new data.

[0136] 6. The emotion detection model training method based on the Transformer architecture of this invention includes an initial hyperparameter, the learning rate. During the first N rounds of initial model training, a Warmup phase is executed, gradually increasing the learning rate from the initial value until a preset maximum learning rate is reached. After the Warmup phase, the learning rate is gradually decreased according to a linear decay strategy until training ends, with the decrease following a predetermined decay path. In the early stages of training, gradually increasing the learning rate through the Warmup mechanism avoids gradient explosion or training instability caused by an excessively high initial learning rate, helping the model smoothly enter the training state and reducing instability during training. In the later stages of training, the linear decay strategy gradually decreases the learning rate, helping the model finely adjust parameters as it approaches the global optimum, avoiding early trapping in local optima and improving the model's global optimization capability. Combining the Warmup and linear decay strategies allows the model to transition more smoothly between different learning rate phases during training, which not only helps stabilize the training process but also results in more refined and stable model performance.

[0137] 7. The Transformer-based sentiment detection model training method of this invention divides the dataset into multiple subsets for batch word segmentation, which improves the efficiency of the word segmentation process and avoids memory overload or slow processing speed by batch processing. Furthermore, batch processing makes the model more stable during training, improves training efficiency, and facilitates parallel processing, further enhancing the overall performance of the system. Word segmentation transforms text into unit forms that the model can understand, and by assigning index tags to each unit, the model can process based on position, preserving sentence structure information. Overlaying word embeddings, sentence tag embeddings, and positional embeddings enhances the model's ability to understand the context of the text, enabling the model to better capture semantic information in the text, thereby improving the accuracy of sentiment classification.

[0138] 8. The present invention also provides an emotion detection system based on the Transformer architecture, which has the same beneficial effects as the emotion detection method based on the Transformer architecture described above, and will not be elaborated here.

[0139] 9. This invention also provides a sentiment detection system based on the Transformer architecture. Based on time series analysis, it effectively improves the real-time performance and accuracy of sentiment detection by combining hybrid expert adaptive matching optimization and Bayesian statistical principles. It avoids the limitations of single-model prediction in traditional methods and further enhances the intelligence and adaptability of the text sentiment detection system.

[0140] 10. The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the above-described emotion detection method based on the Transformer architecture. It has the same beneficial effects as the above-described emotion detection method based on the Transformer architecture, and will not be elaborated upon here.

[0141] The foregoing has provided a detailed description of a training method, system, and storage medium for an emotion detection model based on the Transformer architecture disclosed in the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention. Any modifications, equivalent substitutions, and improvements made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for training a sentiment detection model based on the Transformer architecture, characterized in that: Includes the following steps: Build an initial model based on the Transformer architecture and set the initial hyperparameters; Obtain a standardized dataset with sentiment annotations and confidence scores, and divide the standardized dataset into a training set and a validation set according to a preset ratio; Configure at least two data augmentation operations, each associated with an independent application probability and adjustment parameter. Dynamically select a combination of data augmentation operations based on the application probability, and perform augmentation operations on the text in the training set according to the adjustment parameter of the selected combination to generate an augmented dataset. Iteratively execute the following steps until the validation set metrics reach the preset standard, and output the final model: The initial model is trained using the current reinforcement dataset, and sentiment prediction results are generated through a self-attention mechanism. The difference between the sentiment prediction results and the corresponding confidence levels of the validation set is calculated periodically. When the difference exceeds a preset threshold, the initial hyperparameters are updated based on the difference gradient. The application probability and adjustment parameters are optimized according to the text type and sentiment distribution to generate a new generation of reinforcement dataset.

2. The method for training a sentiment detection model based on the Transformer architecture as described in claim 1, characterized in that: Updating the initial hyperparameters based on the difference gradient includes the following steps: The Softmax activation function is used to transform the sentiment prediction results into a prediction probability distribution for each sentiment category; The difference between the predicted probability distribution and the confidence level is calculated using the cross-entropy loss function, and the loss value is output. The gradient of the loss value with respect to each initial hyperparameter is calculated using the backpropagation algorithm, and the initial hyperparameters are updated using the gradient descent algorithm.

3. The method for training a sentiment detection model based on the Transformer architecture as described in claim 1, characterized in that: The data augmentation operations include: randomly replacing words or phrases, randomly inserting words or phrases, randomly swapping the positions of words or phrases, randomly deleting words or phrases, randomly shuffling sentence order, and data back-translation on the text in the training set.

4. The method for training a sentiment detection model based on the Transformer architecture as described in claim 1, characterized in that: The initial model built on the Transformer architecture can be any one of the following: a pre-trained BERT model, a multilingual BERT model, a lightweight BERT model, a RoBERTa model, or an XLNet model.

5. The method for training a sentiment detection model based on the Transformer architecture as described in claim 1, characterized in that: The standardized dataset after annotation is divided into training, validation and test sets in a ratio of 7:2:1, and the final model is applied to the test set to test its generalization ability.

6. The method for training a sentiment detection model based on the Transformer architecture as described in claim 1, characterized in that: The initial hyperparameters include the learning rate. The initial model training performs a Warmup phase for the first N rounds, gradually increasing the learning rate from the initial value until a preset maximum learning rate is reached. After the Warmup phase, the learning rate is gradually decreased according to a linear decay strategy until training ends. The decrease process follows a set decay path.

7. The method for training a sentiment detection model based on the Transformer architecture as described in claim 1, characterized in that: Training the initial model using the current augmentation dataset includes the following steps: The enhanced dataset is divided into multiple subsets according to a preset number; The text is segmented into units by batch, and a unique index tag is assigned to each unit. At the same time, the position information of the index tag in the standardized dataset is recorded. The index tags are mapped to embedding vectors, and word embeddings, sentence identifier embeddings, and position embeddings are superimposed. The embedding vectors are then used as input to the initial model.

8. A sentiment detection system based on the Transformer architecture, characterized in that: It includes a labeling and partitioning module, which obtains a standardized dataset with sentiment labels and confidence scores, and divides the standardized dataset into a training set and a validation set according to a preset ratio; The data augmentation module is configured with at least two data augmentation operations, each associated with an independent application probability and adjustment parameters. It dynamically selects a combination of data augmentation operations based on the application probabilities and performs augmentation operations on the text in the training set according to the adjustment parameters of the selected combination to generate an augmented dataset. The model building and training module is used to build an initial model based on the Transformer architecture, set initial hyperparameters, train the initial model using the current reinforcement dataset, and generate sentiment prediction results through a self-attention mechanism. The verification module is used to periodically calculate the difference between the sentiment prediction results and the corresponding confidence levels of the verification set. When the difference exceeds a preset threshold, the initial hyperparameters are updated based on the difference gradient. The application probability and adjustment parameters are optimized according to the text type and sentiment distribution to generate a new generation of reinforcement dataset. The output module is used to output the final model when the validation set metrics reach the preset standard.

9. The emotion detection system based on the Transformer architecture as described in claim 8, characterized in that: The emotion detection system also includes a trend analysis module, which incorporates time series analysis and combines it with hybrid expert adaptive matching optimization and Bayesian statistical principles.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the method for training an emotion detection model based on the Transformer architecture as described in any one of claims 1-7.

Images