BERT and LightGBM integrated mental health intelligent monitoring method and system
By combining BERT and LightGBM models, real-time and high-precision monitoring of students' mental health status is achieved, and the problem of insufficient real-time tracking and accuracy in traditional methods is solved, the depth and accuracy of emotion analysis are improved, and the computing efficiency and Generation of interpretive reports that support personalized mental health support.
Patent Information
- Application Number
- CN202410899311.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-04
- Publication Date
- 2024-11-08
AI Technical Summary
Traditional mental health monitoring methods are difficult to achieve real-time tracking of students' mental health status, and there are inefficient and inaccurate problems, so it is impossible to detect potential psychological risks early.
Combining the deep context understanding and sentiment analysis capabilities of the BERT model and the efficient prediction capabilities of the LightGBM model, through data acquisition, preprocessing, feature extraction and model integration, a mental health intelligent monitoring method integrated with BERT and LightGBM is developed to realize Real-time, high-precision emotional-health status mapping.
Improve the depth and accuracy of emotion analysis, enhance the adaptive analysis capabilities of the model, optimize the computing efficiency and resource utilization, promote the intelligence and transparency of mental health monitoring, and provide detailed explanatory reports to support personalized psychology. Health support.
Smart Images

Figure CN118919023A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a method and system for intelligent monitoring of mental health integrating BERT and LightGBM. Background Art
[0002] As the competitiveness of society intensifies, college students face unprecedented learning pressure, life adaptation challenges and interpersonal relationship complexity, which leads to increasingly prominent mental health problems among students. Traditional mental health monitoring relies on manual observation and regular questionnaire surveys, which makes it difficult to track students' psychological dynamics in real time. Therefore, it is particularly important to develop an intelligent method that can automatically and continuously monitor students' mental health status and detect potential psychological risks at an early stage.
[0003] Traditional mental health monitoring methods are still inefficient and inaccurate, so there is an urgent need for an intelligent method that can not only accurately monitor mental health conditions, but also generate detailed explanatory reports to better understand students' mental states. Summary of the invention
[0004] To solve the above problems, the present invention utilizes the deep contextual understanding and sentiment analysis capabilities of the BERT model to identify the sentiment tendency of text, and combines it with the efficient prediction capabilities of the LightGBM model to form a real-time, high-precision emotion-health status mapping model, and provides a mental health intelligent monitoring method and system integrating BERT and LightGBM.
[0005] The first aspect of the present invention provides a method for intelligent monitoring of mental health integrating BERT and LightGBM, comprising the following steps:
[0006] Data collection and preprocessing;
[0007] Data annotation and enhancement;
[0008] Use the pre-trained BERT model to extract text features and classification predictions; use the LSTM network and feature engineering to extract behavioral features;
[0009] Integrate the extracted text features and behavioral features to form an integrated feature vector;
[0010] Use the trained LightGBM model to obtain classification prediction results;
[0011] Fusion of the prediction results of the BERT model and the LightGBM model;
[0012] Generate a report based on the fusion model prediction results.
[0013] The monitoring method of the present invention improves the accuracy of sentiment analysis by integrating BERT and LightGBM models. The BERT model is used to extract emotional text features, and the LSTM (Long Short-Term Memory Network) network and feature engineering are used to extract key behavioral features. These features are input into the LightGBM model for sentiment classification. Finally, the prediction and classification results of the BERT and LightGBM models are fused, which can achieve effective integration of the two and improve the sentiment analysis depth and accuracy of the model.
[0014] Preferably, the collected data includes social platform text data and student card data; the text features come from the social platform text data, and the behavior features come from the student card data. By making full use of campus network information, in addition to the social platform, it also combines student card usage records, including but not limited to school hospital visit information, grade changes, etc. These data record students' daily life behaviors and can serve as supplementary features of students' mental health status.
[0015] Since the text feature data mainly comes from social platforms, preferably, the text feature data preprocessing includes:
[0016] Clean and normalize the collected text data, including removing HTML tags, special characters, and duplicate data, and performing word segmentation, lemmatization, and stop word removal;
[0017] Use NLTK and Spacy libraries for text preprocessing to improve the quality of text data;
[0018] The TF-IDF algorithm is used to vectorize text data. The TF-IDF formula is as follows:
[0019] tf-idf(t,d,D)=tf(t,d)×idf(t,D)
[0020] tf(t, d) represents the frequency of term t in document d. That is, the number of times term t appears in document d. It can be normalized by dividing by the total number of terms in the document to avoid the problem of naturally high term frequency in long documents. idf(t, d) stands for inverse document frequency, which measures the rarity of term t in the entire document set D.
[0021] The data processing of student cards can be done through cleaning, deduplication, and time series construction to prepare for subsequent feature extraction, which is relatively simple.
[0022] Preferably, the “using a pre-trained BERT model to extract text features and classification predictions” includes: pre-training the BERT model, and then extracting text features and classification predictions by fine-tuning the BERT model, wherein the fine-tuning of the BERT model refers to adding a fully connected layer to the last layer and optimizing it using a cross entropy loss function, and the cross entropy loss function formula is as follows:
[0023]
[0024] Where N is the number of samples, C is the number of categories, and yi j Indicates the probability that the i-th sample actually belongs to the j-th class. By fine-tuning the BERT model, it is more suitable for the student mental health text classification task. Preferably, the Stacking strategy is used for model fusion, and the outputs of the two models are weighted combined by training a meta-learner logistic regression to improve the overall prediction performance. The formula is as follows: Among them, σ is the activation function, w1 and w2 are weight coefficients, y^ is the final predicted value, y^ BERT and y^ LightGBM The predicted values of the target variable by the BERT and LightGBM models, respectively. The Stacking strategy is a powerful integration method that learns how to combine the outputs of multiple models through a meta-model in order to obtain better prediction performance than any single model. Model fusion can make full use of the advantages of different models to improve the robustness and accuracy of predictions.
[0025] Preferably, the Bayesian optimization algorithm is used to dynamically adjust the weight distribution of the BERT and LightGBM models. According to the sentiment intensity, context complexity and feature importance of each text, the contribution ratio of the two in the prediction task is optimized. The dynamic weight adjustment strategy can dynamically optimize the model fusion effect according to the data characteristics and improve the accuracy and adaptability of the prediction.
[0026] Preferably, the method further comprises optimizing the BERT model by adjusting gradient checkpoints, quantization technology, or adjusting dynamic batch size to improve the computational efficiency of the BERT model;
[0027] Preferably, the gradient checkpoint saves the gradient every 5 training steps, which can reduce the memory usage by about 20% and shorten the training time by about 15%.
[0028] Preferably, the quantization technology uses 8-bit integers for calculation; compared with using 32-bit floating-point numbers for calculation, the model size is reduced by about 60%, the inference speed is increased by about 2 times, and the performance loss is almost no, only about 3%.
[0029] Preferably, the dynamic batch size ranges from 16 to 128. The dynamic batching technology dynamically adjusts the batch size according to the actual available GPU memory size, ranging from 16 to 128, ultimately increasing GPU utilization by about 20% and reducing training time by about 10%.
[0030] Together, these optimizations improve the computational efficiency of the BERT model, enabling it to handle larger datasets and more complex computations in real time.
[0031] Preferably, it also includes implementing parallel computing and early stopping mechanism for the LightGBM model to ensure efficient training while preventing overfitting.
[0032] Preferably, the multi-threaded parallel computing parameter of the LightGBM model is set to 7, that is, the number of CPU cores minus one, to maximize parallel efficiency, while leaving one core for system processes, which significantly reduces the model training time and can increase the training speed on large-scale data sets by about 40%.
[0033] Preferably, for the application of the early stopping mechanism, by setting its patience parameter to 10, the training is stopped when there is no performance improvement for 10 consecutive rounds, thereby avoiding model overfitting, reducing the overall model training time by about 20%, and improving the generalization ability of the model.
[0034] Preferably, the report is generated by combining a variational autoencoder (VAE) with a recurrent neural network (RNN), which can generate an explainable report, improve the interpretability of mental health monitoring, help educators understand the model prediction logic, promote the scientificity and trustworthiness of decision-making, and thus provide students with more personalized and timely mental health support.
[0035] In the second aspect, the present invention also provides a mental health intelligent monitoring system integrating BERT and LightGBM, comprising:
[0036] Data acquisition module,
[0037] Data preprocessing module,
[0038] Model training module, including BERT model submodule, LSTM network submodule, feature integration module and LightGBM model submodule,
[0039] The BERT model submodule is used to train the BERT model and use the pre-trained BERR model to perform text feature extraction and classification prediction;
[0040] The LSTM network submodule is used to extract behavioral features using LSTM network and feature engineering;
[0041] The feature integration module is used to integrate the text features and behavior features to form an integrated feature vector.
[0042] The LightGBM model submodule uses integrated features to train the LightGBM model, and uses the pre-trained LightGBM model for classification prediction;
[0043] Model fusion module, which fuses the prediction results of the BERT model and the LightGBM model;
[0044] The platform monitoring and early warning module is used to provide a monitoring report based on the prediction results of the model.
[0045] The beneficial effects of the present invention are:
[0046] (1) The monitoring method and system of the present invention can improve the depth and accuracy of the sentiment analysis of the model. The present invention uses the BERT model to extract sentiment text features, uses the LSTM (Long Short-Term Memory Network) network and feature engineering to extract key behavioral features, and inputs these features into the LightGBM model for sentiment classification. Finally, the prediction and classification results of the BERT and LightGBM models are fused to achieve effective integration of the two and improve the depth and accuracy of the sentiment analysis of the model.
[0047] (2) The monitoring method and system of the present invention can enhance the adaptive analysis capability of the model. The present invention introduces a dynamic adaptive fusion strategy and uses the Bayesian optimization algorithm to dynamically adjust the weight configuration of BERT and LightGBM, and intelligently adjusts the analysis strategy according to the specific sentiment intensity, context complexity and feature importance of the text. This innovation improves the flexibility and adaptability of the model in complex situations, surpassing the static analysis limitations of traditional sentiment analysis models.
[0048] (3) The monitoring method and system of the present invention can optimize the computational efficiency and resource utilization of the model. In view of the fact that the BERT model consumes large computational resources, the present invention significantly reduces the computational resource requirements through multi-level optimization measures, such as gradient checkpoints, quantization technology, and dynamic batch processing. At the same time, the application of parallel computing and early stopping strategies to the LightGBM model effectively balances the efficiency and effect of model training, ensures the efficient operation and scalability of the model, and meets the requirements for response speed and resource management in practical applications.
[0049] (4) The monitoring method and system of the present invention can promote the intelligence and transparency of model mental health monitoring. The present invention improves the interpretability of mental health monitoring by introducing a variational autoencoder (VAE) combined with an explanatory report generated by a recurrent neural network (RNN), helps educators understand the model prediction logic, promotes the scientificity and trustworthiness of decision-making, and thus provides students with more personalized and timely mental health support. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 A schematic diagram of a method for intelligent monitoring of mental health according to an embodiment of the present invention;
[0051] Figure 2 A schematic diagram of data collection sources according to an embodiment of the present invention;
[0052] Figure 3 This is the workflow diagram of the BERT model;
[0053] Figure 4 This is a core technology diagram of the LightGBM model:
[0054] Figure 5 This is a brief architecture diagram of the variational autoencoder;
[0055] Figure 6 Schematic diagram of the structure of the mental health intelligent monitoring system according to an embodiment of the present invention. DETAILED DESCRIPTION
[0056] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiment is a module embodiment of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0057] The present invention provides a method for intelligent monitoring of mental health integrating BERT and LightGBM, comprising the following steps:
[0058] S1 Text data collection and preprocessing: This step mainly collects and processes a wide range of text data sources to provide high-quality input for subsequent model training. The data includes social platform text data, student card text data, etc.
[0059] Specifically, the steps may include:
[0060] S11: Multi-source data collection. Figure 2, students' text data can be collected from multiple channels such as campus network social platforms and student medical information, and data can be obtained through API interfaces and web crawler tools Scrapy to ensure the diversity and breadth of data sources. Use data sharding technology for parallel data collection to improve the efficiency of data acquisition. Then divide the data set into training sets and test sets in a 7:3 ratio to ensure the fairness of model training and evaluation. In this process, the privacy of student information should also be guaranteed. In addition to social platform data, the school information system API interface can also be used to collect each student's student card usage records, including but not limited to school hospital medical information, grade changes, etc. These data record students' daily life behaviors and are a valuable source of information reflecting their mental state.
[0061] S12 data preprocessing.
[0062] S121 For the collected social platform text information, since it comes from the network and has a large amount of information, the processing steps may include:
[0063] S1211 cleans and normalizes the collected text data from social platforms, including removing HTML tags, special characters, and duplicate data, and performing word segmentation, lemma restoration, and stop word removal.
[0064] S1212 uses NLTK and Spacy libraries for text preprocessing to improve the quality of text data.
[0065] S1213 uses the TF-IDF algorithm to vectorize text data. TF-IDF is a weighted technique commonly used in information retrieval and text mining, which aims to evaluate the importance of a term to a document or document collection. The TF-IDF formula is as follows:
[0066] tf-idf(t,d,D)=tf(t,d)×idf(t,D)
[0067] tf(t, d) represents the frequency of term t in document d. That is, the number of times term t appears in document d. It can be normalized by dividing by the total number of terms in the document to avoid the problem of naturally high term frequency in long documents. idf(t, d) stands for inverse document frequency, which measures the rarity of term t in the entire document set D.
[0068] S122 Student Card Data Processing
[0069] For the collected student card data, since the data is relatively structured, it can be pre-processed through simple steps, including data cleaning, deduplication, and time series construction, to prepare for subsequent feature extraction.
[0070] S2 Data annotation and enhancement: The purpose of this step is to expand the training data set and provide rich and balanced training samples for the model.
[0071] The following steps may be included:
[0072] S21: Text annotation. The annotation tool Doccano is used to annotate the data. The classification labels include multiple psychological states such as normal, anxious, and depressed. This can provide a high-quality training set for the establishment and training of subsequent models. During the annotation process, the annotation basis and specific psychological state description of each data should be recorded in detail to facilitate accurate learning and understanding of the model.
[0073] S22: Expand the training data set through data augmentation technology, use text replacement, insertion, deletion and other methods to generate diversified text data, and increase the generalization ability of the model. Then use the generative adversarial network (GAN) to generate more training samples to balance the category ratio in the data set. The generative adversarial network (GAN) generates new samples through a pair of mutually adversarial generators and discriminators. The generator uses random noise as input and learns to map to the function of the real data distribution, thereby creating synthetic data that looks real. The discriminator attempts to distinguish these generated samples from real samples. The two are continuously iterated and optimized during the training process until the generator can produce highly realistic data that is difficult for the discriminator to identify its authenticity. In this way, the category ratio of the data set is balanced and the diversity of the training data set is expanded.
[0074] S3 BERT model training and text feature extraction: mainly using social platform text data to pre-train the BERT model, and then extracting text features and predictions by fine-tuning the BERT model; ensuring that the model has good generalization capabilities and provides high-quality feature vectors for classification tasks. Including steps:
[0075] S31 BERK model pre-training.
[0076] The workflow diagram of the BERT model is as follows Figure 3As shown in the flowchart, the input information (input_ids, position_ids and token_type_ids) is converted into corresponding embedding vectors, which are added together to form the initial representation of the input sequence. Next, the multi-head attention mechanism is used to capture the relationship within the input sequence, which is the core part of the BERT model. After being processed by the Transformer encoder with the same structure multiple times, the input signal is further processed. After that, there is a step called "Attention_out_layer", which involves some linear transformations or other operations, followed by "Intermediate_layer" performing some intermediate calculations to prepare for the next step. The results produced by the above complex operations and calculations can be used for various natural language processing tasks, such as text classification, sentiment analysis, etc. The calculation formula of its self-attention mechanism is as follows:
[0077]
[0078] Among them, Q is the query matrix, which indicates based on which information to "query" or focus on other parts of the sequence, K is the key matrix, which represents the "key features" of each position in the sequence or the basis for being queried, and V is the value matrix, which contains the "value" information of each position in the sequence, that is, the part that needs to be queried and weighted.
[0079] The BERK model is trained by annotating the enhanced text dataset and then can be used for feature extraction.
[0080] S32: BERT model fine-tuning.
[0081] The trained BERT model also needs to be fine-tuned on the labeled dataset to adapt it to the student mental health text classification task.
[0082] The fine-tuning process includes adding a fully connected layer to the last layer and optimizing it using the cross entropy loss function, so that the fine-tuned BERT model can output the semantic embedding vector of each text. The cross entropy loss function is as follows:
[0083]
[0084] Where N is the number of samples, C is the number of categories, and yi j It represents the probability that the i-th sample actually belongs to the j-th category. It is usually in the form of one-hot encoding, that is, when sample i belongs to category j, yi j =1, otherwise yi j =0. is the probability that the model predicts that the i-th sample belongs to the j-th class.
[0085] In addition, in order to prevent overfitting, the Dropout layer can also be used to enhance the generalization ability of the model. The Dropout operation is described as follows:
[0086] Dropout(h i )=h i ·Bemoulli(p)
[0087] Here, h ij represents the output value of the i-th neuron in the hidden layer. Bernoulli(p) is a Bernoulli random variable with the probability of generating 0 or 1 respectively p and 1-p, where p is called the retention rate, which represents the probability that each neuron is retained (i.e., its output value remains unchanged) in a training iteration. When Dropout is actually applied, this process is performed independently for each neuron in the hidden layer: for each neuron’s output hi i , keep it at its original value with probability p, and set it to 0 with probability 1-p. In this way, the network structure will change each time you train, which is equivalent to training different sub-networks, thereby reducing the complex co-adaptation relationship between neurons and reducing overfitting.
[0088] S33: BERT model text feature extraction.
[0089] The text feature vector is extracted from the fine-tuned BERT model. Preferably, the feature dimensionality reduction technique PCA is used to reduce the dimensionality of the high-dimensional feature vector to reduce the number of features and improve computational efficiency.
[0090] The dimensionality reduction formula of PCA is Z=XW, where Z is the feature matrix after dimensionality reduction, X is the original feature matrix, and W is the feature vector matrix. Feature dimensionality reduction can effectively reduce data dimensions and computational complexity while retaining most of the useful information.
[0091] S4 behavioral feature extraction;
[0092] Unlike the BERT model, which is suitable for processing text data, the present invention uses long short-term memory network (LSTM) and feature engineering to extract key behavioral features for the structured data of student cards. LSTM is a special recurrent neural network structure designed to solve long-term dependency problems, that is, it can remember long-term historical information when processing sequence data. In this way, by analyzing a student's school hospital visit information, grade changes, etc., it is possible to extract indicators such as the student's physical condition and learning enthusiasm as supplementary features of the student's mental health status.
[0093] S5 feature integration: Integrate the extracted text features and behavioral features to form an integrated feature vector.
[0094] The behavioral features extracted from the student card text information (such as mental health counseling records and grade changes) are combined with the text features of the campus network social platform extracted by the BERT model to form a comprehensive feature vector.
[0095] The text features extracted by the BERT model represent the emotional tendencies and topic preferences of students on social platforms, while the behavioral features extracted from the student card information include important behavioral and historical information such as psychological counseling records and grade changes. These features can be regarded as multidimensional representations of the mental health status of individual students. By integrating these features using weighted average feature fusion technology, the model's predictive and generalization capabilities can be enhanced, ensuring that the model can make more accurate predictions.
[0096] The weighted average formula is as follows:
[0097] F=aF1+(1-α)F2
[0098] Among them, F is the integrated feature vector, F1 and F2 are the feature vectors to be fused, and α is the weight coefficient. α can be a fixed preset value or learned from the data through the grid search method to achieve the best fusion effect. In addition, the weighted average method can be further extended to the fusion scenarios of more feature vectors, just by assigning a weight coefficient to each feature vector and ensuring that the sum of all weight coefficients is equal to 1.
[0099] In this way, the model can make full use of various types of data to gain a more comprehensive understanding of students' mental health status and improve prediction accuracy.
[0100] S6 LightGBM model training and classification prediction: The integrated feature vector is input into the pre-trained LightGBM model to obtain the classification prediction result.
[0101] First, the integrated feature vector is used to train the LightGBM model to classify the students' mental health status, and then the trained model is used for classification.
[0102] The LightGBM framework as a whole includes a series of key technologies to solve the problems of gradient boosted decision trees (GBDT) on large-scale datasets. Figure 4 The four core technologies of LightGBM are demonstrated: gradient unilateral sampling method, mutually exclusive feature bundling algorithm, histogram algorithm, and leaf splitting. These technologies are optimized for computational complexity and memory usage, feature combination, data discretization, and decision tree node classification, respectively, enabling LightGBM to significantly improve training speed and memory efficiency while maintaining high accuracy.
[0103] S7 model fusion: Fusion the prediction results of the BERT model and the LightGBM model.
[0104] In order to improve the prediction performance and stability of the overall model and ensure the accuracy and reliability of the classification results, the features are fused. Preferably, the Stacking strategy is adopted to fuse the prediction results of the BERT model and the LightGBM model. The outputs of the two models are weightedly combined by training a meta-learner logistic regression to improve the overall prediction performance. The Stacking strategy formula is as follows:
[0105]
[0106] Among them, σ is the activation function, w1 and w2 are weight coefficients, y^ is the final predicted value, y^ BERT and y^ LightGBM The predicted values of the target variable by the BERT and LightGBM models, respectively. The Stacking strategy is a powerful integration method that learns how to combine the outputs of multiple models through a meta-model in order to obtain better prediction performance than any single model. Model fusion can make full use of the advantages of different models to improve the robustness and accuracy of predictions.
[0107] S8 generates a report based on the fusion model prediction results.
[0108] Preferably, a variational autoencoder (VAE) combined with a recurrent neural network (RNN) is used to generate an explanatory report on changes in students' mental health status. The generated explanatory report helps educators to understand students' mental health status in a timely manner and provide targeted psychological intervention measures. The report content may include sentiment analysis results, mental health status assessment, change trend analysis, etc.
[0109] The generation formula of the variational autoencoder is as follows:
[0110]
[0111] The variational autoencoder can be broken down into two parts to understand. The first half is the reconstruction loss, which measures the likelihood of reconstructing the original input data X from the latent variable z through the decoder. The second half is the KL divergence, which measures the difference between the posterior distribution q(z|X) and the prior distribution p(z), where q(z|X) is the distribution of the latent variable z about the data point X provided by the encoder, and p(z) is usually a simple prior distribution, such as a standard normal distribution. Minimizing the KL divergence means making the posterior distribution as close as possible to the pre-set prior distribution, which helps make the latent space more regular and interpretable while avoiding overfitting.
[0112] The brief architecture diagram of the variational autoencoder is as follows Figure 5 As shown in the figure, the core idea of the variational autoencoder is mean variance calculation and normal distribution sampling. First, the mean and variance are calculated from real samples (such as X1, X2, X3, etc.). These statistical parameters are used to describe the central tendency and dispersion of the data. Next, these means and variances are used to generate a normal distribution, and variables (such as Z1, Z2, Z3, etc.) are sampled from them. These sampled variables are input into the generator to generate new samples (such as X′1, X′2, X′3, etc.). Finally, the generated samples are compared with the real samples to evaluate the performance of the generator and the quality of the generated samples.
[0113] Model Optimization
[0114] Furthermore, the model can be optimized to further improve the prediction performance and stability of the overall model and ensure the accuracy and reliability of the classification results. This can include:
[0115] Optimize LightGBM model parameters , preferably, the Grid Search method is used to optimize the LightGBM model parameters. The optimized parameters include learning rate, number of trees, maximum depth, etc. The purpose of parameter optimization is to find the best parameter combination to improve the prediction accuracy of the model. The optimization method is as follows:
[0116]
[0117] θ is the parameter of the model, and Θ is the set of all possible parameter values, i.e. the parameter space. It is a loss function used to quantify the difference between the predicted value y^i and the true value y i The degree of mismatch between them, for each sample i, from 1 to n, its loss is calculated.
[0118] BERT and LightGBM weight allocation adjustment and optimization. Preferably, a dynamic weight adjustment strategy is adopted, which can dynamically optimize the model fusion effect according to the data characteristics and improve the accuracy and adaptability of the prediction. In one embodiment, the Bayesian optimization algorithm is used to dynamically adjust the weight allocation of BERT and LightGBM, and the contribution ratio of the two in the prediction task can be optimized according to the sentiment intensity, context complexity and feature importance of each text. The Bayesian optimization algorithm is as follows:
[0119]
[0120] Bayesian optimization is not just a direct process of finding the maximum expected value, but a sequential decision-making process that iteratively updates the model, selects observation points, and gradually approaches the global optimal solution. Although traditional methods such as manual parameter adjustment can find better parameter combinations, they usually require a lot of computing resources and time, especially in high-dimensional parameter spaces. The Bayesian optimization algorithm estimates the distribution of the objective function by constructing a Gaussian process surrogate model, and uses this surrogate model to select the most promising parameters for evaluation. This method significantly reduces the number of parameter combinations that need to be evaluated, thereby speeding up the optimization process. Through specific effect comparison, the introduction of the Bayesian optimization algorithm can reduce the parameter search time by about 40%, and in the same optimization time, the computing resource utilization rate is increased by about 25%.
[0121] BERT model computing efficiency optimization
[0122] To address the efficiency challenges of large-scale data processing, a multi-level computing optimization strategy can be implemented, and the BERT model can be optimized using gradient checkpoints, quantization technology, or dynamic batch processing technology to improve the computational efficiency of the BERT model.
[0123] For example, in one embodiment, the gradient checkpoint technology saves the gradient every 5 training steps, which can reduce memory usage by about 20% and shorten the training time by about 15%.
[0124] In one embodiment, the quantization technology uses 8-bit integers instead of 32-bit floating point numbers for calculations, which reduces the model size by about 60%, increases the inference speed by about 2 times, and has almost no performance loss, only about 3%.
[0125] Dynamic batching technology dynamically adjusts the batch size according to the actual available GPU memory size, ranging from 16 to 128, ultimately improving GPU utilization by about 20% and reducing training time by about 10%.
[0126] These optimizations can be used simultaneously to improve the computational efficiency of the BERT model, enabling it to handle larger data sets and more complex calculations in real time.
[0127] LightGBM model operation optimization
[0128] Parallel computing and early stopping mechanisms are implemented for the LightGBM model to ensure efficient training while preventing overfitting. For example, in one embodiment, parallel computing is performed by setting the multi-threaded parallel computing parameter of LightGBM to 7, that is, the number of CPU cores minus one, to maximize parallel efficiency, while leaving one core for system processes, which significantly reduces the model training time, and the training speed on large-scale data sets can be increased by about 40%. The application of the early stopping mechanism, in one embodiment, by setting its patience parameter to 10, stops training when there is no performance improvement for 10 consecutive rounds, avoids model overfitting, reduces the overall model training time by about 20%, and improves the generalization ability of the model.
[0129] In summary, it can be seen that the monitoring method of the present invention achieves effective integration of the two by fine-tuning the BERT model to extract sentiment features, integrating these features with other features and inputting them into the LightGBM model for sentiment classification, and then fusion of the prediction results of the BERT and LightGBM models, and outputting a report based on the fusion prediction results, which can improve the depth and accuracy of the sentiment analysis of the model. Furthermore, by introducing a dynamic adaptive fusion strategy and using the Bayesian optimization algorithm to dynamically adjust the weight configuration of BERT and LightGBM, the flexibility and adaptability of the model in the face of complex situations are improved, surpassing the static analysis limitations of traditional sentiment analysis models.
[0130] At the same time, through multi-level optimization measures such as gradient checkpoints, quantization technology and dynamic batch processing, the demand for computing resources is significantly reduced. At the same time, the application of parallel computing and early stopping strategies to the LightGBM model effectively balances the efficiency and effect of model training, ensures the efficient operation and scalability of the model, and meets the needs of response speed and resource management in practical applications.
[0131] Moreover, by introducing the explanatory reports generated by the variational autoencoder (VAE) combined with the recurrent neural network (RNN), the interpretability of mental health monitoring is improved, which helps educators understand the model prediction logic, promotes the scientific nature and trustworthiness of decision-making, and thus provides students with more personalized and timely mental health support.
[0132] The embodiment of the present invention also provides a mental health intelligent monitoring system integrating BERT and LightGBM, referring to Figure 6 ,include,
[0133] The data collection module is used to collect student text data from multiple channels, including campus network social platform text data and student card text data;
[0134] Data preprocessing module,
[0135] Model training module, including BERT model submodule, LSTM network submodule, feature integration module and LightGBM model submodule,
[0136] The BERT model submodule is used to train the BERT model and use the pre-trained BERR model to perform text feature extraction and classification prediction;
[0137] The LSTM network submodule is used to extract behavioral features using LSTM network and feature engineering;
[0138] The feature integration module is used to integrate the text features and behavior features to form an integrated feature vector.
[0139] The LightGBM model submodule uses integrated features to train the LightGBM model, and uses the pre-trained LightGBM model for classification prediction;
[0140] Model fusion module, which fuses the prediction results of the BERT model and the LightGBM model;
[0141] The platform monitoring and early warning module is used to provide a monitoring report based on the prediction results of the model.
[0142] The system can be deployed in the actual operating environment and integrated into the school social platform. Through API interface design, efficient call of model services and instant result feedback can be achieved, supporting the system to perform real-time data analysis and processing. The system can monitor students' mental health status in real time and provide personalized mental health advice and intervention measures based on the model prediction results.
[0143] The accuracy and effectiveness of the system can be maintained by regularly updating the training data and model parameters.
[0144] Preferably, a model optimization module is also included, and the model optimization module can optimize the model using the above-mentioned model optimization strategy to improve the prediction performance and stability of the overall model and ensure the accuracy and reliability of the classification results.
[0145] For example, in one embodiment, the model optimization module includes an optimization sub-module one, an optimization sub-module two, and an optimization sub-module three. The optimization sub-module one adopts the above-mentioned dynamic weight adjustment strategy to dynamically adjust the weight distribution of BERT and LightGBM; the optimization sub-module two adopts the grid search method to optimize the LightGBM model parameters; the optimization sub-module three can implement parallel computing and early stopping mechanism for the LightGBM model for computational optimization; the optimization sub-module four optimizes the BERT model by implementing a multi-level computing optimization strategy.
[0146] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.
[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions provided according to the embodiments of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the aforementioned embodiments, it can be understood by those of ordinary skill in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit provided according to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles provided according to the embodiments of the present invention should be included in the protection scope provided according to the embodiments of the present invention.
Claims
1. A mental health intelligent monitoring method integrating BERT and LightGBM, characterized in that: The steps include: Data collection and preprocessing; Data annotation and enhancement; Use the pre-trained BERT model to extract text features and classification predictions; use the LSTM network and feature engineering to extract behavioral features; Integrate the extracted text features and behavioral features to form an integrated feature vector; Use the pre-trained LightGBM model to obtain classification prediction results; Fusion of the prediction results of the BERT model and the LightGBM model; Generate a report based on the fusion model prediction results.
2. The monitoring method according to claim 1, characterized in that: The method comprises the following steps: the data comprises social platform text data and student card data; the text features come from the social platform text data, and the behavior features come from the student card data.
3. The monitoring method according to claim 2, characterized in that: The social platform text data preprocessing includes: Clean and normalize the collected text data; Text preprocessing using NLTK and Spacy libraries, Use the TF-IDF algorithm to vectorize text data. The TF-IDF formula is as follows: tf-idf(t,d,D)=tf(t,d)×idf(t,D) tf(t, d) represents the term frequency of term t in document d, and idf(t, d) represents the inverse document frequency.
4. The monitoring method according to claim 1, characterized in that: "Using a pre-trained BERT model to extract text features and classification predictions" includes the steps of: pre-training the BERT model, and then extracting text features and classification predictions by fine-tuning the BERT model. The fine-tuning of the BERT model means adding a fully connected layer to the last layer and optimizing it using a cross entropy loss function. The cross entropy loss function formula is as follows: Where N is the number of samples, C is the number of categories, and y ij It represents the probability that the i-th sample actually belongs to the j-th class.
5. The monitoring method according to claim 1, characterized in that: The stacking strategy is used to fuse the prediction results. The outputs of the two models are weighted combined by training a meta-learner logistic regression. The formula is as follows: Among them, σ is the activation function, w1 and w2 are weight coefficients, y^ is the final predicted value, and y ^ BERT and y^ L ight GBM They are the predicted values of the target variables by the BERT and LightGBM models respectively.
6. The monitoring method according to claim 5, characterized in that: The Bayesian optimization algorithm is used to dynamically adjust the weight distribution of the BERT and LightGBM models.
7. The monitoring method according to claim 1, characterized in that: It also includes optimizing the BERT model by adjusting the gradient checkpoint, quantization technology, or adjusting the dynamic batch size; preferably, the gradient checkpoint is to save the gradient every 5 training steps; Preferably, the quantization technique is to use 8-bit integers for calculation; Preferably, the dynamic batch size ranges from 16 to 128.
8. The monitoring method according to claim 1, characterized in that: It also includes the implementation of parallel computing and early stopping mechanism for the LightGBM model. Preferably, the multi-threaded parallel computing parameter of the LightGBM model is set to 7; Preferably, the patience parameter of the LightGBM model is set to 10.
9. The monitoring method according to claim 1, characterized in that: Generate reports through variational autoencoders combined with recurrent neural networks.
10. A mental health intelligent monitoring system integrating BERT and LightGBM, characterized in that: include, Data acquisition module, Data preprocessing module, Model training module, including BERT model submodule, LSTM network submodule, feature integration module and LightGBM model submodule, The BERT model submodule is used to train the BERT model and use the pre-trained BERR model to perform text feature extraction and classification prediction; The LSTM network submodule is used to extract behavioral features using LSTM network and feature engineering; The feature integration module is used to integrate the text features and behavior features to form an integrated feature vector. The LightGBM model submodule uses integrated features to train the LightGBM model, and uses the pre-trained LightGBM model for classification prediction; Model fusion module, which fuses the prediction results of the BERT model and the LightGBM model; The platform monitoring and early warning module is used to provide a monitoring report based on the prediction results of the model.
Citation Information
Cited By
English writing grammar error detection system based on deep learning
CN120146036A
