Federal learning-based power marketing short text grading method and system

Abstract

The invention relates to the technical field of electric power text grading, in particular to an electric power marketing short text grading method and system based on federal learning. On the basis of an ERNIE text grading model, aiming at the characteristics of short text data, an ERNIE natural language grading model based on a federated learning framework is provided, and a dynamic multi-head sparse attention mechanism is innovatively introduced into a downstream network of the model. According to the mechanism, by dynamically adjusting the multi-head attention weight and introducing the sparsification strategy, the classification capability of the model on short text data is effectively improved, so that when complex and changeable short texts are processed, key information can be captured more accurately, and the accuracy and robustness of text classification are enhanced. Meanwhile, the model constructs a multi-center data joint training mechanism, and direct sharing of original data is avoided through a distributed cooperative training mode.

Description

Technical Field

[0001] The present invention relates to the technical field of power text classification, and in particular to a method and system for classifying power marketing short texts based on federated learning. Background Art

[0002] Power data is often presented as short text, including field names, descriptions, data types, and permission settings. This text integrates a wealth of structured and unstructured information, which, from a machine learning perspective, constitutes important feature input. Business requirements require classifying data types and determining security levels based on these features. For plain text metadata, features are often expressed in the form of word vectors, and categories are distinguished using one-hot encoding. However, faced with massive and diverse data, how to efficiently and accurately extract key information and provide support to the power industry through analysis remains a key research focus.

[0003] With the continuous breakthroughs in artificial intelligence technology, especially the gradual advancement in the fields of natural language processing (NLP) and deep learning, these technologies have provided a new path for improving data processing and analysis methods and enhancing the application efficiency of power data. Natural language processing technology has shown significant advantages in the automated analysis, intelligent monitoring and abnormal warning of power data, and has received positive reviews in practical applications. With the optimization of algorithms and the improvement of hardware capabilities, the intelligent application of power data is driving the industry to transform to a more accurate and efficient operation model. Accordingly, the present invention securely classifies the raw power data through a negative list mechanism, and combines artificial intelligence technology to construct a multi-level ERNIE data processing model based on federated learning and differential privacy framework, aiming to provide innovative solutions for the intelligent development of the power industry. Summary of the Invention

[0004] To address the problems in the existing technology, this invention provides a method and system for grading short texts in power marketing based on federated learning, which improves data processing efficiency and calculation accuracy. The specific technical solution is as follows:

[0005] A method for grading short texts in power marketing based on federated learning includes the following steps:

[0006] Step S1: collecting and processing historical power marketing short text data samples, and classifying the processed historical power marketing short text data samples into a high security level and a low security level; wherein the historical power marketing short text data samples include several indicators;

[0007] Step S2: performing word segmentation and encoding operations on the power marketing short text data sample, and inputting the data into the ERNIE model to pre-train the ERNIE model;

[0008] Step S3, introducing the dynamic multi-head sparse attention mechanism into the pre-trained ERNIE model to obtain the ERNIE attention mechanism model;

[0009] Step S4: The central server sends the ERNIE attention mechanism model to each client, so that each client uses local data to train the ERNIE attention mechanism model to obtain a local model.

[0010] In step S5, each client uploads the trained local model parameters to the central server. The central server performs a weighted average of the local model updates of each client based on the amount of data, and generates an initial global model.

[0011] In step S6, the central server saves the initial global model and sends it back to each client, so that each client can train the received initial global model. When the accuracy of the trained initial global model converges and reaches a preset value, a trained global model is obtained.

[0012] Step S7: Obtain and input the processed actual power marketing short text data into the trained global model to obtain the probability that the actual power marketing short text data belongs to a high security level and a low security level;

[0013] Step S8: Determine the security level of the power marketing short text data sample based on the comparison result of the probability of belonging to the high security level and the probability of the low security level with the probability threshold.

[0014] Preferably, the processing of historical power marketing short text data samples includes:

[0015] The historical power marketing short text data samples with multiple indicators are processed including missing values, outliers and duplicate values.

[0016] Preferably, the segmentation and encoding of the power marketing short text data samples and inputting them into the ERNIE model, and pre-training the ERNIE model specifically include the following steps:

[0017] Convert the numerical data in the historical power marketing short text data sample with multiple indicators into text representation according to a preset encoding method and add corresponding tags;

[0018] The text data in the historical power marketing short text data samples were segmented and indexed according to the vocabulary corresponding to the ERNIE pre-trained model. Each word was mapped to a corresponding ID sequence, and finally a dataset was generated that can be directly input into the ERNIE model for training and prediction. The dataset was input into the ERNIE model for pre-training.

[0019] Preferably, the word segmentation operation includes:

[0020] The historical power marketing short text data samples are divided into three parts, each corresponding to a masking task, where the three parts are word-level data, phrase-level data and entity-level data.

[0021] Preferably, the method specifically includes the following steps:

[0022] After the dataset is input into the ERNIE model, the model's embedding layer maps each word ID into a vector, adds position embedding and sentence embedding, and then passes through multiple Transformer layers to capture dependencies and contextual information between words. The model uses text feature output and inputs the text features into the dynamic multi-head sparse attention mechanism. The dataset includes word IDs, validity masks, and sentence type information.

[0023] Preferably, the method of introducing a dynamic multi-head sparse attention mechanism into the pre-trained ERNIE model to obtain an ERNIE attention mechanism model specifically includes the following steps:

[0024] The input data comes from the text features of the ERNIE model. The shape of the text features includes the number of samples per training, the number of words in each input sample, and the embedding dimension of each word in the ERNIE model.

[0025] The data is linearly transformed to generate query vectors, key vectors, and value vectors. The linear transformation maps the input text features into three different spaces, corresponding to the query, key, and value in the attention mechanism. The query vector, key vector, and value vector are generated through three linear layers. The output shape includes the number of samples in each training session, the number of words in each input sample, and the embedding dimension of each word in the ERNIE model.

[0026] Split the generated query vector, key vector, and value vector according to the number of attention heads, thereby reshaping the query vector, key vector, and value vector into a multi-head form to obtain a tensor shape;

[0027] The attention score is calculated by the inner product between the query vector and the key vector. The attention score indicates the importance of each word to other words.

[0028] Calculate the highest confidence value according to the preset sparsity parameter. For each attention score vector, only the value with the highest confidence value is retained, and the remaining positions are set to zero, and then renormalized;

[0029] Dynamically select the most important attention heads through a gating mechanism:

[0030] The input preset is averaged along the sequence dimension to obtain a tensor whose shape includes the number of samples per training and the embedding dimension of each word in the ERNIE model;

[0031] The shape of the averaged tensor, which includes the number of samples per training and the embedding dimension of each word in the ERNIE model, is input into a fully connected layer, where the output dimension of the fully connected layer is set to the number of attention heads;

[0032] Apply the Sigmoid activation function to the output of the linear layer and normalize the output value to the interval [0, 1]. The resulting score tensor has the shape of the number of samples and the number of attention heads for each training. Each element of the score tensor represents the importance score of the corresponding attention head.

[0033] After obtaining the output of the scoring tensor whose shape includes the number of samples and the number of attention heads for each training, the number of selected attention heads is calculated according to the preset selection rate to obtain the number of confidence attention heads. For each sample, the highest confidence operation is used to select the number of attention heads with the highest score from the scoring tensor, and the index of the head is obtained;

[0034] Use the selected index to construct a binary mask for each sample. In each sample, the position of the selected confidence attention head is set to 1, and the position of the unselected head is set to 0; use the binary mask to set the attention score corresponding to the unselected head to zero;

[0035] Finally, the filtered text features are obtained based on the selected header.

[0036] Preferably, the calculation formula for performing weighted averaging on the local model update of each client according to the amount of data of the client is:

[0037]

[0038] in, is the global model obtained by round t aggregation; K is the number of local models; represents the kth local model trained in the tth round; |D| represents the total number of samples in each client dataset, |D k | represents the number of samples of local data from the kth client.

[0039] Accordingly, the present invention also provides a power marketing short text grading system based on federated learning, which is applied to the above method and includes:

[0040] A data collection and processing module is used to collect and process historical power marketing short text data samples, and classify the processed historical power marketing short text data samples into a high security level and a low security level; wherein the historical power marketing short text data samples include several indicators;

[0041] The ERNIE model pre-training module is used to perform word segmentation and encoding operations on power marketing short text data samples, and input them into the ERNIE model for pre-training.

[0042] The multi-head sparse attention module is used to introduce the dynamic multi-head sparse attention mechanism into the pre-trained ERNIE model to obtain the ERNIE attention mechanism model;

[0043] The central server is configured to send the ERNIE attention mechanism model to each client, so that each client trains the ERNIE attention mechanism model using local data to obtain a local model; save the initial global model and send it back to each client, so that each client trains the received initial global model; when the accuracy of the trained initial global model converges and reaches a preset value, a trained global model is obtained; obtain and input the processed actual power marketing short text data into the trained global model to obtain the probability that the actual power marketing short text data belongs to a high security level and a low security level; determine the security level of the power marketing short text data sample based on the comparison result of the probability of belonging to the high security level and the low security level with the probability threshold;

[0044] The client uploads the trained local model parameters to the central server. The central server performs a weighted average of the local model updates of each client based on the amount of data to generate an initial global model.

[0045] Accordingly, the present invention also provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the power marketing short text grading method based on federated learning as described above.

[0046] Accordingly, the present invention also provides a processor for running a program, wherein when the program is running, the method for grading short texts of power marketing based on federated learning as described above is executed.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] Based on the ERNIE text grading model, this paper proposes an ERNIE natural language grading model based on a federated learning framework, targeting the characteristics of short text data. It also innovatively introduces a dynamic multi-head sparse attention mechanism into the model's downstream network. This mechanism effectively improves the model's ability to grade short text data by dynamically adjusting the multi-head attention weights and introducing a sparsification strategy, enabling it to more accurately capture key information when processing complex and changeable short texts, thereby enhancing the accuracy and robustness of text grading. At the same time, the model constructs a multi-center data joint training mechanism, which avoids the direct sharing of raw data through distributed collaborative training. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly describes the drawings required for the specific embodiments or the description of the prior art. Similar elements or parts are generally identified by similar reference numerals throughout the drawings. Elements or parts in the drawings are not necessarily drawn to scale.

[0050] Figure 1 It is a schematic diagram of the process of the present invention.

[0051] Figure 2 This is a structural diagram of the ERNIE+DMHA model of the present invention.

[0052] Figure 3 This is a flowchart of the federated learning of the present invention.

[0053] Figure 4 This is a diagram of the training results of each client without federated learning.

[0054] Figure 5 This is a graph of the training results of each client under Bangxue differential privacy.

[0055] Figure 6 This is a system principle diagram of the present invention. DETAILED DESCRIPTION

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0057] It will be understood that when used in this specification and the appended claims, the terms “comprises” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0058] It should also be understood that the terms used in the present specification are only for the purpose of describing particular embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0059] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0060] Example 1:

[0061] Data sets in the power sector are often presented in the form of short texts. Intelligent data grading based on a negative list mechanism can support differentiated governance needs. However, the volume of power data has exceeded manual management capabilities, which may not only lead to improper resource allocation, but also induce chain reactions due to human operational errors. In particular, in the processing of enterprise energy consumption and critical infrastructure data, the security protection level still needs to be improved. In order to efficiently extract short text features, the present invention improves the ERNIE model and designs a dynamic multi-head sparse attention mechanism to optimize the grading performance. At the same time, in order to eliminate the potential privacy leakage risks in traditional centralized learning methods, a multi-center natural language grading training method based on federated learning is proposed by utilizing multi-center data under privacy protection. It avoids sharing of original data through distributed collaboration, integrates differential privacy technology, and combines a multi-layer security level system constructed by a negative list to achieve effective training of diversified data. Experimental results show that this method performs outstandingly in negative list-driven management scenarios, significantly improves processing efficiency and accuracy, and achieves an excellent balance between privacy protection and model performance, providing an innovative path for intelligent classification and grading of power data. The specific plan is as follows:

[0062] like Figure 1As shown, this embodiment provides a method for grading short texts for power marketing based on federated learning. The present invention integrates the ERNIE model into the collaborative grading scheme of federated learning (FL). Similar to centralized machine learning, FL distributes and stores data on local devices rather than on a central server, which not only ensures the privacy and security of user data, but also supports collaborative training of multiple devices. Each client independently completes model training using local data, and uploads the updated parameters to the server, which aggregates and generates a global model. This method not only effectively prevents the risk of data leakage, but also significantly reduces communication costs. The present invention adopts a multi-client collaborative training mechanism based on the FedAvg (federated parameter averaging) algorithm, and jointly constructs a global model through local training and parameter aggregation of each client. This solution can efficiently cope with the challenges of privacy protection and negative list-driven grading in a multi-center distributed environment. The multi-center federated collaborative mechanism breaks through the limitations of single-point training of traditional natural language processing models. Through a distributed architecture, each participant fine-tunes the model based on local data, while ensuring the physical isolation of data, and fully explores the data correlation characteristics across regions or departments. It includes the following steps:

[0063] Step S1, collecting and processing historical power marketing short text data samples, and dividing the processed historical power marketing short text data samples into a high security level and a low security level; wherein the historical power marketing short text data samples include several indicators.

[0064] As a preferred solution, the processing of historical power marketing short text data samples includes:

[0065] The historical power marketing short text data samples with multiple indicators are processed including missing values, outliers and duplicate values.

[0066] The data cleaning process for power marketing short text data with multiple indicators was first performed after removing the raw data read from the power grid's CSV data files. The processing of this sample of power marketing short text data with multiple indicators specifically involved handling missing values, outliers, and duplicate values. For missing values, appropriate filling methods were used based on the characteristics of the specific indicator. For example, for certain indicators with time series correlation, the mean of the previous and next data was used to ensure data integrity. For outliers, reasonable thresholds were set and corrected to values ​​within a reasonable range. Duplicate records in the dataset were also removed to ensure that each piece of data was independent and valid, laying a solid foundation for subsequent processing.

[0067] According to pre-set classification rules, data samples are divided into high security level and low security level according to the importance of each indicator determined in the CSV power grid data. Specifically, the weight of different indicators in the safe operation of the power grid and the degree of their impact on the overall security status when reaching different thresholds are comprehensively considered. By formulating corresponding logical judgment conditions, each data sample is given a corresponding classification label. For example, when the key indicators are continuously at a low level and other related indicators also show abnormal conditions, the sample is marked as a low security level; conversely, when all indicators perform well and meet high safety operation standards, they are marked as a high security level. The hierarchical labels generated in this way serve as the basis for supervised learning and provide clear target guidance for subsequent learning and prediction using the ERNIE model.

[0068] During implementation, the initial power marketing short text data, stored as worksheets in an Excel file, was converted into a DataFrame0 (think of it as a table with rows and columns). The Pandas library function was used to read each power data worksheet from the specified Excel file. This read operation returned a dictionary-type dfs, with the worksheet name as the key and the corresponding DataFrame as the value. Using Pandas library functions, multiple DataFrames were inner-joined using the shared column*table code to generate a new DataFrame df_out. After the merge, each row in df_out contained all field information from the data dictionary and the system data table, with a shape of [n,m]. Rule replacement was then performed to normalize the text and map it to numeric values. The replacement rule was applied to each row of the DataFrame, normalizing or mapping the values ​​in each field that met the rule (for example, converting text to numeric values) to ensure consistency in subsequent comparisons and filtering. The replaced data maintained its shape, but the data content and some data types changed.

[0069] Next, we filter the data by security level and filter df_out using the security level rules to obtain three DataFrames:

[0070] high_security_data: only keep rows that meet high security level rules;

[0071] medium_security_data: retains rows that meet the medium security level rules;

[0072] low_security_data: Keep rows that meet the low security level rules.

[0073] Generate DataFrames that meet high, medium, and low security requirements, with shapes [n_high,m], [n_medium,m], and [n_low,m], respectively. n_high, n_medium, and n_low represent the number of rows of data retained after passing the security level filter (high, medium, and low, respectively). m is the number of columns in the DataFrame, representing the number of fields contained in the data.

[0074] Clean the resulting data. Use dropna() (a very common method in Pandas) on each filtered DataFrame to remove rows with null values ​​to ensure the integrity of the final output data. The number of rows in each DataFrame may be further reduced, with the shape becoming [n_high_clean,m], [n_medium_clean,m], [n_low_clean,m]. Use Pandas' to_csv method to save the security-level filtered and cleaned DataFrames as CSV files, with the file paths corresponding to high, medium, and low security levels, respectively.

[0075] CSV files containing high-security and low-security text data are then used as training data samples. After data initialization, the initial data is stored in a DataFrame, with each row containing a record containing text feature fields (the text_feature_nums column), non-text feature fields (the non_text_feature_nums column), and a label field.

[0076] Step S2: perform word segmentation and encoding operations on the power marketing short text data sample, input the data into the ERNIE model, and pre-train the ERNIE model.

[0077] As a preferred solution, the segmentation and encoding of the power marketing short text data samples and the input of the data into the ERNIE model, and the pre-training of the ERNIE model specifically include the following steps:

[0078] Convert the numerical data in the historical power marketing short text data sample with multiple indicators into text representation according to a preset encoding method and add corresponding tags;

[0079] The text data in the historical power marketing short text data samples were segmented and indexed according to the vocabulary corresponding to the ERNIE pre-trained model. Each word was mapped to a corresponding ID sequence, and finally a dataset was generated that can be directly input into the ERNIE model for training and prediction. The dataset was input into the ERNIE model for pre-training.

[0080] ERNIE is a large-scale pre-trained language model launched by Baidu. By integrating knowledge masking technology at three levels: words, phrases, and entities, the model can have a deeper understanding of language patterns. Compared with BERT, ERNIE shows stronger advantages in text generation tasks. Its core lies in the integration of language units at different levels (including grammatical and semantic knowledge), thereby significantly enhancing language expression capabilities. In terms of model architecture, ERNIE is based on the Encoder structure of Transformer, which is consistent with models such as GPT and BERT. Studies have shown that although both BERT and ERNIE rely on pre-training tasks to learn language knowledge, BERT's dual-task design has limitations in long text generation, while ERNIE's three-level masking mechanism is more efficient in grasping complex language features.

[0081] To run a deep learning model, the data needs to be converted into a format that the deep learning model can accept. The extracted and processed feature data and the corresponding hierarchical labels are integrated and organized into a text sequence format that ERNIE can accept.

[0082] As a preferred solution, the word segmentation operation includes:

[0083] The historical power marketing short text data samples are divided into three parts, each corresponding to a masking task, where the three parts are word-level data, phrase-level data and entity-level data.

[0084] Specifically, word segmentation divides the power marketing short text data samples into three parts, each of which corresponds to a masking task. The three parts are word-level data, phrase-level data, and entity-level data.

[0085] Word-level data: This data is used to enhance the model's basic vocabulary prediction capabilities. Individual words or characters in the original text are randomly masked, and the model must predict these masked words based on the context. This training method helps the model learn word-level language patterns, improving its understanding of basic vocabulary.

[0086] Phrase-level data: This is used to train the model's phrase understanding and prediction capabilities. Phrases or collocations consisting of multiple words are completely obscured, and the model must predict the obscured phrase based on the context. This training method enables the model to better understand the overall semantics and structure of phrases, improving its language processing capabilities at the phrase level.

[0087] Entity-level data: This is used to train the model's entity recognition and semantic understanding capabilities. Named entities or entities with specific semantics in text are obscured, and the model must predict the obscured entities based on the context. This training method helps the model identify and understand key entities, improving its semantic understanding capabilities.

[0088] Through this layered data processing approach, the ERNIE model is able to learn language knowledge more comprehensively, thereby performing well in text generation and comprehension tasks.

[0089] The ERNIE 3.0 series is optimized based on the ERNIE model, integrating the self-supervised contrastive learning pre-training strategy of word mixtures and the self-adversarial fine-tuning technology of word mixture data enhancement, thereby significantly surpassing the performance of the original ERNIE model in many mainstream Chinese downstream tasks. In this invention, we use ERNIE 3.0 as the core model and apply it to the classification task of power grid data. At the same time, in response to the shortcomings of the traditional multi-head self-attention mechanism in long text processing, which has high computational complexity and easy introduction of redundant information, this paper proposes an improved solution-dynamic multi-head sparse attention mechanism. This mechanism aims to improve the efficiency and effect of long text and large-scale data processing by dynamically selecting key attention heads and sparsely sparse attention weights. It not only significantly reduces computational complexity and memory consumption, but also maintains or improves the model's language expression ability and classification accuracy.

[0090] The pre-trained ERNIE model based on the Transformer architecture was selected as the base model for learning the power grid data classification task. The pre-trained ERNIE model was loaded. Through extensive pre-training on corpora, ERNIE learns the contextual relationships and underlying semantic information between words. This enables the model to convert raw text into a vector representation rich in semantic features. The following steps are involved:

[0091] The numerical data in the short text data from power marketing, which includes multiple indicators, was converted to text using a specific encoding method, and corresponding special tags (such as [CLS] and [SEP]) were added to meet the input sequence structure requirements of the ERNIE model. The text data was segmented and indexed according to the vocabulary table corresponding to the ERNIE pre-trained model. Each word was mapped to a corresponding ID sequence, ultimately generating a dataset that can be directly input into the ERNIE model for training and prediction. This dataset was then fed into the ERNIE model for pre-training.

[0092] During implementation, the pandas library function is called to read the CSV files containing the high-security text data and the low-security text data from step S1 into a DataFrame df. The df shape is [n,m], with each row containing multiple fields. Assume that each row in the file represents a record with multiple fields. The first text_feature_nums fields are taken from a row of data and concatenated into a long string (for example, "label 1, label 2, ..., label 10"). The concatenated text is converted into a string data type using the loaded ERNIE tokenizer. Ultimately, it is converted into a fixed-length input (maximum 512 characters) by the ERNIE tokenizer.

[0093] Next, we perform word segmentation on the text. We use the ERNIE word segmenter to convert the concatenated text into model input. The ERNIE word segmenter converts the text into input_ids, attention_mask, and token_type_ids. During word segmentation, the text is truncated to a maximum length of 512 and special tags ([CLS] and [SEP]) are added.

[0094] input_ids: The shape is [batch_size, 512], that is, each word is converted to a number with a length of 512 (padding is performed according to the maximum length setting).

[0095] attention_mask: shape is [batch_size, 512], indicating which tokens are valid (1) or filled (0).

[0096] token_type_ids: The shape is [batch_size, 512], which is used to distinguish sentence A from sentence B. The squeeze operation is used to remove the batch dimension and convert the tensor shape from [1, 512] to

[512] . That is, input_ids, attention_mask, and token_type_ids are all tensors of shape

[512] , where each value is an integer, representing a word ID or mask value.

[0097] Extract the class label (class_label) and level label (level_label) from each row of data. The class label is stored in the second-to-last column and the level label is stored in the last column. Extract non_text_feature_nums non-text features (such as integers) from each row of data and convert these features into tensors. The shape of non_text_feature is [non_text_feature_nums]. For example, if non_text_feature_nums = 5, the shape of non_text_feature is [5], indicating 5 non-text features of the record. self.tokenized_text is a list of dictionaries, each containing three tensors (shape

[512] ). non_text_feature is a list of tensors, each of which has a shape of [non_text_feature_nums]. class_labels_list and level_labels_list are lists of integers with a shape of [n].

[0098] In general, the present invention performs word segmentation and encoding on the text. The RNIE-specific tokenizer converts it into a sequence of token IDs, namely input_ids[batch_size,lens]. Based on the actual length of the text, an attention_mask and token_type_ids are generated. The attention_mask indicates which positions are valid tokens and which are padding. Token_type_ids are generated to distinguish different sentences. Generally speaking, before entering the model, tensors containing input_ids, attention_mask, and token_type_ids are generated. These tensors represent the word ID, validity mask, and sentence type information, respectively; these tensors together constitute the input to ERNIE.

[0099] As a preferred solution, the pre-training of the ERNIE model specifically includes the following steps:

[0100] After the dataset is input into the ERNIE model, the model's embedding layer maps each word ID into a vector, adds position embedding and sentence embedding, and then passes through multiple Transformer layers to capture dependencies and contextual information between words. The model uses text feature output and inputs the text features into the dynamic multi-head sparse attention mechanism. The dataset includes word IDs, validity masks, and sentence type information.

[0101] Pre-training the ERNIE model specifically includes the following steps:

[0102] The dataset contains tensors of word IDs (input_ids), validity masks (attention_mask), and sentence type information (token_type_ids). These tensors represent word IDs, validity masks, and sentence type information respectively.

[0103] After the dataset enters the ERNIE model, each word (token) is converted into a high-dimensional vector through an embedding layer. This vector, combined with positional and sentence information, is then processed through a multi-layer Transformer encoder to capture the complex dependencies between tokens in the text. Specifically, the ERNIE model's embedding layer maps each token ID into a vector (word embedding), adds positional and sentence embeddings, and then passes through multiple Transformer layers to capture inter-token dependencies and contextual information. Ultimately, the model outputs two important representations: a last_hidden_state containing the representation of each token (e.g., shape [batch_size, seq_length, hidden_size]), and a global sentence representation (pooled_output) obtained by pooling [CLS] tokens (e.g., shape [batch_size, hidden_size]). These high-quality semantic representations, when fused with other non-textual features, can significantly improve the performance of downstream tasks such as classification. The model uses the last_hidden_state output as input to a dynamic multi-head sparse attention mechanism. Among them, batch_size is the number of samples for each training, seq_length is the number of words in each input sample, and hidden_size is the embedding dimension of each word in the ERNIE model.

[0104] A dynamic multi-head sparse attention mechanism is used to strengthen the correlation between features. At the same time, according to the classification task objectives, the number of classification head categories of the model is set to 2, corresponding to the two categories in the classification task. In this way, the model can perform subsequent training and prediction work for the power grid data classification task after initialization. After loading the model, it is moved to an available device (such as a GPU) to accelerate the model training and inference process. In order to optimize the model parameters, the AdamW optimizer is selected to update the model parameters, and a learning rate of 1e-5 is set to ensure that the model can stably converge to a better solution during training.

[0105] Step S3: introduce the dynamic multi-head sparse attention mechanism into the pre-trained ERNIE model to obtain the ERNIE attention mechanism model.

[0106] As a preferred solution, the dynamic multi-head sparse attention mechanism is introduced into the pre-trained ERNIE model to obtain the ERNIE attention mechanism model (ERNIE+DMHA model), which specifically includes the following steps:

[0107] The input data comes from the text features of the ERNIE model. The shape of the text features includes the number of samples per training, the number of words in each input sample, and the embedding dimension of each word in the ERNIE model.

[0108] The data is linearly transformed to generate query vectors, key vectors, and value vectors. The linear transformation maps the input text features into three different spaces, corresponding to the query, key, and value in the attention mechanism. The query vector, key vector, and value vector are generated through three linear layers. The output shape includes the number of samples in each training session, the number of words in each input sample, and the embedding dimension of each word in the ERNIE model.

[0109] Split the generated query vector, key vector, and value vector according to the number of attention heads, thereby reshaping the query vector, key vector, and value vector into a multi-head form to obtain a tensor shape;

[0110] The attention score is calculated by the inner product between the query vector and the key vector. The attention score indicates the importance of each word to other words.

[0111] Calculate the highest confidence value according to the preset sparsity parameter. For each attention score vector, only the value with the highest confidence value is retained, and the remaining positions are set to zero, and then renormalized;

[0112] Dynamically select the most important attention heads through a gating mechanism:

[0113] The input preset is averaged along the sequence dimension to obtain a tensor whose shape includes the number of samples per training and the embedding dimension of each word in the ERNIE model;

[0114] The shape of the averaged tensor, which includes the number of samples per training and the embedding dimension of each word in the ERNIE model, is input into a fully connected layer, where the output dimension of the fully connected layer is set to the number of attention heads;

[0115] Apply the Sigmoid activation function to the output of the linear layer and normalize the output value to the interval [0, 1]. The resulting score tensor has the shape of the number of samples and the number of attention heads for each training. Each element of the score tensor represents the importance score of the corresponding attention head.

[0116] After obtaining the output of the scoring tensor whose shape includes the number of samples and the number of attention heads for each training, the number of selected attention heads is calculated according to the preset selection rate to obtain the number of confidence attention heads. For each sample, the highest confidence operation is used to select the number of attention heads with the highest score from the scoring tensor, and the index of the head is obtained;

[0117] Use the selected index to construct a binary mask for each sample. In each sample, the position of the selected confidence attention head is set to 1, and the position of the unselected head is set to 0; use the binary mask to set the attention score corresponding to the unselected head to zero;

[0118] Finally, the filtered text features are obtained based on the selected header.

[0119] Dynamic Multi-Head Sparse Attention is part of the model, such as Figure 2 As shown in Figure 1, the attention mechanism is specifically designed to process input data. It not only uses multi-head self-attention to capture different features, but also optimizes the computational process through sparsity and dynamic selection of attention heads, improving efficiency while enhancing the generalization ability of the model.

[0120] Dynamic multi-head sparse attention has two main purposes. The first is to sparse the attention calculation. In traditional multi-head attention calculation, each token calculates the attention score for all other tokens, which is computationally intensive. By setting a sparsity ratio (for example, 0.5), only the highest part (for example, the top 64 scores when the sequence length is 128) is retained in each attention score vector, and the remaining positions are set to zero, thereby reducing computational complexity and highlighting key information. The second is dynamic head selection. Not all attention heads contribute equally to the task. The importance of each head is scored through a gating mechanism, and the most important attention heads (for example, 12 out of 16 heads) are dynamically selected according to a set selection ratio (for example, 0.8). This can further improve the model's ability to focus on key information while reducing noise interference.

[0121] The dynamic multi-head sparse attention mechanism specifically includes the following steps:

[0122] In step S31, the input data comes from the text feature last_hidden_state of the ERNIE model, and its shape is [batch_size, seq_length, hidden_size], where batch_size is the number of samples for each training, seq_length is the number of tokens in each input sample, and hidden_size is the embedding dimension of each token in the ERNIE model. This part of the data is the representation of the text obtained after pre-training in the ERNIE model. Each input token corresponds to a vector that captures the semantic information of the token in the context.

[0123] In step S32, the data is transformed into a query vector (Q), a key vector (K), and a value vector (V) through three independent linear transformations (query, key, value); the linear transformation maps the input hidden_states to three different spaces, corresponding to the query, key, and value in the attention mechanism, respectively, and generates the query vector Q, key vector K, and value vector V through three linear layers, and the output is [batch_size, seq_length, hidden_size].

[0124] In step S33, perform the head splitting operation and calculate the attention score. Split the generated query vector Q, key vector K, and value vector V into the number of attention heads (num_heads). Each head processes a portion of the input information to capture different features in different subspaces. Reshape the query vector Q, key vector K, and value vector V into a multi-head form, resulting in a tensor of shape [batch_size, num_heads, seq_length, head_dim].

[0125] Step S34, calculate the attention score by the inner product between the query vector Q and the key vector K. The attention score represents the importance of each token to other tokens. To improve computational efficiency, the sparsification operation retains the largest part of the attention score and resets the unimportant attention weights to 0.

[0126] Step S35, calculate the topk (parameters of the sparsification operation) values ​​according to the set sparsity_level (sparseness), and for each attention score vector, only retain the largest topk values, set the remaining positions to 0, and then renormalize.

[0127] Step S36: Dynamically select the most important attention heads through a gating mechanism. Each head has an importance score, and the model selects the most important heads for calculation. Based on the selected heads, a mask is generated and applied to the attention score, retaining only the selected heads for calculation. This specifically includes the following steps:

[0128] Step S361, calculate the mean of the input x along the sequence dimension (the first dimension) to obtain a tensor with a shape of [batch_size, hidden_size]; here, each sample is represented by a vector of hidden_size dimensions to represent the overall information.

[0129] In step S362, the tensor [batch_size, hidden_size] obtained by averaging is input into a fully connected layer (linear layer), and the output dimension of this layer is set to the number of attention heads, i.e., num_heads.

[0130] In step S363, a Sigmoid activation function is applied to the linear layer output to normalize the output value to the interval [0, 1], and a score tensor of shape [batch_size, num_heads] is obtained, where each element of the score tensor represents the importance score of the corresponding attention head.

[0131] Step S364, after obtaining the output of [batch_size, num_heads], calculate the number of selections according to the size of select_ratio (a floating number between 0 and 1) to obtain the number of the most important attention heads to be retained (topk_heads). For each sample, use the topk operation to select the topk_heads attention heads with the highest scores from the score tensor of [batch_size, num_heads], and obtain the indices of these heads.

[0132] In step S365, a binary mask is constructed for each sample using the selected index. In each sample, the positions of the topk_heads selected heads are set to 1, and the positions of the unselected heads are set to 0. This mask will be used in subsequent steps to set the attention scores corresponding to the unselected heads to zero, thereby ensuring that subsequent calculations focus only on the most important attention heads. After the gating mechanism, a weighted summation and output are performed. At this point, after the previous multi-head splitting and sparse processing, the shape of the attention score tensor is [batch_size, num_heads, seq_length, num_token], while the shape of the value tensor V is [batch_size, num_heads, seq_length, head_dim]. The attention scores corresponding to the unselected heads are set to zero using a binary mask.

[0133] In this way, in the subsequent matrix multiplication, only the selected heads will contribute valid information. For each attention head, perform matrix multiplication:

[0134] attn_output_head=attn_scores_masked×V;

[0135] attn_output_head is the attention score tensor after sparse attention mask processing, and attn_scores_masked is the weighted vector result of the final output of each attention head.

[0136] The output shape of each head is [batch_size, num_heads, seq_length, head_dim] (where for non-selected heads, the result is also zero due to the zero attention score). After obtaining the output of each head, each head is merged and the output of multiple heads is transposed first, so that the tensor shape is converted from [batch_size, num_heads, seq_length, head_dim] to [batch_size, seq_length, num_heads, head_dim]. The last two dimensions are merged, and each of the num_heads heads outputs head_dim dimensions, which are concatenated into num_heads×head_dim dimensions in total, thus restoring a single representation. The merged tensor has the same shape as the hidden state output of the original ERNIE, but the key information has been highlighted through sparsification and head selection. Finally, after a linear mapping, the linear layer further integrates the information of each head and may adjust the dimensions, but in the present invention, the output shape still maintains the shape of [batch_size, seq_length, hidden_size].

[0137] Step S366: Finally, the filtered text features are obtained based on the selected header.

[0138] This design not only reduces unnecessary information interference during computation, but also leverages a gating mechanism to dynamically select the most important attention heads, ensuring the model's focus on key features in subsequent tasks. The dynamic multi-head sparse attention mechanism combines the advantages of multi-head attention, sparsification, and dynamic head selection to effectively reduce computational overhead, improving model training efficiency and generalization. By reducing unnecessary computation and memory consumption, this mechanism maintains high efficiency when processing large amounts of data while maintaining model flexibility and adaptability.

[0139] In step S4, the central server sends the ERNIE attention mechanism model to each client, so that each client uses local data to train the ERNIE attention mechanism model to obtain a local model.

[0140] Federated learning is a distributed machine learning approach in which training data is distributed across multiple clients, and model training and updates occur locally on the client rather than on a centralized server. The centralized server collects model updates (typically weight updates) from each client and aggregates them to improve the global model. In this process, each client uses only local data for training, ensuring data privacy and distributed computing.

[0141] The central server sends the pre-trained ERNIE+DMHA model to each client; each client uses local data to train the ERNIE+DMHA model to obtain a trained local model.

[0142] like Figure 3 As shown, the central server sends the current global model to each client; the current global model after the t-1th round of aggregation is distributed to each client as the local model of each client for the tth round of local training. Let the local model of the kth (k=1,2,...,K) local client be

[0143] Client k uses local data to train the model; after each client obtains the model from the central server, it uses the cross entropy loss function and stochastic gradient descent method for training. Denotes the loss function used by client k for training, then the update of the local model is shown in formula (1), where η represents the learning rate during model training:

[0144]

[0145] In step S5, each client uploads the trained local model parameters to the central server. The central server performs weighted averaging on the local model updates of each client based on the amount of data and generates an initial global model.

[0146] Each client sends the trained model to the central server; after training, client k sends its local model to the central server. Send to the central server. The central server aggregates the local model and is responsible for sending the model Aggregation is performed using weighted average, and the amount of data is generally used to represent the weight, as shown below:

[0147]

[0148] in, is the global model obtained by round t aggregation; K is the number of local models; represents the kth local model trained in the tth round; |D| represents the total number of samples in each client dataset, |D k | represents the number of samples of local data from the kth client.

[0149] In addition, for the high-security part of the data, noise is added during the global model training process to protect data privacy. Specifically, we add Gaussian noise to the parameter aggregation step of the federated averaging algorithm. Assume that the parameters received by the central server in the tth round of aggregation are The parameters after adding noise are:

[0150]

[0151] Among them, here As mentioned before Similar, but more formal notation; used to identify the training results uploaded by each client Indicates that the added noise term follows a normal distribution with a mean of 0 and a variance of σ 2 This noise is used for privacy protection to prevent sensitive information of training data from being leaked through model parameters. i is the data volume of client i, N is the sum of the data volume of all clients, K is the number of clients, and for the high-security level part of the data, noise is added during the model training process to protect data privacy.

[0152] This implementation introduces differential privacy technology, classifying raw power data into low-security and high-security categories based on a negative list mechanism. For data marked as high-security under the negative list, differential privacy is incorporated into the model's parameter transfer process, enhancing data privacy protection by adding noise. This approach significantly improves data privacy while maintaining model training effectiveness, ensuring that high-security data is not leaked during the joint training process.

[0153] In each round of federated learning, noise is applied to each client's gradient update. The noise intensity is regulated by the privacy budget (ε). A smaller budget value improves privacy protection, but may also limit model accuracy. By properly allocating the privacy budget, this paper achieves an optimal balance between privacy protection and model performance.

[0154] The goal of federated learning is to learn a global model w from each client with the help of a central server without exchanging data, and to aggregate parameters. Therefore, the overall optimization goal of the model is as follows:

[0155]

[0156] Among them, L k(w) represents the classification loss when client k trains the model, and its formula is as follows:

[0157]

[0158] Among them, E represents the expectation for the data set D k The expectation of all samples (x,y) in (x,y)~D k Represents data D k A data sample (x,y) in , where x represents the sample in the data and y represents the label in the data.

[0159] In step S6, the central server saves the initial global model and sends it back to each client, so that each client can train the received initial global model; when the accuracy of the trained initial global model converges and reaches a preset value, a trained global model is obtained.

[0160] The core idea of ​​federated learning is:

[0161] (1) Local training: Each client performs training locally and updates its own model.

[0162] (2) Global aggregation: After training is completed, each client sends its model update (not the original data) to the server (global model). The server aggregates all models based on each client's model update to obtain a global model.

[0163] The dataset is divided into local data and validation data. Each client (i.e., different data shards) has its own local training data, which is loaded from different files (for example, fed1.xlsx, fed2.xlsx, fed3.xlsx), and each file represents the data of one client. The validation data is a shared dataset (for example, validation_data.xlsx) that all clients can use to verify the performance of the global model. Each client loads data from its corresponding data file and performs preprocessing. Preprocessing includes: concatenating text features (concatenating multiple text fields into one long text); encoding the text using the word segmenter of the pre-trained model and converting it into a format acceptable to the model (such as token ID, attention mask); extracting non-text features (such as integer features) and converting them into tensors. The data of each client is divided into batches and then passed to the local model for training.

[0164] Each client uses its local dataset for training. The client's training process is independent and depends only on the client's data. The data is preprocessed: the text data is concatenated into fields and then encoded using a tokenizer (converted into input_ids, attention_mask, etc.). Non-text features are processed into numerical values. These processed data become inputs and are passed to the local model for training. The data input for each client includes: Text features: including multiple fields, which are concatenated and processed by the model's tokenizer and converted into a fixed-length token sequence (such as input_ids, attention_mask, token_type_ids). Non-text features: including integer features or other features, such as user information, device information, etc., with a shape of [batch_size, non_text_feature_nums]. Label data: such as category labels or level labels for classification tasks.

[0165] The local model is trained on the client's data, calculating the loss and accuracy for each batch. Key training steps include: the client calculates the loss and gradient using its local data; updates the model using an optimization algorithm (such as AdamW); each client does not interact with other clients' data during training, only calculating and updating the model weights locally; and loss calculation: The client calculates the loss for each batch and performs backpropagation to update the model parameters.

[0166] Each client updates its own model parameters during local training. These model parameters are weighted and aggregated based on the amount of data each client has. When all clients complete training, they upload their updated model parameters (e.g., weights and biases). The server aggregates these model updates using a weighted average method to generate a new global model. This model is then returned to each client for the next round of training.

[0167] The updating of the global model is done through the following process:

[0168] Client upload update: Each client uploads its trained model parameters (updated weights) to the server.

[0169] Weighted aggregation: The server performs a weighted average of each client’s model updates based on the amount of data provided to generate a new global model.

[0170] Global model update: After the global model is updated, it is saved and sent back to each client for the next round of training.

[0171] This weighted averaging approach ensures that clients with larger data volumes contribute more to the global model, thereby improving the accuracy of the global model. The federated learning mechanism ensures data privacy, as clients collaborate only through model updates without sharing local data.

[0172] Step S7: Acquire and input the processed actual power marketing short text data into the trained global model to obtain the probability that the actual power marketing short text data belongs to a high security level and a low security level.

[0173] Step S8: Determine the security level of the power marketing short text data sample based on the comparison result of the probability of belonging to the high security level and the probability of the low security level with the probability threshold.

[0174] The preprocessed power marketing short text data is fed into the trained global model, which outputs a probability distribution for each sample belonging to one of two security levels. Based on the probability distribution output by the model, a probability threshold is set to determine the final classification result. This paper chooses 0.5 as the threshold. That is, if the probability of a sample belonging to high security is greater than 0.5, it is classified as high security; otherwise, it is classified as low security.

[0175] In order to evaluate the accuracy of the classification results, the following formula was used to evaluate the correctness.

[0176]

[0177] Among them, Accuracy represents the evaluation accuracy, TP refers to samples correctly predicted as positive, TN refers to samples correctly predicted as negative, FP refers to samples incorrectly predicted as positive, and FN refers to samples incorrectly predicted as negative.

[0178] Performance under normal training conditions, e.g. Figure 4 Figure 2 shows the performance of the ERNIE and ERNIE+DMHA models on three different clients (Client 1, Client 2, and Client 3) under normal training conditions. As can be seen, the ERNIE+DMHA model outperforms the basic ERNIE model on all clients, with the most significant performance on Client 3. This demonstrates that the DMHA (Dynamic Multi-Head Attention) mechanism effectively improves the model's performance, making it more accurate and reliable in the power grid text classification task.

[0179] Performance in federated differential privacy environments, such as Figure 5Figure 2 shows the performance of the ERNIE and ERNIE+DMHA models on three clients (Client 1, Client 2, and Client 3) and a central server (Center) in a federated differential privacy environment. Compared to normal training conditions, overall performance improved in the federated differential privacy environment, particularly on the central server, where the ERNIE+DMHA model achieved a 4.5% performance improvement. This demonstrates that while protecting data privacy, the combination of federated learning and the DMHA mechanism not only maintains performance improvements but actually further optimizes them. This is crucial for balancing data security and model effectiveness in practical applications.

[0180] Example 2:

[0181] like Figure 6 As shown, based on the same inventive concept as Example 1, this embodiment provides a power marketing short text grading system based on federated learning, applying the method described above, including:

[0182] A data collection and processing module is used to collect and process historical power marketing short text data samples, and classify the processed historical power marketing short text data samples into a high security level and a low security level; wherein the historical power marketing short text data samples include several indicators;

[0183] The ERNIE model pre-training module is used to perform word segmentation and encoding operations on power marketing short text data samples, and input them into the ERNIE model for pre-training.

[0184] The multi-head sparse attention module is used to introduce the dynamic multi-head sparse attention mechanism into the pre-trained ERNIE model to obtain the ERNIE attention mechanism model;

[0185] The central server is configured to send the ERNIE attention mechanism model to each client, so that each client trains the ERNIE attention mechanism model using local data to obtain a local model; save the initial global model and send it back to each client, so that each client trains the received initial global model; when the accuracy of the trained initial global model converges and reaches a preset value, a trained global model is obtained; obtain and input the processed actual power marketing short text data into the trained global model to obtain the probability that the actual power marketing short text data belongs to a high security level and a low security level; determine the security level of the power marketing short text data sample based on the comparison result of the probability of belonging to the high security level and the low security level with the probability threshold;

[0186] The client uploads the trained local model parameters to the central server. The central server performs a weighted average of the local model updates of each client based on the amount of data to generate an initial global model.

[0187] Preferably, the processing of historical power marketing short text data samples includes:

[0188] The historical power marketing short text data samples with multiple indicators are processed including missing values, outliers and duplicate values.

[0189] Preferably, the segmentation and encoding of the power marketing short text data samples and inputting them into the ERNIE model, and pre-training the ERNIE model specifically include the following steps:

[0190] Convert the numerical data in the historical power marketing short text data sample with multiple indicators into text representation according to a preset encoding method and add corresponding tags;

[0191] The text data in the historical power marketing short text data samples were segmented and indexed according to the vocabulary corresponding to the ERNIE pre-trained model. Each word was mapped to a corresponding ID sequence, and finally a dataset was generated that can be directly input into the ERNIE model for training and prediction. The dataset was input into the ERNIE model for pre-training.

[0192] Preferably, the word segmentation operation includes:

[0193] The historical power marketing short text data samples are divided into three parts, each corresponding to a masking task, where the three parts are word-level data, phrase-level data and entity-level data.

[0194] Preferably, the method specifically includes the following steps:

[0195] After the dataset is input into the ERNIE model, the model's embedding layer maps each word ID into a vector, adds position embedding and sentence embedding, and then passes through multiple Transformer layers to capture dependencies and contextual information between words. The model uses text feature output and inputs the text features into the dynamic multi-head sparse attention mechanism. The dataset includes word IDs, validity masks, and sentence type information.

[0196] Preferably, the method of introducing a dynamic multi-head sparse attention mechanism into the pre-trained ERNIE model to obtain an ERNIE attention mechanism model specifically includes the following steps:

[0197] The input data comes from the text features of the ERNIE model. The shape of the text features includes the number of samples per training, the number of words in each input sample, and the embedding dimension of each word in the ERNIE model.

[0198] The data is linearly transformed to generate query vectors, key vectors, and value vectors. The linear transformation maps the input text features into three different spaces, corresponding to the query, key, and value in the attention mechanism. The query vector, key vector, and value vector are generated through three linear layers. The output shape includes the number of samples in each training session, the number of words in each input sample, and the embedding dimension of each word in the ERNIE model.

[0199] Split the generated query vector, key vector, and value vector according to the number of attention heads, thereby reshaping the query vector, key vector, and value vector into a multi-head form to obtain a tensor shape;

[0200] The attention score is calculated by the inner product between the query vector and the key vector. The attention score indicates the importance of each word to other words.

[0201] Calculate the highest confidence value according to the preset sparsity parameter. For each attention score vector, only the value with the highest confidence value is retained, and the remaining positions are set to zero, and then renormalized;

[0202] Dynamically select the most important attention heads through a gating mechanism:

[0203] The input preset is averaged along the sequence dimension to obtain a tensor whose shape includes the number of samples per training and the embedding dimension of each word in the ERNIE model;

[0204] The shape of the averaged tensor, which includes the number of samples per training and the embedding dimension of each word in the ERNIE model, is input into a fully connected layer, where the output dimension of the fully connected layer is set to the number of attention heads;

[0205] Apply the Sigmoid activation function to the output of the linear layer and normalize the output value to the interval [0, 1]. The resulting score tensor has the shape of the number of samples and the number of attention heads for each training. Each element of the score tensor represents the importance score of the corresponding attention head.

[0206] After obtaining the output of the scoring tensor whose shape includes the number of samples and the number of attention heads for each training, the number of selected attention heads is calculated according to the preset selection rate to obtain the number of confidence attention heads. For each sample, the highest confidence operation is used to select the number of attention heads with the highest score from the scoring tensor, and the index of the head is obtained;

[0207] Use the selected index to construct a binary mask for each sample. In each sample, the position of the selected confidence attention head is set to 1, and the position of the unselected head is set to 0; use the binary mask to set the attention score corresponding to the unselected head to zero;

[0208] Finally, the filtered text features are obtained based on the selected header.

[0209] Preferably, the calculation formula for performing weighted averaging on the local model update of each client according to the amount of data of the client is:

[0210]

[0211] in, is the global model obtained by round t aggregation; K is the number of local models; represents the kth local model trained in the tth round; |D| represents the total number of samples in each client dataset, |D k | represents the number of samples of local data from the kth client.

[0212] Example 3:

[0213] Based on the same inventive concept as Example 1, this embodiment provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the power marketing short text grading method based on federated learning.

[0214] Example 4:

[0215] Based on the same inventive concept as that of Example 1, this embodiment provides a processor, which is used to run a program, wherein the program executes the power marketing short text classification method based on federated learning when running.

[0216] Those skilled in the art will appreciate that the units of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition of each example has been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0217] In the embodiments provided in the present application, it should be understood that the division of units is merely a logical function division, and there may be other division methods in actual implementation, for example, multiple units can be combined into one unit, one unit can be split into multiple units, or some features can be ignored, etc.

[0218] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0219] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-0nly Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc., various media that can store program code.

[0220] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and description of the present invention.

Claims

1. A method for grading short texts in power marketing based on federated learning, characterized by: The following steps are involved: Step S1: collecting and processing historical power marketing short text data samples, and classifying the processed historical power marketing short text data samples into a high security level and a low security level; wherein the historical power marketing short text data samples include several indicators; Step S2: performing word segmentation and encoding operations on the power marketing short text data sample, and inputting the data into the ERNIE model to pre-train the ERNIE model; Step S3, introducing the dynamic multi-head sparse attention mechanism into the pre-trained ERNIE model to obtain the ERNIE attention mechanism model; Step S4: The central server sends the ERNIE attention mechanism model to each client, so that each client uses local data to train the ERNIE attention mechanism model to obtain a local model. In step S5, each client uploads the trained local model parameters to the central server. The central server performs a weighted average of the local model updates of each client based on the amount of data, and generates an initial global model. In step S6, the central server saves the initial global model and sends it back to each client, so that each client can train the received initial global model. When the accuracy of the trained initial global model converges and reaches a preset value, a trained global model is obtained. Step S7: Obtain and input the processed actual power marketing short text data into the trained global model to obtain the probability that the actual power marketing short text data belongs to a high security level and a low security level; Step S8: Determine the security level of the power marketing short text data sample based on the comparison result of the probability of belonging to the high security level and the probability of the low security level with the probability threshold.

2. The method for grading short texts in power marketing based on federated learning according to claim 1, characterized in that: The processing of historical power marketing short text data samples includes: The historical power marketing short text data samples with multiple indicators are processed including missing values, outliers and duplicate values.

3. The method for grading short texts in power marketing based on federated learning according to claim 1, characterized in that: The power marketing short text data samples are segmented and encoded and input into the ERNIE model, and the ERNIE model is pre-trained, specifically including the following steps: Convert the numerical data in the historical power marketing short text data sample with multiple indicators into text representation according to a preset encoding method and add corresponding tags; The text data in the historical power marketing short text data samples were segmented and indexed according to the vocabulary corresponding to the ERNIE pre-trained model. Each word was mapped to a corresponding ID sequence, and finally a dataset was generated that can be directly input into the ERNIE model for training and prediction. The dataset was input into the ERNIE model for pre-training.

4. A method for grading short texts in power marketing based on federated learning according to any one of claims 1 to 3, characterized in that: The word segmentation operation includes: The historical power marketing short text data samples are divided into three parts, each corresponding to a masking task, where the three parts are word-level data, phrase-level data and entity-level data.

5. The method for grading short texts in power marketing based on federated learning according to claim 3 is characterized in that: The ERNIE model is pre-trained. The following steps are involved: After the dataset is input into the ERNIE model, the model's embedding layer maps each word ID into a vector, adds position embedding and sentence embedding, and then passes through multiple Transformer layers to capture dependencies and contextual information between words. The model uses text feature output and inputs the text features into the dynamic multi-head sparse attention mechanism. The dataset includes word IDs, validity masks, and sentence type information.

6. The method for grading short texts in power marketing based on federated learning according to claim 5, characterized in that: The dynamic multi-head sparse attention mechanism is introduced into the pre-trained ERNIE model to obtain the ERNIE attention mechanism model, which specifically includes the following steps: The input data comes from the text features of the ERNIE model. The shape of the text features includes the number of samples per training, the number of words in each input sample, and the embedding dimension of each word in the ERNIE model. The data is linearly transformed to generate query vectors, key vectors, and value vectors. The linear transformation maps the input text features into three different spaces, corresponding to the query, key, and value in the attention mechanism. The query vector, key vector, and value vector are generated through three linear layers. The output shape includes the number of samples in each training session, the number of words in each input sample, and the embedding dimension of each word in the ERNIE model. Split the generated query vector, key vector, and value vector according to the number of attention heads, thereby reshaping the query vector, key vector, and value vector into a multi-head form to obtain a tensor shape; The attention score is calculated by the inner product between the query vector and the key vector. The attention score indicates the importance of each word to other words. Calculate the highest confidence value according to the preset sparsity parameter. For each attention score vector, only the value with the highest confidence value is retained, and the remaining positions are set to zero, and then renormalized; Dynamically select the most important attention heads through a gating mechanism: The input preset is averaged along the sequence dimension to obtain a tensor whose shape includes the number of samples per training and the embedding dimension of each word in the ERNIE model; The shape of the averaged tensor, which includes the number of samples per training and the embedding dimension of each word in the ERNIE model, is input into a fully connected layer, where the output dimension of the fully connected layer is set to the number of attention heads; Apply the Sigmoid activation function to the output of the linear layer and normalize the output value to the interval [0, 1]. The resulting score tensor has the shape of the number of samples and the number of attention heads for each training. Each element of the score tensor represents the importance score of the corresponding attention head. After obtaining the output of the scoring tensor whose shape includes the number of samples and the number of attention heads for each training, the number of selected attention heads is calculated according to the preset selection rate to obtain the number of confidence attention heads. For each sample, the highest confidence operation is used to select the number of attention heads with the highest score from the scoring tensor, and the index of the head is obtained; Use the selected index to construct a binary mask for each sample. In each sample, the position of the selected confidence attention head is set to 1, and the position of the unselected head is set to 0; use the binary mask to set the attention score corresponding to the unselected head to zero; Finally, the filtered text features are obtained based on the selected header.

7. The method for grading short texts in power marketing based on federated learning according to claim 1, characterized in that: The calculation formula for performing weighted average of local model updates based on the amount of data of each client is: in, is the global model obtained by round t aggregation; K is the number of local models; represents the kth local model trained in the tth round; |D| represents the total number of samples in each client dataset, |D k | represents the number of samples of local data from the kth client.

8. A power marketing short text grading system based on federated learning, characterized by: Applying the method according to any one of claims 1 to 7, comprising: A data collection and processing module is used to collect and process historical power marketing short text data samples, and classify the processed historical power marketing short text data samples into a high security level and a low security level; wherein the historical power marketing short text data samples include several indicators; The ERNIE model pre-training module is used to perform word segmentation and encoding operations on power marketing short text data samples, and input them into the ERNIE model for pre-training. The multi-head sparse attention module is used to introduce the dynamic multi-head sparse attention mechanism into the pre-trained ERNIE model to obtain the ERNIE attention mechanism model; The central server is configured to send the ERNIE attention mechanism model to each client, so that each client trains the ERNIE attention mechanism model using local data to obtain a local model; save the initial global model and send it back to each client, so that each client trains the received initial global model; when the accuracy of the trained initial global model converges and reaches a preset value, a trained global model is obtained; obtain and input the processed actual power marketing short text data into the trained global model to obtain the probability that the actual power marketing short text data belongs to a high security level and a low security level; determine the security level of the power marketing short text data sample based on the comparison result of the probability of belonging to the high security level and the low security level with the probability threshold; The client uploads the trained local model parameters to the central server. The central server performs a weighted average of the local model updates of each client based on the amount of data to generate an initial global model.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the power marketing short text grading method based on federated learning according to any one of claims 1 to 7.

10. A processor, characterized in that: The processor is configured to run a program, wherein the program, when running, executes the method for grading short texts in power marketing based on federated learning as described in any one of claims 1 to 7.

Images