Electricity utilization industry classification method and system based on multi-modal data feature extraction

Abstract

The invention discloses a power utilization industry classification method and system based on multi-modal data feature extraction. The method comprises the following steps: collecting and preprocessing multi-modal data of different power utilization industries; performing format conversion on the preprocessed multi-modal data to obtain a unified input vector; performing fine adjustment on the BERT pre-training model by using the electrical load curves of different industries, inputting the unified input vector into the fine-adjusted BERT pre-training model, and extracting an electrical load feature vector; and on the basis of the extracted electrical load feature vector, parameter updating is performed on the fine-tuned BERT pre-training model in combination with a self-supervised learning mechanism, and the BERT pre-training model after parameter updating is adopted to perform power utilization industry classification. According to the method, complex multi-modal data can be processed and analyzed, the industry feature extraction and classification precision is improved, and the power utilization feature analysis and management level of different specific industries can be remarkably improved in practical application.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal data processing, and relates to a method and system for classifying electricity consumption industries based on multimodal data feature extraction. Background Art

[0002] As the degree of electrification continues to increase, the connection between electricity consumption and industry development has become closer. A detailed analysis of the growth of electricity consumption in various industries is an important basis for scientifically and rationally evaluating market trends and guiding operational decisions. By in-depth research on the patterns of user electricity consumption, it is possible to more accurately evaluate the operating conditions of enterprises and predict the development trends of the industry. Electricity is not only the cornerstone of enterprise operations, but also the key to people's livelihood security. It plays an indispensable role in promoting high-quality social development.

[0003] At present, the rapidly developing marketing business and the complex international economic situation have brought challenges to the electricity management of enterprises. Affected by multiple internal and external factors such as the economic environment and industrial policies, many enterprises have begun to adjust their industrial structure and even turn to flexible operating models such as factory leasing. This change has caused the actual situation of enterprises to be inconsistent with their industry classification in the archives of power grid enterprises, which in turn affects the statistical accuracy of the electricity data of the whole society, the effective implementation of electricity price policies, and the compliance assessment of electricity consumption behavior.

[0004] The data diversity and complexity, data island phenomenon, dynamic changes and real-time, and data labeling difficulties faced in the classification of industry electricity consumption make electricity consumption data highly diverse and complex. Traditional single-modal data processing methods are difficult to fully capture and understand these complex characteristics. In addition, data from different enterprises and industries are usually scattered, lacking a unified data integration and analysis platform, resulting in low data utilization, difficulty in information sharing, and inability to form a global analysis of industry electricity consumption characteristics. The electricity consumption behavior and characteristics of enterprises change frequently over time, such as seasonal production adjustments and the introduction of new technologies, which makes electricity consumption characteristics significantly dynamic. Traditional static data analysis methods cannot reflect these changes in a timely manner, affecting the accuracy of classification and decision-making. At the same time, traditional machine learning methods rely on a large amount of labeled data for training, but obtaining high-quality labeled data requires a lot of manpower and time costs, and the labeling standards of different data sources may be inconsistent, further increasing the difficulty of data labeling.

[0005] In the classification of industrial electricity consumption, traditional machine learning methods such as support vector machine (SVM), random forest (RandomForest) and k-nearest neighbor (k-NN) can achieve classification tasks to a certain extent, but have many shortcomings when dealing with complex multimodal data. These methods usually rely on single modal data for classification, ignoring other important modal data, such as environmental data, policy data and production habit data, and cannot fully utilize the advantages of multimodal data, resulting in inaccurate classification results. In addition, feature extraction often relies on manual design and simple statistical methods, which is not only time-consuming and labor-intensive, but also difficult to capture deep information and complex patterns in the data, especially when facing high-dimensional and nonlinear data. The performance is poor. Actual data often contains noise and missing values, and traditional machine learning methods are not robust enough to deal with these problems and are easily affected by data quality, resulting in unstable and unreliable classification results.

[0006] In addition, as the business environment and industry policies of enterprises change, the electricity consumption behavior and characteristics of enterprises are also constantly changing. Traditional methods lack flexibility and adaptability in the face of such dynamic changes, and require frequent model retraining and adjustment, which not only increases time and computing costs, but may also cause delays in classification results. At the same time, traditional machine learning methods have technical bottlenecks in multimodal data fusion, making it difficult to effectively integrate data from different modalities, resulting in the inability of classification models to fully utilize all available information, affecting classification accuracy and generalization capabilities. Summary of the invention

[0007] In order to address the deficiencies in the prior art, the present invention provides an electricity consumption industry classification method and system based on multimodal data feature extraction. The BERT pre-training model is adopted. By fusing various types of data and combining deep learning and self-supervised learning techniques, effective feature extraction and industry classification of multimodal data can be achieved, the accuracy and robustness of classification can be improved, and accurate analysis and industry classification of enterprise electricity consumption behavior can be achieved. This overcomes many limitations of traditional methods and provides new solutions for enterprise electricity consumption management.

[0008] The present invention adopts the following technical solution.

[0009] A first aspect of the present invention provides a method for classifying electricity consumption industries based on multimodal data feature extraction, comprising:

[0010] Collect multimodal data from different electricity-consuming industries and pre-process the collected multimodal data;

[0011] Convert the preprocessed multimodal data into a unified input vector.

[0012] The BERT pre-trained model is fine-tuned using the power load curves of different industries, and the unified input vector is input into the fine-tuned BERT pre-trained model to extract the power load feature vector;

[0013] Based on the extracted power load feature vector, the parameters of the fine-tuned BERT pre-trained model are updated in combination with the self-supervised learning mechanism, and the BERT pre-trained model with updated parameters is used to classify the power consumption industry.

[0014] Preferably, the electricity-consuming industries include cement industry, steel industry, chemical industry, new energy lithium battery industry, and photovoltaic industry;

[0015] The multimodal data includes policy data, environmental data, electrical quantity data and industry production habit data.

[0016] Preferably, the preprocessing of the collected multimodal data includes data cleaning, data standardization and missing value filling.

[0017] Preferably, the format conversion of the preprocessed multimodal data specifically includes:

[0018] 1) BERT’s text embedding layer converts the preprocessed policy data, environmental data, and industry production habit data into the following embedding vector representation:

[0019] P embed =BERT embed (P),E embed =BERT embed (E),H embed =BERT embed (H)

[0020] Among them, P embed is the embedding vector of the preprocessed policy data P, E embed is the embedding vector of the preprocessed environmental data E, H embed is the embedding vector of the preprocessed industry production habit data H;

[0021] 2) The preprocessed electrical quantity data is standardized according to the following numerical characteristics:

[0022]

[0023] Among them, μ Q and σ Q are the mean and standard deviation of the preprocessed electrical quantity data, Q is the preprocessed electrical quantity data, and Q norm It is the standardized electrical quantity data.

[0024] Preferably, the method of fine-tuning the BERT pre-training model using the power load curves of different industries includes:

[0025] The user's electricity load curve is classified according to industry, and labels are set to build a data set. The BERT model is supervised by cross entropy loss function, and the Adam optimizer is used to update the model parameters.

[0026] Preferably, the Adam optimizer performs model parameter θ t The update formula is as follows:

[0027] m t =β1m t-1 +(1-β1)g t

[0028]

[0029]

[0030] Among them, t is the time step of the current training; m t is the first-order moment estimate of the gradient, representing momentum; β1 is the exponential decay rate of the first-order moment estimate; g t is the gradient of the current time step; v t , v t-1 is the second-order moment estimate of the gradient of the previous time step t-1 at the current time step t, which represents the moving average of the square of the gradient; β2 is the exponential decay rate of the second-order moment estimate; is the bias-corrected momentum estimate; is the moving average of the squared gradient after deviation correction; α is the learning rate; ε is a constant; θ t ,θ t-1 is the model parameter of the current time step t and the previous time step t-1.

[0031] Preferably, the cross entropy loss function is:

[0032]

[0033] Where N is the number of samples, C is the number of categories, and y ic is the true label of sample i, is the probability predicted by the model.

[0034] Preferably, the unified input vector is input into the fine-tuned BERT pre-trained model to extract the power load feature vector:

[0035] z k =BERT(X k )

[0036] Among them, Xk is the feature of the kth unified input vector X;

[0037] The unified input vector X is:

[0038] X={P embed ,E embed ,H embed ,Q norm}

[0039] Among them, P embed is the embedding vector of the preprocessed policy data P, E embed is the embedding vector of the preprocessed environmental data E, H embed is the embedding vector Q of the preprocessed industry production habit data H norm It is the standardized electrical quantity data.

[0040] Preferably, the method of updating the parameters of the fine-tuned BERT pre-trained model based on the extracted power load feature vector in combination with a self-supervised learning mechanism includes:

[0041] The extracted power load feature vectors are used to construct positive and negative sample pairs, where the positive sample pairs are power load feature vectors from the same industry, and the negative sample pairs are power load feature vectors from different industries;

[0042] Based on the constructed positive and negative samples, the following loss function is used for self-supervised learning to update the parameters of the fine-tuned BERT pre-trained model:

[0043]

[0044] Among them, i, i+, ij represent the index of the current sample, the index of its positive sample and the index of its negative sample j respectively, z i ,z i+ ,z i-j is the corresponding feature representation;

[0045] sim(,) is the similarity measurement function; N is the number of samples in the current batch; λ is the global weight parameter; is the temperature parameter; K is the number of negative samples of the current sample i;

[0046] λ j is the dynamic weight parameter; γ j Adaptive temperature parameter for each negative sample.

[0047] A second aspect of the present invention provides an electricity industry classification system based on multimodal data feature extraction, comprising:

[0048] The data collection and processing module is used to collect multimodal data from different electricity-consuming industries and pre-process the collected multimodal data;

[0049] The data conversion module is used to convert the format of the preprocessed multimodal data to obtain a unified input vector;

[0050] The feature extraction module is used to fine-tune the BERT pre-trained model using the power load curves of different industries, and input the unified input vector into the fine-tuned BERT pre-trained model to extract the power load feature vector;

[0051] The industry classification module is used to update the parameters of the fine-tuned BERT pre-trained model based on the power load feature vector and the self-supervised learning mechanism, and use the BERT pre-trained model with updated parameters to classify the power consumption industry.

[0052] A terminal comprises a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to execute the steps of the method.

[0053] A computer-readable storage medium stores a computer program, which implements the steps of the method when executed by a processor.

[0054] Compared with the prior art, the beneficial effects of the present invention include at least:

[0055] 1. Multimodal data fusion: This invention integrates multimodal data such as policy data, environmental data, electrical quantity data, industry production habit data, etc. from traditional industries such as cement, steel, and chemicals, and emerging industries such as new energy lithium batteries and photovoltaics, to achieve effective fusion of complex and diverse data sources, to perform feature extraction and data dimensionality reduction, and to construct high-quality multimodal data feature representation. This multimodal data fusion method improves the comprehensiveness and accuracy of feature extraction and overcomes the limitations of traditional single-modal data processing methods.

[0056] 2. Deep feature extraction: The BERT pre-trained model is used for feature extraction. Through the multi-layer Transformer encoder and self-attention mechanism, the deep features of the data can be automatically learned. This deep feature extraction method based on a multi-layer neural network structure can better capture the complex patterns and associations in the data, realize end-to-end multimodal feature autonomous learning, effectively improve the accuracy and robustness of feature extraction, and improve the richness and accuracy of feature representation, thereby improving the accuracy and generalization ability of classification.

[0057] 3. Model fine-tuning: Based on the electricity consumption behavior of specific users in a group, we studied and constructed a visualized load curve for electricity consumption, and analyzed the electricity consumption characteristics of different industries. Through multimodal data fusion embedding representation, we can accurately determine the electricity consumption patterns and characteristics of specific industries, and improve the accuracy and reliability of electricity consumption characteristic analysis;

[0058] 4. Self-supervised learning mechanism: The specific industry classification and pseudo-label establishment technology are studied, and the self-supervised mechanism is used for industry classification. By introducing the self-supervised mechanism of contrastive learning, there is no need for a large amount of labeled data. By designing reasonable pre-training tasks, it is possible to learn effective data representation in an unsupervised manner, thereby achieving efficient industry classification. Contrastive learning constructs positive and negative sample pairs and optimizes the loss function, so that the model can learn more discriminative and robust feature representations. This self-supervised learning method of electricity consumption patterns with adaptive weights overcomes the limitation of traditional supervised learning relying on a large amount of labeled data, improves the generalization ability of the model, overcomes the shortcomings of traditional supervised and unsupervised learning in classification accuracy, and significantly improves the classification effect. The self-supervised mechanism combines multimodal data features and achieves accurate industry classification through efficient specific industry classification and label generation.

[0059] 5. Optimize the loss function: The loss function used in the present invention for self-supervised learning to update the parameters of the fine-tuned BERT pre-trained model introduces a global weight parameter λ and a dynamic weight parameter λ j and the adaptive temperature parameter γ j , through λ and λ j It can control the degree of attention the model pays to the positive sample and each negative sample, allowing different weights to be applied to the positive sample and different negative samples, so that the influence of similarity can be adjusted for different samples, so that the model can learn features more accurately; through γ j Ability to control the temperature correction for each negative example so that the model can be more sensitive or more tolerant to certain negative examples when needed.

[0060] In summary, the present invention overcomes many limitations of traditional machine learning methods by collecting and preprocessing multimodal data, using the BERT pre-training model for feature extraction, and combining the self-supervision mechanism of contrastive learning for industry classification. It can not only process and analyze complex multimodal data and improve the accuracy of industry feature extraction and classification, but also provide efficient and reliable technical means for improving the level of electricity consumption feature analysis and management in different specific industries in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a schematic diagram of the overall framework flow of the present invention;

[0062] Figure 2 It is a schematic diagram of the electricity load curve of different industries;

[0063] Figure 3 This is a schematic diagram of BERT fine-tuning. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. The embodiments described in this application are only embodiments of a part of the present invention, rather than all embodiments. Based on the spirit of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the protection scope of the present invention.

[0065] Embodiment 1 of the present invention provides a method for classifying electricity consumption industries based on multimodal data feature extraction, such as Figure 1 As shown, the following steps are included:

[0066] S1: Collect multimodal data from different electricity consumption industries and pre-process the collected multimodal data;

[0067] S1.1: Collect multimodal data from different electricity consumption sectors;

[0068] It is further preferred to collect multimodal data of specific industries, such as collecting multimodal data of traditional industries such as cement, steel, and chemicals, and emerging industries such as new energy lithium batteries and photovoltaics. These data include but are not limited to policy data, environmental data, electrical quantity data, and industry production habit data. Define the following parameters:

[0069] Policy data P: text materials such as government regulations, industry standards, etc.

[0070] Environmental data E: Environmental conditions around the factory, such as temperature, humidity, etc.

[0071] Electrical quantity data Q: power consumption, voltage, current, etc.

[0072] Archival data H: production logs, maintenance records, etc.

[0073] S1.2: Preprocess the collected multimodal data;

[0074] Further preferably, in order to ensure the consistency and quality of the data, the collected data needs to be preprocessed, including data cleaning, denoising, standardization, missing value filling, etc. The preprocessed data can better adapt to the subsequent deep learning model training and feature extraction process.

[0075] The data preprocessing steps include:

[0076] S1.2.1 Data cleaning: remove noise and outliers. Noise processing is to reduce the noise data in the data set, which can improve the training effect and prediction accuracy of the model; outliers refer to those unreasonable values ​​in the data set. Outliers in the data set may be caused by sensor failure, manual input errors or abnormal events. These outliers will lead to incorrect conclusions in some scenarios (such as linear regression models, K-means clustering, etc.). Detection of outliers includes simple statistical analysis, box plots, etc.; outlier processing methods include deletion, treating as missing values, mean correction, capping method, binning method, regression interpolation, multiple interpolation and no processing.

[0077] S1.2.2 Data standardization: Convert the data to a format with zero mean and unit variance to eliminate the effects of different dimensions. Use the formula:

[0078]

[0079] Among them, μ and σ are the mean and standard deviation of the original data respectively.

[0080] S1.2.3 Missing value filling: Missing values ​​refer to clustering, grouping, deletion or truncation of data due to lack of information in rough data. It means that the value of one or some attributes in the existing data set is incomplete. For power grid data, data missing problems may occur due to failure of data collection or data preservation at the metering point. For missing data points, mean interpolation, median interpolation or similarity-based interpolation methods can be used.

[0081] The preprocessed data set is represented as D = {P, E, Q, H}.

[0082] S2: Convert the preprocessed multimodal data into a unified input vector.

[0083] Further preferably, the BERT (Bidirectional Encoder Representations from Transformers) pre-trained model adopted in the present invention is a natural language processing model based on deep learning, which can effectively process and analyze complex multimodal data, thereby achieving more accurate industry classification and feature extraction.

[0084] The core architecture of BERT consists of multiple layers of Transformer encoders, each of which contains a self-attention mechanism and a feed-forward neural network.

[0085] The basic building blocks of Transformer are multi-head self-attention mechanism and feedforward neural network.

[0086] The multi-head self-attention mechanism allows the model to focus on different positions in the input sequence in different representation subspaces. It can learn rich context representations, and the calculation formula is as follows:

[0087]

[0088] Where Q, K, and V represent query, key, and value matrices, respectively. k The dimension of the key.

[0089] BERT learns rich contextual feature representations through pre-training on a large-scale corpus. Specifically, BERT uses two pre-training tasks: Masked Language Model (MLM) and Next Sentence Prediction (NSP). MLM learns the contextual representation of words by randomly masking a portion of the words in the input sequence and predicting these words. NSP enhances the model's sentence-level understanding ability by predicting the sequential relationship between sentence pairs.

[0090] In the present invention, the preprocessed multimodal data first needs to be converted into an input form acceptable to BERT, and then input into the BERT model for feature extraction. In this way, a deep feature representation of multimodal data can be obtained:

[0091] Use BERT's pre-trained model to convert text information of policy data, environmental data, and archival data into embedding vectors.

[0092] BERT achieves a deep understanding of the input text by stacking multiple layers of Transformer encoders.

[0093] BERT's text embedding layer is able to convert input text data into a high-dimensional vector representation.

[0094] Input the text content of policy documents, environmental reports, and archival data into BERT to obtain its high-dimensional representation vector. The formula is:

[0095] P embed =BERT embed (P),E embed =BERT embed (E),H embed =BERT embed (H)

[0096] Among them, P embed is the high-dimensional representation vector of the policy document data, E embed is a high-dimensional vector representation of environmental report data, H embed It is a high-dimensional vector representation of the archival data.

[0097] It is understandable that the text embedding layer of BERT is before the encoder. At this time, the preprocessed data is input into the embedding layer of BERT to obtain the embedding vector, which is not counted as the input into the BERT model. The feature vector extracted after the subsequent input into BERT is obtained after the input into the encoder.

[0098] Numerical feature standardization: For electrical quantity data, numerical feature standardization is performed to adapt to the BERT model input requirements. The standardization method is as follows:

[0099]

[0100] Among them, μ Q and σ Q are the mean and standard deviation of the electrical quantity data, Q is the electrical quantity data, Q norm It is the standardized electrical quantity data.

[0101] Through the above steps, the present invention integrates multimodal data such as policy data, environmental data, electrical quantity data, industry production habit data, etc. of traditional industries such as cement, steel, and chemicals, and emerging industries such as new energy lithium batteries and photovoltaics, to perform feature extraction and data dimensionality reduction, and construct high-quality multimodal data feature representation.

[0102] S3: Use the power load curves of different industries to fine-tune the BERT pre-trained model, and input the unified input vector of S2 into the fine-tuned BERT pre-trained model to extract the power load feature vector;

[0103] Further preferably, the electricity load curves of specific users in a group are studied to learn the electricity consumption patterns of different industries; based on the extracted multimodal features, the pre-trained BERT model is fine-tuned using the electricity load data of different industries, and the format-converted data is input into the BERT pre-trained model to extract the electricity load features.

[0104] It can be understood that the electricity load curves of different industries are independent of the unified input vector obtained by S2. The electricity load curves of different industries used to fine-tune the model are not the unified input vector obtained by S2. The unified input vector obtained by S2 is only used to input the fine-tuned model to obtain the electricity load characteristic vector.

[0105] Electricity load curves of different industries are as follows: Figure 2 shown.

[0106] like Figure 3 As shown, during fine-tuning, define an appropriate downstream classification task:

[0107] The user's electricity load curve is classified according to industry, and the BERT model is supervised by setting label data, so that it can predict the industry category to which the user belongs based on the input electricity load curve.

[0108] During the fine-tuning process, the industry's electricity consumption data is used for supervised learning. The specific steps include data segmentation, model training, and model verification and evaluation.

[0109] First, the dataset is divided into training set, validation set and test set to ensure the generalization ability and evaluation effect of the model.

[0110] Next, the BERT model is trained using the training set data and the model parameters are adjusted so that it can better adapt to the electricity consumption characteristics of specific industries.

[0111] Finally, the validation set data is used to evaluate the performance of the model, and the classification and prediction effects of the model are evaluated through indicators such as accuracy, recall, and mean square error. The model structure and parameters are further optimized based on the evaluation results.

[0112] Through in-depth analysis of the power load curve of specific users and fine-tuning of the BERT model, the power consumption patterns of different industries can be learned. These patterns include identifying the peak and valley characteristics of peak and valley time periods, understanding the power consumption patterns of different industries; analyzing the periodic changes in power load, such as daily cycles, weekly cycles, etc., to predict future power consumption trends; identifying abnormal fluctuations in power load, such as equipment failures, abnormal power consumption behaviors, etc., to improve power safety and management levels. In this way, the present invention can effectively study the power load curves of specific users, learn the power consumption patterns of different industries, and improve the accuracy and robustness of power consumption feature analysis and industry classification. This not only helps power grid companies optimize power management, but also provides an important reference for the company's own energy management and cost control.

[0113] Specifically, the power load curve data sets of different industries after standardization and other preprocessing are divided into training set, validation set and test set according to the ratio of 70%, 15% and 15%. The specific operations are as follows:

[0114] Training set: used to train the BERT model, containing 70% of the data.

[0115] Validation set: used to adjust model parameters and avoid overfitting, containing 15% of the data.

[0116] Test set: used to evaluate model performance, contains 15% of the data.

[0117] Optimizer: Use Adam optimizer to update model parameters and set learning rate α. Adam optimizer combines momentum and adaptive learning rate adjustment. The formula is as follows:

[0118] m t =β1m t-1 +(1-β1)g t

[0119]

[0120] Among them, m t is the first-order moment estimate of the gradient, representing momentum; β1 is the exponential decay rate of the first-order moment estimate; g t is the gradient of the current time step; v t is the second-order moment estimate of the gradient, which represents the moving average of the square of the gradient; β2 is the exponential decay rate of the second-order moment estimate; is the bias-corrected momentum estimate; is the moving average of the bias-corrected squared gradient; α is the learning rate; ε is a small constant to prevent division by zero errors.

[0121] Number of training rounds: Set the number of iterations (epochs). In each round of training, the model will traverse the entire training set and update the parameters through backpropagation. The number of training rounds depends on the complexity of the task and the amount of data.

[0122] Supervised learning: By setting label data, the power load curve is classified according to industry. The cross entropy loss function is used to calculate the loss:

[0123]

[0124] Where N is the number of samples, C is the number of categories, and y ic is the true label of sample i, is the probability predicted by the model.

[0125] Model optimization: Gradients are calculated through back-propagation and the Adam optimizer is used to update the model parameters.

[0126] The fine-tuned BERT model can output classification results based on the input power load curve feature vector, achieving accurate prediction of the industry to which the user belongs.

[0127] Input data fusion: Merge the above embedding vector and the normalized numerical vector into a unified input vector:

[0128] X={P embed ,E embed ,H embed ,Q norm}

[0129] Model input: Input the fused input vector into the fine-tuned BERT model and perform forward propagation to obtain the output features:

[0130] zk =BERT(X k )

[0131] Among them, X k is the feature of the kth unified input vector X.

[0132] Through the above steps, the present invention uses deep learning technology to achieve end-to-end multimodal feature autonomous learning, effectively improving the accuracy and robustness of feature extraction; for the electricity consumption behavior of specific users in a group, the power consumption visualization load curve is studied and constructed to analyze the electricity consumption characteristics of different industries. Through multimodal data fusion embedding representation and adaptive weight calculation, the power consumption mode and characteristics of specific industries can be accurately determined, and the accuracy and reliability of power consumption feature analysis can be improved.

[0133] S4: Based on the power load feature vector extracted by S3, the parameters of the fine-tuned BERT pre-trained model are updated in combination with the self-supervised learning mechanism, and the BERT pre-trained model with updated parameters is used to classify the power consumption industry.

[0134] Further preferably, after fine-tuning, although the BERT model is already capable of classification, in order to further improve the robustness and accuracy of classification, the present invention also combines industry classification with a self-supervised learning mechanism. Self-supervised learning learns effective data representations without supervision by designing reasonable pre-training tasks, thereby achieving efficient industry classification. Specifically, a self-supervised method based on contrastive learning can be used to construct positive and negative sample pairs using the feature vectors extracted by BERT, so that the model can learn more discriminative data feature representations.

[0135] It is understandable that after self-supervised learning, only model optimization (parameter update) is completed, and the classification result is not output. The final classification result is obtained by re-acquiring new multimodal data and inputting it into the optimized model.

[0136] Contrastive learning is combined with BERT as follows:

[0137] First, the feature vectors extracted by BERT are used to construct positive and negative sample pairs. The positive sample pairs are feature vectors of the power load curve from the same industry, while the negative sample pairs are feature vectors of the power load curves from different industries. In this way, the feature differences between industries can be significantly represented.

[0138] The goal of optimization (contrastive learning) is to maximize the similarity of positive sample pairs while minimizing the similarity of negative sample pairs. The loss function can be as follows:

[0139]

[0140] Among them, i, i+, ij represent the index of the current sample, the index of its positive sample and the index of its negative sample j respectively, z i ,z i+ ,z i-j is the corresponding feature representation, sim(,) is the similarity measurement function, τ is the temperature parameter, N is the number of samples in the current batch, K is the number of negative samples of the current sample i, λ is the global weight parameter, λ j is the dynamic weight parameter, γ j Adaptive temperature parameter for each negative sample.

[0141] During contrastive learning, BERT's parameters are updated in the process of optimizing the loss function. This means that through contrastive learning training, BERT's feature extraction capabilities will be further enhanced, allowing it to perform better in industry classification tasks. Through contrastive learning, BERT can further enhance its ability to distinguish between different industry characteristics. This combination not only utilizes BERT's powerful feature extraction capabilities, but also improves the model's performance under unsupervised conditions through self-supervised learning, ultimately achieving higher accuracy and robust industry classification.

[0142] Through the above steps, the present invention studies the specific industry classification and pseudo-label establishment technology, and uses the self-supervision mechanism to classify the industry. Develop an adaptive weighted power consumption law analysis method to overcome the shortcomings of traditional supervised and unsupervised learning in classification accuracy and significantly improve the classification effect. The self-supervision mechanism combines multimodal data features to establish an efficient specific industry classification and pseudo-label generation system to achieve accurate industry classification.

[0143] Through this specific implementation scheme, the present invention can achieve effective feature extraction and industry classification of multimodal data, improve the accuracy and robustness of classification, and provide a new solution for enterprise electricity management.

[0144] Embodiment 2 of the present invention provides a system for classifying electricity consumption industries based on multimodal data feature extraction, including:

[0145] The data collection and processing module is used to collect multimodal data from different electricity-consuming industries and pre-process the collected multimodal data;

[0146] The data conversion module is used to convert the format of the preprocessed multimodal data to obtain a unified input vector;

[0147] The feature extraction module is used to fine-tune the BERT pre-trained model using the power load curves of different industries, and input the unified input vector into the fine-tuned BERT pre-trained model to extract the power load feature vector;

[0148] The industry classification module is used to update the parameters of the fine-tuned BERT pre-trained model based on the power load feature vector and the self-supervised learning mechanism, and use the BERT pre-trained model with updated parameters to classify the power consumption industry.

[0149] Embodiment 3 of the present invention provides a terminal, including a processor and a storage medium; the storage medium is used to store instructions;

[0150] The processor is configured to operate according to the instructions to execute the steps of the method.

[0151] Embodiment 4 of the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method when executed by a processor.

[0152] Compared with the prior art, the beneficial effects of the present invention include at least:

[0153] 1. Multimodal data fusion: This invention integrates multimodal data such as policy data, environmental data, electrical quantity data, industry production habit data, etc. from traditional industries such as cement, steel, and chemicals, and emerging industries such as new energy lithium batteries and photovoltaics, to achieve effective fusion of complex and diverse data sources, to perform feature extraction and data dimensionality reduction, and to construct high-quality multimodal data feature representation. This multimodal data fusion method improves the comprehensiveness and accuracy of feature extraction and overcomes the limitations of traditional single-modal data processing methods.

[0154] 2. Deep feature extraction: The BERT pre-trained model is used for feature extraction. Through the multi-layer Transformer encoder and self-attention mechanism, the deep features of the data can be automatically learned. This deep feature extraction method based on a multi-layer neural network structure can better capture the complex patterns and associations in the data, realize end-to-end multimodal feature autonomous learning, effectively improve the accuracy and robustness of feature extraction, and improve the richness and accuracy of feature representation, thereby improving the accuracy and generalization ability of classification.

[0155] 3. Model fine-tuning: Based on the electricity consumption behavior of specific users in a group, we studied and constructed a visualized load curve for electricity consumption, and analyzed the electricity consumption characteristics of different industries. Through multimodal data fusion embedding representation, we can accurately determine the electricity consumption patterns and characteristics of specific industries, and improve the accuracy and reliability of electricity consumption characteristic analysis;

[0156] 4. Self-supervised learning mechanism: The specific industry classification and pseudo-label establishment technology are studied, and the self-supervised mechanism is used for industry classification. By introducing the self-supervised mechanism of contrastive learning, there is no need for a large amount of labeled data. By designing reasonable pre-training tasks, it is possible to learn effective data representation in an unsupervised manner, thereby achieving efficient industry classification. Contrastive learning constructs positive and negative sample pairs and optimizes the loss function, so that the model can learn more discriminative and robust feature representations. This self-supervised learning method of electricity consumption patterns with adaptive weights overcomes the limitation of traditional supervised learning relying on a large amount of labeled data, improves the generalization ability of the model, overcomes the shortcomings of traditional supervised and unsupervised learning in classification accuracy, and significantly improves the classification effect. The self-supervised mechanism combines multimodal data features, and achieves accurate industry classification through efficient specific industry classification and pseudo-label generation.

[0157] 5. Optimize the loss function: The loss function used in the present invention for self-supervised learning to update the parameters of the fine-tuned BERT pre-trained model introduces a global weight parameter λ and a dynamic weight parameter λ j and the adaptive temperature parameter γ j , through λ and λ j It is possible to control the degree of attention the model pays to the positive sample and each negative sample, allowing different weights to be applied to the positive sample and different negative samples. This can adjust the influence of similarity for different samples, allowing the model to learn features more accurately; j Ability to control the temperature correction for each negative example so that the model can be more sensitive or more tolerant to certain negative examples when needed.

[0158] In summary, the present invention overcomes many limitations of traditional machine learning methods by collecting and preprocessing multimodal data, using the BERT pre-training model for feature extraction, and combining the self-supervision mechanism of contrastive learning for industry classification. It can not only process and analyze complex multimodal data and improve the accuracy of industry feature extraction and classification, but also provide efficient and reliable technical means for improving the level of electricity consumption feature analysis and management in different specific industries in practical applications.

[0159] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0160] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0161] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0162] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for classifying electricity consumption industries based on multimodal data feature extraction, characterized in that: include: Collect multimodal data from different electricity-consuming industries and pre-process the collected multimodal data; Convert the preprocessed multimodal data into a unified input vector. The BERT pre-trained model is fine-tuned using the power load curves of different industries, and the unified input vector is input into the fine-tuned BERT pre-trained model to extract the power load feature vector; Based on the extracted power load feature vector, the parameters of the fine-tuned BERT pre-trained model are updated in combination with the self-supervised learning mechanism, and the BERT pre-trained model with updated parameters is used to classify the power consumption industry.

2. The method for classifying electricity consumption industries based on multimodal data feature extraction according to claim 1 is characterized in that: The electricity-consuming industries include cement industry, steel industry, chemical industry, new energy lithium battery industry, and photovoltaic industry; The multimodal data includes policy data, environmental data, electrical quantity data and industry production habit data.

3. The method for classifying electricity consumption industries based on multimodal data feature extraction according to claim 1 is characterized in that: The preprocessing of the collected multimodal data includes data cleaning, data standardization and missing value filling.

4. The method for classifying electricity consumption industries based on multimodal data feature extraction according to claim 1 is characterized in that: The format conversion of the preprocessed multimodal data specifically includes: 1) BERT’s text embedding layer converts the preprocessed policy data, environmental data, and industry production habit data into the following embedding vector representation: P embed =BERT embed (P),E embed =BERT embed (E),H embed =BERT embed (H) Among them, P embed is the embedding vector of the preprocessed policy data P, E embed is the embedding vector of the preprocessed environmental data E, H embed is the embedding vector of the preprocessed industry production habit data H; 2) The preprocessed electrical quantity data is standardized according to the following numerical characteristics: Among them, μ Q and σ Q are the mean and standard deviation of the preprocessed electrical quantity data, Q is the preprocessed electrical quantity data, and Q norm It is the standardized electrical quantity data.

5. The method for classifying electricity consumption industries based on multimodal data feature extraction according to claim 1 is characterized in that: The BERT pre-training model is fine-tuned using the power load curves of different industries, including: The user's electricity load curve is classified according to industry, and labels are set to build a data set. The BERT model is supervised by cross entropy loss function, and the Adam optimizer is used to update the model parameters.

6. The method for classifying electricity consumption industries based on multimodal data feature extraction according to claim 5 is characterized by: The Adam optimizer performs model parameter θ t The update formula is as follows: m t =β1m t-1 +(1-β1)g t Among them, t is the time step of the current training; m t is the first-order moment estimate of the gradient, representing momentum; β1 is the exponential decay rate of the first-order moment estimate; g t is the gradient of the current time step; v t , v t-1 is the second-order moment estimate of the gradient of the previous time step t-1 at the current time step t, which represents the moving average of the square of the gradient; β2 is the exponential decay rate of the second-order moment estimate; is the bias-corrected momentum estimate; is the moving average of the squared gradient after deviation correction; α is the learning rate; ε is a constant; θ t ,θ t-1 is the model parameter of the current time step t and the previous time step t-1.

7. The method for classifying electricity consumption industries based on multimodal data feature extraction according to claim 5 is characterized by: The cross entropy loss function is: Among them, N is the number of samples, C is the number of categories, yic is the true label of sample i, is the probability predicted by the model.

8. The method for classifying electricity consumption industries based on multimodal data feature extraction according to claim 1 is characterized by: The unified input vector is input into the fine-tuned BERT pre-trained model to extract the power load feature vector: z k =BERT(X k ) Among them, X k is the feature of the kth unified input vector X; The unified input vector X is: X={P embed ,E embed ,H embed ,Q norm } Among them, P embed is the embedding vector of the preprocessed policy data P, E embed is the embedding vector of the preprocessed environmental data E, H embed is the embedding vector Q of the preprocessed industry production habit data H norm It is the standardized electrical quantity data.

9. The method for classifying electricity consumption industries based on multimodal data feature extraction according to claim 1 is characterized by: The method of updating the parameters of the fine-tuned BERT pre-trained model based on the extracted power load feature vector and combining the self-supervised learning mechanism includes: The extracted power load feature vectors are used to construct positive and negative sample pairs, where the positive sample pairs are power load feature vectors from the same industry, and the negative sample pairs are power load feature vectors from different industries; Based on the constructed positive and negative samples, the following loss function is used for self-supervised learning to update the parameters of the fine-tuned BERT pre-trained model: Among them, i, i+, ij represent the index of the current sample, the index of its positive sample and the index of its negative sample j respectively, z i ,z i+ ,z i-j is the corresponding feature representation; sim(,) is the similarity measurement function; N is the number of samples in the current batch; λ is the global weight parameter; τ is the temperature parameter; K is the number of negative samples of the current sample i; λ j is the dynamic weight parameter; γ j Adaptive temperature parameter for each negative sample.

10. A system for classifying electricity consumption industries based on multimodal data feature extraction, using the method described in any one of claims 1 to 9, characterized in that: The system comprises: The data collection and processing module is used to collect multimodal data from different electricity-consuming industries and pre-process the collected multimodal data; The data conversion module is used to convert the format of the preprocessed multimodal data to obtain a unified input vector; The feature extraction module is used to fine-tune the BERT pre-trained model using the power load curves of different industries, and input the unified input vector into the fine-tuned BERT pre-trained model to extract the power load feature vector; The industry classification module is used to update the parameters of the fine-tuned BERT pre-trained model based on the power load feature vector and the self-supervised learning mechanism, and use the BERT pre-trained model with updated parameters to classify the power consumption industry.

11. A terminal comprising a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1-9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images