Heterogeneous feature evaluation method in combination with transfer learning

Abstract

The invention discloses a heterogeneous feature evaluation method in combination with transfer learning. The method comprises the following steps: S1, identifying and preprocessing a heterogeneous data source; s2, feature space mapping; s3, feature selection and evaluation; s4, constructing a federal learning framework; s5, fusing and optimizing the model; s6, applying a privacy protection technology; s7, performing real-time feedback and iterative optimization; s8, performing cross-domain knowledge migration exploration; s9, case research and empirical analysis are carried out; in order to solve the problem of fusion of heterogeneous data sources, feature representation of different data sources is effectively unified through feature space mapping and a domain self-adaption technology, and features most critical to a prediction task are accurately recognized by utilizing a model-based feature selection technology of LASSO regression; the introduction of the federated learning framework allows a plurality of participants to cooperatively train the model without sharing original data, which provides a new solution for data sharing.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and in particular to a heterogeneous feature evaluation method combined with transfer learning. Background Art

[0002] The application of big data is one of the core functional features of the digital power grid, and the storage, utilization and service of big data are also important supports for the digital and intelligent transformation of various business functions of the power grid. At present, in the face of the demand for efficient access to the massive multi-source heterogeneous power big data generated under the development trend of the digital power grid, the ability of all-domain data aggregation and opening has been realized through digital construction methods.

[0003] With the rapid growth of the data volume of the power system, how to effectively fuse and evaluate heterogeneous data from different data sources has become a challenge. When dealing with heterogeneous data, there are problems such as inconsistent feature spaces and large differences in data distributions, which affect the effect of data fusion and the generalization ability of the model. Summary of the Invention

[0004] In order to solve the above-mentioned existing technical problems, the present invention provides a heterogeneous feature evaluation method combined with transfer learning.

[0005] The technical solution of the present invention is realized as follows:

[0006] A heterogeneous feature evaluation method combined with transfer learning includes the following steps:

[0007] S1. Identification and preprocessing of heterogeneous data sources: Identify the heterogeneous data sources in the power system and perform preprocessing on them, including data cleaning, standardization and normalization;

[0008] S2. Feature space mapping: Use transfer learning technology to map the data in different feature spaces to a common feature space to reduce the distribution differences between the source domain and the target domain;

[0009] S3. Feature selection and evaluation: Develop a heterogeneous feature evaluation method based on transfer learning, identify key features through feature selection technology, and evaluate their impact on the model performance;

[0010] S4. Construction of a federated learning framework: Construct a federated learning framework that allows multiple participants to jointly train a model while protecting data privacy;

[0011] S5. Model fusion and optimization: Realize the fusion of heterogeneous models within the federated learning framework, use transfer learning technology to integrate the knowledge of different models, and optimize the overall model performance;

[0012] S6. Application of privacy protection technology: Apply privacy protection technology to ensure the data privacy of participants during the data fusion and model training processes;

[0013] S7, Real-time Feedback and Iterative Optimization: Establish a real-time feedback mechanism and continuously iterate and optimize the model using the incremental learning strategy of transfer learning;

[0014] S8, Exploration of Cross-domain Knowledge Transfer: Explore cross-domain knowledge transfer paths to improve the generalization ability of the model;

[0015] S9, Case Study and Empirical Analysis: Verify the effectiveness of the proposed method through case studies, such as load forecasting in smart grids.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] The present invention solves a series of challenges brought about by the rapid growth of data volume in the field of power systems, specifically:

[0018] 1. For the problem of fusing heterogeneous data sources, the present invention effectively unifies the feature representations of different data sources through feature space mapping and domain adaptation techniques, reduces the problems caused by inconsistent feature spaces and data distribution differences, improves the effect of data fusion, and significantly enhances the generalization ability of the model on unknown data;

[0019] 2. The present invention uses the LASSO regression-based model feature selection technique to accurately identify the features that are most critical for the prediction task, further optimizing the performance of the model. While protecting data privacy, the introduction of the federated learning framework allows multiple participants to collaborate in training the model without sharing the original data, providing a new solution for data sharing;

[0020] 3. By using the knowledge distillation technique, the small student model can learn and imitate the knowledge of the large teacher model, thus improving the prediction accuracy of the model while keeping the model lightweight. The establishment of a real-time feedback and iterative optimization mechanism ensures that the model can adapt to the dynamic changes of data and achieve continuous performance improvement. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic flow chart of a method for evaluating heterogeneous features combined with transfer learning according to the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0022] In order to make the objectives, features, and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0023] Example 1

[0024] A heterogeneous feature evaluation method combining transfer learning in the present invention includes the following steps:

[0025] S1. Identification and preprocessing of heterogeneous data sources: Identify heterogeneous data sources in the power system and perform preprocessing on them, including data cleaning, standardization, and normalization;

[0026] Collect data from different power equipment, perform data cleaning, remove invalid and incorrect data records, perform data standardization, convert data with different dimensions into a unified dimension, and apply normalization processing to scale the data to the interval [0, 1];

[0027] Data cleaning ensures data quality and provides accurate input for subsequent analysis. Standardization and normalization ensure that different features have the same importance in model training;

[0028] S2. Feature space mapping: Use transfer learning technology to map data in different feature spaces to a common feature space, reducing the distribution difference between the source domain and the target domain;

[0029] Use domain adaptation technology in transfer learning. Utilize conditional domain adaptation to define feature extractors and classifiers for the source domain and the target domain. Through adversarial training, align the feature distributions of the source domain and the target domain; Adversarial training realizes the mapping of the feature space by minimizing the loss of the domain discriminator;

[0030] S3. Feature selection and evaluation: Develop a heterogeneous feature evaluation method based on transfer learning. Identify key features through feature selection technology and evaluate their impact on model performance;

[0031] Apply a model-based feature selection method, adopt LASSO regression to select important features, and use domain adaptation in transfer learning to evaluate the generalization ability of features in the target domain; Among them, LASSO regression realizes feature selection through a penalty coefficient, enhancing the interpretability of the model;

[0032] Model-based feature selection, adopting LASSO regression:

[0033] Define the objective function:

[0034] Use the objective function of LASSO regression, which combines the squared error loss and the L1 regularization term:

[0035]

[0036] Among them, β is the model parameter, X is the feature matrix, y is the target vector, n is the number of samples, and λ is the regularization parameter;

[0037] Solving parameters:

[0038] Solve the above objective function using a numerical optimization method, and use the coordinate descent method to solve:

[0039]

[0040] where S is the soft threshold function, and λ n is a constant related to the number of samples; obtain the model parameter β;

[0041] Feature selection:

[0042] According to the obtained β, select the features corresponding to non-zero coefficients, because these features have a significant impact on model prediction;

[0043] Domain adaptation evaluation in transfer learning:

[0044] Domain adaptation model training:

[0045] Using transfer learning technology, adopt a domain adversarial network (Domain Adversarial Neural Network, DANN) to train a domain discriminator to distinguish the feature representations of the source domain and the target domain;

[0046] Calculating domain-unrelated feature representations:

[0047] Using the domain adaptation model, transform the features of the source domain and the target domain so that the domain discriminator has difficulty distinguishing which domain these features come from;

[0048] Loss function of the domain discriminator:

[0049] L D =-E x～Ps [logD(x)]-E x～Pt [log(1-D(x))]

[0050] where D is the domain discriminator, and P s and P t are the data distributions of the source domain and the target domain respectively;

[0051] Convert the source domain features into target domain features so that the transformed features have low separability on the domain discriminator;

[0052] Evaluating generalization ability:

[0053] Evaluate the performance of the domain adaptation model on the target domain to evaluate the generalization ability of the feature selection method on the target domain;

[0054] S4. Construction of the federated learning framework: Construct a federated learning framework that allows multiple participants to jointly train a model while protecting data privacy;

[0055] Design a federated learning architecture including a central server and multiple edge nodes. The edge nodes are responsible for local data training, and the central server is responsible for aggregating model parameters. Federated learning allows for model training across multiple data sources while protecting data privacy.

[0056] S5. Model Fusion and Optimization: Achieve the fusion of heterogeneous models within the federated learning framework, utilize transfer learning techniques to integrate the knowledge of different models, and optimize the overall model performance.

[0057] Use knowledge distillation techniques to transfer the knowledge of large models to small models, and in the federated learning framework, achieve the parameter update of the global model and local models. Knowledge distillation improves the performance of the student model through a teacher-student framework.

[0058] It is achieved through knowledge distillation techniques in the federated learning framework, and the specific process can be divided into the following stages:

[0059] Preparation Stage:

[0060] Select the teacher model (T): Select a pre-trained and high-performance large model as the knowledge source.

[0061] Select the student model (S): Select a lightweight model that is expected to learn and imitate the behavior of the teacher model.

[0062] Output of the teacher model: Among them, Z is the original output of the teacher model, and T is the temperature parameter.

[0063] After applying the softmax function, the original output Z of the teacher model is converted into a probability distribution P T , and this probability distribution reflects the prediction confidence of the teacher model for each class and is used to guide the learning of the student model.

[0064] Loss function of the student model:

[0065] Among them, L real is the loss based on the true label, L soft is the soft label loss based on the output of the teacher model, and α is a hyperparameter that balances the two losses.

[0066] Combining the loss L real based on the true label and the soft label loss L soft , the total loss Ls of the student model reflects the performance of the student model in imitating the teacher model and fitting the true label. This total loss is used to guide the training process of the student model. By minimizing the total loss Ls of the student model, the parameters of the student model will be updated.

[0067] Training stage:

[0068] Independent training: Independently train the teacher model and the student model on their respective datasets;

[0069] Knowledge distillation: Use the output of the teacher model (the probability distribution of the softmax layer) as soft labels; When training the student model, in addition to using the true labels, also use these soft labels as auxiliary losses;

[0070] Distillation strategy:

[0071] Loss function: The loss function of the student model consists of two parts, the loss of the true labels and the loss of the teacher model output;

[0072] Temperature adjustment: Use the temperature parameter T to adjust the output distribution of the teacher model to make it smoother;

[0073] Optimization and evaluation:

[0074] Joint optimization: Simultaneously optimize the performance of the student model on true labels and soft labels;

[0075] Evaluation: Evaluate the performance of the student model on the target task to ensure that it reaches or approaches the level of the teacher model; Finally, the performance of the student model is evaluated by classification accuracy, precision, recall, and F1-score metrics, which reflect the effectiveness of the student model in practical applications;

[0076] S6. Application of privacy protection technology: Apply privacy protection technology to ensure the data privacy of participants during data fusion and model training;

[0077] Apply homomorphic encryption or secure multi-party computation technology to protect privacy in model training; Homomorphic encryption allows computations on encrypted data, while SMPC allows multiple parties to jointly compute results while protecting privacy;

[0078] S7. Real-time feedback and iterative optimization: Establish a real-time feedback mechanism and continuously iterate and optimize the model using the incremental learning strategy of transfer learning;

[0079] Establish a feedback loop, compare the model predictions with the actual results, apply the incremental learning strategy, and update the model according to the feedback; Incremental learning allows the model to adapt to new data and achieve continuous optimization; Specifically, it includes:

[0080] Establish a feedback loop: After the model is deployed, continuously collect the difference data between the model predictions and the actual results;

[0081] Performance monitoring: Real-time monitor the key performance metrics of the model, such as accuracy, recall, and F1-score;

[0082] Error analysis: Analyze the model prediction errors, identify the reasons for the decline in model performance, whether due to changes in data distribution or limitations of the model itself;

[0083] Incremental learning: Utilize the incremental learning strategy in transfer learning to gradually integrate newly collected data into the model instead of completely retraining;

[0084] Model fine-tuning: Fine-tune the model parameters according to the feedback data to adapt to the new data distribution;

[0085] Model update: Trigger model updates regularly or based on the threshold of performance decline to integrate new knowledge and fix model biases;

[0086] Automated iteration: Establish an automated process to automatically trigger the iterative optimization process when the model performance is below the predetermined standard;

[0087] Model version control: Maintain different versions of the model to track improvements during the iterative process and allow rolling back to previous versions of the model;

[0088] Feedback from users and domain experts: Collect feedback from users and domain experts to understand the performance of the model in actual applications and optimize the model accordingly.

[0089] S8. Exploration of cross-domain knowledge transfer: Explore cross-domain knowledge transfer approaches to improve the generalization ability of the model;

[0090] Identify the correlation between the source domain and the target domain, apply transfer learning techniques such as domain adaptation to transfer knowledge; Cross-domain transfer improves the performance of the model in the target domain by learning the commonalities between the source domain and the target domain; Specifically:

[0091] Domain analysis: Identify the similarities and differences between different domains and analyze the potential connections between domains;

[0092] Select transfer learning strategy: Select a suitable transfer learning strategy according to the similarities between domains, such as instance-based transfer, feature-based transfer, model-based transfer;

[0093] Data preprocessing: Preprocess the data in the source domain and the target domain, including normalization and denoising, to reduce the distribution differences between domains;

[0094] Feature extraction: Extract the features of the source domain data and evaluate the applicability of these features in the target domain;

[0095] Knowledge transfer: Transfer the knowledge of the source domain to the target domain to improve the learning efficiency and performance of the target domain;

[0096] Domain Adaptation: Use domain adaptation techniques, such as adversarial training, to make the transferred knowledge better adapt to the data distribution of the target domain;

[0097] Model Training and Adjustment: Train the model on the target domain and adjust the transfer learning strategy and parameters according to the model's performance;

[0098] Performance Evaluation: Evaluate the performance of the transferred model on the target domain, such as accuracy and F1 score;

[0099] Iterative Optimization: According to the performance evaluation results, iteratively optimize the transfer learning strategy and model parameters;

[0100] Knowledge Integration: Integrate the transferred knowledge with the knowledge of the target domain itself to form a more robust model.

[0101] S9. Case Study and Empirical Analysis: Verify the effectiveness of the proposed method through case studies, such as smart grid load forecasting.

[0102] Select smart grid load forecasting as a case to train and evaluate the model; the case study verifies the effectiveness and applicability of the method;

[0103] In this embodiment, through heterogeneous data source identification and preprocessing, the availability and processing efficiency of data are improved, laying a solid foundation for subsequent analysis; using feature space mapping ensures that data from different feature spaces can be compatible and inclusive, reducing the distribution difference between the source domain and the target domain; further feature selection and evaluation help identify the most informative features, optimize the feature utilization of the model, and improve the prediction accuracy and efficiency of the model; through the construction of the federated learning framework and the application of privacy protection technology, data privacy protection is strengthened, enabling data sharing and model training to be carried out in a secure environment; using model fusion and optimization, by integrating the knowledge of different models, the overall model performance is improved, achieving a better prediction effect; furthermore, real-time feedback and iterative optimization enable the model to continuously learn and adapt according to the latest data, ensuring the long-term effectiveness and accuracy of the model; and through cross-domain knowledge transfer exploration, the generalization ability of the model in different domains and tasks is improved, increasing the applicability of the model; finally, this embodiment verifies the effectiveness and practicality of the proposed method through the introduction of case studies and empirical analysis, and through specific application cases, such as smart grid load forecasting; accurate data analysis and model prediction provide strong decision-making support for the operation and management of the power system.

Claims

1. A heterogeneous feature evaluation method combined with transfer learning, characterized by: The following steps are involved: S1. Identification and preprocessing of heterogeneous data sources: Identify heterogeneous data sources in the power system and preprocess them, including data cleaning, standardization and normalization; S2, feature space mapping: using transfer learning technology, the data in different feature spaces are mapped to a common feature space to reduce the distribution difference between the source domain and the target domain; S3. Feature selection and evaluation: Develop heterogeneous feature evaluation methods based on transfer learning, identify key features through feature selection techniques, and evaluate their impact on model performance; S4. Construction of a federated learning framework: Build a federated learning framework that allows multiple participants to jointly train models while protecting data privacy; S5. Model fusion and optimization: Achieve the fusion of heterogeneous models within the federated learning framework, use transfer learning technology to integrate the knowledge of different models, and optimize the overall model performance; S6. Application of privacy protection technology: Apply privacy protection technology to ensure the data privacy of participants during data fusion and model training; S7. Real-time feedback and iterative optimization: Establish a real-time feedback mechanism and use the incremental learning strategy of transfer learning to continuously iterate and optimize the model; S8. Exploration of cross-domain knowledge transfer: Explore cross-domain knowledge transfer paths to improve the generalization ability of the model; S9. Case study and empirical analysis: Through case study, the effectiveness of the proposed method is verified.

2. According to claim 1, a heterogeneous feature evaluation method combined with transfer learning is characterized in that: The S1 collects data from different power equipment, performs data cleaning, removes invalid and erroneous data records, performs data standardization, converts data of different dimensions into a unified dimension, applies normalization processing, and scales the data to the [0, 1] interval.

3. The method for evaluating heterogeneous features in combination with transfer learning according to claim 1, characterized in that: The S2 uses the domain adaptation technology in transfer learning and conditional domain adaptation to define feature extractors and classifiers of the source domain and the target domain, and aligns the feature distributions of the source domain and the target domain through adversarial training; Adversarial training achieves mapping of feature space by minimizing the loss of the domain discriminator.

4. The method for evaluating heterogeneous features in combination with transfer learning according to claim 1, characterized in that: The S3 applies a model-based feature selection method, adopts LASSO regression, selects important features, and uses domain adaptation in transfer learning to evaluate the generalization ability of features in the target domain; wherein, LASSO regression realizes feature selection through a penalty coefficient, thereby enhancing the interpretability of the model.

5. The method for evaluating heterogeneous features in combination with transfer learning according to claim 4, characterized in that: The steps of model-based feature selection using LASSO regression include: Define the objective function: The objective function of LASSO regression is combined with squared error loss and L1 regularization term: Among them, β is the model parameter, X is the feature matrix, y is the target vector, n is the number of samples, and λ is the regularization parameter; Solving parameters: Use numerical optimization methods to solve the above objective function, and use coordinate descent method to solve it: Where S is the soft threshold function, λ n is a constant related to the sample size; the model parameter β is obtained; Feature Selection: Based on the solved β, select those features corresponding to non-zero coefficients, because these features have a significant impact on the model prediction; Evaluation of Domain Adaptation in Transfer Learning: Domain Adaptive Model Training: Using transfer learning technology, a domain adversarial neural network (DANN) is used to train a domain discriminator to distinguish the feature representations of the source domain and the target domain. Compute domain-irrelevant feature representation: Use a domain adaptation model to transform the features of the source domain and the target domain, making it difficult for the domain discriminator to distinguish which domain these features come from; The loss function of the domain discriminator is: L D =-E x～Ps [logD(x)]-E x～Pt |[log(1-D(x))] Among them, D is the domain discriminator, P s and P t are the data distribution of the source domain and the target domain respectively; Convert source domain features into target domain features, so that the converted features have lower separability on the domain discriminator; Evaluating generalization ability: The performance of the domain adaptation model is evaluated on the target domain to assess the generalization ability of the feature selection method in the target domain.

6. The method for evaluating heterogeneous features in combination with transfer learning according to claim 1, characterized in that: The S5 uses knowledge distillation technology to transfer the knowledge of the large model to the small model, and implements parameter updates of the global model and the local model in the federated learning framework; knowledge distillation improves the performance of the student model through the teacher-student framework.

7. The method for evaluating heterogeneous features in combination with transfer learning according to claim 6, characterized in that: The S5 is implemented in the federated learning framework through knowledge distillation technology. The specific process can be divided into the following stages: Preparation stage: Select the teacher model (T): select a pre-trained large model with excellent performance as the knowledge source; Select the student model (S): select a lightweight model that is expected to learn and imitate the behavior of the teacher model; Output of the teacher model: Where Z is the original output of the teacher model and T is the temperature parameter; After applying the softmax function, the original output Z of the teacher model is converted into a probability distribution P T , this probability distribution reflects the teacher model’s prediction confidence for each category and is used to guide the learning of the student model; The loss function of the student model is: Among them, L real is the loss based on the true label, L soft is the soft label loss based on the output of the teacher model, and α is a hyperparameter that weighs the two losses; Combined with the true label loss L real and the soft label loss L output by the teacher model soft , the total loss Ls of the student model reflects the performance of the student model in imitating the teacher model and fitting the true label. This total loss is used to guide the training process of the student model; by minimizing the total loss Ls of the student model,, the parameters of the student model will be updated; Training phase: Independent training: Train the teacher model and the student model independently on their respective datasets; Knowledge distillation: Use the output of the teacher model (which is the probability distribution of the softmax layer) as soft labels; when training the student model, in addition to using the real labels, these soft labels are also used as auxiliary losses; Distillation strategy: Loss function: The loss function of the student model consists of two parts, the loss of the true label and the loss of the teacher model output; Temperature adjustment: Use the temperature parameter T to adjust the output distribution of the teacher model to make it smoother; Optimization and evaluation: Joint optimization: optimize the performance of the student model on both true labels and soft labels simultaneously; Evaluation: Evaluate the performance of the student model on the target task to ensure that it reaches or is close to the level of the teacher model; finally, the performance of the student model is evaluated by classification accuracy, precision, recall, and F1 score indicators, which reflect the effectiveness of the student model in practical applications.

8. The method for evaluating heterogeneous features in combination with transfer learning according to claim 1, characterized in that: The S7 establishes a feedback loop, compares the model predictions with the actual results, applies an incremental learning strategy, and updates the model based on the feedback; Incremental learning allows the model to adapt to new data and achieve continuous optimization; specifically: Establish a feedback loop: After the model is deployed, continuously collect data on the difference between the model's predictions and the actual results; Performance monitoring: real-time monitoring of key performance indicators of the model, such as accuracy, recall, and F1 score; Error analysis: Analyze model prediction errors and identify the reasons for the decline in model performance, whether it is due to changes in data distribution or limitations of the model itself; Incremental learning: Using the incremental learning strategy in transfer learning, newly collected data is gradually integrated into the model instead of completely retraining; Model fine-tuning: fine-tune model parameters based on feedback data to adapt to new data distribution; Model updates: Model updates are triggered periodically or based on a threshold of performance degradation to incorporate new knowledge and fix model deviations; Automated iteration: Establish an automated process to automatically trigger the iterative optimization process when the model performance is lower than the predetermined standard; Model versioning: Maintain different versions of the model to track improvements during iterations and allow rolling back to a previous version of the model; User and domain expert feedback: Collect feedback from users and domain experts to understand how the model performs in real-world applications and optimize the model accordingly.

9. The method for evaluating heterogeneous features in combination with transfer learning according to claim 1, characterized in that: S8 applies transfer learning technology, and cross-domain transfer improves the performance of the model in the target domain by learning the commonalities between the source domain and the target domain; specifically: Domain analysis: identifying similarities and differences between different domains and analyzing potential connections between domains; Select transfer learning strategy: Select appropriate transfer learning strategy based on the similarity between fields, such as instance-based transfer, feature-based transfer, and model-based transfer; Data preprocessing: Preprocess the data in the source and target domains, including normalization and denoising, to reduce the distribution differences between domains; Feature extraction: extracting features from source domain data and evaluating the applicability of these features in the target domain; Knowledge transfer: Transferring knowledge from the source domain to the target domain to improve learning efficiency and performance in the target domain; Domain adaptation: Use domain adaptation techniques, such as adversarial training, to make the transferred knowledge better adapt to the data distribution of the target domain; Model training and adjustment: Train the model in the target domain and adjust the transfer learning strategy and parameters based on the model performance; Performance evaluation: Evaluate the performance of the transferred model in the target domain, such as accuracy and F1 score; Iterative optimization: Iteratively optimize the transfer learning strategy and model parameters based on the performance evaluation results; Knowledge integration: Integrate the transferred knowledge with the target domain’s own knowledge to form a more robust model.

Images