Heterogeneous data fusion access method and system based on carbon intelligent management platform

By deploying edge computing devices on the carbon intelligent management platform and using random forest algorithms, the problems of data diversity and complexity in carbon emission management have been solved, efficient and accurate carbon emission data fusion and prediction have been achieved, and management has been improved real-time and transparency.

CN118627692BActive Publication Date: 2024-12-06NANJING JIANGXING LIANJIA INTELLIGENT TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411098887.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-12
Publication Date
2024-12-06
Estimated Expiration
2044-08-12

AI Technical Summary

Technical Problem

Currently, carbon emission management faces the difficulties of diversified data sources, different formats, complexity of data collection, processing and analysis, as well as data transparency, credibility and fair transactions. Traditional centralized processing methods are difficult to meet the needs of real-time and privacy protection.

Method used

Deploy edge computing devices on the carbon intelligent management platform, collect multi-source heterogeneous data in real time and perform preprocessing, use random forest algorithms and semi-supervised learning models to fusion data, and combine deep learning to optimize the model structure to achieve high-quality fusion of data and Feature extraction, multi-model fusion and data augmentation technology are used to process unlabeled and sparse data.

Benefits of technology

It significantly shortens the data processing cycle, improves carbon emission monitoring and response speed, reduces network congestion and data transmission costs, enhances data transparency and immutability, and provides efficient and accurate carbon emission forecasting and management support.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118627692B_ABST
    Figure CN118627692B_ABST
Patent Text Reader

Abstract

The present invention relates to a heterogeneous data fusion access method and system based on a carbon intelligent management platform. Through the deployment of edge nodes, data from the power supply side, load side and energy storage side are collected in real time, and after preprocessing, they are uploaded to the cloud according to the data importance and real-time strategy. The cloud uses multi-source heterogeneous data fusion technology based on random forests, combined with offline and incremental semi-supervised models, to improve data integration quality and availability. Through structured feature modeling, the intrinsic correlation of data is deeply explored to build a high-precision prediction model. For different data attributes, sub-models are optimized and integrated to effectively cope with complex data environments. For unlabeled data and sparsity problems, semi-supervised learning and data enhancement strategies are adopted to ensure the integrity and reliability of model training data. The scheme significantly improves the real-time and accuracy of carbon emission management, and provides strong technical support for achieving carbon neutrality goals.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of carbon intelligent management technology, and specifically to a heterogeneous data fusion access method and system based on a carbon intelligent management platform. Background Art

[0002] The key to achieving the goal of carbon neutrality is to establish an efficient and accurate carbon emission management mechanism, which requires real-time monitoring, accurate accounting and scientific prediction of carbon emission data. However, current carbon emission management faces many challenges, including diverse data sources, different formats, and the complexity of data collection, processing and analysis. In addition, how to ensure the transparency and credibility of data and the realization of fair transactions among multiple entities are also urgent issues to be resolved.

[0003] On this basis, blockchain technology, with its unique characteristics of decentralization, data immutability, and high transparency, provides a new solution for carbon emission management. Blockchain technology can build a reliable carbon emission data recording and trading platform to achieve accurate tracking of carbon footprints, fair trading of carbon emission rights, and transparent certification of carbon emission reduction results. Through smart contracts, carbon emission trading rules can be automatically executed, reducing human intervention and improving efficiency and credibility.

[0004] However, current carbon emission data comes from multiple different sources such as the power supply side, load side, and energy storage side. The data types are diverse and the structures are different. Traditional centralized processing methods are difficult to meet the needs of real-time, privacy protection, and data fusion. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a heterogeneous data fusion access method and system based on a carbon intelligent management platform. The specific technical solutions are as follows:

[0006] A heterogeneous data fusion access method based on a carbon intelligent management platform comprises the following steps:

[0007] S1. Deploy edge computing devices at key nodes on the power supply side, load side, and energy storage side to collect multi-source heterogeneous data in real time, pre-process the data, and intelligently schedule data upload strategies based on the real-time nature and importance of the data;

[0008] S2. The cloud receives and integrates the data uploaded by the edge nodes, applies a fusion algorithm based on random forest, combines the offline semi-supervised model with the incremental semi-supervised model, and uses integrated learning technology to enhance the robustness and adaptability of the model, ensuring the consistency and availability of data fusion, and providing high-quality data sets for subsequent analysis;

[0009] S3. Based on the fused data in step S2, a structured model framework is constructed, and the model structure is further optimized through deep learning or machine learning algorithms to accurately extract key features that affect carbon emissions;

[0010] S4, for the structured feature model constructed in step S3, multiple sub-models are obtained by modeling the structured features of single-source data, each model is optimized for different data attributes, quality and distribution, and then multiple models are fused to achieve efficient adaptation to complex data environments;

[0011] S5. To address the problems of unlabeled data and data sparsity encountered in model training, semi-supervised learning strategies and sparse data processing techniques are adopted, combined with data enhancement techniques to increase data diversity and enhance the integrity and reliability of the data set.

[0012] Furthermore, the specific method of step S1 is:

[0013] S11. Deploy edge computing nodes at key locations on the power supply side, load side, and energy storage side. These nodes are equipped with necessary hardware modules, such as data acquisition modules, processing units, and storage devices, and can be directly connected to on-site sensors, controllers, and other devices to achieve local access to data.

[0014] S12. The edge node receives raw data from different sources in real time. Through the preset data processing algorithm or script, the edge computing device immediately performs preliminary cleaning of the data, removes invalid or erroneous data, performs necessary data smoothing, and reduces noise interference;

[0015] S13. Convert raw data in different formats to ensure consistency of data formats;

[0016] S14. Deploy lightweight analytical algorithms or models on edge nodes to analyze data in real time and quickly identify patterns and anomalies in the data;

[0017] S15. For data that needs to be frequently accessed or monitored in real time, the edge computing node implements a data caching strategy to store the most recent or most important data. At the same time, based on the results of edge analysis, the node implements local decision-making, automatically adjusts equipment operating parameters, and responds to abnormal situations;

[0018] S16. Based on the importance and urgency of the data processing and analysis results, the edge computing node intelligently decides which data needs to be uploaded to the cloud in real time, which data needs to be uploaded on demand or periodically, and which data needs to be stored locally.

[0019] Furthermore, the specific method of step S2 is:

[0020] S21. Based on the collected data, classify the data on the power supply side, load side, and energy storage side, and convert the data in different formats into a standardized data format;

[0021] S22. Data quality check and cleaning: Check the integrity of the collected data, remove missing or erroneous data records, identify outliers in the data, and choose to remove, modify or retain them according to the specific situation;

[0022] S23. Extract features meaningful for carbon emission prediction from raw data and convert unstructured or semi-structured data into structured data;

[0023] S24. Use the random forest algorithm to handle the diversity of heterogeneous data, build offline and incremental learning models, adjust the random forest parameters to improve the fusion effect according to data heterogeneity, use data interpolation technology to fill sparse data, and use semi-supervised learning algorithms to process unlabeled data to enhance the generalization ability of the model;

[0024] S25. Data fusion verification and evaluation: Evaluate the quality of the fused data and the predictive ability of the model, analyze the improvement of the fused data compared with a single data source, and tune the fusion algorithm based on the evaluation results.

[0025] Furthermore, the specific method of step S3 is:

[0026] S31. Identify features that are highly correlated with carbon emissions, and create new derived features based on the characteristics of carbon emissions and data features;

[0027] S32. Normalize or standardize continuous numerical features, and use one-hot encoding, label encoding, or ordinal encoding for categorical data so that the model can understand and process such features;

[0028] S33. Evaluate the contribution of each feature to the model prediction results, eliminate irrelevant or redundant features, and quantitatively analyze the strength of the relationship between the features and the target variable;

[0029] S34. Divide the integrated feature data set into a training set, a validation set, and a test set in proportion. According to the characteristics of carbon emission prediction, select a suitable machine learning model for model training, optimize the model performance by adjusting the model hyperparameters, and improve the model prediction accuracy;

[0030] S35. Evaluate the performance of the model on the test set, analyze the importance of the features of the model output, and identify the key parameters for carbon emission prediction.

[0031] Furthermore, the features with a high correlation with the carbon emissions include, for example, time series, energy type, equipment type, and environmental parameters.

[0032] Furthermore, the specific method of step S4 is:

[0033] S41. Sub-model construction and verification: Select or customize multiple models for different dimensions and influencing factors of carbon emissions. Each model is responsible for a specific prediction or analysis task. The integrated and cleaned data is divided into training sets, verification sets, and test sets according to model requirements. Each sub-model is trained independently, and the performance of the sub-model is verified. The model parameters are adjusted according to the verification results to ensure that each sub-model reaches the optimal state.

[0034] S42, using an ensemble learning method to integrate the prediction results of multiple sub-models;

[0035] S43. Schedule the model to run on the edge or cloud based on the amount of real-time data and computing requirements, optimize resource allocation, adjust model parameters in real time based on newly collected data, maintain the timeliness of model predictions, and continuously optimize model performance through feedback mechanisms;

[0036] S43. Evaluate the overall performance of multi-structure model collaboration through historical data backtesting and real-time data tracking, and test the stability of model collaboration under different environments and data conditions.

[0037] Furthermore, the specific method of step S5 is:

[0038] S51. Before processing unlabeled data, identify potential outliers through statistical methods or machine learning algorithms and filter or correct them;

[0039] S52. Use feature selection algorithms to remove features that contribute less to the model, or use dimensionality reduction techniques to reduce the dimensions of the data and reduce sparsity. Convert sparse data into a set of sparse codes, each code corresponding to an atom, thereby revealing the potential structure of the data and enhancing the model's ability to express sparse data;

[0040] S53. Regularly clean the data, remove duplicates, correct erroneous values, and perform integrity checks on the data to ensure the accuracy and completeness of the data;

[0041] S54. Effect evaluation and iterative optimization: Evaluate the performance of the processed data set in model training, including prediction accuracy, model stability, and training efficiency, and adjust the data processing strategy based on the evaluation results.

[0042] Furthermore, the step S53 also includes, for a specific type of unlabeled data, using data synthesis technology to increase data diversity, wherein the data synthesis technology includes one or more combinations of noise injection, rotation, and translation.

[0043] The present invention also discloses a heterogeneous data fusion access system based on a carbon intelligent management platform, including: an edge centralized data acquisition module, a multi-source heterogeneous data fusion module, a structured feature modeling module, a multi-structure model collaboration module, and an unlabeled data and sparse data processing module;

[0044] The edge centralized data acquisition module is used to deploy edge computing devices at key nodes on the power supply side, load side and energy storage side, collect multi-source heterogeneous data in real time, pre-process the data, and intelligently schedule data upload strategies based on the real-time and importance of the data;

[0045] The multi-source heterogeneous data fusion module is used to receive and integrate data uploaded by edge nodes in the cloud, and uses a fusion algorithm based on random forests, combined with an offline semi-supervised model and an incremental semi-supervised model, to enhance the robustness and adaptability of the model through integrated learning technology, ensure the consistency and availability of data fusion, and provide a high-quality data set for subsequent analysis;

[0046] The structured feature modeling module is used to build a structured model framework based on the fused data, further optimize the model structure through deep learning or machine learning algorithms, and accurately extract key features that affect carbon emissions;

[0047] The multi-structure model collaboration module is used to construct a structured feature model, which obtains multiple sub-models through single-source data structured feature modeling. Each model is optimized for different data attributes, quality and distribution, and then multi-model fusion is performed to achieve efficient adaptation to complex data environments;

[0048] The unlabeled data and sparse data processing module is used to address the unlabeled data and data sparsity problems encountered in model training, adopt semi-supervised learning strategies and sparse data processing techniques, and combine data enhancement techniques to increase data diversity and enhance the integrity and reliability of the data set.

[0049] Compared with the prior art, the present invention has the following beneficial effects: 1. By deploying data processing capabilities at the edge nodes, the present invention realizes real-time data collection and preliminary analysis, significantly shortens the data processing cycle, and reduces the latency of traditional cloud processing. This near-end processing mode greatly improves the monitoring and response speed of carbon emissions, which is crucial for real-time regulation of energy consumption, timely detection and response to abnormal emissions, and greatly enhances the real-time and efficiency of carbon management.

[0050] 2. The use of edge computing effectively reduces the burden on cloud servers, avoids the centralized transmission of massive data, and reduces network congestion and data transmission costs. By preprocessing and screening data at the edge and only transmitting key data to the cloud, network resources and cloud computing resources are significantly saved, distributed optimization of data processing is achieved, and the energy efficiency of the overall system is improved.

[0051] The multi-source heterogeneous data fusion method based on random forest effectively solves the challenges of diverse data formats and uneven quality. By integrating offline and incremental learning models, it not only improves the consistency and availability of data, but also improves the prediction accuracy and stability of the model through integrated learning. This fusion technology can deeply analyze carbon emission data from multiple dimensions and levels, providing strong technical support for the accurate prediction and management of carbon emissions.

[0052] 3. Localized data processing through edge computing reduces the distance and number of data transmission in the network, reducing the risk of data leakage. Combined with blockchain and other technologies, it can further improve the transparency and immutability of data, ensure the integrity and authenticity of carbon emission data, provide a reliable data foundation for carbon trading, carbon footprint tracing, etc., and enhance the trust of participants. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 The present invention is a flow chart of the heterogeneous data fusion access method based on the carbon intelligent management platform. DETAILED DESCRIPTION

[0054] The following is combined with Figure 1 The present invention is described in further detail. Example

[0055] Reference Figure 1 This embodiment discloses a heterogeneous data fusion access method based on a carbon intelligent management platform, comprising the following steps:

[0056] S1. Deploy edge computing devices at key nodes on the power supply side, load side, and energy storage side to collect multi-source heterogeneous data in real time, pre-process the data, and intelligently schedule data upload strategies based on the real-time and importance of the data. The specific steps are as follows:

[0057] S11. Edge node deployment and data access: Edge computing nodes are deployed at key locations on the power supply side, load side, and energy storage side. These nodes are equipped with necessary hardware modules, such as data acquisition modules, processing units, and storage devices. They can be directly connected to on-site sensors, controllers, and other equipment to achieve local access to data.

[0058] S12. Real-time data reception and preliminary processing: Edge nodes receive raw data from different sources in real time, which may include time series data, status information, alarm signals, etc. Through the preset data processing algorithm or script, the edge computing device immediately performs preliminary cleaning of the data, removes invalid or erroneous data, such as removing outliers that exceed the normal range, and performs necessary data smoothing to reduce noise interference.

[0059] S13, Data format standardization and conversion: Implement data format standardization on the edge side, convert raw data in different formats (such as CSV, JSON, XML, etc.), ensure data format consistency, and facilitate subsequent integration and analysis. This step usually includes defining a unified data model and protocol so that data from different devices can be seamlessly connected.

[0060] S14. Lightweight data processing and analysis: Deploy lightweight analysis algorithms or models at edge nodes, such as simple statistical analysis, trend forecasting, etc., to analyze data instantly and quickly identify patterns and anomalies in the data without waiting for data to be uploaded to the cloud for processing. This helps with rapid response and local decision-making, and reduces reliance on central servers.

[0061] S15. Data caching and local decision-making: For data that needs to be frequently accessed or monitored in real time, edge computing nodes can implement data caching strategies to store the most recent or most important data for quick access and instant feedback. At the same time, based on the results of edge analysis, nodes can implement local decisions, such as automatically adjusting equipment operating parameters and responding to abnormal situations, without waiting for cloud instructions, significantly reducing response delays.

[0062] S16. Data transmission strategy optimization: According to the importance and urgency of data processing and analysis results, edge computing nodes intelligently decide which data needs to be uploaded to the cloud in real time and which data can be uploaded on demand or regularly. In this way, unnecessary data transmission is reduced, network bandwidth usage is optimized, and the processing burden of the central server is reduced.

[0063] Through step S1, edge computing not only reduces the delay of data processing and improves the efficiency of data processing, but also ensures the security and privacy protection of data, providing a solid foundation for carbon intelligent management.

[0064] S2. The cloud receives and integrates the data uploaded by the edge nodes, uses a fusion algorithm based on random forest, combines the offline semi-supervised model with the incremental semi-supervised model, and uses integrated learning technology to enhance the robustness and adaptability of the model, ensure the consistency and availability of data fusion, and provide a high-quality data set for subsequent analysis. The specific steps are as follows:

[0065] S21. Data source integration and standardization:

[0066] Based on the data collected in the first step, the data on the power supply side, load side, and energy storage side are classified in detail to clarify the source, format, and meaning of each type of data. Data format standardization: For data in different formats, such as CSV, JSON, XML, etc., a conversion interface is developed to convert them into a standardized data format for subsequent processing.

[0067] S22. Data quality check and cleaning:

[0068] Data integrity check: Perform integrity check on the collected data and eliminate missing or erroneous data records.

[0069] Outlier detection and processing: Apply statistical methods or machine learning algorithms to identify outliers in the data and choose to eliminate, modify or retain them based on the specific situation.

[0070] Data consistency check: ensure the consistency of data from different sources at the same time point and resolve data conflicts.

[0071] S23, feature extraction and structured processing:

[0072] Feature selection: Based on business needs, extract features that are meaningful for carbon emission prediction from the original data, such as time, location, energy consumption, equipment status, etc.

[0073] Structured transformation: Convert unstructured or semi-structured data into structured data to facilitate model processing. For example, text log data is converted into numerical features through natural language processing technology.

[0074] S24. Application of heterogeneous data fusion algorithm

[0075] Fusion strategy based on random forest: Use the strong adaptability and robustness of the random forest algorithm to handle the diversity of heterogeneous data. Build offline and incremental learning models to adapt to real-time changes in data.

[0076] Data fusion algorithm optimization: In view of data heterogeneity, random forest parameters such as the number of trees and node splitting criteria are adjusted to improve the fusion effect.

[0077] Sparse data and unlabeled data processing: Use data interpolation technology to fill in sparse data, and use semi-supervised learning algorithms to process unlabeled data to enhance the generalization ability of the model.

[0078] S25. Data fusion verification and evaluation

[0079] Model validation: Use methods such as cross-validation and A / B testing to evaluate the quality of the fused data and the predictive ability of the model.

[0080] Result analysis: Analyze the improvement of fused data compared with single data source, including accuracy, recall, F1 score and other indicators.

[0081] Continuous optimization: According to the evaluation results, the fusion algorithm is tuned, such as adjusting the fusion weight, optimizing feature selection, etc., to achieve the best fusion effect.

[0082] Through step S2, efficient integration of multi-source heterogeneous data is achieved, providing an accurate and unified data foundation for the carbon intelligent management platform and providing strong support for carbon emission prediction and management decisions.

[0083] S3. Based on the fused data in step S2, a structured model framework is constructed, and the model structure is further optimized through deep learning or machine learning algorithms to accurately extract key features that affect carbon emissions. The specific steps are as follows;

[0084] S31. Based on business understanding and data exploration, identify features that are highly correlated with carbon emissions, such as time series (hours, days, months), energy types, equipment types, environmental parameters, etc. Create new derivative features based on the characteristics of carbon emissions and data features, such as time series lag features, holiday identifiers, temperature change rates, etc., to enhance the model's ability to capture complex relationships.

[0085] S32. Normalize or standardize continuous numerical features, such as z-score standardization and min-max scaling, to ensure fairness between features of different scales. Use one-hot encoding, label encoding, or ordinal encoding for categorical data so that the model can understand and process such features.

[0086] S33. Use feature importance scoring: Use algorithms such as random forest and gradient boosting tree to evaluate the contribution of each feature to the model prediction results and eliminate irrelevant or redundant features.

[0087] Methods based on statistical tests: Statistical methods such as chi-square test and ANOVA are used to quantitatively analyze the strength of the relationship between features and target variables.

[0088] S34. Construction of structured feature model:

[0089] Data segmentation: Divide the integrated feature data set into training set, validation set and test set in proportion to ensure the generalization ability of the model.

[0090] Model selection and training: According to the characteristics of carbon emission prediction, select appropriate machine learning models, such as linear regression, random forest, neural network, etc., for model training.

[0091] Model optimization: By adjusting model hyperparameters and using methods such as cross-validation, we can optimize model performance, reduce the risk of overfitting, and improve model prediction accuracy.

[0092] S35. Feature model evaluation:

[0093] Performance indicator calculation: Use indicators such as R² score, mean square error (MSE), and root mean square error (RMSE) to evaluate the performance of the model on the test set.

[0094] Feature interpretation: Analyze the importance of the features output by the model to understand which features are most critical to carbon emission prediction and provide a basis for subsequent strategy formulation.

[0095] Through step S3, the construction process from raw data to structured feature model is completed, providing a solid feature foundation for carbon emission prediction and ensuring that the model can accurately and efficiently analyze and predict carbon emission trends.

[0096] S4: For the structured feature model constructed in step S3, multiple sub-models are obtained by modeling the structured features of single-source data. Each model is optimized for different data attributes, quality and distribution, and then multiple models are fused to achieve efficient adaptation to complex data environments. The specific steps are as follows:

[0097] S41. Select or customize multiple models for different dimensions and influencing factors of carbon emissions, such as time series models (ARIMA, LSTM), machine learning models (random forest, XGBoost) and deep learning models (convolutional neural network, Transformer), each model focusing on specific prediction or analysis tasks;

[0098] The integrated and cleaned data is divided into training set, validation set and test set according to the model requirements, and each sub-model is trained independently;

[0099] The performance of the sub-model is verified through cross-validation, grid search and other methods, and the model parameters are adjusted according to the verification results to ensure that each sub-model reaches the optimal state.

[0100] S42. During the sub-model training process, explore the complementarity of features between different models, find the most effective feature set through feature combination experiments, and improve the predictive ability of the model.

[0101] Ensemble learning methods (such as bagging, boosting, and stacking) are used to combine the prediction results of multiple sub-models, such as through weighted averaging, voting, or gradient boosting, to improve the accuracy and stability of the overall prediction.

[0102] S43. Design a model scheduling mechanism based on cloud-edge collaboration. According to the real-time data volume and computing requirements, the intelligent scheduling model runs on the edge or cloud to optimize resource allocation.

[0103] Real-time update and feedback: Establish a model update mechanism to adjust model parameters in real time based on newly collected data, maintain the timeliness of model predictions, and continuously optimize model performance through feedback mechanisms.

[0104] S44. Evaluate the overall performance of multi-structure model collaboration through historical data backtesting and real-time data tracking, including prediction accuracy, response speed, resource utilization, etc.

[0105] Stability and robustness analysis: Test the stability of the model's collaborative work under different environments and data conditions to ensure the model's predictive robustness under extreme conditions.

[0106] Through step S4, efficient collaboration of multi-structure models is achieved, which not only improves the accuracy of carbon emission prediction, but also optimizes the use of computing resources, providing a more accurate and efficient decision support system for the carbon intelligent management platform.

[0107] S5. To address the unlabeled data and data sparsity issues encountered during model training, we adopt semi-supervised learning strategies and sparse data processing techniques, combined with data enhancement techniques to increase data diversity and enhance the integrity and reliability of the data set. The specific steps are as follows:

[0108] S51, Unlabeled data processing:

[0109] Semi-supervised learning strategy: Using semi-supervised learning methods such as self-learning or collaborative training, a small amount of labeled data is used to guide the learning process of a large amount of unlabeled data, and the model's ability to understand unlabeled data is gradually improved through data self-propagation and model iteration.

[0110] Outlier detection: Before processing unlabeled data, potential outliers are identified through statistical methods or machine learning algorithms, and then filtered or corrected to prevent these data points from interfering with model learning.

[0111] Active learning: During data processing, the most representative unlabeled samples are actively selected for manual labeling based on the uncertainty or prediction error of the model to gradually enrich the labeled data set.

[0112] S52. Sparse data processing:

[0113] For sparse data, neighborhood-based mean or median filling, model-based prediction filling (such as using features of historical data or similar data for prediction), and matrix decomposition-based filling methods such as singular value decomposition (SVD) or non-negative matrix factorization (NMF) can be used.

[0114] Feature selection and dimensionality reduction: Before processing, use feature selection algorithms (such as recursive feature elimination, model-based feature importance) to remove features that contribute less to the model, or use dimensionality reduction techniques (PCA, t-SNE) to reduce the dimension of the data and reduce sparsity.

[0115] Sparse representation learning: Through sparse coding, dictionary learning and other technologies, sparse data is converted into a set of sparse codes, each code corresponding to an atom, thereby revealing the potential structure of the data and enhancing the model's ability to express sparse data.

[0116] S53. Improved data quality and integrity:

[0117] Data cleaning and verification: Perform data cleaning regularly to remove duplicates, correct erroneous values, and perform integrity verification on the data to ensure its accuracy and completeness.

[0118] Data enhancement: For specific types of unlabeled data, data synthesis techniques (such as noise injection, rotation, translation and other transformations) can be used to increase data diversity, especially in machine learning model training, which helps improve the model's generalization ability.

[0119] S54. Effect evaluation and iterative optimization

[0120] Through cross-validation, A / B testing and other methods, the performance of the processed dataset in model training is evaluated, including prediction accuracy, model stability and training efficiency.

[0121] According to the evaluation results, the data processing strategy is adjusted, such as adjusting the filling strategy, optimizing feature selection, etc., to form a closed-loop feedback mechanism and continuously improve the data processing effect.

[0122] Through step S5, unlabeled data and sparse data are effectively processed, which enhances the integrity and quality of the data set, provides more reliable data input for the carbon intelligent management platform, and further improves the accuracy and reliability of carbon emission forecasting and management decisions. Example

[0123] This embodiment discloses a heterogeneous data fusion access system based on a carbon intelligent management platform, including: an edge centralized data acquisition module, a multi-source heterogeneous data fusion module, a structured feature modeling module, a multi-structure model collaboration module, and an unlabeled data and sparse data processing module;

[0124] The edge centralized data acquisition module is used to deploy edge computing devices at key nodes on the power supply side, load side and energy storage side, collect multi-source heterogeneous data in real time, pre-process the data, and intelligently schedule data upload strategies based on the real-time and importance of the data;

[0125] The multi-source heterogeneous data fusion module is used to receive and integrate data uploaded by edge nodes in the cloud, and uses a fusion algorithm based on random forests, combined with an offline semi-supervised model and an incremental semi-supervised model, to enhance the robustness and adaptability of the model through integrated learning technology, ensure the consistency and availability of data fusion, and provide a high-quality data set for subsequent analysis;

[0126] The structured feature modeling module is used to build a structured model framework based on the fused data, further optimize the model structure through deep learning or machine learning algorithms, and accurately extract key features that affect carbon emissions;

[0127] The multi-structure model collaboration module is used to construct a structured feature model, which obtains multiple sub-models through single-source data structured feature modeling. Each model is optimized for different data attributes, quality and distribution, and then multi-model fusion is performed to achieve efficient adaptation to complex data environments;

[0128] The unlabeled data and sparse data processing module is used to address the unlabeled data and data sparsity problems encountered in model training, adopt semi-supervised learning strategies and sparse data processing techniques, and combine data enhancement techniques to increase data diversity and enhance the integrity and reliability of the data set.

[0129] The above are all preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A heterogeneous data fusion access method based on a carbon intelligent management platform is characterized by: The following steps are involved: S1. Deploy edge computing devices at key nodes on the power supply side, load side, and energy storage side to collect multi-source heterogeneous data in real time, pre-process the data, and intelligently schedule data upload strategies based on the real-time nature and importance of the data; S2. The cloud receives data uploaded by edge nodes, uses a fusion algorithm based on random forest, combines offline semi-supervised models with incremental semi-supervised models to process the received multi-source heterogeneous data, and enhances the robustness and adaptability of the model through integrated learning technology; S3. Based on the data fused in step S2, a structured model framework is constructed, and the model structure is further optimized through deep learning or machine learning algorithms to accurately extract key features that affect carbon emissions. The specific method of step S3 is: S31. Identify features that are highly correlated with carbon emissions, and create new derived features based on the characteristics of carbon emissions and data features; S32. Normalize or standardize continuous numerical features, and use one-hot encoding, label encoding, or ordinal encoding for categorical data so that the model can understand and process such features; S33. Evaluate the contribution of each feature to the model prediction results, eliminate irrelevant or redundant features, and quantitatively analyze the strength of the relationship between the features and the target variable; S34. Divide the integrated feature data set into a training set, a validation set, and a test set in proportion. According to the characteristics of carbon emission prediction, select a suitable machine learning model for model training, optimize the model performance by adjusting the model hyperparameters, and improve the model prediction accuracy; S35. Evaluate the performance of the model on the test set, analyze the importance of the features output by the model, and identify the key parameters for carbon emission prediction; S4, for the structured feature model constructed in step S3, multiple sub-models are obtained by modeling the structured features of single-source data, each model is optimized for different data attributes, quality and distribution, and then multiple models are fused; the specific method of step S4 is: S41. Sub-model construction and verification: Select or customize multiple models for different dimensions and influencing factors of carbon emissions. Each model is responsible for the corresponding prediction or analysis task. The integrated and cleaned data is divided into training set, verification set and test set according to the model requirements. Each sub-model is trained independently, and the performance of the sub-model is verified. The model parameters are adjusted according to the verification results to ensure that each sub-model reaches the optimal state. S42, using an ensemble learning method to integrate the prediction results of multiple sub-models; S43. Schedule the model to run on the edge or cloud based on the amount of real-time data and computing requirements, optimize resource allocation, adjust model parameters in real time based on newly collected data, maintain the timeliness of model predictions, and continuously optimize model performance through feedback mechanisms; S43. Evaluate the overall performance of multi-structure model collaboration through historical data backtesting and real-time data tracking, and test the stability of model collaboration under different environments and data conditions; S5. To address the problems of unlabeled data and data sparsity encountered in model training, a semi-supervised learning strategy and sparse data processing are adopted, combined with data enhancement technology to increase data diversity and enhance the integrity and reliability of the data set.

2. The heterogeneous data fusion access method based on the carbon intelligent management platform according to claim 1 is characterized in that: The specific method of step S1 is: S11. Deploy edge computing nodes at key locations on the power supply side, load side, and energy storage side. These nodes are equipped with hardware modules, including data acquisition modules, processing units, and storage devices, and can be directly connected to field sensors and controller devices to achieve local access to data. S12. The edge node receives raw data from different sources in real time. Through the preset data processing algorithm or script, the edge computing device immediately performs preliminary cleaning of the data, removes invalid or erroneous data, performs data smoothing, and reduces noise interference; S13. Convert raw data in different formats to ensure consistency of data formats; S14. Deploy lightweight analytical algorithms or models on edge nodes to analyze data in real time and quickly identify patterns and anomalies in the data; S15. For data that needs to be frequently accessed or monitored in real time, the edge computing node implements a data caching strategy to store the most recent or most important data. At the same time, based on the results of edge analysis, the node implements local decision-making, automatically adjusts equipment operating parameters, and responds to abnormal situations; S16. Based on the importance and urgency of the data processing and analysis results, the edge computing node intelligently decides which data needs to be uploaded to the cloud in real time, which data needs to be uploaded on demand or periodically, and which data needs to be stored locally.

3. The heterogeneous data fusion access method based on the carbon intelligent management platform according to claim 1 is characterized in that: The specific method of step S2 is: S21. Based on the collected data, classify the data on the power supply side, load side, and energy storage side, and convert the data in different formats into a standardized data format; S22. Data quality check and cleaning: Check the integrity of the collected data, remove missing or erroneous data records, identify outliers in the data, and choose to remove, modify or retain them according to the specific situation; S23. Extract features meaningful for carbon emission prediction from raw data and convert unstructured or semi-structured data into structured data; S24. Use the random forest algorithm to handle the diversity of heterogeneous data, build offline and incremental learning models, adjust the random forest parameters to improve the fusion effect according to data heterogeneity, use data interpolation technology to fill sparse data, and use semi-supervised learning algorithms to process unlabeled data to enhance the generalization ability of the model; S25. Data fusion verification and evaluation: Evaluate the quality of the fused data and the predictive ability of the model, analyze the improvement of the fused data compared with a single data source, and tune the fusion algorithm based on the evaluation results.

4. The heterogeneous data fusion access method based on the carbon intelligent management platform according to claim 1 is characterized in that: The features with a high correlation with carbon emissions include time series, energy type, equipment type, and environmental parameters.

5. The heterogeneous data fusion access method based on the carbon intelligent management platform according to claim 1 is characterized in that: The specific method of step S5 is: S51. Before processing unlabeled data, identify potential outliers through statistical methods or machine learning algorithms and filter or correct them; S52. Use feature selection algorithms to remove features that contribute less to the model, or use dimensionality reduction techniques to reduce the dimension of the data, reduce sparsity, and convert sparse data into a set of sparse codes, each code corresponding to an atom, thereby revealing the potential structure of the data and enhancing the model's ability to express sparse data; S53. Regularly clean the data, remove duplicates, correct erroneous values, and perform integrity checks on the data to ensure the accuracy and completeness of the data; S54. Effect evaluation and iterative optimization: Evaluate the performance of the processed data set in model training, including prediction accuracy, model stability, and training efficiency, and adjust the data processing strategy based on the evaluation results.

6. The heterogeneous data fusion access method based on the carbon intelligent management platform according to claim 5 is characterized by: The step S53 also includes using data synthesis technology to increase data diversity for a specific type of unlabeled data, and the data synthesis technology includes one or more combinations of noise injection, rotation, and translation.

7. A heterogeneous data fusion access system for a carbon intelligent management platform based on the method described in any one of claims 1 to 6, characterized in that: include: Edge centralized data acquisition module, multi-source heterogeneous data fusion module, structured feature modeling module, multi-structure model collaboration module, unlabeled data and sparse data processing module; The edge centralized data acquisition module is used to deploy edge computing devices at key nodes on the power supply side, load side and energy storage side, collect multi-source heterogeneous data in real time, pre-process the data, and intelligently schedule data upload strategies based on the real-time and importance of the data; The multi-source heterogeneous data fusion module is used to receive and integrate data uploaded by edge nodes in the cloud, and uses a fusion algorithm based on random forests, combined with an offline semi-supervised model and an incremental semi-supervised model, to enhance the robustness and adaptability of the model through integrated learning technology, ensure the consistency and availability of data fusion, and provide a high-quality data set for subsequent analysis; The structured feature modeling module is used to build a structured model framework based on the fused data, further optimize the model structure through deep learning or machine learning algorithms, and accurately extract key features that affect carbon emissions; The multi-structure model collaboration module is used to construct a structured feature model, which obtains multiple sub-models through single-source data structured feature modeling. Each model is optimized for different data attributes, quality and distribution, and then multi-model fusion is performed to achieve efficient adaptation to complex data environments; The unlabeled data and sparse data processing module is used to address the unlabeled data and data sparsity problems encountered in model training, adopt semi-supervised learning strategies and sparse data processing techniques, and combine data enhancement techniques to increase data diversity and enhance the integrity and reliability of the data set.