Isomerization energy data processing method, device and storage medium

Abstract

The application discloses a heterogeneous energy data processing method and device and a storage medium, and belongs to the technical field of multi-modal data processing. The application realizes real-time access to multi-source heterogeneous energy flow data from kafka through a preset Spark Streaming, generates regular data streams according to different types of data sources, divides the regular data streams into micro batches through the Spark Streaming, extracts corresponding features from the divided data streams, obtains a feature vector stream, stores the feature vector stream into a corresponding time sequence database, and enables relevant personnel to call the feature vector stream from the time sequence database, so that the real-time data accessed from kafka is processed in the form of micro batch segmentation through the Spark Streaming, and a corresponding feature vector stream is generated, thereby realizing efficient real-time processing of heterogeneous energy data and guaranteeing accurate and timely data support for real-time monitoring, analysis and decision-making in the energy field.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a heterogeneous energy data processing method, device and storage medium. Background Technology

[0002] In recent years, with the widespread application of technologies such as smart sensors and the Internet of Things in the energy sector, the scale of heterogeneous energy data has grown exponentially, and the data types have become increasingly complex and diverse, covering structured equipment operating parameters, semi-structured log files, and unstructured text reports.

[0003] However, real-time processing of heterogeneous energy data faces challenges in practical applications: heterogeneous energy data comes from a wide range of sources and has diverse formats, making it difficult to stably and efficiently access various types of energy stream data from different Kafka clusters. Furthermore, multi-source heterogeneous energy data has complex structures, including structured, semi-structured, and unstructured data. Existing cleaning methods cannot effectively process data with different structures, and time-series alignment is difficult.

[0004] Therefore, it is currently impossible to effectively process heterogeneous energy data in real time, which results in the inability to provide accurate and timely data support for real-time monitoring, analysis, and decision-making in the energy sector. Summary of the Invention

[0005] The main objective of this application is to provide a heterogeneous energy data processing method, device, and storage medium, aiming to solve the technical problem that heterogeneous energy data cannot be effectively processed in real time, thus failing to provide accurate and timely data support for real-time monitoring, analysis, and decision-making in the energy field.

[0006] To achieve the above objectives, this application provides a heterogeneous energy data processing method, which includes the following steps:

[0007] The system uses Spark Streaming to receive heterogeneous energy stream data from multiple sources in real time from Kafka and generates regular data streams based on the different data source types.

[0008] The Spark Streaming method is used to perform micro-batch segmentation on the regular data stream, and corresponding features are extracted from the segmented data stream to obtain a feature vector stream.

[0009] The feature vector stream is stored in the corresponding time series database so that relevant personnel can retrieve the feature vector stream from the time series database.

[0010] In one embodiment, the step of real-time access to heterogeneous energy stream data from multiple sources via a preset Spark Streaming method from Kafka, and generating a regularized data stream according to the different data source types, includes:

[0011] The system uses a pre-defined Spark Streaming mechanism to receive heterogeneous energy stream data from multiple sources in real time from Kafka, and uses this data to generate an initial data stream.

[0012] Based on the different data source types, the initial data stream is subjected to distributed cleaning, and the timestamp field of the data provided by each data source in the heterogeneous energy stream data is extracted;

[0013] Based on the timestamp field, determine the common factor of the time intervals between the data sources when providing data, and set the corresponding event time window according to the common factor and the accuracy requirements of the current data processing;

[0014] Based on the event time window, the initial data stream after distributed cleaning is aligned with multiple sources in time sequence to obtain a regularized data stream.

[0015] In one embodiment, the step of performing distributed cleaning on the initial data stream according to different data source types includes:

[0016] If the initial data stream is structured data, then obtain the standard parameter range corresponding to the structured data, and remove data in the structured data that falls outside the standard parameter range;

[0017] If the initial data stream is semi-structured or unstructured data, the matching degree between the text corresponding to each data in the initial data stream and the preset regular expression is calculated, and data whose matching degree does not meet the preset requirements is removed.

[0018] In one embodiment, the step of performing micro-batch segmentation on the regularized data stream using Spark Streaming includes:

[0019] Obtain the generation rate of the heterogeneous energy stream data determined by the Kafka endpoint, and determine the resource usage of the local system;

[0020] Based on the generation rate, resource usage, and preset dynamic coefficients, determine the micro-batch size that best suits the local system's maximum operating efficiency under the current conditions;

[0021] The regularized data stream is divided into micro-batches using the Spark Streaming and micro-batch specifications.

[0022] In one embodiment, the step of performing micro-batch segmentation on the regularized data stream using the Spark Streaming and the micro-batch specification includes:

[0023] The regularized data stream is pre-segmented using Spark Streaming and the micro-batch specification.

[0024] The pre-segmentation results are evaluated based on the multi-dimensional partitioning requirements corresponding to the data characteristics of the heterogeneous energy flow data.

[0025] If the evaluation results do not meet the multi-dimensional segmentation requirements, the micro-batch specifications are optimized, and the regularized data stream is segmented into micro-batches based on the optimized results.

[0026] In one embodiment, the step of evaluating the pre-segmentation results based on the multi-dimensional partitioning requirements corresponding to the data characteristics of the heterogeneous energy flow data includes:

[0027] The multi-dimensional segmentation requirements include at least the requirements for data continuity, data contextual coherence, and data integrity.

[0028] Obtain the application requirements proposed by the user when the current data is micro-batch segmented, and assign corresponding requirement weights to the data continuity requirement, the data context coherence requirement, and the data integrity requirement based on the application requirements;

[0029] The pre-segmentation results are evaluated based on the required weights, the data continuity requirements, the data context coherence requirements, and the data integrity requirements.

[0030] If the evaluation results do not meet the multi-dimensional segmentation requirements, the step of optimizing the micro-batch specification includes:

[0031] If the evaluation results do not meet the multi-dimensional segmentation requirements, the data splitting positions corresponding to the micro-batch specifications will be adjusted according to the requirement weights, data continuity requirements, data context coherence requirements, and data integrity requirements, so as to control the data length corresponding to each micro-batch specification to meet the multi-dimensional segmentation requirements.

[0032] In one embodiment, the step of extracting corresponding features from the segmented data stream to obtain a feature vector stream includes:

[0033] If the data in the segmented data stream is structured data, then based on the current data processing requirements, key data dimensions are selected as features to obtain the first processing result;

[0034] If the data in the segmented data stream is semi-structured or unstructured, then the TF-IDF algorithm is used to determine the keyword set from the data in the segmented data stream, and the weight of each keyword in the keyword set is determined. The weight is then used as a feature to obtain the second processing result.

[0035] The first processing result and the second processing result are combined into a feature vector, and multiple feature vectors are output in time sequence to form a feature vector stream.

[0036] In one embodiment, the step of storing the feature vector stream to a corresponding time-series database includes:

[0037] Determine the load and available resources of the local system, and determine the priority of the current data task;

[0038] Configure the appropriate number of executors based on the load, priority, and amount of idle resources;

[0039] Determine the access frequency and data characteristics of the heterogeneous energy flow data;

[0040] Configure the corresponding storage level based on the access frequency and the data characteristics;

[0041] The feature vector stream is stored in the corresponding time-series database according to the configured executor and storage level.

[0042] Furthermore, to achieve the above objectives, this application also provides a heterogeneous energy data processing device, which includes:

[0043] The data generation module is used to receive heterogeneous energy stream data from multiple sources from Kafka in real time through the preset Spark Streaming, and generate regular data streams according to the different data source types.

[0044] The data processing module is used to perform micro-batch segmentation on the regular data stream through Spark Streaming, and extract corresponding features from the segmented data stream to obtain a feature vector stream;

[0045] The data storage module is used to store the feature vector stream to a corresponding time series database so that relevant personnel can retrieve the feature vector stream from the time series database.

[0046] In addition, to achieve the above objectives, this application also provides a heterogeneous energy data processing device, which includes: a memory, a processor, and a heterogeneous energy data processing program stored in the memory and executable on the processor. The heterogeneous energy data processing program is configured to implement the steps of the heterogeneous energy data processing method described above.

[0047] In addition, to achieve the above objectives, this application also provides a computer-readable storage medium storing a heterogeneous energy data processing program, which, when executed by a processor, implements the steps of the heterogeneous energy data processing method described above.

[0048] The one or more technical solutions proposed in this application have at least the following technical effects: Real-time access to heterogeneous energy stream data from multiple sources via a preset Spark Streaming method, generating a regularized data stream based on the different data source types; Spark Streaming performs micro-batch segmentation on the regularized data stream, extracting corresponding features from the segmented data streams to obtain a feature vector stream; The feature vector stream is stored in a corresponding time-series database for relevant personnel to retrieve from the time-series database. Specifically, real-time access to heterogeneous energy data from multiple sources via a preset Spark Streaming method, micro-batch segmentation of the processed data stream using Spark Streaming, extraction of corresponding features from each segmented data stream, generation of a specific feature vector stream, and storage of this stream ensures that data accessed from Kafka is stored in the form of a regularized data stream and a micro-batch processed feature vector stream according to the specific formats of different data sources. This improves the real-time processing effect of heterogeneous energy data from multiple sources, thereby ensuring accurate and timely data support for real-time monitoring, analysis, and decision-making. Attached Figure Description

[0049] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart illustrating an embodiment of the heterogeneous energy data processing method of this application.

[0052] Figure 2 This is a flowchart illustrating Embodiment 2 of the heterogeneous energy data processing method of this application;

[0053] Figure 3 This is a flowchart illustrating Embodiment 3 of the heterogeneous energy data processing method of this application;

[0054] Figure 4 This is a schematic diagram of the module structure of the heterogeneous energy data processing device according to an embodiment of this application;

[0055] Figure 5 This is a schematic diagram of the hardware operating environment involved in the heterogeneous energy data processing method in this application embodiment.

[0056] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0057] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0058] It should be noted that in recent years, with the widespread application of technologies such as smart sensors and the Internet of Things in the energy sector, the scale of heterogeneous energy data has grown exponentially, and the data types have become increasingly complex and diverse, encompassing structured equipment operating parameters, semi-structured log files, and unstructured text reports. At the same time, energy systems are increasingly demanding real-time, accurate, and comprehensive data processing capabilities to enable timely and informed decision-making and ensure the efficient and stable operation of energy systems.

[0059] However, due to the inherent characteristics of heterogeneous energy data, the limitations of existing processing technologies, and various system architecture issues, real-time processing of heterogeneous energy data faces numerous challenges in practical applications: (1) Difficulty in accessing heterogeneous energy data: Heterogeneous energy data comes from a wide range of sources and has diverse formats. Traditional methods struggle to stably and efficiently access various energy stream data from different Kafka clusters, leading to interruptions or data loss during data acquisition, which affects subsequent processing and analysis. (2) Difficulty in cleaning and aligning heterogeneous energy data: Multi-source heterogeneous energy data has complex structures, including structured, semi-structured, and unstructured data. Existing cleaning methods cannot effectively process data with different structures, and time-series alignment is difficult, resulting in low data quality and difficulty in meeting the needs of subsequent accurate analysis. (3) Inefficient use of system resources: In the process of heterogeneous energy data processing, traditional methods use fixed resource allocation strategies, such as a fixed number of Executors and a uniform memory storage level, which cannot be dynamically adjusted according to the characteristics of data processing tasks (such as task priority, data size, and complexity), resulting in resource waste or low processing efficiency.

[0060] Based on the above problems, this application proposes a heterogeneous energy data processing method based on Spark Streaming (a real-time stream processing framework). The aim is to break through the limitations of traditional stream and batch processing separation, realize efficient real-time processing and in-depth analysis of heterogeneous energy data, and improve the level of intelligence in energy management.

[0061] Reference Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the heterogeneous energy data processing method of this application.

[0062] In the first embodiment, the heterogeneous energy data processing method includes the following steps:

[0063] S10 uses the preset Spark Streaming to receive heterogeneous energy stream data from multiple sources in real time from Kafka, and generates regular data streams according to the different data source types.

[0064] Understandably, Kafka is a distributed stream processing platform designed to efficiently process large-scale real-time data streams, supporting high-throughput and low-latency message delivery. It is widely used in building real-time data pipelines and streaming applications. In this embodiment, Kafka mainly needs to push energy stream data of different structure types and different data sources to Spark Streaming in real time.

[0065] The heterogeneous energy stream data pushed by Kafka includes at least energy data with different structural types, such as structured data, semi-structured data, and unstructured data. In addition, Kafka will generate real-time data streams based on the data generation time.

[0066] Spark Streaming is a core component specifically designed for processing real-time data streams. It primarily employs a micro-batch processing architecture, dividing a continuous data stream into a series of small batches and leveraging Spark's distributed computing engine for efficient processing. In this embodiment, Spark Streaming is mainly used to receive heterogeneous energy stream data from Kafka and perform corresponding processing on this stream data, including generating a regular data stream from the received stream data, micro-batch splitting of the data stream, and vector extraction.

[0067] Understandably, after Spark Streaming receives heterogeneous energy stream data from Kafka, the data is quite complex, with differences in data structure, content, and timestamps from different data sources. Therefore, Spark Streaming is needed to perform preliminary data preprocessing on this heterogeneous energy stream data to transform it into a regular data stream, including data cleaning, data alignment, and data format unification.

[0068] S20, using Spark Streaming, the regular data stream is divided into micro-batch segments, and corresponding features are extracted from the segmented data stream to obtain a feature vector stream;

[0069] Understandably, Spark Streaming uses a micro-batch processing architecture to divide a continuous data stream into a series of small batches and uses Spark's distributed computing engine for efficient processing. In this embodiment, this feature of Spark Streaming is also used to divide the preprocessed regular data stream into micro-batches, extract the corresponding features from the divided data stream, and form a feature vector stream based on the features.

[0070] It should be noted that since the regularized data stream includes data of various different structural types and data sources, different types and contents of data will be extracted when extracting features. It is necessary to take into account the above data situation, use different methods to extract features, and fuse the different features into a feature vector stream.

[0071] S30, the feature vector stream is stored in the corresponding time series database so that relevant personnel can retrieve the feature vector stream from the time series database.

[0072] It should be noted that writing the feature vector stream into the corresponding time series database (since the data contains timestamps, indicating its corresponding time series characteristics, it needs to be written into the time series database accordingly) involves writing the data points of the feature vector stream into a time series database suitable for energy data storage. This is done by converting the data into measurement names, labels, timestamps, and field information according to the required format before writing. This ensures the integrity of the data storage and allows relevant personnel to retrieve the corresponding feature vector stream from the time series database during subsequent data use, providing accurate and timely data support for real-time monitoring, analysis, and decision-making.

[0073] At the same time, it strictly follows the RESTful architecture principles and JDBC specifications, encapsulates standardized RESTful / JDBC interfaces, realizes long-term stable storage of energy data, and provides a unified and standardized data access method for various external systems, such as energy management platforms and data analysis tools, thereby improving the universality and ease of use of data and promoting the circulation and application of energy data in the entire energy ecosystem.

[0074] In this embodiment, the step of storing the feature vector stream to the corresponding time-series database includes:

[0075] Determine the load and available resources of the local system, and determine the priority of the current data task;

[0076] Configure the appropriate number of executors based on the load, priority, and amount of idle resources;

[0077] Determine the access frequency and data characteristics of the heterogeneous energy flow data;

[0078] Configure the corresponding storage level based on the access frequency and the data characteristics;

[0079] The feature vector stream is stored in the corresponding time-series database according to the configured executor and storage level.

[0080] Understandably, when storing feature vector streams to the corresponding time-series database, it is necessary to consider the current local system load and available resources. Based on these factors, resources can be rationally allocated to perform the feature vector stream storage. Furthermore, this implementation also incorporates additional storage levels for heterogeneous energy stream data, avoiding the use of fixed resource allocation strategies, such as a fixed number of Executors and a uniform memory storage level, common in traditional methods. In this embodiment, the number of Executors is intelligently and dynamically adjusted based on real-time monitoring of system load, available resources, data processing task priorities, and data size and complexity. Simultaneously, the memory storage level is rationally configured according to the access frequency and importance of the energy data, achieving efficient utilization of system resources, reducing system operating costs, and ensuring stable and efficient system operation under different workloads.

[0081] Among them, the priority of data tasks refers to whether the data content corresponding to the current data task is urgently needed by the current user or whether the importance and timeliness requirements of the data content are high.

[0082] The storage level of the feature vector stream is mainly determined by the access frequency and data characteristics of the heterogeneous energy stream data. If the access frequency of the stream data is high (the frequency at which users need to use the data is high), the storage level of the feature vector stream corresponding to the heterogeneous energy stream data will be higher. The data characteristics refer to indicators such as the real-time performance, importance and security of the data. If the indicators are higher, the storage level of the corresponding feature vector stream will be higher.

[0083] Specifically, when dynamically adjusting the number of Executors to achieve the desired configuration effect, the adjustment is mainly based on the current system load (loadIndex) and the priority of energy data processing tasks. The change in the number of Executors is calculated through executorChange, thereby dynamically increasing or decreasing the number of Executors.

[0084] ;

[0085] Where c is a coefficient adjusted according to the actual situation, and idleResources is the amount of idle resources.

[0086] Specifically, when configuring the corresponding memory storage level, different storage levels are set mainly based on the access frequency and data characteristics of each data in the heterogeneous energy flow data (in this embodiment, importance is used as an example to illustrate; importance may also include other characteristics such as timeliness), and the storage level is calculated to determine the storage level.

[0087] ;

[0088] Among them, MEMORY_AND_DISK_SER means that data is stored in memory first, and if memory is insufficient, it is serialized and spilled to disk. DISK_ONLY means that data is stored entirely on disk and does not occupy memory space. The corresponding judgment condition for this storage level is:

[0089] If accessFrequency > th resh oldf and importance > th resh oldi

[0090] If the memory boundary is set to MEMORY_AND_DISK_SER, then it is set to DISK_ONLY.

[0091] Where th resh oldf is the access frequency threshold, and th resh oldi is the threshold corresponding to the data characteristics.

[0092] This embodiment uses a pre-defined Spark Streaming method to receive heterogeneous energy stream data from multiple sources in real time from Kafka. Based on the different data source types, it generates regularized data streams. Spark Streaming then performs micro-batch segmentation on the regularized data streams and extracts corresponding features from the segmented data streams to obtain feature vector streams. These feature vector streams are stored in a corresponding time-series database for retrieval by relevant personnel. In essence, the embodiment uses a pre-defined Spark Streaming method to receive heterogeneous energy data from multiple sources in real time from Kafka. The received data streams are then processed and split into micro-batch segments using Spark Streaming, and corresponding features are extracted from each segmented data stream to generate specific feature vector streams. These feature vector streams are then stored to ensure that the data received from Kafka is stored in the form of regularized data streams and micro-batch processed feature vector streams according to the specific formats of different data sources. This improves the real-time processing effect of heterogeneous energy data from multiple sources, thereby ensuring accurate and timely data support for real-time monitoring, analysis, and decision-making.

[0093] like Figure 2 As shown, based on the first embodiment, a second embodiment of the heterogeneous energy data processing method of this application is proposed. In this embodiment, the method further includes:

[0094] S11 uses pre-defined Spark Streaming to receive heterogeneous energy stream data from multiple sources in real time from Kafka and generates an initial data stream accordingly.

[0095] S12, Distributed cleaning is performed on the initial data stream according to the different data source types, and the timestamp field of the data provided by each data source in the heterogeneous energy stream data is extracted;

[0096] S13, Based on the timestamp field, determine the common factor of the time interval between the data sources when providing data, and set the corresponding event time window according to the common factor and the accuracy requirements of the current data processing;

[0097] S14. Based on the event time window, perform multi-source time-series alignment on the initial data stream after distributed cleaning to obtain a regularized data stream.

[0098] Understandably, when using Spark Streaming to receive heterogeneous energy streams from multiple sources in real time from Kafka, the system can generate an initial data stream based on the data push time and the data sequence, quickly and stably collecting various types of heterogeneous energy data and providing a continuous and comprehensive raw data foundation for subsequent processing.

[0099] In this embodiment, a variety of customized cleaning rules are used to perform distributed cleaning of energy data with different structures, and multi-source time series are accurately aligned based on event time windows, which significantly improves data quality and provides a reliable foundation for subsequent energy analysis models, thereby improving the accuracy and reliability of the models.

[0100] Specifically, after generating the initial data stream, it is necessary to clean the heterogeneous energy stream data from multiple sources to remove any abnormal data. The cleaned data also needs to be time-aligned to ensure that the processed data stream can generate a uniform and regular data stream for different data source types, thus obtaining a regular data stream.

[0101] In this embodiment, the address parameters, port parameters, and topic parameters of the Kafka cluster are determined through a preset Spark Streaming, and the address parameters, port parameters, and topic parameters are used as data source configuration parameters to access heterogeneous energy data from multiple sources in real time from Kafka and generate an initial data stream.

[0102] Understandably, this embodiment mainly uses the Spark Streaming Kafka data source connector. By setting the Kafka cluster address parameter kafka_adress, port parameter kafka_port, and topic parameter topic, it can receive multi-source energy stream data such as energy data, reports, and other structured data, logs, reports, and other semi-structured data, as well as images and other unstructured data in real time from Kafka. The required data source configuration parameters are determined through Config, and an initial data stream is generated.

[0103] Config=(kafka_adress,kafka_port,topic);

[0104] Specifically, through a specific configuration method, it is possible to access multi-source heterogeneous energy stream data from Kafka in real time and stably, and it can flexibly adapt to different Kafka cluster environments, ensuring the continuity of energy data acquisition and meeting the data access needs of various energy application scenarios. At the same time, by leveraging the streaming processing capabilities of Spark Streaming, the Kafka data source connector can be configured to flexibly set key parameters such as the address, port, and topic of the Kafka cluster, enabling real-time and stable access to multi-source heterogeneous energy stream data from Kafka and transforming it into an initial data stream. This improves the flexibility and stability of data access and provides continuous and reliable raw data support for subsequent data processing.

[0105] In this embodiment, the step of performing distributed cleaning on the initial data stream according to different data source types includes:

[0106] If the initial data stream is structured data, then obtain the standard parameter range corresponding to the structured data, and remove data in the structured data that falls outside the standard parameter range;

[0107] If the initial data stream is semi-structured or unstructured data, the matching degree between the text corresponding to each data in the initial data stream and the preset regular expression is calculated, and data whose matching degree does not meet the preset requirements is removed.

[0108] Understandably, to address the diversity of heterogeneous energy flow data, a variety of customized cleaning rules are employed. For structured data, data is filtered according to a pre-defined reasonable range; for semi-structured and unstructured data, regular expressions and other methods are used to calculate the matching degree for cleaning. Simultaneously, the time series of the cleaned multi-source data is precisely aligned based on event time windows. Through these meticulous and customized operations, data quality is improved, time discrepancies between different data sources are eliminated, and a highly accurate and consistent data foundation is provided for subsequent energy analysis models.

[0109] Specifically, for numerical structured data such as equipment operating parameters, data that exceeds a reasonable range is removed by judging isValid.

[0110] ;

[0111] Where x is the data, and minValue and maxValue are the minimum and maximum values ​​determined according to the standard parameter range of the device operation. The standard parameter range corresponding to the structured data is determined by the two values, thereby removing data that falls outside the standard parameter range.

[0112] Specifically, for semi-structured and unstructured data such as device logs and energy report texts, abnormal data is removed by calculating the matching degree formula matchDegree (used to calculate the matching degree between the text corresponding to each data in the initial data stream and the preset regular expression).

[0113] ;

[0114] Where matchedLength(text, regex) is the length of the match between the text text and the preset regular expression regex, and length(text) is the total length of the text text.

[0115] In this embodiment, a sliding window algorithm is used based on the timestamp field in the energy data, where the event time window size is windowsSize, the sliding step size is slideStep, and the energy data time series is... After window alignment, a regular data stream is output.

[0116] In particular, considering the special characteristics of the multi-source heterogeneous energy flow data to be processed in this implementation, the generation time of structured data, semi-structured data and unstructured data in the heterogeneous energy flow data cannot be kept consistent. The generation time interval of structured data is relatively short (for example, the operation data automatically generated by the equipment, which generates new data every minute), while the generation time interval of unstructured data is relatively long (for example, manually filled text data, which requires a certain amount of time to fill in by humans, but the image data in unstructured data is generated efficiently). Therefore, when aligning the above heterogeneous energy flow data in time sequence, it is necessary to consider the timestamps of the generation time of different data, and extract the common factor of the time interval of each type of data that needs to be aligned according to the generation time interval of different data sources, and use this to define an appropriate event time window.

[0117] In this embodiment, simply setting a time window is not enough to fully satisfy the multi-source time-series alignment effect of heterogeneous energy flow data. Therefore, it is necessary to further adjust the size of the time window based on the defined time window, combined with the data source of the flow data and its corresponding data, as well as the accuracy requirements of the data processing.

[0118] Specifically, for example, the timeliness of data content corresponding to hand-written text data and image data collected by devices differs. Hand-written text is usually limited to a certain period of time, and this data can meet operations such as real-time analysis. However, image data usually refers to information in a moment or a continuous short period of time composed of multiple pictures. Due to the difference in their timeliness, different timeliness accuracy analysis requirements need to be adopted for each. Therefore, it is necessary to increase the time window for hand-written text data and decrease the time window for image data, so as to obtain the time intervals of various types of data in multi-source heterogeneous energy flow data and the event situations they represent, and obtain the event time window applicable to the current situation.

[0119] This embodiment uses a pre-defined Spark Streaming method to receive heterogeneous energy stream data from multiple sources in real time from Kafka, generating an initial data stream. Based on the different data source types, the initial data stream undergoes distributed cleaning, and then multi-source time-series alignment is performed on the cleaned initial data stream according to a pre-defined event time window, resulting in a regularized data stream. In other words, it innovatively employs multiple customized cleaning rules to address the diversity of heterogeneous energy data, and precisely aligns the time series of multi-source data based on event time windows. Through these refined and customized operations, data quality is improved, time discrepancies between different data sources are eliminated, and a highly accurate and consistent data foundation is provided for subsequent energy analysis models.

[0120] like Figure 3As shown, based on the first embodiment, a third embodiment of the heterogeneous energy data processing method of this application is proposed. In this embodiment, the method further includes:

[0121] S21, obtain the generation rate of the heterogeneous energy flow data determined by the Kafka terminal, and determine the resource usage of the local system;

[0122] S22, based on the generation rate, the resource usage, and the preset dynamic coefficient, determine the micro-batch specification that best suits the local system's maximum operating efficiency under the current conditions;

[0123] S23, the regularized data stream is divided into micro-batches using the Spark Streaming and the micro-batch specification.

[0124] In this embodiment, the cleaned and time-aligned data stream is intelligently divided into micro-batches based on factors such as the scale and processing complexity of the energy data. This optimizes memory usage and processing efficiency. Various energy analysis models, such as energy consumption prediction models and equipment fault diagnosis models, are pre-loaded in memory. Each micro-batch of data is analyzed in depth to accurately extract key features and form a feature vector stream. This process extracts core features from massive amounts of energy data and reduces data dimensionality.

[0125] Specifically, based on Spark Streaming technology, a micro-batch processing approach is used to transform real-time streaming data into short batch processing jobs, providing a unified expression for real-time and offline data processing. By defining unified data processing logic and operation operators, the same code system can be compatible with processing streaming and batch data, greatly simplifying the processing flow, reducing development and maintenance costs, and effectively improving the efficiency of streaming-batch fusion.

[0126] Specifically, the micro-batch partitioning is determined based on the generation rate of heterogeneous energy stream data as determined by the Kafka endpoint and the local system resource usage, and the micro-batch rule S is calculated:

[0127] ;

[0128] in, , , R is a preset dynamic coefficient that is adjusted according to actual conditions, and R is the generation rate of heterogeneous energy flow data. and For the resource usage of the local system, in detail This represents the percentage of remaining available resources in the local system's central processing unit. To utilize the remaining available memory space on the local system, the regular data stream is divided according to the micro-batch rule S.

[0129] In this embodiment, the step of extracting corresponding features from the segmented data stream to obtain a feature vector stream includes:

[0130] If the data in the segmented data stream is structured data, then based on the current data processing requirements, key data dimensions are selected as features to obtain the first processing result;

[0131] If the data in the segmented data stream is semi-structured or unstructured, then the TF-IDF algorithm is used to determine the keyword set from the data in the segmented data stream, and the weight of each keyword in the keyword set is determined. The weight is then used as a feature to obtain the second processing result.

[0132] The first processing result and the second processing result are combined into a feature vector, and multiple feature vectors are output in time sequence to form a feature vector stream.

[0133] In this embodiment, custom parsers and converters are developed for heterogeneous energy data with different formats and semantics. Various data parsing and conversion methods are used to integrate data of different structural types into a unified format, ensuring data integrity and accuracy, and significantly improving the ability to process the diversity and complexity of heterogeneous energy data.

[0134] For structured energy data, such as metering data for electricity and gas, key data dimensions are selected. As features, they form the corresponding feature vectors. In order to obtain the first processing result.

[0135] Among them, the key data dimensions mainly refer to the core attribute framework used for the systematic classification, analysis and application of energy data.

[0136] These key data dimensions can be analyzed cross-sectionally (e.g., "region × time × energy type") to reveal energy consumption patterns, optimize resource allocation, and provide decision support for smart energy systems. In practical applications, the combination of dimensions needs to be selected based on the specific scenario; for example, carbon emission monitoring requires linking energy type with spatiotemporal dimensions.

[0137] Specifically, when selecting key data dimensions for cross-analysis, it is necessary to distinguish between primary and secondary analysis dimensions. For example, from "region × time × energy type", region and energy type can be used as secondary analysis dimensions, while time can be used as the primary analysis dimension, thus generating a feature vector of (time, region × energy type).

[0138] Furthermore, the selection of key dimension data can be freely changed according to the needs of actual application scenarios. Specifically, corresponding weight coefficients can be set to divide different main analysis dimensions and auxiliary analysis dimensions in different application scenarios, so as to flexibly meet the data processing needs of different scenarios.

[0139] For semi-structured or unstructured energy data, a keyword set is obtained using the TF-IDF (Term Frequency-Inverse Document Frequency) algorithm. Calculate the weight of each keyword. Keyword weights are used as features to form a feature vector. In order to obtain the second processing result.

[0140] The TF-IDF algorithm evaluates the importance of words in a document set by combining term frequency (TF) and inverse document frequency (IDF) to extract keywords. Specifically, it includes the following steps:

[0141] Text preprocessing: The document is split into words or phrases, punctuation, numbers, stop words and other irrelevant characters are removed, and specific parts of speech (such as nouns and verbs) are retained to improve keyword quality;

[0142] The term frequency (TF) is calculated using the formula: TF(t,d) = the total number of words in document d and the number of times word t appears in document d; this measures the local importance of a word in a single document.

[0143] The inverse document frequency (IDF) is calculated using the formula: IDF(t,D) = log(number of documents containing word t + 1 total number of documents in document set D) (adding 1 avoids the denominator being zero). This penalizes common words and highlights rare words.

[0144] The TF-IDF value is calculated using the formula: TF-IDF(t,d,D)=TF(t,d)×IDF(t,D), which combines local and global importance.

[0145] Sorting and Keyword Extraction Steps: 1. Sort all words in the document in descending order of their TF-IDF values. 2. Select the Top-N (e.g., top 5 or top 10) words as keywords.

[0146] It should be further explained that, in this embodiment, multiple keywords can be optimized and divided according to the classification structure of the knowledge graph of energy field knowledge (the keywords mainly refer to the general term of the same type of data after classification, and multiple replacement words are generated based on the general term). The keywords can be combined with the form of TF-IDF algorithm to selectively select the corresponding keywords, or the keywords corresponding to the knowledge graph can be used as a supplement based on the keywords generated by TF-IDF algorithm to further optimize the accuracy of feature vector generation when extracting semi-structured and unstructured data.

[0147] In summary, the feature vectors extracted from structured and unstructured data are combined into a complete feature vector. Multiple such feature vectors Output them sequentially to form a feature vector stream.

[0148] In this embodiment, the step of performing micro-batch segmentation on the regularized data stream using the Spark Streaming and the micro-batch specification includes:

[0149] The regularized data stream is pre-segmented using Spark Streaming and the micro-batch specification.

[0150] The pre-segmentation results are evaluated based on the multi-dimensional partitioning requirements corresponding to the data characteristics of the heterogeneous energy flow data.

[0151] If the evaluation results do not meet the multi-dimensional segmentation requirements, the micro-batch specifications are optimized, and the regularized data stream is segmented into micro-batches based on the optimized results.

[0152] Understandably, based on the data generation efficiency and system resource usage, the micro-batch specifications for dynamic micro-batch segmentation that meet the data generation efficiency and system resource requirements can be calculated using the above formula. However, when performing micro-batch segmentation on data, it is also necessary to consider whether the data obtained after micro-batch segmentation retains the characteristics of the data before segmentation. For example, if a data jump occurs within a certain period due to equipment failure or other reasons, the event corresponding to this data jump is short, and the information carried by the data has a high degree of continuity. If it is directly split into at least two micro-batch data, it will destroy the continuity characteristic of the data. For another example, if a segment of data is semi-structured data, in which each important piece of information is described in multiple text segments, directly splitting these multiple text segments into multiple micro-batch data will destroy the continuity of the text data, resulting in a significant deviation in the meaning represented by the data when viewing a single segment. For yet another example, some continuous signals or periodically output data have high integrity requirements. If this data is split, its integrity will be destroyed, and the corresponding periodic changes in the data will be lost.

[0153] Therefore, in this application, the data is pre-segmented according to the micro-batch specification, and the results of the pre-segmentation are evaluated according to the multi-dimensional segmentation requirements of the data characteristics (continuity, integrity or text context relevance) corresponding to the multi-source heterogeneous energy flow data. If the evaluation fails, the micro-batch specification is optimized, and the regular data flow is micro-batch segmented according to the optimized results.

[0154] In this embodiment, the step of evaluating the pre-segmentation results based on the multi-dimensional partitioning requirements corresponding to the data characteristics of the heterogeneous energy flow data includes:

[0155] The multi-dimensional segmentation requirements include at least the requirements for data continuity, data contextual coherence, and data integrity.

[0156] Obtain the application requirements proposed by the user when the current data is micro-batch segmented, and assign corresponding requirement weights to the data continuity requirement, the data context coherence requirement, and the data integrity requirement based on the application requirements;

[0157] The pre-segmentation results are evaluated based on the required weights, the data continuity requirements, the data context coherence requirements, and the data integrity requirements.

[0158] If the evaluation results do not meet the multi-dimensional segmentation requirements, the step of optimizing the micro-batch specification includes:

[0159] If the evaluation results do not meet the multi-dimensional segmentation requirements, the data splitting positions corresponding to the micro-batch specifications will be adjusted according to the requirement weights, data continuity requirements, data context coherence requirements, and data integrity requirements, so as to control the data length corresponding to each micro-batch specification to meet the multi-dimensional segmentation requirements.

[0160] Understandably, when the evaluation results do not meet the requirements for multi-dimensional segmentation, it is necessary to optimize and adjust the micro-batch specifications by combining the requirements for demand weight, data continuity, data context coherence, and data integrity. On the one hand, the length of the data after splitting each micro-batch specification is controlled. By controlling the data length, it is ensured that the split streaming data contains data that meets the requirements for multi-dimensional segmentation, thereby ensuring that the optimized micro-batch rules can meet the user's application requirements and multi-dimensional segmentation requirements.

[0161] Specifically, optimizations may include using specific requirements (continuity, integrity, or textual contextual relevance) as the standard for micro-batch specifications to ensure that the data in each segmented data stream meets these specific requirements.

[0162] Specifically, the optimization may also include considering redundant segmentation with overlapping content when segmenting regular data streams. That is, the data at both ends of the time sequence of the segmented micro-batch data will overlap with some content in other micro-batch data.

[0163] It should be noted that, since user application requirements influence the tendency of various multi-dimensional segmentation requirements as a weight, different optimization effects may occur during the adjustment of micro-batch specifications, depending on the actual needs. Furthermore, the optimization process must consider system resource utilization; that is, system resource utilization should be recorded when calculating micro-batch specifications. After micro-batch specification optimization, the rationality of the optimization result should be further judged based on application requirements, multi-dimensional segmentation requirements, and system resource usage. Specifically, after ensuring data integrity, to avoid excessively long data segments in each micro-batch specification and considering the possibility of redundant batch splitting during optimization, the data length during micro-batch specification optimization should be kept less than a preset data length. This preset data length is calculated proportionally based on system resource usage. If the proportion of idle system resources to total resources is greater than 50%, the data length is increased; if it is less than 50%, the data length is decreased. The specific increase / decrease ratio can be set according to actual needs.

[0164] Understandably, the multi-dimensional segmentation requirements corresponding to data characteristics include at least the requirements for data continuity (such as the continuity of fault mutation signals), data contextual coherence (the specific textual semantic content of semi-structured data), and data integrity (the integrity of some short-cycle data).

[0165] It is understandable that different data characteristics correspond to different needs, but these may not necessarily meet the user's needs. For example, long-cycle data does not need to guarantee data integrity, and concise text content (each text segment is independent) does not need to consider contextual coherence. Therefore, when performing micro-batch partitioning on the current heterogeneous energy flow data, the user's actual application needs can be introduced. These application needs mainly involve the user making corresponding demand judgments and adjustments based on the content of the data generated and pushed in real time by the Kafka end. The main content of these application needs includes setting demand weights corresponding to multi-dimensional partitioning needs, setting a certain bias, evaluating the results of micro-batch pre-segmentation based on the demand weights and corresponding multi-dimensional partitioning needs, and further influencing the specific standards of micro-batch specifications based on the evaluation results.

[0166] This embodiment obtains the generation rate of the heterogeneous energy stream data determined by the Kafka endpoint and determines the resource usage of the local system. Based on the generation rate, resource usage, and preset dynamic coefficients, it determines the micro-batch specification that best suits the local system's maximum operating efficiency under the current conditions. Through Spark Streaming and the micro-batch specification, the regularized data stream is segmented into micro-batches. This intelligently segments the cleaned and time-aligned regularized data stream into micro-batches based on factors such as the scale and processing complexity of the energy data, thereby optimizing memory usage and processing efficiency. Furthermore, each micro-batch of data is analyzed in depth to accurately extract key features and form a feature vector stream, thus extracting core features from massive amounts of energy data and reducing data dimensionality.

[0167] Furthermore, embodiments of this application also propose a heterogeneous energy data processing device, referring to... Figure 4 The heterogeneous energy data processing device includes:

[0168] The data generation module 10 is used to access heterogeneous energy stream data from multiple sources in real time from Kafka through the preset Spark Streaming, and generate regular data streams according to the different data source types.

[0169] The data processing module 20 is used to perform micro-batch segmentation on the regular data stream through the Spark Streaming, and extract corresponding features from the segmented data stream to obtain a feature vector stream;

[0170] The data storage module 30 is used to store the feature vector stream to a corresponding time series database so that relevant personnel can retrieve the feature vector stream from the time series database.

[0171] This embodiment uses a pre-defined Spark Streaming method to receive heterogeneous energy stream data from multiple sources in real time from Kafka. Based on the different data source types, it generates regularized data streams. Spark Streaming then performs micro-batch segmentation on the regularized data streams and extracts corresponding features from the segmented data streams to obtain feature vector streams. These feature vector streams are stored in a corresponding time-series database for retrieval by relevant personnel. In essence, the embodiment uses a pre-defined Spark Streaming method to receive heterogeneous energy data from multiple sources in real time from Kafka. The received data streams are then processed and split into micro-batch segments using Spark Streaming, and corresponding features are extracted from each segmented data stream to generate specific feature vector streams. These feature vector streams are then stored to ensure that the data received from Kafka is stored in the form of regularized data streams and micro-batch processed feature vector streams according to the specific formats of different data sources. This improves the real-time processing effect of heterogeneous energy data from multiple sources, thereby ensuring accurate and timely data support for real-time monitoring, analysis, and decision-making.

[0172] It should be noted that each module in the above-mentioned device can be used to implement each step in the above-mentioned method and achieve the corresponding technical effect. This embodiment will not elaborate further here.

[0173] Reference Figure 5 , Figure 5 This is a schematic diagram of the hardware operating environment of the device involved in the embodiments of this application.

[0174] like Figure 5 As shown, the device may include: a processor 1001, such as a CPU; a communication bus 1002; a user interface 1003; a network interface 1004; and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0175] Those skilled in the art will understand that Figure 5 The structure shown does not constitute a limitation on the device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0176] like Figure 5 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a heterogeneous energy data processing program.

[0177] exist Figure 5 In the device shown, the network interface 1004 is mainly used for data communication with an external network; the user interface 1003 is mainly used for receiving user input commands; the device calls the heterogeneous energy data processing program stored in the memory 1005 through the processor 1001 and performs the following operations:

[0178] The system uses Spark Streaming to receive heterogeneous energy stream data from multiple sources in real time from Kafka and generates regular data streams based on the different data source types.

[0179] The Spark Streaming method is used to perform micro-batch segmentation on the regular data stream, and corresponding features are extracted from the segmented data stream to obtain a feature vector stream.

[0180] The feature vector stream is stored in the corresponding time series database so that relevant personnel can retrieve the feature vector stream from the time series database.

[0181] Furthermore, the processor 1001 can call the heterogeneous energy data processing program stored in the memory 1005 and also perform the following operations:

[0182] The system uses a pre-defined Spark Streaming mechanism to receive heterogeneous energy stream data from multiple sources in real time from Kafka, and uses this data to generate an initial data stream.

[0183] Based on the different data source types, the initial data stream is subjected to distributed cleaning, and the timestamp field of the data provided by each data source in the heterogeneous energy stream data is extracted;

[0184] Based on the timestamp field, determine the common factor of the time intervals between the data sources when providing data, and set the corresponding event time window according to the common factor and the accuracy requirements of the current data processing;

[0185] Based on the event time window, the initial data stream after distributed cleaning is aligned with multiple sources in time sequence to obtain a regularized data stream.

[0186] Furthermore, the processor 1001 can call the heterogeneous energy data processing program stored in the memory 1005 and also perform the following operations:

[0187] If the initial data stream is structured data, then obtain the standard parameter range corresponding to the structured data, and remove data in the structured data that falls outside the standard parameter range;

[0188] If the initial data stream is semi-structured or unstructured data, the matching degree between the text corresponding to each data in the initial data stream and the preset regular expression is calculated, and data whose matching degree does not meet the preset requirements is removed.

[0189] Furthermore, the processor 1001 can call the heterogeneous energy data processing program stored in the memory 1005 and also perform the following operations:

[0190] Obtain the generation rate of the heterogeneous energy stream data determined by the Kafka endpoint, and determine the resource usage of the local system;

[0191] Based on the generation rate, resource usage, and preset dynamic coefficients, determine the micro-batch size that best suits the local system's maximum operating efficiency under the current conditions;

[0192] The regularized data stream is divided into micro-batches using the Spark Streaming and micro-batch specifications.

[0193] Furthermore, the processor 1001 can call the heterogeneous energy data processing program stored in the memory 1005 and also perform the following operations:

[0194] The regularized data stream is pre-segmented using Spark Streaming and the micro-batch specification.

[0195] The pre-segmentation results are evaluated based on the multi-dimensional partitioning requirements corresponding to the data characteristics of the heterogeneous energy flow data.

[0196] If the evaluation results do not meet the multi-dimensional segmentation requirements, the micro-batch specifications are optimized, and the regularized data stream is segmented into micro-batches based on the optimized results.

[0197] Furthermore, the processor 1001 can call the heterogeneous energy data processing program stored in the memory 1005 and also perform the following operations:

[0198] The multi-dimensional segmentation requirements include at least the requirements for data continuity, data contextual coherence, and data integrity.

[0199] Obtain the application requirements proposed by the user when the current data is micro-batch segmented, and assign corresponding requirement weights to the data continuity requirement, the data context coherence requirement, and the data integrity requirement based on the application requirements;

[0200] The pre-segmentation results are evaluated based on the required weights, the data continuity requirements, the data context coherence requirements, and the data integrity requirements.

[0201] If the evaluation results do not meet the multi-dimensional segmentation requirements, the data splitting positions corresponding to the micro-batch specifications will be adjusted according to the requirement weights, data continuity requirements, data context coherence requirements, and data integrity requirements, so as to control the data length corresponding to each micro-batch specification to meet the multi-dimensional segmentation requirements.

[0202] Furthermore, the processor 1001 can call the heterogeneous energy data processing program stored in the memory 1005 and also perform the following operations:

[0203] If the data in the segmented data stream is structured data, then based on the current data processing requirements, key data dimensions are selected as features to obtain the first processing result;

[0204] If the data in the segmented data stream is semi-structured or unstructured, then the TF-IDF algorithm is used to determine the keyword set from the data in the segmented data stream, and the weight of each keyword in the keyword set is determined. The weight is then used as a feature to obtain the second processing result.

[0205] The first processing result and the second processing result are combined into a feature vector, and multiple feature vectors are output in time sequence to form a feature vector stream.

[0206] Furthermore, the processor 1001 can call the heterogeneous energy data processing program stored in the memory 1005 and also perform the following operations:

[0207] Determine the load and available resources of the local system, and determine the priority of the current data task;

[0208] Configure the appropriate number of executors based on the load, priority, and amount of idle resources;

[0209] Determine the access frequency and data characteristics of the heterogeneous energy flow data;

[0210] Configure the corresponding storage level based on the access frequency and the data characteristics;

[0211] The feature vector stream is stored in the corresponding time-series database according to the configured executor and storage level.

[0212] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0213] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0214] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the heterogeneous energy data processing method in the above embodiments.

[0215] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0216] The aforementioned computer-readable storage medium may be included in the heterogeneous energy data processing device; or it may exist independently and not assembled into the heterogeneous energy data processing device.

[0217] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the heterogeneous energy data processing device, cause the heterogeneous energy data processing device to:

[0218] The system uses Spark Streaming to receive heterogeneous energy stream data from multiple sources in real time from Kafka and generates regular data streams based on the different data source types.

[0219] The Spark Streaming method is used to perform micro-batch segmentation on the regular data stream, and corresponding features are extracted from the segmented data stream to obtain a feature vector stream.

[0220] The feature vector stream is stored in the corresponding time series database so that relevant personnel can retrieve the feature vector stream from the time series database.

[0221] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0222] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0223] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0224] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described heterogeneous energy data processing method, thereby solving the technical problems of heterogeneous energy data processing. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the heterogeneous energy data processing method provided in the above embodiments, and will not be repeated here.

[0225] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

[0226] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0227] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0228] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0229] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for processing heterogeneous energy data, characterized in that, The heterogeneous energy data processing method includes the following steps: The system uses Spark Streaming to receive heterogeneous energy stream data from multiple sources in real time from Kafka and generates regular data streams based on the different data source types. The Spark Streaming method is used to perform micro-batch segmentation on the regular data stream, and corresponding features are extracted from the segmented data stream to obtain a feature vector stream. The feature vector stream is stored in the corresponding time series database so that relevant personnel can retrieve the feature vector stream from the time series database; The step of performing micro-batch segmentation on the regularized data stream using Spark Streaming includes: Obtain the generation rate of the heterogeneous energy stream data determined by the Kafka endpoint, and determine the resource usage of the local system; Based on the generation rate, resource usage, and preset dynamic coefficients, determine the micro-batch size that best suits the local system's maximum operating efficiency under the current conditions; micro-batch size S : ;in, , , These are preset dynamic coefficients that are adjusted according to actual conditions. R The generation rate of heterogeneous energy flow data. and For the resource usage of the local system, in detail This represents the percentage of remaining available resources in the local system's central processing unit. To utilize the remaining available memory space on the local system, the regular data stream is divided into micro-batch sizes S; The regularized data stream is divided into micro-batches using the Spark Streaming and micro-batch specifications.

2. The method as described in claim 1, characterized in that, The steps of receiving heterogeneous energy stream data from multiple sources in real time from Kafka via a preset Spark Streaming method, and generating a regularized data stream according to different data source types, include: The system uses a pre-defined Spark Streaming mechanism to receive heterogeneous energy stream data from multiple sources in real time from Kafka, and uses this data to generate an initial data stream. Based on the different data source types, the initial data stream is subjected to distributed cleaning, and the timestamp field of the data provided by each data source in the heterogeneous energy stream data is extracted; Based on the timestamp field, determine the common factor of the time intervals between the data sources when providing data, and set the corresponding event time window according to the common factor and the accuracy requirements of the current data processing; Based on the event time window, the initial data stream after distributed cleaning is aligned with multiple sources in time sequence to obtain a regularized data stream.

3. The method as described in claim 2, characterized in that, The step of performing distributed cleaning of the initial data stream according to different data source types includes: If the initial data stream is structured data, then obtain the standard parameter range corresponding to the structured data, and remove data in the structured data that falls outside the standard parameter range; If the initial data stream is semi-structured or unstructured data, the matching degree between the text corresponding to each data in the initial data stream and the preset regular expression is calculated, and data whose matching degree does not meet the preset requirements is removed.

4. The method as described in claim 1, characterized in that, The step of performing micro-batch segmentation on the regularized data stream using the Spark Streaming and the micro-batch specification includes: The regularized data stream is pre-segmented using Spark Streaming and the micro-batch specification. The pre-segmentation results are evaluated based on the multi-dimensional partitioning requirements corresponding to the data characteristics of the heterogeneous energy flow data. If the evaluation results do not meet the multi-dimensional segmentation requirements, the micro-batch specifications are optimized, and the regularized data stream is segmented into micro-batches based on the optimized results.

5. The method as described in claim 4, characterized in that, The step of evaluating the pre-segmentation results based on the multi-dimensional partitioning requirements corresponding to the data characteristics of the heterogeneous energy flow data includes: The multi-dimensional segmentation requirements include at least the requirements for data continuity, data contextual coherence, and data integrity. Obtain the application requirements proposed by the user when the current data is micro-batch segmented, and assign corresponding requirement weights to the data continuity requirement, the data context coherence requirement, and the data integrity requirement based on the application requirements; The pre-segmentation results are evaluated based on the required weights, the data continuity requirements, the data context coherence requirements, and the data integrity requirements. If the evaluation results do not meet the multi-dimensional segmentation requirements, the step of optimizing the micro-batch specification includes: If the evaluation results do not meet the multi-dimensional segmentation requirements, the data splitting positions corresponding to the micro-batch specifications will be adjusted according to the requirement weights, data continuity requirements, data context coherence requirements, and data integrity requirements, so as to control the data length corresponding to each micro-batch specification to meet the multi-dimensional segmentation requirements.

6. The method as described in claim 1, characterized in that, The step of extracting corresponding features from the segmented data stream to obtain a feature vector stream includes: If the data in the segmented data stream is structured data, then based on the current data processing requirements, key data dimensions are selected as features to obtain the first processing result; If the data in the segmented data stream is semi-structured or unstructured, then the TF-IDF algorithm is used to determine the keyword set from the data in the segmented data stream, and the weight of each keyword in the keyword set is determined. The weight is then used as a feature to obtain the second processing result. The first processing result and the second processing result are combined into a feature vector, and multiple feature vectors are output in time sequence to form a feature vector stream.

7. The method as described in claim 1, characterized in that, The step of storing the feature vector stream into the corresponding time-series database includes: Determine the load and available resources of the local system, and determine the priority of the current data task; Configure the appropriate number of executors based on the load, priority, and amount of idle resources; Determine the access frequency and data characteristics of the heterogeneous energy flow data; Configure the corresponding storage level based on the access frequency and the data characteristics; The feature vector stream is stored in the corresponding time-series database according to the configured executor and storage level.

8. A heterogeneous energy data processing device, characterized in that, The heterogeneous energy data processing device includes: a memory, a processor, and a heterogeneous energy data processing program stored in the memory and executable on the processor, wherein the heterogeneous energy data processing program is configured to implement the steps of the heterogeneous energy data processing method as described in any one of claims 1 to 7.

9. A storage medium, characterized in that, The storage medium stores a program for implementing a heterogeneous energy data processing method, and the program for implementing the heterogeneous energy data processing method is executed by a processor to implement the steps of the heterogeneous energy data processing method as described in any one of claims 1 to 7.

Images