Heterogeneous energy data processing method and device and storage medium

Abstract

The invention discloses a heterogeneous energy data processing method and device and a storage medium, and belongs to the technical field of multi-modal data processing. The method comprises the following steps: accessing multi-source heterogeneous energy flow data from kafka in real time through preset Spark Streaming, and generating a regular data flow according to the types of different data sources; through the Spark Streaming, performing micro-batch segmentation on the regular data stream, and extracting corresponding features from the segmented data stream to obtain a feature vector stream; according to the method, the feature vector flow is stored in a corresponding time sequence database so that related personnel can call the feature vector flow from the time sequence database, and real-time data accessed from kafka is subjected to micro-batch segmentation processing through Spark Streaming to generate the corresponding feature vector flow, so that the effect of efficiently processing heterogeneous energy data in real time is achieved, and the real-time performance of the heterogeneous energy data is improved. And accurate and timely data support is provided for real-time monitoring, analysis and decision-making in the energy field.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a heterogeneous energy data processing method, device, and storage medium. Background Art

[0002] In recent years, with the widespread application of technologies such as smart sensors and the Internet of Things in the energy field, the scale of heterogeneous energy data has grown exponentially, and the data types have become increasingly complex and diverse, covering structured equipment operating parameters, semi-structured log files, and unstructured text reports.

[0003] However, real-time processing of heterogeneous energy data faces challenges in practical applications: heterogeneous energy data comes from a wide range of sources and in various formats, making it difficult to stably and efficiently access various types of energy flow data from different Kafka clusters. In addition, the structure of multi-source heterogeneous energy data is complex, including structured, semi-structured and unstructured data. Existing cleaning methods cannot effectively process data of different structures, and time series alignment is difficult.

[0004] Therefore, it is currently impossible to effectively process heterogeneous energy data in real time, resulting in the inability to provide accurate and timely data support for real-time monitoring, analysis and decision-making in the energy field. Summary of the Invention

[0005] The main purpose of this application is to provide a heterogeneous energy data processing method, device and storage medium, aiming to solve the technical problem of being unable to effectively process heterogeneous energy data in real time, thereby failing to provide accurate and timely data support for real-time monitoring, analysis and decision-making in the energy field.

[0006] To achieve the above objectives, the present application provides a method for processing heterogeneous energy data, which includes the following steps: Through the preset Spark Streaming, it accesses heterogeneous energy flow data from multiple sources in real time from Kafka, and generates regular data streams based on the types of different data sources; By using Spark Streaming, the regularized data stream is divided into micro-batches, and corresponding features are extracted from the divided data stream to obtain a feature vector stream; The feature vector stream is stored in a corresponding time series database so that relevant personnel can retrieve the feature vector stream from the time series database.

[0007] In one embodiment, the step of accessing heterogeneous energy flow data from multiple sources from Kafka in real time through the preset Spark Streaming and generating a regularized data stream based on the types of different data sources includes: Use the preset Spark Streaming to access heterogeneous energy flow data from multiple sources from Kafka in real time and generate the initial data stream; Perform distributed cleaning on the initial data stream according to the type of different data sources, and extract the timestamp field of the data provided by each data source in the heterogeneous energy flow data; Determine, based on the timestamp field, a common factor of the time intervals between the data sources providing data, and set a corresponding event time window based on the common factor and the accuracy requirement of the current data processing; According to the event time window, the distributed cleaned initial data stream is aligned with multiple sources to obtain a regular data stream.

[0008] In one embodiment, the step of performing distributed cleaning on the initial data stream according to different data source types includes: If the initial data stream is structured data, obtaining a standard parameter range corresponding to the structured data, and removing data outside the standard parameter range in the structured data; If the initial data stream is semi-structured data or unstructured data, the matching degree between the text corresponding to each data in the initial data stream and the preset regular expression is calculated, and the data whose matching degree does not meet the preset requirement is removed.

[0009] In one embodiment, the step of dividing the regular data stream into micro-batches by Spark Streaming includes: Obtaining a generation rate of the heterogeneous energy flow data determined by the Kafka end, and determining resource usage of the local system; Determining a micro-batch size adapted to the maximum operating efficiency of the local system under current circumstances based on the generation rate, the resource usage, and a preset dynamic coefficient; The regular data stream is divided into micro-batches using the Spark Streaming and the micro-batch specification.

[0010] In one embodiment, the step of dividing the regular data stream into micro-batches using Spark Streaming and the micro-batch specification includes: Pre-segmenting the regularized data stream using the Spark Streaming and micro-batch specifications; Evaluate the pre-segmentation results according to the multi-dimensional segmentation requirements corresponding to the data characteristics of the heterogeneous energy flow data; If the evaluation result does not meet the multi-dimensional partitioning requirements, the micro-batch specifications are optimized, and the regularized data stream is divided into micro-batches according to the optimized results.

[0011] In one embodiment, the step of evaluating the pre-segmentation result according to the multi-dimensional segmentation requirements corresponding to the data characteristics of the heterogeneous energy flow data includes: The multi-dimensional partitioning requirements include at least data continuity requirements, data context consistency requirements, and data integrity requirements; Obtaining application requirements proposed by the user during the current data micro-batch segmentation, and assigning corresponding requirement weights to the data continuity requirement, the data context coherence requirement, and the data integrity requirement based on the application requirements; Evaluating the pre-segmentation result according to the requirement weight, the data continuity requirement, the data context coherence requirement, and the data integrity requirement; If the evaluation result does not meet the multi-dimensional partitioning requirements, the step of optimizing the micro-batch size includes: If the evaluation result does not meet the multi-dimensional partitioning requirement, the data splitting position corresponding to the micro-batch specification is adjusted according to the requirement weight, the data continuity requirement, the data context consistency requirement and the data integrity requirement, so as to control the data length corresponding to each micro-batch specification to meet the multi-dimensional partitioning requirement.

[0012] In one embodiment, the step of extracting corresponding features from the segmented data stream to obtain a feature vector stream includes: If the data in the segmented data stream is structured data, then according to the current data processing requirements, the key data dimension is selected as the feature to obtain the first processing result; If the data in the segmented data stream is semi-structured data or unstructured data, determining a keyword set from the data in the segmented data stream using a TF-IDF algorithm, determining a weight of each keyword in the keyword set, and using the weight as a feature to obtain a second processing result; The first processing result and the second processing result are combined into a feature vector, and a plurality of the feature vectors are output in time sequence to form a feature vector stream.

[0013] In one embodiment, the step of storing the feature vector stream in a corresponding time series database includes: Determine the load and idle resources of the local system and prioritize the current data tasks; Allocate a corresponding number of executors according to the load condition, the priority, and the amount of idle resources; determining access frequency and data characteristics of the heterogeneous energy flow data; Configuring a corresponding storage level according to the access frequency and the data characteristics; According to the configured executor and storage level, the feature vector stream is stored in the corresponding time series database.

[0014] In addition, to achieve the above-mentioned purpose, the present application further provides a heterogeneous energy data processing device, the heterogeneous energy data processing device comprising: The data generation module is used to access heterogeneous energy flow data from multiple sources from Kafka in real time through the preset Spark Streaming, and generate regular data streams based on the types of different data sources; A data processing module is used to divide the regular data stream into micro batches through Spark Streaming, and extract corresponding features from the divided data stream to obtain a feature vector stream; The data storage module is used to store the feature vector stream in a corresponding time series database so that relevant personnel can retrieve the feature vector stream from the time series database.

[0015] In addition, to achieve the above-mentioned purpose, the present application also provides a heterogeneous energy data processing device, which includes: a memory, a processor, and a heterogeneous energy data processing program stored on the memory and executable on the processor, wherein the heterogeneous energy data processing program is configured to implement the steps of the heterogeneous energy data processing method as described above.

[0016] In addition, to achieve the above-mentioned purpose, the present application also provides a computer-readable storage medium, on which a heterogeneous energy data processing program is stored. When the heterogeneous energy data processing program is executed by a processor, the steps of the heterogeneous energy data processing method described above are implemented.

[0017] One or more technical solutions proposed in this application have at least the following technical effects: accessing multi-source heterogeneous energy flow data from Kafka in real time through the preset Spark Streaming, and generating a regularized data stream according to the types of different data sources; through the Spark Streaming, the regularized data stream is micro-batch segmented, and corresponding features are extracted from the segmented data stream to obtain a feature vector stream; the feature vector stream is stored in a corresponding time series database so that relevant personnel can retrieve the feature vector stream from the time series database, that is, accessing multi-source heterogeneous energy data from Kafka in real time through the preset Spark Streaming, and the processed data stream is micro-batch segmented by using Spark Streaming, and corresponding features are respectively extracted from the segmented data streams, thereby generating a specific feature vector stream, and storing it, so as to ensure that the data accessed from the Kafka end is stored in the form of a regularized data stream and a feature vector stream after micro-batch processing according to the specific format of different data sources, so as to improve the real-time processing effect of multi-source heterogeneous energy data, thereby ensuring the provision of accurate and timely data support for real-time monitoring, analysis and decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 A flowchart of the first embodiment of the heterogeneous energy data processing method of this application is provided; Figure 2 A flow chart illustrating the second embodiment of the heterogeneous energy data processing method of this application; Figure 3 A flowchart of the third embodiment of the heterogeneous energy data processing method of this application is provided; Figure 4 This is a schematic diagram of the module structure of the heterogeneous energy data processing device according to an embodiment of the present application; Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the heterogeneous energy data processing method in the embodiment of the present application.

[0021] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0022] It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0023] It's important to note that in recent years, with the widespread adoption of technologies like smart sensors and the Internet of Things in the energy sector, the volume of heterogeneous energy data has grown exponentially. The data types are becoming increasingly complex and diverse, encompassing structured equipment operating parameters, semi-structured log files, and unstructured text reports. At the same time, energy systems are increasingly demanding the real-time, accurate, and comprehensive nature of data processing to enable timely, informed decision-making and ensure the efficient and stable operation of energy systems.

[0024] However, due to the inherent characteristics of heterogeneous energy data, the limitations of existing processing technologies, and system architecture, real-time processing of heterogeneous energy data faces many challenges in practical applications: (1) Difficulty in accessing heterogeneous energy data: Heterogeneous energy data comes from a wide range of sources and in various formats. Traditional methods make it difficult to stably and efficiently access various energy flow data from different Kafka clusters, resulting in interruptions or data loss in the data acquisition process, which affects subsequent processing and analysis. (2) Dilemma in cleaning and aligning heterogeneous energy data: Multi-source heterogeneous energy data has a complex structure, including structured, semi-structured, and unstructured data. Existing cleaning methods cannot effectively process data of different structures, and time series alignment is difficult, resulting in low data quality and difficulty in meeting subsequent accurate analysis requirements. (3) Unreasonable utilization of system resources: In the process of processing heterogeneous energy data, traditional methods use fixed resource allocation strategies, such as a fixed number of executors and a unified memory storage level, which cannot be dynamically adjusted according to the characteristics of data processing tasks (such as task priority, data size and complexity, etc.), resulting in resource waste or low processing efficiency.

[0025] Based on the above problems, this application proposes a heterogeneous energy data processing method based on Spark Streaming (real-time stream processing framework), aiming to break the limitations of traditional stream and batch processing separation, realize efficient real-time processing and in-depth analysis of heterogeneous energy data, and improve the intelligent level of energy management.

[0026] Reference Figure 1 , Figure 1 This is a flow chart of the first embodiment of the heterogeneous energy data processing method of this application.

[0027] In a first embodiment, the heterogeneous energy data processing method includes the following steps: S10 uses the preset Spark Streaming to access heterogeneous energy flow data from multiple sources in real time from Kafka, and generates regular data streams based on the types of different data sources; It is understandable that Kafka is a distributed stream processing platform that is designed to efficiently process large-scale real-time data streams, supports high-throughput, low-latency message delivery, and is widely used in building real-time data pipelines and streaming applications. In this embodiment, Kafka mainly needs to push energy flow data of different structure types and different data sources to Spark Streaming in real time.

[0028] Among them, the heterogeneous energy flow data pushed by the Kafka end includes at least energy data with different structural types such as structured data, semi-structured data and unstructured data. At the same time, Kafka will generate real-time push data streams according to the time when the data is generated.

[0029] Spark Streaming is a core component specifically designed for processing real-time data streams. It primarily employs a micro-batch processing architecture, dividing continuous data streams into a series of small batches and leveraging Spark's distributed computing engine for efficient processing. In this embodiment, Spark Streaming is primarily used to access heterogeneous energy stream data from Kafka and process it accordingly, including generating regular data streams from the streams, splitting the data streams into micro-batches, and extracting vectors.

[0030] It is understandable that after Spark Streaming accesses heterogeneous energy flow data from the Kafka end, due to the complex composition of the data, the data structure, data content, and timestamps generated by different data sources are different. Therefore, Spark Streaming is needed to perform preliminary data preprocessing on the heterogeneous energy flow data in order to process it into a regular data stream, including operations such as data cleaning, data alignment, and data unified format processing.

[0031] S20, dividing the regularized data stream into micro-batches through Spark Streaming, and extracting corresponding features from the divided data stream to obtain a feature vector stream; It is understandable that Spark Streaming adopts a micro-batch processing architecture to divide the continuous data stream into a series of small batches and utilize Spark's distributed computing engine for efficient processing. In this embodiment, this feature of Spark Streaming is also adopted to divide the preprocessed regular data stream into micro-batches, and extract the corresponding features from the segmented data stream, and form a feature vector stream based on the features.

[0032] It should be noted that since the regular data stream includes data of various structural types and different data sources, when extracting features, data of different types and contents will be extracted. It is necessary to comprehensively consider the above data, use different methods to extract features, and fuse different features into a feature vector stream.

[0033] S30: storing the feature vector stream in a corresponding time series database, so that relevant personnel can retrieve the feature vector stream from the time series database.

[0034] It should be noted that when writing the feature vector stream into the corresponding time series database (since the data contains a timestamp, indicating its corresponding time series characteristics, it needs to be written into the time series database accordingly), the data points of the feature vector stream are written into a time series database suitable for energy data storage, converted into measurement names, labels, timestamps and field information according to the format requirements and written, so as to ensure the regularity of data storage and enable relevant personnel to retrieve the corresponding feature vector stream from the time series database during the subsequent use of the data, providing accurate and timely data support for subsequent relevant personnel to perform real-time monitoring, analysis and decision-making operations.

[0035] At the same time, it strictly adheres to the RESTful architecture principles and JDBC specifications, encapsulates standardized RESTful / JDBC interfaces, realizes long-term and stable storage of energy data, and provides a unified and standard data access method for various external systems, such as energy management platforms and data analysis tools, thereby improving the versatility and usability of data and promoting the circulation and application of energy data in the entire energy ecosystem.

[0036] In this embodiment, the step of storing the feature vector stream in the corresponding time series database includes: Determine the load and idle resources of the local system and prioritize the current data tasks; Allocate a corresponding number of executors according to the load condition, the priority, and the amount of idle resources; determining access frequency and data characteristics of the heterogeneous energy flow data; Configuring a corresponding storage level according to the access frequency and the data characteristics; According to the configured executor and storage level, the feature vector stream is stored in the corresponding time series database.

[0037] It is understandable that when storing the feature vector stream in the corresponding time series database, it is necessary to consider the current local system load and idle resources. Therefore, the corresponding resources can be reasonably allocated according to the current local system load and idle resources to perform the feature vector stream storage work. At the same time, in this embodiment, an additional storage level corresponding to heterogeneous energy flow data is designed to avoid the traditional approach of adopting a fixed resource allocation strategy, such as a fixed number of executors and a unified memory storage level. That is, in this embodiment, the number of executors is intelligently and dynamically adjusted based on multiple factors such as the real-time monitoring of the system load, idle resources, the priority of the data processing task, and the data size and complexity. At the same time, according to the access frequency and importance of energy data, the memory storage level is reasonably configured to achieve efficient utilization of system resources, reduce system operating costs, and ensure that the system can operate stably and efficiently under different workloads.

[0038] The priority of a data task refers to whether the data content corresponding to the current data task is urgently needed by the current user or whether the importance and timeliness of the data content are high.

[0039] Among them, the storage level of the characteristic vector flow is mainly determined by the access frequency and data characteristics of the heterogeneous energy flow data. If the access frequency of the flow data is high (the frequency with which users need to use the data is high), the storage level of the characteristic vector flow corresponding to the heterogeneous energy flow data will be higher. The data characteristics refer to indicators such as the real-time nature, importance, and security of the data. The higher the indicator, the higher the storage level of the corresponding characteristic vector flow.

[0040] Specifically, when dynamically adjusting the number of executors to achieve the configuration effect, the adjustment is mainly based on the current system load condition loadIndex and the priority of the energy data processing task. The change in the number of executors is calculated through executorChange, and the number of executors is dynamically increased or decreased.

[0041] ; Among them, c is a coefficient adjusted according to actual conditions, and idleResources is the amount of idle resources.

[0042] Specifically, when configuring the corresponding memory storage level, different storage levels are set mainly based on the access frequency accessFrequency and data characteristics of each data in the heterogeneous energy flow data (in this embodiment, importance is used as an example, and importance may also include other characteristics such as timeliness), and storageLevel is calculated to determine the storage level.

[0043] ; MEMORY_AND_DISK_SER means that data is stored in memory first. If memory is insufficient, data is serialized and then written to disk. DISK_ONLY means that data is stored completely on disk without occupying memory space. The corresponding judgment conditions for this storage level are: If accessFrequency>th resh oldf and importance>th resh oldi, Then the storage level is determined to be MEMORY_AND_DISK_SER, otherwise it is DISK_ONLY; Wherein, th resh oldf is the access frequency threshold, and th resh oldi is the threshold corresponding to the data characteristic.

[0044] This embodiment uses the preset Spark Streaming to access multi-source heterogeneous energy flow data from Kafka in real time, and generates regularized data streams according to the types of different data sources; uses the Spark Streaming to split the regularized data stream into micro-batches, and extracts corresponding features from the split data streams to obtain feature vector streams; and stores the feature vector streams in a corresponding time series database so that relevant personnel can retrieve the feature vector streams from the time series database. That is, the preset Spark Streaming is used to access multi-source heterogeneous energy data from Kafka in real time, and the processed data streams are split into micro-batches by using Spark Streaming, and corresponding features are extracted from the split data streams respectively, thereby generating specific feature vector streams, and storing them. This ensures that the data accessed from the Kafka end is stored in the form of regularized data streams and feature vector streams after micro-batch processing according to the specific formats of different data sources, thereby improving the real-time processing effect of multi-source heterogeneous energy data, thereby ensuring accurate and timely data support for real-time monitoring, analysis, and decision-making.

[0045] like Figure 2 As shown, based on the first embodiment, a second embodiment of the heterogeneous energy data processing method of the present application is proposed. In this embodiment, the method further includes: S11, through the preset Spark Streaming, accesses multi-source heterogeneous energy flow data from Kafka in real time and generates the initial data stream; S12, performing distributed cleaning on the initial data stream according to different data source types, and extracting the timestamp field of the data provided by each data source in the heterogeneous energy flow data; S13, determining a common factor of the time intervals between the data sources when providing data based on the timestamp field, and setting a corresponding event time window based on the common factor and the accuracy requirement of the current data processing; S14: Perform multi-source time series alignment on the initial data stream after distributed cleaning according to the event time window to obtain a regularized data stream.

[0046] It is understandable that when the preset Spark Streaming is used to access heterogeneous energy flow data from multiple sources in real time from Kafka, the corresponding initial data stream can be generated according to the data push time and the time sequence of the data, and various types of heterogeneous energy data can be collected quickly and stably, providing a continuous and comprehensive original data foundation for subsequent processing.

[0047] In this embodiment, a variety of customized cleaning rules are used to perform distributed cleaning on energy data of different structures, and multi-source time series are accurately aligned based on event time windows, which significantly improves data quality, provides a reliable foundation for subsequent energy analysis models, and can enhance the accuracy and reliability of the models.

[0048] Specifically, after generating the initial data stream, it is necessary to clean the heterogeneous energy flow data from multiple sources accordingly to remove the abnormal data therein, and to align the cleaned data in time sequence to ensure that the processed data stream can generate a data stream in a unified and regular format for different data source types, that is, to obtain a regular data stream.

[0049] In this embodiment, the address parameters, port parameters and topic parameters of the Kafka cluster are determined through the preset Spark Streaming, and the address parameters, port parameters and topic parameters are used as data source configuration parameters to access heterogeneous energy data from multiple sources from Kafka in real time and generate an initial data stream.

[0050] It is understandable that this embodiment mainly adopts the Kafka data source connector configured with Spark Streaming. By setting the address parameter kafka_adress, port parameter kafka_port, and topic parameter topic of the Kafka cluster, it accesses energy data, reports and other structured data, logs, reports and other semi-structured data, images and other unstructured multi-source energy flow data from Kafka in real time. The required data source configuration parameters are determined through Config to generate the initial data stream.

[0051] Config=(kafka_adress,kafka_port,topic); Specifically, through a specific configuration method, it is possible to access multi-source heterogeneous energy flow data from Kafka in real time and stably, and can flexibly adapt to different Kafka cluster environments to ensure the continuity of energy data acquisition and meet the data access needs of various energy application scenarios. At the same time, with the help of Spark Streaming stream processing capabilities, the Kafka data source connector is configured, and the address, port, topic and other key parameters of the Kafka cluster are flexibly set to achieve real-time and stable access to multi-source heterogeneous energy flow data from Kafka, and convert it into the initial data stream, thereby improving the flexibility and stability of data access and providing continuous and reliable original data support for subsequent data processing processes.

[0052] In this embodiment, the step of performing distributed cleaning on the initial data stream according to different data source types includes: If the initial data stream is structured data, obtaining a standard parameter range corresponding to the structured data, and removing data outside the standard parameter range in the structured data; If the initial data stream is semi-structured data or unstructured data, the matching degree between the text corresponding to each data in the initial data stream and the preset regular expression is calculated, and the data whose matching degree does not meet the preset requirement is removed.

[0053] Understandably, to address the diversity of heterogeneous energy flow data, we primarily employ a variety of customized cleaning rules. For structured data, we filter data based on pre-set reasonable ranges; for semi-structured and unstructured data, we clean it by calculating matching scores using methods such as regular expressions. Furthermore, we precisely align the time series of cleaned multi-source data based on event time windows. These sophisticated and customized operations improve data quality, eliminate time skew between different data sources, and provide a highly accurate and consistent data foundation for subsequent energy analysis models.

[0054] Specifically, for numerical structured data such as equipment operating parameters, data outside the reasonable range is removed by judging isValid.

[0055] ; Here, x is the data, minValue and maxValue are the minimum and maximum values ​​determined based on the standard parameter range of the device operation. The standard parameter range corresponding to the structured data is determined by the two, so as to remove the data outside the standard parameter range.

[0056] Specifically, for semi-structured and unstructured data such as equipment logs and energy report texts, abnormal data is removed by calculating the matching formula matchDegree (used to calculate the matching degree between the text corresponding to each data in the initial data stream and the preset regular expression).

[0057] ; Among them, matchedLength(text,regex) is the length of the text text that matches the preset regular expression regex, and length(text) is the total length of the text text.

[0058] In this embodiment, according to the timestamp field in the energy data, a sliding window algorithm is adopted, where the event time window size is windowsSize, the sliding step is slideStep, and the energy data time series is , after window alignment, the regular data stream is output.

[0059] Among them, considering the particularity of the multi-source heterogeneous energy flow data that needs to be processed in this implementation, the generation time of structured data, semi-structured data and unstructured data in the heterogeneous energy flow data cannot be consistent. The generation time interval of structured data is relatively short (for example, the operation data automatically generated by the equipment operation generates new data every minute), while the generation time interval of unstructured data is relatively long (for example, manually filled text materials require a certain amount of time to fill in manually, but the image data in the unstructured data is efficiently generated). Therefore, when performing time series alignment on the above-mentioned heterogeneous energy flow data, it is necessary to consider the timestamps of the generation time of different data, and extract the common factors of the time intervals of each type of data that needs to be aligned based on the generation time intervals of different data sources, and use this to define a suitable event time window.

[0060] In this embodiment, setting a time window alone cannot fully meet the multi-source timing alignment effect of heterogeneous energy flow data. Therefore, it is necessary to further adjust the size of the time window based on the defined time window, combined with the data source of the flow data and its data corresponding situation, as well as the accuracy requirements of the data processing.

[0061] Specifically, for example, there are differences in the timeliness of the data content corresponding to manual text data and image data collected by the equipment. Manual text is usually limited to a period of time, and the data meets the requirements of real-time analysis and other operations, while image data usually refers to information within a moment or a continuous short period of time composed of multiple pictures. Due to the difference in their timeliness, different timeliness accuracy analysis requirements need to be adopted for the two. Therefore, it is necessary to increase the time window for manual text data and reduce the time window for image data, so as to obtain the time intervals of various types of data that meet the timeliness of multi-source heterogeneous energy flow data and the event conditions represented by them, and obtain the event time window suitable for the current situation.

[0062] This embodiment uses the preset Spark Streaming to access multi-source heterogeneous energy flow data from Kafka in real time, and generates an initial data stream based on this. The initial data stream is distributed cleaned according to the type of different data sources, and the multi-source time series alignment is performed on the distributed cleaned initial data stream according to the preset event time window to obtain a regular data stream. That is, in response to the diversity of heterogeneous energy data, a variety of customized cleaning rules are innovatively adopted, and the time series of multi-source data are accurately aligned based on the event time window. Through these sophisticated and customized operations, data quality is improved, time deviations between different data sources are eliminated, and a highly accurate and consistent data foundation is provided for subsequent energy analysis models.

[0063] like Figure 3 As shown, based on the first embodiment, a third embodiment of the heterogeneous energy data processing method of the present application is proposed. In this embodiment, the method further includes: S21, obtaining the generation rate of the heterogeneous energy flow data determined by the Kafka end, and determining the resource usage of the local system; S22, determining a micro-batch size adapted to the maximum operating efficiency of the local system under the current circumstances based on the generation rate, the resource usage, and a preset dynamic coefficient; S23, dividing the regular data stream into micro-batches using the Spark Streaming and the micro-batch specification.

[0064] In this embodiment, the cleaned and time-aligned regular data stream is intelligently divided into micro-batches based on factors such as the scale and processing complexity of the energy data to optimize memory usage and processing efficiency. Various energy analysis models pre-loaded in the memory are called, such as energy consumption prediction models and equipment fault diagnosis models. Each micro-batch data is mainly analyzed in depth, key features are accurately extracted, and a feature vector stream is formed to extract core features from massive energy data and reduce data dimensions.

[0065] Specifically, based on Spark Streaming technology, micro-batch processing is used to convert real-time streaming data into short batch processing jobs, providing a unified expression for real-time and offline data processing. By defining unified data processing logic and operation operators, the same set of code systems can be compatible with processing streaming data and batch data, greatly simplifying the processing process, reducing development and maintenance costs, and effectively improving the efficiency of stream-batch fusion.

[0066] The micro-batch division is determined based on the generation rate of heterogeneous energy flow data determined by the Kafka end and the local system resource usage, and the micro-batch rule S is calculated: ; in, 、 、 is the preset dynamic coefficient adjusted according to actual conditions, R is the generation rate of heterogeneous energy flow data, and The resource usage of the local system, specifically The remaining available resources ratio of the local system's CPU. To make up for the remaining available space in the local system's memory, the regular data stream is split according to the micro-batch rule S.

[0067] In this embodiment, the step of extracting corresponding features from the segmented data stream to obtain a feature vector stream includes: If the data in the segmented data stream is structured data, then according to the current data processing requirements, the key data dimension is selected as the feature to obtain the first processing result; If the data in the segmented data stream is semi-structured data or unstructured data, determining a keyword set from the data in the segmented data stream using a TF-IDF algorithm, determining a weight of each keyword in the keyword set, and using the weight as a feature to obtain a second processing result; The first processing result and the second processing result are combined into a feature vector, and a plurality of the feature vectors are output in time sequence to form a feature vector stream.

[0068] In this embodiment, custom parsers and converters are written for heterogeneous energy data of different formats and semantics, and multiple data parsing and conversion methods are used to integrate data of different structural types into a unified format, ensuring data integrity and accuracy, and significantly improving the ability to handle the diversity and complexity of heterogeneous energy data.

[0069] Among them, for structured energy data, such as electricity, gas and other metering data, select key data dimensions As features, form the corresponding feature vector , to obtain the first processing result.

[0070] Among them, key data dimensions mainly refer to the core attribute framework used to systematically classify, analyze and apply energy data.

[0071] These key data dimensions can be analyzed across regions (e.g., region × time × energy type) to reveal energy consumption patterns, optimize resource allocation, and provide decision support for smart energy systems. In practical applications, the combination of dimensions must be tailored to the specific scenario. For example, carbon emissions monitoring requires linking energy type with both time and space.

[0072] Specifically, when selecting key data dimensions for cross-analysis, it is necessary to divide them into primary analysis dimensions and auxiliary analysis dimensions. For example, from "region × time × energy type", region and energy type are used as auxiliary analysis dimensions, and time is used as the primary analysis dimension to generate a feature vector of (time, region × energy type).

[0073] In addition, the selection of key dimension data can be freely changed according to the needs of the actual application scenario. Specifically, the corresponding weight coefficient can be set. In different application scenarios, different main analysis dimensions and auxiliary analysis dimensions can be divided to flexibly respond to the data processing needs of different scenarios.

[0074] For semi-structured or unstructured energy data, the keyword set is obtained through the TF-IDF (Term Frequency-Inverse Document Frequency) algorithm. , calculate the weight of each keyword , taking the keyword weight as a feature to form a feature vector , to obtain the second processing result.

[0075] The TF-IDF algorithm combines term frequency (TF) and inverse document frequency (IDF) to evaluate the importance of words in a document collection, thereby extracting keywords. The specific steps include: Text preprocessing: Split documents into words or phrases, remove irrelevant characters such as punctuation, numbers, and stop words, and retain specific parts of speech (such as nouns and verbs) to improve keyword quality; Calculate word frequency using the formula: TF(t,d) = the total number of words in document d and the number of times word t appears in document d; this is used to measure the local importance of a word in a single document; Calculate the inverse document frequency using the formula: IDF(t,D) = log(number of documents containing word t + 1 total number of documents in the document set D) (add 1 to avoid the denominator being zero). This is used to penalize common words and highlight rare words. Calculate the TF-IDF value using the formula: TF-IDF(t,d,D)=TF(t,d)×IDF(t,D) to combine local and global importance; Sorting and keyword extraction steps: 1. Sort all words in the document by TF-IDF value in descending order. 2. Take the top-N (such as the top 5 or top 10) as keywords.

[0076] It should be noted that, in this embodiment, multiple keywords can be optimized and divided according to the classification structure of the knowledge graph of energy field knowledge (the keyword mainly refers to the general term for the same type of data after classification, and multiple replacement words are generated based on the general term). The keyword and the form of the TF-IDF algorithm can be combined to tend to select corresponding keywords, or the keywords corresponding to the knowledge graph can be used as a supplement on the basis of the keywords generated by the TF-IDF algorithm to further optimize the accuracy of feature vector generation when extracting semi-structured data and unstructured data.

[0077] In summary, the feature vectors extracted from structured and unstructured data are combined into a complete feature vector , multiple such feature vectors Output in sequence to form a feature vector stream.

[0078] In this embodiment, the step of dividing the regular data stream into micro-batches using Spark Streaming and the micro-batch specification includes: Pre-segmenting the regularized data stream using the Spark Streaming and micro-batch specifications; Evaluate the pre-segmentation results according to the multi-dimensional segmentation requirements corresponding to the data characteristics of the heterogeneous energy flow data; If the evaluation result does not meet the multi-dimensional partitioning requirements, the micro-batch specifications are optimized, and the regularized data stream is divided into micro-batches according to the optimized results.

[0079] It is understandable that, based on the data generation efficiency and system resource usage, the above formula can be used to calculate the micro-batch specifications for dynamic micro-batch segmentation that meet these requirements. However, when performing micro-batch segmentation on data, it is also necessary to consider whether the data obtained after micro-batch segmentation retains the data characteristics before segmentation. For example, data jumps caused by equipment failure or other faults within a certain period of time may correspond to short events, and the information carried by the data is highly continuous. Directly splitting the data into at least two micro-batches will destroy the continuity of the data. For example, a certain segment of data is semi-structured data, in which each important information is expressed in the form of corresponding multiple paragraphs of text description. Directly splitting these multiple paragraphs into multiple micro-batches will cause the continuity of the text data to be destroyed, resulting in significant deviations in the meaning represented by the data when viewing a single segment of data. For example, certain continuous signals or periodically output data have high integrity requirements. Splitting this data will destroy its integrity and lose the corresponding periodic changes in the data.

[0080] Therefore, in this application, the data is pre-segmented according to the micro-batch specifications, and the pre-segmentation results are evaluated for the corresponding dimensional requirements based on the multi-dimensional division requirements of the data characteristics (continuity, integrity or text context relevance) corresponding to the multi-source heterogeneous energy flow data. If the evaluation fails, the micro-batch specifications are optimized, and the regular data stream is segmented into micro-batches according to the optimized results.

[0081] In this embodiment, the step of evaluating the pre-segmentation result according to the multi-dimensional segmentation requirements corresponding to the data characteristics of the heterogeneous energy flow data includes: The multi-dimensional partitioning requirements include at least data continuity requirements, data context consistency requirements, and data integrity requirements; Obtaining application requirements proposed by the user during the current data micro-batch segmentation, and assigning corresponding requirement weights to the data continuity requirement, the data context coherence requirement, and the data integrity requirement based on the application requirements; Evaluating the pre-segmentation result according to the requirement weight, the data continuity requirement, the data context coherence requirement, and the data integrity requirement; If the evaluation result does not meet the multi-dimensional partitioning requirements, the step of optimizing the micro-batch size includes: If the evaluation result does not meet the multi-dimensional partitioning requirement, the data splitting position corresponding to the micro-batch specification is adjusted according to the requirement weight, the data continuity requirement, the data context consistency requirement and the data integrity requirement, so as to control the data length corresponding to each micro-batch specification to meet the multi-dimensional partitioning requirement.

[0082] It is understandable that when the evaluation results do not meet the requirements of multi-dimensional partitioning, it is necessary to optimize and adjust the micro-batch specifications in combination with the demand weights, data continuity requirements, data context consistency requirements and data integrity requirements. On the one hand, the length of the data after splitting corresponding to each micro-batch specification is controlled. Through the data length, it is ensured that the split stream data contains data that meets the requirements of multi-dimensional partitioning, thereby ensuring that the optimized micro-batch rules can meet the user's application requirements and multi-dimensional partitioning requirements.

[0083] Specifically, the optimization may include using each special requirement (continuity, completeness, or text context relevance) as a standard for micro-batch specifications to ensure that the data in each segmented data stream meets the above special requirements.

[0084] Specifically, the optimization content may also include, when segmenting the regular data stream, considering redundant segmentation with overlapping content of some segments, that is, the data at both ends of the corresponding time series of the segmented micro-batch data will overlap with part of the content in other micro-batch data.

[0085] Among them, it should be noted that since the user's application needs act as a weight to influence the tendency of each multi-dimensional division demand, different optimization effects can be produced according to actual demand conditions during the optimization of micro-batch specifications. At the same time, during the optimization process, it is also necessary to combine the utilization of system resources, that is, when calculating the micro-batch specifications, the system resource utilization is recorded, and after the micro-batch specifications are optimized and adjusted, the application needs, multi-dimensional division requirements, and system resource usage are combined to further judge whether the optimization results are reasonable. Specifically, after ensuring data integrity, avoid the data of each micro-batch specification being too long, and considering the possibility of redundant batch segmentation in the optimization, the data length when controlling the micro-batch specification optimization is less than the preset data length. The preset data length is obtained by proportional conversion based on the usage of system resources. If the proportion of system idle resources to all resources is greater than 50%, the data length will be increased, and if the proportion is less than 50%, the data length will be reduced. The specific increase and decrease ratios can be set according to actual needs.

[0086] It is understandable that the multi-dimensional division requirements corresponding to data characteristics include at least data continuity requirements (such as the continuity of fault mutation signals), data context coherence (specific text semantic content of semi-structured data) and data integrity requirements (the integrity of some short-cycle data).

[0087] It is understandable that data characteristics correspond to different types of demand properties, but they do not necessarily meet the needs of users. For example, long-cycle data does not need to ensure data integrity, and data with refined text content (each text paragraph is independent) does not need to consider contextual coherence, etc. Therefore, when actually performing micro-batch segmentation on the current heterogeneous energy flow data, the actual application needs of users can be introduced. This application requirement is mainly for users to make corresponding demand judgments and adjustments based on the content of the data currently generated and pushed in real time by the Kafka end. The main content of this application requirement includes setting demand weights corresponding to multi-dimensional segmentation requirements, and setting a certain tendency in a way. According to the demand weights and the corresponding multi-dimensional segmentation requirements, the results of micro-batch pre-segmentation are evaluated accordingly, and based on the results of the evaluation, the specific standards of the micro-batch specifications are further affected.

[0088] This embodiment obtains the generation rate of the heterogeneous energy flow data determined by the Kafka end and determines the resource usage of the local system; determines the micro-batch specification that adapts to the maximum operating efficiency of the local system under the current circumstances based on the generation rate, the resource usage and the preset dynamic coefficient; divides the regular data stream into micro-batches through the Spark Streaming and the micro-batch specification, so that the cleaned and time-aligned regular data stream is intelligently divided into micro-batches according to factors such as the scale and processing complexity of the energy data to optimize memory usage and processing efficiency, and conducts in-depth analysis on each micro-batch data, accurately extracts key features, forms a feature vector stream, and realizes the extraction of core features from massive energy data and reduces data dimensions.

[0089] In addition, the present application also proposes a heterogeneous energy data processing device, referring to Figure 4 , the heterogeneous energy data processing device includes: The data generation module 10 is used to access heterogeneous energy flow data from multiple sources from Kafka in real time through the preset Spark Streaming, and generate regular data streams according to the types of different data sources; The data processing module 20 is used to divide the regular data stream into micro batches through Spark Streaming, and extract corresponding features from the divided data stream to obtain a feature vector stream; The data storage module 30 is used to store the feature vector stream in a corresponding time series database so that relevant personnel can retrieve the feature vector stream from the time series database.

[0090] This embodiment uses the preset Spark Streaming to access multi-source heterogeneous energy flow data from Kafka in real time, and generates regularized data streams according to the types of different data sources; uses the Spark Streaming to split the regularized data stream into micro-batches, and extracts corresponding features from the split data streams to obtain feature vector streams; and stores the feature vector streams in a corresponding time series database so that relevant personnel can retrieve the feature vector streams from the time series database. That is, the preset Spark Streaming is used to access multi-source heterogeneous energy data from Kafka in real time, and the processed data streams are split into micro-batches by using Spark Streaming, and corresponding features are extracted from the split data streams respectively, thereby generating specific feature vector streams, and storing them. This ensures that the data accessed from the Kafka end is stored in the form of regularized data streams and feature vector streams after micro-batch processing according to the specific formats of different data sources, thereby improving the real-time processing effect of multi-source heterogeneous energy data, thereby ensuring accurate and timely data support for real-time monitoring, analysis, and decision-making.

[0091] It should be noted that the modules in the above-mentioned device can be used to implement the various steps in the above-mentioned method and achieve the corresponding technical effects, which will not be described in detail in this embodiment.

[0092] Reference Figure 5 , Figure 5 This is a schematic diagram of the structure of the equipment in the hardware operating environment involved in the embodiment of the present application.

[0093] like Figure 5 As shown, the device may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to implement communication between these components. The user interface 1003 may include a display and an input unit such as a keyboard. Optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory, such as a disk storage device. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0094] Those skilled in the art will understand that Figure 5 The structure shown in the figure does not constitute a limitation of the device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0095] like Figure 5 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a heterogeneous energy data processing program.

[0096] exist Figure 5 In the device shown, the network interface 1004 is mainly used to communicate data with the external network; the user interface 1003 is mainly used to receive user input instructions; the device calls the heterogeneous energy data processing program stored in the memory 1005 through the processor 1001 and performs the following operations: Through the preset Spark Streaming, it accesses heterogeneous energy flow data from multiple sources in real time from Kafka, and generates regular data streams based on the types of different data sources; By using Spark Streaming, the regularized data stream is divided into micro-batches, and corresponding features are extracted from the divided data stream to obtain a feature vector stream; The feature vector stream is stored in a corresponding time series database so that relevant personnel can retrieve the feature vector stream from the time series database.

[0097] Furthermore, the processor 1001 may call the heterogeneous energy data processing program stored in the memory 1005 and perform the following operations: Use the preset Spark Streaming to access heterogeneous energy flow data from multiple sources from Kafka in real time and generate the initial data stream; Perform distributed cleaning on the initial data stream according to the type of different data sources, and extract the timestamp field of the data provided by each data source in the heterogeneous energy flow data; Determine, based on the timestamp field, a common factor of the time intervals between the data sources providing data, and set a corresponding event time window based on the common factor and the accuracy requirement of the current data processing; According to the event time window, the distributed cleaned initial data stream is aligned with multiple sources to obtain a regular data stream.

[0098] Furthermore, the processor 1001 may call the heterogeneous energy data processing program stored in the memory 1005 and perform the following operations: If the initial data stream is structured data, obtaining a standard parameter range corresponding to the structured data, and removing data outside the standard parameter range in the structured data; If the initial data stream is semi-structured data or unstructured data, the matching degree between the text corresponding to each data in the initial data stream and the preset regular expression is calculated, and the data whose matching degree does not meet the preset requirement is removed.

[0099] Furthermore, the processor 1001 may call the heterogeneous energy data processing program stored in the memory 1005 and perform the following operations: Obtaining a generation rate of the heterogeneous energy flow data determined by the Kafka end, and determining resource usage of the local system; Determining a micro-batch size adapted to the maximum operating efficiency of the local system under current circumstances based on the generation rate, the resource usage, and a preset dynamic coefficient; The regular data stream is divided into micro-batches using the Spark Streaming and the micro-batch specification.

[0100] Furthermore, the processor 1001 may call the heterogeneous energy data processing program stored in the memory 1005 and perform the following operations: Pre-segmenting the regularized data stream using the Spark Streaming and micro-batch specifications; Evaluate the pre-segmentation results according to the multi-dimensional segmentation requirements corresponding to the data characteristics of the heterogeneous energy flow data; If the evaluation result does not meet the multi-dimensional partitioning requirements, the micro-batch specifications are optimized, and the regularized data stream is divided into micro-batches according to the optimized results.

[0101] Furthermore, the processor 1001 may call the heterogeneous energy data processing program stored in the memory 1005 and perform the following operations: The multi-dimensional partitioning requirements include at least data continuity requirements, data context consistency requirements, and data integrity requirements; Obtaining application requirements proposed by the user during the current data micro-batch segmentation, and assigning corresponding requirement weights to the data continuity requirement, the data context coherence requirement, and the data integrity requirement based on the application requirements; Evaluating the pre-segmentation result according to the requirement weight, the data continuity requirement, the data context coherence requirement, and the data integrity requirement; If the evaluation result does not meet the multi-dimensional partitioning requirement, the data splitting position corresponding to the micro-batch specification is adjusted according to the requirement weight, the data continuity requirement, the data context consistency requirement and the data integrity requirement, so as to control the data length corresponding to each micro-batch specification to meet the multi-dimensional partitioning requirement.

[0102] Furthermore, the processor 1001 may call the heterogeneous energy data processing program stored in the memory 1005 and perform the following operations: If the data in the segmented data stream is structured data, then according to the current data processing requirements, the key data dimension is selected as the feature to obtain the first processing result; If the data in the segmented data stream is semi-structured data or unstructured data, determining a keyword set from the data in the segmented data stream using a TF-IDF algorithm, determining a weight of each keyword in the keyword set, and using the weight as a feature to obtain a second processing result; The first processing result and the second processing result are combined into a feature vector, and a plurality of the feature vectors are output in time sequence to form a feature vector stream.

[0103] Furthermore, the processor 1001 may call the heterogeneous energy data processing program stored in the memory 1005 and perform the following operations: Determine the load and idle resources of the local system and prioritize the current data tasks; Allocate a corresponding number of executors according to the load condition, the priority, and the amount of idle resources; determining access frequency and data characteristics of the heterogeneous energy flow data; Configuring a corresponding storage level according to the access frequency and the data characteristics; According to the configured executor and storage level, the feature vector stream is stored in the corresponding time series database.

[0104] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0105] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0106] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, a computer program) stored thereon, and the computer-readable program instructions are used to execute the heterogeneous energy data processing method in the above-mentioned embodiment.

[0107] The computer-readable storage medium provided herein may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including, but not limited to, wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0108] The computer-readable storage medium may be included in the heterogeneous energy data processing device; or may exist independently without being assembled into the heterogeneous energy data processing device.

[0109] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a heterogeneous energy data processing device, the heterogeneous energy data processing device: Through the preset Spark Streaming, it accesses heterogeneous energy flow data from multiple sources in real time from Kafka, and generates regular data streams based on the types of different data sources; By using Spark Streaming, the regularized data stream is divided into micro-batches, and corresponding features are extracted from the divided data stream to obtain a feature vector stream; The feature vector stream is stored in a corresponding time series database so that relevant personnel can retrieve the feature vector stream from the time series database.

[0110] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0111] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0112] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0113] The computer-readable storage medium provided in this application stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned heterogeneous energy data processing method, thereby resolving the technical challenges of heterogeneous energy data processing. Compared to the prior art, the computer-readable storage medium provided in this application offers the same beneficial effects as the heterogeneous energy data processing method provided in the aforementioned embodiments, and will not be further elaborated upon here.

[0114] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

[0115] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0116] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0117] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0118] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for processing heterogeneous energy data, characterized in that: The heterogeneous energy data processing method comprises the following steps: Through the preset Spark Streaming, it accesses heterogeneous energy flow data from multiple sources in real time from Kafka, and generates regular data streams based on the types of different data sources; By using Spark Streaming, the regularized data stream is divided into micro-batches, and corresponding features are extracted from the divided data stream to obtain a feature vector stream; The feature vector stream is stored in a corresponding time series database so that relevant personnel can retrieve the feature vector stream from the time series database.

2. The method according to claim 1, wherein The steps of accessing heterogeneous energy flow data from multiple sources from Kafka in real time through the preset Spark Streaming and generating regularized data streams according to the types of different data sources include: Use the preset Spark Streaming to access heterogeneous energy flow data from multiple sources from Kafka in real time and generate the initial data stream; Perform distributed cleaning on the initial data stream according to the type of different data sources, and extract the timestamp field of the data provided by each data source in the heterogeneous energy flow data; Determine, based on the timestamp field, a common factor of the time intervals between the data sources providing data, and set a corresponding event time window based on the common factor and the accuracy requirement of the current data processing; According to the event time window, the distributed cleaned initial data stream is aligned with multiple sources to obtain a regular data stream.

3. The method according to claim 2, wherein The step of performing distributed cleaning on the initial data stream according to different data source types includes: If the initial data stream is structured data, obtaining a standard parameter range corresponding to the structured data, and removing data outside the standard parameter range in the structured data; If the initial data stream is semi-structured data or unstructured data, the matching degree between the text corresponding to each data in the initial data stream and the preset regular expression is calculated, and the data whose matching degree does not meet the preset requirement is removed.

4. The method according to claim 1, wherein The step of dividing the regular data stream into micro-batches by Spark Streaming includes: Obtaining a generation rate of the heterogeneous energy flow data determined by the Kafka end, and determining resource usage of the local system; Determining a micro-batch size adapted to the maximum operating efficiency of the local system under current circumstances based on the generation rate, the resource usage, and a preset dynamic coefficient; The regular data stream is divided into micro-batches using the Spark Streaming and the micro-batch specification.

5. The method according to claim 4, wherein The step of dividing the regular data stream into micro-batches using Spark Streaming and the micro-batch specification includes: Pre-segmenting the regularized data stream using the Spark Streaming and micro-batch specifications; Evaluate the pre-segmentation results according to the multi-dimensional segmentation requirements corresponding to the data characteristics of the heterogeneous energy flow data; If the evaluation result does not meet the multi-dimensional partitioning requirements, the micro-batch specifications are optimized, and the regularized data stream is divided into micro-batches according to the optimized results.

6. The method according to claim 5, wherein The step of evaluating the pre-segmentation result according to the multi-dimensional segmentation requirements corresponding to the data characteristics of the heterogeneous energy flow data includes: The multi-dimensional partitioning requirements include at least data continuity requirements, data context consistency requirements, and data integrity requirements; Obtaining application requirements proposed by the user during the current data micro-batch segmentation, and assigning corresponding requirement weights to the data continuity requirement, the data context coherence requirement, and the data integrity requirement based on the application requirements; Evaluating the pre-segmentation result according to the requirement weight, the data continuity requirement, the data context coherence requirement, and the data integrity requirement; If the evaluation result does not meet the multi-dimensional partitioning requirements, the step of optimizing the micro-batch size includes: If the evaluation result does not meet the multi-dimensional partitioning requirement, the data splitting position corresponding to the micro-batch specification is adjusted according to the requirement weight, the data continuity requirement, the data context consistency requirement and the data integrity requirement, so as to control the data length corresponding to each micro-batch specification to meet the multi-dimensional partitioning requirement.

7. The method according to claim 4, wherein The step of extracting corresponding features from the segmented data stream to obtain a feature vector stream includes: If the data in the segmented data stream is structured data, then according to the current data processing requirements, the key data dimension is selected as the feature to obtain the first processing result; If the data in the segmented data stream is semi-structured data or unstructured data, determining a keyword set from the data in the segmented data stream using a TF-IDF algorithm, determining a weight of each keyword in the keyword set, and using the weight as a feature to obtain a second processing result; The first processing result and the second processing result are combined into a feature vector, and a plurality of the feature vectors are output in time sequence to form a feature vector stream.

8. The method according to claim 1, wherein The step of storing the feature vector stream in a corresponding time series database includes: Determine the load and idle resources of the local system and prioritize the current data tasks; Allocate a corresponding number of executors according to the load condition, the priority, and the amount of idle resources; determining access frequency and data characteristics of the heterogeneous energy flow data; Configuring a corresponding storage level according to the access frequency and the data characteristics; According to the configured executor and storage level, the feature vector stream is stored in the corresponding time series database.

9. A heterogeneous energy data processing device, characterized in that: The heterogeneous energy data processing device includes: a memory, a processor, and a heterogeneous energy data processing program stored in the memory and executable on the processor, wherein the heterogeneous energy data processing program is configured to implement the steps of the heterogeneous energy data processing method according to any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium stores a program for implementing the heterogeneous energy data processing method. The program for implementing the heterogeneous energy data processing method is executed by a processor to implement the steps of the heterogeneous energy data processing method according to any one of claims 1 to 8.

Images