Stream batch data rapid fusion method

Through the integrated flow batch architecture and related technical means, the problems of slow fusion of flow batch data and difficulty in management are solved, rapid fusion and efficient processing are achieved, and the decision-making efficiency and data processing capabilities of the enterprise are improved.

CN119026070APending Publication Date: 2024-11-26GUANGDONG POWER GRID CO LTD +1

Patent Information

Application Number
CN202410910249.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-09
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The fusion speed of existing stream batch data is slow during the rapid fusion process, and there are difficulties in managing and controlling the processing and analysis of stream batch data.

Method used

Adopt streaming and batch integrated architecture, streaming and batch unified interface, data preprocessing and cache, real-time streaming data windowing, data format standardization, parallel computing and distributed processing, real-time streaming data storage and real-time streaming data index to realize streaming data and batches Fast fusion and efficient processing of data.

Benefits of technology

Accelerate the speed of streaming batch data processing, improve data processing efficiency and performance, enable enterprises to make real-time decisions and respond to emergencies more quickly, simplify data access complexity, reduce integration complexity and cost, and meet large-scale Data processing requirements.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention discloses a stream batch data fast fusion method which comprises a stream batch integrated framework, a stream batch unified interface, data preprocessing and caching, real-time stream data windowing, data format standardization, parallel computing and distributed processing, real-time stream data storage and real-time stream data indexing. The stream batch integrated architecture mainly comprises the steps of firstly collecting various stream batch data and batch processing data, then analyzing and formatting the data, and then integrating real-time stream data processing and batch data processing through the stream batch integrated architecture, generally comprising a real-time calculation engine and a batch processing engine. According to the method, real-time streaming data and batch data can be quickly fused through a streaming and batch integrated architecture, so that an enterprise can more quickly carry out real-time decision making and coping with emergencies, the advantages of real-time processing and batch processing can be fully utilized, resources can be flexibly allocated according to different requirements, the data processing efficiency and performance are improved, and the data processing cost is reduced. And the data analysis is more comprehensive.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of data fusion, and in particular relates to a method for fast fusion of stream and batch data. Background Art

[0002] Rapid fusion of stream and batch data refers to the process of quickly integrating and processing real-time stream data and batch data. Real-time stream data refers to data that is continuously generated and transmitted, while batch data refers to data accumulated within a specific period of time. The goal of rapid fusion of stream and batch data is to achieve real-time fusion and analysis of these two types of data so that enterprises can make decisions, provide services or perform other business processing more quickly. Rapid fusion of stream and batch data is of great significance for real-time big data processing and business applications. It can help enterprises better understand and use data, improve decision-making efficiency, and provide better services.

[0003] In the existing rapid fusion process of stream and batch data, the fusion speed is slow, and there are still great problems in managing and controlling the processing and analysis of stream and batch data. Therefore, the analysis and management of stream and batch data after the fusion is completed needs to be greatly improved. Summary of the invention

[0004] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a method for rapid fusion of stream and batch data, which has the advantage of accelerating the processing speed of stream and batch data.

[0005] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a method for rapid fusion of stream and batch data, including a stream-batch integrated architecture, a stream-batch unified interface, data preprocessing and caching, real-time stream data windowing, data format standardization, parallel computing and distributed processing, real-time stream data storage and real-time stream data indexing. The stream-batch integrated architecture mainly includes: first collecting various stream batch data and batch processing data, then parsing and formatting the data, and then integrating real-time stream data processing and batch data processing through the stream-batch integrated architecture, which usually includes a real-time computing engine and a batch processing engine, and can process and analyze real-time stream data and batch data on the same platform. The use of the stream-batch integrated architecture can make stream data and batch data processed in the same system, reduce the complexity of data conversion and integration, and improve processing efficiency.

[0006] Preferably, a unified stream-batch interface: After completing the stream-batch integrated architecture, a unified interface is designed that can support the input and output of both stream data and batch data. This interface can integrate stream data and batch data together, making the data processing flow more efficient and unified. Through the unified interface, the processing and fusion process of stream data and batch data can be better managed and controlled, thereby improving the efficiency of data processing.

[0007] Preferably, data preprocessing and caching: in the process of processing and merging stream data and batch data, preprocessing the stream data and caching it can speed up the processing of the stream data. At the same time, preprocessing the batch data allows it to be merged with the stream data more quickly. Using caching technology can improve the locality and reusability of data, speed up data processing, and thus merge stream and batch data more quickly.

[0008] Preferably, real-time streaming data is windowed: real-time streaming data is processed in a windowed manner, and streaming data can be converted into batch data for processing. This method can make streaming data easier to merge and analyze with batch data. Through windowing, real-time streaming data can be segmented by time or other dimensions and converted into batch data, thereby facilitating processing and fusion by a batch processing engine.

[0009] Preferably, data format standardization: Standardizing the formats of stream data and batch data before merging them can make the two types of data easier to merge and analyze. Standardized data formats can reduce the complexity of data conversion and integration and improve the efficiency of data processing. Through a unified data format, the processing and analysis process of stream and batch data can be better managed and controlled.

[0010] Preferably, parallel computing and distributed processing: The use of parallel computing and distributed processing methods can speed up the processing of streaming batch data. By decomposing the data and distributing it to multiple computing nodes for processing, the efficiency of data processing can be improved. Using parallel computing and distributed processing technology, streaming batch data can be fused and analyzed more quickly, thereby improving the processing speed.

[0011] Preferably, real-time streaming data storage: real-time streaming data is stored so as to be merged with batch data. Storing real-time streaming data allows the real-time streaming data to be processed by a batch processing engine, making it easier to merge and analyze with batch data. Through real-time streaming data storage, the processing and analysis process of real-time streaming data can be better managed and controlled.

[0012] Preferably, real-time stream data indexing: indexing real-time stream data can speed up the retrieval and analysis of real-time stream data. By indexing real-time stream data, the data retrieval speed can be increased, and fusion and analysis can be performed faster. Real-time stream data indexing can make it easier to merge and analyze real-time stream data with batch data.

[0013] Compared with the prior art, the present invention has the following beneficial effects:

[0014] 1. Through the integrated stream-batch architecture, real-time stream data and batch data can be quickly integrated, allowing enterprises to make real-time decisions and respond to emergencies more quickly. They can fully utilize the advantages of real-time processing and batch processing, flexibly allocate resources according to different needs, improve data processing efficiency and performance, and make data analysis more comprehensive.

[0015] 2. Through the unified stream-batch interface, the access methods of stream data and batch data are unified, which simplifies the complexity of data access and improves the accessibility and availability of data. Through the unified stream-batch interface, data from different data sources can be more easily integrated together, reducing the complexity and cost of integration and improving the efficiency of data integration.

[0016] 3. Through data preprocessing and caching, real-time streaming data windowing and data format standardization, the locality and reusability of data can be improved, the data processing speed can be accelerated, and the streaming and batch data can be integrated faster. The processing and analysis process of streaming and batch data can be better managed and controlled. The real-time streaming data can be segmented by time or other dimensions and converted into batch data, which is convenient for batch processing engines to process and integrate.

[0017] 4. Through parallel computing and distributed processing, large-scale data can be processed effectively. By processing data simultaneously on multiple computers, it can meet the needs of large-scale data processing and meet the growing data processing needs of enterprises. Large-scale data can be divided into multiple small tasks for parallel processing, thereby improving the speed and efficiency of data processing and shortening the time of data processing. DETAILED DESCRIPTION

[0018] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0019] The present invention provides a technical solution: a method for rapid fusion of stream and batch data, including a stream-batch integrated architecture, a stream-batch unified interface, data preprocessing and caching, real-time stream data windowing, data format standardization, parallel computing and distributed processing, real-time stream data storage and real-time stream data indexing. The stream-batch integrated architecture mainly includes: first collecting various stream batch data and batch processing data, then parsing and formatting the data, and then integrating real-time stream data processing and batch data processing through the stream-batch integrated architecture, usually including a real-time computing engine and a batch processing engine, which can process and analyze real-time stream data and batch data on the same platform. The stream-batch integrated architecture can enable stream data and batch data to be processed in the same system, reducing the complexity of data conversion and integration and improving processing efficiency.

[0020] Unified stream and batch interface: After completing the stream-batch integrated architecture, a unified interface is designed to support the input and output of both stream data and batch data. This interface can integrate stream data and batch data together, making the data processing process more efficient and unified. Through the unified interface, the processing and fusion process of stream data and batch data can be better managed and controlled, thereby improving the efficiency of data processing.

[0021] Data preprocessing and caching: In the process of processing and fusing streaming data and batch data, preprocessing the streaming data and caching it can speed up the processing of streaming data. At the same time, preprocessing the batch data allows it to be fused with streaming data more quickly. Using caching technology can improve the locality and reusability of data, speed up data processing, and thus fuse streaming and batch data faster.

[0022] Windowing of real-time streaming data: By windowing real-time streaming data, streaming data can be converted into batch data for processing. This method makes it easier to merge and analyze streaming data with batch data. Through windowing, real-time streaming data can be segmented by time or other dimensions and converted into batch data, making it easier for batch processing engines to process and merge them.

[0023] Data format standardization: Before merging streaming data and batch data, standardizing the formats of streaming data and batch data can make the two types of data easier to merge and analyze. Standardizing the data format can reduce the complexity of data conversion and integration and improve the efficiency of data processing. Through a unified data format, the processing and analysis process of streaming and batch data can be better managed and controlled.

[0024] Parallel computing and distributed processing: The use of parallel computing and distributed processing methods can speed up the processing of streaming batch data. By decomposing the data and distributing it to multiple computing nodes for processing, the efficiency of data processing can be improved. Using parallel computing and distributed processing technology, streaming batch data can be fused and analyzed more quickly, thereby improving processing speed.

[0025] Real-time streaming data storage: Storing real-time streaming data so that it can be merged with batch data. Storing real-time streaming data allows the real-time streaming data to be processed by the batch processing engine, making it easier to merge and analyze with batch data. Through real-time streaming data storage, the processing and analysis process of real-time streaming data can be better managed and controlled.

[0026] Real-time streaming data indexing: Indexing real-time streaming data can speed up the retrieval and analysis of real-time streaming data. By indexing real-time streaming data, the data retrieval speed can be improved, and fusion and analysis can be performed faster. Real-time streaming data indexing can make it easier to merge and analyze real-time streaming data with batch data.

[0027] The working principle and usage process of the present invention are as follows: first, various stream batch data and batch processing data are collected, then the data is parsed and formatted, and then real-time stream data processing and batch data processing are integrated through a stream-batch integrated architecture, which usually includes a real-time computing engine and a batch processing engine. Real-time stream data and batch data can be processed and analyzed on the same platform. The use of a stream-batch integrated architecture allows stream data and batch data to be processed in the same system.

[0028] Secondly, after completing the stream-batch integrated architecture, a unified interface is designed that can integrate stream data and batch data together, making the data processing process more efficient and unified. In the process of processing and merging stream data and batch data, the stream data is preprocessed and cached, and then the real-time stream data is windowed and converted into batch data for processing. In addition, before merging stream data and batch data, the formats of stream data and batch data are standardized to improve data processing efficiency.

[0029] Finally, by using parallel computing and distributed processing methods, the data is decomposed and distributed to multiple computing nodes for processing, and the real-time streaming data is stored so that it can be merged with batch data. By indexing the real-time streaming data, the data retrieval speed can be improved, and the fusion and analysis can be performed faster.

[0030] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for rapid fusion of stream and batch data, including a stream-batch integrated architecture, a stream-batch unified interface, data preprocessing and caching, real-time stream data windowing, data format standardization, parallel computing and distributed processing, real-time stream data storage and real-time stream data indexing, characterized in that: The stream-batch integration architecture mainly includes: first collecting various stream-batch data and batch processing data, then parsing and formatting the data, and then integrating real-time stream data processing and batch data processing through the stream-batch integration architecture, which usually includes a real-time computing engine and a batch processing engine, and can process and analyze real-time stream data and batch data on the same platform. The use of the stream-batch integration architecture can enable stream data and batch data to be processed in the same system, reducing the complexity of data conversion and integration and improving processing efficiency.

2. The method for fast fusion of stream and batch data according to claim 1, characterized in that: Unified stream and batch interface: After completing the stream-batch integrated architecture, a unified interface is designed to support the input and output of both stream data and batch data. This interface can integrate stream data and batch data together, making the data processing process more efficient and unified. Through the unified interface, the processing and fusion process of stream data and batch data can be better managed and controlled, thereby improving the efficiency of data processing.

3. The method for fast fusion of stream and batch data according to claim 1, characterized in that: Data preprocessing and caching: In the process of processing and fusing streaming data and batch data, preprocessing the streaming data and caching it can speed up the processing of streaming data. At the same time, preprocessing the batch data allows it to be fused with streaming data more quickly. Using caching technology can improve the locality and reusability of data, speed up data processing, and thus fuse streaming and batch data faster.

4. The method for fast fusion of stream and batch data according to claim 1, characterized in that: Windowing of real-time streaming data: By windowing real-time streaming data, streaming data can be converted into batch data for processing. This method makes it easier to merge and analyze streaming data with batch data. Through windowing, real-time streaming data can be segmented by time or other dimensions and converted into batch data, making it easier for batch processing engines to process and merge them.

5. The method for fast fusion of stream and batch data according to claim 1, characterized in that: Data format standardization: Before merging streaming data and batch data, standardizing the formats of streaming data and batch data can make the two types of data easier to merge and analyze. Standardizing the data format can reduce the complexity of data conversion and integration and improve the efficiency of data processing. Through a unified data format, the processing and analysis process of streaming and batch data can be better managed and controlled.

6. The method for fast fusion of stream and batch data according to claim 1, characterized in that: Parallel computing and distributed processing: The use of parallel computing and distributed processing methods can speed up the processing of streaming batch data. By decomposing the data and distributing it to multiple computing nodes for processing, the efficiency of data processing can be improved. Using parallel computing and distributed processing technology, streaming batch data can be fused and analyzed more quickly, thereby improving processing speed.

7. The method for fast fusion of stream and batch data according to claim 1, characterized in that: Real-time streaming data storage: Storing real-time streaming data so that it can be merged with batch data. Storing real-time streaming data allows the real-time streaming data to be processed by the batch processing engine, making it easier to merge and analyze with batch data. Through real-time streaming data storage, the processing and analysis process of real-time streaming data can be better managed and controlled.

8. The method for fast fusion of stream and batch data according to claim 1, characterized in that: Real-time streaming data indexing: Indexing real-time streaming data can speed up the retrieval and analysis of real-time streaming data. By indexing real-time streaming data, the data retrieval speed can be improved, and fusion and analysis can be performed faster. Real-time streaming data indexing can make it easier to merge and analyze real-time streaming data with batch data.

Citation Information

Patent Citations

  • Operation data processing method and system

    CN103761309A

  • Information processing method and device for batch stream fusion, and storage medium

    CN113918771A

  • Integrated processing system based on big data stream batch and working method thereof

    CN116186173A

  • Data real-time efficient fusion processing method based on FCM clustering algorithm model

    CN117112871A

  • Stream batch integrated data processing method and device, equipment and storage medium

    CN117271012A

Cited By

  • Multi-system dynamic scheduling processing method for public service real-time streaming data

    CN120471365A

  • Streaming and batch adaptive fusion processing method and system for heterogeneous data source

    CN122309091A