Batch flow integrated data processing system and processing method
A data processing system and data processing technology, which is applied in the field of batch-stream integrated data processing systems, can solve the problem of separate processing of batch data and stream data, slow processing efficiency, and inability to satisfy complex, relational, and non-relational data distribution Acquisition, calculation, filtering and analysis, etc., to achieve the effect of enriching data analysis interface and analysis function, reducing establishment time and shortening task extraction time
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
Method used
Image
Examples
Embodiment 1
[0047]The batch-flow integrated data processing system of the present invention includes a front-end interaction subsystem, a configuration library, and a back-end service subsystem. The front-end interaction subsystem interacts with administrators and users through the front-end interaction interface, and is used to support administrators to configure component models, components The model is used to define the rules of data reading, data processing and data writing, and is used to support users to configure tasks and submit task requests based on the component model; the configuration library interacts with the front-end interaction module to store component models and configured tasks; The end service subsystem interacts with the front-end interaction subsystem, and is used to execute tasks and monitor task execution based on task requests, generate task operation new and abnormal alarms, and return task operation information and exceptions based on task query requests from t...
Embodiment 2
[0058] The batch-flow integrated data processing method of the present invention processes batch data and flow data through the batch-flow integrated data processing system disclosed in Embodiment 1, and the method includes the following steps:
[0059] S100. Configure a component model, where the component model is used to define rules for data reading, data processing, and data writing;
[0060] S200. Create a task based on the component model and submit a task request;
[0061] S300. Execute tasks based on task requests and monitor task operation, generate task operation information and abnormal alarms, and return task operation information and abnormal alarms based on task query requests, follow the following processing logic when executing tasks: stream processing as processing logic, batch as A special case of flow exists. When batch data flows in, the batch data is converted into flow data through the time window to realize the unification of batch flow data.
[0062] ...
PUM
Abstract
Description
Claims
Application Information
- R&D Engineer
- R&D Manager
- IP Professional
- Industry Leading Data Capabilities
- Powerful AI technology
- Patent DNA Extraction
Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.
© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com