Batch flow integrated data processing system and processing method

A data processing system and data processing technology, which is applied in the field of batch-stream integrated data processing systems, can solve the problem of separate processing of batch data and stream data, slow processing efficiency, and inability to satisfy complex, relational, and non-relational data distribution Acquisition, calculation, filtering and analysis, etc., to achieve the effect of enriching data analysis interface and analysis function, reducing establishment time and shortening task extraction time

Inactive Publication Date: 2022-02-18
INSPUR SOFTWARE TECH CO LTD
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0002] With the continuous development of computer technology and the continuous improvement of informatization, such rapidly growing, huge and complex data resources have brought great challenges to traditional data analysis and processing technologies.
There are some defects in traditional data processing software, mainly including: using a pseudo-distributed architecture, batch data and stream data need to be processed separately, and the processing efficiency is slow, which can no longer satisfy complex, relational, and non-relational data. Distributed acquisition, computation, filtering and analysis

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Batch flow integrated data processing system and processing method
  • Batch flow integrated data processing system and processing method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0047]The batch-flow integrated data processing system of the present invention includes a front-end interaction subsystem, a configuration library, and a back-end service subsystem. The front-end interaction subsystem interacts with administrators and users through the front-end interaction interface, and is used to support administrators to configure component models, components The model is used to define the rules of data reading, data processing and data writing, and is used to support users to configure tasks and submit task requests based on the component model; the configuration library interacts with the front-end interaction module to store component models and configured tasks; The end service subsystem interacts with the front-end interaction subsystem, and is used to execute tasks and monitor task execution based on task requests, generate task operation new and abnormal alarms, and return task operation information and exceptions based on task query requests from t...

Embodiment 2

[0058] The batch-flow integrated data processing method of the present invention processes batch data and flow data through the batch-flow integrated data processing system disclosed in Embodiment 1, and the method includes the following steps:

[0059] S100. Configure a component model, where the component model is used to define rules for data reading, data processing, and data writing;

[0060] S200. Create a task based on the component model and submit a task request;

[0061] S300. Execute tasks based on task requests and monitor task operation, generate task operation information and abnormal alarms, and return task operation information and abnormal alarms based on task query requests, follow the following processing logic when executing tasks: stream processing as processing logic, batch as A special case of flow exists. When batch data flows in, the batch data is converted into flow data through the time window to realize the unification of batch flow data.

[0062] ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a batch-stream integrated data processing system and processing method, belongs to the technical field of big data processing analysis, and aims to solve the technical problem of how to provide a distributed batch-stream integrated data processing mode and improve the real-time performance and convenience of data processing calculation. The system comprises a front-end interaction subsystem used for supporting an administrator to configure a component model, and the component model is used for defining rules of data reading, data processing and data writing and supporting a user to configure a task based on the component model and submit a task request; the configuration library is used for storing component models and configured tasks; the rear-end service subsystem is used for executing a task based on a task request, monitoring task operation, generating a task operation new type and an abnormal alarm, and returning task operation information and the abnormal alarm, and the processing logic of the rear-end service subsystem is as follows: stream processing is taken as the processing logic, a batch is taken as a special case of the stream, and when batch data flows in, the task operation new type and the abnormal alarm are generated; and converting the batch data into stream data through a time window.

Description

technical field [0001] The invention relates to the technical field of big data processing and analysis, in particular to a batch-flow integrated data processing system and processing method. Background technique [0002] With the continuous development of computer technology and the continuous improvement of informatization, such rapidly growing, huge and complex data resources have brought great challenges to traditional data analysis and processing technologies. There are some defects in traditional data processing software, mainly including: using a pseudo-distributed architecture, batch data and stream data need to be processed separately, and the processing efficiency is slow, which can no longer satisfy complex, relational, and non-relational data. Distributed acquisition, computation, filtering and analysis. Therefore, distributed data processing systems are attracting more and more people's attention, and among the technologies related to big data computing, the di...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/445G06F9/448G06F9/451G06F16/25
CPCG06F9/4451G06F9/451G06F9/4482G06F16/252G06F16/254
Inventor 袁富强路国隋李存冰王方
Owner INSPUR SOFTWARE TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap