Streaming data processing system based on metadata

A streaming data and processing system technology, applied in the field of data processing, can solve the problems that the process cannot be implemented quickly, it is difficult to fully utilize, and the lack of data management, etc., to achieve the effect of shortening the implementation delivery cycle, lowering the threshold, and reducing maintenance costs

Pending Publication Date: 2021-12-21
CHINACCS INFORMATION IND
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

When there are many streaming data processing businesses, there is a lack of unified management, rapid business evolution, and rapid changes in requirements. Designing new data processing processes or modifying existing processes cannot be implemented quickly
[0004] In view of the rapid development of smart security, smart city, smart agriculture, smart transportation and other industries, a large amount of data has been accumulated. The aggregation and management of these data often needs to be completed by each platform, which creates a lot of repetitive work and lack of data management. , it is difficult to take full advantage of

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Streaming data processing system based on metadata
  • Streaming data processing system based on metadata
  • Streaming data processing system based on metadata

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0043] Embodiment one, see Figure 1-Figure 5 , the present invention is achieved through the following technical solutions: a streaming data processing system based on metadata, including a metadata management module, the metadata management module can define the structure of the metamodel according to the form and storage mode of the data; The metamodel includes: an interface metamodel, a logical metamodel, and a physical metamodel; according to the metamodel, the structure of corresponding metadata is defined, and the metadata includes: interface metadata, logical metadata, and physical metadata;

[0044] Metadata management can help enterprise personnel to clearly see what data the enterprise has and where it exists, and at the same time help to sort out the enterprise data dictionary, quickly query and locate data. Such as figure 2As shown, an interface package is defined, which defines the meta-model of the interface model, including the meta-metadata of the interface ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a streaming data processing system based on metadata, and belongs to the technical field of data processing. According to the technical scheme, the streaming data processing system based on metadata comprises; a metadata management module, where the metadata management module can define the structure of a meta-model according to the form and storage mode of data; the process management module that comprises a process management component and a process arrangement component; the flow scheduling module that can obtain a flow execution graph, distributes execution tasks according to the flow execution graph and sends out corresponding execution instructions; and the process execution module that can receive the execution instruction, execute the corresponding execution task and start a corresponding operator to start calculation. The invention has the beneficial effects that metadata definition, flow definition and flow arrangement are carried out through a visualization method, and a streaming processing flow is created by using a dragging component and a model-driven logic.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a stream data processing system based on metadata. Background technique [0002] At present, with the rapid development of the Internet and Internet of Things technology, information browsing, retrieval, data reporting of Internet of Things devices, and many common life products such as e-commerce and the Internet have all been online. The requirements for real-time are further improved, and the interaction and communication of information is developing from point-to-point to information chain and even information network. This will inevitably bring about cross-correlation of data in various dimensions, and data explosion is inevitable. Therefore, stream processing comes into being. It solves the problem of real-time framework and large-scale calculation of data. [0003] At present, real-time processing requires mastering various big data technologies and stream process...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/2455G06F16/28G06F8/34G06F8/38
CPCG06F16/24568G06F16/284G06F16/287G06F8/34G06F8/38
Inventor 陶志强魏晟坤蒲凌云马新成
Owner CHINACCS INFORMATION IND
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products