Unlock instant, AI-driven research and patent intelligence for your innovation.

Data processing method and system

A data processing system and data processing technology, applied in the field of data processing, can solve problems such as inconsistency and data processing delay, and achieve the effect of avoiding data processing delay and inconsistency

Inactive Publication Date: 2020-12-01
长沙树根互联技术有限公司
View PDF5 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] However, the inventors have found through research that in the prior art, when the metadata needs to be updated when the task is restarted, it is necessary to stop the task of real-time data processing and then update the metadata. Data processing delays and inconsistencies

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method and system
  • Data processing method and system
  • Data processing method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0036] In the existing technology, the acquired real-time data is processed through the stateful distributed real-time data processing framework - Flink, and the Flink distributed real-time processing framework uses three methods (event-driven analysis), use different computing resources (K8s, Yarn, Mesos, etc.) and storage resources (HDFS, S3, NFS, etc.) to process different types of data sources (real-time events, databases, file systems, and key-value storage, etc.) After that, it is provided to the application for consumption or persisted into different data streams. When the task restarts and needs to update the metadata, it is necessary to stop the task of real-time data processing and then update the metadata, so there is a problem of low data processing efficiency.

[0037] In order to solve the above technical problems, the embodiments of the present application provide a data processing method and system, and the technical solution of the present application will be ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the invention provides a data processing method and system, and relates to the technical field of data processing. The data processing method is applied to the data processing system. The data processing system comprises a data processing device and a metadata processing device. The data processing method comprises the following steps: a data processing unit acquires real-time data, initiates a metadata request to a metadata cache manager according to the real-time data, requests to acquire metadata corresponding to the real-time data, processes the real-time data according to the metadata, and stores the processed real-time data, wherein the real-time data represents the working condition information of the data processing device, and the metadata represents the processing logic of the working condition information; a metadata processing unit receives a metadata updating request and requests to acquire updated metadata from a metadata database. Through the setting, real-time data and metadata can be respectively processed, so that the influence caused by data processing delay and inconsistency due to metadata updating in the real-time processing process is reduced.

Description

technical field [0001] The present application relates to the technical field of data processing, and in particular, to a data processing method and system. Background technique [0002] In the context of Internet big data, real-time streaming data processing frameworks with high throughput, low latency and easy horizontal expansion are gradually emerging. As business requirements continue to change, the logic of real-time processing will evolve iteratively and frequently, and the real-time processing tasks deployed online need to be updated accordingly. [0003] However, the inventors have found through research that in the prior art, when the metadata needs to be updated when the task is restarted, it is necessary to stop the task of real-time data processing and then update the metadata. data processing delays and inconsistencies. Contents of the invention [0004] In view of this, the purpose of this application is to provide a data processing method and system to im...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/23G06F12/0866G06F12/123
CPCG06F12/0866G06F12/123G06F16/2365
Inventor 顾敬潇焦丽梅陈垚亮
Owner 长沙树根互联技术有限公司