Unlock instant, AI-driven research and patent intelligence for your innovation.

Data processing method and device and computer equipment

A data processing and real-time data processing technology, applied in the field of data processing, can solve the problems of reducing real-time data and achieve the effect of ensuring consistency and real-time

Pending Publication Date: 2021-04-16
大众问问(北京)信息科技有限公司
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The transactional write operation combines Flink’s consistency checkpoint Checkpoint mechanism to ensure that it only affects the external output once, but only the data confirmed by the Checkpoint can be written to the outside. Since there is a certain time interval between Checkpoints, it will Reduce the real-time performance of data

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method and device and computer equipment
  • Data processing method and device and computer equipment
  • Data processing method and device and computer equipment

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0027] figure 1 It is a flow chart of a data processing method provided by Embodiment 1 of the present invention. The embodiment of the present invention is applicable to processing any type of data based on the Flink real-time processing framework to ensure the consistency and real-time performance of the data processing process. The method can be executed by the data processing device provided in the embodiment of the present invention, the device can be implemented in the form of software and / or hardware, and can generally be integrated into computer equipment.

[0028] Such as figure 1 As shown, the data processing method provided in this embodiment specifically includes:

[0029] S110. Acquire data to be processed, and add the data to be processed to the first data message queue.

[0030] Wherein, the data to be processed may be log data of various sources, types and formats, for example, buried point data, log files or external data, and the like. That is, the data to...

Embodiment 2

[0061] figure 2 It is a flowchart of a data processing method provided in Embodiment 2 of the present invention. This embodiment is embodied on the basis of the above embodiments, wherein the data to be processed can be added to the first data message queue, specifically:

[0062] Determine the serial number of the first target partition and the position of the first target partition corresponding to the data to be processed;

[0063] The data to be processed is added to the first target data partition in the first data message queue according to the sequence number of the first target partition and the position of the first target partition.

[0064] Further, processing the data to be processed in the first data message queue by using streaming data processing based on the Flink real-time processing framework may include:

[0065] Determine the current partition serial number corresponding to the current data partition and the data processing progress indicator;

[0066] ...

Embodiment 3

[0104] Figure 5 It is a schematic structural diagram of a data processing device provided in Embodiment 3 of the present invention. The present invention is applicable to the situation where any type of data is processed based on the Flink real-time processing framework to ensure the consistency and real-time performance of the data processing process. The device can be implemented in the form of software and / or hardware, and generally can be integrated into computer equipment.

[0105] Such as Figure 5 As shown, the data query device specifically includes: a first data message queue generation module 510 , a queue processing data generation module 520 , a second data message queue generation module 530 and a real-time data processing module 540 . in,

[0106] The first data message queue generating module 510 is configured to acquire data to be processed, and add the data to be processed to the first data message queue;

[0107] The queue processing data generating modul...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the invention discloses a data processing method and device and computer equipment. The method comprises the steps of obtaining to-be-processed data, and adding the to-be-processed data to a first data message queue; processing the to-be-processed data in the first data message queue by adopting a streaming data processing mode based on an Flink real-time processing framework to obtain queue processing data; adding the queue processing data to a second data message queue; and performing real-time data processing on the queue processing data in the second data message queue based on the Flink real-time processing framework. According to the technical scheme, the consistency and real-time performance of the data processing process can be ensured.

Description

technical field [0001] The embodiments of the present invention relate to the technical field of data processing, and in particular, to a data processing method, device, and computer equipment. Background technique [0002] With the rapid development of the Internet, there are more and more multivariate data, and these data are often real-time. When processing big data, it needs to rely on technologies such as distributed processing or distributed database, and ensuring data consistency and real-time performance during data processing is always an important issue in data processing. [0003] At present, in the field of data processing, there are generally two types of tasks: batch computing and real-time stream computing. Flink is an open source data platform for both distributed real-time stream processing and batch data processing. It can provide support for both stream processing and batch processing when running on the same Flink real-time processing framework. When en...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/215G06F16/2457G06F16/2458G06F16/25G06F16/27
Inventor 唐杰
Owner 大众问问(北京)信息科技有限公司