Data processing method, device and system, electronic equipment and storage medium

A technology of data processing and storage media, which is applied in the computer field and can solve the problem of repeated consumption of data A

Pending Publication Date: 2021-01-05
BEIJING KINGSOFT CLOUD NETWORK TECH CO LTD
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

In this case, when the consumer device returns to normal, it will read and store the data

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method, device and system, electronic equipment and storage medium
  • Data processing method, device and system, electronic equipment and storage medium
  • Data processing method, device and system, electronic equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0066] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0067] Currently, the producer device can send the produced data to the Kafka message queue to cache the data through the Kafka message queue. In addition, consumer devices can obtain data from the Kafka consumption queue for consumption. Among them, the Kafka message queue is a high-throughput distributed publish / subscribe message queue.

[0068] In addition, the consumer device can be configured: after successfully writing the read data, submit the offset (...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention provides a data processing method, device and system, electronic equipment and a storage medium. The method comprises the steps: reading to-be-processed data in a preset data consumption queue, and executing a first preprocessing operation used for preprocessing the to-be-processed data, a second preprocessing operation used for pre-storing the to-be-processed dataand a third preprocessing operation used for pre-reporting offset, and when execution of at least one of the first preprocessing operation, the second preprocessing operation and the third preprocessing operation fails, recovering the data state to a state before the to-be-processed data are read from the preset data consumption queue. According to the embodiment of the invention, when the execution of at least one of the first, second and third preprocessing operations fails, the whole first, second and third preprocessing operations fail, so that the data consumption process meets the atomicity, and the problems of data loss and repeated data consumption caused by an automatic offset submitting mode are solved.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a data processing method, device, system, electronic equipment and storage medium. Background technique [0002] Currently, the producer device can send the produced data to the Kafka message queue to store the data through the Kafka message queue. In addition, consumer devices can obtain data from the Kafka consumption queue for consumption. Among them, the Kafka message queue is a high-throughput distributed publish / subscribe message queue. [0003] In addition, the consumer device can be configured: after successfully writing the read data, submit the offset (offset) corresponding to the read data to the Kafka consumption queue, that is, manually submit the offset to inform Kafka consumption Queue where this consumer device has read data. [0004] In the process of implementing the present invention, the inventor found that the consumer device may have the followi...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/54G06F9/50
CPCG06F9/5016G06F9/546
Inventor 张永鹏
Owner BEIJING KINGSOFT CLOUD NETWORK TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap