Method and system for processing data

A data processing and data processing unit technology, applied in the field of computer networks, can solve the problems of data receiving and processing update loss, etc., to release hard disk space, ensure processing efficiency, and avoid update loss.

Active Publication Date: 2015-08-12
BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD +1
View PDF5 Cites 25 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0010] In view of this, the present invention provides a method and system for data processing, which solves the problem of data reception and processing when the kafka client unexpectedly goes down, and the problem of update loss caused by simultan

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and system for processing data
  • Method and system for processing data
  • Method and system for processing data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0029] Exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present invention to facilitate understanding, and they should be regarded as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

[0030] figure 1 is a schematic diagram of main steps of a data processing method according to an embodiment of the present invention. like figure 1 As shown, a data processing method in the embodiment of the present invention mainly includes the following steps S11 to S13.

[0031] Step S11: The server device distributes the grouped data to multiple groups of client dev...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a method and system for processing data. The data can be processed efficiently and fast. The method comprises the steps that a server side device distributes the grouped data to multiple sets of client side devices; the client side devices receive the grouped data, and store the data; the client side devices read the stored data, then fragment the read data, and carry out single-thread processing on the fragmented data. According to the technical scheme, the problems of data receiving and processing in the process that a single kafka client side fails and updating loss caused by simultaneous updating of multiple threads are solved, data processing tasks can be manually controlled to be suspended and restarted, meanwhile, data loss caused by data overstock or processing failures is avoided, the data processing reliability is improved, and the accuracy and real-time performance of the data are guaranteed.

Description

technical field [0001] The invention relates to the technical field of computer networks, in particular to a data processing method and system. Background technique [0002] With the wide application of computer networks and the rapid development of the e-commerce industry, more and more people pay attention to the importance of data, especially for digital operations, the processing and analysis of data, while reducing operating costs, also It can greatly improve the accuracy of operation direction, among which real-time data is the focus of attention. However, the processing of massive real-time data has always been a difficulty. [0003] In the existing technology, for the processing of massive real-time data, distributed message queues, such as kafka, are generally used to send, receive and manage messages. The Kafka distributed message queue uses keywords (topics) for message management, and each topic is divided into multiple groups (partitions), and partitions are u...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/46H04L29/08
Inventor 魏仁言孙海滨吕科
Owner BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap