Data processing method, related device, related equipment and related system

A data processing and data source technology, applied in the computer field, can solve the problems that cannot satisfy the real-time processing of hundreds of billions of data per day, and achieve the effect of wide access business scope, wide application, and no processing congestion

Active Publication Date: 2019-09-06
TENCENT TECH (SHENZHEN) CO LTD
View PDF7 Cites 5 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The technical problem to be solved by the embodiments of the present invention is to provide a data processing method, a data processing device, a data processing device, a data processing system, and a computer-readable storage medium to solve Technical problems that cannot meet the requirements for real-time processing of hundreds of billions of data per day

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method, related device, related equipment and related system
  • Data processing method, related device, related equipment and related system
  • Data processing method, related device, related equipment and related system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0033] The technical solutions in the embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention.

[0034] It should also be understood that the terminology used in the description of the present invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the present invention.

[0035] It should also be further understood that the term "and / or" used in the description of the present invention and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations .

[0036] In a specific implementation, the terminals or devices described in the embodiments of the present invention include, but are not limited to, desktop computers, laptop computers, tablet computers, smart terminals, and the like.

[0037] In order to better understand a data processing method, a da...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data processing method, which comprises the following steps of generating access indication information according to resource information needing to be allocated by a data source and real-time resource conditions of processing nodes of a cluster; sending the access indication information to each processing node of the cluster; enabling each processing node of the clusterto access the data source according to the access indication information, performing parallel processing on the accessed data source according to a configured thread, and outputting a processing result, wherein each processing node of the cluster is configured with a thread used for processing data, and the threads independently process the data in parallel. The invention further discloses a dataprocessing device, equipment and system. The technical problem that the requirement for real-time processing of hundred million-level data every day cannot be met in the prior art is solved.

Description

technical field [0001] The present invention relates to the computer field, in particular to a data processing method, a related device, a related device and a system. Background technique [0002] With the continuous development of information technology, massive data has become the most valuable wealth in recent years. Today, with the rapid spread of information, all kinds of data permeate our lives, and they grow at an exponential rate. The data explosion brings us into the era of big data. [0003] In the existing technology, it is still difficult for the processing and analysis tools for massive logs and other massive data to guarantee the real-time processing and analysis of hundreds of billions of logs. For example, the log analysis system (Elasticsearch+logstash+kibana, ELK) is an open source log processing solution or log management solution. Logstash in ELK, the complex transformation of the intermediate data structure leads to low performance, and in the process ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F11/30
CPCG06F11/3006
Inventor 曾东旭徐炯陈敏
Owner TENCENT TECH (SHENZHEN) CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap