Unlock instant, AI-driven research and patent intelligence for your innovation.

Source data acquisition method and system in big data offline computing

An off-line computing and source data technology, applied in the field of data processing, can solve problems such as low utilization rate and inability to obtain source data across platforms, and achieve the effect of improving utilization rate

Active Publication Date: 2019-12-06
TENCENT TECH (SHENZHEN) CO LTD +1
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] In order to solve the technical problem that source data cannot be obtained across platforms in related technologies, so that the application of source data is limited to the platform and the utilization rate is low, this disclosure provides a method for collecting source data in offline computing of big data and system

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Source data acquisition method and system in big data offline computing
  • Source data acquisition method and system in big data offline computing
  • Source data acquisition method and system in big data offline computing

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0034] Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with aspects of the invention as recited in the appended claims.

[0035] figure 1 is a schematic diagram of an implementation environment according to the present disclosure. The implementation environment includes an acquisition cluster 110 and an offline computing database 130 . The collection cluster 110 and the offline computing database 130 will construct a set of data collection system.

[0036] Among them, the collection cluster 110 is used to realize th...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a source data collection method and system in big data offline computing. The method comprises the steps of downloading source data from storage clusters through a collection cluster in a distribution mode, wherein the source data is reported to the storage clusters in a streaming mode; uploading the downloaded source data to an offline computing database; and storing the source data to the offline computing database. The source data in the storage clusters is collected. Cross-platform source data collection can be carried out according to a demand through utilization of the process, namely, the source data of a plurality of storage clusters is collected. The application of the source data is no long limited within a platform, namely a single storage cluster, and autilization rate of the source data is improved.

Description

technical field [0001] The present disclosure relates to the technical field of data processing, and in particular to a method and system for collecting source data in big data off-line computing. Background technique [0002] With the rapid development of Internet applications, various applications based on big data are gradually increasing, and then provide users with accurate services in various scenarios. In the implementation of applications based on big data, it first includes various terminals for collecting source data and storage systems for storing source data. [0003] However, in the existing application implementation based on big data, after the terminal completes the collection of its own source data, it reports the source data to the storage system, and the source data received in the storage system is continuous and massive. processing to achieve the corresponding services. [0004] That is to say, the currently collected source data is stored in the platf...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): H04L29/08G06F16/182
CPCG06F16/182H04L67/06
Inventor 万志颖阮华曾凡李家昌何瑞史晓茸
Owner TENCENT TECH (SHENZHEN) CO LTD