File generation method and system based on massive data

A technology for file generation and massive data, which is applied in the computer field and can solve problems such as slow file generation

Pending Publication Date: 2019-02-05
PING AN TECH (SHENZHEN) CO LTD
View PDF6 Cites 12 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] In view of this, the embodiment of the present invention provides a method and system for generating files based on massive data to solve the problem of slow file generation in the prior art based on massive data

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • File generation method and system based on massive data
  • File generation method and system based on massive data
  • File generation method and system based on massive data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0028] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system structures and technologies are set forth in order to provide a thorough understanding of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.

[0029] In order to illustrate the technical solutions of the present invention, the following specific embodiments are used for description.

[0030] The embodiments of the present invention provide a method for generating files based on massive data. combine figure 1 , the method includes:

[0031] S101, a client sends a first request message to the first management n...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a file generation method and system based on massive data. The method comprises the following steps: a client sends a first request message to a first management node, wherein the first request message carries a storage path of N data blocks and a task type corresponding to each data block, and comprises a CPU intensive task and an I / O intensive task; the first management node sequentially obtains the processing capability of each computing node to process two types of tasks, and respectively allocates a sub-task to each of N computing nodes according to the task types of the N data blocks, and the computing nodes read the data in the data blocks and process the data; and the client generates a file corresponding to the data according to data processing results of the N computing nodes. A plurality of computing nodes in a spark cluster process the massive data in parallel to generate the file, and the management node in the spark cluster allocates the data blocksto the computing nodes having high capability of processing the type of tasks according to the task types corresponding to the database, so that data processing speed is improved on the basis of achieving load balancing.

Description

technical field [0001] The invention belongs to the technical field of computers, and in particular relates to a method and system for generating files based on massive data. Background technique [0002] With the rapid development of computer technology and Internet technology, the network penetration rate and the scale of Internet users are also increasing year by year. The double stimulation of the continuous increase in the scale of users and the rapid growth of data processing volume has brought new challenges to Internet applications. [0003] For example, data interaction between fund systems is basically in the form of text files. As the number of users grows, more than 30 GB of data files need to be generated in a day. It takes hours to generate files with ordinary methods, which seriously affects the efficiency of business. Moreover, as the amount of data increases, the performance of the system is also getting higher and higher. Therefore, in the face of massive ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): H04L29/08G06F9/50
CPCH04L67/1008H04L67/1097G06F9/5027G06F9/5066
Inventor 安栋王斌宋先优郭锦玉
Owner PING AN TECH (SHENZHEN) CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap