Check patentability & draft patents in minutes with Patsnap Eureka AI!

Intermediate file processing method, client, server and system

A technology of intermediate files and processing methods, applied in the Internet field, can solve problems such as the inability to dynamically adjust the read and write operations of intermediate files, and achieve the effect of maintaining load balance and preventing slowdown

Active Publication Date: 2017-01-18
ALIBABA CLOUD COMPUTING LTD
View PDF5 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] In view of this, the application provides an intermediate file processing method, client, server and system to solve the problem in the prior art that the read and write operations for intermediate files cannot be dynamically adjusted

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Intermediate file processing method, client, server and system
  • Intermediate file processing method, client, server and system
  • Intermediate file processing method, client, server and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0031] The implementation of the present invention will be described in detail below with reference to the drawings and examples, so as to fully understand and implement the implementation process of how to use technical means to solve technical problems and achieve technical effects in the present invention.

[0032] figure 1 It is a schematic topology diagram of an intermediate file processing system provided by the embodiment of the present application. The system includes a first server 10 , a second server 11 , a cluster management client 12 , a first client 13 and a second client 14 .

[0033] The first client 13 is a client for writing an intermediate file to the first server 10 , and the second client 14 is a client for reading the intermediate file from the first server 10 . Multiple tasks (Tasks) are included in the overall job (Job) execution plan, the first client 13 is the executor of the upper-level task, and the second client 14 is the executor of the lower-leve...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses an intermediate file processing method, a client, a server and a system. The method comprises following steps: receiving a message, from a first client, of writing intermediate files to a first server; requesting a second server to create cluster information for the intermediate files; after the cluster information is successively created, receiving cluster information returned from the second server, wherein the cluster information comprises cluster name and priority; sending the cluster information to the first client and the second client, wherein the first client uploads the intermediate files to the first server so that the first server writes in the intermediate files according to local disk load and priority, the second client reads the intermediate files from the first server according to the cluster information. Task of high priority is guaranteed to be timely processed to prevent task of high priority from being slowed down and to maintain the balance in disk load.

Description

technical field [0001] The invention belongs to the technical field of the Internet, and in particular relates to an intermediate file processing method, a client, a server and a system. Background technique [0002] In distributed data computing and processing (such as MapReduce, Spark), each job (Job) for data processing contains multiple tasks (Task), and each task performs corresponding conversion operations on data, usually executed by multiple tasks ( Task worker) threads process different data fragments in parallel to complete. A large amount of intermediate result data usually needs to be exchanged between task execution threads, and these intermediate result data are accessed in the form of intermediate files. How to efficiently process intermediate files is a key technology of distributed data processing. [0003] The existing technical solutions mainly include: each task processing thread directly accesses the local file system of the server to store the output ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F3/06
CPCG06F16/122G06F16/17G06F16/144H04L67/06H04L67/01
Inventor 朱家稷姚文辉谢巍
Owner ALIBABA CLOUD COMPUTING LTD
Features
  • R&D
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com