Check patentability & draft patents in minutes with Patsnap Eureka AI!

Intermediate file processing method, client, server and system

A technology of intermediate files and processing methods, applied in the Internet field, can solve problems such as the inability to dynamically adjust the read and write operations of intermediate files, and achieve the effect of maintaining load balance and preventing slowdown

Active Publication Date: 2020-04-10
ALIBABA CLOUD COMPUTING LTD
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] In view of this, the application provides an intermediate file processing method, client, server and system to solve the problem in the prior art that the read and write operations for intermediate files cannot be dynamically adjusted

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Intermediate file processing method, client, server and system
  • Intermediate file processing method, client, server and system
  • Intermediate file processing method, client, server and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0031] The implementation of the present invention will be described in detail below with reference to the drawings and examples, so as to fully understand and implement the implementation process of how to use technical means to solve technical problems and achieve technical effects in the present invention.

[0032] figure 1 It is a schematic topology diagram of an intermediate file processing system provided by the embodiment of the present application. The system includes a first server 10 , a second server 11 , a cluster management client 12 , a first client 13 and a second client 14 .

[0033] The first client 13 is a client for writing an intermediate file to the first server 10 , and the second client 14 is a client for reading the intermediate file from the first server 10 . Multiple tasks (Tasks) are included in the overall job (Job) execution plan, the first client 13 is the executor of the upper-level task, and the second client 14 is the executor of the lower-leve...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses an intermediate file processing method, client, server and system, including: receiving a message from a first client to write an intermediate file to a first server; requesting the second server to create cluster information of an intermediate file; After the information is successfully created, the cluster information returned by the second server is received; the cluster information includes the cluster name and priority; the cluster information is sent to the first client and the second client, and the first client uploads it to the first server The intermediate file is to make the first server write the intermediate file according to the local disk load and priority, and the second client read the intermediate file from the first server according to the cluster information. Ensure that tasks of higher priority jobs are processed in a timely manner, prevent higher priority jobs from being slowed down, and maintain disk load balance.

Description

technical field [0001] The invention belongs to the technical field of the Internet, and in particular relates to an intermediate file processing method, a client, a server and a system. Background technique [0002] In distributed data computing and processing (such as MapReduce, Spark), each job (Job) for data processing contains multiple tasks (Task), and each task performs corresponding conversion operations on data, usually executed by multiple tasks ( Taskworker) threads process different data slices in parallel to complete. A large amount of intermediate result data usually needs to be exchanged between task execution threads, and these intermediate result data are accessed in the form of intermediate files. How to efficiently process intermediate files is a key technology of distributed data processing. [0003] The existing technical solutions mainly include: each task processing thread directly accesses the local file system of the server to store the output inte...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F3/06
CPCG06F16/122G06F16/17G06F16/144H04L67/06H04L67/01
Inventor 朱家稷姚文辉谢巍
Owner ALIBABA CLOUD COMPUTING LTD
Features
  • R&D
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com