Data processing method and device, electronic equipment and computer storage medium

A data processing and computer program technology, applied in the field of data transmission, can solve the problems of limited FTP server resources and difficult to adapt to large-scale data transmission scenarios, and achieve the effect of facilitating load balancing

Pending Publication Date: 2021-11-30
CHINA MOBILE SUZHOU SOFTWARE TECH CO LTD +1
View PDF3 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The present invention expects to provide a technical solution for data processing, which can solve the problem that it is difficult to adapt to the scene of large-scale data transmission due to the limited resources of the FTP server

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method and device, electronic equipment and computer storage medium
  • Data processing method and device, electronic equipment and computer storage medium
  • Data processing method and device, electronic equipment and computer storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0048] In the research on related technologies, the inventor found that: in the process of actual data transmission, when the HDFS-OVER-FTP server reaches a stable transmission state, the transmission rate of a single server node is 300-500MB / s, due to the machine disk input Due to the limitation of output (IO) and network IO, the transfer rate cannot reach higher, so that the method of using FTP service to transfer data to DFS cannot cope with the scene of large-scale data transfer; thus, in order to apply to the scene of large-scale data transfer, Multiple concurrent HDFS-OVER-FTP services are required to achieve a higher data transfer rate, and horizontal expansion of multiple HDFS-OVER-FTP servers requires setting the same data transfer configuration parameters for each server separately, which is not convenient for HDFS-OVER -Configuration management of the FTP server; in a specific example, for each HDFS-OVER-FTP server, two transmission control protocol (Transmission Con...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a data processing method and device, electronic equipment and a computer storage medium. The method comprises the following steps that: after a data transmission instruction sent by an FTP client is received, each registered data transmission process is determined, wherein each data transmission process represents a process for realizing data interaction between the FTP client and the distributed file system (DFS), and the process number of the registered data transmission processes is greater than 1; and at least one data transmission process in the registered data transmission processes is controlled to perform data interaction between the FTP client and the DFS. Thus, a server side can realize data interaction between the FTP client side and the DFS through the plurality of data transmission processes, the problem that resources of a single FTP server are limited in the prior art is solved, and the method and the device can be applied to scenes of large-scale data transmission.

Description

technical field [0001] The present invention relates to data transmission technology, in particular to a data processing method, device, electronic equipment and computer storage medium, which can be applied to data reading and writing of a distributed file system (Distributed File System, DFS). Background technique [0002] In related technologies, with the continuous development of cloud computing and big data technology, more and more enterprises need to use DFS to store massive data, and use some computing frameworks (such as tez, mapreduce, spark, flink) to process these Data analysis, processing and processing; for example, DFS can be Hadoop distributed file system (Hadoop Distributed File System, HDFS), Ceph distributed file system, GFS (Google File System) distributed file system or Luster file system; computing framework can be It is a computing framework such as Tez, Mapreduce, Spark, and Flink; some big data tools (such as Flume tools or Kafka tools) can quickly u...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/50H04L29/08
CPCG06F9/5083G06F9/505H04L67/06
Inventor 卢华超廖光贤陶捷
Owner CHINA MOBILE SUZHOU SOFTWARE TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap