Data Transmission Method, Proxy Server, Storage Medium and Electronic Device

Abstract

An embodiment of the present invention provides a data transmission method, a proxy server, a storage medium, and an electronic device. The method includes: The proxy server receives a target request sent by a client. When the target request is a viewing request, the proxy server obtains the target attribute information of the target file from the saved copy of the attribute information and returns it to the client; when the target request is a transmission request, the proxy server sends the target request to the target server in the data server, where the data server is a double-layer stacked FTPServer multi-instance distributed cluster server, the data server includes multiple first servers, and the proxy server records the range of processing FTP users reported by each first server. Through the embodiment of the present invention, the problem of low file transmission efficiency is solved, and thus the effect of improving the file transmission efficiency is achieved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of file transfer under 5G intelligent networks. Specifically, the embodiments relate to a data transfer method, a proxy server, a storage medium, and an electronic device. Background Art

[0002] The existing methods for supporting FTP file transfer mainly include the following categories:

[0003] An efficient single-instance FTP server, a single FTP server, which uses the local file system, NIO, etc. to efficiently support FTP file transfer.

[0004] An FTP distributed server, which can expand multiple FTP server instances according to the scale to support a larger scale. According to the file sharing methods between multiple FTP server instances, there are mainly the following 3 categories:

[0005] (1) File not shared;

[0006] (2) File sharing by file copying;

[0007] (3) File sharing by distributed file systems such as NFS and CEPH;

[0008] Generally, an FTP reverse proxy is used to solve the network connectivity problem to ensure that files can be transferred between an FTP client and an FTP server that is not directly accessible.

[0009] However, for the above methods, in the single-instance FTP service, since there is only one FTP server instance, the upper limit of the support capacity is limited by the processing capacity of the single instance and cannot meet the large-scale multi-connection scenario. The existing implementation of the multi-instance FTP server cannot well solve the efficiency problem in the high-concurrency connection scenario.

[0010] That is to say, in the prior art, there is a problem of low file transfer efficiency in the process of data transfer. Summary of the Invention

[0011] The embodiments of the present invention provide a data transfer method, a proxy server, a storage medium, and an electronic device to at least solve the problem of low file transfer efficiency in the related art.

[0012] According to an embodiment of the present invention, a data transmission method is provided, including: a proxy server receives a target request sent by a client, where the target request is a viewing request or a transmission request, the viewing request is used to request to view target attribute information of a target file in a data server, and the transmission request is used to request to transmit a target file to the data server or obtain the target file from the data server; when the target request is a viewing request, the proxy server obtains the target attribute information of the target file from a saved copy of the attribute information and returns it to the client, where the corresponding relationship between the target file identifier of the target file and the target attribute information is recorded in the copy of the attribute information; when the target request is a transmission request, the proxy server sends the target request to a target server in the data server, where the data server is a double-layer stacked FTPServer multi-instance distributed cluster server, the data server includes multiple first servers, and the proxy server records the range of processing FTP users reported by each of the first servers.

[0013] In an exemplary embodiment, before the proxy server receives the target request sent by the client, the method further includes: obtaining the file identifiers of all files in the data server and the attribute information of all files; saving the file identifiers of all files in the data server and the attribute information of all files to the copy of the attribute information.

[0014] In an exemplary embodiment, before the proxy server receives the target request sent by the client, the method further includes: obtaining the corresponding relationship between the user identifier and the first server, where different user identifiers correspond to different first servers; saving the corresponding relationship to the memory.

[0015] In an exemplary embodiment, when the target request is a viewing request, obtaining the attribute information of the target file from the copy of the attribute information saved in the proxy server and returning it to the client includes: obtaining the target file identifier of the target request; obtaining the target attribute information corresponding to the target file identifier in the copy of the attribute information; returning the target attribute information to the client.

[0016] In an exemplary embodiment, when the target request is a transmission request, before sending the target request to the target server in the data server, the method further includes: obtaining the target user identifier of the client; searching for the target server corresponding to the target user identifier in the memory.

[0017] In an exemplary embodiment, the method of finding the target server corresponding to the target user identifier in the memory includes: when multiple servers corresponding to the target user identifier are found, selecting the server with the lowest load balance from the multiple servers as the target server, or determining the target server from the multiple servers using a polling algorithm.

[0018] In an exemplary embodiment, when the target request is a transmission request, after sending the target request to the target server in the data server, the method further includes: establishing a data transmission channel between the client and the target server; transmitting the target file between the client and the target server through the data transmission channel.

[0019] In an exemplary embodiment, before the proxy server receives the target request sent by the client, the method further includes: when the file size of the property information copy is less than a predetermined threshold, saving the property information copy in the memory; when the file size of the property information copy is greater than or equal to the predetermined threshold, saving the property information copy to the memory.

[0020] In an exemplary embodiment, when the target request is a transmission request, before the proxy server sends the target request to the target server in the data server, the method further includes: forming a one-layer stacked structure using multiple FTPServer instances located on the same first server; forming a two-layer stacked structure using multiple first servers to obtain the data server.

[0021] In an exemplary embodiment, after forming a two-layer stacked structure using multiple first servers to obtain the data server, the method further includes: mounting multiple FTPServer instances on the same first server on the same magnetic array LUN high-speed storage.

[0022] According to another embodiment of the present invention, a data transmission device is provided, including: a receiving unit, configured to receive a target request sent by a client, where the target request is a viewing request or a transmission request, the viewing request is used to request to view the target attribute information of a target file in a data server, and the transmission request is used to request to transmit the target file to the data server or obtain the target file from the data server; a first obtaining unit, configured to, when the target request is a viewing request, obtain the target attribute information of the target file from a saved copy of the attribute information and return it to the client, where the corresponding relationship between the target file identifier of the target file and the target attribute information is recorded in the copy of the attribute information; a first sending unit, configured to, when the target request is a transmission request, send the target request to a target server in the data server, where the data server is a double-layer stacked FTPServer multi-instance distributed cluster server, the data server includes a plurality of first servers, and the range of processing FTP users reported by each of the first servers is recorded in the proxy server.

[0023] In an exemplary embodiment, the proxy server further includes: a second obtaining unit, configured to obtain the file identifiers of all files in the data server and the attribute information of all files before the proxy server receives the target request sent by the client; a first saving unit, configured to save the file identifiers of all files in the data server and the attribute information of all files to the copy of the attribute information.

[0024] In an exemplary embodiment, the proxy server further includes: a third obtaining unit, configured to obtain the corresponding relationship between the user identifier and the first server before the proxy server receives the target request sent by the client, where different user identifiers correspond to different first servers; a second saving unit, configured to save the corresponding relationship to the memory.

[0025] In an exemplary embodiment, the first obtaining unit includes: a first obtaining module, configured to obtain the target file identifier of the target file in the target request; a second obtaining module, configured to obtain the target attribute information corresponding to the target file identifier in the copy of the attribute information; a returning module, configured to return the target attribute information to the client.

[0026] In an exemplary embodiment, the proxy server further includes: a fourth obtaining unit, configured to obtain a target user identifier of the client before sending the target request to the target server in the data server when the target request is a transmission request; a searching unit, configured to search for the target server corresponding to the target user identifier in the memory.

[0027] In an exemplary embodiment, the searching unit includes: a processing module, configured to, when multiple servers corresponding to the target user identifier are found, select the server with the lowest load balancing as the target server from the multiple servers, or determine the target server from the multiple servers using a polling algorithm.

[0028] In an exemplary embodiment, the proxy server further includes: a establishing unit, configured to establish a data transmission channel between the client and the target server after sending the target request to the target server in the data server when the target request is a transmission request; a transmitting unit, configured to transmit the target file between the client and the target server through the data transmission channel.

[0029] In an exemplary embodiment, the proxy server further includes: a third saving unit, configured to save the copy of the attribute information in the memory when the file size of the copy of the attribute information is less than a predetermined threshold before the proxy server receives a target request sent by the client; a fourth saving unit, configured to save the copy of the attribute information to a memory when the file size of the copy of the attribute information is greater than or equal to the predetermined threshold.

[0030] According to another embodiment of the present invention, there is also provided a computer-readable storage medium, in which a computer program is stored, and wherein the computer program is configured to execute the steps in any one of the above method embodiments when running.

[0031] According to another embodiment of the present invention, there is also provided an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0032] Through the embodiments of the present invention, since during the file transmission process, file scheduling can be performed through the proxy server, the view request can directly return the target attribute information through the proxy server, and the transmission request can be sent to different target servers for processing, therefore, the problem of low file transmission efficiency can be solved, and the effect of improving the file transmission efficiency can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Hardware structure block diagram of a mobile terminal for the data transmission method according to an embodiment of the present invention;

[0034] Figure 2 Flowchart of the data transmission method according to an embodiment of the present invention;

[0035] Figure 3 Schematic diagram of the architecture and process of the data transmission method according to an embodiment of the present invention;

[0036] Figure 4 Flowchart of shunt and addressing of the data transmission method according to an embodiment of the present invention;

[0037] Figure 5 Flowchart of the distributed coordination algorithm of the data transmission method according to an embodiment of the present invention;

[0038] Figure 6 Schematic diagram of affinity and HA of distributed FTP clusters for the data transmission method according to an embodiment of the present invention;

[0039] Figure 7 Structure block diagram of a proxy server according to an embodiment of the present invention. Detailed implementation manners

[0040] Embodiments of the present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0041] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence.

[0042] File transfer

[0043] File transfer is a general technology. In the 5G intelligent network scenario, file transfer is everywhere. For example, for southbound out-of-band network elements, due to the massive access characteristics of 5G, the number of access network elements has increased significantly. File transfer is a common technology in network element performance management and version management. Since security can be guaranteed from the physical network of the access network element, a simple and efficient file transfer protocol FTP server has become a reasonable choice for file transfer. The massive access network elements trigger a large number of FTP concurrent connection requests. At the same time, the 5G low-latency intelligent network further requires FTP to achieve high efficiency in file transfer while supporting a large number of connections.

[0044] Distributed cluster

[0045] Distributed means that multiple systems cooperate to complete a specific task. Distributed is to solve the problem of centralized management. Stacking all tasks on one node for processing is too slow. Therefore, a large problem is split into multiple small problems and solved separately, and finally they cooperate. The main work of distributed is to decompose tasks and disassemble functions. The main use case of a cluster is to share the pressure of requests, that is, deploy the same application on several servers to share client requests.

[0046] FTP reverse proxy

[0047] Due to network environment restrictions, when it is necessary to upload and download files from the internal network (which cannot be directly connected to the external network) to an external FTP, an FTP reverse proxy can only be set up on the gateway server. The gateway server can be connected to both the external network and the internal network at the same time. FTP users on the internal network connect to the FTP proxy and access the external real FTP server through it.

[0048] The method embodiments provided in the embodiments of the present application can be executed on a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 is a hardware structure block diagram of a mobile terminal of a data transmission method according to an embodiment of the present invention. As Figure 1 shown, the mobile terminal may include one or more ( Figure 1 only one is shown in Figure 1 the processor 102 (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Among them, the above mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those of ordinary skill in the art can understand that Figure 1 the structure shown in Figure 1 is only schematic and does not limit the structure of the above mobile terminal. For example, the mobile terminal may further include more or fewer components than

[0049] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the data transmission method in the embodiments of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implements the above-mentioned data transmission method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory 104 may further include a memory remotely disposed relative to the processor 102, and these remote memories can be connected to the mobile terminal through a network. Examples of the above network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0050] The transmission device 106 is used to receive or send data via a network. Specific examples of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one instance, the transmission device 106 includes a network adapter (Network Interface Controller, abbreviated as NIC), which can be connected to other network devices through a base station and thus can communicate with the Internet. In one instance, the transmission device 106 can be a radio frequency (Radio Frequency, abbreviated as RF) module, which is used to communicate with the Internet wirelessly.

[0051] In this embodiment, a data transmission method is provided. Figure 2 It is a flowchart of the data transmission method according to the embodiments of the present invention, as Figure 2 shown, and this process includes the following steps:

[0052] Step S202, the proxy server receives a target request sent by the client, where the target request is a viewing request or a transmission request. The viewing request is used to request to view the target attribute information of a target file in the data server, and the transmission request is used to request to transmit a target file to the data server or obtain the target file from the data server;

[0053] Step S204, when the target request is a viewing request, the proxy server obtains the target attribute information of the target file from the saved copy of the attribute information and returns it to the client, where the corresponding relationship between the target file identifier of the target file and the target attribute information is recorded in the copy of the attribute information;

[0054] Step S206, when the target request is a transmission request, the proxy server sends the target request to the target server in the data server. The data server is a double-layer stacked FTPServer multi-instance distributed cluster server, which includes multiple first servers. The proxy server records the range of processing FTP users reported by each first server.

[0055] Before the proxy server receives the target request sent by the client, the method further includes: obtaining the file identifiers of all files in the data server and the attribute information of all files; saving the file identifiers of all files in the data server and the attribute information of all files to the attribute information copy.

[0056] Before the proxy server receives the target request sent by the client, the method further includes: obtaining the correspondence between the user identifier and the first server, where different user identifiers correspond to different first servers; saving the correspondence to the memory.

[0057] When the target request is a viewing request, obtaining the attribute information of the target file from the attribute information copy saved in the proxy server and returning it to the client includes: obtaining the target file identifier of the target file in the target request; obtaining the target attribute information corresponding to the target file identifier in the attribute information copy; returning the target attribute information to the client.

[0058] Before sending the target request to the target server in the data server when the target request is a transmission request, the method further includes: obtaining the target user identifier of the client; searching for the target server corresponding to the target user identifier in the memory.

[0059] Searching for the target server corresponding to the target user identifier in the memory includes: when multiple servers corresponding to the target user identifier are found, selecting the server with the lowest load balance as the target server from the multiple servers, or determining the target server from the multiple servers using the round-robin algorithm.

[0060] After sending the target request to the target server in the data server when the target request is a transmission request, the method further includes: establishing a data transmission channel between the client and the target server; transmitting the target file between the client and the target server through the data transmission channel.

[0061] Before the proxy server receives the target request sent by the client, the method further includes: when the file size of the copy of the attribute information is less than a predetermined threshold, saving the copy of the attribute information in the memory; when the file size of the copy of the attribute information is greater than or equal to the predetermined threshold, saving the copy of the attribute information to the memory.

[0062] Before the proxy server sends the target request to the target server in the data server when the target request is a transfer request, the method further includes: forming a stacked structure with multiple FTPServer instances located on the same first server; forming a two-layer stacked structure with multiple first servers to obtain the data server.

[0063] After forming a two-layer stacked structure with multiple first servers to obtain the data server, the method further includes: mounting multiple FTPServer instances on the same first server on the same magnetic array LUN high-speed storage.

[0064] Through the embodiments of the present invention, since in the process of file transmission, the file can be scheduled through the proxy server, the view request can directly return the target attribute information through the proxy server, and the transfer request can be sent to different target servers for processing, therefore, the problem of low file transmission efficiency can be solved, and the effect of improving the file transmission efficiency can be achieved.

[0065] A specific example is used for illustration.

[0066] The proxy server in this application can be an ftp reverse proxy server. The first server can be a server node, and an FTP SERVER instance (FTPServer microservice) runs on the server node. Multiple first servers form a two-layer stacked structure to form the data server.

[0067] FTP Reverse Proxy FTPProxy Shunt Addressing

[0068] The FTPProxy reverse proxy is a primitive layer implementation of the FTP protocol. For the client, it is a standard FTP server, but many actual file operation requests are passed to the backend FTPServer microservice cluster for processing. The FTPProxy reverse proxy in this application focuses on local shunting and cooperating with the FTPServer two-layer stacked addressing design.

[0069] First, introduce the reverse proxy. The reverse proxy is to shield the differences caused by multiple distributed instances of the backend FTPServers to the client, and cooperate with traffic splitting and addressing. The FTP client does not need to care about details such as how many FTPServer instances there are, which server nodes they are running on, what the addresses of each instance are, and whether a certain instance has stopped. The FTP client does not directly request the FTPServer microservice, but connects to the FTPProxy, which acts as a proxy for the FTPServer to provide FTP protocol capabilities. The FTPProxy provides a monitoring console where the status of the FTPProxy itself, all current FTP connection information, and the information and status of all backend ServerNodes and FTPServer instances can be viewed.

[0070] By analyzing FTP requests, the ftp reverse proxy can classify FTP requests into two major categories:

[0071] View requests: File viewing requests, such as LIST, NLST, PWD, STAT. These requests are mainly to view file information at the file system level.

[0072] Transfer requests: File operation and file transfer requests, which are other requests except for viewing requests, such as RETR, STOR, DELE, CWD, RNTO, etc. We collectively call them file transfer requests.

[0073] To relieve the pressure on the FTPServer and improve the response efficiency of file viewing requests, traffic splitting capabilities can be provided in the reverse proxy server FTPProxy.

[0074] After receiving a request from the client, the FTPProxy determines the request type. If it is a file viewing request, it is directly processed locally and no longer passed to the backend FTPServer. To support file viewing requests to be processed in the FTPProxy, a copy of the file directory tree and attribute information of the backend FTPServer needs to be maintained in the FTPProxy. Since the single server nodes where the backend FTPServers are located share the local file system, for the same server node, this file tree information does not need to be merged in the FTPProxy. The local file systems of each server node are not shared and are independent of each other. Therefore, the FTPProxy needs to maintain a directory tree for each server node where all FTPServers are located, taking the server node as a unit. As subsequent FTP actions occur, the content of the FTPServer's file system will change, and the local cache of the FTPProxy needs to be updated synchronously. Since all FTP requests pass through the FTPProxy reverse proxy, these changes can be made simultaneously in the directory tree maintained by the FTPProxy to keep the consistency with the content of the backend FTPServer's file system.

[0075] To improve efficiency, the directory tree information maintained by the FTPProxy can be stored in memory. If the number of files held by the backend FTPServer is too large, this part of the information can be persisted to storage to relieve memory pressure.

[0076] For file transfer requests, after reaching the proxy, the subsequent file transfer channel is directly established between the client and the FTPServer instance. The file transfer does not pass through the proxy but is directly connected and transferred directly, reducing one forwarding by the FTPProxy, avoiding the FTPProxy from becoming a bottleneck, and further improving efficiency.

[0077] The FTPProxy is responsible for distributing FTP requests to multiple FTPServer microservice instances at the backend. Because of the double-layer stacking design of the FTPServer, the number of FTP users that each FTPServer instance can support is within a certain range. Therefore, the FTPProxy needs to determine which FTPServer instance at the backend to address according to the currently connected FTP user. Of course, there may be multiple such instances. At this time, requests can be evenly distributed according to the load of the FTPServer to maximize efficiency, or the Round-Robin polling algorithm can be used, which is simple and efficient.

[0078] In the trend of cloud-network integration in 5G intelligent networks, microservice architectures are widely used in operation and maintenance management, network management, etc. Multi-instance expansion has become a basic requirement of cloud-native for each service component to better support distributed, high-concurrency large-scale, and dual-machine cluster system HA. Therefore, in this solution, the FTPServer, as a file transfer service component, is implemented in the form of microservice containers and supports multi-instance operation. At the same time, to improve file transfer efficiency, the FTPServer instances on the same server node share the local high-efficiency storage logical unit number LUN, and multiple server nodes are stacked to form the entire FTP distributed cluster system to meet the requirements of high-concurrency and high-efficiency file transfer.

[0079] The number of FTP concurrent connections that a single-instance FTPServer can support is limited, and its ability cannot be infinitely improved with the vertical expansion of resources. Because of the thread scheduling and lock waiting inside a single instance, the processing ability has an upper limit. Generally, the concurrent connections reach the limit at about 1000, and allocating additional extended resources cannot improve it. Therefore, to support more concurrent connections, an FTP server cluster needs to be implemented.

[0080] In the embodiments of the present invention, the FTP Server runs in a container manner, meeting the requirements of cloud native, supporting multi-instance expansion, supporting scaling, and being able to pop up multiple instances for running according to configurations, loads, such as the number of connections, connection activity, CPU and memory usage. A server node can run multiple FTP Server instances, and multiple server nodes can be deployed. Thus, multiple FTP Server instances form an FTP distributed cluster, and its life cycle is managed by a container management system, such as K8s, to achieve scaling scheduling and HA.

[0081] Among them, the FTP Server instances on the same server node share the attached magnetic array LUN on the local storage.

[0082] The biggest problem to be solved by the FTP server cluster is the consistency of the file system. Multiple FTP Server instances need to see a completely consistent file view so that file upload / download and file operations can ensure the correct business. To ensure this consistency, whether it is the file replication method or the distributed file system method, the efficiency issue is the most prominent problem. The increased IO and conflicts inside the cluster greatly reduce the file transfer efficiency. The embodiments of the present invention solve this problem.

[0083] To solve the file system consistency, it is designed that the FTP Server instances on the same server share the attached magnetic array LUN on the local storage, and a distributed coordination mechanism is used to solve file access conflicts, so that the FTP Server instances on the same server see a unified file system view. This method uses the efficient IO of the magnetic array LUN to support efficient file transfer. Of course, this requires the corresponding FTP Server instances to run on the same server node, and the high configuration of the server node resources is used to support multiple FTP Server instances. For example, when the capacity limit of a single FTP Server instance is a resource requirement of 1C2G (1-core CPU, 2G memory), currently high-configured servers can basically reach a configuration of 100C300G, that is, the same server node can support the horizontal expansion of 100 FTP Server instances, reaching the support capacity of 100,000 FTP concurrent connections.

[0084] The multiple FTP Server instances running on a single server node form an FTP server cluster, but limited by the server node configuration, the processing capacity of this cluster also has an upper limit, which to a certain extent restricts the support scale of the distributed FTP server cluster. Therefore, it is necessary to break the limitation of a single server node and use multiple server nodes. However, the attached magnetic array LUN on the local storage cannot be efficiently accessed across multiple server nodes, otherwise a distributed file system similar to NFS must be constructed, and efficiency problems will occur.

[0085] Combined with the business characteristics of FTP, the main working directory of a specific FTP user is determined, and there is no overlap between the files in different main working directories. That is, the operations on files by FTP are always restricted to the main working directory of this user. Therefore, we can bind the main working directory of a certain FTP user to a fixed server node, and bind the different main working directories of other FTP users to other server nodes. In this way, different server nodes handle different main working directories, and there is no requirement for a file sharing view between multiple server nodes. The basis for this division is the main working directory of the FTP user, rather than the FTP user, because different FTP users can point to the same working directory. The final result is that a specific working directory only exists on a specific server node.

[0086] Pre-divide the corresponding relationship between the main working directories of each FTP user and the server nodes. A server node can handle the file transfer requests of the corresponding FTP user as long as it contains the corresponding working directories. Eventually, the relationship between the server nodes and the FTP users is formed. All requests of a certain FTP user are processed by a fixed server node (multiple FTPServer instances). We can continue to utilize the magnetic array LUN attached to the local storage, and at the same time, we can also utilize multiple FTPServer instances running on this server node to support the clustering ability for this user. In this way, the FTPServer is divided into many categories. The FTP user directories processed by a certain category of FTPServer (multiple instances) are determined and do not overlap with other categories of FTPServer, and they run on fixed server nodes.

[0087] According to the previous resource estimation, a single user running on a single server node can reach the order of magnitude of 100,000 concurrent connections. Different services generally use different FTP users. Therefore, combined with the multi-server node expansion in this solution, the entire system can provide a higher order of magnitude of concurrent support ability.

[0088] In the embodiments of the present invention, the range of FTP users that the FTPServer on different server nodes can handle is different. A specific FTPServer instance can only run on a specific server node and scale in and out on this node. This requires the use of the label mechanism of the container management system K8s, which can schedule the server node where the FTPServer instance runs through labels. Add labels to each server node, and at the same time configure the labels of a certain category of FTPServer, so as to control the running of its instances on the server nodes with corresponding labels. Other categories of FTPServer with different working directory ranges can be configured with other labels to control the running of their instances on other server nodes with corresponding labels.

[0089] According to the plan, the server node corresponding to the label on which an instance of a certain type of FTPServer can run can only be one in order to utilize the magnetic array LUN stored locally. However, once this server node fails, HA cannot be guaranteed. Therefore, the number of server nodes corresponding to a certain label must be greater than 1. But it is also necessary to ensure that all instances of the same type of FTPServer can only be scheduled to the same server nodes. This requires the use of the affinity strategy of the container management system K8s. Affinity means that once the first instance of a certain type of FTPServer is scheduled to a certain server node according to the label rules, subsequent instances of this type of FTPServer will all be scheduled to the same server node, even if there are multiple server nodes with the same label. In this way, it is ensured that all instances of the same type of FTPServer run on the same server node, can share the magnetic array LUN stored locally, and at the same time guarantee HA. When a certain server node goes down, all FTPServer instances on this server node are switched as a whole to another server node that conforms to the label rules, and the magnetic array LUN is also remounted to the new server node.

[0090] Since all FTPServer instances on the same server node will read and write to the same magnetic array LUN, a distributed coordination mechanism is required to coordinate the IO operations of these FTPServer instances so that they do not conflict with each other. Mainly, the read and write operations on the same file should not conflict with each other, and at the same time, only one FTPServer instance can write to the same file. When writing, reading is not allowed simultaneously, and there are no other restrictions. Inside the same FTPServer instance, the multi-threads are controlled by memory locks and do not require distributed coordination.

[0091] This distributed coordination mechanism is implemented through a distributed lock across containers: when writing a file, a corresponding lock is generated based on the file name and released after writing. For other read and write operations, it is checked whether this lock exists. If it exists, other read and write operations on this file are not allowed.

[0092] This distributed coordination is a distributed lock implemented based on temporary files and file sorting. This lock is not reentrant. When applying for a lock, a temporary directory is generated for each lock, corresponding to the specific file to be coordinated for reading and writing. Each FTPServer instance applying for the lock generates a temporary file under this directory according to its own container ID and locks it. Judgments are made before and after locking. If multiple containers apply simultaneously (that is, there are multiple temporary files), the container that succeeds in locking is selected according to the fixed algorithm for sorting file names. Finally, the goal that only one container obtains this lock is achieved, thus realizing distributed coordination.

[0093] By using the method of the embodiment of the present invention, compared with the prior art, the operation of viewing response files is faster, it can support a large number of concurrent FTP connections, and in the case of a large number of FTP connections, since there is no file copying or network file system, the IO inside the FTP server cluster is effectively reduced, and the file transfer efficiency is improved.

[0094] Through the FTPProxy reverse proxy to shunt and address multiple FTPServer instances at the back end, file transfer requests skip the proxy and are directly connected and transferred directly, improving the file transfer efficiency. Combining the characteristics of the FTP service, a two-layer stacked multi-instance distributed cluster (data server) of FTPServers is designed. Using the affinity strategy and distributed coordination, similar FTPServer instances can share the local magnetic array LUN for efficient storage, further improving the file upload and download efficiency.

[0095] As Figure 3 shown, Figure 3 It is a schematic diagram of the method architecture and process of this application. The FTPProxy can be deployed at the gateway, acting as a reverse proxy, and cooperating with the FTPServer instances addressed at the back end. When the first generated FTPServer instance on each ServerNode starts, it passes the working directory tree information in the local LUN to the FTPProxy for caching, so that it can quickly respond to file viewing requests. The FTPProxy maintains the working directory trees of all LUNs and updates them dynamically with subsequent FTP operations. At the same time, the FTPProxy provides a monitoring console to collect various information and status of the FTP distributed cluster system. It can view the status of the FTPProxy itself, the current information of all FTP connections, and the information and status of all ServerNodes and FTPServer instances at the back end.

[0096] The client FTPClient first connects to the FTPProxy for FTP user authentication, and the FTPProxy records the FTP user information corresponding to each FTP connection. When subsequent file transfer requests are initiated, the FTPServer instance corresponding to this user is addressed, and the FTPClient and this FTPServer instance directly establish a file transfer channel for subsequent file transfer.

[0097] Each instance that composes the FTPServer distributed cluster runs on the corresponding ServerNode server node according to the label rules. This server node mounts the corresponding magnetic array LUN, and its working directory range determines the range of FTP users that can be served. The FTPServer instances on the same ServerNode form a layer of stacking, and multiple ServerNodes form a two-layer stacking, thus forming the entire FTPServer distributed cluster. It can serve a large number of high-concurrency connections of FTP users.

[0098] The FTPServer instances run as microservices in container form on the ServerNode. All the FTPServer containers on the same ServerNode are mounted with the same array LUN high-speed storage, and access is shared through distributed coordination to improve the file transfer efficiency by using efficient I / O, thus achieving the high-concurrency and high-efficiency design purpose of the present invention.

[0099] As Figure 4 shown, Figure 4 is the flowchart of FTPProxy shunting and addressing

[0100] S401, the FTP request arrives at the FTPProxy;

[0101] S402, the FTPProxy analyzes the request according to the FTP RFC protocol specification;

[0102] S403, determine that if it is a file transfer request, go to step S405; otherwise S404, directly respond through the local cache information;

[0103] S405, find the corresponding FTPServer instance according to the current FTP user information; if there are multiple FTPServer instances, an appropriate FTPServer instance can be addressed according to the load balancing algorithm or the Round-Robin polling algorithm;

[0104] S406, establish a direct file transfer channel between the FTPClient and the FTPServer instance;

[0105] S407, perform file transfer (upload or download).

[0106] As Figure 5 shown, Figure 5 is the flowchart of the distributed coordination algorithm.

[0107] Multiple FTPServer container instances on the same server node need to ensure that the read and write operations on the same file do not conflict with each other, and only one FTPServer instance can perform write operations on the same file, and reading and writing cannot be performed simultaneously. This is achieved through cross-container distributed locks. The specific distributed lock algorithm is as follows:

[0108] Preconditions:

[0109] The FTPServer containers on the same server node use the same directory clusterlocks as the distributed lock temporary directory.

[0110] All distributed locks provide a keyword (or file path), which is represented in hexadecimal encoding. Each lock creates a unique directory under clusterlocks with this keyword, and a file is created under this directory according to the container ID that applies for the lock. In this way, a unique temporary file clusterlocks / <lock keyword> / <container ID> will be generated when each container applies for a lock.

[0111] To prevent the container from exiting abnormally after obtaining the lock, each lock has an expiration time of 60s (determine whether it has expired according to the last modification time of the file).

[0112] The process of applying for a distributed lock is as follows:

[0113] 1. Obtain the directory corresponding to the file lock.

[0114] 2. Check whether there is a file starting with locked in the lock directory (clusterlocks / <lock keyword>). If not, jump to step 4. If there is and the locked file has not expired, it means the lock has been obtained, and the lock acquisition fails.

[0115] 3. If the locked file has expired, delete the locked file.

[0116] 4. Check whether there is only one file in the directory. If so, jump to step 5. Otherwise, sort the multiple files, and then determine whether the first file name is the container ID of this container. If not, the lock acquisition fails; if so, jump to step 6.

[0117] 5. Check whether the file name of this one file is the container ID of this container. If not, the lock acquisition fails; if so, continue.

[0118] 6. Add the locked prefix to the file name.

[0119] 7. Sleep for 10ms, and confirm again whether it is the only locked file. If not, jump to step 8. If so, confirm again whether the lock file name is locked + the container ID name. If so, the lock acquisition is successful; otherwise, the lock acquisition fails.

[0120] 8. If there are multiple locked files, sort them again. The container corresponding to the first file name obtains the lock, and the remaining containers fail to obtain the lock and clean up the files they generated.

[0121] 9. The process ends.

[0122] In the above process, after the program obtains the distributed lock, if the usage time is too long and exceeds 60s, it is necessary to update the last modification time of the lock file during the usage process.

[0123] As shown in Table 1 below, a distributed FTPServer cluster is instantiated

[0124] Table 1

[0125]

[0126] There are 3 server nodes in the system: ServerNode-A, ServerNode-B, ServerNode-C.

[0127] Among them, ServerNode-A is labeled with: "FTP-A", "FTP-C", indicating that it can run 2 types of FTPServer instances, namely "FTP-A" and "FTP-C"; ServerNode-B is labeled with "FTP-A", "FTP-B", and ServerNode-C is labeled with "FTP-B", "FTP-C".

[0128] There are 3 types of FTPServer instances: FTP-A, FTP-B, FTP-C. Among them, FTP-A can serve FTP users user-A1, user-A2, user-A3; the corresponding magnetic array LUN where the working directory is located is LUN-A; FTP-A is configured with 3 running instances: FTPServer-A-1, FTPServer-A-2, FTPServer-A-3. FTP-A can only run on server nodes labeled with "FTP-A", that is, ServerNode-A or ServerNode-B.

[0129] Figure 6 It is a schematic diagram of an optional distributed FTP cluster affinity and HA.

[0130] Due to the affinity principle:

[0131] All FTPServer instances of the FTP-A type: FTPServer-A-1, FTPServer-A-2, FTPServer-A-3 are running on ServerNode-A labeled with "FTP-A", and at the same time the corresponding magnetic array LUN-A is also mounted on ServerNode-A.

[0132] All FTPServer instances of the FTP-B type: FTPServer-B-1, FTPServer-B-2 are running on ServerNode-B labeled with "FTP-B", and at the same time the corresponding magnetic array LUN-B is also mounted on ServerNode-B.

[0133] All FTPServer instances of the FTP-C type: FTPServer-C-1, FTPServer-C-2, FTPServer-C-3 are running on ServerNode-C with the "FTP-C" label, and the corresponding magnetic array LUN-C is also mounted on ServerNode-C.

[0134] At this time, assume that the system has an exception and ServerNode-A crashes. The cluster management system K8s discovers that all FTPServer instances of the FTP-A type: FTPServer-A-1, FTPServer-A-2, FTPServer-A-3 were originally running on ServerNode-A. According to the label principle, it finds the server node ServerNode-B that can run FTP-A type instances (because it has the label "FTP-B"), then first mounts the corresponding magnetic array LUN-A on ServerNode-B, and then switches all FTPServer instances of the FTP-A type: FTPServer-A-1, FTPServer-A-2, FTPServer-A-3 to run on ServerNode-B. At this time, ServerNode-B is running both FTPServer instances of the FTP-B and FTP-A types, and LUN-B and LUN-A are mounted at the same time.

[0135] In this way, the running form after switching meets the label and affinity policies, can use the corresponding local magnetic array LUN, and achieves HA in case of exceptions.

[0136] As Figure 6 shown, when the ServerNode-A node goes down, the FTPServer-A-1, FTPServer-A-2, FTPServer-A-3 instances are migrated to the ServerNode-B node to run as a whole, and Lun-A is mounted on the ServerNode-B node during the cluster migration.

[0137] Among them, the File Transfer Protocol (FTP) is a set of standard protocols for file transfer on the network, and the Logical Unit Number (LUN) is an independent storage unit that can be directly recognized by the server.

[0138] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present invention, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions for causing a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of the present invention.

[0139] In this embodiment, a data transmission device is further provided. This device is used to implement the above embodiments and preferred embodiments, and those that have been described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that can achieve a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0140] Figure 7 is a structural block diagram of a proxy server according to an embodiment of the present invention, as Figure 7 shown, the proxy server includes:

[0141] A receiving unit 702, configured to receive a target request sent by a client. Among them, the target request is a viewing request or a transmission request. The viewing request is used to request to view the target attribute information of a target file in a data server, and the transmission request is used to request to transmit a target file to the data server or obtain the target file from the data server;

[0142] A first obtaining unit 704, configured to, when the target request is a viewing request, obtain the target attribute information of the target file from a saved copy of the attribute information and return it to the client. Among them, the corresponding relationship between the target file identifier of the target file and the target attribute information is recorded in the copy of the attribute information;

[0143] A first sending unit 706, configured to, when the target request is a transmission request, send the target request to a target server in the data server. Among them, the data server is a double-layer stacked FTPServer multi-instance distributed cluster server, the data server includes multiple first servers, and the proxy server records the range of processing FTP users reported by each of the first servers.

[0144] Through the above proxy server, since file scheduling can be performed through the proxy server during the file transmission process, the view request can directly return the target attribute information through the proxy server, and the transmission request can be sent to different target servers for processing. Therefore, the problem of low file transmission efficiency can be solved, and the effect of improving file transmission efficiency can be achieved.

[0145] An embodiment of the present invention also provides a computer-readable storage medium, in which a computer program is stored. Wherein, the computer program is set to execute the steps in any one of the above method embodiments when running.

[0146] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: USB flash drive, read-only memory (ROM for short), random access memory (RAM for short), mobile hard disk, magnetic disk or optical disc and other media that can store computer programs.

[0147] An embodiment of the present invention also provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is set to run the computer program to execute the steps in any one of the above method embodiments.

[0148] In an exemplary embodiment, the above electronic device may further include a transmission device and an input / output device. Wherein, the transmission device is connected to the above processor, and the input / output device is connected to the above processor.

[0149] The specific examples in this embodiment may refer to the examples described in the above embodiments and exemplary embodiments, and will not be repeated here.

[0150] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general-purpose computing device. They can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. They can be implemented by program codes executable by the computing device. Thus, they can be stored in a storage device and executed by the computing device. And in some cases, the steps shown or described can be executed in a different order than here, or they can be separately made into individual integrated circuit modules, or multiple modules or steps of them can be made into a single integrated circuit module to implement. In this way, the present invention is not limited to any specific combination of hardware and software.

[0151] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the present invention. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A data transmission method, characterized in that: include: The proxy server receives a target request sent by the client, wherein the target request is a viewing request or a transfer request, the viewing request is used to request viewing target attribute information of a target file in the data server, and the transfer request is used to request transmitting the target file to the data server or obtaining the target file from the data server; In a case where the target request is a viewing request, the proxy server obtains the target attribute information of the target file from the stored attribute information copy and returns it to the client, wherein the attribute information copy records the correspondence between the target file identifier of the target file and the target attribute information; Using multiple FTPServer instances on the same first server to form a one-layer stacking structure; using multiple first servers to form a two-layer stacking structure to obtain the data server; mounting the multiple FTPServer instances on the same first server on the same disk array LUN high-speed storage; When the target request is a transmission request, the proxy server sends the target request to the target server in the data server, wherein the data server is a double-stacked FTPServer multi-instance distributed cluster server, the data server includes multiple first servers, and the proxy server records the scope of processing FTP users reported by each first server.

2. The method according to claim 1, characterized in that Before the proxy server receives the target request sent by the client, the method further includes: Obtaining file identifiers and attribute information of all files in the data server; The file identifications of all files in the data server and the attribute information of all files are saved in the attribute information copy.

3. The method according to claim 1, characterized in that Before the proxy server receives the target request sent by the client, the method further includes: Acquire a correspondence between a user identifier and the first server, wherein different user identifiers correspond to different first servers; The corresponding relationship is saved in the memory.

4. The method according to claim 1, wherein When the target request is a viewing request, obtaining the attribute information of the target file from the attribute information copy stored in the proxy server and returning the attribute information to the client includes: Obtaining the target file identifier of the target file in the target request; Acquire the target attribute information corresponding to the target file identifier in the attribute information copy; The target attribute information is returned to the client.

5. The method according to claim 1, wherein In a case where the target request is a transmission request, before sending the target request to the target server among the data servers, the method further includes: Obtaining a target user identifier of the client; Searching for the target server corresponding to the target user identifier in the memory; The transmission request is sent to the target server.

6. The method according to claim 5, characterized in that The searching for the target server corresponding to the target user identifier in the memory includes: When multiple servers corresponding to the target user identifier are found, a server with the lowest load balance is selected from the multiple servers as the target server, or the target server is determined from the multiple servers using a polling algorithm.

7. The method according to claim 5, characterized in that In a case where the target request is a transmission request, after sending the target request to the target server among the data servers, the method further includes: Establishing a data transmission channel between the client and the target server; The target file is transmitted between the client and the target server through the data transmission channel.

8. The method according to claim 1, characterized in that Before the proxy server receives the target request sent by the client, the method further includes: If the file size of the attribute information copy is smaller than a predetermined threshold, storing the attribute information copy in a memory; When the file size of the attribute information copy is greater than or equal to a predetermined threshold, the attribute information copy is saved in a memory.

9. A proxy server, characterized in that: include: a receiving unit, configured to receive a target request sent by a client, wherein the target request is a viewing request or a transfer request, the viewing request is used to request viewing target attribute information of a target file in a data server, and the transfer request is used to request transferring a target file to the data server or obtaining the target file from the data server; a first acquiring unit, configured to, when the target request is a viewing request, acquire the target attribute information of the target file from a stored attribute information copy and return the acquired attribute information to the client, wherein the attribute information copy records a correspondence between a target file identifier of the target file and the target attribute information; The first sending unit is used to use multiple FTPServer instances located on the same first server to form a one-layer stacking structure; use multiple first servers to form a double-layer stacking structure to obtain the data server; mount multiple FTPServer instances on the same first server on the same magnetic array LUN high-speed storage; when the target request is a transmission request, send the target request to the target server in the data server, wherein the data server is a double-layer stacked FTPServer multi-instance distributed cluster server, the data server includes multiple first servers, and the proxy server records the scope of processing FTP users reported by each first server.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 8 when executed.

11. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 8.

Images