Data transmission method, device, equipment, system, medium and program product

Abstract

The present application relates to a data transmission method, device, equipment, system, medium and program product. The method comprises: firstly obtaining a file transfer request sent by each user terminal, then obtaining load information of multiple file transfer protocol FTP servers connected to a target load balancer, and finally distributing each file transfer request to a corresponding FTP server according to the load information of each FTP server by using a preset load balancing strategy, wherein the file transfer request in the FTP server is used to instruct the corresponding FTP server to operate a distributed file system HDFS according to the file transfer request to realize data transmission. The method can distribute the received file transfer request according to the load information and load balancing strategy of the FTP server, and the transmission efficiency is higher.

Description

Technical Field

[0001] The present application relates to the field of data communication technology, and in particular to a data transmission method, device, equipment, system, medium and program product. Background Art

[0002] The distributed file system HDFS (Hadoop Distributed File System) can provide high-throughput data access and is very suitable for applications on large-scale data sets. Hdfs-over-ftp is a tool developed based on the FTP server ApacheftpServer and the distributed system infrastructure Apache hadoop. It can manage the distributed file system HDFS. The Hdfs-over-ftp tool can quickly download and upload files in the distributed file system HDFS.

[0003] However, the current HDFS-over-FTP transmission efficiency is not high. When multiple clients access the distributed file system HDFS, if the number of concurrent service requests from each client is large, there may be a problem that HDFS cannot respond to the service requests of some clients. Summary of the invention

[0004] Based on this, it is necessary to provide a data transmission method, device, equipment, system, medium and program product with higher data transmission efficiency in response to the above technical problems.

[0005] In a first aspect, the present application provides a data transmission method, the method comprising:

[0006] Obtain file transfer requests sent by each client;

[0007] Obtain load information of multiple FTP servers connected to the target load balancer;

[0008] According to the load information of each FTP server, a preset load balancing strategy is used to distribute each file transfer request to the corresponding FTP server; the file transfer request in the FTP server is used to instruct the corresponding FTP server to operate the distributed file system HDFS according to the file transfer request to realize data transmission.

[0009] In one of the embodiments, the preset load balancing strategy includes at least one of the following: a polling strategy, a weighted polling strategy, a minimum connection number scheduling strategy, and a target address hash strategy.

[0010] In this embodiment, the target load balancer can distribute the file transfer requests according to different load balancing strategies, thereby ensuring load balancing of each FTP server.

[0011] In one embodiment, the target load balancer is a master load balancer or a slave load balancer, and the method further comprises:

[0012] According to the heartbeat information between the master load balancer and the slave load balancer, detect whether the master load balancer and the slave load balancer are faulty;

[0013] If any of the master load balancer and the slave load balancer fails, the other load balancer that has not failed is determined as the target load balancer.

[0014] In this embodiment, by detecting the heartbeat information between the master load balancer and the slave load balancer, it is determined that the target load balancer provides services for the user, thereby ensuring the high availability of the system.

[0015] In a second aspect, the present application also provides a data transmission method, the method comprising:

[0016] Obtain the file transfer request sent by the target load balancer; the file transfer request is the file transfer request sent by each client received by the target load balancer, and the corresponding FTP server is determined according to the load information of each FTP server and the preset load balancing strategy, and then distributed to the FTP server;

[0017] Operate HDFS according to file transfer requests to achieve data transfer.

[0018] In this embodiment, the lightweight thread design of the FTP server can realize the simultaneous processing of file transfer requests from multiple clients, thereby improving the concurrency of the service.

[0019] In one embodiment, the file transfer request includes a write request, and operating the HDFS according to the file transfer request includes:

[0020] Determine the target resource corresponding to the write request in HDFS;

[0021] The target resource is locked and data is written to the target resource according to the write request.

[0022] In one embodiment, if the file transfer request of each user terminal includes multiple write requests, a lock operation is performed on the target resource, and data writing operations are performed in the target resource according to the write requests, including:

[0023] Determine the priority of each write request;

[0024] The target resource is locked in sequence according to the priority order of each write request, and data is written in the target resource in sequence according to each write request.

[0025] In this embodiment, when the FTP server processes file transfer requests from multiple clients in parallel, the conflict caused by the simultaneous operations of multiple user threads is avoided by locking the target resource, thereby improving the stability of the system.

[0026] In a third aspect, the present application further provides a data transmission device, the device comprising:

[0027] A first acquisition module, used to acquire a file transfer request sent by each user terminal;

[0028] A second acquisition module is used to acquire load information of multiple file transfer protocol FTP servers connected to the target load balancer;

[0029] The distribution module is used to distribute each file transfer request to the corresponding FTP server according to the load information of each FTP server using a preset load balancing strategy; the file transfer request in the FTP server is used to instruct the corresponding FTP server to operate the distributed file system HDFS according to the file transfer request to realize data transmission.

[0030] In a fourth aspect, the present application further provides a data transmission device, the device comprising:

[0031] The request acquisition module is used to acquire the file transfer request sent by the target load balancer; the file transfer request is the file transfer request sent by each client received by the target load balancer, and the corresponding FTP server is determined according to the load information of each FTP server and the preset load balancing strategy, and then distributed to the FTP server;

[0032] The operation module is used to operate HDFS according to the file transfer request to realize data transmission.

[0033] In a fifth aspect, the present application further provides a load balancer, comprising a memory and a processor, wherein the memory stores a computer program, and wherein the processor implements the data transmission method described in any one of the first aspects above when executing the computer program.

[0034] In a sixth aspect, the present application further provides an FTP server, comprising a memory and a processor, wherein the memory stores a computer program, and wherein the processor implements the data transmission method described in any one of the second aspects when executing the computer program.

[0035] In a seventh aspect, the present application further provides a file transfer system, including a load balancer, an FTP server and HDFS,

[0036] The load balancer is used to perform the steps of the method according to any one of claims 1 to 3;

[0037] The FTP server is used to perform the steps of the method according to any one of claims 4 to 6;

[0038] The HDFS is used to respond to the file transfer request of the FTP server and return data corresponding to the file transfer request.

[0039] In an eighth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the data transmission method of the first aspect and / or the second aspect described above is implemented.

[0040] In a ninth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the data transmission method of the first aspect and / or the second aspect mentioned above.

[0041] The above-mentioned data transmission method, device, equipment, system, medium and program product first obtain the file transfer request sent by each user terminal, then obtain the load information of multiple file transfer protocol FTP servers connected to the target load balancer, and finally distribute each file transfer request to the corresponding FTP server according to the load information of each FTP server using a preset load balancing strategy, wherein the file transfer request in the FTP server is used to instruct the corresponding FTP server to operate the distributed file system HDFS according to the file transfer request to realize data transmission. In this way, after receiving the file transfer request sent by each user terminal, the load balancer distributes the received file transfer request to the corresponding FTP server according to the load information and load balancing strategy of the FTP server, so that when the number of concurrent file transfer requests from each user terminal is too large, the file transfer request distributed to the FTP server cannot be responded due to the excessive load of individual FTP servers. Furthermore, the received file transfer request is distributed according to the load information and load balancing strategy of the FTP server, and the transmission efficiency is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 is an application environment diagram of a data transmission method in an embodiment;

[0044] Figure 2A schematic diagram of a flow chart of a data transmission method in an embodiment;

[0045] Figure 3 A schematic diagram of a flow chart of a data transmission method in another embodiment;

[0046] Figure 4 A schematic diagram of a flow chart of a data transmission method in another embodiment;

[0047] Figure 5 A schematic diagram of a multi-threaded response of an FTP server in one embodiment;

[0048] Figure 6 A schematic diagram of a multi-threaded response of an FTP server in another embodiment;

[0049] Figure 7 A schematic diagram of a flow chart of a data transmission method in another embodiment;

[0050] Figure 8 is a schematic diagram of target resource types in another embodiment;

[0051] Fig. 9 A schematic diagram of a flow chart of a data transmission method in another embodiment;

[0052] Fig.10 A schematic diagram of a flow chart of a data transmission method in another embodiment;

[0053] Fig.11 A schematic diagram of the RPC structure of HDFS in another embodiment;

[0054] Fig.12 is a structural block diagram of a data transmission device in one embodiment;

[0055] Fig.13 is a structural block diagram of a data transmission device in another embodiment;

[0056] Fig.14 is an internal structure diagram of a load balancer in one embodiment;

[0057] Fig.15 FIG. 1 is a diagram showing the internal structure of an FTP server in one embodiment. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] The data transmission method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the load balancer 10 is connected to multiple FTP servers 20 in communication, the FTP server 20 is connected to HDFS30 in communication, the user terminal 40 is connected to the load balancer 10 through the network, the user can send a file transfer request to the network through the user terminal 40, and the file transfer request is distributed to the FTP server 20 through the load balancer 10. The FTP service 20 operates HDFS30 according to the file transfer request and then returns the corresponding result of the file transfer request. Among them, the load balancer 10 can be LVS (Linux Virtual Server), and the FTP server 20 can be an independent server or a server cluster composed of multiple servers. The user terminal 40 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc.

[0060] The following embodiment first describes the data transmission method on the load balancer side.

[0061] In an exemplary embodiment, Figure 2 As shown, a data transmission method is provided, which is applied to Figure 1 Taking the load balancer in as an example, the method may include the following steps:

[0062] S202: Obtain a file transfer request sent by each client.

[0063] Each client can send multiple file transfer requests at the same time. The file transfer request may include the user's operation on the file in the system, including the read operation on the file, the write operation on the file, and the permission modification operation on the file. The load balancer can obtain the file transfer request sent by each client through the network.

[0064] S204, obtaining load information of multiple File Transfer Protocol FTP servers connected to the target load balancer.

[0065] Among them, in order to ensure the high availability of the system, two load balancers can be used for dual-machine hot standby service, and the target load balancer is a load balancer that works normally. Optionally, a heartbeat connection is established between the target load balancer and multiple FTP servers, and heartbeat information can be sent to each other through the heartbeat connection. The target load balancer can determine whether the operating status of the FTP server is normal through the heartbeat information of the FTP server. The load information of the FTP server may include the current number of connections of the FTP server or the CPU occupancy rate of the FTP server, etc. The current operating status of the FTP server can be determined based on the load information of the FTP server.

[0066] S206: Distribute each file transfer request to a corresponding FTP server using a preset load balancing strategy according to the load information of each FTP server.

[0067] Among them, the file transfer request in the FTP server is used to instruct the corresponding FTP server to operate the distributed file system HDFS according to the file transfer request to realize data transmission. Optionally, the FTP server can operate HDFS through the HDFS client, and call the HDFS read and write file module through RPC (Remote Procedure Call Protocol) to operate the HDFS file. The target load balancer determines the operating status of each FTP server based on the load information of each FTP server obtained, and then adopts a preset load balancing strategy, such as a weighted polling strategy or a minimum number of connections strategy, to distribute the file transfer requests received from each user terminal to the appropriate FTP server for processing.

[0068] In the above data transmission method, the file transfer request sent by each user terminal is first obtained, and then the load information of multiple file transfer protocol FTP servers connected to the target load balancer is obtained. Finally, according to the load information of each FTP server, a preset load balancing strategy is used to distribute each file transfer request to the corresponding FTP server, wherein the file transfer request in the FTP server is used to instruct the corresponding FTP server to operate the distributed file system HDFS according to the file transfer request to realize data transmission. In this way, after receiving the file transfer request sent by each user terminal, the load balancer distributes the received file transfer request to the corresponding FTP server according to the load information and load balancing strategy of the FTP server, so that when the number of concurrent file transfer requests of each user terminal is too large, the file transfer request distributed to the FTP server cannot be responded due to the excessive load of individual FTP servers. Furthermore, the received file transfer request is distributed according to the load information and load balancing strategy of the FTP server, and the transmission efficiency is higher.

[0069] In another embodiment, the above-mentioned preset load balancing strategy includes at least one of the following: a polling strategy, a weighted polling strategy, a minimum connection number scheduling strategy, and a target address hash strategy.

[0070] Among them, the polling strategy is that the target load balancer allocates file transfer requests to each FTP server in sequence. This method can ensure that each FTP server can share the load evenly. The weighted round-robin strategy is that the target load balancer assigns a weight value to each FTP server, and then allocates the proportion of file transfer requests according to the weight value. The FTP server with a higher weight value will be allocated more requests. Optionally, the weight value can be set according to the hardware configuration of the FTP server. The higher the hardware configuration of the FTP server, the higher the weight value. The minimum connection number scheduling strategy is that the target load balancer allocates file transfer requests to the FTP server with the smallest current number of connections. This method can ensure that the load of each FTP server is as balanced as possible. The target address hashing strategy is that the target load balancer allocates file transfer requests to the FTP server through a hash function based on the IP address of the user end.

[0071] In this embodiment, the target load balancer can distribute the file transfer requests according to different load balancing strategies, thereby ensuring load balancing of each FTP server.

[0072] In another embodiment, a method for determining a target load balancer is provided, wherein the target load balancer is a master load balancer or a slave load balancer, such as Figure 3 As shown, the steps include:

[0073] S302: Detect whether the master load balancer and the slave load balancer fail according to the heartbeat information between the master load balancer and the slave load balancer.

[0074] The load balancer includes a master load balancer and a slave load balancer. The master load balancer and the slave load balancer are connected by a heartbeat, and can send heartbeat information to each other to detect whether the working state of the other party fails. Optionally, LVS is used as a load balancer. LVS is a load balancer working at layer 4, a TCP / IP protocol stack working in the kernel space, and is part of the Linux standard kernel. KeepAlived is used as a dual-machine hot standby service for the load balancer LVS.

[0075] S304: If any one of the master load balancer and the slave load balancer fails, another load balancer that is not failed is determined as the target load balancer.

[0076] In this step, when the slave load balancer does not receive the heartbeat information sent by the master load balancer for a period of time, it is determined that the master load balancer has failed. At this time, if the slave load balancer has not failed, the slave load balancer will be determined as the target load balancer to receive the user's file transfer request and distribute it. When the slave load balancer is the target load balancer, after the master load balancer fails, if the master load balancer does not receive the heartbeat information sent by the slave load balancer for a period of time, it is determined that the slave load balancer has failed. At this time, if the master load balancer has not failed, the master load balancer will be determined as the target load balancer.

[0077] In this embodiment, by detecting the heartbeat information between the master load balancer and the slave load balancer, it is determined that the target load balancer provides services for the user, thereby ensuring the high availability of the system.

[0078] The following embodiments illustrate the data transmission method on the FTP server side.

[0079] In an exemplary embodiment, Figure 4 As shown, a data transmission method is provided, which is applied to Figure 1 The following is an example of an FTP server in the example, including:

[0080] S402: Obtain a file transfer request sent by a target load balancer.

[0081] Among them, the target load balancer receives the file transfer request sent by each user terminal, and then determines the corresponding FTP server based on the load information of each FTP server and adopts the preset load balancing strategy to distribute it to the FTP server. Each FTP server obtains the file transfer request distributed to each FTP server by the target load balancer. Among them, the FTP server can use the lightweight thread feature of the Go language to process file transfer requests from multiple users in parallel, thereby improving the concurrency of the FTP service. Figure 5 As shown in the figure, the FTP server can receive requests from multiple clients. Client threads 1 to 3 are threads of three clients. The select() function interface of the FTP server can detect the read status, write status and error status of multiple handles at the same time, so a server system that provides independent question-and-answer services for multiple clients can be constructed. As shown in the figure, fd1 is the handle of user thread 1, fd2 is the handle of user thread 2, and fd3 is the handle of user thread 3. The client thread sends a request through send(). A specific execution cycle is as follows: Figure 6As shown in the figure, the FTP server thread finds that a handle has captured a readable event in the Select() function, that is, a client thread has sent a file transfer request. The FTP server program promptly performs the Receive() operation and prepares the data to be sent based on the received file transfer request data. If Select() captures a writable event, the program promptly performs the Send() operation and prepares for the next file transfer request detection.

[0082] S404, operating HDFS according to the file transfer request to realize data transmission.

[0083] Each FTP server operates HDFS through the HDFS client according to the file transfer request, and then responds to the file transfer request from the user according to the return result of HDFS to complete the data transfer.

[0084] In this embodiment, the lightweight thread design of the FTP server can realize the simultaneous processing of file transfer requests from multiple clients, thereby improving the concurrency of the service.

[0085] In another embodiment, another data transmission method is provided. Based on the above embodiment, the file transfer request includes a write request. The steps of operating the HDFS according to the file transfer request are as follows: Figure 7 As shown, including:

[0086] S702, determine the target resource corresponding to the write request in HDFS.

[0087] When the file transfer request is a write request, the target resource is the content that the write request needs to write. When the write request needs to operate on a file resource, the target resource is the file resource; when the write request needs to operate on a path resource, the target resource is the path resource.

[0088] S704: Lock the target resource and write data to the target resource according to the write request.

[0089] When a user thread of the FTP server makes a write request to a target resource, other user threads cannot operate on the target resource. Therefore, it is necessary to lock the target resource, that is, add a mutex tree lock to the target resource, and then write data to the target resource according to the write request. Figure 8 If the target resource is a file resource, the corresponding file is locked; if the target resource is a path resource, all resources under the path tree are locked.

[0090] In another embodiment, based on the above embodiment, if the file transfer request of each user terminal includes multiple write requests, the target resource is locked, and the steps of performing data writing operation in the target resource according to the write request are as follows: Fig. 9 As shown, including:

[0091] S902: Determine the priority of each write request.

[0092] When multiple user threads simultaneously perform write operations on the target resource, according to the priority of the write request of each user thread, optionally, the priority may be the sending time of each user write request, and the earlier the sending time, the higher the priority.

[0093] S904: Lock the target resource in sequence according to the priority order of each write request, and write data in the target resource in sequence according to each write request.

[0094] The target resource is locked according to the priority order of each write request. When a user thread performs a write request operation on the target resource, other user threads queue up and wait for the mutex tree lock to be released before the next user thread continues processing.

[0095] Optionally, the file transfer request also includes resource change operations such as file permission changes. The resource change operations are the same as the write request operations described above, and both require locking the target resource. The file transfer request also includes a read request. The current user thread performs a read request operation on the target resource. When a new user thread also performs a read request operation on the target resource, they can be processed in parallel and the contents of the target resource can be read at the same time. When a new user thread needs to perform a write request or resource change operation on the target resource, it also needs to be queued and processed after the read request operation of the current user thread is completed.

[0096] In this embodiment, when the FTP server processes file transfer requests from multiple clients in parallel, the conflict caused by the simultaneous operations of multiple user threads is avoided by locking the target resource, thereby improving the stability of the system.

[0097] In the embodiments of the present application, in order to facilitate the understanding of the reader, the data transmission method provided in the present application is illustrated by way of example. For the specific process, please refer to Fig.10. The client sends a file transfer request to the target load balancer. The target load balancer and the FTP server exchange heartbeat information. The target load balancer distributes the file transfer request to the FTP server according to the preset load balancing strategy. The FTP server checks the file transfer request and determines the type of the file transfer request. Then, it determines whether to lock the target resource based on the type of the file transfer request. Then, it calls the HDFS interface, operates HDFS through the HDFS client, and returns the result of the read request or write request corresponding to the file transfer request. The FTP server responds to the file transfer request from the client. During the processing of the file transfer request, if an exception occurs in any step of the load balancer, FTP server, HDFS client, or HDFS, the corresponding exception is returned to the client.

[0098] Optionally, the HDFS client module is also developed in Go language. It calls the HDFS API to read and write files, receives read requests or write requests and data streams sent by the FTP server, and calls the HDFS API through the RPC protocol to operate HDFS files. Fig.11 The figure shows the RPC structure of the HDFS client. The client provides methods such as add, delete, modify, and query, and the client is an RPC client. When the FTP server sends a write request, the HDFS client obtains the write request and data stream, calls the corresponding local method, and calls the remote server, that is, HDFS, through the RPC interface to complete the operation. In the design of the RPC data set, the lightweight data serialization protocol Protobuf is used. This data serialization protocol is simple to maintain data, so it is very suitable for using a simple and fast Protobuf data structure.

[0099] In this embodiment, the file transfer efficiency of the HDFS client can be improved through the setting of the HDFS client.

[0100] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0101] Based on the same inventive concept, the embodiment of the present application also provides a data transmission device for implementing the data transmission method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data transmission device embodiments provided below can refer to the limitations on the data transmission method above, and will not be repeated here.

[0102] In an exemplary embodiment, Fig.12 As shown, a data transmission device is provided, comprising: a first acquisition module, a second acquisition module and a distribution module, wherein:

[0103] The first acquisition module is used to acquire the file transfer request sent by each user terminal.

[0104] The second acquisition module is used to acquire load information of multiple File Transfer Protocol FTP servers connected to the target load balancer.

[0105] The distribution module is used to distribute each file transfer request to the corresponding FTP server according to the load information of each FTP server using a preset load balancing strategy; the file transfer request in the FTP server is used to instruct the corresponding FTP server to operate the distributed file system HDFS according to the file transfer request to realize data transmission.

[0106] In one embodiment, the preset load balancing strategy includes at least one of the following: a polling strategy, a weighted polling strategy, a minimum connection number scheduling strategy, and a target address hash strategy.

[0107] In one embodiment, the target load balancer is a master load balancer or a slave load balancer, and further includes a detection module:

[0108] A fault detection unit, used to detect whether the master load balancer and the slave load balancer have faults according to the heartbeat information between the master load balancer and the slave load balancer;

[0109] The determination unit is used to determine another load balancer that has not failed as the target load balancer if any one of the master load balancer and the slave load balancer fails.

[0110] Each module in the above data transmission device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0111] In an exemplary embodiment, Fig.13As shown, a data transmission device is provided, including: a request acquisition module and an operation module, wherein:

[0112] The request acquisition module is used to acquire the file transfer request sent by the target load balancer; the file transfer request is the file transfer request sent by each user terminal received by the target load balancer, and the corresponding FTP server is determined based on the load information of each FTP server and the preset load balancing strategy, and then distributed to the FTP server.

[0113] This operation module is used to operate HDFS according to the file transfer request to realize data transmission.

[0114] In one embodiment, the file transfer request includes a write request, and the operation module includes:

[0115] A determination unit is used to determine the target resource corresponding to the write request in HDFS;

[0116] The locking unit is used to lock the target resource and perform data writing operations in the target resource according to the write request.

[0117] In one embodiment, if the file transfer request of each user terminal includes multiple write requests, the locking unit is specifically used to determine the priority of each write request; lock the target resource in order of the priority of each write request, and perform data writing operations in the target resource in order according to each write request.

[0118] Each module in the above data transmission device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0119] In an exemplary embodiment, a load balancer is provided, and its internal structure diagram can be shown as follows: Fig.14As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store file transfer request data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data transmission method is implemented.

[0120] Those skilled in the art will understand that Fig.14 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0121] In an exemplary embodiment, a server is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the following steps when executing the computer program: obtaining a file transfer request sent by each user terminal; obtaining load information of multiple file transfer protocol FTP servers connected to a target load balancer; distributing each file transfer request to a corresponding FTP server using a preset load balancing strategy according to the load information of each FTP server; the file transfer request in the FTP server is used to instruct the corresponding FTP server to operate a distributed file system HDFS according to the file transfer request to realize data transmission.

[0122] In one embodiment, the preset load balancing strategy includes at least one of the following: a polling strategy, a weighted polling strategy, a minimum connection number scheduling strategy, and a target address hash strategy.

[0123] In one embodiment, the target load balancer is a master load balancer or a slave load balancer, and the processor further implements the following steps when executing the computer program: based on the heartbeat information between the master load balancer and the slave load balancer, detect whether the master load balancer and the slave load balancer fail; if any one of the master load balancer and the slave load balancer fails, determine the other load balancer that has not failed as the target load balancer.

[0124] In an exemplary embodiment, a FTP server is provided, whose internal structure diagram can be shown as follows: Fig.15 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store file transfer request data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data transmission method is implemented.

[0125] Those skilled in the art will understand that Fig.15 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0126] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the following steps when executing the computer program: obtaining a file transfer request sent by a target load balancer; after the target load balancer receives the file transfer request sent by each user terminal, the target load balancer determines the corresponding FTP server based on the load information of each FTP server obtained and adopts a preset load balancing strategy and then distributes it to the FTP server; and operating HDFS according to the file transfer request to realize data transmission.

[0127] In one embodiment, the file transfer request includes a write request, and the processor implements the following steps when executing the computer program: determining the target resource corresponding to the write request in HDFS; locking the target resource, and performing data writing operations in the target resource according to the write request.

[0128] In one embodiment, if the file transfer request of each user terminal includes multiple write requests, the processor implements the following steps when executing the computer program: determining the priority of each write request; locking the target resource in order of the priority of each write request, and performing data writing operations in the target resource in order according to each write request.

[0129] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: obtaining a file transfer request sent by each user terminal; obtaining load information of multiple file transfer protocol FTP servers connected to a target load balancer; distributing each file transfer request to a corresponding FTP server using a preset load balancing strategy based on the load information of each FTP server; the file transfer request in the FTP server is used to instruct the corresponding FTP server to operate a distributed file system HDFS according to the file transfer request to achieve data transmission.

[0130] In one embodiment, the preset load balancing strategy includes at least one of the following: a polling strategy, a weighted polling strategy, a minimum connection number scheduling strategy, and a target address hash strategy.

[0131] In one embodiment, the target load balancer is a master load balancer or a slave load balancer, and the computer program also implements the following steps when executed by the processor: based on the heartbeat information between the master load balancer and the slave load balancer, detect whether the master load balancer and the slave load balancer fail; if any one of the master load balancer and the slave load balancer fails, determine the other load balancer that has not failed as the target load balancer.

[0132] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining a file transfer request sent by the target load balancer; the file transfer request is a file transfer request sent by each user terminal received by the target load balancer, and the corresponding FTP server is determined based on the load information of each FTP server and a preset load balancing strategy, and then distributed to the FTP server; HDFS is operated according to the file transfer request to realize data transmission.

[0133] In one embodiment, the file transfer request includes a write request, and the computer program, when executed by the processor, further implements the following steps: determining a target resource corresponding to the write request in HDFS; locking the target resource, and performing a data write operation in the target resource according to the write request.

[0134] In one embodiment, if the file transfer request of each user terminal includes multiple write requests, the computer program also implements the following steps when executed by the processor: determining the priority of each write request; locking the target resource in order of the priority of each write request, and performing data writing operations in the target resource in order according to each write request.

[0135] In one embodiment, a computer program product is provided, including a computer program, which implements the following steps when executed by a processor: obtaining a file transfer request sent by each user terminal; obtaining load information of multiple file transfer protocol FTP servers connected to a target load balancer; distributing each file transfer request to a corresponding FTP server using a preset load balancing strategy based on the load information of each FTP server; the file transfer request in the FTP server is used to instruct the corresponding FTP server to operate a distributed file system HDFS according to the file transfer request to achieve data transmission.

[0136] In one embodiment, the preset load balancing strategy includes at least one of the following: a polling strategy, a weighted polling strategy, a minimum connection number scheduling strategy, and a target address hash strategy.

[0137] In one embodiment, the target load balancer is a master load balancer or a slave load balancer, and the computer program also implements the following steps when executed by the processor: based on the heartbeat information between the master load balancer and the slave load balancer, detect whether the master load balancer and the slave load balancer fail; if any one of the master load balancer and the slave load balancer fails, determine the other load balancer that has not failed as the target load balancer.

[0138] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining a file transfer request sent by the target load balancer; the file transfer request is a file transfer request sent by each user terminal received by the target load balancer, and the corresponding FTP server is determined based on the load information of each FTP server and a preset load balancing strategy, and then distributed to the FTP server; HDFS is operated according to the file transfer request to realize data transmission.

[0139] In one embodiment, the file transfer request includes a write request, and the computer program, when executed by the processor, further implements the following steps: determining a target resource corresponding to the write request in HDFS; locking the target resource, and performing a data write operation in the target resource according to the write request.

[0140] In one embodiment, if the file transfer request of each user terminal includes multiple write requests, the computer program also implements the following steps when executed by the processor: determining the priority of each write request; locking the target resource in order of the priority of each write request, and performing data writing operations in the target resource in order according to each write request.

[0141] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0142] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0143] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0144] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data transmission method, characterized in that: The method comprises: Obtain file transfer requests sent by each client; Obtain load information of multiple File Transfer Protocol (FTP) servers connected to the target load balancer; According to the load information of each FTP server, a preset load balancing strategy is adopted to distribute each file transfer request to the corresponding FTP server; the file transfer request in the FTP server is used to instruct the corresponding FTP server to operate the distributed file system HDFS according to the file transfer request to realize data transmission.

2. The method according to claim 1, characterized in that: The preset load balancing strategy includes at least one of the following: a polling strategy, a weighted polling strategy, a minimum connection number scheduling strategy, and a target address hash strategy.

3. The method according to claim 1, characterized in that The target load balancer is a master load balancer or a slave load balancer, and the method further comprises: Detecting whether the master load balancer and the slave load balancer fail according to the heartbeat information between the master load balancer and the slave load balancer; If any one of the master load balancer and the slave load balancer fails, another load balancer that has not failed is determined as the target load balancer.

4. A data transmission method, characterized in that: The method comprises: Obtaining a file transfer request sent by a target load balancer; the file transfer request is a file transfer request sent by each client received by the target load balancer, and after the target load balancer determines the corresponding FTP server based on the load information of each FTP server obtained and adopts a preset load balancing strategy, the file transfer request is distributed to the FTP server; The HDFS is operated according to the file transfer request to realize data transmission.

5. The method according to claim 4, characterized in that The file transfer request includes a write request, and the operating the HDFS according to the file transfer request includes: Determine the target resource corresponding to the write request in the HDFS; A locking operation is performed on the target resource, and a data writing operation is performed in the target resource according to the write request.

6. The method according to claim 5, characterized in that If the file transfer request of each of the user terminals includes multiple write requests, the locking operation on the target resource and performing a data writing operation in the target resource according to the write request include: Determining the priority of each of the write requests; The target resource is locked in sequence according to the priority order of each write request, and data writing operation is performed in sequence in the target resource according to each write request.

7. A data transmission device, characterized in that: The device comprises: A first acquisition module, used to acquire a file transfer request sent by each user terminal; A second acquisition module is used to acquire load information of multiple file transfer protocol FTP servers connected to the target load balancer; The distribution module is used to distribute each file transfer request to the corresponding FTP server according to the load information of each FTP server using a preset load balancing strategy; the file transfer request in the FTP server is used to instruct the corresponding FTP server to operate the distributed file system HDFS according to the file transfer request to realize data transmission.

8. A data transmission device, characterized in that: The device comprises: A request acquisition module is used to acquire a file transfer request sent by a target load balancer; the file transfer request is a file transfer request sent by each client received by the target load balancer, and the target load balancer determines the corresponding FTP server based on the load information of each FTP server acquired and adopts a preset load balancing strategy and then distributes it to the FTP server; The operation module is used to operate the HDFS according to the file transfer request to realize data transmission.

9. A load balancer, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 3 are implemented.

10. An FTP server, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 4 to 6 are implemented.

11. A file transfer system, comprising a load balancer, an FTP server and HDFS, characterized in that: The load balancer is used to perform the steps of the method according to any one of claims 1 to 3; The FTP server is used to perform the steps of the method according to any one of claims 4 to 6; The HDFS is used to respond to the file transfer request of the FTP server and return data corresponding to the file transfer request.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images