Data transmission method, device, equipment, medium and product

Abstract

The invention relates to the technical field of computers, and particularly provides a data transmission method and device, equipment, a medium and a product. The method comprises the following steps: receiving a target data transmission task from an FTP client; determining a load distribution weight of each candidate data agent node based on node performance information and node use information of a plurality of candidate data agent nodes obtained from a registration center; determining a target data agent node based on each load distribution weight and the connection number information of each candidate data agent node; and sending the target data transmission task to the target data agent node. And determining a node load utilization rate condition through the node performance information and the node use information of the proxy node so as to determine a load distribution weight of each candidate data proxy node. Therefore, whether each node has spare resources or not can be accurately judged, and all tasks can be prevented from being distributed to the previous nodes with spare resources in combination with the connection number information. Therefore, the data transmission efficiency can be improved.

Description

Technical Field

[0001] This disclosure belongs to the field of computer technology, and specifically relates to a data transmission method, apparatus, device, medium and product. Background Technology

[0002] Hadoop's HDFS (Hadoop Distributed File System) is highly fault-tolerant and provides high-throughput data access, making it ideal for applications with large-scale datasets. The hdfs-over-ftp component is an FTP (File Transfer Protocol) server built on top of HDFS, providing tools for accessing HDFS via the FTP protocol. Users can easily upload and download files to and from the HDFS file system. The hdfs-over-ftp component includes FTP server nodes and data proxy nodes. The FTP server nodes can distribute data transfer tasks to appropriate data proxy nodes for execution.

[0003] In related technologies, when selecting data proxy nodes, the node with the lowest load is often chosen based on its own performance to perform data transmission tasks. However, this data transmission method considers only a single load-affecting metric and cannot achieve end-to-end load balancing on the server. Therefore, it reduces data transmission efficiency. Summary of the Invention

[0004] This disclosure addresses some of the shortcomings mentioned in the background art by providing a data transmission method, apparatus, device, medium, and product that can improve data transmission efficiency.

[0005] In a first aspect, embodiments of this disclosure provide a data transmission method applied to an FTP server node, the method comprising: Receive target data transfer tasks from FTP clients; Based on the node performance information and node usage information of multiple candidate data proxy nodes obtained from the registration center, the load allocation weight of each candidate data proxy node is determined. The target data proxy node is determined based on the load allocation weights and the connection count information of each candidate data proxy node. The target data transmission task is sent to the target data proxy node.

[0006] Optionally, determining the load allocation weight of each candidate data proxy node based on node performance information and node usage information obtained from the registration center includes: For any candidate data proxy node, based on the node performance information, a first node parameter of the candidate data proxy node is determined, wherein the first node parameter is used to characterize the maximum processing capacity of the candidate data proxy node. Based on the node usage information, the difference between node warm-up time and service running time, the second node parameters of the candidate data proxy node are determined, and the second node parameters characterize the actual processing capability of the candidate data proxy node. Based on the node usage information, determine the load allocation weight influencing factor; Based on the first node parameters, the second node parameters, and the load allocation weight influence factor, the load allocation weight of each candidate data proxy node is determined.

[0007] Optionally, the node performance information includes at least one of the following: CPU processing speed, memory size, network bandwidth, and disk read / write speed; The node usage information includes at least one of the following: CPU usage, memory usage, network usage, and disk read / write usage.

[0008] Optionally, determining the first node parameters of any candidate data proxy node based on the node performance information includes: Multiple performance parameters are obtained by multiplying the CPU processing speed, memory size, network bandwidth, disk read / write speed, and the corresponding node performance information. The sum of the various performance parameters is calculated to obtain the first node parameter.

[0009] Optionally, determining the second node parameters of the candidate data proxy node based on the node usage information, the difference between the node warm-up time and the service runtime includes: When the service runtime difference is greater than or equal to the node warm-up time, the product of the CPU utilization, memory utilization, network utilization, disk read / write utilization and the corresponding node usage information is obtained to get multiple usage parameters; the sum of the multiple usage parameters is obtained to get the second node parameter; the service runtime difference is the difference between the current time and the startup time of the candidate data proxy node service; the node warm-up time is the time taken for the candidate data proxy node to run to the first state; When the difference in service runtime is less than the node warm-up time, the product of the CPU utilization, memory utilization, network utilization, disk read / write utilization and the corresponding node usage information is obtained to get multiple usage parameters; based on the sum of the multiple usage parameters, the node warm-up time and the difference in service runtime, the second node parameter is calculated.

[0010] Optionally, determining the load allocation weighting influencing factor based on the node usage information includes: Multiple usage parameters are obtained by multiplying the CPU utilization, memory utilization, network utilization, disk read / write utilization, and the corresponding node usage information. Determine the maximum value among the CPU utilization, memory utilization, network utilization, and disk read / write utilization; The load allocation weighting influence factor is determined based on the maximum value and multiple usage parameters.

[0011] Optionally, determining the target data proxy node based on the load allocation weights and the connection count information of each candidate data proxy node includes: For any first node among the candidate data proxy nodes, when the second node is a node other than the last node among the multiple candidate data proxy nodes, the product of the connection number information of the first node and the load distribution weight of the second node is determined sequentially, and the product of the connection number information of the second node and the load distribution weight of the first node is determined; the second node is the next node after the first node. Update the first node and / or the second node based on the judgment result; Until the second node is the last of the multiple candidate data proxy nodes, the current first node is taken as the target data proxy node.

[0012] Optionally, the connection count information is positively correlated with the connection type and the number of connections.

[0013] Optionally, the method further includes: Before the FTP server node executes FTP type commands, the current working directory is changed; the command to change the current working directory is used to change the target namespace of the FTP type commands, and the target namespace is an uninitialized namespace.

[0014] Optionally, changing the current working directory includes: Intercept the FTP type commands using a global interceptor; When the FTP type command is to display the current working directory and the access path includes a namespace, the access path is reset to the target access path, wherein the target access path corresponds to the target namespace.

[0015] Optionally, the method further includes: Monitor the node thread pool; When a configuration file update indication exists in the node thread pool, a hot reload update of the configuration file is performed.

[0016] Optionally, the step of performing a hot reload update of the configuration file when a configuration file update indication exists in the node thread pool includes: Intercept the configuration file update instruction using a global interceptor; Determine whether the global variable configuration field in the configuration file update instruction is incomplete; If so, determine whether the user who executed the configuration file exists in the user information of the FTP server node after the update; If so, determine whether the configuration information of the executing user has been updated; if so, execute the configuration file update instruction. Based on the configuration information of the executing user, update the FTP session information of the executing user, and update the global variable configuration field to "completed".

[0017] Optionally, the method further includes: When the target data transmission task is received from the target data proxy node, the target data proxy node is re-determined.

[0018] In a second aspect, embodiments of this disclosure provide a data transmission method applied to a target data proxy node, wherein the target data proxy node is determined based on load balancing weights and connection count information of each candidate data proxy node, the method comprising: Receive the target data transfer task from the FTP server node; The target data transfer task is sent to the HDFS distributed file system.

[0019] Optionally, the method further includes: Determine whether the node weight update object of the target data proxy node is locked; If locked, stop updating node performance information and node usage information; If the lock is released, the node performance information and the node usage information are updated.

[0020] Optionally, the method further includes: When it is determined multiple times that the number of tasks in the thread pool of the target data proxy node is greater than a preset threshold, the target data transmission task is returned to the FTP server node.

[0021] In a third aspect, embodiments of this disclosure provide a data transmission apparatus applied to an FTP server node, the apparatus comprising: The first receiving module is used to receive the target data transfer task from the FTP client. The first determining module is used to determine the load allocation weight of each candidate data proxy node based on the node performance information and node usage information of multiple candidate data proxy nodes obtained from the registration center. The second determining module is used to determine the target data proxy node based on the load allocation weights and the connection number information of each candidate data proxy node. The first sending module is used to send the target data transmission task to the target data proxy node.

[0022] In a fourth aspect, embodiments of this disclosure provide a data transmission apparatus applied to a target data proxy node, the apparatus comprising: The second receiving module is used to receive the target data transmission task from the FTP server node; The second sending module is used to send the target data transmission task to the HDFS distributed file system.

[0023] In a fifth aspect, embodiments of this disclosure provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described data transmission method.

[0024] In a sixth aspect, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the above-described data transmission method.

[0025] In a seventh aspect, embodiments of this disclosure provide a computer program product including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device executes the above-described data transmission method.

[0026] In this disclosure, a target data transfer task is received from an FTP client; based on node performance and usage information of multiple candidate data proxy nodes obtained from a registry center, the load allocation weight of each candidate data proxy node is determined; based on the load allocation weight and the connection count information of each candidate data proxy node, a target data proxy node is determined; and the target data transfer task is sent to the target data proxy node. By using the node performance and usage information of the proxy node, the node load utilization rate is determined, thereby determining the load allocation weight of each candidate data proxy node. This allows for accurate assessment of whether each node has spare resources, and by combining connection count information, it avoids distributing all tasks to previously resource-spared nodes, preventing sudden increases in node connection counts from reducing task processing efficiency, and ensuring that the target data transfer task is sent to a suitable data proxy node for execution. Therefore, data transfer efficiency can be improved.

[0027] It should be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide further illustration of the claimed technology. Attached Figure Description

[0028] Figure 1 A flowchart of a data transmission method provided in this disclosure.

[0029] Figure 2 This is a functional architecture diagram of data transmission provided in this disclosure.

[0030] Figure 3 A flowchart for the target data proxy node provided in this disclosure.

[0031] Figure 4 Another flowchart of a data transmission method provided in this disclosure.

[0032] Figure 5 This is a schematic diagram of a data transmission device provided in this disclosure.

[0033] Figure 6 This is another structural schematic diagram of a data transmission device provided in this disclosure.

[0034] Figure 7 This is a hardware block diagram of an electronic device provided in this disclosure.

[0035] Figure 8 This is a schematic diagram of a computer program product provided in this disclosure. Detailed Implementation

[0036] To enable those skilled in the art to better understand the technical solution of this application, the application scenario of this application will be described first below.

[0037] In recent years, with the continuous development of cloud computing and big data technologies, more and more people are using big data technologies to analyze and store massive amounts of data. Hadoop's HDFS (Hadoop Distributed File System) has high fault tolerance and provides high-throughput data access, making it very suitable for applications on large-scale datasets. The hdfs-over-ftp component is an FTP (File Transfer Protocol) server built on top of HDFS, providing a tool for accessing HDFS via the FTP protocol. Users can easily upload and download data from the HDFS file system. The hdfs-over-ftp component includes an FTP server node and a data proxy node. The FTP server node can distribute data transfer tasks to the appropriate data proxy node for execution. In large-scale data transfer scenarios, deploying a single HDFS-over-FTP service for data transmission is problematic because the CPU, memory, network bandwidth, and disk I / O resources of a single node are limited. This can lead to excessive pressure on the node and time-consuming data transfer. In such cases, the only solution is to deploy multiple HDFS-over-FTP services. However, these services need to expose multiple addresses, requiring manual allocation of specific business logic to different addresses. Adding or removing HDFS-over-FTP servers necessitates modifications to the application's configuration or code to connect to the services, which is opaque to the application, hinders service expansion, and makes load balancing between multiple services impossible.

[0038] In related technologies, a distributed FTP tool is considered for building on HDFS. Users can achieve unified data access using a single FTP address without manually configuring multiple FTP addresses or pre-dividing requests, and tasks are evenly distributed based on node load. When selecting a data proxy node, its performance is often assessed using a dynamic weighted load balancing algorithm and performance information to determine the data proxy node with the lowest load among available nodes, which is then used to execute data transmission tasks. However, this data transmission method considers only a single load-affecting metric and cannot achieve end-to-end load balancing on the server. Therefore, it reduces data transmission efficiency.

[0039] To address the aforementioned technical problems, this disclosure provides an inventive concept: by determining the node load utilization rate through the node performance and usage information of proxy nodes, the load allocation weight of each candidate data proxy node can be determined. This allows for accurate assessment of whether each node has spare resources. Combined with connection count information, it can prevent all tasks from being distributed to previously resource-spared nodes in the event of a sudden increase in concurrency within a period, thus avoiding a surge in node connections, reduced task processing efficiency, and prolonged task execution time. This ensures that target data transmission tasks can be sent to appropriate data proxy nodes for execution, thereby balancing the service load. Therefore, data transmission efficiency can be improved.

[0040] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present disclosure and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the drawings, not the entire structure.

[0041] Figure 1 This is a flowchart illustrating a data transmission method provided in this disclosure. Figure 1 As shown, this method is applied to an FTP server node and includes: S101: Receive the target data transfer task from the FTP client.

[0042] Specifically, Figure 2 The functional architecture diagram of data transmission provided in this disclosure is as follows: Figure 2 As shown. This disclosure mainly includes several parts: FTP Client, keepalived+haproxy, zookeeper cluster, FTP Server, dataproxy, and HDFS distributed file system.

[0043] The Keepalived+HAProxy setup utilizes HAProxy for load balancing. Keepalived provides a virtual VIP and ensures high availability for HAProxy. Distributed FTP services can achieve unified data access using this virtual VIP. The ZooKeeper cluster acts as the registry center, providing registration, listening, heartbeat functionality for the FTP Server and DataProxy services, as well as node performance and usage information for each node, including CPU, memory, network utilization, disk I / O, and connection count. The FTP Server handles command interaction and task scheduling with the FTP Client. Upon application startup, it registers node IP, CPU, memory, load, connection count, and waiting task count with the registry center and updates node information at configurable intervals. The FTP Server reads available DataProxy nodes from the registry center, obtains CPU, memory, load, and network utilization information for each DataProxy node in the ZooKeeper cluster, distributes tasks to specific DataProxy nodes for execution based on the corresponding load balancing algorithm, and updates the connection count information of each DataProxy node. Dataproxy acts as a data proxy node, serving as an intermediary layer for data transmission. It is responsible for accepting task scheduling from the FTP server, completing file data transfers, and notifying the FTP server of task transfer nodes, ports, and results. When the application starts, it registers node information such as IP address, CPU usage, memory, load, network utilization, and connection count with the registry center, and updates this information at configurable time intervals. The Hadoop Distributed File System (HDFS) needs to read from the Hadoop Distributed File System.

[0044] Furthermore, the interaction process between the FTP client, FTP server, and dataproxy is as follows: 1) The FTP client initiates a data transfer request, and the FTP server receives authentication information containing the username and password. 2) After successful authentication, the FTP server obtains the corresponding dataproxy node according to the load balancing algorithm and updates the connection count information of that node. 3) In passive mode (PASV), the dataproxy node is notified to listen. 4) After receiving the FTP server's request to open a port, the dataproxy opens a data service port to wait for the FTP client's connection and sends the opened data service port to the FTP server. 5) After receiving the dataproxy message, the FTP server returns the node address and port to the FTP client, and the FTP client connects to the corresponding address to transfer data to perform file read and write operations with HDFS. 6) In active mode (PORT), the client actively sends the data connection port to the FTP server, and the FTP server notifies the node to establish a data connection with the client's port. 7) After the data transfer is completed, the dataproxy sends a data connection closure message to the FTP server. The FTP server sends a data connection closure message to the dataproxy, releases the port number allocated by the execution command, and updates the connection count information of the dataproxy node. In active mode (PORT), the client actively sends the data connection port to the FTP server, and the FTP server notifies the node to establish a data connection with the client port.

[0045] In other words, during the data transfer process, the FTP client needs to provide the target data transfer task. After unified access to the data is achieved through a virtual VIP, the task is transferred to the FTP client. The hdfs-over-ftp tool then performs load balancing, transferring the task to a suitable data proxy, and finally sending it to the HDFS distributed file system to store the data.

[0046] The data transmission method in this embodiment, through an improved load balancing algorithm and enhanced functionality of the open-source hdfs-over-ftp tool, achieves load balancing among transmission nodes, thereby improving the scalability, robustness, availability, and business continuity of the entire system. It is applicable to scenarios such as high concurrency and large-scale data transmission.

[0047] S102: Based on the node performance information and node usage information of multiple candidate data proxy nodes obtained from the registration center, determine the load distribution weight of each candidate data proxy node.

[0048] Specifically, such as Figure 2As shown, the registry center, i.e., the ZooKeeper cluster, is used to provide registration, listening, heartbeat, and node performance and usage information for FTP Server and DataProxy services. Therefore, by obtaining the performance and usage information of each candidate data proxy node from the registry center, the FTP server node can determine the load distribution weight of each candidate node based on the current performance and usage status of each node, and identify the candidate data proxy nodes with relatively idle resources.

[0049] S103: Determine the target data proxy node based on the load distribution weights and the connection count information of each candidate data proxy node.

[0050] Specifically, after determining the load distribution weight of each node, it is also necessary to combine the connection number information of each node to avoid a surge in transmission tasks received by an idle node, balance the load of each transmission node, and accurately determine the target data proxy node suitable for transmitting the target data transmission task.

[0051] S104: Send the target data transmission task to the target data proxy node.

[0052] Specifically, the FTP server node sends the target data transfer task to the appropriate target data proxy node, so that the target data proxy node can send the data of the task to the HDFS distributed file system to complete the data transfer.

[0053] In this disclosure, a target data transfer task is received from an FTP client; based on node performance and usage information of multiple candidate data proxy nodes obtained from a registry center, the load allocation weight of each candidate data proxy node is determined; based on the load allocation weight and the connection count information of each candidate data proxy node, a target data proxy node is determined; and the target data transfer task is sent to the target data proxy node. By using the node performance and usage information of the proxy node, the node load utilization rate is determined, thereby determining the load allocation weight of each candidate data proxy node. This allows for accurate assessment of whether each node has spare resources, and by combining connection count information, it avoids distributing all tasks to previously resource-spared nodes, preventing sudden increases in node connection counts from reducing task processing efficiency, and ensuring that the target data transfer task is sent to a suitable data proxy node for execution. Therefore, data transfer efficiency can be improved.

[0054] In one possible implementation, an exemplary method for determining the load allocation weight of each candidate data proxy node based on node performance information and node usage information obtained from a registry center includes: For any candidate data proxy node, determine the first node parameter of the candidate data proxy node based on the node performance information.

[0055] For example, node performance information includes at least one of the following: CPU processing speed, memory size, network bandwidth, and disk read / write speed.

[0056] Specifically, the first node parameter is used to characterize the maximum processing capacity of the candidate data proxy nodes. The node performance information of each node is obtained and sent to the registry center for registration. After the FTP server node and data proxy node services start, at regular intervals, information such as node IP, CPU processing speed, memory size, network bandwidth, disk read / write speed, CPU utilization, memory utilization, network utilization, and disk read / write utilization is registered with the registry center ZooKeeper for subsequent use in determining the load allocation weights of each candidate data proxy node.

[0057] For example, for any candidate data proxy node, the first node parameters of the candidate data proxy node are determined based on node performance information, including: The product of CPU processing speed, memory size, network bandwidth, disk read / write speed and the corresponding node performance information is used to obtain multiple performance parameters; the sum of multiple performance parameters is calculated to obtain the first node parameter.

[0058] Specifically, since the node performance of each candidate data proxy node is not necessarily the same, the maximum processing capacity of the nodes is also different.

[0059] An example formula for determining the parameters of the first node is as follows:

[0060] in, For the first node parameters, For CPU processing speed, For memory size, For network bandwidth, For disk read / write speed, As a weight for CPU processing speed, Weights for memory size Weights for network bandwidth Weights for disk read / write speeds.

[0061] In this embodiment, the sum of the performance information of each node is configured to 1, and each weight can be customized.

[0062] Based on node usage information, the difference between node warm-up time and service runtime, the second node parameters of the candidate data proxy node are determined.

[0063] Specifically, the second node parameter characterizes the actual processing capacity of the candidate data proxy node. For example, node usage information includes at least one of the following: CPU utilization, memory utilization, network utilization, and disk read / write utilization. Thread tasks are executed at regular intervals to synchronize node usage information, thereby eliminating nodes that do not meet the threshold conditions and avoiding waste of computing resources. If any one of the candidate data proxy node's CPU utilization, memory utilization, network utilization, disk read / write utilization, or number of connections exceeds a preset configuration value, then that node will not participate in load balancing, i.e., it will not be allocated traffic.

[0064] For example, based on node usage information, the difference between node warm-up time and service runtime, the second node parameters of the candidate data proxy node are determined, including: When the service runtime difference is greater than or equal to the node warm-up time, obtain the product of CPU utilization, memory utilization, network utilization, disk read / write utilization, and the corresponding node usage information to obtain multiple usage parameters; obtain the sum of multiple usage parameters to obtain the second node parameter; when the service runtime difference is less than the node warm-up time, obtain the product of CPU utilization, memory utilization, network utilization, disk read / write utilization, and the corresponding node usage information to obtain multiple usage parameters; calculate the second node parameter based on the sum of multiple usage parameters, the node warm-up time, and the service runtime difference.

[0065] Specifically, the service runtime difference is the difference between the current time and the service startup time of the candidate data proxy node, which is the actual trial service runtime of the data proxy node; the node warm-up time is the time taken for the candidate data proxy node to run to the first state, which is the time required for the data proxy node to reach its optimal state.

[0066] To ensure the FTP server node and data proxy node services gradually reach their optimal state, they need to be warmed up. After service startup, they run at low power for a period of time, allowing the node's weight to gradually reach its maximum. In this embodiment, the node warm-up time is set to ten minutes by default. The current state of the data proxy node is determined based on its current service running time, and the calculation method for the second node parameters is determined based on different running states.

[0067] When the data proxy node reaches its optimal operating state, the following is an example formula for determining the parameters of the second node:

[0068] in, For the second node parameters, For CPU utilization, For memory usage, For network usage, For disk read / write usage, As a weight for CPU utilization, As a weight for memory usage, As a weight for network usage, Weights for disk read / write usage Preheating time for nodes Service runtime difference. In this embodiment, the sum of the weights of the information used by each node is set to 1, and each weight can be customized.

[0069] When the data proxy node has not yet reached its optimal operating state, an example formula for determining the parameters of the second node is as follows:

[0070] in, For the second node parameters, For CPU utilization, For memory usage, For network usage, For disk read / write usage, As a weight for CPU utilization, As a weight for memory usage, As a weight for network usage, Weights for disk read / write usage Preheating time for nodes Service uptime difference.

[0071] Based on node usage information, determine the load allocation weight influencing factors.

[0072] Specifically, during file transfers, FTP services can experience extremely high network and disk read / write usage, sometimes even saturating network bandwidth, while other metrics such as CPU and memory usage remain relatively low. Therefore, this disclosure employs a weighted product approach to calculate the node load allocation weighting factor.

[0073] For example, based on the node usage information, the load allocation weighting influencing factor is determined, including: The system obtains multiple usage parameters by multiplying CPU utilization, memory utilization, network utilization, and disk read / write utilization with the corresponding node usage information; it then determines the maximum value among CPU utilization, memory utilization, network utilization, and disk read / write utilization; based on the maximum value and multiple usage parameters, it determines the load allocation weighting influencing factor.

[0074] Specifically, an exemplary formula for determining the load allocation weighting influencing factor is as follows:

[0075]

[0076] in, For CPU utilization, For memory usage, For network usage, For disk read / write usage, As a weight for CPU utilization, As a weight for memory usage, As a weight for network usage, Weights for disk read / write usage Assign weighting factors to the load.

[0077] Based on the parameters of the first node, the parameters of the second node, and the load allocation weight influence factor, the load allocation weight of each candidate data proxy node is determined.

[0078] Specifically, in existing technologies, JIT optimization is not considered when determining target data proxy nodes, resulting in the service not reaching its optimal operating state upon startup. Suddenly increasing service traffic to normal levels could lead to numerous request timeouts or even system crashes. Furthermore, in live network use, as business volume increases, new users continuously join the distributed FTP service. User information is stored in configuration files by default; each time a user is added, modified, or deleted, the distributed FTP service needs to be restarted, interrupting user services and reducing the tool's availability and flexibility. Related load balancing algorithms weight nodes based on various metrics. If a load metric reaches 100%, the node's load weight should be close to 1, indicating that the server is extremely busy and cannot be assigned tasks. However, using weighted summation to calculate node load weights fails to accurately determine whether a node has available resources in such extreme load utilization scenarios.

[0079] In this disclosure, an exemplary formula for determining the load allocation weight based on the first node parameters, the second node parameters, and the load allocation weight influence factor is as follows:

[0080] in, Assign weights to the load. Assign weighting factors to the load. For the second node parameters, For the first node parameter.

[0081] In one possible implementation, an exemplary method for determining the target data proxy node based on the load allocation weights and the connection count information of each candidate data proxy node includes: For any first node among the candidate data proxy nodes, when the second node is a node other than the last node among the multiple candidate data proxy nodes, the connection number information of the first node and the product of the load distribution weight of the second node are judged sequentially, and the product of the connection number information of the second node and the load distribution weight of the first node is judged. The first node and / or the second node are updated based on the judgment result. This continues until the second node is the last node among the multiple candidate data proxy nodes, and the current first node is taken as the target data proxy node.

[0082] Specifically, the second node is the node following the first node. Figure 3 A flowchart of the target data proxy node provided in this disclosure, such as Figure 3 As shown.

[0083] The real-time connection count information of each first node is compared with the real-time load distribution weight to calculate the data proxy node with the dynamic weighted minimum connection count, and this node is used as the target data proxy node.

[0084] An example formula for determining the weighted minimum number of connections for a data proxy node is as follows:

[0085] in, This represents the number of connections already established for the first node i. This represents the number of connections already established for the second node j. Assign weights to the load of the second node. Assign weights to the load of the first node.

[0086] Furthermore, a sudden surge in tasks during the heartbeat cycle of a task startup might cause tasks to be directly assigned to the dataproxy node with the lowest weight in the load balancing algorithm, while other dataproxy nodes might not receive any tasks. The concurrency of a dataproxy is achieved through a connection pool with a limited size set in the configuration file. Once the maximum number of threads in the connection pool is exceeded, subsequent tasks will queue, which lengthens the overall task execution time. Since FTP connections are divided into long connections and short connections, and certain commands such as `list` and `pwd` are quick, while file upload and download commands such as `put` and `get` are more time-consuming, when the FTP server assigns a dataproxy and records the connection count information of that dataproxy node, it can store the number of connections and connection types in ZooKeeper based on the received commands. In one possible implementation, the connection count information is positively correlated with the connection type and the number of connections.

[0087] Therefore, an exemplary formula for calculating the number of connections is as follows:

[0088] in, Weights for long-connection counts. Weights for short connections. The number of long connections. This represents the number of short connections. In this embodiment, the sum of the weights for long connections and short connections is 1, and the weight for long connections is greater than the weight for short connections.

[0089] like Figure 3 As shown, when selecting a dataproxy, if the number of connections of the dataproxy is less than the size of the dataproxy thread pool, the dataproxy is selected according to the improved dynamic weighted minimum connection number algorithm; if the number of connections of the dataproxy is greater than the size of the dataproxy thread pool, the dataproxy node with the smallest weighted connection number is selected; if the weighted connection numbers are equal, the dataproxy is selected according to the improved dynamic weighted minimum connection number algorithm. The final target data proxy node is hi.

[0090] In one possible implementation, the method further includes: Change the current working directory before executing FTP type commands on the FTP server node.

[0091] Specifically, as cluster size increases, the limitations of a single NameNode become increasingly apparent in large Hadoop clusters with over 1000 nodes. HDFS's Federation architecture solves the problem of NameNode metadata storage, giving the NameNode near-infinite horizontal scaling capabilities; however, the open-source hdfs-over-ftp does not support this. In related technologies, when a user has files on multiple NameNodes (NSs), and needs to transfer files from one NNS to another cluster, the transfer will fail if the user's files are not in the default FS of the HDFS cluster. When a user specifies the corresponding HDFS ns in the `ftpserver.user.xxx.homedirectory` value of the `users.properties` configuration file for HDFS-over-FTP, for example, `ftpserver.user.xxx.homedirectory=hdfs: / / ns2 / `, and `fs.defaultFS` in the cluster is `hdfs: / / ns1`, HDFS-over-FTP will throw the error `java.lang.IllegalArgumentEcxeption: Wrong FS:hdfs: / / ns2 / , expected:hdfs: / / ns1.`.

[0092] In this disclosure, the command to change the current working directory is used to change the target namespace of FTP type commands. The target namespace is an uninitialized namespace. hdfs-over-ftp supports a multi-namespace (ns, namespace) scheme, enabling users to access files under the corresponding namespace.

[0093] For example, changing the current working directory includes: FTP type commands are intercepted using a global interceptor; when an FTP type command displays the current working directory and the access path includes a namespace, the access path is reset to the target access path, where the target access path corresponds to the target namespace.

[0094] Specifically, by setting a global interceptor to intercept FTP-type commands, and determining whether the access path needs to be reset when the command carries a namespace, this disclosure enhances the functionality of the open-source hdfs-over-ftp tool by providing a hot-reloading method and supporting multiple HDFS namespaces. When user information changes, such as adding, deleting, or updating user information, the application does not need to be restarted, improving service availability and flexibility, and ensuring the continuity and security of user services. Users can configure corresponding namespaces to read and write files within a specific namespace, solving the problem that user files cannot be transferred when they are not in the default defaultFS of the HDFS cluster.

[0095] Furthermore, an exemplary method for changing the current working directory is as follows: Specify the corresponding HDFS ns in the `ftpserver.user.xxx.homedirectory` value of the user configuration file, for example, `ftpserver.user.xxx.homedirectory=hdfs: / / ns1 / `. When a user logs in for authentication, the `execute` method in the `PASS` class of the FTP Server package is triggered. This method calls the `createFileSystemView` method of the `FileSystemFactory` interface. The `HdfsFileSystemFactory` class inherits from the `FileSystemFactory` interface and implements the `createFileSystemView` method. The `createFileSystemView` implementation creates an `HdfsFileSystemView` object, whose constructor sets the current user's root directory and current directory. The `homeDirectory` parameter in the constructor is processed, setting it to `fs.defaultFS` for that user, and setting the path after `fs.defaultFS` to the user's root directory and current directory.

[0096] The main code is as follows:

[0097] When users dynamically set ns during use, they only need to execute the cd command (to change the current working directory) before executing other operation commands. The command should include the corresponding ns, for example, cd hdfs: / / ns2 / tmp / .

[0098] To achieve this function, the specific implementation process is as follows: Add a command type to the global interceptor. If it's CWD and the path contains ns, reset the user's corresponding fs.defaultFS, setting the path after fs.defaultFS to the user's root directory and current directory. Create a new HdfsFileSystemView object, and set ATTRIBUTE_FILE_SYSTEM to the latest HdfsFileSystemView object in FtpSession. First, override the CWD command, inheriting the Command interface of ftpserver-core, and implement the execute method. Extract the argument from the FtpRequest to obtain the path after ns as the actual path to switch to. Obtain the FileSystemView object from FtpIoSession, call its changeWorkingDirectory method to switch to the directory the user wants to enter, and then call the getWorkingDirectory method of the FileSystemView object again to get file information. Add the custom CWD command using addCommand in CommandFactoryFactory when the ftpserver starts.

[0099] The custom CWD command is added as follows: FtpServerFactory factory = new FtpServerFactory(); CommandFactoryFactory factoryFactory = new CommandFactoryFactory(); factoryFactory.addCommand("CWD",new CWD()); factory.setCommandFactory(factoryFactory.createCommandFactory()); At this point, user operations such as ls, put, and get will operate on the relevant files or directories under the user-defined ns directory.

[0100] In one possible implementation, the method further includes: Monitor the node thread pool; when there is a configuration file update indication in the node thread pool, perform a hot reload update of the configuration file.

[0101] Specifically, HDFS-over-FTP's user authentication method loads a configuration file containing user information when the application starts and performs authentication by comparison during user login. However, once the service is running, any modification to user information requires a service restart. Since user login information may be dynamically added, modified, and deleted when providing user services, it's impossible to read all user data when the HDFS-over-FTP service starts, and online applications cannot be frequently restarted. Furthermore, if a user is performing operations while connected to the FTP server and their configuration information changes (e.g., deleting user information or changing passwords), all user operations must be disabled for security reasons. Therefore, hot-loading of the HDFS-over-FTP application's configuration file is necessary. When user information needs to be changed, it should be written to the configuration file, and the application should be able to load the new configuration information in real time, allowing users to perform relevant operations based on the updated configuration.

[0102] The WatchService interface utilizes the operating system's built-in file monitor to monitor directories and files. When the monitored object changes, a signal is sent, allowing for efficient detection of changes. First, a new WatchService monitor is created based on the operating system. The directory or file containing the configuration file to be monitored is selected, and events to be monitored, such as creation, deletion, and editing, are subscribed to. The monitor is registered with the monitored location. A monitoring thread is started to handle change logic, retrieving changes from the monitoring pool in an infinite loop, waiting if no changes are found. All pending modification events are retrieved and iterated through. If the file name in the modification event matches the specified configuration file, a hot update operation is performed on the configuration file. The monitor is reset after each change. A monitor hook is registered so that the monitor can be shut down when the JVM shuts down. At this point, we only need to call the method in the implementation class that listens for user configuration file modifications after the FTP Server service in the distributed HDFS-over-FTP starts.

[0103] In this embodiment, changes to the configuration file in the node thread pool are monitored in real time. When there is a modification / update instruction for the file, the business implementation method is rewritten so that the running service can load the latest modified configuration. This allows newly added users to perform related operations, while deleted users cannot continue to perform operations. For changed users, it is determined whether they can perform related operations based on the specific changes. The application service is not restarted, thus achieving hot reloading of the service.

[0104] In one possible implementation, an exemplary method for performing a hot reload update of the configuration file when a configuration file update indication exists in the node thread pool includes: Intercept the configuration file update instruction using a global interceptor; determine if the global variable configuration field in the configuration file update instruction is incomplete; if so, determine if the configuration file exists in the user information of the FTP server node after the update for the executing user; if so, determine if the configuration information of the executing user has been updated; if so, execute the configuration file update instruction; based on the configuration information of the executing user, update the FTP session information of the executing user and update the global variable configuration field to complete.

[0105] Specifically, a global interceptor is defined to process the FTP commands before execution. A global variable is defined to determine whether `isConfigured` (whether configuration is complete) is loaded. The `volatile` keyword is added to `isConfigured` to ensure the visibility and atomicity of shared variables during concurrent access by multiple threads. When the configuration file changes, the `refresh` method in the `FTPPropertiesUserManager` class is called to reload the properties file containing user information, and `isConfigured` is set to false.

[0106] When `isConfigured` is false, it indicates whether the global variable configuration field in the configuration file update indicator is incomplete. The username is retrieved from the `FtpSession` object, and the system checks if the executing user is included in the user information loaded in real-time after the configuration file change. If the executing user is not present, the user's command will result in an error. The password is retrieved from the `FtpSession` object, decrypted, and then the password loaded in real-time after the configuration file change is checked against the username. If they do not match, the user's command will result in an error.

[0107] If the user being executed exists, check if any of the user's other configuration information has been changed. If so, update the FtpSession information and set isConfigured to true.

[0108] The configuration change information includes at least one of the following: user home directory, maximum user idle time, maximum number of user logins, maximum number of user login IPs, user permission status, and user upload and download speeds. Further: 1) When a user's home directory changes, a new HdfsFileSystemView object needs to be created based on the latest user configuration information, and the FtpSession needs to set ATTRIBUTE_FILE_SYSTEM to the latest HdfsFileSystemView object. This way, when the user previously performed operations such as pwd, cd, and ls in the home directory, they will get the correct results based on the newly configured home directory.

[0109] 2) When a user's maximum idle time changes, the maximum idle time of FtpSession needs to be set to the latest value, and the user's connection disconnection time will use this value.

[0110] 3) If the user's permission to use the service changes, and the user's command is no longer allowed, an error will occur.

[0111] 4) When a user's upload and download speeds change, retrieve the user's latest information and set the ATTRIBUTE_USER property of the FtpSession to the latest user object. When the user performs upload and download operations again, the speed will be limited according to the latest value.

[0112] 5) If the maximum number of user logins or the maximum number of user login IPs changes, and the modified value is less than the original value, subsequent user operations will result in errors. To view the real-time login count for each user, define a map object with the username as the key and user login information as the value, including the total number of logins, login addresses, and their counts. The login count is incremented by 1 each time a user successfully logs in, and decremented by 1 each time the user logs out.

[0113] In one possible implementation, the method further includes: When a target data transmission task is received from the target data proxy node, the target data proxy node is re-determined.

[0114] Specifically, if the target data proxy node reassigns the target data transmission task multiple times but still fails to meet the conditions, the task needs to be returned to the FTP server node so that the FTP server node can reassign the task.

[0115] Figure 4 This is another flowchart illustrating a data transmission method provided in this disclosure. Figure 4 As shown, this method is applied to the target data proxy node and includes: S401: Receive target data transfer task from FTP server node.

[0116] Specifically, in this embodiment, the node load utilization is determined by analyzing the node performance and usage information of the proxy nodes, thereby determining the load allocation weight for each candidate data proxy node. This allows for accurate assessment of whether each node has spare resources. Combined with connection count information, it avoids distributing all tasks to previously resource-spared nodes, preventing sudden increases in node connections from reducing task processing efficiency, and ensuring that target data transmission tasks are sent to appropriate data proxy nodes for execution.

[0117] In other words, the target data proxy node is determined based on the load distribution weights and the connection count information of each candidate data proxy node. This target data proxy node receives the target data transfer task from the FTP server node.

[0118] S402: Send the target data transfer task to the HDFS distributed file system.

[0119] Specifically, in order to achieve data transmission, the target data proxy node needs to send the target data transmission task to the HDFS distributed file system to store the data in the task.

[0120] In one possible implementation, the method further includes: Determine if the node weight update object of the target data proxy node is locked; if locked, stop updating node performance information and node usage information; if the lock is released, update node performance information and node usage information.

[0121] Specifically, when updating the load balancing weight of a node, it's necessary to determine whether the node is currently in use. In this embodiment, a node weight update object is set to determine whether the node's load balancing weight can be updated. If it's currently in use, it cannot be updated immediately and must wait for the node to be released before updating. For example, it checks whether the node weight update object (map object) is locked. This is because after command distribution, the load balancing algorithm needs to be called to retrieve information from this map object, preventing exceptions caused by multiple threads updating and retrieving the same object. If the map object is not locked, its content is updated; the updated content includes at least one of the following: CPU utilization, memory utilization, network utilization, connection count information, disk read / write utilization, service startup time, node status, etc. The lock is released after the update is complete. Otherwise, it waits for the lock to be released.

[0122] In one possible implementation, the method further includes: When the number of tasks in the thread pool of the target data proxy node is determined to be greater than the preset threshold multiple times, the target data transfer task is returned to the FTP server node.

[0123] Specifically, since dataproxy updates the waiting queue size in the registry center with each heartbeat, if a large number of tasks are started within a heartbeat cycle and the connection count has not been updated within the current heartbeat cycle, after dataproxy receives the task assigned by the FTP server, if the number of tasks in the dataproxy thread pool is greater than the set value, then every certain period of time (e.g., 1 second), it will check again whether the number of tasks in the dataproxy thread pool is greater than the set value. If it is still greater than the set value, after a certain number of loops, if the condition is still not met, the task will be returned to the FTP server so that the FTP server can reassign the task.

[0124] Figure 5 This is a schematic diagram of a data transmission device provided in this disclosure. Figure 5 As shown, the device 500 is applied to an FTP server node and includes: a first receiving module 510, a first determining module 520, a second determining module 530, and a first sending module 540.

[0125] The first receiving module 510 is used to receive the target data transmission task from the FTP client. The first determining module 520 is used to determine the load allocation weight of each candidate data proxy node based on the node performance information and node usage information of multiple candidate data proxy nodes obtained from the registration center. The second determining module 530 is used to determine the target data proxy node based on the load allocation weights and the connection number information of each candidate data proxy node. The first sending module 540 is used to send the target data transmission task to the target data proxy node.

[0126] Optionally, the first determining module includes: The first determining submodule is used to determine, for any candidate data proxy node, a first node parameter of the candidate data proxy node based on the node performance information, wherein the first node parameter is used to characterize the maximum processing capacity of the candidate data proxy node. The second determining submodule is used to determine the second node parameters of the candidate data proxy node based on the node usage information, the difference between the node warm-up time and the service running time. The second node parameters characterize the actual processing capability of the candidate data proxy node. The third determining submodule is used to determine the load allocation weight influencing factor based on the node usage information; The fourth determination submodule is used to determine the load allocation weight of each candidate data proxy node based on the first node parameters, the second node parameters, and the load allocation weight influence factor.

[0127] Optionally, the node performance information includes at least one of the following: CPU processing speed, memory size, network bandwidth, and disk read / write speed; The node usage information includes at least one of the following: CPU usage, memory usage, network usage, and disk read / write usage.

[0128] Optionally, the first determining submodule is configured to: Multiple performance parameters are obtained by multiplying the CPU processing speed, memory size, network bandwidth, disk read / write speed, and the corresponding node performance information. The sum of the various performance parameters is calculated to obtain the first node parameter.

[0129] Optionally, the second determining submodule is used for: When the service runtime difference is greater than or equal to the node warm-up time, the product of the CPU utilization, memory utilization, network utilization, disk read / write utilization and the corresponding node usage information is obtained to get multiple usage parameters; the sum of the multiple usage parameters is obtained to get the second node parameter; the service runtime difference is the difference between the current time and the startup time of the candidate data proxy node service; the node warm-up time is the time taken for the candidate data proxy node to run to the first state; When the difference in service runtime is less than the node warm-up time, the product of the CPU utilization, memory utilization, network utilization, disk read / write utilization and the corresponding node usage information is obtained to get multiple usage parameters; based on the sum of the multiple usage parameters, the node warm-up time and the difference in service runtime, the second node parameter is calculated.

[0130] Optionally, the third determining submodule is used for: Multiple usage parameters are obtained by multiplying the CPU utilization, memory utilization, network utilization, disk read / write utilization, and the corresponding node usage information. Determine the maximum value among the CPU utilization, memory utilization, network utilization, and disk read / write utilization; The load allocation weighting influence factor is determined based on the maximum value and multiple usage parameters.

[0131] Optionally, the second determining module includes: The first judgment submodule is used to, for any first node among the candidate data proxy nodes, when the second node is a node other than the last node among the multiple candidate data proxy nodes, sequentially judge the product of the connection number information of the first node and the load distribution weight of the second node, and the product of the connection number information of the second node and the load distribution weight of the first node; the second node is the next node after the first node. The first update submodule is used to update the first node and / or the second node based on the judgment result; The fifth determining submodule is used to determine the target data proxy node until the second node is the last node among the plurality of candidate data proxy nodes.

[0132] Optionally, the connection count information is positively correlated with the connection type and the number of connections.

[0133] Optionally, the device further includes: The change module is used to change the current working directory before the FTP server node executes FTP type commands; the change current working directory command is used to change the target namespace of the FTP type command, and the target namespace is an uninitialized namespace.

[0134] Optionally, the change module includes: The first interception submodule is used to intercept the FTP type command using a global interceptor; The second update submodule is used to reset the access path to the target access path when the FTP type command is to display the current working directory and the access path carries a namespace, wherein the target access path corresponds to the target namespace.

[0135] Optionally, the device further includes: The monitoring module is used to monitor the node thread pool; The update module is used to perform hot reloading of the configuration file when there is a configuration file update indication in the node thread pool.

[0136] Optionally, the update module includes: The second interception submodule is used to intercept the configuration file update instruction using a global interceptor; The second judgment submodule is used to determine whether the global variable configuration field in the configuration file update instruction is incomplete; The third judgment submodule is used to determine, if yes, whether the configuration file is executed by the user in the user information of the FTP server node after the update. The fourth judgment submodule is used to determine whether the configuration information of the executing user has been updated if so, and to execute the configuration file update instruction if so. The third update submodule is used to update the FTP session information of the executing user based on the configuration information of the executing user, and update the global variable configuration field to "completed".

[0137] Optionally, the device further includes: The third determining module is used to re-determine the target data proxy node when it receives the target data transmission task returned by the target data proxy node.

[0138] Figure 6 This is another structural schematic diagram of a data transmission device provided in this disclosure. Figure 6 As shown, the device 600 is applied to a target data proxy node, which is determined based on the load distribution weights and the connection number information of each candidate data proxy node, and includes: a second receiving module 610 and a second sending module 620.

[0139] The second receiving module 610 is used to receive the target data transmission task from the FTP server node. The second sending module 620 is used to send the target data transmission task to the HDFS distributed file system.

[0140] Optionally, the device further includes: The judgment module is used to determine whether the node weight update object of the target data proxy node is locked. The second update module is used to stop updating node performance information and node usage information if the node is locked. The third update module is used to update the node performance information and the node usage information if the lock is released.

[0141] Optionally, the device further includes: The third sending module is used to return the target data transmission task to the FTP server node when it is determined multiple times that the number of tasks in the thread pool of the target data proxy node is greater than a preset threshold.

[0142] This application also provides an electronic device for performing the above-described data transmission method. Please refer to... Figure 7 It illustrates a schematic diagram of an electronic device provided by some embodiments of this application. For example... Figure 7 As shown, the electronic device 70 includes: a processor 700, a memory 701, a bus 702, and a communication interface 703. The processor 700, the communication interface 703, and the memory 701 are connected via the bus 702. The memory 701 stores a computer program that can run on the processor 700. When the processor 700 runs the computer program, it executes the data transmission method provided in any of the foregoing embodiments of this application.

[0143] The memory 701 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between the device network element and at least one other network element is achieved through at least one communication interface 703 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0144] Bus 702 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 701 is used to store programs. After receiving an execution instruction, the processor 700 executes the program. The data transmission method disclosed in any of the foregoing embodiments of this application can be applied to the processor 700, or implemented by the processor 700.

[0145] The processor 700 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 700 or by instructions in software form. The processor 700 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 701. Processor 700 reads the information in memory 701 and, in conjunction with its hardware, completes the steps of the above method.

[0146] The electronic device and the data transmission method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0147] This application also provides a computer-readable storage medium corresponding to the data transmission method provided in the foregoing embodiments. The computer-readable storage medium shown may be an optical disc, on which a computer program is stored. When the computer program is run by a processor, it executes the data transmission method provided in any of the foregoing embodiments.

[0148] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0149] The computer-readable storage medium provided in the above embodiments of this application and the data transmission method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0150] This application also provides a computer program product 800, such as... Figure 8 As shown. This computer program product carries a computer program 801. The instructions included in the program code can be used to execute the steps of the data transmission method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0151] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0152] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0153] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0154] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0155] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0156] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0157] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0158] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A data transmission method, characterized in that, Applied to an FTP server node, the method includes: Receive target data transfer tasks from FTP clients; Based on the node performance information and node usage information of multiple candidate data proxy nodes obtained from the registration center, the load allocation weight of each candidate data proxy node is determined. The target data proxy node is determined based on the load allocation weights and the connection count information of each candidate data proxy node. The target data transmission task is sent to the target data proxy node.

2. The method according to claim 1, characterized in that, The step of determining the load allocation weight of each candidate data proxy node based on node performance and usage information obtained from the registration center includes: For any candidate data proxy node, based on the node performance information, a first node parameter of the candidate data proxy node is determined, wherein the first node parameter is used to characterize the maximum processing capacity of the candidate data proxy node. Based on the node usage information, the difference between node warm-up time and service running time, the second node parameters of the candidate data proxy node are determined, and the second node parameters characterize the actual processing capability of the candidate data proxy node. Based on the node usage information, determine the load allocation weight influencing factor; Based on the first node parameters, the second node parameters, and the load allocation weight influence factor, the load allocation weight of each candidate data proxy node is determined.

3. The method according to claim 2, characterized in that, The node performance information includes at least one of the following: CPU processing speed, memory size, network bandwidth, and disk read / write speed; The node usage information includes at least one of the following: CPU usage, memory usage, network usage, and disk read / write usage.

4. The method according to claim 3, characterized in that, The step of determining the first node parameters of any candidate data proxy node based on the node performance information includes: Multiple performance parameters are obtained by multiplying the CPU processing speed, memory size, network bandwidth, disk read / write speed with the weighted product of the corresponding node performance information; The sum of the various performance parameters is calculated to obtain the first node parameter.

5. The method according to claim 3, characterized in that, The step of determining the second node parameters of the candidate data proxy node based on the node usage information, the difference between the node warm-up time and the service runtime includes: When the service runtime difference is greater than or equal to the node warm-up time, the weighted product of the CPU utilization, memory utilization, network utilization, disk read / write utilization, and the corresponding node usage information is obtained to get multiple usage parameters; the sum of the multiple usage parameters is obtained to get the second node parameter; the service runtime difference is the difference between the current time and the startup time of the candidate data proxy node service; the node warm-up time is the time taken for the candidate data proxy node to run to the first state; When the difference in service runtime is less than the node warm-up time, the product of the CPU utilization, memory utilization, network utilization, disk read / write utilization and the corresponding node usage information is obtained to get multiple usage parameters; based on the sum of the multiple usage parameters, the node warm-up time and the difference in service runtime, the second node parameter is calculated.

6. The method according to claim 3, characterized in that, The step of determining the load allocation weight influencing factor based on the node usage information includes: Multiple usage parameters are obtained by multiplying the CPU utilization, memory utilization, network utilization, disk read / write utilization, and the corresponding node usage information. Determine the maximum value among the CPU utilization, memory utilization, network utilization, and disk read / write utilization; The load allocation weighting influence factor is determined based on the maximum value and multiple usage parameters.

7. The method according to claim 1, characterized in that, The step of determining the target data proxy node based on the load allocation weights and the connection number information of each candidate data proxy node includes: For any first node among the candidate data proxy nodes, when the second node is a node other than the last node among the multiple candidate data proxy nodes, the product of the connection number information of the first node and the load distribution weight of the second node is determined sequentially, and the product of the connection number information of the second node and the load distribution weight of the first node is determined; the second node is the next node after the first node. Update the first node and / or the second node based on the judgment result; Until the second node is the last of the multiple candidate data proxy nodes, the current first node is taken as the target data proxy node.

8. The method according to claim 1, characterized in that, The method further includes: Before the FTP server node executes FTP type commands, the current working directory is changed; the command to change the current working directory is used to change the target namespace of the FTP type commands, and the target namespace is an uninitialized namespace.

9. A data transmission method, characterized in that, The method, applied to a target data proxy node determined based on load balancing weights and connection count information of each candidate data proxy node, includes: Receive the target data transfer task from the FTP server node; The target data transfer task is sent to the HDFS distributed file system.

10. A data transmission device, characterized in that, The device, applied to an FTP server node, includes: The first receiving module is used to receive the target data transfer task from the FTP client. The first determining module is used to determine the load allocation weight of each candidate data proxy node based on the node performance information and node usage information of multiple candidate data proxy nodes obtained from the registration center. The second determining module is used to determine the target data proxy node based on the load allocation weights and the connection number information of each candidate data proxy node. The first sending module is used to send the target data transmission task to the target data proxy node.

Images