Data processing method, device, electronic device and computer storage medium
By managing multiple data transmission processes on the server side and load balancing is carried out based on the connection load information, the problem of limited resources of FTP servers is solved, and the efficiency and flexibility of large-scale data transmission is improved.
Patent Information
- Application Number
- CN202010451615.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-05-25
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2040-05-25
AI Technical Summary
In the prior art, the resources of the FTP server are limited and difficult to adapt to large-scale data transmission scenarios, resulting in low transmission rates and high complexity when expanding data transmission capabilities and realizing connection load balancing.
By determining and controlling multiple data transmission processes on the server side, selecting appropriate processes based on the connection load information to perform data interaction between the FTP client and DFS, load balancing is achieved, and multiple data transmission processes are allowed to run on different FTP servers. .
It improves the transmission rate of large-scale data transmission, simplifies the data transmission process, reduces the complexity of configuration management, realizes load balancing, and is suitable for large-scale data transmission scenarios.
Smart Images

Figure CN113722084B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to data transmission technology, and in particular to a data processing method, device, electronic equipment and computer storage medium, which can be applied to data reading and writing of a distributed file system (DFS). Background Art
[0002] In related technologies, with the continuous development of cloud computing and big data technologies, more and more enterprises need to use DFS to store massive amounts of data, and use some computing frameworks (such as tez, mapreduce, spark, flink) to analyze, process and process these data; for example, DFS can be Hadoop Distributed File System (HDFS), Ceph distributed file system, GFS (Google File System) distributed file system or Lustre file system; the computing framework can be Tez, Mapreduce, Spark, Flink and other computing frameworks; some big data tools (such as Flume tool or Kafka tool) can quickly upload the collected data to DFS, but the codes of these tools are complex, and the configuration, deployment, use and operation and maintenance of the tools require professional big data staff, that is, the deployment and use of these tools are relatively complex; using the File Transfer Protocol (FTP) service to transfer data to DFS is a simple and efficient method.
[0003] For example, Figure 1 FIG. 1 is a schematic diagram of a structure in which FTP service is used to transfer data to DFS in a related technology. Figure 1 As shown, in the related technology, the HDFS-OVER-FTP server is an FTP server based on the apache ftp open source that works on HDFS. The HDFS-OVER-FTP is encapsulated with the HDFS client, so that the HDFS-OVER-FTP server can read and write HDFS files, thereby allowing users or businesses to use any FTP client and FTP commands to access HDFS; at the same time, the HDFS-OVER-FTP server can also perform secure access based on the kerberos protocol and support all HDFS permission settings; in summary, you only need to use the HDFS-OVER-FTP server to realize data transmission between the FTP client and HDFS. The use of the HDFS-OVER-FTP server is the same as that of the linux vsftp server, and it is simple and convenient to use.
[0004] However, when using FTP service to transfer data to DFS, due to the limited resources of FTP server, it is often difficult to adapt to the scenario of large-scale data transmission. Summary of the invention
[0005] The present invention is intended to provide a technical solution for data processing, which can solve the problem that the FTP server is difficult to adapt to the scenario of large-scale data transmission due to limited resources.
[0006] The present invention provides a data processing method, the method comprising:
[0007] After receiving the data transmission instruction sent by the FTP client, determining each registered data transmission process, each data transmission process represents a process for realizing data interaction between the FTP client and the DFS, and the number of the registered data transmission processes is greater than 1;
[0008] Control at least one of the registered data transmission processes to perform data interaction between the FTP client and the DFS.
[0009] Optionally, the method further comprises:
[0010] Storing connection load information of each registered data transmission service process;
[0011] The controlling at least one of the registered data transmission processes to perform data interaction between the FTP client and the DFS includes:
[0012] According to the connection load information of each registered data transmission service process, a data transmission process is selected from each registered data transmission process, and data interaction between the FTP client and the DFS is performed according to the selected data transmission process.
[0013] Optionally, selecting a data transmission process from the registered data transmission processes according to the connection load information of the registered data transmission service processes includes:
[0014] Among the registered data transmission processes, the data transmission process with the smallest connection load is selected, or the data transmission process with a connection load less than a set threshold is selected.
[0015] Optionally, the method further comprises:
[0016] When the connection load of at least one of the registered data transmission processes changes, the stored connection load information of the registered data transmission service processes is updated according to the changed connection load of the data transmission process.
[0017] Optionally, determining each registered data transmission process includes:
[0018] According to the registration information of the data transmission process stored in the server, each of the registered data transmission processes is determined.
[0019] Optionally, the method further comprises:
[0020] When at least one of the registered data transmission processes is deleted, the registration information of the at least one data transmission process stored in the server is deleted.
[0021] Optionally, the method further comprises:
[0022] When a new data transmission process appears, the registration information of the new data transmission process is stored in the server.
[0023] The present invention also provides a data processing device, characterized in that it is applied to a server, and the device comprises: a first processing module and a second processing module, wherein:
[0024] A first processing module, configured to determine each registered data transmission process after receiving a data transmission instruction sent by the FTP client, each data transmission process representing a process for realizing data interaction between the FTP client and the DFS, and the number of the registered data transmission processes is greater than 1;
[0025] The second processing module is used to control at least one of the registered data transmission processes to perform data interaction between the FTP client and the DFS.
[0026] Optionally, the second processing module is further used to store the connection load information of each registered data transmission service process;
[0027] The second processing module is used to control at least one of the registered data transmission processes to perform data interaction between the FTP client and the DFS, including:
[0028] According to the connection load information of each registered data transmission service process, a data transmission process is selected from each registered data transmission process, and data interaction between the FTP client and the DFS is performed according to the selected data transmission process.
[0029] Optionally, the second processing module is used to select a data transmission process from each registered data transmission process according to the connection load information of each registered data transmission service process, including:
[0030] Among the registered data transmission processes, the data transmission process with the smallest connection load is selected, or the data transmission process with a connection load less than a set threshold is selected.
[0031] Optionally, the second processing module is also used to update the stored connection load information of each registered data transmission service process according to the changed connection load of at least one of the registered data transmission processes when the connection load of the data transmission process changes.
[0032] Optionally, the first processing module is used to determine each registered data transmission process, including:
[0033] According to the registration information of the data transmission process stored in the server, each of the registered data transmission processes is determined.
[0034] Optionally, the first processing module is further configured to delete the registration information of the at least one data transmission process stored on the server side when at least one data transmission process among the registered data transmission processes is deleted.
[0035] Optionally, the first processing module is further configured to store registration information of a newly added data transmission process to the server side when a newly added data transmission process appears.
[0036] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any one of the above-mentioned data processing methods when executing the program.
[0037] The present invention also provides a computer storage medium on which a computer program is stored. When the computer program is executed by a processor of a terminal, any of the above data processing methods is implemented.
[0038] In the virtual machine network traffic determination method, device, terminal and computer storage medium proposed by the present invention, after receiving the data transmission instruction sent by the FTP client, each registered data transmission process is determined, each data transmission process represents a process for realizing data interaction between the FTP client and the DFS, and the number of the registered data transmission processes is greater than 1; at least one of the registered data transmission processes is controlled to perform data interaction between the FTP client and the distributed file system DFS. In this way, the server side can realize data interaction between the FTP client and the DFS through multiple data transmission processes, which is conducive to solving the problem of limited resources of a single FTP server in the related technology and can be applied to the scenario of large-scale data transmission.
[0039] Furthermore, since the registration information of multiple data transmission processes can be obtained, it is convenient to uniformly control the data transmission process of multiple data transmission processes on the server side; since the selected data transmission process is obtained based on the connection load information of each data transmission service process, it is convenient to achieve load balancing between the various data transmission processes on the server side.
[0040] Furthermore, based on the data processing method of the embodiment of the present invention, the control connection and the data connection responsible for data transmission can be separated, allowing multiple data transmission processes to run on multiple different FTP servers.
[0041] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] The accompanying drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present invention and, together with the specification, are used to explain the technical solutions of the present invention.
[0043] Figure 1 A schematic diagram of a structure for transferring data to DFS using FTP service in the related art;
[0044] Figure 2 is a flow chart of the data processing method of the present invention;
[0045] Figure 3 A schematic diagram of an architecture for implementing data interaction between an FTP client and DFS in the present invention;
[0046] Figure 4 A schematic diagram of the structure of a data processing device of the present invention;
[0047] Figure 5 It is a schematic structural diagram of the electronic device of the present invention. DETAILED DESCRIPTION
[0048] In the study of related technologies, the inventors found that: during the actual data transmission process of the HDFS-OVER-FTP server, when the stable transmission state is reached, the transmission rate of a single server node is 300-500MB / s. Due to the limitations of the machine disk input and output (IO) and network IO, the transmission rate cannot be higher, making the method of using FTP service to transfer data to DFS unable to cope with the scenario of large-scale data transmission; thus, in order to be applied to the scenario of large-scale data transmission, multiple HDFS-OVER-FTP services are required to achieve a higher data transmission rate in parallel, and horizontal expansion of multiple HDFS-OVER-FTP servers requires setting the same data transmission configuration parameters for each server separately, which is not convenient for configuration management of the HDFS-OVER-FTP server; in a specific example, for each HDFS-OVER-FTP server, two transmission control protocols (Transmission Control Protocols) need to be established between the FTP client and the HDFS-OVER-FTP server. The first connection is a FTP client-side control protocol (TCP) connection, and the second connection is a data connection. The control connection is used for command interaction between the FTP client and the HDFS-OVER-FTP server, and the data connection is used for data transmission between the FTP client and the HDFS-OVER-FTP server. The control connection and the data connection are implemented through the same process set by the HDFS-OVER-FTP server.
[0049] In addition, the inventors also found that: in order to be applied to the scenario of large-scale data transmission, the FTP client needs to be connected to each HDFS-OVER-FTP server in a dispersed manner, and then the FTP client needs to achieve load balancing of the connection; when adding or reducing HDFS-OVER-FTP servers, the FTP client needs to be notified to modify the corresponding addressing range, and the data transmission load balancing needs to be achieved by modifying the FTP client code development method, which is relatively complicated to implement; in summary, in the scenario of large-scale data transmission, the above-mentioned method of using FTP service to transfer data to DFS is not easy to implement in terms of expanding data transmission capabilities and achieving connection load balancing.
[0050] In view of the above technical problems, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the embodiments provided herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the embodiments provided below are partial embodiments for implementing the present invention, rather than providing all embodiments for implementing the present invention. In the absence of conflict, the technical solutions recorded in the embodiments of the present invention can be implemented in any combination.
[0051] It should be noted that, in the embodiments of the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a method or apparatus including a series of elements includes not only the elements explicitly recorded, but also includes other elements not explicitly listed, or also includes elements inherent to the implementation of the method or apparatus. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other related elements (such as steps in a method or units in a device, for example, a unit may be a part of a circuit, a part of a processor, a part of a program or software, etc.) in the method or apparatus including the element.
[0052] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.
[0053] For example, the data processing method provided by the embodiment of the present invention includes a series of steps, but the data processing method provided by the embodiment of the present invention is not limited to the recorded steps. Similarly, the data processing determination device provided by the embodiment of the present invention includes a series of modules, but the data processing device provided by the embodiment of the present invention is not limited to including the modules explicitly recorded, and may also include modules required to obtain relevant information or perform processing based on information.
[0054] The embodiments of the present invention may be implemented based on electronic devices, where the electronic devices may be thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics products, network personal computers, small computer systems, and the like.
[0055] Electronic devices may be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. In general, program modules may include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems may be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules may be located on local or remote computing system storage media including storage devices.
[0056] Based on the application scenarios described above, the following embodiments are proposed.
[0057] The embodiment of the present invention proposes a data processing method, which can be applied to a server side, where the server side may include a control node, and the server side may be set with at least two data transmission processes, wherein the data transmission process is used to realize data transmission between an FTP client and a DFS; the control node may form a communication connection with the FTP client, and may control the data transmission process to realize data transmission between the FTP client and the DFS.
[0058] Figure 2 is a flow chart of the data processing method of the present invention, such as Figure 2 As shown, the process may include:
[0059] Step 201: after receiving a data transmission instruction sent by an FTP client, determine each registered data transmission process, each data transmission process represents a process for realizing data interaction between the FTP client and the DFS, and the number of the registered data transmission processes is greater than one.
[0060] In actual applications, the control node at the server end can exchange data with the FTP client and can receive the data transmission instruction sent by the FTP client; as an implementation method, the FTP client can first send account authentication information to the control node, and when the account authentication is passed, it can send the data transmission instruction to the control node. In the embodiment of the present invention, the data transmission instruction can be a passive mode (PASV) control command or other types of data transmission instructions, which are not limited here.
[0061] In an embodiment of the present invention, the server side can pre-register the data transmission process that has been set up and store the registration information of the data transmission process; after receiving the above-mentioned data transmission instruction, the control node can realize data interaction between the FTP client and the DFS according to each registered data transmission process; in actual application, the server side is provided with a first node, and the first node can store the registration information of the data transmission process that has been set up by the server side.
[0062] In one implementation, the registered data transmission processes may be determined according to the registration information of the data transmission processes stored on the server.
[0063] In one implementation, when at least one of the registered data transmission processes is deleted, the registration information of the at least one data transmission process stored on the server side can be deleted; when a new data transmission process appears on the server side, the registration information of the new data transmission process can be stored on the server side. It can be seen that the embodiment of the present invention can dynamically update the stored registration information of the data transmission process according to the data transmission process actually existing on the server side, without the need for the FTP client to modify the corresponding addressing range when the data transmission process changes, so that it is convenient to realize data interaction between the FTP client and the DFS according to the actually existing data transmission process.
[0064] Step 202: Control at least one of the registered data transmission processes to perform data interaction between the FTP client and the DFS.
[0065] In practical applications, this step can be implemented by the above-mentioned control node; in a specific example, the control node can obtain the port information of the above-mentioned at least one data transmission process, and send the port information of the above-mentioned at least one data transmission process to the FTP client; the FTP client can establish the above-mentioned at least one data transmission process according to the received port information, thereby realizing data interaction with the DFS through the above-mentioned at least one data transmission process; optionally, the FTP client can perform read operations and / or write operations on the data between the DFS.
[0066] In practical applications, steps 201 to 202 can be implemented based on at least one processor on the server side, and the processor can be at least one of an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a CPU, a controller, a microcontroller, and a microprocessor.
[0067] It can be seen that in the embodiment of the present invention, the server side can realize data interaction between the FTP client and the DFS through multiple data transmission processes, which is conducive to solving the problem of limited resources of a single FTP server in the related technology and can be applicable to the scenario of large-scale data transmission; further, since the registration information of multiple data transmission processes can be obtained, it is convenient to uniformly control the data transmission process of multiple data transmission processes on the server side; based on the data processing method of the embodiment of the present invention, the control connection and data connection responsible for data transmission can be separated, allowing multiple data transmission processes to run on multiple different FTP servers.
[0068] In one implementation, the connection load information of each registered data transmission service process may be stored; here, the connection load information indicates the connection load of the data transmission that the data transmission process is currently responsible for, and the connection load here indicates the disk IO, network IO and other resources consumed by the data transmission process.
[0069] In practical applications, the second node may store the connection load information of each registered data transmission service process; in the embodiment of the present invention, the first node and the second node may be implemented by different electronic devices or by the same electronic device.
[0070] Optionally, a data transmission process may be selected from the registered data transmission processes according to the connection load information of the registered data transmission service processes, and data interaction between the FTP client and the DFS may be performed according to the selected data transmission process.
[0071] It can be seen that since the selected data transmission process is obtained based on the connection load information of each data transmission service process, it is convenient to achieve load balancing between the various data transmission processes on the server side.
[0072] Optionally, a data transmission process with the smallest connection load may be selected from among the registered data transmission processes, or a data transmission process with a connection load less than a set threshold may be selected; in this way, load balancing among the data transmission processes may be achieved.
[0073] Optionally, when the connection load of at least one of the registered data transmission processes changes, the stored connection load information of the registered data transmission service processes may be updated according to the changed connection load of the data transmission process. In this way, it is convenient to accurately achieve load balancing among the various data transmission service processes according to the accurate connection load information of the various data transmission service processes.
[0074] A specific application embodiment of the present invention is described below in conjunction with the accompanying drawings; in the application embodiment, the server side includes a distributed FTP server and a service registration and status maintenance node.
[0075] Figure 3 FIG. 1 is a schematic diagram of an architecture for realizing data interaction between an FTP client and a DFS in the present invention. Figure 3 As shown, the distributed FTP server may include the control node described above, and the distributed FTP server is also provided with multiple data transmission processes, each of which is encapsulated with a DFS client for accessing DFS, and each of which can provide data transmission services between the FTP client and the DFS. The service registration and status maintenance node can implement the functions of the first node and the second node described above.
[0076] The control node can interact with the FTP client through control commands based on the FTP protocol; the control node can also communicate remotely with the data transmission process, access the information in the service registration and status maintenance node, and give priority to allocating the data transmission process with a smaller connection load to establish a data connection with the FTP client; the control node can also request the corresponding data transmission process to allocate a data service port, and then return the identifier of the requested data service port and the address of the corresponding data transmission process to the FTP client.
[0077] The data transmission process may receive a request from the control node and allocate a corresponding data service port to wait for a connection from the FTP client. Here, the data transmission process may send an identifier of the data service port to the control node;
[0078] The data transmission process can receive a data connection request sent by the FTP client, and establish a data connection with the FTP client according to the received data connection request to realize data transmission between the FTP client and the DFS. For example, when data needs to be read from the DFS, the data sent by the FTP client to the DFS includes the content of the data to be read and / or the address in the DFS, and the data sent by the DFS to the FTP client may include the read data; when data needs to be written to the DFS, the data sent by the FTP client to the DFS may include the data to be written, and the data sent by the DFS to the FTP client may include the response information of the completion of data writing. After the data transmission between the FTP client and the DFS is completed, the data transmission process can send notification information to the control node, so that the control node determines that the data transmission task of the corresponding data transmission process is completed. In the data transmission process, the data transmission process realizes the process of data transmission between the FTP client and the DFS. The data transmission process can also call the access interface of the DFS and is responsible for interacting with the DFS to complete the data reading and writing of the DFS.
[0079] In one example, the data transmission process can use a proxy mechanism to access the distributed file system. At this time, the FTP client's access rights to the distributed file system are determined based on the account information for logging into the FTP client. The account information here may include information such as a user name.
[0080] The data transmission process can also send registration information to the access service registration and status maintenance node when it starts, so that the service registration and status maintenance node receives the registration information of the corresponding data transmission process; the data transmission process can also update the connection load information in the service registration and status maintenance node when starting the data transmission between the FTP client and the DFS, and update the connection load information in the service registration and status maintenance node when ending the data transmission between the FTP client and the DFS.
[0081] The service registration and status maintenance node can realize the functions of the first node and the second node mentioned above.
[0082] DFS can be responsible for data storage.
[0083] based on Figure 3 The data processing method of the embodiment of the present invention may include:
[0084] Step A1: The FTP client disconnects from the control node and can send account authentication information to the control node. After the control node determines that the account authentication is successful, it can send a response message of successful authentication to the FTP client; here, the account authentication information can include a user name and password.
[0085] Step A2: When data transmission between the FTP client and the DFS is required, the FTP client may send a PASV control command to the control node.
[0086] Step A3: After receiving the PASV control command, the control node accesses the information in the service registration and status maintenance node, and sends an open port request to the data transmission process with the smallest connection load.
[0087] Step A4: After receiving the open port request from the control node, the data transmission process opens a data service port to wait for the connection of the FTP client. At the same time, the data transmission process can update the connection load information of the process in the service registration and status maintenance node. The data transmission process can send the identifier of the open data service port to the control node.
[0088] Step A5: After receiving the identifier of the data service port, the control node may return the response information of the PASV control command to the FTP client. Here, the response information of the ASV control command may include the identifier of the open data service port and the address of the corresponding data transmission process.
[0089] Step A6: After receiving the response information of the PASV control command, the FTP client can send a data connection request to the corresponding data transmission process according to the response information to establish a data connection with the data transmission process.
[0090] Step A7: After the FTP client establishes a data connection with the data transmission process, the FTP client and the data transmission process perform data transmission; in specific implementation, the data transmission process can implement the FTP client's data read and write operations on the DFS.
[0091] Step A8: After the data transmission is completed, the FTP client is connected to the data transmission process port data, and the data transmission process updates the connection load information of the process in the service registration and status maintenance node.
[0092] Step A9: The FTP client is connected to the control node.
[0093] It can be seen that the embodiment of the present invention can deploy multiple data transmission processes on the server side. Through the expansion of the data transmission process, it is helpful to solve the resource bottleneck problems such as disk IO and network IO when a single FTP server performs data transmission in the related technology; through the information update in the service registration and status maintenance node, the connection load information of the data transmission process can be updated, without the need for the FTP client to modify the configuration information such as the addressing range.
[0094] In addition, in large-scale data transmission scenarios, there is no need to develop or deploy complex big data services such as Flume and Kafka; there is no need to modify the FTP client code to achieve data transmission load balancing; you only need to deploy Figure 3 The distributed FTP server shown can realize data access to DFS and achieve load balancing between various data transmission processes.
[0095] Based on the data processing method proposed in the above-mentioned embodiment, the embodiment of the present invention further proposes a data processing device, which can be applied to the server side.
[0096] Figure 4 FIG. 1 is a schematic diagram of the structure of a data processing device of the present invention. Figure 4 As shown, the device may include: a first processing module 401 and a second processing module 402, wherein:
[0097] The first processing module 401 is used to determine each registered data transmission process after receiving the data transmission instruction sent by the FTP client, each data transmission process represents a process for realizing data interaction between the FTP client and the DFS, and the number of the registered data transmission processes is greater than 1;
[0098] The second processing module 402 is used to control at least one of the registered data transmission processes to perform data interaction between the FTP client and the DFS.
[0099] Optionally, the second processing module 402 is further configured to store the connection load information of each registered data transmission service process;
[0100] The second processing module 402 is used to control at least one of the registered data transmission processes to perform data interaction between the FTP client and the DFS, including:
[0101] According to the connection load information of each registered data transmission service process, a data transmission process is selected from each registered data transmission process, and data interaction between the FTP client and the DFS is performed according to the selected data transmission process.
[0102] Optionally, the second processing module 402 is configured to select a data transmission process from the registered data transmission processes according to the connection load information of the registered data transmission service processes, including:
[0103] Among the registered data transmission processes, the data transmission process with the smallest connection load is selected, or the data transmission process with a connection load less than a set threshold is selected.
[0104] Optionally, the second processing module 402 is also used to update the stored connection load information of each registered data transmission service process according to the changed connection load of at least one of the registered data transmission processes when the connection load of the data transmission process changes.
[0105] Optionally, the first processing module 401 is used to determine each registered data transmission process, including:
[0106] According to the registration information of the data transmission process stored in the server, each of the registered data transmission processes is determined.
[0107] Optionally, the first processing module 401 is further configured to delete the registration information of the at least one data transmission process stored on the server side when at least one data transmission process among the registered data transmission processes is deleted.
[0108] Optionally, the first processing module 401 is further configured to store registration information of a newly added data transmission process to the server side when a newly added data transmission process appears.
[0109] The first processing module 401 and the second processing module 402 can both be implemented by at least one processor located in the server side, and the processor is at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor.
[0110] In addition, each functional module in this embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or software functional modules.
[0111] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.
[0112] Specifically, a computer program instruction corresponding to a data processing method in this embodiment can be stored on a storage medium such as a CD, a hard disk, or a USB flash drive. When the computer program instruction corresponding to a data processing method in the storage medium is read or executed by a processor of a terminal, any data processing method in the aforementioned embodiments is implemented.
[0113] Based on the same technical concept as the above embodiments, see Figure 5 , which shows an electronic device 50 provided by the present invention, which may include: a memory 51, a processor 52, and a computer program stored in the memory 51 and executable on the processor 52; wherein,
[0114] A memory 51, for storing computer programs and data;
[0115] The processor 52 is used to execute the computer program stored in the memory to implement any one of the data processing methods in the foregoing embodiments.
[0116] In practical applications, the memory 51 may be a volatile memory, such as RAM; or a non-volatile memory, such as ROM, flash memory, hard disk drive (HDD) or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 52.
[0117] The processor 52 may be at least one of an ASIC, a DSP, a DSPD, a PLD, a FPGA, a CPU, a controller, a microcontroller, and a microprocessor.
[0118] In some embodiments, the functions or modules included in the apparatus provided in the embodiments of the present invention can be used to execute the method described in the above method embodiment. The specific implementation can refer to the description of the above method embodiment. For the sake of brevity, it will not be repeated here.
[0119] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other. For the sake of brevity, this article will not repeat them.
[0120] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.
[0121] The features disclosed in the various product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.
[0122] The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.
[0123] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.
[0124] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.
Claims
1. A data processing method, characterized in that: Applied to the server side, the method includes: After receiving a data transmission instruction sent by a file transfer protocol FTP client, determining each registered data transmission process, each data transmission process represents a process for realizing data interaction between the FTP client and a distributed file system DFS, and the number of the registered data transmission processes is greater than 1; According to the connection load information of each registered data transmission service process, a data transmission process is selected from the registered data transmission processes, and data interaction is performed between the FTP client and the DFS according to the selected data transmission process, and the selected data transmission process is requested to allocate a data service port, and the identifier of the requested data service port and the address of the selected data transmission process are returned to the FTP client together.
2. The method according to claim 1, characterized in that The method further comprises: The connection load information of each registered data transmission service process is stored.
3. The method according to claim 2, characterized in that The selecting a data transmission process from each of the registered data transmission service processes according to the connection load information of each of the registered data transmission service processes includes: Among the registered data transmission processes, the data transmission process with the smallest connection load is selected, or the data transmission process with a connection load less than a set threshold is selected.
4. The method according to claim 2, characterized in that: The method further comprises: When the connection load of at least one of the registered data transmission processes changes, the stored connection load information of the registered data transmission service processes is updated according to the changed connection load of the data transmission process.
5. The method according to claim 1, characterized in that The step of determining each registered data transmission process includes: According to the registration information of the data transmission process stored in the server, each of the registered data transmission processes is determined.
6. The method according to claim 5, characterized in that The method further comprises: When at least one of the registered data transmission processes is deleted, the registration information of the at least one data transmission process stored in the server is deleted.
7. The method according to claim 5, characterized in that The method further comprises: When a new data transmission process appears, the registration information of the new data transmission process is stored in the server.
8. A data processing device, characterized in that: Applied to a server, the device comprises: a first processing module and a second processing module, wherein: A first processing module is used to determine each registered data transmission process after receiving a data transmission instruction sent by a file transfer protocol FTP client, each data transmission process represents a process for realizing data interaction between the FTP client and a distributed file system DFS, and the number of the registered data transmission processes is greater than 1; The second processing module is used to select a data transmission process from the registered data transmission service processes according to the connection load information of the registered data transmission service processes, perform data interaction between the FTP client and the DFS according to the selected data transmission process, and request the selected data transmission process to allocate a data service port, and return the identifier of the requested data service port and the address of the selected data transmission process to the FTP client.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.
10. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor of a terminal, the method according to any one of claims 1 to 7 is implemented.
