File transmission method and system of distributed file storage system based on FTP (File Transfer Protocol)

Abstract

The invention provides a file transmission method and system of a distributed file storage system based on FTP, and relates to the technical field of computers.The method comprises the steps that firstly, a target file uploaded by a client side through the FTP is received, the target file is subjected to fragmentation processing, and a plurality of file fragments are generated; then, the multiple file fragments are sent to different distributed storage nodes for storage, and uploading and storage of the files are achieved; and downloading different file fragments from a plurality of distributed storage nodes at the same time by the client, and merging to generate a target file, thereby realizing the downloading of the file. By means of the method, the technical problems that a traditional method is low in file transmission efficiency and low in system stability are solved, and the technical effects of improving the file transmission efficiency and improving the system reliability are achieved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a file transmission method and system of a distributed file storage system based on FTP. Background Art

[0002] With the rapid development of informatization, the demand for file transfer and storage is increasing. Traditional file transfer protocols (such as File Transfer Protocol, FTP) have been widely used in various network environments, but with the advent of the big data era, a single server can hardly meet the needs of efficient storage and fast transmission of massive data. Distributed storage systems can improve data processing and storage capabilities through the collaboration of multiple server nodes. However, achieving stable and efficient file transfer in a distributed environment still faces many challenges, such as load balancing, fault tolerance, data security, etc.

[0003] That is to say, the conventional FTP file transfer protocol used in the prior art for file transfer in a distributed file storage system has the technical problems of low transmission efficiency and low system stability. Summary of the invention

[0004] The object of the present invention is to provide a file transmission method and system for a distributed file storage system based on FTP, so as to alleviate the technical problems of low transmission efficiency and low system stability existing in the prior art.

[0005] In a first aspect, an embodiment of the present invention provides a file transfer method for a distributed file storage system based on FTP, the method being applied to a file transfer system for a distributed file storage system based on FTP, the system comprising: a client, an FTP server and a plurality of distributed storage nodes; the method comprising: the FTP server receiving a target file uploaded by a client via the FTP protocol, and performing segmentation processing on the target file to generate a plurality of file segments; the FTP server sending the plurality of the file segments to different distributed storage nodes for storage; the client simultaneously downloading different file segments from the plurality of distributed storage nodes determined according to the storage locations of the file segments, and merging them to generate a target file.

[0006] In some optional implementations, the FTP server receives the target file uploaded by the client via the FTP protocol, and slices the target file to generate a number of file slices, including: the client uploads the target file to the distributed storage system via the FTP protocol; the FTP server receives the target file, and slices the target file according to a predetermined sharding strategy to generate a number of file slices; the sharding strategy includes the number and size of the file slices.

[0007] In some optional implementations, the FTP server includes a load balancing module; the FTP server sends several of the above-mentioned file fragments to different distributed storage nodes for storage, including: the load balancing module of the above-mentioned FTP server dynamically allocates the above-mentioned file fragments to different distributed storage nodes for storage according to the storage capacity and load conditions of the nodes.

[0008] In some optional implementations, the above-mentioned distributed storage nodes include a primary distributed storage node and a secondary distributed storage node; the method also includes: the above-mentioned FTP server fragments the above-mentioned target file according to a predetermined fragmentation strategy to generate several copies of the file fragments; the above-mentioned load balancing module sends several copies of the above-mentioned file fragments to different secondary distributed storage nodes for storage.

[0009] In some optional implementations, the method further includes: performing integrity verification on the target file based on the file segments and generating metadata of the file segments; and determining storage locations of the file segments based on the client's download request and the metadata of the file segments.

[0010] In some optional implementations, the integrity of the target file is checked based on the file segments, and metadata of the file segments is generated, including: the FTP server generates metadata of the file segments based on the file segments, and sends them to corresponding distributed storage nodes; the metadata includes the file name, segment number, and storage location; each distributed storage node receives and stores the file segments and the metadata of the file segments, and sends feedback information to the FTP server; the FTP server checks the integrity of the file based on the feedback information.

[0011] In some optional implementations, the above method also includes: monitoring the above-mentioned primary distributed storage node, and when the above-mentioned primary distributed storage node fails, determining the metadata of the above-mentioned file shards stored on the above-mentioned primary distributed storage node; and determining the secondary distributed storage node storing a copy of the above-mentioned file shards based on the above-mentioned metadata.

[0012] In the second aspect, an embodiment of the present invention provides a file transfer system of a distributed file storage system based on FTP, the system comprising: a client, an FTP server and multiple distributed storage nodes; the FTP server is used to receive a target file uploaded by a client through the FTP protocol, and to segment the target file to generate a plurality of file segments; the FTP server is used to send the plurality of the file segments to different distributed storage nodes for storage; the client is used to simultaneously download different file segments from the plurality of distributed storage nodes determined according to the storage locations of the file segments, and merge them to generate a target file.

[0013] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and when the processor executes the computer program, the steps of any one of the methods described in the first aspect are implemented.

[0014] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores machine-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions cause the processor to execute any method described in the first aspect above.

[0015] The present invention provides a file transmission method and system of a distributed file storage system based on FTP, the method comprising: first receiving a target file uploaded by a client through the FTP protocol, and performing fragmentation processing on the target file to generate a plurality of file fragments; then sending the plurality of file fragments to different distributed storage nodes for storage to achieve file upload and storage; and then downloading different file fragments from multiple distributed storage nodes at the same time by the client, merging and generating a target file to achieve file download. The above method solves the technical problems of low file transmission efficiency and low system stability of the traditional method, and achieves the technical effect of improving file transmission efficiency and improving system reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A schematic diagram of the structure of a file transmission system of a distributed file storage system based on FTP provided by an embodiment of the present invention;

[0018] Figure 2 A schematic diagram of a flow chart of a file transmission method of a distributed file storage system based on FTP provided by an embodiment of the present invention;

[0019] Figure 3 A schematic diagram of a process of uploading a file in a file transmission method of a distributed file storage system based on FTP provided by an embodiment of the present invention;

[0020] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0022] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. Some embodiments of the present invention are described in detail below in conjunction with the drawings. The following embodiments and features in the embodiments can be combined with each other without conflict.

[0024] With the rapid development of informatization, the demand for file transfer and storage is increasing. The traditional file transfer protocol FTP has been widely used in various network environments, but with the advent of the big data era, a single server can hardly meet the needs of efficient storage and fast transmission of massive data. Distributed storage systems can improve data processing and storage capabilities through the collaboration of multiple server nodes. However, achieving stable and efficient file transfer in a distributed environment still faces many challenges, such as load balancing, fault tolerance, data security, etc. In other words, the application of the traditional FTP file transfer protocol in the prior art for file transfer in a distributed file storage system has technical problems such as low transmission efficiency and low system stability.

[0025] Based on this, the embodiment of the present invention provides a file transmission method and system for a distributed file storage system based on FTP, so as to effectively solve the deficiencies in the prior art and meet the requirements of efficient and secure file transmission in a big data environment.

[0026] To facilitate understanding of this embodiment, firstly, the structure of a file transmission system based on a distributed file storage system of FTP disclosed in an embodiment of the present invention is introduced. Figure 1The structure diagram of a file transmission system of a distributed file storage system based on FTP is shown, and the system includes: a client 110, an FTP server 120 and a plurality of distributed storage nodes 130. The FTP server and the plurality of distributed storage nodes constitute a distributed file storage system for implementing distributed storage of files. The client communicates with the distributed file storage system through the FTP protocol and transfers the target file to the distributed storage node.

[0027] In one embodiment, the FTP server includes a load balancing module, which is used to allocate file storage nodes according to the storage capacity and load of each distributed storage node. Specifically, the allocation of tasks can be determined by a dynamic load balancing algorithm based on the real-time load status information of the server, such as the least connection method, the weighted least connection method, etc.

[0028] Next, in conjunction with the above-mentioned file transmission system of the distributed file storage system based on FTP, a file transmission method of the distributed file storage system based on FTP disclosed in an embodiment of the present invention is introduced in detail. The method is applied to the file transmission system of the distributed file storage system based on FTP. Figure 2 The flowchart of a file transmission method of a distributed file storage system based on FTP is shown. The method can be executed by an electronic device and mainly includes the following steps S210 to S230:

[0029] S210: The FTP server receives the target file uploaded by the client through the FTP protocol, and performs segmentation processing on the target file to generate a plurality of file segments;

[0030] In one embodiment, the above step S210 may include:

[0031] (S211) The client uploads the target file to the distributed storage system via the FTP protocol;

[0032] (S212) The FTP server receives the target file, and segments the target file according to a predetermined segmentation strategy to generate a plurality of file segments.

[0033] The predetermined sharding strategy includes the number and size of file shards, etc. For example, the size range of a file shard is greater than or equal to 5MB and less than or equal to 5GB; the number of shards cannot exceed 10,000.

[0034] S220: The FTP server sends the plurality of file fragments to different distributed storage nodes for storage;

[0035] In one embodiment, the above steps include: (S221) the load balancing module of the FTP server dynamically allocates file fragments to different distributed storage nodes for storage according to the storage capacity and load conditions of the nodes.

[0036] Among them, the FTP server includes a load balancing module, which can determine the distribution of tasks based on the real-time load status information of the server through a dynamic load balancing algorithm, such as the least connection method, the weighted least connection method, etc. That is, the optimal storage node is selected through the load balancing algorithm, and the fragmented data is transmitted to different nodes respectively.

[0037] In this embodiment, the file is fragmented when it is uploaded, and the large file is divided into several small pieces, which are distributed and stored in different nodes, thereby improving the efficiency of file uploading. In particular, when transmitting large files, the upload time is significantly reduced through fragmented transmission.

[0038] In order to ensure data integrity, each storage node needs to perform data verification after receiving the file slice. That is, in one embodiment, the method may also include: performing integrity verification on the target file based on the file slice, and generating metadata of the file slice.

[0039] As a specific example, the steps include:

[0040] (1) The FTP server generates metadata of the file segments based on the file segments and sends it to the corresponding distributed storage nodes; the metadata of the file segments includes information such as the file name, segment number, and storage location;

[0041] (2) Each distributed storage node receives and stores the file segments and their corresponding metadata, and sends feedback information to the FTP server;

[0042] (3) The FTP server performs integrity check on the file based on the feedback information.

[0043] The feedback information may be the metadata of the file shard or the shard number in the metadata. For example, after the distributed storage node receives the file shard numbered 5, it sends feedback information of shard numbered 5 to the FTP server. The FTP server receives feedback information from all nodes that have stored the file shards, and compares the number of all shard numbers fed back with the number of shards originally generated. If the two numbers are consistent, the integrity check of the file is completed.

[0044] In another specific example, the FTP server receives feedback information from all nodes that store file fragments, and matches all the fragment numbers fed back with the fragment numbers in the metadata of the originally generated file fragments one by one. If the comparison results are consistent, the integrity check of the file is completed.

[0045] In this embodiment, after each storage node receives the file shard, it performs integrity check by the number of shards, which can improve the speed of the check; it performs integrity check by comparing the shard numbers one by one, which can ensure the accuracy of the check; it can also perform double checks on the number of shards and the shard numbers in sequence, that is, after passing the number check, it performs number comparison check, which can not only improve the speed of the check, but also ensure the accuracy of the check. Through the integrity check of the above file shards, the data integrity is guaranteed during the data upload process.

[0046] In addition, in order to improve the data fault tolerance of the system, a redundant storage mechanism is introduced for data storage. Redundant storage refers to duplicating and backing up data in the storage system to improve data reliability and availability. Its main purpose is to ensure data integrity and accessibility when a data storage device fails.

[0047] In one embodiment, the multiple distributed storage nodes constituting the distributed file storage system may be divided into primary distributed storage nodes and secondary distributed storage nodes.

[0048] That is to say, in this embodiment, the above method includes: first, the FTP server segments the target file according to a predetermined segmentation strategy to generate several file segments; then, the load balancing module of the FTP server sends the several file segments to different primary distributed storage nodes for storage.

[0049] At the same time, the FTP server fragments the target file according to a predetermined fragmentation strategy to generate several copies of the file fragments; then the load balancing module of the FTP server sends the copies of the several file fragments to different secondary distributed storage nodes for storage.

[0050] In this embodiment, each primary distributed storage node may correspond to one or more secondary distributed storage nodes. Therefore, when the target file is uploaded for sharding, the file shards are copied, at least one copy of the file shard is generated, and the file shard is transmitted to the primary distributed storage node, and at least one copy of the file shard is transmitted to at least one secondary distributed node.

[0051] Through the above redundant storage mechanism, when a primary distributed storage node fails or the file shard stored by the primary distributed storage node becomes unavailable due to various reasons, the file shard copy stored by the secondary distributed storage node can be used to complete the transmission of the file shard. This improves the fault tolerance of the system and ensures the integrity, high availability and data security of the file. S230: The client downloads different file shards from multiple distributed storage nodes determined according to the storage location of the file shards at the same time, and merges them to generate the target file.

[0052] In one embodiment, the above method also includes: based on the client's download request, according to the metadata of the file shard, determining the storage location of the file shard; the storage location is the distributed storage node storing the file shard, and the metadata of the file shard can be used as an index for searching the storage node.

[0053] That is to say, the client uses parallel transmission technology to download different file fragments from multiple distributed storage nodes at the same time, and merge them locally to generate the target file to achieve the download of the target file.

[0054] In this embodiment, the efficiency of file downloading is improved by parallel downloading, especially when large files are transmitted, the download time is significantly reduced.

[0055] In addition, during the download process, the integrity of the shard data can also be automatically verified. If a node fails, the system automatically obtains the shard data from the backup node (i.e., the secondary distributed storage node that stores a copy of the file shard on the node).

[0056] That is, in one embodiment, the above method further includes:

[0057] (1) monitoring the primary distributed storage node, and when the primary distributed storage node fails, determining the metadata of the file shards stored on the primary distributed storage node;

[0058] Among them, a distributed storage node failure generally refers to a communication failure between the node and the FTP server.

[0059] (2) determining a secondary distributed storage node storing a copy of the file shard based on the metadata;

[0060] (3) The client downloads a copy of the corresponding file segment from the above-mentioned secondary distributed storage node, and merges it with other file segments to generate a target file.

[0061] In this embodiment, the system monitors the status of each storage node, detects node failures in real time, and starts the secondary distributed storage node that stores the file shard copy to complete the download of the corresponding file shard, thereby ensuring data integrity and improving system stability.

[0062] In addition, in this embodiment, when a node is unavailable, the system can automatically copy the file shard copies stored on the node to other healthy nodes; and during the data transmission process, if a network failure occurs, the system supports breakpoint resumption to ensure the reliability of transmission.

[0063] In the above embodiment, during the process of uploading and downloading files, the system encrypts the file segment data to ensure the security of the transmitted data; specifically, the encryption process may be to use a channel encryption protocol to transmit the file segment during the transmission process, or to encrypt the data packets of the file segment before transmission. This provides a more secure file transmission environment and ensures the confidentiality and security of the data.

[0064] In the above embodiment, when the file segments are stored on the distributed storage nodes, a permission control mechanism is used to ensure that only authorized users can access the files, thereby providing a more secure file storage environment and ensuring the confidentiality and security of the data.

[0065] As a specific example, combining Figure 3 As shown, this embodiment discloses a specific process of uploading a file through the above method, including:

[0066] S31: The client uploads the target file to the distributed storage system through the FTP protocol;

[0067] S32: The FTP server receives the target file, and performs fragmentation processing on the target file according to a predetermined fragmentation strategy, generating a plurality of file fragments and their metadata, and copies of the file fragments and their metadata;

[0068] S33: The FTP server selects the best storage node through a load balancing algorithm, and sends a number of file fragments and their metadata, and copies of the file fragments and their metadata to different primary distributed storage nodes and secondary distributed storage nodes for storage;

[0069] S34: Each distributed storage node receives and stores the file segments and their corresponding metadata, and sends feedback information to the FTP server;

[0070] S35: The FTP server performs integrity check on the file based on the feedback information.

[0071] That is to say, the specific process of uploading files disclosed in this embodiment includes: first, the client connects to the FTP server through the FTP protocol; then the target file to be uploaded is processed in pieces, and the piece data selects a storage node for transmission through a load balancing algorithm; after each node receives the piece, data verification is performed, and the metadata of the piece is recorded in the central index library of the system. The central index library can be stored in a separate server or directly in the FTP server.

[0072] This embodiment also discloses a file download process, that is, when a client requests to download a file, the FTP server obtains the storage location information of the file fragments from the central index library, and instructs the client to download the fragment data from each distributed node in parallel. After the download is completed, the client merges the fragments and restores them to the original file.

[0073] This embodiment also discloses the fault tolerance and data recovery mechanism of the system. When a primary distributed storage node fails, the system detects that the node is unavailable and automatically obtains data from the secondary distributed storage node with redundant storage, and copies the file shard copies on the failed node to other healthy nodes. The system also supports breakpoint resume. When an interruption occurs during the transmission process, the transmission can be continued at the breakpoint to ensure the reliability of file transmission.

[0074] The present invention provides a method for file transmission in a distributed file storage system based on FTP, which can improve the reliability and security of the system while ensuring the transmission efficiency, effectively solves the deficiencies in the prior art, and meets the requirements for efficient and secure file transmission in a big data environment.

[0075] The embodiment of the present invention also provides a file transmission system based on a distributed file storage system of FTP, the system comprising: a client, an FTP server and a plurality of distributed storage nodes.

[0076] The FTP server is used to receive the target file uploaded by the client through the FTP protocol, and to segment the target file into several file segments;

[0077] The FTP server is used to send several file fragments to different distributed storage nodes for storage;

[0078] The client is used to simultaneously download different file fragments from multiple distributed storage nodes determined according to the storage locations of the file fragments, and merge them to generate a target file.

[0079] In other words, the FTP server and multiple distributed storage nodes form a distributed file storage system for implementing distributed storage of files. The client communicates with the distributed file storage system through the FTP protocol to upload target file fragments to the distributed storage nodes for storage, and download different file fragments from multiple distributed storage nodes at the same time and merge them to generate the target file.

[0080] In one embodiment, the FTP server includes a load balancing module, which is used to allocate file storage nodes according to the storage capacity and load of each distributed storage node. The file transfer system based on the FTP distributed file storage system provided in the embodiment of the present application can be specific hardware on the device or software or firmware installed on the device. The device provided in the embodiment of the present application, its implementation principle and the technical effect produced are the same as those in the aforementioned method embodiment. For the sake of brief description, the parts not mentioned in the device embodiment can refer to the corresponding contents in the aforementioned method embodiment. Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here. The file transfer system based on the FTP distributed file storage system provided in the embodiment of the present application has the same technical features as the file transfer method based on the FTP distributed file storage system provided in the above embodiment, so it can also solve the same technical problems and achieve the same technical effects.

[0081] In one embodiment, the system includes multiple distributed storage nodes, each of which includes a storage module, a verification module, and a security module. The client interacts with the system through FTP client software to upload and download files. During the file upload process, the file is first fragmented and distributed to multiple nodes; during the download process, the client obtains the fragmented data from multiple nodes through parallel download technology, and performs verification and merging.

[0082] In an actual application scenario, a user needs to upload a large file of more than 1TB to a distributed storage system. The file is automatically divided into 100 small file fragments, and each fragment is transmitted to a different storage node. When the client downloads the file, the system downloads the 100 fragments from multiple nodes in parallel based on the metadata index, significantly improving the download speed.

[0083] The embodiment of the present invention provides a file transmission method and system based on a distributed file storage system of FTP, which has the characteristics of high efficiency, safety and reliability, can well solve the transmission and storage requirements of large-scale files, and is suitable for data management applications in enterprise-level storage systems and cloud computing environments. The system has good scalability and can flexibly increase storage nodes according to demand to meet the needs of large-scale data storage and transmission.

[0084] An embodiment of the present application further provides an electronic device. Specifically, the electronic device includes a processor and a storage device. The storage device stores a computer program, and when the computer program is executed by the processor, it executes the method described in any one of the above-mentioned embodiments.

[0085] Figure 4A structural schematic diagram of an electronic device provided in an embodiment of the present application, the electronic device 400 includes: a processor 40, a memory 41, a bus 42 and a communication interface 43, wherein the processor 40, the communication interface 43 and the memory 41 are connected via the bus 42; the processor 40 is used to execute an executable module stored in the memory 41, such as a computer program.

[0086] The memory 41 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 43 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0087] The bus 42 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0088] Among them, the memory 41 is used to store programs, and the processor 40 executes the program after receiving the execution instruction. The method executed by the process definition device disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 40 or implemented by the processor 40.

[0089] The processor 40 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 40. The above processor 40 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor to execute, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 41, and the processor 40 reads the information in the memory 41 and completes the steps of the above method in combination with its hardware.

[0090] Corresponding to the above method, an embodiment of the present application also provides a computer-readable storage medium, which stores machine-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the steps of the above method.

[0091] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0092] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0093] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0094] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, electronic device, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0095] It should be noted that similar numbers and letters represent similar items in the accompanying drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A file transmission method of a distributed file storage system based on FTP, characterized in that: A file transfer system applied to a distributed file storage system based on FTP, the system comprising: a client, an FTP server and a plurality of distributed storage nodes; the method comprising: The FTP server receives the target file uploaded by the client through the FTP protocol, and performs fragmentation processing on the target file to generate a plurality of file fragments; The FTP server sends the plurality of file segments to different distributed storage nodes for storage; The client simultaneously downloads different file fragments from a plurality of distributed storage nodes determined according to the storage locations of the file fragments, and merges them to generate a target file.

2. The file transmission method of the distributed file storage system based on FTP according to claim 1, characterized in that: The FTP server receives a target file uploaded by a client through the FTP protocol, and performs fragmentation processing on the target file to generate a plurality of file fragments, including: The client uploads the target file to the distributed storage system via the FTP protocol; The FTP server receives the target file, and performs fragmentation processing on the target file according to a predetermined fragmentation strategy to generate a plurality of file fragments; the fragmentation strategy includes the number and size of the file fragments.

3. The file transmission method of the distributed file storage system based on FTP according to claim 2, characterized in that: The FTP server includes a load balancing module; the FTP server sends a plurality of the file fragments to different distributed storage nodes for storage, including: The load balancing module of the FTP server dynamically allocates the file fragments to different distributed storage nodes for storage according to the storage capacity and load conditions of the nodes.

4. The file transmission method of the distributed file storage system based on FTP according to claim 3, characterized in that: The distributed storage node includes a primary distributed storage node and a secondary distributed storage node; the method further includes: The FTP server performs fragmentation processing on the target file according to a predetermined fragmentation strategy to generate a plurality of copies of the file fragments; The load balancing module sends copies of several file shards to different secondary distributed storage nodes for storage.

5. The file transmission method of the distributed file storage system based on FTP according to claim 4, characterized in that: The method further comprises: Performing integrity check on the target file based on the file slices, and generating metadata of the file slices; Based on the download request of the client, the storage location of the file segment is determined according to the metadata of the file segment.

6. The file transmission method of the distributed file storage system based on FTP according to claim 5, characterized in that: Performing integrity check on the target file based on the file slices and generating metadata of the file slices includes: The FTP server generates metadata of the file segment based on the file segment and sends it to the corresponding distributed storage node; the metadata includes the file name, segment number, and storage location; Each distributed storage node receives and stores the file segments and metadata of the file segments, and sends feedback information to the FTP server; The FTP server performs integrity check on the file based on the feedback information.

7. The file transmission method of the distributed file storage system based on FTP according to claim 5, characterized in that: Also includes: Monitoring the primary distributed storage node, and when the primary distributed storage node fails, determining metadata of the file shards stored on the primary distributed storage node; A secondary distributed storage node storing a copy of the file shard is determined based on the metadata.

8. A file transfer system based on a distributed file storage system of FTP, characterized in that: The system comprises: a client, an FTP server and a plurality of distributed storage nodes; The FTP server is used to receive the target file uploaded by the client through the FTP protocol, and to perform fragmentation processing on the target file to generate a plurality of file fragments; The FTP server is used to send the plurality of file fragments to different distributed storage nodes for storage; The client is used to simultaneously download different file fragments from a plurality of distributed storage nodes determined according to the storage locations of the file fragments, and merge them to generate a target file.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores machine-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions cause the processor to execute the method according to any one of claims 1 to 7.

Images