FTP horizontal expansion method and FTP horizontal expansion system

Abstract

The present application relates to FTP horizontal expansion method and FTP horizontal expansion system. FTP horizontal expansion method, including: (1) changing the storage mode of FTP, using distributed storage mode instead of single computer storage mode of FTP; (2) replacing the single disk directory structure of FTP, changing the original local directory mode of FTP into virtual file directory, using relational database to store the directory structure of FTP, and simulating the upper and lower level relationship of directory through the self-connection mode of relational database; (3) an FTP protocol compatible with standard FTP client communication program, combining ESS and VD to form the complete storage architecture of FTP-AS scalable storage system; (4) migrating historical files from the previous FTP server to the new distributed storage engine, and mapping the previous directory structure to the virtual file directory of step (2).

Description

Technical Field

[0001] This invention belongs to the field of FTP server storage technology, and specifically relates to FTP horizontal expansion methods and FTP horizontal expansion systems. Background Technology

[0002] As a product of the early development of computer science, FTP enjoys a unique popularity, and almost every company uses it to some extent. However, it also has its inherent drawbacks, such as the problem of single point of failure and the limitation of storage capacity to a single physical machine.

[0003] When a single point of failure occurs, RAID technology is needed to redundancy the disks, but this comes at the cost of sacrificing storage space (single machine storage space is inherently limited and not very durable).

[0004] When insufficient capacity is reached and the storage limit is exceeded, the only way to free up space is to delete useless files or archive historical files to another location.

[0005] These shortcomings increase maintenance costs and add complexity that is not part of the business. When carrying out business operations, these storage-related technical issues must also be considered.

[0006] Despite its obvious drawbacks, FTP is very popular, and directly replacing it with a modern distributed storage server would be risky, require significant system modifications, and unnecessarily increase company costs. Summary of the Invention

[0007] The technical problem this invention aims to solve is to overcome the limitation of traditional FTP storage space by the capacity of a single physical machine. It provides a method and system for horizontal FTP expansion by adding new physical machines, partially addressing the limitation of single-machine FTP storage capacity and the issue of seamless / unobtrusive integration with business systems. Another technical problem this invention aims to solve is to provide a migration scheme combined with virtual directory services, enabling existing FTP-based business systems to seamlessly / unobtrusively integrate with the FTP horizontal expansion method and system. A further technical problem this invention aims to solve is to provide a method and system for horizontal FTP expansion achieved by modifying the traditional FTP structure; by adding a fully FTP protocol-compatible proxy server (FTPAgentServer) to extend traditional FTP services, allowing business system development to focus on the business itself rather than irrelevant storage technology issues. A fourth technical problem this invention aims to solve is to provide a method and system for horizontal FTP expansion that retains the original FTP protocol, allowing business systems to obtain scalable FTP servers without modification, eliminating concerns about insufficient storage capacity.

[0008] The technical solution of the present invention is the FTP horizontal scaling method, which is characterized by including the following steps:

[0009] (1) Expandable Storage Module (ESS): Changes the storage method of FTP to use distributed storage instead of the single-machine storage mode of FTP;

[0010] (2) Virtual Directory Service (VD): Replaces the single-machine disk directory structure of FTP, changes the original local directory method of FTP to a virtual file directory, uses a relational database to store the directory structure of FTP, and simulates the upper and lower hierarchy relationships of the directory through the self-join method of the relational database;

[0011] (3) FTP Proxy Server (FTP-AS): A program compatible with the FTP protocol that can communicate with standard FTP clients. Combined with ESS and VD, it forms the complete storage architecture of the scalable storage system of FTP-AS.

[0012] (4) Migration tool: Migrate historical files from the previous FTP server to the new distributed storage engine, and map the previous directory structure to the virtual file directory in step (2).

[0013] As a preferred embodiment, the virtual directory service described in step (2) further includes:

[0014] (2.1) Introducing ESS for expansion: Since files will be distributed across different machines, directories must be separated from disks. This separation allows for the management of files distributed across multiple machines under ESS.

[0015] (2.1.1) User manually creates the directory;

[0016] (2.1.2) User manually deletes the directory;

[0017] (2.1.3) When a user deletes a file, one file will be removed from the directory;

[0018] (2.1.4) When a user adds a file, it will result in an additional file in the directory;

[0019] (2.1.5) When a user uploads a file in chunks, the size of the files in the directory will change.

[0020] (2.1.6) Establish the following table structure:

[0021]

[0022] As a preferred option: Step 2 further includes: taking the example of uploading an employee's .exe file [employee.xlsx] from the OA system to the new FT-AS system and storing it by year and month:

[0023] A. The root directory assigned to the FTP user is: / ftp / oa_user, and the file needs to be uploaded to the directory / ftp / oa_user / employees / 2022 / 05;

[0024] B. Upload files:

[0025] (1) Establish a TCP connection using a standard FTP client and FTP-AS;

[0026] (2) Check if the directory exists. If it does not exist, create the directory:

[0027] / ftp / oa_user / employees / 2022 / 05, note the difference from traditional FTP: it does not create a real directory on the server disk where FTP-AS is located, but only registers several directory records in the database table, and checks in turn whether the directories employees, 2022, and 05 exist. If none of them exist, then three database directory records are created.

[0028] (3) FTP-AS establishes a connection with the underlying Scalable Storage Service (ESS) and stores the employee.xlsx file uploaded by the FTP client in ESS. ESS returns the fileId to FTP-AS. The purpose of this ID is to uniquely identify the location of the file in ESS.

[0029] (4) FTP-AS combines the fileId returned by ESS and records it in the directory as the ess_path attribute of ftp_virtual_directory;

[0030] (5) FTP-AS uses the standard FTP protocol to inform the FTP client that the file upload was successful;

[0031] Download file

[0032] ① The FTP client and FTP-AS establish a TCP connection, using the path

[0033] Send a GET command with the parameter / ftp / oa_user / employees / 2022 / 05 / employee.xlsx;

[0034] ②When FTP-AS receives the get command, it finds the file from the virtual directory based on the path parameter and extracts the ess_path parameter;

[0035] ③FTP-AS establishes a TCP connection with ESS, retrieves the file from ESS according to ess_path, and forwards it synchronously to the business client;

[0036] Delete file

[0037] ① The FTP client establishes a TCP connection with FTP-AS and sends the rm command with the path / ftp / oa_user / employees / 2022 / 05 / employee.xlsx as the parameter;

[0038] ②FTP-AS; Upon receiving the rm command, it locates the file from the virtual directory based on the path parameters and extracts the ess_path parameter;

[0039] ③FTP-AS establishes a TCP connection with ESS and deletes the file from ESS according to ess_path;

[0040] ④ FTP-AS deletes the virtual directory record corresponding to the file; it then sends a notification to the business system client, confirming successful deletion.

[0041] View files in the directory

[0042] ① The FTP client establishes a TCP connection with FTP-AS and sends the ls command with the directory / ftp / oa_user / employees / 2022 / 05 as the parameter;

[0043] ②When FTP-AS receives the ls command, it queries the database and retrieves the directory's UUID by comparing the directory parameters with the full_path.

[0044] ③ By passing the UUID as the parent_uuid parameter, you can obtain all direct child files in this directory;

[0045] ④ Feedback is sent to the business system.

[0046] As a preferred option: Step 3 further includes: using file-based object storage services as the ESS solution, specifically including: no need to care which server the file is stored on, the ESS server uniformly assigns a fileId to the client; the corresponding storage method is a block-based storage method, which divides the file into the smallest blocks supported by the operating system, which can improve disk utilization and reduce disk fragmentation. Enterprises with requirements for storage utilization can choose a suitable storage solution.

[0047] The storage model is divided into two main modules: Tracker and Storage. Multiple Storage units form a Group, which is a virtual management unit, unlike the Tracker and Storage units which are specific programs.

[0048] The management characteristics of a group are as follows:

[0049] (3.1) Multiple Storages within the same Group are redundant with each other, forming a mutual backup relationship;

[0050] (3.2) Generally speaking, two Storages form a Group. If backup is not required, a single Storage can form a Group.

[0051] (3.3) The storage capacity of a Group is equal to the capacity of the smallest storage device in the same group, that is, Volume(Group) = Volume(Min(Storage1,Storage2,...,StorageN)); the disk space of Storage devices in the same group must be the same size;

[0052] (3.4) The capacity of the entire cluster is equal to the sum of the capacities of the multiple Groups:

[0053] Volume(Cluster) = Volume(Group1) + Volume(Group2) + Volume(Group N); Later, to increase space, simply add a new Group, such as for backup expansion.

[0054] Preferably, the storage mode includes the following control flow: a tracker maintains the network topology of the storage; guides multiple storage instances within the same group to directly synchronize data; and provides real-time feedback on the network health of the storage to ensure that each file upload is allocated to a valid storage instance. For example, regarding file uploads:

[0055] (1) FTP-AS received the file uploaded by the business system;

[0056] (2) FTP-AS requests the Tracker module under ESS to ask which group has the most free space and returns the group with the most free space to FTP-AS. For example, if Group003 has the highest free space ratio, then Group003 is returned to FTP-AS.

[0057] (3) FTP-AS randomly selects a storage device from Group003, let's say it's A01. FTP-AS establishes a connection with A01 and uploads the file to A01.

[0058] (4) At this point, the file upload operation on the business system side is complete, but ESS itself still needs to complete the next steps;

[0059] (5) The memory A01 will synchronously report the received files to the tracker;

[0060] (6) The tracker synchronizes this file record to other storage devices in the same group as A01, and the other storage devices begin to request file synchronization;

[0061] (7) After all storage devices in the same group have been synchronized, the tracker records the status of the file as synchronized, and the design of the file storage module ESS is now complete.

[0062] As a preferred option: Step (4) further includes: Due to the fundamental changes in the underlying storage, in order for the business system to operate seamlessly, it is also necessary to migrate historical files. Migration can be divided into static migration and dynamic migration.

[0063] (4.1) The static migration strategy is as follows: The original FTP server files are stored on the disk, requiring only a recursive algorithm to traverse the folders, starting from the top-level folder, to synchronize historical data to the new architecture; the traversal algorithm is as follows:

[0064] (4.1.1) Pass in the top-level directory and record the top-level directory in the virtual directory table;

[0065] (4.1.2) Obtain all files in the top-level directory and read them in a loop;

[0066] (4.1.3) If it is a file, call the interface of the ESS module to store it under the ESS module, get the fileId assigned by the ESS module, and store it in the virtual directory table in combination with the parent directory;

[0067] (4.1.4) If it is a folder, return to step (4.1.1) and perform recursive judgment;

[0068] (4.1.5) The migration ends when the last file is stored in ESS and synchronized to the virtual directory;

[0069] (4.2) The dynamic migration strategy is as follows: If the number of files is too large or too many, causing static migration to take a long time, and the business system cannot be interrupted, then dynamic migration is needed to integrate the migration process into FTP-AS; the process is as follows:

[0070] (4.2.1) The business system requests to download a certain historical file;

[0071] (4.2.2) FTP-AS then detects that the historical files have not yet been synchronized, so it acts as an FTP client, establishes a connection with the original FTP server, and caches the files locally.

[0072] (4.2.3) Cache the files:

[0073] a. Return the information to the business system;

[0074] b. Create an asynchronous task, upload it to ESS, and store a file record in the virtual directory so that it can be downloaded from ESS on the next request.

[0075] Another technical solution of the present invention is the FTP horizontal expansion system, which includes a business system that uses FTP service internally. The special feature is that the business system that uses FTP service internally communicates with the expandable storage structure through an FTP client. The communication method includes permission authentication / file operation.

[0076] Preferably, the scalable storage structure includes a file system gateway node cluster, a tracker cluster, and a storage cluster. The file system gateway node cluster communicates with the storage cluster through the tracker cluster, and the communication includes file operations through querying and health feedback.

[0077] Preferably, the business systems that use FTP services internally include CRM systems, OA systems, risk control systems, and integrated management backends.

[0078] Preferably, the tracker cluster includes at least one virtual machine tracker;

[0079] The memory cluster comprises several groups of physical machines and several groups of physical machines (backup expansion).

[0080] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0081] (1) This invention eliminates the need to worry about insufficient FTP storage space: there is no need to delete business files or archived historical files due to insufficient space, thus reducing the company's operation and maintenance costs.

[0082] (2) The historical files stored in the old FTP server can be migrated to the new FTP server.

[0083] (3) The business system of this invention is unaware of this underlying change, that is, it is not intrusive to the original business system and is seamlessly connected.

[0084] (4) The redesigned architecture is much more complex. The FTP server that interfaces with the business system has become an FTP proxy server (FTP-AS), and the requirement has been changed from only one server to at least one. Multiple FTP-AS servers can establish a shared authorization system through a database. From the redesigned architecture, FTP-AS no longer directly stores files; instead, the storage task is delegated to ESS, enabling horizontal scaling.

[0085] (5) Although the modified architecture is much more complex, the top-level business system still uses a traditional FTP client to interact with the underlying storage, regardless of whether it is before or after the modification. This means that the business system does not need to be modified at all, which is one of the core problems solved by this invention. Attached Figure Description

[0086] Figure 1 This is a diagram of the traditional FTP architecture before the modification;

[0087] Figure 2 This is a diagram of the modified FTP architecture of this invention;

[0088] Figure 3 This is a schematic diagram of the expandable storage structure of the present invention. Detailed Implementation

[0089] The present invention will now be described in further detail with reference to the accompanying drawings:

[0090] The FTP horizontal scaling method includes the following steps:

[0091] (1) Expandable Storage Module (ESS): Changes the storage method of FTP to use the currently mainstream distributed storage method instead of the traditional single-machine storage mode of FTP;

[0092] (2) Virtual Directory Service (VD): Replaces the single-machine disk directory structure of FTP, changes the original local directory method of FTP to a virtual file directory, uses a relational database to store the directory structure of FTP, and simulates the upper and lower hierarchy relationships of the directory through the self-join method of the relational database;

[0093] (3) FTP Proxy Server (FTP-AS): A program compatible with the FTP protocol that can communicate with standard FTP clients. Combined with ESS and VD, it forms the complete storage architecture of the scalable storage system of FTP-AS.

[0094] (4) Migration tool: Migrate historical files from the previous FTP server to the new distributed storage engine, and map the previous directory structure to the virtual file directory in step (2).

[0095] Virtual directories in FTP-AS:

[0096] Traditional FTP uses local disks for file storage, so directories are bound to the disk, similar to folders created on the C and D drives of a Windows computer. However, with the introduction of ESS for expansion, files will be distributed across different machines, and directories must be separated from the disks.

[0097] To successfully achieve separation, we need to understand the essence of a directory. Typically, a file directory is used to categorize files for easier file management; more abstractly, photo albums also categorize photos, which is essentially a type of directory. The function of a directory is clearly categorization. From a computer's perspective, a directory is essentially an address identifier, its essence being a unique identifier for a file. Through this understanding, from a computer's perspective, files and directories are essentially the same—address markers. Based on this essence, once the purpose of categorization is achieved, we can redesign the directory functionality to manage files distributed across multiple machines under ESS. The following situations are related to directories:

[0098] (1) User manually creates directories

[0099] (2) User manually deletes the directory

[0100] (3) When a user deletes a file, one file will be removed from the directory.

[0101] (4) When a user adds a file, it will result in an additional file in the directory.

[0102] (5) When a user uploads a file in chunks (note: this is an extended feature and not part of the standard FTP protocol), the file size in the directory will change.

[0103] ⑹ View the files in a directory

[0104] In summary, all of the above situations essentially involve changes in file metadata. Based on these requirements, we choose a relational database, such as Huawei's Gaussian Database or the open-source MySQL database, to manage directory entries. We can create the following table structure:

[0105]

[0106] Table 1 Virtual Directory Design

[0107] Note: The main focus of this invention is to solve the problems of limited storage capacity on a single FTP machine and seamless / unnoticeable integration with business systems. Therefore, permission issues are not described, and it is assumed that permission checks pass.

[0108] Taking the uploading of an employee's Excel file [employee.xlsx] from the OA system to the new FT-AS system and storing it by year and month as an example.

[0109] The root directory assigned to the FTP user is / ftp / oa_user, and the file needs to be uploaded to the directory / ftp / oa_user / employees / 2022 / 05.

[0110] Upload File

[0111] 1. Establish a TCP connection using a standard FTP client and FTP-AS.

[0112] 2. Check if the directory exists; if not, create the directory:

[0113] / ftp / oa_user / employees / 2022 / 05, note the difference from traditional FTP: it doesn't create a real directory on the server disk where FTP-AS resides. Instead, it simply registers several directory records in a database table, checking if the directories employees, 2022, and 05 exist. If none of them exist, it creates three database directory records.

[0114] 3. FTP-AS establishes a connection with the underlying Scalable Storage Service (ESS) and stores the employee.xlsx file uploaded by the FTP client in ESS. ESS returns the fileId to FTP-AS. This ID is used to uniquely identify the location of the file in ESS.

[0115] 4. FTP-AS combines the fileId returned by ESS with the ess_path attribute of ftp_virtual_directory and records it in the directory.

[0116] 5. FTP-AS uses the standard FTP protocol to inform FTP clients that the file upload was successful.

[0117] Download file

[0118] (1) The FTP client and FTP-AS establish a TCP connection, using the path

[0119] Send a GET command with the parameter / ftp / oa_user / employees / 2022 / 05 / employee.xlsx.

[0120] (2) When FTP-AS receives the get command, it locates the file in the virtual directory based on the path parameter and extracts the ess_path parameter.

[0121] (3) FTP-AS establishes a TCP connection with ESS, retrieves the file from ESS according to ess_path, and forwards it synchronously to the business client.

[0122] Delete file

[0123] (1) The FTP client and FTP-AS establish a TCP connection, using the path

[0124] The file / ftp / oa_user / employees / 2022 / 05 / employee.xlsx is used as a parameter to send the rm command;

[0125] (2) When FTP-AS receives the rm command, it finds the file from the virtual directory according to the path parameter and extracts the ess_path parameter.

[0126] (3) FTP-AS establishes a TCP connection with ESS and deletes the file from ESS according to ess_path;

[0127] (4) FTP-AS deletes the virtual directory record corresponding to the file. It then sends a notification to the business system client confirming successful deletion.

[0128] View files in the directory

[0129] (1) The FTP client and FTP-AS establish a TCP connection to the directory.

[0130] Send the ls command with the parameter / ftp / oa_user / employees / 2022 / 05.

[0131] (2) When FTP-AS receives the ls command, it queries the database and retrieves the directory's UUID by comparing the directory parameter with the full_path.

[0132] (3) By using the UUID as the parent_uuid parameter, you can obtain all direct sub-files (including direct subdirectories) under this directory;

[0133] (4) Feedback is sent to the business system;

[0134] FTP-AS Expandable Storage System

[0135] Distributed storage is currently the mainstream storage method in computer science. The latest business system development will basically choose distributed storage as the first solution. There are many implementation schemes for distributed storage. In order to be compatible with FTP itself, we will retain the storage method with files as the storage unit, that is, object storage service with files as the storage unit as our ESS solution.

[0136] Object storage services (ESS) mean that clients don't need to worry about which server a file is stored on; the ESS server assigns a unique fileId to each client. Another corresponding storage method is block-based storage, which divides files into the smallest blocks supported by the operating system. This improves disk utilization and reduces disk fragmentation, but it's not the core focus of this invention. Enterprises with high storage utilization requirements can choose a suitable storage solution.

[0137] Our storage model is divided into two main modules: Tracker and Storage. Multiple Storage units can be combined into a Group. Note that a Group is a virtual management unit, unlike the Tracker and Storage units which are specific programs; let's first look at the Group from an overall perspective.

[0138] The management characteristics of a group are as follows:

[0139] (1) Multiple Storages within the same Group are redundant and form a mutual backup relationship.

[0140] (2) Generally speaking, it is more appropriate to form a group with two storages. If backup is not required, a single storage can also form a group.

[0141] (3) The storage capacity of a Group is equal to the capacity of the smallest storage device in the same group, that is, Volume(Group) = Volume(Min(Storage1,Storage2,......,StorageN)). Therefore, the disk space of Storage in the same group should ideally be kept at the same size, otherwise it will be easy to waste, because the extra disk space will not be usable due to the constraints of mutual backup.

[0142] (4) The capacity of the entire cluster is equal to the sum of the capacities of multiple Groups: Volume(Cluster) = Volume(Group1) + Volume(Group2) ... + Volume(Group N). Therefore, to increase the space later, you only need to add new Groups, as shown in the backup expansion section in the lower right corner of the diagram above.

[0143] Tracker / Storage workflow:

[0144] Compared to Storage, Tracker's work is simpler and lighter. Its main function is to maintain the Storage network topology, such as identifying which Storage instances belong to the same group, and guiding data synchronization between multiple storage instances within the same group. It also provides real-time feedback on Storage network health, ensuring that every file upload is assigned to a valid Storage instance.

[0145] Taking file upload as an example:

[0146] (1) FTP-AS received the file uploaded by the business system;

[0147] (2) FTP-AS requests the Tracker module under ESS to ask which group has the most free space and returns the group with the most free space to FTP-AS. Assuming that Group003 in the above figure has the highest free space ratio, Group003 is returned to FTP-AS.

[0148] (3) FTP-AS randomly selects a storage device from Group003, let's say it's A01 in the diagram above. FTP-AS establishes a connection with A01 and uploads the file to A01.

[0149] (4) At this point, the file upload operation on the business system side is complete, but ESS itself still needs to complete the next steps;

[0150] (5) The memory A01 will synchronously report the received files to the tracker;

[0151] (6) The tracker synchronizes this file record to other storage devices in the same group as A01, and the other storage devices begin to request file synchronization;

[0152] (7) After all storage devices in the same group have completed synchronization, the tracker records the file's status as synchronized. This synchronization status is crucial. If synchronization is incomplete but a download request is received, the storage device where the file was first written to disk will be used first to provide download service. If synchronization is complete, a storage device will be randomly selected to provide download service.

[0153] The file download process has already been covered in the file upload section, and the file deletion process is similar; thus, the ESS file storage module design is complete. In actual implementation, the Tracker can also be integrated with FTP-AS as a single program, reducing architectural complexity.

[0154] Historical document migration

[0155] Although this invention maintains protocol compatibility in its design, enabling seamless integration with business systems, the underlying storage has undergone fundamental changes. For business systems to operate without being affected, historical files still need to be migrated. Migration can be divided into static migration and dynamic migration. In actual implementation, the best results are achieved through a combination of both, minimizing the impact on business systems.

[0156] The static migration strategy (offline migration) is as follows:

[0157] The original FTP server files were stored on the disk. A simple recursive algorithm to traverse the folders is all that's needed to synchronize historical data to the new architecture, starting from the top-level folder. The traversal algorithm is as follows:

[0158] (1) Pass in the top-level directory and record the top-level directory in the virtual directory table;

[0159] (2) Obtain all files (including subfolders) in the top-level directory and read them in a loop;

[0160] (3) If it is a file, call the interface of the ESS module to store it under the ESS module, get the fileId assigned by the ESS module, and store it in the virtual directory table in combination with the parent directory.

[0161] (4) If it is a folder, return to step 1 and perform recursive judgment;

[0162] (5) The migration ends when the last file is stored in ESS and synchronized to the virtual directory;

[0163] The dynamic migration strategy (online migration) is as follows:

[0164] If the number of files is too large or too many, static migration may take a long time, but the business system cannot be interrupted. In this case, dynamic migration is needed, and the migration process should be integrated into FTP-AS. The process is as follows:

[0165] (1) The business system requests to download a certain historical file;

[0166] (2) FTP-AS then detects that the historical files have not yet been synchronized, so it acts as an FTP client, establishes a connection with the original FTP server, and caches the files locally.

[0167] (3) Cache the files:

[0168] a. Return the information to the business system;

[0169] b. Create an asynchronous task, upload it to ESS, and store a file record in the virtual directory so that it can be downloaded from ESS on the next request.

[0170] Please see Figure 1, Figure 2 As shown, the FTP horizontal expansion system includes an internal business system that uses FTP services and an expandable storage structure. The internal business system that uses FTP services performs permission authentication / file operations with the expandable storage structure through an FTP client.

[0171] Please see Figure 3 As shown, the scalable storage structure includes a file system gateway node cluster, a tracker cluster, and a storage cluster. The file system gateway node cluster communicates with the storage cluster through the tracker cluster, and the communication includes file operations through querying and health feedback.

[0172] The internal business systems that use FTP services include CRM systems, OA systems, risk control systems, and integrated management backends.

[0173] The tracker cluster includes at least one virtual machine tracker;

[0174] The memory cluster comprises several groups of physical machines and several groups of physical machines (backup expansion).

[0175] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made within the scope of the claims of the present invention should be covered by the claims of the present invention.

Claims

1. A method for horizontal scaling of FTP, comprising the following steps: (1) Expandable Storage Module (ESS): Changes the storage method of FTP to use distributed storage instead of the single-machine storage mode of FTP; Its features are, It also includes the following steps: (2) Virtual Directory Service (VD): Replaces the single-machine disk directory structure of FTP, changes the original local directory method of FTP to a virtual file directory, uses a relational database to store the directory structure of FTP, and simulates the upper and lower hierarchy relationships of the directory through the self-join method of the relational database. The virtual directory service further includes: (2.1) Introducing ESS for expansion: Since files will be distributed across different machines, directories must be separated from disks. This separation allows for the management of files distributed across multiple machines under ESS. (2.1.1) User manually creates the directory; (2.1.2) User manually deletes the directory; (2.1.3) When a user deletes a file, one file will be removed from the directory; (2.1.4) When a user adds a file, it will result in an additional file in the directory; (2.1.5) When a user uploads a file in chunks, the size of the files in the directory will change. (2.1.6) Establish the following table structure: (3) FTP Proxy Server FTP-AS: A program compatible with the FTP protocol that can communicate with standard FTP clients. Combined with ESS and VD, it forms the complete storage architecture of the scalable storage system of FTP-AS. (4) Migration tool: Migrate historical files from the previous FTP server to the new distributed storage engine, and map the previous directory structure to the virtual file directory in step (2).

2. The FTP horizontal scaling method according to claim 1, characterized in that, Step (2) further includes: Uploading an employee's .exe file [employee.xlsx] from the OA system to the new FT-AS system and storing it by year and month. A. The root directory assigned to the FTP user is: / ftp / oa_user, and the file needs to be uploaded to the directory / ftp / oa_user / employees / 2022 / 05; B. Upload files: (1) Establish a TCP connection using a standard FTP client and FTP-AS; (2) Check if the directory exists. If it does not exist, create the directory: / ftp / oa_user / employees / 2022 / 05. Note the difference between this and traditional FTP: it does not create a real directory on the server disk where FTP-AS is located, but only registers several directory records in the database table. Check if the directories employees, 2022, and 05 exist in turn. If they do not exist, create three database directory records. (3) FTP-AS establishes a connection with the lower-level scalable storage service ESS, stores the employee.xlsx file uploaded by the FTP client in ESS, and ESS returns the fileId to FTP-AS. The purpose of this ID is to uniquely identify the location of the file in ESS. (4) FTP-AS combines the fileId returned by ESS and records it in the directory as the ess_path attribute of ftp_virtual_directory; (5) FTP-AS uses the standard FTP protocol to inform the FTP client that the file upload was successful; Download file ① The FTP client establishes a TCP connection with FTP-AS and sends a GET command with the path / ftp / oa_user / employees / 2022 / 05 / employee.xlsx as the parameter; ②When FTP-AS receives the get command, it finds the file from the virtual directory based on the path parameter and extracts the ess_path parameter; ③FTP-AS establishes a TCP connection with ESS, retrieves the file from ESS according to ess_path, and forwards it synchronously to the business client; Delete file ① The FTP client establishes a TCP connection with FTP-AS and sends the rm command with the path / ftp / oa_user / employees / 2022 / 05 / employee.xlsx as the parameter; ②FTP-AS; Upon receiving the rm command, it locates the file from the virtual directory based on the path parameters and extracts the ess_path parameter; ③FTP-AS establishes a TCP connection with ESS and deletes the file from ESS according to ess_path; ④ FTP-AS deletes the virtual directory record corresponding to the file; it then sends a notification to the business system client, confirming successful deletion. View files in the directory ① The FTP client establishes a TCP connection with FTP-AS and sends the ls command with the directory / ftp / oa_user / employees / 2022 / 05 as the parameter; ②When FTP-AS receives the ls command, it queries the database and retrieves the directory's UUID by comparing the directory parameters with the full_path. ③ By passing the UUID as the parent_uuid parameter, you can obtain all direct child files in this directory; ④ Feedback is sent to the business system.

3. The FTP horizontal scaling method according to claim 1, characterized in that, Step (3) further includes: using file-based object storage services as the storage unit as the ESS solution, specifically including: There's no need to worry about which server the file is stored on; the ESS server assigns a unique fileId to each client. The corresponding storage method is block-based, which divides the file into the smallest blocks supported by the operating system. This improves disk utilization and reduces disk fragmentation. Enterprises with high storage utilization requirements can choose the storage solution that best suits their needs. The storage model is divided into two main modules: Tracker and Storage. Multiple Storage units form a Group, which is a virtual management unit, unlike the Tracker and Storage which are specific programs. The management characteristics of a group are as follows: (3.1) Multiple Storages within the same Group are redundant with each other, forming a mutual backup relationship; (3.2) Two Storages form a Group. If backup is not required, a single Storage can form a Group. (3.3) The storage capacity of a Group is equal to the capacity of the smallest storage device in the same group, that is, Volume(Group) = Volume(Min(Storage1,Storage2,......,StorageN)); the disk space of Storage devices in the same group must be the same size; (3.4) The capacity of the entire cluster is equal to the sum of the capacities of the multiple Groups: Volume(Cluster) = Volume(Group1) + Volume(Group2) + Volume(GroupN); Later, to increase the space, simply add a new Group, i.e., the backup expansion portion.

4. The FTP horizontal scaling method according to claim 3, characterized in that, The storage mode includes the following control flow: A tracker maintains the network topology of the storage; guides multiple storage instances within the same group to directly synchronize data; and provides real-time feedback on the network health of the storage, ensuring that each file upload is allocated to a valid storage instance. (1) FTP-AS received the file uploaded by the business system; (2) FTP-AS requests the Tracker module under ESS to ask which group has the most free space and returns the group with the most free space to FTP-AS. For example, if Group003 has the highest free space ratio, then Group003 is returned to FTP-AS. (3) FTP-AS randomly selects a storage device from Group003, let's say it's A01. FTP-AS establishes a connection with A01 and uploads the file to A01. (4) At this point, the file upload operation on the business system side is complete, but ESS itself still needs to complete the next steps; (5) The memory A01 will synchronously report the received files to the tracker; (6) The tracker synchronizes this file record to other storage devices in the same group as A01, and the other storage devices begin to request file synchronization; (7) After all storage devices in the same group have been synchronized, the tracker records the status of the file as synchronized, and the design of the file storage module ESS is now complete.

5. The FTP horizontal scaling method according to claim 1, characterized in that, Step (4) further includes: Due to the fundamental changes in the underlying storage, in order for the business system to operate seamlessly, it is also necessary to migrate historical files. Migration can be divided into static migration and dynamic migration. (4.1) The static migration strategy is as follows: The original FTP server files are stored on the disk, requiring only a recursive algorithm to traverse the folders, starting from the top-level folder, to synchronize historical data to the new architecture; the traversal algorithm is as follows: (4.1.1) Pass in the top-level directory and record the top-level directory in the virtual directory table; (4.1.2) Obtain all files in the top-level directory and read them in a loop; (4.1.3) If it is a file, call the interface of the ESS module to store it under the ESS module, get the fileId assigned by the ESS module, and store it in the virtual directory table in combination with the parent directory; (4.1.4) If it is a folder, return to step (4.1.1) and perform recursive judgment; (4.1.5) The migration ends when the last file is stored in ESS and synchronized to the virtual directory; (4.2) The dynamic migration strategy is as follows: If the number of files is too large or too many, causing static migration to take a long time, and the business system cannot be interrupted, then dynamic migration is needed to integrate the migration process into FTP-AS; the process is as follows: (4.2.1) The business system requests to download a certain historical file; (4.2.2) FTP-AS then detects that the historical files have not yet been synchronized, so it acts as an FTP client, establishes a connection with the original FTP server, and caches the files locally. (4.2.3) Cache the files: a. Return the information to the business system; b. Create an asynchronous task, upload it to ESS, and store a file record in the virtual directory so that it can be downloaded from ESS on the next request.

Images