A parallel file transfer method, system, device and storage medium

Abstract

This application relates to a parallel file transfer method, system, device, and storage medium, applied in the field of file transfer technology. The method is based on the SMB and SCP protocols. The method includes: constructing a usable SMB connection pool according to the SMB protocol; constructing a usable SCP connection pool according to the SCP protocol; receiving SMB file groups using the SMB connection pools; filtering the SMB file groups according to preset conditions using the SMB protocol to obtain several files to be downloaded; constructing parallel file transfer instructions based on the several files to be downloaded; and using the SCP connection pool to transfer the several files to be downloaded in parallel according to the parallel file transfer instructions, and integrating the several files to be downloaded to obtain a target file group. The technical effect of this application is: improved file transfer efficiency.

Description

Technical Field

[0001] This application relates to the field of file transfer, and in particular to a parallel file transfer method, system, device, and storage medium. Background Technology

[0002] With the rapid development of computer, communication and information technology, remote file transfer technology has emerged and become the core means of realizing file sharing across servers.

[0003] Existing remote file transfer technologies mainly include rsync (an open-source, high-efficiency file synchronization tool), SCP, and SMB. However, these technologies suffer from the following problems:

[0004] Using rsync for transmission only allows for manual command execution or timed transmission, making dynamic scheduling impossible and resulting in low transmission efficiency. When using the SMB protocol to transmit a single large file, a single TCP connection is typically used, and its speed is limited by network bandwidth, latency, and the single-threaded processing capability of the server. Using the SCP protocol for transmission involves transmitting data through a single SCP process, which can only read and send files sequentially, leading to low transmission efficiency.

[0005] Therefore, these problems lead to low efficiency in large file transfers, small file batch transfers, and file transfers in complex network environments, while also increasing the difficulty of system management and maintenance. Summary of the Invention

[0006] To improve file transfer efficiency, this application provides a parallel file transfer method, system, device, and storage medium.

[0007] Firstly, this application provides the following technical solution:

[0008] Construct a usable SMB connection pool according to the SMB protocol, and construct a usable SCP connection pool according to the SCP protocol;

[0009] Receive SMB file groups using the SMB connection pool;

[0010] Using the SMB protocol, the SMB file group is filtered and selected according to preset conditions to obtain several files to be downloaded;

[0011] Construct parallel file transfer instructions based on several of the files to be downloaded;

[0012] According to the parallel file transfer instruction, several files to be downloaded are transferred in parallel using the SCP connection pool, and the files to be downloaded are integrated to obtain a target file group.

[0013] Through the above technical solutions, the purpose of this invention is to provide a method and system for transferring files by combining the SMB and SCP protocols. This allows access to remote paths as if they were local paths via the SMB protocol, while reducing the number of SSH connection channels. At the same time, it avoids the bottleneck of SMB downloads and achieves near-full-load download efficiency with SCP. This fully utilizes computer network bandwidth and disk I / O speed, thereby achieving efficient, stable, and resource-controllable parallel file downloads.

[0014] In one specific implementation scheme, constructing a usable SMB connection pool according to the SMB protocol includes:

[0015] An SMB connection pool is constructed according to the SMB protocol, and an SMB session is created using the SMB connection pool.

[0016] Determine whether the SMB session is in a connected state;

[0017] If the SMB session is in a connected state, then the SMB connection pool is determined to be ready for use.

[0018] Conversely, if the connection pool is not used, an SMB session is recreated and the number of reconnections is recorded.

[0019] Determine whether the SMB session can remain connected before the number of reconnections reaches a preset maximum value;

[0020] If the SMB session is in a connected state before the number of reconnections reaches a preset maximum value, then the SMB connection pool is determined to be usable.

[0021] Conversely, if the SMB connection pool construction fails, an alarm message will be issued.

[0022] In one specific feasible implementation, constructing a usable SCP connection pool according to the SCP protocol includes:

[0023] After constructing the SCP connection pool according to the SCP protocol, user information verification is performed.

[0024] Determine whether the user information verification passed;

[0025] If the user information verification passes, the SCP connection pool is deemed ready for use.

[0026] Conversely, if the user information is not verified, the number of verifications will be recorded.

[0027] Determine whether the user information verification can be completed before the number of verification attempts reaches a preset maximum value;

[0028] If the user information verification is completed before the number of verifications reaches the preset maximum value, then the SCP connection pool is determined to be ready for use.

[0029] Conversely, if the SCP connection pool construction fails, an alarm message will be issued.

[0030] By using the above technical solutions, a shared SMB / SCP session is created and maintained to reduce the time and performance consumption of each connection session. This significantly reduces the time and resource consumption required to establish a new connection for each file transfer. This method improves system response speed and overall efficiency by reusing established connections. Especially in scenarios requiring frequent file transfers, this method can greatly reduce network latency and system overhead.

[0031] In a specific feasible implementation, the step of using the SMB protocol to filter and select the SMB file group according to preset conditions to obtain several files to be downloaded includes:

[0032] The directory list and file list of the SMB filegroup are read using the SMB protocol to obtain several SMB files;

[0033] Parse several SMB files to obtain file feature information of the SMB files;

[0034] Based on preset conditions and the file feature information, the SMB file group is filtered to obtain several files to be downloaded.

[0035] The above technical solution involves using the SMB protocol to filter and select SMB file groups according to preset conditions to obtain several files to be downloaded, including:

[0036] The directory list and file list of the SMB filegroup are read using the SMB protocol to obtain several SMB files;

[0037] Parse several SMB files to obtain file feature information of the SMB files;

[0038] Based on preset conditions and the file feature information, the SMB file group is filtered to obtain several files to be downloaded.

[0039] In a specific feasible implementation, the step of using the SMB protocol to filter and select the SMB file group according to preset conditions to obtain several files to be downloaded includes:

[0040] The directory list and file list of the SMB filegroup are read using the SMB protocol to obtain several SMB files;

[0041] Parse several SMB files to obtain file feature information of the SMB files;

[0042] Based on preset conditions and the file feature information, the SMB file group is filtered to obtain several files to be downloaded.

[0043] By employing the above technical solution, using system-level parallel tasks for file downloads can fully utilize the system's multi-core processing capabilities, significantly improving the parallelism and efficiency of file transfer. This method can not only handle the download of multiple files simultaneously but also perform parallel downloading of single large files in chunks. Compared to traditional single-threaded download methods, this parallel download method can better utilize network bandwidth and reduce overall download time, especially when processing a large number of small files or extremely large files, where the effect is more pronounced.

[0044] In one specific implementation, after the parallel transfer of several files to be downloaded using an SCP connection pool according to the parallel task, the method further includes:

[0045] If any of the files to be downloaded fail to download, then the files that failed to download will be retransmitted in parallel using the SCP connection pool, and the number of retransmissions will be recorded.

[0046] Determine whether a number of the failed download files can be successfully transmitted before the number of retransmissions reaches a preset maximum value;

[0047] If several of the downloaded files that failed to download are successfully transmitted before the number of retransmissions reaches a preset maximum value, then the parallel transmission is determined to be complete.

[0048] Conversely, if the parallel transmission fails, an alarm message will be issued.

[0049] The above technical solution can dynamically adjust the transmission strategy according to the file size and network conditions, avoiding the limitations of a single connection of SMB and SCP. At the same time, it can achieve near-full-load download efficiency through SCP, making full use of computer network bandwidth and disk I / O speed.

[0050] By sending instant messages with task status and error reports, system administrators can monitor the execution of file transfer tasks in real time, receiving immediate notifications, especially when errors occur. This method significantly improves problem response speed and reduces the duration of system downtime. Compared to traditional log viewing methods, instant message pushes are more proactive and timely, helping administrators to identify and resolve issues faster, thereby improving the overall reliability and efficiency of the system.

[0051] In one specific implementation scheme, the integration of several files to be downloaded to obtain a target file group includes:

[0052] Retrieve user configuration information;

[0053] Generate file directory partitioning information for several files to be downloaded based on user configuration information;

[0054] Based on the file directory partitioning information, several files to be downloaded are integrated to obtain a target file group.

[0055] The above technical solution receives data files from servers of various download connections and re-forms a new file group in the target directory according to the file order, source, file name, task group, and other rules in the user configuration information, which facilitates subsequent file processing after file transfer.

[0056] Secondly, this application provides a parallel file transfer system, which is based on the SMB and SCP protocols and adopts the following technical solution: The system includes:

[0057] The connection pool construction module is used to construct a usable SMB connection pool according to the SMB protocol and a usable SCP connection pool according to the SCP protocol.

[0058] The file receiving module is used to receive SMB file groups using the SMB connection pool;

[0059] The file filtering module is used to filter the SMB file group according to preset conditions using the SMB protocol to obtain a number of files to be downloaded.

[0060] The file transfer instruction generation module is used to construct parallel file transfer instructions based on several files to be downloaded.

[0061] The file transfer module is used to transfer several files to be downloaded in parallel using the SCP connection pool according to the parallel file transfer instruction, and to integrate the several files to be downloaded to obtain a target file group.

[0062] Thirdly, this application provides a computer device that adopts the following technical solution: it includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed as described above in a parallel file transfer method.

[0063] Fourthly, this application provides a computer-readable storage medium that stores a computer program capable of being loaded by a processor and executed using the aforementioned parallel file transfer method.

[0064] In summary, this application has the following beneficial technical effects:

[0065] (1) The purpose of this invention is to provide a method and system for transferring files by combining the SMB protocol and the SCP protocol. This allows access to remote paths as if they were local paths via the SMB protocol, while reducing the number of SSH connection channels. At the same time, it avoids the bottleneck of SMB downloads and achieves near-full-load download efficiency with SCP. This fully utilizes computer network bandwidth and disk I / O rate, thereby achieving efficient, stable and resource-controllable parallel file downloads.

[0066] (2) By creating and maintaining a shared SMB / SCP session, the time and performance consumption of each connection session can be reduced. This can significantly reduce the time and resource consumption required to establish a new connection for each file transfer. This method improves the system's response speed and overall efficiency by reusing established connections. Especially in scenarios where frequent file transfers are required, this method can greatly reduce network latency and system overhead.

[0067] (3) Using the SMB protocol to quickly read file directories can significantly improve the speed of file list retrieval, especially when dealing with a large number of files or complex directory structures. Compared with protocols such as SCP, the SMB protocol has a clear advantage in reading directory structures. In addition, filtering based on user needs can reduce unnecessary data transmission and improve overall transmission efficiency. This method enables the system to identify and locate the files that need to be transmitted more quickly, thereby optimizing the entire file transmission process.

[0068] (4) Using system-level parallel tasks for file downloads can fully utilize the system's multi-core processing capabilities, significantly improving the parallelism and efficiency of file transfer. This method can not only handle the download of multiple files simultaneously, but also perform parallel download of single large files in chunks. Compared to the traditional single-threaded download method, this parallel download method can better utilize network bandwidth and reduce the overall download time, especially when processing a large number of small files or very large files, the effect is more obvious.

[0069] (5) The transmission strategy is dynamically adjusted according to the file size and network conditions, avoiding the limitations of a single connection of SMB and SCP. At the same time, the download efficiency is close to full load through SCP, which can make full use of computer network bandwidth and disk I / O rate.

[0070] (6) By sending instant messages to push task status and error reports, system administrators can understand the execution status of file transfer tasks in real time, especially receiving immediate notifications when errors occur. This method greatly improves the speed of problem response and reduces the duration of system failures. Compared with traditional log viewing methods, instant message push is more proactive and timely, helping administrators to discover and resolve problems more quickly, thereby improving the reliability and efficiency of the entire system. Attached Figure Description

[0071] Figure 1 This is a flowchart of a parallel file transfer method according to an embodiment of this application.

[0072] Figure 2 This is a file transfer framework diagram.

[0073] Figure 3 This is a structural block diagram of a parallel file transfer method according to an embodiment of this application.

[0074] Reference numerals: 301, Connection pool construction module; 302, File receiving module; 303, File filtering module; 304, File transfer instruction generation module; 305, File transfer module. Detailed Implementation

[0075] The following is in conjunction with the appendix Figures 1-3 This application will be described in further detail.

[0076] This application discloses a parallel file transfer method to improve file transfer efficiency.

[0077] With the rapid development of computer, communication and information technology, remote file transfer technology has emerged and become the core means of realizing file sharing across servers.

[0078] Existing remote file transfer technologies mainly include rsync (an open-source, high-efficiency file synchronization tool), SCP, and SMB. However, these technologies suffer from the following problems:

[0079] The existing rsync can perform bidirectional file synchronization, but it can only be done by manually executing commands or by scheduling, and cannot achieve dynamic scheduling. In actual tests, its transmission efficiency cannot reach that of SCP or SMB transmission.

[0080] When transferring a single large file, the standard SMB protocol typically uses a single TCP connection. Its speed is limited by network bandwidth, latency, and the single-threaded processing capacity of the server. It lacks a unified, intelligent solution for dynamically managing parallel connections in an SMB environment. Furthermore, SMB's transmission efficiency is lower than that of the SCP method, and it is even less efficient for transferring small files.

[0081] Transferring files using the SCP protocol essentially involves transferring data over an SSH connection via a single SCP process (typically a remote scp command). This process reads and sends files sequentially, resulting in low transfer efficiency. The SCP protocol is simple and lacks native chunking support. Unlike SMB, the SCP protocol doesn't have primitives like "byte range read"; it's designed to transfer entire files or file streams "in one go."

[0082] In addition, existing remote file transfer technologies suffer from the problem of delayed task failures, meaning that task failures cannot be detected in a timely manner, requiring manual inspection of the task status or review of logs for error judgment and resolution.

[0083] Therefore, this application proposes a parallel file transfer method to improve file transfer efficiency.

[0084] like Figure 1 As shown, the method includes:

[0085] S10: Construct a usable SMB connection pool according to the SMB protocol, and construct a usable SCP connection pool according to the SCP protocol.

[0086] Specifically, such as Figure 2 The diagram shows the file transfer framework. A usable SMB connection pool is built according to the SMB protocol, and a usable SCP connection pool is built according to the SCP protocol. A shared SMB and SCP session management pool is created and maintained to reduce the time and performance consumption of each connection session.

[0087] In addition, it is necessary to install and configure the dependent environments such as parallel, SMB, and SCP in advance for the system.

[0088] S20, Receive SMB file groups using the SMB connection pool.

[0089] Specifically, it receives SMB filegroups, which include several SMB files, such as... Figure 2 The file shown is received, processed, and transmitted via an SMB connection pool.

[0090] S30 uses the SMB protocol to filter and select SMB file groups according to preset conditions to obtain several files to be downloaded.

[0091] Specifically, based on several conditions provided by the user, such as filename matching rules, file creation time, file size, and other characteristic information, the shared directory files to be accessed are filtered and selected to obtain several files to be downloaded.

[0092] S40 constructs parallel file transfer instructions based on several files to be downloaded.

[0093] Specifically, the download command is constructed by combining the above-mentioned files to be downloaded; for example, `parallel -j 5scp {} user@host: / path ::: files*.txt` means that a maximum of 5 parallel tasks will be executed.

[0094] S50: Based on the parallel file transfer instruction, use the SCP connection pool to transfer several files to be downloaded in parallel, and integrate the several files to be downloaded to obtain the target file group.

[0095] Specifically, according to the parallel file transfer instruction, several files to be downloaded are transferred in parallel using the SCP connection pool. Based on the file order, source, file name, task group, and other rules, a new target file group is formed in the target directory.

[0096] The purpose of this invention is to provide a method and system for transferring files by combining the SMB and SCP protocols. This allows access to remote paths as if they were local paths via the SMB protocol, while reducing the number of SSH connection channels. At the same time, it avoids the bottleneck of SMB downloads and achieves near-full-load download efficiency with SCP. This fully utilizes computer network bandwidth and disk I / O speed, thereby achieving efficient, stable, and resource-controllable parallel file downloads.

[0097] In one embodiment, to improve file transfer efficiency, the step of building a usable SMB connection pool according to the SMB protocol can be specifically performed as follows:

[0098] First, an SMB connection pool is constructed according to the SMB protocol, and an SMB session is created using the SMB connection pool. Specifically, the SMB connection pool refers to a pre-established set of multiple reusable network connection resources, which is implemented by using multi-threading technology to maintain the connection instance queue, in order to reduce the time overhead of repeatedly establishing connections. An SMB session refers to a communication channel between the client and the server established based on the SMB protocol, which is created by negotiating the protocol version and the identity authentication process, and serves as the basic carrier for file transfer operations.

[0099] Then, it is determined whether the SMB session is in a connected state. If the SMB session is in a connected state, it is determined that the SMB connection pool can be put into use. Otherwise, the SMB session is recreated using the SMB connection pool, and the number of reconnections is recorded. Specifically, the connection status determination refers to detecting whether the session has the ability to transmit data. By sending test messages and verifying the response results, it is determined whether the connection failure is due to network interruption or service abnormality. When the SMB session is not in a connected state, a reconnection will be performed.

[0100] Next, it is determined whether the SMB session can be in a connected state before the preset maximum number of reconnections is reached. If the SMB session is in a connected state before the preset maximum number of reconnections is reached, the SMB connection pool is determined to be usable; otherwise, the SMB connection pool construction is determined to have failed and an alarm message is issued. Specifically, when the first session establishment fails, the system automatically enters the retry process, monitors in real time whether the number of retryes exceeds a preset threshold, and dynamically switches the processing strategy: continuously attempts to restore the connection within the threshold range, and terminates the retry and triggers an alarm if the threshold is exceeded. This mechanism is implemented through a dual verification structure: the initial connection verification is used to handle normal scenarios, the retry mechanism deals with temporary network fluctuations, and the threshold control prevents excessive resource consumption.

[0101] Compared to existing technologies, traditional SMB connection establishment methods only perform a single connection attempt, making them prone to connection failure due to momentary faults in unstable network environments. This solution introduces a reconnection mechanism with a limited number of attempts, effectively addressing short-term network jitter and avoiding the waste of system resources caused by infinite retries through preset threshold control. Simultaneously, by employing dual judgment logic based on connection status and retry count, it ensures rapid response under normal conditions while providing a controllable recovery window for abnormal scenarios, significantly improving the robustness of connection pool construction.

[0102] In one embodiment, to improve file transfer efficiency, the step of constructing a usable SCP connection pool according to the SCP protocol can be specifically performed as follows:

[0103] First, after constructing an SCP connection pool according to the SCP protocol, user information is verified. Specifically, SSH key pairs or password authentication are used to verify user information. Access permissions are confirmed by verifying whether the identity credentials provided by the user match the authorization information stored on the server.

[0104] Then, it is determined whether the user information verification is successful. If the user information verification is successful, the SCP connection pool is determined to be ready for use. Otherwise, the user information verification is performed again, and the number of verifications is recorded. Specifically, when constructing the SCP connection pool, the user identity verification process is executed first. If the first verification is successful, the connection pool initialization is completed immediately. When the first verification fails, a limited number of retry mechanisms are automatically triggered.

[0105] Next, it is determined whether the user information verification can be completed before the number of verification attempts reaches the preset maximum value. If the user information verification is completed before the number of verification attempts reaches the preset maximum value, the SCP connection pool is determined to be usable. Otherwise, the SCP connection pool construction is determined to have failed and an alarm message is issued. Specifically, during the retry process, the number of verification attempts is continuously monitored to see if it exceeds the preset threshold. If the verification is completed before reaching the threshold, the connection pool is determined to be usable. If the number of retry attempts is exhausted and the verification is still not passed, the connection pool construction process is terminated and an alarm mechanism is triggered.

[0106] Compared to existing technologies, traditional SCP connection establishment terminates the connection directly if the initial user authentication fails, failing to handle occasional failures caused by network jitter or temporary authentication service anomalies. This solution, however, introduces a controllable retry mechanism to automatically resume the verification process within preset security boundaries. This avoids connection pool unavailability due to a single verification failure and prevents malicious brute-force attacks by setting a verification count limit. Existing technologies lack effective monitoring and resource release mechanisms for verification failures, easily leading to wasted system resources. This solution achieves precise resource management through verification count threshold control and alarm linkage mechanisms.

[0107] Creating and maintaining a shared SMB / SCP session reduces the time and performance overhead of each connection session. This significantly reduces the time and resource consumption required to establish a new connection for each file transfer. By reusing established connections, this approach improves system responsiveness and overall efficiency. Especially in scenarios requiring frequent file transfers, this method can greatly reduce network latency and system overhead.

[0108] In one embodiment, to improve file transfer efficiency, the step of filtering and selecting SMB file groups according to preset conditions using the SMB protocol to obtain several files to be downloaded can be specifically performed as follows:

[0109] First, the directory list and file list of the SMB file group are read using the SMB protocol to obtain several SMB files. Specifically, the directory traversal function of the SMB protocol is used to read the file list and directory list to obtain several SMB files.

[0110] Then, several SMB files are parsed to obtain file feature information of several SMB files. Specifically, file feature information is extracted by parsing the file attribute fields returned by the SMB protocol through the metadata coordinator. The file feature information mainly includes file creation time, modification time, file size and extension, etc.

[0111] Next, the SMB file group is filtered based on preset conditions and file feature information to obtain several files to be downloaded. Specifically, the user-defined dynamic filtering rules are the preset conditions. The SMB file group is filtered based on the preset filtering rules, such as only keeping files with a specific extension and a modification time within a specified range. The set of files to be downloaded is updated in real time during the filtering process to obtain several files to be downloaded.

[0112] Utilizing the SMB protocol for fast file directory reading can significantly improve the speed of file list retrieval, especially when handling large numbers of files or complex directory structures. Compared to protocols like SCP, SMB has a clear advantage in reading directory structures. Furthermore, filtering based on user needs can reduce unnecessary data transmission and improve overall transmission efficiency. This method enables the system to more quickly identify and locate the files that need to be transferred, thereby optimizing the entire file transfer process.

[0113] In one embodiment, to improve file transfer efficiency, the step of using an SCP connection pool to transfer several files to be downloaded in parallel according to a parallel file transfer instruction can be specifically executed as follows:

[0114] Parse the parallel file transfer instructions to obtain the files to be transferred in parallel and the number of files to be transferred in parallel; generate and invoke a parallel task based on the files to be transferred in parallel and the number of files to be transferred in parallel using the SCP connection pool based on the parallel task.

[0115] Specifically, upon receiving a parallel file transfer instruction, the instruction parsing module first extracts the file path list and concurrency parameter. The concurrency parameter is dynamically adjusted based on system resource availability, for example, set to 80% of the current idle thread count. Then, the task scheduling module generates a parallel task script containing file fragmentation information and thread allocation rules based on the parsing results. This script allocates an independent connection channel to each transfer thread by calling a user-verified connection instance from the SCP connection pool. During execution, each thread obtains an available connection from the connection pool and transfers the target file in segments according to the fragmentation rules.

[0116] Using system-level parallel tasks for file downloads fully leverages the system's multi-core processing power, significantly improving the parallelism and efficiency of file transfers. This method can not only handle the download of multiple files simultaneously but also perform parallel downloading of single large files in chunks. Compared to traditional single-threaded download methods, this parallel download method makes better use of network bandwidth and reduces overall download time, especially when processing a large number of small files or extremely large files.

[0117] In one embodiment, to improve file transfer efficiency, after transferring several files to be downloaded in parallel using the SCP connection pool according to the parallel task, the following steps can also be performed:

[0118] First, if there are files that failed to download among the files to be downloaded, these files will be retransmitted in parallel using the SCP connection pool, and the number of retransmissions will be recorded. Specifically, during the download process, there may be files that fail to download due to network fluctuations, abnormal permissions, or file corruption. If there are no files that fail to download, it means that all files have been transmitted. If there are files that fail to download, they need to be added back to the task list for retransmission.

[0119] Then, it determines whether several failed download files can be successfully transmitted before the preset maximum number of retransmissions is reached. If several failed download files are successfully transmitted before the preset maximum number of retransmissions is reached, the parallel transmission is considered complete; otherwise, the parallel transmission is considered a failure and an alarm message is issued. Specifically, if a file is successfully transmitted within the preset number of retransmissions, the file status is marked as complete; if all failed files are successfully transmitted, the overall process ends. If any file still fails after reaching the maximum number of retryes, the transmission process is terminated and the alarm module is triggered. Timely message notifications are sent through various user-configured methods, such as Lark and WeChat Work, pushing the specific file failure information and reasons to the responsible person.

[0120] The transmission strategy can be dynamically adjusted according to file size and network conditions, avoiding the limitations of a single connection of SMB and SCP. At the same time, SCP can achieve near-full-load download efficiency, making full use of computer network bandwidth and disk I / O speed.

[0121] By sending instant messages with task status and error reports, system administrators can monitor the execution of file transfer tasks in real time, receiving immediate notifications, especially when errors occur. This method significantly improves problem response speed and reduces the duration of system downtime. Compared to traditional log viewing methods, instant message pushes are more proactive and timely, helping administrators to identify and resolve issues faster, thereby improving the overall reliability and efficiency of the system.

[0122] In one embodiment, to improve file transfer efficiency, the step of merging several files to be downloaded into a target file group can be specifically performed as follows:

[0123] Retrieve user configuration information; generate file directory partitioning information for several files to be downloaded based on user configuration information; integrate several files to be downloaded based on file directory partitioning information to obtain the target file group.

[0124] After file transfer is complete, the system automatically reads the user-defined configuration file, which contains the mapping between file types and target directories. The attribute information of each file to be downloaded is extracted and compared with the matching rules in the configuration file. For example, files with the ".log" extension are categorized into the " / logs" directory, and filenames containing the characters "2023" are stored in the " / archive / 2023" path. The system dynamically generates a directory tree structure based on the matching results, automatically creating the target directory if it does not exist. Finally, a batch command moves all files to the corresponding terminal locations, ensuring that the file storage structure is completely consistent with the user-preset directory partitioning.

[0125] This application implements intelligent storage management of transferred files, effectively avoiding retrieval difficulties caused by scattered file storage. The file directory structure can be dynamically generated according to user needs, eliminating the manual sorting process and reducing the risk of file misalignment due to manual operation errors. The configurability of file classification rules enables the system to adapt to the storage needs of different business scenarios, improving the scalability and reusability of the file management system.

[0126] like Figure 2 As shown, after obtaining the target file group that has been transferred, it is persisted and other business logic is called to process the target file group.

[0127] Based on the above method, this application also discloses a parallel file transfer system. For example... Figure 3 The system includes the following modules:

[0128] Connection pool construction module 301 is used to construct a usable SMB connection pool according to the SMB protocol and a usable SCP connection pool according to the SCP protocol.

[0129] File receiving module 302 is used to receive SMB file groups using an SMB connection pool;

[0130] The file filtering module 303 is used to filter and select SMB file groups according to preset conditions using the SMB protocol to obtain several files to be downloaded.

[0131] File transfer instruction generation module 304 is used to construct parallel file transfer instructions based on several files to be downloaded;

[0132] The file transfer module 305 is used to transfer several files to be downloaded in parallel using the SCP connection pool according to the parallel file transfer instruction, and to integrate the several files to be downloaded to obtain the target file group.

[0133] In one embodiment, the connection pool construction module 301 is specifically used to construct an SMB connection pool according to the SMB protocol and create an SMB session using the SMB connection pool.

[0134] Determine if the SMB session is in a connected state; if the SMB session is in a connected state, the SMB connection pool is deemed usable; otherwise, recreate the SMB session using the SMB connection pool and record the number of reconnections; determine if the SMB session can remain connected before the number of reconnections reaches a preset maximum; if the SMB session remains connected before the number of reconnections reaches the preset maximum, the SMB connection pool is deemed usable; otherwise, the SMB connection pool construction is deemed to have failed and an alarm message is issued.

[0135] In one embodiment, the connection pool construction module 301 is specifically used to construct an SCP connection pool according to the SCP protocol and then perform user information verification; determine whether the user information verification passes; if the user information verification passes, the SCP connection pool is determined to be usable; otherwise, the user information verification is performed again, and the number of verifications is recorded; determine whether the user information verification can be completed before the number of verifications reaches a preset maximum value; if the user information verification is completed before the number of verifications reaches the preset maximum value, the SCP connection pool is determined to be usable; otherwise, the SCP connection pool construction is determined to have failed and an alarm message is issued.

[0136] In one embodiment, the file filtering module 303 is specifically used to read the directory list and file list of the SMB file group using the SMB protocol to obtain several SMB files; parse the several SMB files to obtain the file feature information of the several SMB files; and perform filtering processing on the SMB file group according to preset conditions and file feature information to obtain several files to be downloaded.

[0137] In one embodiment, the file transfer module 305 is specifically used to parse the parallel file transfer instruction to obtain the parallel transfer files and the number of parallel transfers; generate and invoke a parallel task based on the parallel transfer files and the number of parallel transfers; and use the SCP connection pool to transfer several files to be downloaded in parallel according to the parallel task.

[0138] In one embodiment, the file transfer module 305 is specifically used to: if there are files that have failed to download among the files to be downloaded, then retransmit the files that have failed to download in parallel using the SCP connection pool and record the number of retransmissions; determine whether the files that have failed to download can be successfully transmitted before the number of retransmissions reaches a preset maximum value; if the files that have failed to download can be successfully transmitted before the number of retransmissions reaches the preset maximum value, then the parallel transmission is determined to be complete; otherwise, the parallel transmission is determined to have failed and an alarm message is issued.

[0139] In one embodiment, the file transfer module 305 is specifically used to call user configuration information; generate file directory partitioning information for several files to be downloaded based on the user configuration information; and integrate several files to be downloaded based on the file directory partitioning information to obtain a target file group.

[0140] This application also discloses a computer device.

[0141] Specifically, the computer device includes a memory and a processor, and the memory stores a computer program that can be loaded by the processor and executed using the aforementioned parallel file transfer method.

[0142] This application also discloses a computer-readable storage medium.

[0143] Specifically, the computer-readable storage medium stores a computer program that can be loaded by a processor and executed as described above in a parallel file transfer method. The computer-readable storage medium includes, for example, various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0144] This specific embodiment is merely an explanation of the present invention and is not intended to limit the invention. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they are within the scope of the claims of the present invention.

Claims

1. A parallel file transfer method, characterized in that, The method is based on the SMB and SCP protocols, and includes: Construct a usable SMB connection pool according to the SMB protocol, and construct a usable SCP connection pool according to the SCP protocol; Receive SMB file groups using the SMB connection pool; Using the SMB protocol, the SMB file group is filtered and selected according to preset conditions to obtain several files to be downloaded; Construct parallel file transfer instructions based on several of the files to be downloaded; According to the parallel file transfer instruction, several files to be downloaded are transferred in parallel using the SCP connection pool, and the files to be downloaded are integrated to obtain a target file group.

2. The method according to claim 1, characterized in that, The step of constructing a usable SMB connection pool according to the SMB protocol includes: An SMB connection pool is constructed according to the SMB protocol, and an SMB session is created using the SMB connection pool. Determine whether the SMB session is in a connected state; If the SMB session is in a connected state, then the SMB connection pool is determined to be ready for use. Conversely, if the connection pool is not used, an SMB session is recreated and the number of reconnections is recorded. Determine whether the SMB session can remain connected before the number of reconnections reaches a preset maximum value; If the SMB session is in a connected state before the number of reconnections reaches a preset maximum value, then the SMB connection pool is determined to be usable. Conversely, if the SMB connection pool construction fails, an alarm message will be issued.

3. The method according to claim 1, characterized in that, The construction of a usable SCP connection pool according to the SCP protocol includes: After constructing the SCP connection pool according to the SCP protocol, user information verification is performed. Determine whether the user information verification passed; If the user information verification passes, the SCP connection pool is deemed ready for use. Conversely, if the user information is not verified, the number of verifications will be recorded. Determine whether the user information verification can be completed before the number of verification attempts reaches a preset maximum value; If the user information verification is completed before the number of verifications reaches the preset maximum value, then the SCP connection pool is determined to be ready for use. Conversely, if the SCP connection pool construction fails, an alarm message will be issued.

4. The method according to claim 1, characterized in that, The process of filtering and selecting the SMB file group according to preset conditions using the SMB protocol to obtain several files to be downloaded includes: The directory list and file list of the SMB filegroup are read using the SMB protocol to obtain several SMB files; Parse several SMB files to obtain file feature information of the SMB files; Based on preset conditions and the file feature information, the SMB file group is filtered to obtain several files to be downloaded.

5. The method according to claim 1, characterized in that, The parallel transfer of several files to be downloaded using the SCP connection pool according to the parallel file transfer instruction includes: The parallel file transfer instructions are parsed to obtain the number of files to be transferred in parallel and the number of files to be transferred in parallel. Generate and invoke parallel tasks based on the parallel transfer files and the number of parallel transfers; According to the parallel task, several files to be downloaded are transmitted in parallel using an SCP connection pool.

6. The method according to claim 5, characterized in that, After the parallel transfer of several files to be downloaded using the SCP connection pool according to the parallel task, the method further includes: If any of the files to be downloaded fail to download, then the files that failed to download will be retransmitted in parallel using the SCP connection pool, and the number of retransmissions will be recorded. Determine whether a number of the failed download files can be successfully transmitted before the number of retransmissions reaches a preset maximum value; If several of the downloaded files that failed to download are successfully transmitted before the number of retransmissions reaches a preset maximum value, then the parallel transmission is determined to be complete. Conversely, if the parallel transmission fails, an alarm message will be issued.

7. The method according to claim 1, characterized in that, The process of integrating several files to be downloaded to obtain a target file group includes: Retrieve user configuration information; Generate file directory partitioning information for several files to be downloaded based on user configuration information; Based on the file directory partitioning information, several files to be downloaded are integrated to obtain a target file group.

8. A parallel file transfer system, characterized in that, The system is based on the SMB and SCP protocols, and includes: The connection pool construction module (301) is used to construct a usable SMB connection pool according to the SMB protocol and to construct a usable SCP connection pool according to the SCP protocol. The file receiving module (302) is used to receive SMB file groups using the SMB connection pool; The file filtering module (303) is used to filter the SMB file group according to preset conditions using the SMB protocol to obtain several files to be downloaded. The file transfer instruction generation module (304) is used to construct parallel file transfer instructions based on several files to be downloaded; The file transfer module (305) is used to transfer several files to be downloaded in parallel using the SCP connection pool according to the parallel file transfer instruction, and to integrate the several files to be downloaded to obtain a target file group.

9. A computer device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer program is stored that can be loaded by a processor and executed according to any one of claims 1 to 7.

Images