A file distribution transmission method based on sftp network protocol

Abstract

This invention discloses a file distribution and transmission method based on the SFTP network protocol, belonging to the field of network file transmission technology. The method is executed on the file distribution scheduling node: extracting file features and classifying them into static HTML or attachment types; sensing the status of each target server node, generating a distribution transmission plan using a scheduling strategy network trained with a near-end strategy optimization algorithm, and dynamically adjusting the SFTP connection pool capacity; fragmenting and compressing attachment files before parallel transmission through a multi-channel pipeline, prioritizing the transmission of lightly compressed static HTML files, supporting fragment-level breakpoint resumption and incremental synchronization of file fingerprints; implementing fault detection, task migration, and exponential backoff retries through heartbeat monitoring, and continuously optimizing the scheduling strategy based on performance baselines. This invention improves the efficiency, reliability, and resource utilization of file transmission in multi-node scenarios.

Description

Technical Field

[0001] This invention relates to the field of network file transfer technology, and in particular to a file distribution transfer method based on the SFTP network protocol. Background Technology

[0002] Content Management Systems (CMS) generate two core types of data during daily operation: static HTML files and attachment files (images, documents, audio, video, etc.). The former are numerous and have high real-time requirements, while the latter are large in size but relatively few in number. In actual deployment, these files need to be distributed and transmitted to multiple target server nodes (such as origin nodes, CDN edge nodes, backup nodes, etc.) to support high-concurrency access, disaster recovery, and edge acceleration requirements.

[0003] SFTP (SSH File Transfer Protocol), as the de facto standard for enterprise-level file transfer, boasts advantages such as encrypted transmission, authentication, and rich operational features. However, existing SFTP-based transfer solutions have the following shortcomings when dealing with large-scale distributed file transfers in CMS: First, the transmission strategy is simplistic, failing to distinguish the essential differences between static HTML files and attachment files in terms of file size, quantity, and real-time requirements, resulting in low transmission efficiency; second, multi-node scheduling is crude, often employing serial or simple parallel methods to push data node by node, lacking global awareness and intelligent scheduling of network bandwidth and node load, leading to low resource utilization; third, fault tolerance mechanisms are weak, lacking refined breakpoint resumption and intelligent error recovery capabilities, often requiring retransmission from the beginning after a transmission interruption.

[0004] Therefore, it is necessary to design a file distribution transmission method based on the SFTP network protocol, which can realize efficient and reliable distributed transmission of CMS files among multiple nodes through differentiated transmission strategies, intelligent scheduling decisions and a sound fault tolerance and recovery mechanism. Summary of the Invention

[0005] This invention proposes a file distribution transmission method based on the SFTP network protocol, aiming to overcome the shortcomings of existing CMS file distribution transmission schemes, such as insufficient differentiated processing capabilities, lack of scheduling and coordination mechanisms, and weak error recovery capabilities. This invention achieves a synergistic improvement in file transmission efficiency, reliability, and resource utilization in large-scale, multi-node scenarios by constructing an end-to-end execution process of "file feature perception - intelligent distribution scheduling - dynamic fault-tolerant recovery."

[0006] To achieve the above objectives, this invention provides a file distribution and transfer method based on the SFTP network protocol. This method is executed on a file distribution and scheduling node, which is located in a distributed transmission system containing multiple target server nodes. The method includes the following steps:

[0007] Step A, File Feature Perception and Classification: Receive the set of files to be transmitted from the content management system, extract feature vectors such as file size, file extension, MIME type, and file entropy value for each file, and classify the files into static HTML files or attachment files based on the feature vectors, providing a basis for decision-making for subsequent differentiated transmission strategies.

[0008] Step B, Intelligent Distributed Scheduling: This includes a multi-node state awareness sub-step, a reinforcement learning-based scheduling decision sub-step, and an adaptive concurrency control sub-step.

[0009] Multi-node state awareness sub-step: Real-time acquisition of network bandwidth, storage capacity, CPU load and SFTP connection count of each target server node through heartbeat mechanism, and construction of node state vector;

[0010] The scheduling decision sub-step based on reinforcement learning is as follows: taking the feature vector of each file and the state vector of the node as input, and taking the transmission completion time, node load balancing, bandwidth utilization and transmission success rate as optimization objectives, a near-end strategy optimization algorithm is used to pre-train the scheduling strategy network with historical data. In the online stage, the optimal target server node selection and transmission priority are output according to the current system state to generate a distributed transmission plan.

[0011] Adaptive concurrency control sub-step: Dynamically adjust the SFTP connection pool capacity based on network conditions and target server node load.

[0012] Step C, Efficient Transfer Execution: Establish an SFTP session channel according to the distributed transfer plan and execute file transfer. This step includes:

[0013] File fragmentation and differential compression sub-steps: File fragmentation and high compression ratio compression are performed on attachment type files, and lightweight compression or direct transmission is performed on static HTML type files;

[0014] Multi-channel pipeline transmission sub-steps: Using the producer-consumer model, multiple files on the same target server node are transmitted in parallel using a pipeline method through a bounded buffer, and multiple file fragments of the same file are uploaded in parallel using multiple channels;

[0015] Resume interrupted download and incremental synchronization sub-steps: record the transmission status of file fragments, resume from the breakpoint when interrupted, and use file fingerprint comparison to achieve incremental synchronization.

[0016] Step D, Dynamic Fault Tolerance and Recovery: Monitor the transmission status of each SFTP session channel in real time, and automatically trigger task migration or retry mechanisms when a target server node failure or transmission anomaly is detected to ensure the eventual consistency of distributed transmission tasks.

[0017] Step E, Transmission Monitoring and Adaptive Optimization: Collect operational metrics from the above steps and establish a transmission performance baseline. Based on the confidence level of the network output of the scheduling strategy, dynamically adjust the balance between exploration and utilization to achieve continuous evolution of the transmission strategy. Encapsulate the transmission completion status of each target server node into a standardized transmission report for querying and tracing by the content management system.

[0018] Furthermore, the specific process of generating a distributed transmission plan based on the reinforcement learning-based scheduling decision sub-step is as follows: constructing a state space containing file feature vectors and node state vectors; constructing an action space for selecting target server nodes; designing a reward function with transmission completion time, node load balancing, bandwidth utilization, and transmission success rate as optimization objectives; using a near-end policy optimization algorithm to pre-train the scheduling policy network with historical data; and outputting the optimal target server node selection and priority ranking based on the current state during the online phase, thereby establishing or allocating SFTP session channels accordingly.

[0019] Furthermore, in the adaptive concurrency control sub-step, an SFTP connection pool is maintained for each target server node. In each scheduling cycle, the network health is calculated based on the round-trip latency and packet loss rate of the target server node, the node busyness is calculated based on the CPU load and the number of SFTP connections, and the SFTP connection pool capacity is dynamically adjusted according to preset rules.

[0020] Furthermore, in the file fragmentation and differential compression sub-steps, the file fragment size is determined based on the size of the attachment type file and compressed using the zstd or lz4 algorithm; for static HTML type files, it is checked whether they contain compressed resources. If they do, they are transmitted directly; otherwise, gzip lightweight compression is used; the compressed file fragments or files and metadata are packaged into transmission units.

[0021] Furthermore, in the multi-channel pipelined transmission sub-step, transmission units destined for the same target server node are prioritized and then sent in parallel in a pipelined manner; multiple file fragments of the same file are uploaded in parallel using multiple SFTP session channels in the SFTP connection pool; the transmission rate of each SFTP session channel is dynamically monitored, and when the rate of a certain channel is lower than a preset threshold, the file fragments currently being transmitted on that channel are reallocated to an idle SFTP session channel.

[0022] Furthermore, in the breakpoint resume and incremental synchronization sub-steps, a temporary transmission directory and temporary file fragments are created on the target server node for the target file before transmission; the transmission status record table is updated after each preset data block is transmitted; when an SFTP session channel interruption is detected, breakpoint information is recorded, and a resume request with offset parameters is sent after the session is restored; after all steps are completed, the file fingerprint calculated using the SHA-256 algorithm is compared, and only the changed data blocks are transmitted for files with inconsistent fingerprints; after verification, the temporary file is atomically renamed to the official file.

[0023] Further, step D specifically involves: establishing heartbeat monitoring for the SFTP session channel, sending SSH_MSG_IGNORE messages at preset time intervals to detect channel activity; marking a target server node as unavailable and suspending task allocation when a target server node's heartbeat times out or the SFTP session channel is abnormally disconnected; reallocating unfinished tasks to other available target server nodes and synchronizing the status of transmitted file fragments; re-adding the target server node to the scheduling candidate set after it recovers, and prioritizing the allocation of low-priority tasks for warm-up; retrying files that fail to transmit using an exponential backoff strategy, generating an alarm after exceeding a preset number of attempts.

[0024] Further, step E specifically involves: defining a set of core performance indicators, statistically analyzing their historical distribution using a sliding window approach, and establishing a transmission performance baseline; calculating the deviation between the current indicator and the transmission performance baseline in each scheduling cycle, and triggering an alarm when the deviation exceeds a threshold; dynamically adjusting the exploration rate based on the action confidence output by the scheduling strategy network; and periodically using actual transmission results as feedback samples to incrementally fine-tune the scheduling strategy network.

[0025] Furthermore, the overall execution flow of the method is as follows: the content management system submits the set of files to be transmitted to the file distribution scheduling node; step A is executed to extract feature vectors and classify the files; step B is executed to obtain the status of the target server nodes and generate a distributed transmission plan; step C is executed to transmit the files; step D is executed to monitor and handle anomalies in real time; step E is executed to collect indicators and optimize the scheduling strategy network; after each target server node completes the reception, it returns an acknowledgment, and the file distribution scheduling node encapsulates the transmission completion status of each target server node into a transmission report and feeds it back to the content management system.

[0026] By adopting the above solution, the beneficial effects achieved by the present invention are as follows:

[0027] This invention overcomes the shortcomings of traditional SFTP transmission schemes that do not distinguish between CMS file types by constructing a collaborative architecture of "file feature perception - intelligent distributed scheduling - efficient transmission execution". Static HTML files are compressed and prioritized for transmission to ensure real-time performance, while attachment files are compressed in chunks and transmitted in parallel through multiple channels to fully utilize bandwidth resources. This can alleviate the transmission efficiency problem in CMS scenarios where there are both many small files and large large files.

[0028] This invention applies reinforcement learning to SFTP multi-node file distribution scheduling scenarios. By constructing a scheduling decision environment using file feature vectors and node state vectors, and using multi-dimensional network transmission indicators such as transmission completion time, load balancing, bandwidth utilization, and transmission success rate as optimization objectives, the scheduling strategy network is trained to learn the optimal mapping from system state to scheduling decisions. This achieves intelligent scheduling of transmission tasks in large-scale multi-node scenarios, which is beneficial to improving network resource utilization and transmission efficiency. The adaptive concurrency control sub-step dynamically adjusts the SFTP connection pool capacity according to network conditions and target server node load, which can reduce the risk of target server node overload or bandwidth idleness.

[0029] This invention improves the reliability and efficiency of large-scale file transfers through multi-channel pipelined transmission and a refined breakpoint resumption mechanism. Multiple files on the same target server node are sent in parallel via pipeline; file fragments of the same file are uploaded in parallel using multiple SFTP session channels, with real-time monitoring of each channel's rate for load balancing. Simultaneously, the breakpoint resumption granularity is refined to the file fragment and data block levels, and incremental synchronization is achieved using file fingerprint comparison, avoiding retransmission from the beginning after an interruption.

[0030] This invention provides the system with fault tolerance and adaptability in complex network environments through dynamic fault tolerance and recovery steps, as well as transmission monitoring and adaptive optimization steps. Heartbeat monitoring can detect the availability of target server nodes in real time; in case of failure, it can automatically migrate unfinished tasks to other available target server nodes and retry using an exponential backoff strategy. Transmission performance baseline combined with Z-score anomaly detection enables real-time monitoring; the scheduling policy network confidence can be used to adjust the balance between exploration and exploitation; and the scheduling policy network is updated through incremental learning, giving the system the potential for continuous performance improvement. Attached Figure Description

[0031] Figure 1 This is a system architecture block diagram of a file distribution transmission method based on the SFTP network protocol proposed in this invention;

[0032] Figure 2 This is a flowchart illustrating the intelligent distributed scheduling steps in an embodiment of the present invention. Detailed Implementation

[0033] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0034] Example 1:

[0035] See Figure 1 This invention provides a file distribution and transmission method based on the SFTP network protocol, applicable to scenarios where static HTML files and attachment files generated by a content management system are distributed and transmitted among multiple target server nodes. The system used in this method consists of a file distribution scheduling node and multiple target server nodes, and the method is executed on the file distribution scheduling node according to the following steps.

[0036] Step S1, File Feature Perception and Classification: The file distribution scheduling node receives a set of files to be transmitted from the content management system and extracts a feature vector for each file, including file size, file extension, MIME type, and file entropy value. This step first performs initial classification based on the file extension and MIME type: if the file extension is .html, .htm, .shtml, or .xhtml, or the MIME type is text / html, it is initially judged as a candidate static HTML file, and the file header information is further parsed to detect whether it contains...<!DOCTYPE html> The file will be identified by the `<file>` tag. If the check passes, it will be classified as a static HTML file; otherwise, it will be classified as an attachment file. For attachment files, they will be marked as large attachments or small attachments depending on whether the file size exceeds 100 MB.

[0037] Step S2, Intelligent Distributed Scheduling: This step consists of a multi-node state awareness sub-step, a reinforcement learning-based scheduling decision sub-step, and an adaptive concurrency control sub-step. The multi-node state awareness sub-step uses a heartbeat mechanism to obtain the network bandwidth, storage capacity, CPU load, and current SFTP connection count of each target server node in real time, forming a node state vector. The reinforcement learning-based scheduling decision sub-step takes the feature vector of each file and the node state vector as input, and uses a pre-trained scheduling policy network based on the near-end policy optimization algorithm to select the optimal target server node for each file to be transmitted and allocate transmission priority. The system generates a distributed transmission plan. The adaptive concurrency control sub-step maintains an SFTP connection pool for each target server node, with an initial capacity of 8. In each scheduling cycle (30 seconds in this embodiment), the network health is calculated based on the round-trip latency and packet loss rate of each target server node, and the node busyness is calculated based on the CPU load and the number of SFTP connections. The SFTP connection pool capacity is dynamically adjusted accordingly: when the network health is higher than 0.8 and the node busyness is lower than 0.6, the capacity is increased by 2; when the network health is lower than 0.5 or the node busyness is higher than 0.8, the capacity is decreased by 2, but remains no less than 2.

[0038] Step S3, Efficient Transmission Execution: Establish SFTP session channels with each target server node according to the distributed transmission plan and execute file transfer; this step includes the following sub-steps:

[0039] File fragmentation and differential compression sub-steps: Large attachments are fragmented into 16MB file segments, and small attachments into 4MB file segments. Each file segment is compressed using the zstd algorithm (compression level 3). Static HTML files are checked for compressed resources (such as images, styles, and scripts). If they are, they are transmitted directly; otherwise, they are compressed using gzip level 1. The compressed file segments or files are packaged with metadata (including the original filename, file segment number, file fingerprint, and compression algorithm identifier) ​​into a transmission unit.

[0040] Multi-channel pipelined transmission sub-steps: After prioritizing transmission units destined for the same target server node, parallel transmission is performed using a producer-consumer pipelined architecture. A bounded buffer of size 16 decouples transmission preparation from data transmission. For large attachments with at least four file fragments, multiple SFTP session channels in the SFTP connection pool are used to upload different file fragments in parallel. Each file fragment is evenly distributed to available SFTP session channels according to its sequence number. Each SFTP session channel reports its current transmission rate every 5 seconds. The scheduling thread of the file distribution scheduling node calculates the average transmission rate. (in, (This refers to the total number of currently available SFTP session channels), when a certain channel rate... When this happens, the file fragments currently being transmitted on that channel are reassigned to the highest-speed idle SFTP session channel.

[0041] The breakpoint resumption and incremental synchronization sub-steps are as follows: Before the transmission begins, a temporary transmission directory and temporary file fragment files are created on the target server node for the target file. The local transmission status record table is updated after every 1MB of data is transmitted. When the SFTP session channel is abnormally interrupted, the system records the breakpoint information (including file path, file fragment sequence number, and offset of transmitted bytes). After the session is restored, a resumption request with offset parameters is sent to the target server node. After all files are transmitted, the SHA-256 file fingerprint of the transmitted files on the target server node is calculated and compared with the source. Incremental synchronization is initiated for files with inconsistent file fingerprints (only the changed data blocks are transmitted). After successful verification, the temporary files are atomically renamed to the official files and the temporary directory is cleaned up.

[0042] Step S4, Dynamic Fault Tolerance and Recovery: Establish heartbeat monitoring for each SFTP session channel, sending an SSH_MSG_IGNORE message every 15 seconds to check channel activity. When a target server node fails to respond to heartbeats three times consecutively (timeout period 45 seconds) or the SFTP session channel is abnormally disconnected, mark the target server node as "unavailable" and suspend new task allocation. Simultaneously, reassign unfinished transmission tasks on the target server node to other available target server nodes, and synchronize the status of transmitted file fragments to the target server nodes to resume transmission from the point of interruption. After the target server node becomes available again (two consecutive normal heartbeat responses), it is added back to the scheduling candidate set, and only low-priority tasks are allocated for warm-up in the first 5 minutes. For failed file or file fragment transmissions, an exponential backoff strategy is used for retrying (intervals of 2 seconds, 4 seconds, 8 seconds, 16 seconds, and 32 seconds respectively). After more than 5 retries, an alarm message is generated and recorded in the transmission report.

[0043] Step S5, Transmission Monitoring and Adaptive Optimization: Collect core performance indicators such as transmission throughput, average task completion time, node load balancing variance, and transmission success rate for each target server node. Calculate the mean and standard deviation of each indicator using a sliding window (window size of 100 transmission tasks) to establish a transmission performance baseline. Within each scheduling cycle, calculate the deviation of the current indicator from the mean of the transmission performance baseline. Trigger an alarm when the deviation exceeds twice the standard deviation. Dynamically adjust the exploration rate based on the action confidence output by the scheduling policy network: set the exploration rate to 0.05 when the confidence is higher than 0.9 and to 0.3 when it is lower than 0.6. Add the actual transmission results as feedback samples to the training dataset monthly to incrementally fine-tune the scheduling policy network.

[0044] Example 2:

[0045] This embodiment, based on Embodiment 1, focuses on illustrating the workflow of the reinforcement learning-based scheduling decision sub-step in step B (intelligent distributed scheduling). See also... Figure 2 This sub-step generates a distributed transmission plan through the following process:

[0046] Step B1: Construct the state space State space It consists of a set of feature vectors for each file to be transferred and a set of node state vectors for each target server node. In this embodiment, the file feature vector is 5-dimensional, including the normalized file size, file type identifier (0 for static HTML, 1 for attachments), file entropy value, normalized file age value, and hot file identifier based on historical access frequency. The node state vector is 4-dimensional, including available bandwidth (Mbps), storage space (GB), CPU load (0~1 normalized value), and normalized value of the current SFTP connection count. Let the number of files to be transferred be... The number of target server nodes is Then the state space The total dimensions are 5 .

[0047] Step B2: Constructing the Action Space Action space A decision combination for selecting target server nodes for each file to be transferred; the action for each file is as follows: Select one target server node from a set of target servers; the total action space size is [value missing]. To reduce the complexity of the action space, a hierarchical decision-making strategy is adopted, that is, firstly, the files are grouped according to file type and size, and then the target server node combination is decided for each group of files.

[0048] Step B3: Design the reward function The reward function comprehensively considers transmission completion time, target server node load balancing, network bandwidth utilization, and transmission success rate; its expression is:

[0049] ;

[0050] in, The total time for all file transfers to complete. The preset maximum allowed transmission time; This represents the variance normalized value of the load on each target server node; For bandwidth utilization; The transmission success rate is used; each weight coefficient is used to balance the contribution of each sub-objective to the total reward, and the values ​​are: weight of transmission completion time item. Weight of load balancing item Bandwidth utilization item weight Weight of transmission success rate The above weight values ​​have been verified through simulation experiments, and they can maximize the overall reward in a typical CMS file transfer scenario.

[0051] Step B4: Pre-train the scheduling policy network using a proximal policy optimization algorithm; use historical transmission task data as training samples, and iteratively optimize the scheduling policy network parameters through interaction with the environment; the scheduling policy network adopts a multilayer perceptron structure with two hidden layers, each layer having 128 neurons, and the activation function being ReLU; during training, a discount factor is applied. GAE parameters PPO trimming parameters The initial learning rate is Linear decay was employed; the training data came from historical transmission logs over the past 6 months, totaling approximately 100,000 transmission task records.

[0052] Step B5: In the online scheduling phase, the current state space is... Input the trained scheduling policy network, which outputs the probability distribution of each file to be transmitted on each target server node. Select the target server node with the highest probability as the target node for that file. At the same time, determine the transmission priority based on the file type and size: static HTML files have the highest priority (priority 1), small attachments have the second highest priority (priority 2), and large attachments have the lowest priority (priority 3). Finally, generate a distributed transmission plan containing the triple of "file-target server node-priority".

[0053] Example 3:

[0054] This embodiment is based on Embodiment 1 and focuses on illustrating the collaborative execution process of the multi-channel pipeline transmission sub-step (belonging to step C), the dynamic fault tolerance and recovery step (step D), and the transmission monitoring and adaptive optimization step (step E) in the method.

[0055] The coordinated execution of the multi-channel pipeline transmission mechanism: For multiple transmission units destined for the same target server node, they are first sorted by transmission priority, with static HTML files transmitted first, and attachment files arranged in ascending order of size; the system adopts a producer-consumer pipeline architecture, using a bounded buffer of size 16 to decouple transmission preparation from data transmission; for large attachment files with at least 4 file fragments, multiple SFTP session channels in the target server node's SFTP connection pool are used to upload different file fragments in parallel, and each file fragment is evenly distributed to available SFTP session channels according to its sequence number; each SFTP session channel reports its current transmission rate every 5 seconds. The scheduling thread of the file distribution scheduling node calculates the average transmission rate based on this. When a certain channel rate When this happens, a load balancing operation is triggered, which pauses the file fragments currently being transmitted on that channel and reassigns them to the idle SFTP session channel with the highest transmission rate.

[0056] The coordinated execution of dynamic fault tolerance and recovery mechanisms: The file distribution scheduling node maintains a heartbeat timer for each SFTP session channel, sending an SSH_MSG_IGNORE message every 15 seconds to check channel activity. If no response is received within a 45-second timeout period (i.e., three consecutive heartbeat failures), the SFTP session channel or target server node is determined to be unreachable, marked as "unavailable," and new task allocation is suspended. The system queries the transmission status record table to obtain all incomplete transmission tasks on the unavailable target server node, selects the node with the lowest load from the set of available target server nodes as the migration target, and synchronizes the status information of the transmitted file fragments to the migration target node to continue transmission from the breakpoint. After the target server node becomes available again (two consecutive normal heartbeat responses), it is re-added to the scheduling candidate set and prioritized for low-priority tasks for warm-up. For files or file fragments that fail to transmit, an exponential backoff strategy is used for retrying (intervals of 2 seconds, 4 seconds, 8 seconds, 16 seconds, and 32 seconds respectively), and an alarm is generated after more than 5 attempts.

[0057] Coordinated execution of transmission monitoring and adaptive optimization mechanisms: a set of core performance indicators defined by the system. These correspond to transmission throughput, average task completion time, target server node load balancing variance, and transmission success rate, respectively; in the initial stage of operation, a sliding window (window size) is used. Calculate the mean of each indicator. with standard deviation Establish an initial transmission performance baseline. Subsequently, the transmission performance baseline is dynamically updated after each transmission task is completed; at the end of each scheduling cycle, the values ​​of each indicator are calculated. degree of deviation ,when A mild alarm is triggered when... A critical alarm is triggered at any time; the scheduling strategy network outputs the action confidence level. Exploration rate Adjustments will be made dynamically accordingly: hour, ; hour, ; hour, At the beginning of each month (or after a cumulative total of 5,000 new transmission tasks), the actual transmission results are added to the training dataset as feedback samples to incrementally fine-tune the scheduling policy network.

[0058] The present invention and its embodiments have been described above. This description is not restrictive. The accompanying drawings are only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if a person skilled in the art is inspired by this description and designs a similar structure and embodiment without departing from the spirit of the present invention, such design should fall within the protection scope of the present invention.

Claims

1. A file distribution transmission method based on the SFTP network protocol, applied to a distributed transmission system comprising a file distribution scheduling node and multiple target server nodes, characterized in that, Includes the following steps: Step A, File Feature Perception and Classification: The file distribution scheduling node receives a set of files to be transmitted from the content management system, extracts a file feature vector for each file, and classifies the file into static HTML type files or attachment type files based on the file feature vector; Step B, Intelligent Distributed Scheduling: Obtain the status information of each target server node and construct the node status vector; Based on reinforcement learning scheduling decisions, a distributed transmission plan is generated according to file feature vectors and node state vectors. And dynamically adjust the SFTP connection pool capacity; Step C, Efficient Transmission Execution: Establish an SFTP session channel according to the distributed transmission plan and execute file transfer, including file fragmentation and differential compression processing, transmission using a multi-channel pipeline method, and execution of breakpoint resume and incremental synchronization operations; Step D, Dynamic Fault Tolerance and Recovery: When a target server node failure or transmission anomaly is detected, a task migration or retry mechanism is triggered. Step E, Transmission Monitoring and Adaptive Optimization: Collect operational metrics for each step and establish a transmission performance baseline, and continuously evolve the transmission strategy based on the transmission performance baseline.

2. The method according to claim 1, characterized in that, The file feature vector extracted in step A includes file size, file extension, MIME type, and file entropy value; and in step A, files are initially classified according to file extension and MIME type, the HTML candidate files obtained in the initial screening are confirmed by parsing the file header tags, and attachment type files are divided into large attachments and small attachments according to whether the file size exceeds a preset threshold.

3. The method according to claim 1, characterized in that, The specific process of generating a distributed transmission plan based on reinforcement learning scheduling decision in step B is as follows: First, a system state space containing file feature vectors and node state vectors is constructed, and an action space adapted to the selection of target server nodes is constructed; second, based on the state space and action space, a reward function is designed with transmission completion time, node load balancing, bandwidth utilization, and transmission success rate as optimization objectives; then, a near-end policy optimization algorithm is used, and the scheduling policy network is pre-trained using historical transmission task data as training samples. During the online scheduling phase, the current system status is input into the pre-trained scheduling strategy network, which outputs the optimal target server node selection scheme and the corresponding file transfer priority ranking result, so as to establish or allocate SFTP session channels.

4. The method according to claim 3, characterized in that, The method for dynamically adjusting the SFTP connection pool capacity in step B is as follows: maintain an SFTP connection pool for each target server node, calculate the network health based on the round-trip latency and packet loss rate of the target server node in each scheduling cycle, calculate the node busyness based on CPU load and the number of SFTP connections, and dynamically adjust the SFTP connection pool capacity according to preset rules.

5. The method according to claim 1, characterized in that, The file fragmentation and differential compression process in step C is as follows: For attachment type files, determine the file fragment size based on the size and compress using the zstd or lz4 algorithm; for static HTML type files, check whether they contain compressed resources. If they do, transmit them directly; otherwise, use gzip lightweight compression; and package the compressed file fragments or files and metadata into transmission units.

6. The method according to claim 5, characterized in that, The multi-channel pipelined transmission in step C specifically involves: prioritizing transmission units destined for the same target server node and sending them in parallel in a pipelined manner; uploading multiple file fragments of the same file in parallel using multiple SFTP session channels in the SFTP connection pool; dynamically monitoring the transmission rate of each SFTP session channel, and reallocating the file fragments currently being transmitted in that channel to an idle SFTP session channel when the rate of a certain channel is lower than a preset threshold.

7. The method according to claim 6, characterized in that, The breakpoint resumption and incremental synchronization operations in step C are as follows: Before transmission, a temporary transmission directory and temporary file fragments are created on the target server node for the target file; the transmission status record table is updated after each preset data block is transmitted; when an SFTP session channel interruption is detected, the breakpoint information is recorded, and a resumption request with offset parameters is sent after the session is restored; after all operations are completed, the file fingerprint calculated using the SHA-256 algorithm is compared, and only the changed data blocks are transmitted for files with inconsistent fingerprints; after verification, the temporary file is atomically renamed to the official file.

8. The method according to claim 1, characterized in that, Step D specifically involves: establishing heartbeat monitoring for the SFTP session channel, sending SSH_MSG_IGNORE messages at preset time intervals to check channel activity; marking the target server node as unavailable and pausing task allocation when a target server node's heartbeat times out or the SFTP session channel is abnormally disconnected; and reallocating unfinished tasks to other available target server nodes and synchronizing the status of transferred file fragments. Once the target server node recovers, it is re-added to the scheduling candidate set and prioritized for warm-up tasks with low priority. For files that fail to be transmitted, an exponential backoff strategy is used to retry them. An alarm is generated after the preset number of attempts is exceeded.

9. The method according to claim 1, characterized in that, Step E specifically involves: defining a set of core performance indicators, statistically analyzing their historical distribution using a sliding window approach, and establishing a transmission performance baseline; calculating the deviation of the current indicator from the transmission performance baseline in each scheduling cycle, and triggering an alarm when the deviation exceeds a threshold. based on The exploration rate is dynamically adjusted based on the action confidence output by the scheduling strategy network; the actual transmission results are periodically used as feedback samples to incrementally fine-tune the scheduling strategy network, thereby achieving continuous iterative optimization of the transmission strategy.

Images