Large file fast transmission method and system oriented to complex cross-domain environment

Through intelligent path selection and fault prediction mechanism, the efficiency and reliability problems in large file transmission in complex cross-domain environments are solved, and efficient and stable data transmission in complex network environments are achieved.

CN119030965APending Publication Date: 2024-11-26NAT COMP NETWORK & INFORMATION SECURITY MANAGEMENT CENT +1
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410892693.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-04
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In complex cross-domain environments, traditional large file transfer methods have problems such as low transmission efficiency, insufficient reliability, difficulty in path selection and insufficient prediction of influencing factors, resulting in slow transmission speed, unreliable and susceptible to network failures.

Method used

The intelligent path selection mechanism is adopted to monitor the health of nodes and network quality in real time, calculate the network quality score and path score between nodes, select the optimal path, and perform fault prediction and avoidance, ensuring the stability and reliability of the transmission process.

Benefits of technology

It improves the efficiency and reliability of data transmission, enhances network stability and business continuity, dynamically adapts to changes in the network environment, and reduces operation and maintenance costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119030965A_ABST
    Figure CN119030965A_ABST
Patent Text Reader

Abstract

The invention discloses a large file fast transmission method and system oriented to a complex cross-domain environment, and belongs to the field of digital information transmission. The method comprises the following steps of: monitoring utilization rates and temperatures of a CPU (Central Processing Unit), a memory and a disk of each node in real time aiming at a multi-data center connected with a hierarchical structure, and calculating a node health degree score; calculating a network quality score between the nodes according to the actual bandwidth rate, the maximum bandwidth rate, the packet loss retransmission rate and the network delay; selecting an optimal path for large file transmission according to the superior node health degree score and the inter-node network quality score; and when the optimal path is connected with the superior node, calculating the path health degree score at the current moment, carrying out fault prediction and avoiding the predicted fault path. According to the method, the optimal path can be selected according to the real-time network condition, the data transmission efficiency is improved, possible influence factors can be predicted and dealt with, and the stability and reliability of the transmission process are ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a large file fast transmission method and system in a complex cross-domain environment, belonging to the field of digital information transmission. Background Art

[0002] As the globalization process continues to deepen, the demand for cross-domain data transmission between enterprises and organizations is growing. Especially in the era of big data, the transmission of large files is becoming more and more common, which may involve fast transmission between data centers or nodes in different geographical locations and different network environments. The current mainstream large file transmission technology solutions include Xunlei, KuaiTransfer, Aspera, Signiant, FileCatalyst, Dropbox, Google Drive, OneDrive and other cloud storage services. However, traditional data transmission methods often face a series of challenges when facing complex cross-domain environments:

[0003] 1. Low transmission efficiency: Traditional data transmission methods may be limited by factors such as network bandwidth and latency, resulting in slow transmission of large files and affecting business efficiency.

[0004] 2. Insufficient transmission reliability: In complex cross-domain environments, network conditions may be unstable, and traditional methods may not be able to cope with problems such as network interruptions and packet loss, resulting in unreliable data transmission or even transmission failure.

[0005] 3. Difficulty in path selection: Cross-domain data transmission may involve multiple network paths. Selecting the optimal path becomes a challenge. Traditional methods often lack intelligent path selection mechanisms.

[0006] 4. Insufficient prediction of influencing factors: Traditional methods often find it difficult to predict and adapt to possible transmission failures in complex cross-domain environments, such as network congestion and packet loss, and lack targeted adjustment measures.

[0007] Therefore, in order to meet the demand for fast transmission of large files in complex cross-domain environments, a new technology is needed to overcome the limitations of traditional methods and improve transmission efficiency and reliability. Summary of the invention

[0008] The purpose of the present invention is to provide a method and system for fast transmission of large files in a complex cross-domain environment, which has intelligent path selection capability, can select the optimal path according to the real-time network conditions, and improve the data transmission efficiency; can predict and respond to possible influencing factors, and ensure the stability and reliability of the transmission process.

[0009] The technical solution adopted by the present invention to achieve the above-mentioned purpose is as follows:

[0010] A method for fast transmission of large files in a complex cross-domain environment includes the following steps:

[0011] For multiple data centers connected in a hierarchical structure, each data center serves as the root node and other nodes at all levels according to its hierarchical position, and users upload large files from the root node;

[0012] Monitor the CPU, memory and disk usage and temperature of each node in real time, and calculate the node health score;

[0013] Calculate the network quality score between nodes based on the actual bandwidth rate, maximum bandwidth rate, packet loss retransmission rate and network delay;

[0014] The lower-level node selects the optimal path for large file transmission based on the health score of the upper-level node and the network quality score between nodes;

[0015] When the lower-level node is connected to the upper-level node according to the optimal path, the path health score at the current moment is calculated, and faults are predicted based on the path health score to avoid the path predicted to have a fault.

[0016] Furthermore, the algorithm for calculating the node health score is:

[0017]

[0018] Among them, H is the node health score, U c is the CPU usage, T c is the CPU temperature, U m is the memory usage, T m is the memory temperature, U d is the disk usage, T d is the disk temperature; w 1 ,w 2 ,w 3 is the weight of CPU, memory, and disk.

[0019] Furthermore, the algorithm for calculating the network quality score between nodes is:

[0020]

[0021] Among them, Q is the network quality score between nodes, B is the actual bandwidth rate, and B max is the maximum bandwidth rate, R is the packet loss retransmission rate, Δt is the network delay; w 4 ,w 5 ,w 6 are the weights of bandwidth rate, network delay, and packet loss retransmission rate.

[0022] Furthermore, the path score is calculated according to the health score of the upper node and the network quality score between nodes. According to the scores of all paths, the path with the highest score is selected as the optimal path for large file transmission.

[0023] Furthermore, the algorithm for calculating the path score is:

[0024] S=w 7 H+*w 8 Q

[0025] Among them, S is the path score; w 7 ,w 8 It is the weight of the health score of the upper node and the network quality score between nodes.

[0026] Furthermore, the step of calculating the path health score at the current moment includes:

[0027] Calculate the health score of the parent node at the current moment based on the time sliding window;

[0028] Calculate the network quality score between nodes at the current moment based on the time sliding window;

[0029] The product of the health score of the upper node at the current moment and the network quality score between nodes is calculated to obtain the path health score at the current moment.

[0030] Furthermore, the algorithm for calculating the health score of the upper node at the current moment based on the time sliding window is:

[0031]

[0032] Among them, H t is the health score of the parent node at the current time t, t-1 to t-10 are 10 time windows divided from the time t, is the sum of the node health scores of the five time windows closest to time t, It is the sum of the node health scores of the five time windows farthest from time t.

[0033] Furthermore, the algorithm for calculating the network quality score between nodes at the current moment based on the time sliding window is:

[0034]

[0035] Among them, Q t is the inter-node network quality score at the current time t, t-1 to t-10 are 10 time windows divided from the time t, is the sum of the network quality scores between nodes in the five time windows closest to time t, It is the total network quality score among nodes in the 5 time windows farthest from the t moment.

[0036] Further, the steps of performing fault prediction based on the path health score and avoiding the predicted faulty path include: comparing the path health score P at the current moment t with a preset fault threshold g. If P t < g, it is predicted that there is a fault risk in the current optimal path; if the large file is not being transmitted on the current optimal path, a new optimal path is reselected and switched; if the large file is being transmitted on the current optimal path, switch to the reselected optimal path after the transmission ends.

[0037] A large file fast transmission system for a complex cross - domain environment, including:

[0038] Multi - data centers connected in a hierarchical structure. Each data center serves as the root node and other levels of nodes according to its hierarchical position. The root node provides a port for users to upload large files.

[0039] A transmission optimization management module deployed on each node. This module is responsible for real - time monitoring of the CPU, memory, disk usage rate and temperature of each node, calculating the node health score; calculating the network quality score among nodes according to the actual bandwidth rate, maximum bandwidth rate, packet loss retransmission rate and network latency; selecting the optimal path for large file transmission according to the health score of the upper - level node and the network quality score among nodes; calculating the path health score at the current moment when connecting according to the optimal path to the upper - level node, and performing fault prediction based on the path health score and avoiding the predicted faulty path.

[0040] The present invention has the following advantages:

[0041] 1. Improve network stability and reliability: Node health assessment can help detect potential node failures or performance degradation in a timely manner, so as to connect to nodes that may have faults. Network quality assessment among nodes helps to select paths with higher network quality, reduce problems such as network latency and packet loss, and further improve the stability and reliability of the transmission network.

[0042] 2. Optimize network performance: By selecting paths with better node health and higher network quality, network performance can be optimized, the efficiency and speed of data transmission can be improved, which helps to enhance the user experience, reduce data transmission latency, and improve the response speed.

[0043] 3. Enhance network stability: Fault prediction and avoidance technologies can identify potential transmission problems in advance and take corresponding avoidance measures, thereby enhancing the network stability.

[0044] 4. Dynamically adapt to changes in the network environment: Node health assessment and network quality assessment can be dynamically adjusted and optimized according to real-time changes in the network environment. When the node health changes or the network quality decreases, the path selection technology can automatically adjust and select the optimal path to adapt to the changing network conditions.

[0045] 5. Reduce network operation and maintenance costs: Automated path selection technology can reduce the workload of network administrators, improve network management efficiency, and reduce network operation and maintenance costs.

[0046] 6. Enhanced business continuity: When certain nodes or paths fail or performance degrades, path selection technology based on node health assessment and network quality assessment can quickly switch to other optimal paths to ensure business continuity and stability and reduce the risk of business interruption. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a connection architecture diagram of multiple data centers.

[0048] Figure 2 This is an example diagram of communication between the upper node and the lower node.

[0049] Figure 3 This is an example diagram of calculating the node health score and inter-node network quality score based on the time sliding window.

[0050] Figure 4 It is a comparison chart of the file upload performance of Doris and FastDFS of the present invention in the test.

[0051] Figure 5 It is a diagram of resource usage of Doris of the present invention during testing. DETAILED DESCRIPTION

[0052] In order to make the various technical features and advantages or technical effects in the above technical solutions of the present invention more obvious and easy to understand, they are described in detail below with reference to the accompanying drawings.

[0053] Figure 1This is an example of a multi-data center architecture, including a root data center and level I, level II, and level III data centers. The multi-data centers are connected in a hierarchical structure. Starting from level II, each node is configured with multiple upper nodes that are allowed to connect, but only one can be selected for connection at the same time. A user can upload a large file from the root node, and the large file needs to be sent from the root node to each node of the multi-data center in sequence. In order to enable large files to be efficiently and reliably transferred between different nodes in this multi-data center, the embodiment of the present invention specifically proposes a large file fast transmission method for complex cross-domain environments, introduces technical means such as node health evaluation, network quality evaluation between nodes, optimal path selection, and fault prediction and avoidance mechanism optimization. By evaluating the health of the nodes and evaluating the network quality between the nodes, the optimal path selection and fault prediction and avoidance are performed. These technical means can be executed by the transmission optimization management module deployed on each node, forming a large file fast transmission system (Doris) for complex cross-domain environments. The detailed processing steps of this method are described below.

[0054] 1. Monitor the CPU, memory and disk usage of the node in real time and calculate the node health score. The specific formula is as follows:

[0055]

[0056] Among them, H is the node health score, U c is the CPU usage, T c is the CPU temperature, U m is the memory usage, T m is the memory temperature, U d is the disk usage, T d is the disk temperature; w 1 ,w 2 ,w 3 are the weights of CPU, memory, and disk, all of which are 1 by default and can be adjusted according to the specific environment and expert experience.

[0057] like Figure 2 In the example shown, the health scores of nodes B1, B2, and B3 are 0.141, 0.299, and 0.712, respectively.

[0058] 2. Calculate the network quality score between nodes based on the actual bandwidth rate, maximum bandwidth rate, packet loss retransmission rate and network delay. The specific formula is as follows:

[0059]

[0060] Among them, Q is the network quality score between nodes, B is the actual bandwidth rate, and B maxis the maximum bandwidth rate, R is the packet loss retransmission rate, Δt is the network delay; w 4 ,w 5 ,w 6 The weights of bandwidth rate, network delay, and packet loss retransmission rate are all 1 by default and can be adjusted according to the specific environment and expert experience.

[0061] like Figure 2 In the example shown, the maximum bandwidth rate is 100 Mbps, the actual bandwidth rate is 80 Mbps, the network delay is 4 ms, and the packet loss retransmission rate is 1%. The network quality score between the B1-C1 nodes calculated by the above formula is 0.762.

[0062] 3. Select the optimal path based on the health score of the upper node and the network quality score between nodes. The specific steps are as follows:

[0063] a) The lower-level node requests the health score of the upper-level node that it is allowed to connect to, and calculates the network quality score of its own node and each upper-level node. If the upper-level node has not been connected and data is missing, the average value of the existing data is taken.

[0064] b) Calculate the path score based on the health score of the upper node and the network quality score between nodes. The formula is as follows:

[0065] S=w 7 H+*w 8 Q

[0066] Among them, S is the path score; w 7 ,w 8 The weights of the health score of the upper-level node and the network quality score between nodes are both 1 by default and can be adjusted according to the specific environment and expert experience.

[0067] d) After scoring all paths, select the path with the highest score as the optimal path. Figure 2 In the example shown, the optimal path is the B3-C1 node path.

[0068] 4. Calculate the path health score based on the health score of the upper node and the network quality score between nodes, and predict and avoid faults based on the path health score. The specific instructions are as follows:

[0069] During the connection between the lower-level node and the upper-level node, based on the idea of ​​time sliding window, the lower-level node periodically requests the health of the upper-level node, calculates the network quality score between the current nodes, and performs fault prediction. Figure 3 Taking the example shown as an example, the health scores of the upper-level nodes and the network quality scores between nodes for the last 10 times are calculated, specifically:

[0070] a) Calculate the health score of the parent node at the current moment based on the time sliding window:

[0071]

[0072] Among them, H t is the health score of the parent node at the current time t, t-1 to t-10 are 10 time windows divided from the time t, is the sum of the node health scores of the five time windows closest to time t, It is the sum of the node health scores of the five time windows farthest from time t.

[0073] b) Calculate the network quality score between nodes at the current moment based on the time sliding window:

[0074]

[0075] Among them, Q t is the inter-node network quality score at the current time t, t-1 to t-10 are 10 time windows divided from the time t, is the sum of the network quality scores between nodes in the five time windows closest to time t, It is the sum of the network quality scores between nodes in the five time windows farthest from time t.

[0076] c) Calculate the product of the upper node health score and the network quality score between nodes to get the path health score at the current moment:

[0077] P t =H t *Q t

[0078] exist Figure 3 In the example shown, H is calculated t =0.218, Q t =0.771, P t =0.218*0.771=0.168.

[0079] d) The path health score P at time t t When it is less than the fault threshold g, it means that the connection may fail. The threshold can be set according to the actual situation. If there is no large file being transferred at this time, the optimal path is switched immediately; if there is a large file being transferred at this time, the optimal path is switched after the large file transfer is completed.

[0080] Experimental test:

[0081] 1. Control test:

[0082] This test compares the system Doris proposed in the present invention with the current mainstream lightweight distributed file system FastDFS.

[0083] Environment configuration: 15G memory, 80G disk.

[0084] Test results: Figure 4 As shown, Figure 4 Figure (a) shows that under single-core CPU conditions, as the file data size increases, Doris's performance is shortened from 70% slower than FastDFS to 5% slower; Figure 4 Figure (b) shows that under the condition of 4-core CPU, the performance of Doris is improved from 57% slower than FastDFS to 32% faster than FastDFS. The test results show that the overall performance of Doris for uploading large files is better than FastDFS, and the performance advantage becomes more obvious as the file data size increases and the number of cores increases.

[0085] 2. Pressure comparison test:

[0086] Compare the time and hardware resource usage of Doris and FastDFS when transferring the same file. In this test, a 4-level tree structure is constructed, and the time to distribute files of different data volumes to 10,000 machines is calculated.

[0087] Environment configuration: CPU 4 cores, memory 15G, disk 80G, 10G bandwidth.

[0088] Test results: See the test data in Tables 1 and 2 below, and Figure 5 Doris's processor usage and memory usage are shown.

[0089] Table 1 The time taken for Doris and fastDFS to send files during the test

[0090]

[0091]

[0092] Table 2 The maximum CPU and memory usage of Doris and FastDFS during the test

[0093] Maximum CPU usage Maximum memory usage Doris 370% Memory 8GB, cache 317.8GB FastDFS 390% Memory 12GB, cache 370GB

[0094] From the comparison data in Table 1 and Table 2, we can see that Doris has lower CPU usage and lower memory usage during the process, so its performance is better than FastDFS.

[0095] Although the present invention has been disclosed as above by way of embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent substitutions of the technical solutions of the present invention made by ordinary technicians in the field should all be included in the protection scope of the present invention. The protection scope of the present invention shall be based on what is defined in the claims.

Claims

1. A method for fast transmission of large files in a complex cross-domain environment, characterized in that: The following steps are involved: For multiple data centers connected in a hierarchical structure, each data center serves as the root node and other nodes at all levels according to its hierarchical position, and users upload large files from the root node; Monitor the CPU, memory and disk usage and temperature of each node in real time, and calculate the node health score; Calculate the network quality score between nodes based on the actual bandwidth rate, maximum bandwidth rate, packet loss retransmission rate and network delay; The lower-level node selects the optimal path for large file transmission based on the health score of the upper-level node and the network quality score between nodes; When the lower-level node is connected to the upper-level node according to the optimal path, the path health score at the current moment is calculated, and faults are predicted based on the path health score to avoid the path predicted to have a fault.

2. The method according to claim 1, characterized in that The algorithm for calculating the node health score is: Among them, H is the node health score, U c is the CPU usage, T c is the CPU temperature, U m is the memory usage, T m is the memory temperature, U d is the disk usage, T d is the disk temperature; w1, w2, w3 are the weights of CPU, memory, and disk.

3. The method according to claim 1, characterized in that The algorithm for calculating the network quality score between nodes is: Among them, Q is the network quality score between nodes, B is the actual bandwidth rate, and B max is the maximum bandwidth rate, R is the packet loss retransmission rate, Δt is the network delay; w4, w5, w6 are the weights of bandwidth rate, network delay, and packet loss retransmission rate.

4. The method according to claim 1, characterized in that The path score is calculated based on the health score of the upper node and the network quality score between nodes. Based on the scores of all paths, the path with the highest score is selected as the optimal path for large file transmission.

5. The method according to claim 4, characterized in that The algorithm for calculating the path score is: S=w7H+*w8Q Among them, S is the path score; w7 and w8 are the weights of the upper-level node health score and the network quality score between nodes.

6. The method according to claim 1, characterized in that The steps to calculate the path health score at the current moment include: Calculate the health score of the parent node at the current moment based on the time sliding window; Calculate the network quality score between nodes at the current moment based on the time sliding window; The product of the health score of the upper node at the current moment and the network quality score between nodes is calculated to obtain the path health score at the current moment.

7. The method according to claim 6, characterized in that The algorithm for calculating the health score of the parent node at the current moment based on the time sliding window is: Among them, H t is the health score of the parent node at the current time t, t-1 to t-10 are 10 time windows divided from the time t, is the sum of the node health scores of the five time windows closest to time t, It is the sum of the node health scores of the five time windows farthest from time t.

8. The method according to claim 6, characterized in that The algorithm for calculating the network quality score between nodes at the current moment based on the time sliding window is: Among them, Q t is the inter-node network quality score at the current time t, t-1 to t-10 are 10 time windows divided from the time t, is the sum of the network quality scores between nodes in the five time windows closest to time t, It is the sum of the network quality scores between nodes in the five time windows farthest from time t.

9. The method according to claim 1, characterized in that The steps of predicting faults according to the path health score and avoiding the predicted faulty path include: t Compared with the preset fault threshold g, if P t <g, it is predicted that the current optimal path has a risk of failure; if the current optimal path is not transmitting large files, a new optimal path is reselected and switched; if the current optimal path is transmitting large files, it is switched to the reselected optimal path after the transmission is completed.

10. A system for fast large file transmission in a complex cross-domain environment, implementing the method described in any one of claims 1 to 9, characterized in that: include: Multiple data centers connected in a hierarchical structure, each data center serves as the root node and other nodes at all levels according to its hierarchical position. The root node provides a port for users to upload large files. The transmission optimization management module deployed on each node is responsible for real-time monitoring of the CPU, memory and disk usage and temperature of each node, and calculating the node health score; it calculates the network quality score between nodes based on the actual bandwidth rate, maximum bandwidth rate, packet loss retransmission rate and network delay; The optimal path for large file transfer is selected based on the health score of the upper node and the network quality score between nodes. When connecting to the upper node according to the optimal path, the path health score at the current moment is calculated, and faults are predicted based on the path health score to avoid paths with predicted faults.

Citation Information

Cited By

  • Large file transmission state monitoring method based on multi-service collaboration

    CN121530964A

  • Systems and Methods for Extranet Application Support

    US20260163963A1