A network data transmission method and system

Abstract

The present disclosure relates to the technical field of network data transmission, and provides a network data transmission method and system. The method comprises: dividing network request data into a primary channel queue and a secondary channel queue according to real-time indexes; obtaining real-time transmission indexes, calculating a fast rate threshold and a slow rate threshold by using an exponential weighted moving average algorithm, and obtaining a fusion threshold by weighting and fusing the two thresholds; monitoring the queue length and the bandwidth utilization of the primary channel and the secondary channel in real time; when the bandwidth utilization of the primary channel is low and the queue of the secondary channel exceeds the fusion threshold, the secondary channel can borrow the bandwidth of the primary channel; and when the queue of the primary channel exceeds the fusion threshold, the borrowed bandwidth is recovered. The present disclosure can dynamically allocate bandwidth resources according to the real-time state of the network, and improve the transmission efficiency and stability of the network.

Description

TECHNICAL FIELD

[0001] The present disclosure belongs to the technical field of network data transmission, and particularly relates to a network data transmission method and system. BACKGROUND

[0002] With the vigorous development of cloud computing, Internet of Things and edge computing technology, in order to adapt to the communication of various physical devices in the perception layer, network layer and application layer, a wired network and a wireless network are usually combined to realize the unification of short-distance communication and long-distance communication.

[0003] Considering the differences in communication processing capability of different devices, working mode of different communication technologies and network bandwidth, etc., some existing data transmission technologies and their problems are as follows:

[0004] The data transmission adopts a fixed rate threshold to divide the priority, and the communication rate is bound to the priority. Different communication tasks adopt different priorities. Although this method is simple to implement and easy to manage, it cannot adapt to the dynamic changes of the network environment (such as bandwidth fluctuation and burst traffic) in a complex network environment, and cannot realize the dynamic allocation of network resources. SUMMARY

[0005] To solve the above problems, the present disclosure provides a network data transmission method and system, which adopts a dynamic threshold and a cross-layer allocation technology, can intelligently divide the primary and secondary channel queues, dynamically controls the resources by fusing the threshold, realizes bandwidth borrowing and recycling, optimizes the queue processing and intelligently switches the transmission path, and improves the bandwidth utilization rate and the task response efficiency.

[0006] The technical content of the present disclosure is as follows:

[0007] A network data transmission method, characterized in that it comprises:

[0008] obtaining network request data, and dividing the request data into a primary channel queue and a secondary channel queue based on a real-time index;

[0009] obtaining real-time network data transmission indexes, calculating the instantaneous threshold of each network data transmission index as a fast rate threshold by using an exponential weighted moving average algorithm, calculating the long-term trend threshold of each network data transmission index as a slow rate threshold, and fusing the fast rate threshold and the slow rate threshold to obtain a fusion threshold of each network data transmission index;

[0010] real-time monitoring the queue length and the bandwidth utilization rate of the primary channel and the secondary channel; if the primary channel bandwidth utilization rate is lower than a preset lower limit and the secondary channel queue length exceeds a queue length fusion threshold, the secondary channel borrows the primary channel bandwidth for data transmission and reception; if the primary channel queue length exceeds the queue length fusion threshold, the primary channel recycles the borrowed primary channel bandwidth from the secondary channel.

[0011] Furthermore,

[0012] The method of dividing the request data into the primary channel queue and the secondary channel queue comprises:

[0013] Input the requested data into the random forest model and output the classification results of the primary channel or secondary channel.

[0014] Furthermore,

[0015] The instantaneous threshold of each network data transmission indicator is calculated as the fast rate threshold; the long-term trend threshold of each network data transmission indicator is calculated as the slow rate threshold; the calculation formula is:

[0016]

[0017]

[0018] in, is the fast rate threshold at time t; is the network data transmission indicator value at time t; is the fast rate threshold at time t-1; is the slow rate threshold at time t; is the slow rate threshold at time t-1; α is the fast rate smoothing factor, is the slow rate smoothing factor, α> .

[0019] Furthermore,

[0020] The fusion threshold of each network data transmission indicator is obtained by fusion of the fast rate threshold and the slow rate threshold, including:

[0021] If the network's real-time response requirement is higher than the threshold, the maximum value of the fast rate threshold and the slow rate threshold is used as the fusion threshold;

[0022] If the network's real-time response requirement is lower than the threshold, the fast rate threshold and the slow rate threshold are weighted and fused to obtain the fusion threshold, whose formula is:

[0023] + (1- )

[0024] in, , is the fusion threshold, is the fast rate threshold, Slow rate threshold

[0025] Furthermore,

[0026] The main channel adopts a weighted fair queue algorithm to control the flow; and the secondary channel adopts a token bucket algorithm to control the flow.

[0027] Further,

[0028] Further comprising: when network data is transmitted, real-time acquisition of state information of a network transmission link, triggering transmission path switching when network transmission link congestion is detected; when the path is switched, the new path is enabled first and then the old path is closed.

[0029] Further,

[0030] For the aggregation layer and the edge layer in the network architecture, the request data in the main channel queue and the secondary channel queue are sorted according to priority groups, wherein the high-priority group can preempt the bandwidth resources of the low-priority group, and the tasks in the group use a round-robin scheduling mechanism to occupy network bandwidth.

[0031] For the perception layer and the execution layer in the network architecture, the request data in the main channel queue and the secondary channel queue are sorted in ascending order according to the task deadline, and when the deadline of a new task is earlier than that of the current execution task, a preemption mechanism is triggered, and the interrupted task is added to the tail of the queue to wait for scheduling.

[0032] A network data transmission system, comprising:

[0033] A queue allocation module for dividing request data into a main channel queue and a secondary channel queue based on real-time indicators;

[0034] A threshold calculation module for obtaining real-time network data transmission indicators, calculating the instantaneous threshold of each network data transmission indicator as a fast rate threshold using an exponential weighted moving average algorithm, calculating the long-term trend threshold of each network data transmission indicator as a slow rate threshold, and fusing the fast rate threshold and the slow rate threshold to obtain a fusion threshold of each network data transmission indicator;

[0035] A control module for real-time monitoring of the queue length and bandwidth utilization of the main channel and the secondary channel; if the bandwidth utilization of the main channel is lower than a preset lower limit and the queue length of the secondary channel exceeds the queue length fusion threshold, the secondary channel borrows the bandwidth of the main channel for data transmission and reception; if the queue length of the main channel exceeds the queue length fusion threshold, the main channel recovers the borrowed bandwidth of the main channel from the secondary channel.

[0036] Further,

[0037] Further comprising:

[0038] A path switching module for real-time detection of bandwidth, delay, and packet loss rate information of a transmission device when network data is transmitted, triggering transmission path switching when the bandwidth, delay, and packet loss rate exceed a threshold.

[0039] Further,

[0040] Also includes:

[0041] A backup path generation module is used to build a historical database using the bandwidth, delay, and packet loss rate information, predict network link congestion points based on the historical database, and set backup paths based on the link congestion points;

[0042] When it is detected that the bandwidth, delay, or packet loss rate exceeds the threshold, the backup path is switched.

[0043] Compared with the prior art, the present disclosure has the following advantages:

[0044] The present disclosure implements dynamic classification of task priorities by dividing network request data into primary channel queues and secondary channel queues based on real-time indicators. It uses an exponentially weighted moving average algorithm to calculate a fast rate threshold reflecting the instantaneous state of the network and a slow rate threshold reflecting the long-term trend, and then fuses the two into a fused threshold. This constructs a dynamic threshold system to replace the traditional fixed rate threshold.

[0045] By real-time monitoring of the queue lengths of the primary and secondary channels, the secondary channel is allowed to borrow bandwidth from the primary channel when the bandwidth utilization of the primary channel falls below the preset lower limit and the queue length of the secondary channel exceeds the convergence threshold. The borrowed bandwidth is reclaimed when the queue length of the primary channel exceeds the convergence threshold. This dynamic threshold generation and intelligent bandwidth allocation mechanism can flexibly adjust resource allocation strategies based on the real-time network status (such as bandwidth fluctuations and burst traffic). This not only avoids the defect of fixed thresholds that cannot adapt to dynamic network changes, but also realizes dynamic allocation of network resources through elastic resource borrowing between the primary and secondary channels, thus effectively solving the problem of low resource utilization and inability to dynamically adapt to environmental changes in traditional technologies in complex network environments.

[0046] Other features and advantages of the present disclosure will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present disclosure. The purposes and other advantages of the present disclosure can be realized and obtained by the structures indicated in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 A schematic diagram of the disclosed method is shown;

[0049] Figure 2A schematic diagram of the disclosed system is shown;

[0050] Figure 3 The flowchart of the fusion threshold calculation disclosed in the present invention is shown. DETAILED DESCRIPTION

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0052] like Figure 1 This is a schematic diagram of the method disclosed herein, and the specific steps are as follows:

[0053] 1. Step 1: Arrange the request data, including request classification, threshold calculation, and sequence arrangement:

[0054] 1) Request classification:

[0055] After receiving the request data, we classify it into two request queues: "primary channel" (important and urgent) and "secondary channel" (delayable), based on information such as source, type, and service agreement, and real-time indicators (including: important and urgent, and deferrable). (For example, live video uploads go through the primary channel, while log uploads go through the secondary channel.)

[0056] The data structure of the request data includes the source IP address, data type, SLA (Service Level Agreement), etc. The real-time index is a key attribute used to measure the real-time transmission requirements of network request data. It can be obtained through the following methods:

[0057] 1) Obtained through source IP: Identifies the network address of the data sender, which can be used to prioritize requests from different devices or systems (for example, requests from core devices may be more urgent);

[0058] 2) By data type: For example, latency-sensitive data such as real-time video and audio streams are usually classified as primary channels, while non-real-time data such as log uploads and file transfers are classified as secondary channels.

[0059] 3) Obtained through SLA (Service Level Agreement): The performance indicators of data transmission (such as latency limit, bandwidth guarantee, etc.) are agreed upon, which directly reflects the real-time requirements of data transmission.

[0060] Specifically, a lightweight ML model (such as a random forest model) can be used to predict the criticality level of requests and output the request queue allocation marked as primary and secondary.

[0061] 2) Threshold calculation:

[0062] Two thresholds are calculated using "fast rate indicators" (real-time traffic, queue length, etc., reflecting the instantaneous state of the system, sensitive to sudden changes) and "slow rate indicators" (long-term trends, smoothing short-term fluctuations). The two thresholds are combined to form a dynamic threshold, and if the real-time system indicators (such as traffic, queue depth) exceed the threshold, control actions such as flow control, expansion, etc. are taken on the channel;

[0063] Specifically, the algorithm for threshold calculation is as shown in Figure 3 , specifically:

[0064] Initialize parameters, including: fast rate smoothing factor α (large, such as 0.9), slow rate smoothing factor β (small, such as 0.1), initial threshold , , which can be set to the historical average or empirical value;

[0065] Obtain the network data transmission indicators Xt (such as delay, packet loss rate, bandwidth utilization, QoS level, packet size, queue length, CPU load, etc.) of the current system, and update the fast rate threshold and slow rate threshold based on the network data transmission indicators; The formula is:

[0066]

[0067]

[0068] Select the maximum value of the fast rate threshold and the slow rate threshold, or weight and fuse the two to obtain the dynamic threshold, the formula is:

[0069] If the network's response real-time demand is higher than the threshold, the maximum value of the fast rate threshold and the slow rate threshold is taken as the fusion threshold, the formula is: ,

[0070] If the network's response real-time demand is lower than the threshold, the fast rate threshold and the slow rate threshold are weighted and fused to obtain the fusion threshold, the formula is: + (1- )

[0071] Wherein, , and Adjust according to the real-time state of the system, for example, reduce when the false positive rate is high.

[0072] 3) Use mixed queue management strategy to arrange request queue (grouped by priority or deadline): ​

[0073] Specifically, this strategy combines priority preemption with fairness constraints. The overall fairness strategy is to prioritize meeting the minimum network resource requirements and then allocate the remaining network resources.

[0074] For the aggregation layer / edge layer: Group by priority. The high-priority group can preempt the bandwidth of the low-priority group. Tasks within the group are executed in a round-robin manner to occupy network bandwidth.

[0075] For the perception layer / execution layer: prioritize tasks with close deadlines, and interrupt current tasks when new tasks are more urgent.

[0076] 2. Step 2: Before data transmission between the data orchestration and network layer transmission, the cross-layer coordinator monitors the bandwidth utilization and queue length of the primary and secondary channels. Based on the monitoring results, corresponding adjustment measures are taken (such as borrowing bandwidth from the secondary channel when the primary channel is idle and reclaiming bandwidth when the primary channel is congested). The rate controller (WFQ on the primary channel and token bucket on the secondary channel) is used to adjust the transmission rate of the primary and secondary channels.

[0077] Specifically:

[0078] 1) The corresponding adjustment measures based on the monitoring situation of the cross-layer coordinator are:

[0079] When the primary channel is idle (e.g., the idle rate is >30% for 5 seconds), the secondary channel can use the bandwidth of the primary channel to transmit data.

[0080] When the main channel is busy (for example, there are more than 100 queued requests), the borrowed bandwidth is immediately reclaimed to prioritize the main channel.

[0081] Using a cross-layer coordinator can solve the following problems:

[0082] 1. In a wireless narrowband communication network environment where data processing uses priority communication, high-priority tasks may occupy bandwidth resources for a long time. This will cause low-priority but urgent requests to be blocked, resulting in the urgent tasks not being responded to in a timely manner.

[0083] 2. Due to the complexity of communication networks, a mechanism is usually adopted in which primary and secondary rate channels operate independently in the communication network to achieve controllability of each segment and layer of the network. However, this communication technology lacks cross-layer optimization, resulting in low overall bandwidth utilization and wasted bandwidth resources.

[0084] 2) Use the rate controller (WFQ on the primary channel and token bucket on the secondary channel) to adjust the transmission rate of the primary and secondary channels. Specifically:

[0085] For the main channel: use "weighted fair queue" to ensure that important data has at least a fixed transmission rate and is not delayed;

[0086] For the secondary channel: Use the "token bucket" mechanism to handle burst traffic, allowing temporary over-transmission of data without exceeding the limit to avoid overwhelming the primary channel.

[0087] 3. Step 3: During data transmission, use SDN-based bandwidth fast switching technology to dynamically optimize and manage network traffic (such as real-time monitoring, dynamic path switching, and pre-calculated backup paths)

[0088] 1) Real-time monitoring: Check the bandwidth, latency, and packet loss rate of each link at any time, and switch links within milliseconds if problems are detected.

[0089] 2) Dynamic path switching: Set bandwidth usage thresholds, triggering switchover when they are exceeded, and calculate backup paths in advance to reduce latency.

[0090] Specifically, when switching paths, the "add first, then delete" strategy is adopted, first enabling the new path and then shutting down the old path to avoid data loss and ensure smooth transmission;

[0091] 3) Intelligent prediction: Use historical data to predict congestion points and plan alternative routes in advance to avoid congestion.

[0092] like Figure 2 This is a schematic diagram of the disclosed system, including:

[0093] 1) Data orchestration layer, which includes queue allocation module, threshold calculation module, and queue management module;

[0094] The queue allocation module is used to predict the request criticality level based on data request metadata (including source IP, data type, SLA agreement, etc.) through a random forest model, and divide the request data into the primary channel queue or secondary channel queue;

[0095] The threshold calculation module is used to obtain real-time network data transmission indicators. It uses the exponentially weighted moving average algorithm to calculate the instantaneous threshold of each network data transmission indicator as the fast rate threshold; calculate the long-term trend threshold of each network data transmission indicator as the slow rate threshold; and fuse the fast rate threshold and the slow rate threshold to obtain the fused threshold of each network data transmission indicator.

[0096] The queue management module is used to manage queues. For the convergence layer and edge layer in the network architecture, the request data in the primary channel queue and secondary channel queue are sorted by priority group. The high-priority group can preempt the bandwidth resources of the low-priority group. The tasks within the group use a round-robin scheduling mechanism to occupy the network bandwidth.

[0097] For the perception layer and execution layer in the network architecture, the request data in the main channel queue and the secondary channel queue are sorted in ascending order according to the task deadline. When the deadline of the new task is earlier than the current executing task, the preemption mechanism is triggered and the interrupted task is re-added to the tail of the queue waiting for scheduling.

[0098] 2) Data transmission layer, including the cross-layer coordinator module and the rate controller module;

[0099] The cross-layer coordinator module is used to monitor the bandwidth utilization and queue length of the primary and secondary channels and perform corresponding adjustment measures based on the monitoring results;

[0100] The rate controller module is used to adjust the transmission rate of the primary and secondary channels.

[0101] 3) Network layer, which is used to dynamically optimize and manage network traffic based on SDN-based bandwidth fast switching technology, including a path switching module and a backup path generation module.

[0102] The path switching module is used to detect the bandwidth, delay, and packet loss rate information of the transmission equipment in real time when transmitting data on the network. When it is detected that the bandwidth, delay, or packet loss rate exceeds the threshold, the transmission path switching is triggered.

[0103] The backup path generation module is used to build a historical database using bandwidth, delay, and packet loss rate information, predict network link congestion points based on the historical database, and set backup paths based on the link congestion points.

[0104] Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A network data transmission method, characterized in that: include: Obtain network request data and divide the request data into primary channel queue and secondary channel queue based on real-time indicators; Obtain real-time network data transmission indicators, use the exponentially weighted moving average algorithm to calculate the instantaneous threshold of each network data transmission indicator as the fast rate threshold; calculate the long-term trend threshold of each network data transmission indicator as the slow rate threshold; and fuse the fast rate threshold and the slow rate threshold to obtain the fusion threshold of each network data transmission indicator; The queue length and bandwidth utilization of the primary and secondary channels are monitored in real time. If the primary channel bandwidth utilization falls below the preset lower limit and the secondary channel queue length exceeds the queue length convergence threshold, the secondary channel borrows the primary channel bandwidth for data transmission and reception. If the primary channel queue length exceeds the queue length convergence threshold, the primary channel reclaims the borrowed primary channel bandwidth from the secondary channel.

2. A network data transmission method according to claim 1, characterized in that: The method of dividing the request data into the primary channel queue and the secondary channel queue comprises: Input the requested data into the random forest model and output the classification results of the primary channel or secondary channel.

3. A network data transmission method according to claim 1, characterized in that: The instantaneous threshold of each network data transmission indicator is calculated as the fast rate threshold; the long-term trend threshold of each network data transmission indicator is calculated as the slow rate threshold; the calculation formula is: in, is the fast rate threshold at time t; is the network data transmission indicator value at time t; is the fast rate threshold at time t-1; is the slow rate threshold at time t; is the slow rate threshold at time t-1; α is the fast rate smoothing factor, is the slow rate smoothing factor, α> .

4. A network data transmission method according to claim 1, characterized in that: The fusion threshold of each network data transmission indicator is obtained by fusion of the fast rate threshold and the slow rate threshold, including: If the network's real-time response requirement is higher than the threshold, the maximum value of the fast rate threshold and the slow rate threshold is used as the fusion threshold; If the network's real-time response requirement is lower than the threshold, the fast rate threshold and the slow rate threshold are weighted and fused to obtain the fusion threshold, whose formula is: +（1- ） in, , is the fusion threshold, is the fast rate threshold, is the slow rate threshold.

5. A network data transmission method according to claim 1, characterized in that: The weighted fair queue algorithm is used to control traffic on the primary channel, and the token bucket algorithm is used to control traffic on the secondary channel.

6. A network data transmission method according to claim 1, characterized in that: Also includes: During network data transmission, the status information of the network transmission link is obtained in real time. When congestion of the network transmission link is detected, the transmission path switching is triggered; when the path is switched, the new path is enabled first and then the old path is closed.

7. A network data transmission method according to claim 1, characterized in that: For the convergence layer and edge layer in the network architecture, the request data in the primary and secondary channel queues are sorted by priority. The high-priority group can preempt the bandwidth resources of the low-priority group. The tasks within the group use a round-robin scheduling mechanism to occupy the network bandwidth. For the perception layer and execution layer in the network architecture, the request data in the main channel queue and the secondary channel queue are sorted in ascending order according to the task deadline. When the deadline of the new task is earlier than the current executing task, the preemption mechanism is triggered and the interrupted task is re-added to the tail of the queue waiting for scheduling.

8. A network data transmission system, characterized in that: include: The queue allocation module is used to divide the request data into the primary channel queue and the secondary channel queue based on the real-time index; The threshold calculation module is used to obtain real-time network data transmission indicators and use the exponentially weighted moving average algorithm to calculate the instantaneous threshold of each network data transmission indicator as the fast rate threshold; calculate the long-term trend threshold of each network data transmission indicator as the slow rate threshold; and fuse the fast rate threshold and the slow rate threshold to obtain the fused threshold of each network data transmission indicator; The control module is used to monitor the queue length and bandwidth utilization of the primary and secondary channels in real time. If the bandwidth utilization of the primary channel falls below a preset lower limit and the queue length of the secondary channel exceeds a queue length fusion threshold, the secondary channel borrows the bandwidth of the primary channel for data transmission and reception. If the queue length of the primary channel exceeds the queue length fusion threshold, the primary channel reclaims the borrowed primary channel bandwidth from the secondary channel.

9. The system according to claim 8, characterized in that Also includes: The path switching module is used to detect the bandwidth, delay, and packet loss rate information of the transmission equipment in real time when transmitting data on the network. When it is detected that the bandwidth, delay, or packet loss rate exceeds the threshold, the transmission path switching is triggered.

10. The system according to claim 9, characterized in that Also includes: A backup path generation module is used to build a historical database using the bandwidth, delay, and packet loss rate information, predict network link congestion points based on the historical database, and set backup paths based on the link congestion points; When it is detected that the bandwidth, delay, or packet loss rate exceeds the threshold, the backup path is switched.

Images