Network data transmission method and system

Abstract

The invention relates to the technical field of network data transmission, and provides a network data transmission method and system, and the method comprises the steps: distributing network request data to a primary channel queue and a secondary channel queue according to a real-time index; acquiring a real-time transmission index, calculating a fast rate threshold value and a slow rate threshold value by using an exponential weighted moving average algorithm, and performing weighted fusion on the fast rate threshold value and the slow rate threshold value to obtain a fusion threshold value; the queue length and the bandwidth utilization rate of the main channel and the secondary channel are monitored in real time, when the bandwidth utilization rate of the main channel is low and the queue of the secondary channel exceeds the fusion threshold value, the secondary channel can borrow the bandwidth of the main channel, and when the queue of the main channel exceeds the fusion threshold value, the borrowed bandwidth is recycled. The bandwidth resources can be dynamically allocated according to the real-time state of the network, and the network transmission efficiency and stability are improved.

Description

Technical Field

[0001] The present disclosure belongs to the technical field of network data transmission, and in particular to a network data transmission method and system. Background Art

[0002] With the vigorous development of cloud computing, Internet of Things, and edge computing technologies, in order to adapt to the communication of various physical devices in the perception layer, network layer, and application layer, the integration of wired networks and wireless networks is usually adopted to achieve the unification of short-distance communication and long-distance communication.

[0003] Considering the differences in communication processing capabilities of different devices, working methods of different communication technologies, and network bandwidth, the current state of some data transmission technologies and their existing problems are as follows: Data transmission uses a fixed rate threshold to divide priorities, binding the communication rate to the priority. Different communication tasks use different priorities. Although this method is simple to implement and easy to manage, it cannot adapt to dynamically changing network environments (such as bandwidth fluctuations and burst traffic) in complex network environments and cannot achieve dynamic allocation of network resources. Summary of the Invention

[0004] To solve the above problems, the present disclosure provides a network data transmission method and system, which adopts dynamic thresholds and cross-layer allocation technology, can intelligently divide the primary and secondary channel queues, dynamically regulate resources by integrating thresholds, realize bandwidth borrowing and recovery, optimize queue processing and intelligently switch transmission paths, thereby improving bandwidth utilization and task response efficiency.

[0005] The following is the technical content of this disclosure: A network data transmission method, characterized by comprising: Obtain network request data and divide the request data into primary channel queue and secondary channel queue based on real-time indicators; Obtain real-time network data transmission indicators, use the exponentially weighted moving average algorithm to calculate the instantaneous threshold of each network data transmission indicator as the fast rate threshold; calculate the long-term trend threshold of each network data transmission indicator as the slow rate threshold; and fuse the fast rate threshold and the slow rate threshold to obtain the fusion threshold of each network data transmission indicator; The queue length and bandwidth utilization of the primary and secondary channels are monitored in real time. If the primary channel bandwidth utilization falls below the preset lower limit and the secondary channel queue length exceeds the queue length convergence threshold, the secondary channel borrows the primary channel bandwidth for data transmission and reception. If the primary channel queue length exceeds the queue length convergence threshold, the primary channel reclaims the borrowed primary channel bandwidth from the secondary channel.

[0006] Furthermore, The method of dividing the request data into the primary channel queue and the secondary channel queue comprises: Input the requested data into the random forest model and output the classification results of the primary channel or secondary channel.

[0007] Further, The instantaneous threshold of each network data transmission indicator is calculated as the fast rate threshold; the long-term trend threshold of each network data transmission indicator is calculated as the slow rate threshold; the calculation formula is:

[0008]

[0009] in, is the fast rate threshold at time t; is the network data transmission indicator value at time t; is the fast rate threshold at time t-1; is the slow rate threshold at time t; is the slow rate threshold at time t-1; α is the fast rate smoothing factor, is the slow rate smoothing factor, α> .

[0010] Further, The fusion threshold of each network data transmission indicator is obtained by fusion of the fast rate threshold and the slow rate threshold, including: If the network's real-time response requirement is higher than the threshold, the maximum value of the fast rate threshold and the slow rate threshold is used as the fusion threshold; If the network's real-time response requirement is lower than the threshold, the fast rate threshold and the slow rate threshold are weighted and fused to obtain the fusion threshold, whose formula is: + (1- )

[0011] in, , is the fusion threshold, is the fast rate threshold, Slow rate threshold

[0012] Further, The weighted fair queue algorithm is used to control traffic on the primary channel, and the token bucket algorithm is used to control traffic on the secondary channel.

[0013] Further, It also includes: obtaining the status information of the network transmission link in real time during network data transmission, and triggering transmission path switching when congestion of the network transmission link is detected; when switching the path, first enabling the new path and then closing the old path.

[0014] Furthermore, For the convergence layer and edge layer in the network architecture, the request data in the primary and secondary channel queues are sorted by priority. The high-priority group can preempt the bandwidth resources of the low-priority group. The tasks within the group use a round-robin scheduling mechanism to occupy the network bandwidth. For the perception layer and execution layer in the network architecture, the request data in the main channel queue and the secondary channel queue are sorted in ascending order according to the task deadline. When the deadline of the new task is earlier than the current executing task, the preemption mechanism is triggered and the interrupted task is re-added to the tail of the queue waiting for scheduling.

[0015] A network data transmission system, comprising: The queue allocation module is used to divide the request data into the primary channel queue and the secondary channel queue based on the real-time index; The threshold calculation module is used to obtain real-time network data transmission indicators and use the exponentially weighted moving average algorithm to calculate the instantaneous threshold of each network data transmission indicator as the fast rate threshold; calculate the long-term trend threshold of each network data transmission indicator as the slow rate threshold; and fuse the fast rate threshold and the slow rate threshold to obtain the fused threshold of each network data transmission indicator; The control module is used to monitor the queue length and bandwidth utilization of the primary and secondary channels in real time. If the bandwidth utilization of the primary channel falls below a preset lower limit and the queue length of the secondary channel exceeds a queue length fusion threshold, the secondary channel borrows the bandwidth of the primary channel for data transmission and reception. If the queue length of the primary channel exceeds the queue length fusion threshold, the primary channel reclaims the borrowed primary channel bandwidth from the secondary channel.

[0016] Furthermore, Also includes: The path switching module is used to detect the bandwidth, delay, and packet loss rate information of the transmission equipment in real time when transmitting data on the network. When it is detected that the bandwidth, delay, or packet loss rate exceeds the threshold, the transmission path switching is triggered.

[0017] Furthermore, Also includes: A backup path generation module is used to build a historical database using the bandwidth, delay, and packet loss rate information, predict network link congestion points based on the historical database, and set backup paths based on the link congestion points; When it is detected that the bandwidth, delay, or packet loss rate exceeds the threshold, the backup path is switched.

[0018] Compared with the prior art, the present disclosure has the following advantages: The present disclosure implements dynamic classification of task priorities by dividing network request data into primary channel queues and secondary channel queues based on real-time indicators. It uses an exponentially weighted moving average algorithm to calculate a fast rate threshold reflecting the instantaneous state of the network and a slow rate threshold reflecting the long-term trend, and then fuses the two into a fused threshold. This constructs a dynamic threshold system to replace the traditional fixed rate threshold. By real-time monitoring of the queue lengths of the primary and secondary channels, the secondary channel is allowed to borrow bandwidth from the primary channel when the bandwidth utilization of the primary channel falls below the preset lower limit and the queue length of the secondary channel exceeds the convergence threshold. The borrowed bandwidth is reclaimed when the queue length of the primary channel exceeds the convergence threshold. This dynamic threshold generation and intelligent bandwidth allocation mechanism can flexibly adjust resource allocation strategies based on the real-time network status (such as bandwidth fluctuations and burst traffic). This not only avoids the defect of fixed thresholds that cannot adapt to dynamic network changes, but also realizes dynamic allocation of network resources through elastic resource borrowing between the primary and secondary channels, thus effectively solving the problem of low resource utilization and inability to dynamically adapt to environmental changes in traditional technologies in complex network environments.

[0019] Other features and advantages of the present disclosure will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present disclosure. The purposes and other advantages of the present disclosure can be realized and obtained by the structures indicated in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 A schematic diagram of the disclosed method is shown; Figure 2 A schematic diagram of the disclosed system is shown; Figure 3 The flowchart of the fusion threshold calculation of the present disclosure is shown. DETAILED DESCRIPTION

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0023] like Figure 1 This is a schematic diagram of the method disclosed herein, and the specific steps are as follows: 1. Step 1: Arrange the request data, including request classification, threshold calculation, and sequence arrangement: 1) Request classification: After receiving the request data, we classify it into two request queues: "primary channel" (important and urgent) and "secondary channel" (delayable), based on information such as source, type, and service agreement, and real-time indicators (including: important and urgent, and deferrable). (For example, live video uploads go through the primary channel, while log uploads go through the secondary channel.) The data structure of the request data includes the source IP address, data type, SLA (Service Level Agreement), etc. The real-time index is a key attribute used to measure the real-time transmission requirements of network request data. It can be obtained through the following methods: 1) Obtained through source IP: Identifies the network address of the data sender, which can be used to prioritize requests from different devices or systems (for example, requests from core devices may be more urgent); 2) By data type: For example, latency-sensitive data such as real-time video and audio streams are usually classified as primary channels, while non-real-time data such as log uploads and file transfers are classified as secondary channels. 3) Obtained through SLA (Service Level Agreement): The performance indicators of data transmission (such as latency limit, bandwidth guarantee, etc.) are agreed upon, which directly reflects the real-time requirements of data transmission.

[0024] Specifically, a lightweight ML model (such as a random forest model) can be used to predict the criticality level of requests and output the request queue allocation marked as primary and secondary.

[0025] 2) Threshold calculation: Two thresholds are calculated using "fast rate indicators" (real-time traffic flow, queue length, etc., which reflect the instantaneous system state and are sensitive to sudden changes) and "slow rate indicators" (long-term trends, which smooth out short-term fluctuations). These two thresholds are combined to form a dynamic threshold. If real-time system indicators (such as traffic flow and queue depth) exceed the threshold, control actions such as flow limiting and capacity expansion are taken on the channel. Specifically, the threshold calculation algorithm is as follows Figure 3 As shown, specifically: Initialization parameters, including: fast rate smoothing factor α (large, such as 0.9), slow rate smoothing factor β (small, such as 0.1), initial threshold , , can be set to the historical average or experience value; Obtain the current system's network data transmission indicators Xt (such as delay, packet loss rate, bandwidth utilization, QoS level, packet size, queue length, CPU load, etc.), and calculate and update the fast rate threshold and slow rate threshold based on the network data transmission indicators; the formula is:

[0026]

[0027] Select the maximum value of the fast rate threshold and the slow rate threshold, or perform a weighted fusion of the two to obtain a dynamic threshold. The formula is: If the network's real-time response requirement is higher than the threshold, the maximum value of the fast rate threshold and the slow rate threshold is used as the fusion threshold. The formula is: , ) If the network's real-time response requirement is lower than the threshold, the fast rate threshold and the slow rate threshold are weighted and fused to obtain the fusion threshold, whose formula is: + (1- )

[0028] in, 、 and Adjust according to the real-time status of the system, such as reducing the false alarm rate when the false alarm rate is high. .

[0029] 3) Use a hybrid queue management strategy to arrange request queues (by priority group or deadline): Specifically, this strategy combines priority preemption with fairness constraints. The overall fairness strategy is to prioritize meeting the minimum network resource requirements and then allocate the remaining network resources. For the aggregation layer / edge layer: Group by priority. The high-priority group can preempt the bandwidth of the low-priority group. Tasks within the group are executed in a round-robin manner to occupy network bandwidth. For the perception layer / execution layer: prioritize tasks with close deadlines, and interrupt current tasks when new tasks are more urgent.

[0030] 2. Step 2: Before data transmission between the data orchestration and network layer transmission, the cross-layer coordinator monitors the bandwidth utilization and queue length of the primary and secondary channels. Based on the monitoring results, corresponding adjustment measures are taken (such as borrowing bandwidth from the secondary channel when the primary channel is idle and reclaiming bandwidth when the primary channel is congested). The rate controller (WFQ on the primary channel and token bucket on the secondary channel) is used to adjust the transmission rate of the primary and secondary channels. Specifically: 1) The corresponding adjustment measures based on the monitoring situation of the cross-layer coordinator are: When the primary channel is idle (e.g., the idle rate is >30% for 5 seconds), the secondary channel can use the bandwidth of the primary channel to transmit data. When the main channel is busy (for example, there are more than 100 queued requests), the borrowed bandwidth is immediately reclaimed to prioritize the main channel.

[0031] Using a cross-layer coordinator can solve the following problems: 1. In a wireless narrowband communication network environment where data processing uses priority communication, high-priority tasks may occupy bandwidth resources for a long time. This will cause low-priority but urgent requests to be blocked, resulting in the urgent tasks not being responded to in a timely manner.

[0032] 2. Due to the complexity of communication networks, a mechanism is usually adopted in which primary and secondary rate channels operate independently in the communication network to achieve controllability of each segment and layer of the network. However, this communication technology lacks cross-layer optimization, resulting in low overall bandwidth utilization and wasted bandwidth resources.

[0033] 2) Use the rate controller (WFQ on the primary channel and token bucket on the secondary channel) to adjust the transmission rate of the primary and secondary channels. Specifically: For the main channel: use "weighted fair queue" to ensure that important data has at least a fixed transmission rate and is not delayed; For the secondary channel: Use the "token bucket" mechanism to handle burst traffic, allowing temporary over-transmission of data without exceeding the limit to avoid overwhelming the primary channel.

[0034] 3. Step 3: During data transmission, use SDN-based bandwidth fast switching technology to dynamically optimize and manage network traffic (such as real-time monitoring, dynamic path switching, and pre-calculated backup paths) 1) Real-time monitoring: Check the bandwidth, latency, and packet loss rate of each link at any time, and switch links within milliseconds if problems are detected. 2) Dynamic path switching: Set bandwidth usage thresholds, triggering switchover when they are exceeded, and calculate backup paths in advance to reduce latency. Specifically, when switching paths, the "add first, then delete" strategy is adopted, first enabling the new path and then shutting down the old path to avoid data loss and ensure smooth transmission; 3) Intelligent prediction: Use historical data to predict congestion points and plan alternative routes in advance to avoid congestion.

[0035] like Figure 2 This is a schematic diagram of the disclosed system, including: 1) Data orchestration layer, which includes queue allocation module, threshold calculation module, and queue management module; The queue allocation module is used to predict the request criticality level based on data request metadata (including source IP, data type, SLA agreement, etc.) through a random forest model, and divide the request data into the primary channel queue or secondary channel queue; The threshold calculation module is used to obtain real-time network data transmission indicators. It uses the exponentially weighted moving average algorithm to calculate the instantaneous threshold of each network data transmission indicator as the fast rate threshold; calculate the long-term trend threshold of each network data transmission indicator as the slow rate threshold; and fuse the fast rate threshold and the slow rate threshold to obtain the fused threshold of each network data transmission indicator. The queue management module is used to manage queues. For the convergence layer and edge layer in the network architecture, the request data in the primary channel queue and secondary channel queue are sorted by priority group. The high-priority group can preempt the bandwidth resources of the low-priority group. The tasks within the group use a round-robin scheduling mechanism to occupy the network bandwidth. For the perception layer and execution layer in the network architecture, the request data in the main channel queue and the secondary channel queue are sorted in ascending order according to the task deadline. When the deadline of the new task is earlier than the current executing task, the preemption mechanism is triggered and the interrupted task is re-added to the tail of the queue waiting for scheduling.

[0036] 2) Data transmission layer, including the cross-layer coordinator module and the rate controller module; The cross-layer coordinator module is used to monitor the bandwidth utilization and queue length of the primary and secondary channels and perform corresponding adjustment measures based on the monitoring results; The rate controller module is used to adjust the transmission rate of the primary and secondary channels.

[0037] 3) Network layer, which is used to dynamically optimize and manage network traffic based on SDN-based bandwidth fast switching technology, including a path switching module and a backup path generation module.

[0038] The path switching module is used to detect the bandwidth, delay, and packet loss rate information of the transmission equipment in real time when transmitting data on the network. When it is detected that the bandwidth, delay, or packet loss rate exceeds the threshold, the transmission path switching is triggered.

[0039] The backup path generation module is used to build a historical database using bandwidth, delay, and packet loss rate information, predict network link congestion points based on the historical database, and set backup paths based on the link congestion points.

[0040] Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A network data transmission method, characterized in that: include: Obtain network request data and divide the request data into primary channel queue and secondary channel queue based on real-time indicators; Obtain real-time network data transmission indicators, use the exponentially weighted moving average algorithm to calculate the instantaneous threshold of each network data transmission indicator as the fast rate threshold; calculate the long-term trend threshold of each network data transmission indicator as the slow rate threshold; and fuse the fast rate threshold and the slow rate threshold to obtain the fusion threshold of each network data transmission indicator; The queue length and bandwidth utilization of the primary and secondary channels are monitored in real time. If the primary channel bandwidth utilization falls below the preset lower limit and the secondary channel queue length exceeds the queue length convergence threshold, the secondary channel borrows the primary channel bandwidth for data transmission and reception. If the primary channel queue length exceeds the queue length convergence threshold, the primary channel reclaims the borrowed primary channel bandwidth from the secondary channel.

2. A network data transmission method according to claim 1, characterized in that: The method of dividing the request data into the primary channel queue and the secondary channel queue comprises: Input the requested data into the random forest model and output the classification results of the primary channel or secondary channel.

3. A network data transmission method according to claim 1, characterized in that: The instantaneous threshold of each network data transmission indicator is calculated as the fast rate threshold; the long-term trend threshold of each network data transmission indicator is calculated as the slow rate threshold; the calculation formula is: in, is the fast rate threshold at time t; is the network data transmission indicator value at time t; is the fast rate threshold at time t-1; is the slow rate threshold at time t; is the slow rate threshold at time t-1; α is the fast rate smoothing factor, is the slow rate smoothing factor, α> .

4. A network data transmission method according to claim 1, characterized in that: The fusion threshold of each network data transmission indicator is obtained by fusion of the fast rate threshold and the slow rate threshold, including: If the network's real-time response requirement is higher than the threshold, the maximum value between the fast rate threshold and the slow rate threshold is used as the fusion threshold; If the network's real-time response requirement is lower than the threshold, the fast rate threshold and the slow rate threshold are weighted and fused to obtain the fusion threshold, whose formula is: +（1- ） in, , is the fusion threshold, is the fast rate threshold, is the slow rate threshold.

5. A network data transmission method according to claim 1, characterized in that: The weighted fair queue algorithm is used to control traffic on the primary channel, and the token bucket algorithm is used to control traffic on the secondary channel.

6. A network data transmission method according to claim 1, characterized in that: Also includes: During network data transmission, the status information of the network transmission link is obtained in real time. When congestion of the network transmission link is detected, the transmission path switching is triggered; when the path is switched, the new path is enabled first and then the old path is closed.

7. A network data transmission method according to claim 1, characterized in that: For the convergence layer and edge layer in the network architecture, the request data in the primary and secondary channel queues are sorted by priority. The high-priority group can preempt the bandwidth resources of the low-priority group. The tasks within the group use a round-robin scheduling mechanism to occupy the network bandwidth. For the perception layer and execution layer in the network architecture, the request data in the main channel queue and the secondary channel queue are sorted in ascending order according to the task deadline. When the deadline of the new task is earlier than the current executing task, the preemption mechanism is triggered and the interrupted task is re-added to the tail of the queue waiting for scheduling.

8. A network data transmission system, characterized in that: include: The queue allocation module is used to divide the request data into the primary channel queue and the secondary channel queue based on the real-time index; The threshold calculation module is used to obtain real-time network data transmission indicators and use the exponentially weighted moving average algorithm to calculate the instantaneous threshold of each network data transmission indicator as the fast rate threshold; calculate the long-term trend threshold of each network data transmission indicator as the slow rate threshold; and fuse the fast rate threshold and the slow rate threshold to obtain the fused threshold of each network data transmission indicator; The control module is used to monitor the queue length and bandwidth utilization of the primary and secondary channels in real time. If the bandwidth utilization of the primary channel falls below a preset lower limit and the queue length of the secondary channel exceeds a queue length fusion threshold, the secondary channel borrows the bandwidth of the primary channel for data transmission and reception. If the queue length of the primary channel exceeds the queue length fusion threshold, the primary channel reclaims the borrowed primary channel bandwidth from the secondary channel.

9. The system according to claim 8, characterized in that Also includes: The path switching module is used to detect the bandwidth, delay, and packet loss rate information of the transmission equipment in real time when transmitting data on the network. When it is detected that the bandwidth, delay, or packet loss rate exceeds the threshold, the transmission path switching is triggered.

10. The system according to claim 9, characterized in that Also includes: A backup path generation module is used to build a historical database using the bandwidth, delay, and packet loss rate information, predict network link congestion points based on the historical database, and set backup paths based on the link congestion points; When it is detected that the bandwidth, delay, or packet loss rate exceeds the threshold, the backup path is switched.

Images