Internet data transmission method and system based on large-scale node scheduling

Abstract

The invention discloses an internet data transmission method and system based on large-scale node scheduling, and the method comprises the steps: collecting network flow data, node load data and data transmission demands, and carrying out the preprocessing of the data; performing feature extraction on the preprocessed data to obtain feature vectors; iterative training is carried out on the decision network through the feature vectors, and training is stopped until the loss function L converges; using the trained decision network to carry out scheduling decision so as to adjust a data transmission path; by improving the feature extraction process and designing the decision network, the adaptive capacity to the dynamic network environment is enhanced, the data transmission path can be effectively adjusted, and the data transmission efficiency and the overall performance of the network are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data transmission, and in particular to an Internet data transmission method and system based on large-scale node scheduling. Background Art

[0002] In today's large-scale Internet environment, data transmission technology continues to evolve to cope with the growing traffic and increasingly complex network structure. Early Internet data transmission methods mainly rely on static routing and scheduling strategies, which are based on fixed network topologies and preset rules. Although this method can effectively transmit data when the network load is relatively stable, its flexibility and adaptability are insufficient when facing network traffic fluctuations, node load changes and other dynamic factors. With the expansion of network scale and the diversification of application requirements, traditional static scheduling methods have gradually exposed their limitations in being unable to cope with dynamic loads and complex network environments. Summary of the invention

[0003] In order to overcome the defects in the above-mentioned prior art, the present invention provides an Internet data transmission method based on large-scale node scheduling, including: collecting network traffic data, node load data and data transmission requirements, and preprocessing them; performing feature extraction on the preprocessed data to obtain a feature vector; iteratively training a decision network through the feature vector until the loss function L converges, then stopping the training; and using the trained decision network to make scheduling decisions to adjust the data transmission path.

[0004] As a preferred solution of the Internet data transmission method based on large-scale node scheduling described in the present invention, it includes: network traffic data including inbound and outbound traffic, data packet size, and transmission delay of each node; node load data including node CPU utilization, memory utilization, and network bandwidth utilization; data transmission requirements including priority, timeliness requirements, and data volume of data packet transmission.

[0005] As a preferred solution of the Internet data transmission method based on large-scale node scheduling described in the present invention, it is characterized in that: it includes: performing preliminary feature extraction on the preprocessed data through a convolutional autoencoder to generate low-dimensional semantic features, and performing dimensionality reduction and compression on the preprocessed data; using deep separable convolution and jump connection strategies to convolve and downsample the compressed generated low-dimensional semantic features to obtain feature vectors.

[0006] As a preferred solution of the Internet data transmission method based on large-scale node scheduling described in the present invention, wherein: the decision network includes a prediction unit and a decision unit; the prediction unit includes a first input layer, an LSTM layer, a first gated loop layer, a second gated loop layer and a first fully connected layer; the decision unit includes a second input layer, a second fully connected layer, a third fully connected layer and an output layer; the second input layer is connected to the first fully connected layer; the feature vector enters the prediction unit through the first input layer, and outputs the flow, load, and demand prediction results; the decision unit adjusts the data transmission path, load distribution and task scheduling strategy based on the flow, load and demand prediction results.

[0007] As a preferred solution of the Internet data transmission method based on large-scale node scheduling described in the present invention, it includes: calculating the target value y of the decision network of the decision network ′ :

[0008] y ′ =r+α×max(next_state)

[0009] Use the target value y ′ Iteratively train the decision network until the target loss function value converges. The target loss function L is:

[0010]

[0011] Where r is the immediate reward obtained by taking an action in the current state, α is the discount factor, which represents the weight of future rewards; max(next_state) represents the maximum Q value of all actions for the next state next_state in the target decision network; μ is a hyperparameter, and y is the predicted value of the decision network.

[0012] As a preferred solution of the Internet data transmission system based on large-scale node scheduling described in the present invention, it includes: a data acquisition module, which is configured to collect network traffic data, node load data and data transmission requirements, and preprocess them; a feature extraction module, which is configured to perform feature extraction on the preprocessed data to obtain a feature vector; a training module, which is configured to perform iterative training of the decision network through the feature vector until the loss function L converges, and then stop the training; a decision module, which is configured to perform scheduling decisions using the trained decision network to adjust the data transmission path.

[0013] As a preferred solution of the Internet data transmission system based on large-scale node scheduling described in the present invention, it includes: network traffic data including inbound and outbound traffic, data packet size, and transmission delay of each node; node load data including node CPU utilization, memory utilization, and network bandwidth utilization; data transmission requirements including priority, timeliness requirements, and data volume of data packet transmission.

[0014] As a preferred solution of the Internet data transmission system based on large-scale node scheduling described in the present invention, the feature extraction module is specifically configured to execute: performing preliminary feature extraction on the preprocessed data through a convolutional autoencoder to generate low-dimensional semantic features, and performing dimensionality reduction and compression on the features; and convolving and down-sampling the compressed generated low-dimensional semantic features using a deep separable convolution and skip connection strategy to obtain a feature vector.

[0015] As a preferred solution of the Internet data transmission system based on large-scale node scheduling described in the present invention, wherein: the training module is specifically configured to execute: the decision network includes a prediction unit and a decision unit; the prediction unit includes a first input layer, an LSTM layer, a first gated loop layer, a second gated loop layer and a first fully connected layer; the decision unit includes a second input layer, a second fully connected layer, a third fully connected layer and an output layer; the second input layer is connected to the first fully connected layer; the feature vector enters the prediction unit through the first input layer, and outputs the traffic, load, and demand prediction results; the decision unit adjusts the data transmission path, load distribution and task scheduling strategy based on the traffic, load and demand prediction results.

[0016] As a preferred solution of the Internet data transmission system based on large-scale node scheduling described in the present invention, the decision module is specifically configured to execute: calculating the target value y of the decision network ′ :

[0017] y ′ =r+α×max(next_state)

[0018] Use the target value y ′ Iteratively train the decision network until the target loss function value converges. The target loss function L is:

[0019]

[0020] Where r is the immediate reward obtained by taking an action in the current state, α is the discount factor, which represents the weight of future rewards; max(next_state) represents the maximum Q value of all actions for the next state next_state in the target decision network; μ is a hyperparameter, and y is the predicted value of the decision network.

[0021] The beneficial effects of the present invention are as follows: the present invention enhances the adaptability to dynamic network environments by improving the feature extraction process and designing a decision network, can effectively adjust the data transmission path, and improves the efficiency of data transmission and the overall performance of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0023] Figure 1 A schematic diagram of a flow chart of an Internet data transmission method based on large-scale node scheduling according to a first embodiment of the present invention;

[0024] Figure 2 It is a schematic diagram of the structure of the decision network described in the first embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0026] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0027] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0028] The present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0029] At the same time, in the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "upper, lower, inner and outer" are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first, second or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0030] In the present invention, unless otherwise clearly specified and limited, the terms "install, connect, connect" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integral connection; it can also be a mechanical connection, an electrical connection or a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0031] Example 1

[0032] Reference Figure 1-2 , which is the first embodiment of the present invention, provides an Internet data transmission method based on large-scale node scheduling, including:

[0033] S1: Collect network traffic data, node load data, and data transmission requirements, and pre-process them.

[0034] Collect network traffic data from switches or routers, including inbound and outbound traffic, packet size, and transmission delay for each node;

[0035] Use system monitoring tools (such as Prometheus) to collect node load data, including the node's CPU usage, memory usage, and network bandwidth utilization;

[0036] The task management system records data transmission requirements, including the priority of data packet transmission, timeliness requirements (i.e. task deadline, delay duration, etc.), and data volume (i.e. the amount of data involved in each transmission task).

[0037] Furthermore, the network traffic data, node load data and data transmission requirements are preprocessed as follows:

[0038] (1) Remove noise data and repair missing values;

[0039] (2) Standardize the data to a uniform range to facilitate analysis and processing by subsequent decision-making networks.

[0040] S2: Extract features from the preprocessed data to obtain feature vectors.

[0041] (1) Perform preliminary feature extraction on the preprocessed data through a convolutional autoencoder to generate low-dimensional semantic features, and then perform dimensionality reduction and compression on them;

[0042] The convolutional autoencoder includes an encoder and a decoder, and a residual connection is established between each layer of the encoder and the corresponding layer of the decoder to obtain more low-dimensional semantic features.

[0043] Furthermore, principal component analysis (PCA) can be used to reduce the dimension and compress these features to reduce redundancy.

[0044] (2) The compressed low-dimensional semantic features are convolved and downsampled using the depthwise separable convolution and skip connection strategy with a downsampling stride of 2 to obtain the feature vector, thereby effectively reducing the amount of computation while maintaining the feature information.

[0045] S3: Iteratively train the decision network through the feature vector until the loss function L converges, then stop training.

[0046] Reference Figure 2 , the decision network includes a prediction unit 100 and a decision unit 200;

[0047] The prediction unit 100 includes a first input layer 101, an LSTM layer 102, a first gated recurrent layer 103, a second gated recurrent layer 104 and a first fully connected layer 105;

[0048] Specifically, the number of neurons in the LSTM layer 102 is 64, and the return sequence is True (maintaining the output of the time step), and the Dropout is 0.2 (preventing overfitting); the number of neurons in the first gated recurrent layer 103 is 64, and the return sequence is True (maintaining the output of the time step), and the Dropout is 0.2 (preventing overfitting); the number of neurons in the second gated recurrent layer 104 is 32, and the return sequence is False (outputting the state of the last time step), and the Dropout is 0.2 (preventing overfitting); the number of neurons in the first fully connected layer 105 is 16, and the ReLU activation function is used.

[0049] Preferably, the present application greatly simplifies the network structure by introducing a gated recurrent layer, thereby reducing the computational burden and improving prediction efficiency.

[0050] The decision unit 200 includes a second input layer 201, a second fully connected layer 202, a third fully connected layer 203 and an output layer 204; the second input layer 201 is connected to the first fully connected layer 105;

[0051] Specifically, the number of neurons in the second fully connected layer 202 is 64, and the ReLU activation function is adopted; the number of neurons in the third fully connected layer 203 is 32, and the ReLU activation function is adopted; the number of neurons in the output layer 204 is num_actions (the number of actions), and the Linear activation function is adopted (the Q value of each action is output).

[0052] The feature vector enters the prediction unit 100 through the first input layer 101 and outputs the flow, load and demand prediction results; the decision unit 200 adjusts the data transmission path, load distribution and task scheduling strategy based on the flow, load and demand prediction results.

[0053] Furthermore, in order to improve the prediction accuracy of the decision network, the decision network needs to be optimized and trained. The specific steps are as follows:

[0054] Calculate the target value y of the decision network ′ :

[0055] y ′ =r+α×max(next_state)

[0056] Use the target value y ′ Iteratively train the decision network until the target loss function value converges. The target loss function L is:

[0057]

[0058] Where r is the immediate reward obtained by taking an action in the current state, α is the discount factor, which represents the weight of future rewards; max(next_state) represents the maximum Q value of all actions for the next state next_state in the target decision network; μ is a hyperparameter that determines the switching point of the linear and square parts of the loss function, and y is the predicted value of the decision network.

[0059] Preferably, this embodiment combines DQN to predict Q values ​​and improves the stability of training through the Huber loss function, thereby improving the robustness of the decision network to outliers.

[0060] S4: Use the trained decision network to make scheduling decisions to adjust the data transmission path.

[0061] Taking data center network traffic scheduling as an example, the trained decision network is used to make scheduling decisions. The input data is as follows:

[0062] (1) Network traffic data:

[0063] Node A: Inbound traffic: 1.4 Gbps, outbound traffic: 1.3 Gbps, packet size: 500 MB, transmission delay: 28 ms;

[0064] Node B: Inbound traffic: 1.1 Gbps, outbound traffic: 1.0 Gbps, packet size: 500 MB, transmission delay: 20 ms;

[0065] Node C: Inbound traffic: 0.9 Gbps, outbound traffic: 0.85 Gbps, packet size: 500 MB, transmission delay: 30 ms.

[0066] (2) Node load data:

[0067] Node A: CPU usage: 85%, memory usage: 75%, bandwidth usage: 90%;

[0068] Node B: CPU usage: 60%, memory usage: 55%, bandwidth usage: 70%;

[0069] Node C: CPU utilization: 55%, memory utilization: 50%, bandwidth utilization: 70%.

[0070] (3) Data transmission requirements:

[0071] Data packet 1: Priority: high, timeliness requirement: 20ms, data volume: 500MB;

[0072] Data packet 2: Priority: medium, timeliness requirement: 30ms, data volume: 500MB;

[0073] Data packet 3: Priority: low, timeliness requirement: 40ms, data volume: 500MB.

[0074] Furthermore, based on the input data, the traffic demand and load changes of each node in the next 5 minutes are predicted. The prediction results are as follows:

[0075] Node A:

[0076] Forecasted traffic: Inbound traffic increases to 1.5Gbps, outbound traffic increases to 1.4Gbpsp;

[0077] Predicted load: CPU utilization is expected to increase to 90%, memory utilization to 80%, and bandwidth utilization to 95%;

[0078] Node B:

[0079] Forecasted traffic: Inbound traffic is maintained at 1.1 Gbps, and outbound traffic is maintained at 1.0 Gbps;

[0080] Predicted load: CPU utilization is maintained at 60%, memory utilization is maintained at 55%, and bandwidth utilization is maintained at 70%;

[0081] Node C:

[0082] Forecasted traffic: Inbound traffic increases to 1.0 Gbps, outbound traffic increases to 0.95 Gbps;

[0083] Predicted load: CPU utilization is maintained at 55%, memory utilization is maintained at 50%, and bandwidth utilization is maintained at 70%.

[0084] Demand Forecast:

[0085] Packet 1 (high priority): It is recommended to be routed to Node B (because Node B has lower latency and lighter load, meeting the 20ms timeliness requirement);

[0086] Packet 2 (medium priority): It is recommended to be routed to node C (the latency of node C is slightly higher, but within 30ms, and the load is moderate);

[0087] Data packet 3 (low priority): It is recommended to route to node A (although node A has a high load, the timeliness requirement of data packet 3 is relaxed);

[0088] Use the prediction results to formulate a scheduling strategy, taking into account the node load and data transmission requirements, optimize traffic distribution, and adjust the strategy as follows:

[0089] Adjustments based on forecast results:

[0090] Packet 1:

[0091] Select node: Node B (predicted delay 18ms, actual 18ms, appropriate load);

[0092] Traffic shaping: Optimize the traffic routing of node B and give priority to high-priority data packets 1;

[0093] Load distribution: Increase the processing priority of data packet 1 on node B to ensure that timeliness requirements are met.

[0094] Packet 2:

[0095] Select node: Node C (predicted delay 29ms, actual 29ms, moderate load);

[0096] Traffic shaping: Traffic routing at node C is optimized to ensure that the medium priority of packet 2 is processed;

[0097] Load distribution: Adjust the load balance of node C so that data packet 2 can obtain appropriate processing resources.

[0098] Packet 3:

[0099] Select node: Node A (predicted delay 35ms, actual 36ms. Although the load is high, the timeliness requirement of data packet 3 is loose).

[0100] Traffic shaping: Optimize traffic routing on node A and allocate more bandwidth to data packet 3;

[0101] Load distribution: Perform load balancing adjustments on node A to reduce resource usage of low-priority data packets to alleviate load pressure.

[0102] It can be seen that the decision network of the present application accurately predicts the traffic changes, load status and data transmission requirements of the nodes, effectively optimizes the data transmission path, load distribution and task scheduling, and ensures that the needs of data packets of different priorities are met.

[0103] Example 2

[0104] This embodiment is different from the first embodiment in that it provides an Internet data transmission system based on large-scale node scheduling, including:

[0105] The data collection module is configured to collect network traffic data, node load data and data transmission requirements, and pre-process them; network traffic data includes the inbound and outbound traffic, data packet size, and transmission delay of each node; node load data includes the node's CPU utilization, memory utilization, and network bandwidth utilization; data transmission requirements include the priority, timeliness requirements, and data volume of data packet transmission.

[0106] The feature extraction module is configured to perform feature extraction on the preprocessed data to obtain a feature vector; the feature extraction module is specifically configured to perform: preliminary feature extraction on the preprocessed data through a convolutional autoencoder to generate low-dimensional semantic features, and perform dimensionality reduction and compression on the features; and convolute and downsample the compressed generated low-dimensional semantic features using a deep separable convolution and skip connection strategy to obtain a feature vector.

[0107] A training module is configured to perform iterative training of the decision network through the feature vector until the loss function L converges, and then stop the training;

[0108] The decision module is configured to perform scheduling decisions using the trained decision network to adjust the data transmission path.

[0109] The training module is specifically configured to execute:

[0110] The decision network includes a prediction unit 100 and a decision unit 200;

[0111] The prediction unit 100 includes a first input layer 101, an LSTM layer 102, a first gated recurrent layer 103, a second gated recurrent layer 104 and a first fully connected layer 105;

[0112] The decision unit 200 includes a second input layer 201, a second fully connected layer 202, a third fully connected layer 203 and an output layer 204; the second input layer 201 is connected to the first fully connected layer 105;

[0113] The feature vector enters the prediction unit 100 through the first input layer 101 and outputs the flow, load and demand prediction results; the decision unit 200 adjusts the data transmission path, load distribution and task scheduling strategy based on the flow, load and demand prediction results.

[0114] The decision module is specifically configured to execute:

[0115] Calculate the target value y of the decision network ′ :

[0116] y ′ =r+α×max(next_state)

[0117] Use the target value y ′ Iteratively train the decision network until the target loss function value converges. The target loss function L is:

[0118]

[0119] Where r is the immediate reward obtained by taking an action in the current state, α is the discount factor, which represents the weight of future rewards; max(next_state) represents the maximum Q value of all actions for the next state next_state in the target decision network; μ is a hyperparameter, and y is the predicted value of the decision network.

[0120] It should be appreciated that embodiments of the present invention may be implemented or enforced by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The method may be implemented in a computer program using standard programming techniques - including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner - according to the methods and drawings described in the specific embodiments. Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program may be implemented in an assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed ASIC for this purpose.

[0121] Furthermore, the operations of the processes described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors, by hardware, or a combination thereof. The computer program includes a plurality of instructions that may be executed by one or more processors.

[0122] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or part thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described herein includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques described in the present invention, the present invention also includes the computer itself. The computer program can be applied to input data to perform the functions described herein, thereby converting the input data to generate output data stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the converted data represents a physical and tangible object, including a specific visual depiction of the physical and tangible object produced on a display.

[0123] As used in this application, the terms "component", "module", "system", etc. are intended to refer to a computer-related entity, which can be hardware, firmware, a combination of hardware and software, software, or software in operation. For example, a component can be, but is not limited to: a process running on a processor, a processor, an object, an executable file, a thread in execution, a program, and / or a computer. As an example, an application running on a computing device and the computing device can both be components. One or more components can exist in a process and / or thread in execution, and a component can be located in a computer and / or distributed between two or more computers. In addition, these components can be executed from various computer-readable media having various data structures thereon. These components can communicate in a local and / or remote process manner, such as according to a signal having one or more data packets (e.g., data from a component that interacts with another component in a local system, a distributed system, and / or interacts with other systems in a signal manner through a network such as the Internet).

[0124] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An Internet data transmission method based on large-scale node scheduling, characterized in that: include: Collect network traffic data, node load data, and data transmission requirements, and pre-process them; Perform feature extraction on the preprocessed data to obtain feature vectors; Iteratively training the decision network using the feature vector until the loss function L converges, then stopping the training; The trained decision network is used to make scheduling decisions to adjust the data transmission path.

2. The Internet data transmission method based on large-scale node scheduling according to claim 1, characterized in that: include: Network traffic data includes inbound and outbound traffic, packet size, and transmission delay of each node; Node load data includes the node’s CPU usage, memory usage, and network bandwidth utilization; Data transmission requirements include the priority of data packet transmission, timeliness requirements, and data volume.

3. The Internet data transmission method based on large-scale node scheduling according to claim 1 or 2, characterized in that: include: The convolutional autoencoder is used to perform preliminary feature extraction on the preprocessed data, generate low-dimensional semantic features, and perform dimensionality reduction and compression on them; The compressed low-dimensional semantic features are convolved and downsampled using depthwise separable convolution and skip connection strategies to obtain feature vectors.

4. The Internet data transmission method based on large-scale node scheduling as claimed in claim 3, characterized in that: The decision network includes a prediction unit and a decision unit; The prediction unit includes a first input layer, an LSTM layer, a first gated recurrent layer, a second gated recurrent layer and a first fully connected layer; The decision unit includes a second input layer, a second fully connected layer, a third fully connected layer and an output layer; the second input layer is connected to the first fully connected layer; The feature vector enters the prediction unit through the first input layer and outputs the flow, load and demand prediction results; the decision unit adjusts the data transmission path, load distribution and task scheduling strategy based on the flow, load and demand prediction results.

5. The Internet data transmission method based on large-scale node scheduling as claimed in claim 4, characterized in that: include: Calculate the target value y of the decision network ′ : y ′ =r+α×max(next_state) Use the target value y ′ Iteratively train the decision network until the target loss function value converges. The target loss function L is: In the formula, r is the immediate reward obtained by taking an action in the current state, and α is the discount factor, which represents the weight of future rewards; max(next_state) represents the maximum Q value of all actions for the next state next_state in the target decision network; μ is a hyperparameter, and y is the predicted value of the decision network.

6. An Internet data transmission system based on large-scale node scheduling, characterized in that: include: The data collection module is configured to collect network flow data, node load data and data transmission requirements, and pre-process them; A feature extraction module is configured to perform feature extraction on the preprocessed data to obtain a feature vector; A training module is configured to perform iterative training on the decision network through the feature vector until the loss function L converges, and then stop the training; The decision module is configured to perform scheduling decisions using the trained decision network to adjust the data transmission path.

7. The Internet data transmission system based on large-scale node scheduling as claimed in claim 6, characterized in that: include: Network traffic data includes inbound and outbound traffic, packet size, and transmission delay of each node; Node load data includes the node’s CPU usage, memory usage, and network bandwidth utilization; Data transmission requirements include the priority of data packet transmission, timeliness requirements, and data volume.

8. The Internet data transmission system based on large-scale node scheduling according to claim 6 or 7, characterized in that: The feature extraction module is specifically configured to perform: The convolutional autoencoder is used to perform preliminary feature extraction on the preprocessed data, generate low-dimensional semantic features, and perform dimensionality reduction and compression on them; The compressed low-dimensional semantic features are convolved and downsampled using depthwise separable convolution and skip connection strategies to obtain feature vectors.

9. The Internet data transmission system based on large-scale node scheduling as claimed in claim 8, characterized in that: The training module is specifically configured to perform: The decision network includes a prediction unit and a decision unit; The prediction unit includes a first input layer, an LSTM layer, a first gated recurrent layer, a second gated recurrent layer and a first fully connected layer; The decision unit includes a second input layer, a second fully connected layer, a third fully connected layer and an output layer; the second input layer is connected to the first fully connected layer; The feature vector enters the prediction unit through the first input layer and outputs the flow, load and demand prediction results; the decision unit adjusts the data transmission path, load distribution and task scheduling strategy based on the flow, load and demand prediction results.

10. The Internet data transmission system based on large-scale node scheduling according to claim 9, characterized in that: The decision module is specifically configured to perform: Calculate the target value y of the decision network ′ : y ′ =r+α×max(next_state) Use the target value y ′ Iteratively train the decision network until the target loss function value converges. The target loss function L is: In the formula, r is the immediate reward obtained by taking an action in the current state, and α is the discount factor, which represents the weight of future rewards; max(next_state) represents the maximum Q value of all actions for the next state next_state in the target decision network; μ is a hyperparameter, and y is the predicted value of the decision network.

Images