Data packet transmission method for heterogeneous transmission network and related device

Abstract

The data packet transmission method and related device of the heterogeneous transmission network provided by the embodiments of the present application, the heterogeneous transmission network comprising a plurality of transmission links, the method comprising: firstly, obtaining a link state feature sequence of each transmission link in a preset scheduling time period; then, inputting all the link state feature sequences into a link scheduling model for data processing to obtain a link selection probability matrix of the plurality of transmission links; finally, determining at least one target transmission link in the next preset scheduling time period from the plurality of transmission links based on the link selection probability matrix, and using the target transmission link for data packet transmission, thereby further improving the success reliability of data packet transmission in the heterogeneous transmission network.

Description

Technical Field

[0001] This application relates to the field of data communication technology, and in particular to data packet transmission methods and related equipment for heterogeneous transmission networks. Background Technology

[0002] With the rapid development of multi-interface technology, heterogeneous transmission networks with multi-link transmission are gradually becoming an important part of data communication. Simultaneous data transmission through multiple interfaces and paths can significantly improve network bandwidth utilization and transmission reliability. However, since the number of transmission links and data packets in heterogeneous transmission networks is usually large, selecting a suitable transmission link for each data packet is a complex problem.

[0003] In related technologies, transmission links are typically pre-assigned to each data packet with different transmission requirements based on the statistical transmission capacity of different transmission links, and then the data packets are transmitted. However, since the real-time transmission capacity of the actual transmission links changes dynamically, using the original transmission link allocation for data packet transmission can easily lead to situations where the actual transmission capacity of the transmission links cannot meet the transmission requirements of the data packets, resulting in insufficient reliability of data packet transmission. Summary of the Invention

[0004] This application provides a data packet transmission method and related equipment for heterogeneous transmission networks, which can improve transmission reliability when transmitting data packets in heterogeneous transmission networks.

[0005] To achieve the above objectives, a first aspect of this application proposes a data packet transmission method for a heterogeneous transmission network, wherein the heterogeneous transmission network includes multiple transmission links, and the method includes:

[0006] Obtain the link status feature sequence of each transmission link within a preset scheduling time period;

[0007] All the link state feature sequences are input into the link scheduling model for data processing to obtain the link selection probability matrix of multiple transmission links;

[0008] Based on the link selection probability matrix, at least one target transmission link is determined from the multiple transmission links for the next preset scheduling time period, and the target transmission link is used for data packet transmission.

[0009] In some embodiments, the preset scheduling time period includes multiple time slots, and obtaining the link state feature sequence of each transmission link within the preset scheduling time period includes:

[0010] Obtain the link status information of each transmission link in each time slot;

[0011] The link status information of each transmission link is concatenated to obtain a link status information sequence;

[0012] The link state information sequence is subjected to attention processing to obtain the link state feature sequence.

[0013] In some embodiments, the step of inputting all the link state feature sequences into the link scheduling model for data processing to obtain a link selection probability matrix for multiple transmission links includes:

[0014] All the link state feature sequences are input into the link scheduling model for data processing to obtain the link selection probability of each of the transmission links in each time slot;

[0015] The multiple link selection probabilities of each transmission link are concatenated to obtain the link selection probability sequence;

[0016] The link selection probability matrix is ​​obtained by concatenating multiple link selection probability sequences.

[0017] In some embodiments, the preset scheduling time period includes multiple time slots, and determining at least one target transmission link in the next preset scheduling time period from multiple transmission links based on the link selection probability matrix includes:

[0018] Based on the link selection probability corresponding to multiple transmission links in each time slot, a sampling method selection probability is generated;

[0019] Based on the sampling method selection probability, a target sampling method is selected from binary Bernoulli sampling and Monte Carlo sampling;

[0020] Based on the target sampling method and the link selection probability, the target transmission link corresponding to each time slot is selected one by one from the multiple transmission links.

[0021] In some embodiments, when the target sampling method is Monte Carlo sampling, the step of selecting the target transmission link corresponding to each time slot from the plurality of transmission links one by one based on the target sampling method and the link selection probability includes:

[0022] Based on the multiple link selection probabilities within each time slot, multiple sampling sub-intervals corresponding to the transmission links and a sequence number identifier corresponding to each sampling sub-interval are generated.

[0023] Generate random sample values, and select the sample sub-interval whose sequence number matches the random sample value from the multiple sample sub-intervals as the target sample sub-interval;

[0024] The transmission link corresponding to the target sampling sub-interval is taken as the target transmission link.

[0025] In some embodiments, the generation process of the link scheduling model includes:

[0026] Obtain the link state feature parameters corresponding to the link state feature sequence, obtain the link selection parameters of multiple transmission links, obtain the initial policy network parameters of the policy network and the initial value function network parameters of the value function network, and generate the aggregate throughput ideal difference based on the link state feature parameters.

[0027] A state space is generated based on the link state characteristic parameters;

[0028] Based on the link selection, an action space is generated for transmission.

[0029] A reward space is generated based on the ideal difference in aggregated throughput;

[0030] An initial link scheduling model is generated based on the initial policy network parameters, the initial value function network parameters, the state space, the action space, and the reward space. The initial link scheduling model is then trained multiple times, and the link scheduling model is obtained based on the trained initial link scheduling model.

[0031] In some embodiments, the link-state characteristic parameters include the throughput parameters of each transmission link and the actual aggregate throughput of the heterogeneous transmission network. Generating the ideal difference in aggregate throughput based on the link-state characteristic parameters includes:

[0032] The ideal aggregate throughput is obtained by summing all the aforementioned throughput parameters;

[0033] A first throughput term is obtained based on the sum of the ideal aggregate throughput and the actual aggregate throughput; a second throughput term is obtained based on the difference between the ideal aggregate throughput and the actual aggregate throughput; and an aggregate throughput difference is obtained based on the ratio between the first throughput term and the second throughput term.

[0034] The aggregate throughput difference is logarithmically processed to obtain the ideal aggregate throughput difference.

[0035] In some embodiments, the training process of the initial link scheduling model includes:

[0036] Obtain the training state sequence, training action sequence, and training reward sequence;

[0037] Based on the training state sequence, the current policy state value, the subsequent policy state value, the current value function state value, and the subsequent value function state value are obtained.

[0038] The future policy reward is obtained based on the training reward sequence, the current policy state value, and the subsequent policy state value;

[0039] The initial policy network parameters are updated using gradient descent based on the future policy reward.

[0040] The updated state value of the value function is obtained based on the subsequent value function state value and the training reward sequence. The value function error is obtained based on the difference between the current value function state value and the updated state value of the value function. The initial value function network parameters are then updated using gradient descent based on the value function error.

[0041] To achieve the above objectives, a second aspect of this application provides a data packet transmission apparatus for a heterogeneous transmission network, wherein the heterogeneous transmission network includes multiple transmission links, and the apparatus includes:

[0042] The status acquisition module is used to acquire the link status feature sequence of each transmission link within a preset scheduling time period;

[0043] The data processing module is used to input all the link state feature sequences into the link scheduling model for data processing to obtain the link selection probability matrix of multiple transmission links;

[0044] The target transmission link determination module is used to determine at least one target transmission link in the next preset scheduling time period from multiple transmission links based on the link selection probability matrix, and to use the target transmission link for data packet transmission.

[0045] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the data packet transmission method for a heterogeneous transmission network as described in the first aspect.

[0046] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the data packet transmission method for heterogeneous transmission networks described in the first aspect.

[0047] The data packet transmission method and related equipment for heterogeneous transmission networks proposed in this application include multiple transmission links. The method includes: first, obtaining the link state feature sequence of each transmission link within a preset scheduling time period; then, inputting all link state feature sequences into a link scheduling model for data processing to obtain a link selection probability matrix of multiple transmission links; finally, determining at least one target transmission link from the multiple transmission links within the next preset scheduling time period based on the link selection probability matrix, and using the target transmission link for data packet transmission. This application embodiment utilizes the dynamic link state feature sequence of multiple transmission links changing within a preset scheduling time period in a heterogeneous transmission network to characterize the real-time transmission capability of the current multiple transmission links. Combined with a link scheduling model pre-constructed based on the data packet transmission process in the heterogeneous transmission network, a link selection probability matrix is ​​obtained, which determines the probability of selecting multiple transmission links that match the link state feature sequence for data packet transmission in the next preset scheduling time period. This improves the reliability of data packet transmission. Based on the link selection probability matrix, at least one target transmission link is selected from multiple transmission links for data packet transmission in the next preset scheduling time period, thereby avoiding getting trapped in local optima and further improving the success reliability of data packet transmission in the heterogeneous transmission network.

[0048] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of a heterogeneous transmission network provided in one embodiment of this application.

[0050] Figure 2 This is a flowchart of a data packet transmission method for a heterogeneous transmission network provided in another embodiment of this application.

[0051] Figure 3 yes Figure 2 The flowchart for step 201.

[0052] Figure 4 This is a schematic diagram of the attention feature extraction process provided in another embodiment of this application.

[0053] Figure 5 yes Figure 2 The flowchart for step 202.

[0054] Figure 6 This is a flowchart of a generation link scheduling model provided in another embodiment of this application.

[0055] Figure 7 This is a schematic diagram of the structure of a policy network provided in another embodiment of this application.

[0056] Figure 8 This is a schematic diagram of the structure of a value function network provided in another embodiment of this application.

[0057] Figure 9 yes Figure 6 The flowchart for step 601.

[0058] Figure 10 This is a flowchart of a training initial link scheduling model provided in another embodiment of this application.

[0059] Figure 11 This is a schematic diagram of a scheduling model based on a Markov decision process, provided in another embodiment of this application.

[0060] Figure 12 yes Figure 2 The flowchart for step 203.

[0061] Figure 13 yes Figure 12 The flowchart for step 1203.

[0062] Figure 14 This is a schematic diagram illustrating the overall concept of a data packet transmission method for a heterogeneous transmission network according to an embodiment of this application.

[0063] Figure 15 This is a schematic diagram of the structure of a data packet transmission device for a heterogeneous transmission network provided in an embodiment of this application.

[0064] Figure 16 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0066] It should be noted that although functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0068] First, let's analyze some of the terms used in this application:

[0069] Deep reinforcement learning, a branch of machine learning, combines the powerful representation learning capabilities of deep learning with the goal-oriented mechanisms of reinforcement learning, enabling agents to learn how to make optimal decisions in an uncertain environment. Specifically, deep reinforcement learning uses deep neural networks as function approximators to estimate the value of actions or directly output action policies, allowing the agent to learn the optimal path to achieve a specific goal through trial and error while interacting with the environment. This method significantly improves the model's learning and generalization abilities, especially when dealing with high-dimensional input data such as images and natural language.

[0070] With the rapid development of multi-interface technology, heterogeneous transmission networks with multi-link transmission are gradually becoming an important part of data communication. Simultaneous data transmission through multiple interfaces and paths can significantly improve network bandwidth utilization and transmission reliability. However, since the number of transmission links and data packets in heterogeneous transmission networks is usually large, selecting a suitable transmission link for each data packet is a complex problem.

[0071] In related technologies, transmission links are typically pre-assigned to each data packet with different transmission requirements based on the statistical transmission capacity of different transmission links, and then the data packets are transmitted. However, since the real-time transmission capacity of the actual transmission links changes dynamically, using the original transmission link allocation for data packet transmission can easily lead to situations where the actual transmission capacity of the transmission links cannot meet the transmission requirements of the data packets, resulting in insufficient reliability of data packet transmission.

[0072] To improve transmission reliability in heterogeneous transmission networks, this application utilizes a dynamic link state feature sequence of multiple transmission links changing within a preset scheduling time period to characterize the real-time transmission capability of these links. Combined with a link scheduling model pre-constructed based on the data packet transmission process in the heterogeneous transmission network, a link selection probability matrix is ​​obtained, indicating which transmission links, matching the link state feature sequence, will be selected for data packet transmission in the next preset scheduling time period. This improves data packet transmission reliability. Based on this link selection probability matrix, at least one target transmission link is selected from the multiple transmission links for data packet transmission in the next preset scheduling time period, thus avoiding local optima and further enhancing the success reliability of data packet transmission in heterogeneous transmission networks.

[0073] To better describe the data packet transmission method for heterogeneous transmission networks provided in this application, the heterogeneous transmission network in which the data packet transmission method is applied is first described below. (Refer to...) Figure 1 This is a schematic diagram of a heterogeneous transmission network provided in an embodiment of this application. Figure 1 As shown, the heterogeneous transmission network includes transmitting devices, receiving devices, mobile routers (SMRs), aggregation servers (SARs), and multiple transmission links. The transmitting devices send data packets to the receiving devices via the mobile routers and aggregation servers. During the process of the mobile routers sending data packets to the aggregation servers, a target transmission link is selected from the multiple transmission links for the transmission of the data packet.

[0074] Based on the above description of heterogeneous transmission networks, the data packet transmission method and related devices for heterogeneous transmission networks provided in this application embodiment will be further described below. The data packet transmission method for heterogeneous transmission networks provided in this application embodiment can be applied to transmitting devices, mobile routers, or processor devices, etc., in heterogeneous transmission networks.

[0075] The data packet transmission method for heterogeneous transmission networks in the embodiments of this application will be described in detail below. (Refer to...) Figure 2 This is an optional flowchart of a data packet transmission method for a heterogeneous transmission network provided in an embodiment of this application. Figure 2 The method may include, but is not limited to, steps 201 to 203. It is also understood that this embodiment... Figure 2 The order of steps 201 to 203 is not specifically limited. The order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0076] Step 201: Obtain the link status feature sequence of each transmission link within the preset scheduling time period.

[0077] Step 201 is described in detail below.

[0078] In some embodiments, in response to a data packet transmission request, in order to select a suitable target transmission link for the data packet in the heterogeneous transmission network so as to transmit data in the next preset scheduling time period and improve the reliability of data packet transmission, it is necessary to first obtain the link state feature sequence of all transmission links in the heterogeneous transmission network in the most recent preset scheduling time period, so as to use the link state feature sequence to characterize the real-time transmission capability of the transmission link, so as to use the link state feature sequence to select the target transmission link in the future.

[0079] The data information in the link-state feature sequence typically includes the number of data packets, the latency, throughput, and congestion status of each transmission link. The following section will further describe how to obtain the link-state feature sequence.

[0080] Reference Figure 3 The process involves obtaining the link status feature sequence of each transmission link within a preset scheduling time period, including the following steps 301 to 303.

[0081] Step 301: Obtain the link status information of each transmission link in each time slot.

[0082] Step 302: Concatenate multiple link status information from each transmission link to obtain a link status information sequence.

[0083] Step 303: Perform attention processing on the link state information sequence to obtain the link state feature sequence.

[0084] Steps 301 to 303 are described in detail below.

[0085] In some embodiments, a probe management module is provided in the heterogeneous transmission network. This probe management module is used to obtain the link status information of each transmission link in real time within each time slot of a preset scheduling time period (which typically includes information such as the latency, throughput, and congestion status of each transmission link).

[0086] Then, the link status information of each transmission link in all time slots is spliced ​​together to obtain the link status information sequence of all link status information of each transmission link within the preset scheduling time period.

[0087] After obtaining the link-state information sequence, a Transformer encoder framework based on a multi-head self-attention mechanism is used to extract feature representations of the state sequence, thus obtaining the link-state feature sequence corresponding to the link-state information sequence. This encoding framework computes an abstract representation of the current network state through positional encoding and multi-layer self-attention encoding, capturing element features at various positions within the input sequence, enabling the subsequent policy network to obtain more information about changes in the network environment. (Refer to...) Figure 4 This is a schematic diagram of an attention feature extraction process provided in an embodiment of this application. Figure 4 As shown, the network first transforms the input sequence into an embedding vector through an embedding layer, adjusting its magnitude to control the model's gradient. Next, the embedding vector is encoded sequentially through a positional encoding layer, an encoding layer, and a mask encoding layer. Multi-layer encoding can extract richer features, generating abstract representations of the data from multiple perspectives, suitable for network state spaces with multi-dimensional attributes. Finally, a linear layer performs a linear transformation on the encoded embedding vector, adjusting it back to the original state sequence dimension for subsequent action inference by the policy network.

[0088] Meanwhile, the detection management module will also perform real-time management and data backup of the link status information and link status information sequence obtained from the detection, and deliver the relevant information of the link status information to the online scheduling module on a regular basis so that the online scheduling module can perform scheduling inference work in the future, while the backup data file is used for training the policy network.

[0089] Through steps 301 to 303 above, all link status information of each time slot in the heterogeneous transmission network within the most recent preset scheduling time period is spliced ​​together to obtain a link status information sequence that represents the dynamic changes of each transmission link. Then, a multi-attention mechanism is used to extract features from the link status information sequence to obtain a link status feature sequence that is more suitable for subsequent data processing, thereby improving the accuracy of selecting target transmission links in the heterogeneous transmission network.

[0090] Step 202: Input all link state feature sequences into the link scheduling model for data processing to obtain the link selection probability matrix of multiple transmission links.

[0091] Step 202 will be described in detail below.

[0092] In some embodiments, after obtaining the link state feature sequence that characterizes the real-time transmission capability of each transmission link, all the link state feature sequences are input into a link scheduling model pre-constructed based on the data packet transmission process in the heterogeneous transmission network for data processing, so as to obtain a link selection probability matrix that the multiple transmission links corresponding to the link state feature sequence are selected for data packet transmission in the next preset scheduling time period, so as to select the target transmission link from the multiple transmission links according to the link selection probability matrix.

[0093] The following describes how to obtain the link selection probability matrix of the corresponding multiple transmission links based on the sequence of all link state characteristics.

[0094] Reference Figure 5 All link state feature sequences are input into the link scheduling model for data processing to obtain the link selection probability matrix of multiple transmission links, including the following steps 501 to 503.

[0095] Step 501: Input all link state feature sequences into the link scheduling model for data processing to obtain the link selection probability of each transmission link in each time slot.

[0096] Step 501 will be described in detail below.

[0097] In some embodiments, after obtaining the link state feature sequence characterizing the real-time transmission capability of each transmission link, all link state feature sequences are used together as input parameters and input into the link scheduling model for data processing to obtain the link selection probability corresponding to the selection of each transmission link in each time slot within a preset scheduling time period. This facilitates the subsequent selection of the target transmission link corresponding to each time slot from multiple transmission links based on the link selection probability.

[0098] The following section will first describe how to generate the link scheduling model.

[0099] Reference Figure 6 The process of generating the link scheduling model includes the following steps 601 to 605.

[0100] Step 601: Obtain the link state feature parameters corresponding to the link state feature sequence, obtain the link selection parameters of multiple transmission links, obtain the initial policy network parameters of the policy network and the initial value function network parameters of the value function network, and generate the aggregate throughput ideal difference based on the link state feature parameters.

[0101] Step 601 will be described in detail below.

[0102] In some embodiments, in order to generate a suitable link scheduling model, it is necessary to first obtain the link state feature parameters corresponding to the link state feature sequence. These parameters typically include the data volume of the data packets in each time slot, the latency of each transmission link, the packet loss rate, the number of unreply data packets, the number of unsent data packets, etc. This information can fully characterize the transmission quality of each transmission link in the current heterogeneous transmission network, enabling the agent used to select the target transmission link to adjust the data packet allocation behavior more precisely.

[0103] Furthermore, it is also necessary to obtain the link selection parameters of multiple transmission links in the heterogeneous transmission network, that is, the set of transmission links that can be selected in each time slot.

[0104] Additionally, it obtains the initial policy network parameters θ and the initial value function network parameters ω of the policy network pre-generated by the A2C algorithm in the deep reinforcement learning module.

[0105] Understandably, the A2C network comprises two key models: a policy network and a value function network. The policy network learns new scheduling policies based on environmental states and feedback rewards, thus enabling the online scheduling algorithm to be adaptive. The value function network, on the other hand, evaluates the advantages of each action through the temporal difference error between state values, assisting in updating the policy network.

[0106] The A2C algorithm reduces the variance of the policy gradient method by guiding the algorithm through the value function, thereby improving its stability and making it easier to converge to a better policy. In actual packet scheduling, flexible transmission mechanisms need to make frequent decisions for a large number of packets, requiring high immediacy; a fast policy learning rate has a greater advantage. Furthermore, the aggregation performance of multipath transmission depends not only on the immediate reward of the current decision but also on the long-term benefits. The idea of ​​co-optimization between the policy network and the value function network helps to balance the impact of immediate performance rewards and long-term performance benefits on the scheduling algorithm. The policy network and the value function network will be further described below.

[0107] Reference Figure 7 This is a schematic diagram of the structure of a policy network provided in an embodiment of this application. Figure 7As shown, the policy network sets the state feature extraction network as a submodule of the policy network and includes two fully connected layers. The ReLU activation function is used between these layers, and the action probabilities are output through the Softmax function. This network works in parallel with the online scheduling algorithm, responsible for training network parameters, optimizing the scheduling policy, and synchronizing parameters with the online scheduling module in real time. After the agent executes an action according to the online scheduling algorithm, the detected environmental state and corresponding feedback reward are input into the policy training module. This module uses the policy gradient method to maximize the expected reward to guide the network optimization direction, increasing the probability of high-reward actions and decreasing the probability of low-reward actions. After training, the policy training model copies the updated parameters to the policy network of the online scheduling algorithm, enabling the algorithm to adjust its policy as the network changes.

[0108] The value function estimation model is responsible for predicting the expected reward an agent might receive from its current state, denoted as the state value V(s). This model effectively evaluates the environmental quality of multiple transmission links; a better transmission environment indicates greater potential for reward and a higher state value. When reasoning about agent actions, it's necessary to consider not only immediate feedback rewards but also future rewards. The value function estimation model quantifies the impact of actions on long-term gains by evaluating the transmission environment quality of each time slot, thus guiding the optimization direction of the policy network. (See reference...) Figure 8 This is a schematic diagram of the structure of a value function network provided in an embodiment of this application. Figure 8 As shown, the value function network is responsible for calculating the state based on the network state. It consists of two fully connected layers, with nonlinear representations introduced between layers via the ReLU activation function. To learn the state features within each time slot, the input to the value function network is actually a state feature vector obtained by deforming and reshaping the state sequence, with a dimension of [missing information]. Its output is the corresponding state value.

[0109] In addition, in order to generate a suitable reward model, the ideal difference of aggregate throughput will also be generated based on the link state characteristic parameters, as described below.

[0110] Reference Figure 9 The ideal difference in aggregate throughput is generated based on the link state characteristic parameters, including the following steps 901 to 903.

[0111] Step 901: Sum all throughput parameters to obtain the ideal aggregate throughput.

[0112] Step 902: Obtain the first throughput term based on the sum of the ideal aggregate throughput and the actual aggregate throughput, obtain the second throughput term based on the difference between the ideal aggregate throughput and the actual aggregate throughput, and obtain the aggregate throughput difference based on the ratio of the first throughput term and the second throughput term.

[0113] Step 903: Perform logarithmic processing on the aggregate throughput difference to obtain the ideal aggregate throughput difference.

[0114] Steps 901 to 903 are described in detail below.

[0115] In some embodiments, the throughput parameter of each transmission link is extracted from the link state characteristic parameters. It is expressed as the throughput of the s-th transmission link in the t-th time slot, and then the throughput parameters of all transmission links in the t-th time slot are accumulated. To obtain the ideal aggregate throughput

[0116] It is also necessary to extract the common throughput of all transmission links within each time slot (e.g., the t-th time slot) from the link state characteristic parameters, i.e., the actual aggregate throughput.

[0117] Then, the first throughput term is obtained based on the sum of the ideal aggregate throughput and the actual aggregate throughput. The second throughput term is obtained based on the difference between the ideal aggregate throughput and the actual aggregate throughput. The aggregate throughput difference is obtained based on the ratio of the first throughput term and the second throughput term, as shown in the following formula (1).

[0118]

[0119] It can be seen that the actual aggregated throughput is close to the ideal aggregated throughput, indicating a good aggregation effect; conversely, it indicates that the actual aggregated bandwidth is small and the aggregation performance is poor.

[0120] Through steps 901 to 903 above, the sum of the throughput parameters of all transmission links within a time slot is used as a representation of the theoretically maximum aggregate throughput that multi-path transmission can achieve, and the actual aggregate throughput of all transmission links within a time slot is used to obtain the aggregate throughput difference, which represents the difference between the actual and theoretical throughput. This aggregate throughput difference can be used to describe the aggregation performance in heterogeneous transmission networks, so that the reward model of the initial link scheduling model can be constructed using this aggregate throughput difference. This allows the training process of the initial link scheduling model to aim at improving aggregation performance, thereby improving the reliability of data packet transmission.

[0121] Step 602: Generate state space based on link state characteristic parameters.

[0122] Step 603: Generate action space based on link selection transmission.

[0123] Step 604: Generate a reward space based on the ideal difference in aggregated throughput.

[0124] Step 605: Generate an initial link scheduling model based on the initial policy network parameters, initial value function network parameters, state space, action space, and reward space, and train the initial link scheduling model multiple times. Based on the trained initial link scheduling model, obtain the link scheduling model.

[0125] Steps 602 to 605 are described in detail below.

[0126] To select a suitable target transmission link from multiple transmission links for data packet transmission, this embodiment transforms the data packet scheduling decision problem into a Markov decision process based on the state of heterogeneous transmission networks. The Markov decision process includes six key parameters: agent, state space, agent's action space, environmental feedback (i.e., reward space) obtained after the agent makes a decision, and decision policy.

[0127] In the scheduling decision-making process, the intelligent agent refers to the scheduling mechanism used in the heterogeneous transmission network (or the executor itself that executes the data packet transmission method of the heterogeneous transmission network in the embodiments of this application), which is responsible for interacting with the heterogeneous transmission network and executing corresponding actions according to the strategy.

[0128] Then, after obtaining the link state feature parameters corresponding to the link state feature sequence, these parameters are used as the state space. This includes information such as the data volume of each time slot, the latency of each transmission link, packet loss rate, number of unreply packets, and number of unsent packets. This fully characterizes the transmission quality of the current heterogeneous transmission network, allowing the agent to adjust packet allocation behavior more precisely. After the detection and management module completes the detection, the above n link information and global information are sequentially concatenated to obtain a state space of S, with a sequence dimension of [missing information].

[0129] For action decisions, the agent takes corresponding actions according to the current state and strategy. During transmission across multiple links, the scheduling algorithm selects links for data packets based on the current environment state. Therefore, after obtaining the link selection parameters for multiple transmission links, the agent's actual behavior is the link selection process, and its set is the set of available links. Assuming there are n transmission links, the agent's action space is A = (a1, a2, ..., a...). n ), sequence dimension is

[0130] Regarding the reward space, after the agent performs an action, new data packets are allocated to various links, and the current heterogeneous transmission network environment will inevitably change, affecting the overall transmission performance. The agent's behavior directly impacts the aggregated transmission performance during this process. If a better link is selected for the data packets, the transmission performance will improve. Conversely, if the agent selects a poor link, the transmission performance may gradually decline, or even cause severe performance degradation. To evaluate the quality of the agent's current behavior, the agent uses the relevant algebraic formula of the multi-path transmission aggregation rate as the evaluation value of the immediate reward to guide the optimization direction of the decision-making strategy. Therefore, the reward space is determined based on the above-mentioned ideal difference in aggregated throughput, i.e., the above formula (1). The larger the reward value, the closer the actual aggregated throughput is to the ideal aggregated throughput, and the better the aggregation effect; conversely, the smaller the actual aggregated bandwidth, the worse the aggregation performance. When the actual aggregation effect is much smaller than the ideal aggregated throughput, the reward value is approximately 0.

[0131] However, in practical applications, more attention needs to be paid to the performance during long-term transmission. Therefore, evaluating the transmission quality of multiple links does not depend solely on the aggregation effect of a single time slot, but also on the average aggregation transmission performance over the long term. Under this consideration, the concept of future reward is introduced, which is the expected value of the weighted sum of the immediate future rewards after the agent performs the current action, as shown in the following formula (2).

[0132]

[0133] Among them, G t R represents the accumulated reward after time t. t+k+1 Represents the immediate reward for subsequent time slots. γ k The discount factor, representing the discount at subsequent time k, is responsible for balancing the importance of immediate rewards and future accumulated rewards, enabling the agent to more comprehensively evaluate the effects of its actions. A larger discount factor makes the agent focus more on future accumulated rewards, while a smaller discount factor emphasizes the impact of immediate rewards.

[0134] Regarding the decision policy, in the Markov decision process, the agent calculates the conditional probability distribution of actions according to the policy based on the current state. That is, in the current state s, the probability of the agent performing action a is shown in the following formula (3).

[0135] π(a|s)=p(A=a|S=s) (3)

[0136] In the transmission process across multiple transmission links, the scheduling algorithm calculates the probability distribution of data packets being allocated to n transmission links based on the current environmental state of the heterogeneous transmission network. Assume that in time slot t, the agent's probability of choosing transmission link i is p.i Then the probability distribution of data packets being allocated to each link is as follows:

[0137] In actual multi-link transmission, if the detection interval is too large, a lot of detailed network information will be lost, making it difficult for the scheduling algorithm to perceive the current network changes. However, if the online scheduling algorithm updates the policy according to the feedback reward after each detection, the detection granularity is too small, making it difficult to characterize the impact of scheduling actions on the link state from a global perspective. At the same time, the deep learning model used by the online scheduling algorithm requires a certain amount of computation time when inferring the probability distribution of actions, which is generally greater than the detection interval. Therefore, in order to balance the frequency of network information detection and the frequency of policy update, the actual input of the online scheduling algorithm is not the state vector of a single time slot defined in the state space, but a state sequence composed of state vectors detected within a preset scheduling time period (i.e., multiple time slots). Correspondingly, the actual output of the algorithm is a sequence of action probability distributions of the same length as the sequence, and the feedback reward is the aggregated performance gain after the agent has performed a series of actions. This can preserve the details of network changes and match the detection frequency with the policy update frequency. The specific sequence expression is shown in the following formula (4).

[0138]

[0139] After constructing the state space, action space, reward space, initial decision network parameters, and initial value function parameters corresponding to the Markov decision, an initial link scheduling model will be constructed based on the idea of ​​the A2C algorithm. Then, the initial link scheduling model will be trained multiple times using training samples to obtain a practically usable link scheduling model.

[0140] This application utilizes the Advantage Action-Evaluation (A2C) reinforcement learning framework to train the policy network used in the initial link scheduling model. The A2C network combines the advantages of policy gradient and value function estimation, aiming to learn policies that maximize expected rewards, consistent with the purpose of this application to improve transmission aggregation performance in multi-transmission links. Whenever the probe management module completes the phased collection of state-action-reward tuples, the offline training module sequentially stores the tuple data into the experience replay cache and performs preprocessing, which is then used to train the policy network and value function network, respectively. The training of the initial link scheduling model will be detailed below.

[0141] Reference Figure 10 The initial link scheduling model training process includes the following steps 1001 to 1005.

[0142] Step 1001: Obtain the training state sequence, training action sequence, and training reward sequence.

[0143] Step 1002: Obtain the current policy state value, subsequent policy state value, current value function state value, and subsequent value function state value based on the training state sequence.

[0144] Step 1003: Obtain the future policy reward based on the training reward sequence, the current policy state value, and the subsequent policy state value.

[0145] Step 1004: Update the initial policy network parameters using gradient descent based on future policy rewards.

[0146] Step 1005: Obtain the updated state value of the value function based on the subsequent value function state value and the training reward sequence. Obtain the value function error based on the difference between the current value function state value and the updated state value of the value function. Then, update the initial value function network parameters using gradient descent based on the value function error.

[0147] Steps 1001 to 1005 are described in detail below.

[0148] In some embodiments, the policy network is trained to maximize accumulated rewards in the hope of generating a better policy π. θ (a t |s t This allows the agent to select appropriate actions and reap greater rewards. The training process employs a policy gradient approach, calculating the action advantage to determine the gradient direction of the expected reward, thus updating the parameters in the direction that maximizes reward growth.

[0149] Therefore, firstly, training samples for M preset scheduling time periods used to train the initial link scheduling model are obtained, which correspond to training state sequences, training action sequences, and training reward sequences, i.e., τ′={(S1,A1,r1), (S2,A2,r2), ..., (S M A M r M Then, based on the training state sequence, the current policy state value V(S) is obtained. i ), and subsequent policy state value V(S) i+1 ).

[0150] Since the sampling of actions has a certain degree of randomness, the feedback rewards generated by different action sequences may vary greatly, resulting in a high variance in the gradient estimation of the accumulated reward expectation, making it difficult to converge to a stable state quickly. Therefore, the policy training module uses the state value predicted by the value function estimation module as the baseline for obtaining future rewards after executing actions, and calculates the future policy reward for each action that exceeds the baseline, as shown in the following formula (5).

[0151] ADV i =ri +γV(S i+1 )-V(S i (5)

[0152] Among them, S i and r i S is the state sequence and feedback reward at the i-th time step. i+1 S is the state sequence for the next time step, γ is the discount factor responsible for balancing the importance of immediate rewards and accumulated rewards, and V(·) is the state value calculated by the value function estimation module based on the state sequence, representing the average level of future accumulated rewards that the agent will reap under this state sequence. Therefore, when the agent is in state sequence S i Execute action sequence A i And receive an instant reward r i At that time, the reward that can be obtained from time step i is r. i +γV(S i+1 This reward value is related to the average future reward V(S) of the current state sequence. i The difference between the two values ​​is the relative reward (ADV) for the agent's action. i This reflects the potential of each action. A larger relative reward indicates that the action sequence is more advantageous and can generate more rewards. Furthermore, relative rewards compare the advantages of different actions, helping to eliminate the influence of random noise, reduce variance, and make training more stable.

[0153] Based on the above analysis, the training objective function expression of the policy network is shown in the following formula (6).

[0154] J(θ) = log(π) θ (A i |S i ADV i +β·H(π θ (Ai|S i (6)

[0155] The function consists of two parts: a policy reward and an entropy regularization term. The former determines the direction of parameter updates based on the advantage of the action, maximizing the accumulated reward through training. The latter adds a degree of exploratory nature to the training process, because when the entropy value is large, the probability distribution of actions is more even, and the agent's action selection is more random.

[0156] Where θ is the policy network parameter, logπ θ (A i |S i ) represents the log-likelihood probability of the action, ADV i For relative future rewards, β is the entropy coefficient, and H(π) θ (A i |Si ADV is the entropy of the policy network. i The sign of β determines the direction of parameter updates in the policy network. A negative value indicates that the action's reward is below average, and its probability should be reduced; parameters are updated along the negative gradient. Conversely, when an action is advantageous, parameters are updated along the positive gradient to increase the probability of that action. Policy entropy represents the degree of randomness in the policy network; a large β indicates that parameter updates tend to maintain the randomness of the policy.

[0157] The above process involves calculating action probabilities and relative future rewards. However, after the policy network infers the action probability distribution, Bernoulli sampling and Monte Carlo sampling are still needed to determine the actual actions to be executed, and there is a certain probability that multiple actions will be selected simultaneously. Therefore, the log-likelihood probability shown in the following formula (7) is actually the sum of the log probabilities of the selected actions. Assuming that k actions are selected, the log-likelihood probability of the actions should be:

[0158]

[0159] In summary, the policy training model updates the parameters in the direction of increasing reward by using the policy gradient method based on the relative future reward of the action. Therefore, the initial decision parameters are updated by gradient descent as shown in the following formula (8), where α is the learning rate and determines the update step size.

[0160]

[0161] The training process is shown in Algorithm 1 below. In practical network training, to simplify mathematical calculations, the reinforcement learning training process is often expressed as a minimization optimization problem. Therefore, Algorithm 1 introduces a negative sign into the objective function, transforming the problem of maximizing accumulated reward into minimizing policy loss.

[0162]

[0163]

[0164] For the value function network V(S; ω), which is responsible for predicting the average reward the agent will receive in the current state, the training objective is to minimize the error between the network's predicted reward and the actual reward. Therefore, after obtaining the training state sequence, the value of the current policy state is obtained based on the training state sequence. Subsequent strategy state value

[0165] The value function network uses the mean square error method to train the network parameters, and its objective function is shown in the following formula (9).

[0166]

[0167] During training, the initial value function network parameters are updated in the direction of the negative gradient of the loss function in order to reduce the error value. The expression for updating the initial value function network parameters is shown in the following formula (10).

[0168]

[0169] In summary, the training process of the value function network is shown in Algorithm 2 below.

[0170]

[0171]

[0172] Reference Figure 11 This is a schematic diagram of a scheduling model based on a Markov decision process provided in an embodiment of this application. Figure 11 As shown, the current environmental state of the heterogeneous transmission network is composed of network metrics such as latency, packet loss rate, and congestion status of each transmission link. The environmental states of different time slots constitute the state space, which characterizes the changing process of the heterogeneous transmission network environment. The scheduler proposed in this section is regarded as an intelligent agent that continuously interacts with the environment. Its main behavior is to allocate data packets on various links. Therefore, the set of actions of the intelligent agent is the set of available transmission links. The scheduler's selection of the best transmission link for data packets from different sub-streams can be regarded as a decision-making process in which the intelligent agent selects different behaviors based on different system states, and the aggregated transmission performance of the transmission system is the feedback of the decision. In this framework, each action of the intelligent agent will have a corresponding environmental feedback reward, namely, a change in aggregated transmission performance. The intelligent agent adopts an action-feedback-learning strategy to learn the effects of different interactive behaviors with the network environment in different states, thereby adaptively adjusting its subsequent behaviors to obtain more rewards.

[0173] Through steps 601 to 605 and steps 1001 to 1005 above, an initial link scheduling model corresponding to the actual data packets during transmission in a heterogeneous transmission network is constructed using reinforcement learning. The difference in aggregated throughput, which characterizes the aggregation performance of the heterogeneous transmission network, is used as the training reward. Multiple training samples are used to train the decision network and value function network in the initial link scheduling model to improve the aggregation performance of the decisions obtained by the decision network. Therefore, when the decision network in the trained initial link scheduling model is used as the actual link scheduling model for target transmission link selection, the aggregation performance corresponding to the decision can be improved, thereby improving the reliability of data packet transmission.

[0174] Step 502: Concatenate the multiple link selection probabilities of each transmission link to obtain a link selection probability sequence.

[0175] Step 503: Concatenate multiple link selection probability sequences to obtain a link selection probability matrix.

[0176] Steps 502 to 503 are described in detail below.

[0177] In some embodiments, all link state feature sequences are input together as input parameters into the link scheduling model for data processing to obtain the link selection probability corresponding to the selection of each transmission link in each time slot within a preset scheduling time period. Then, the link selection probabilities of each transmission link within multiple time slots of a preset scheduling time period are concatenated to obtain a link selection probability sequence. Then, the link selection probability sequence p of multiple transmission links is further processed. n The data is spliced ​​together to obtain a link selection probability matrix for multiple transmission links in multiple time slots within a preset scheduling time period.

[0178] Step 203: Based on the link selection probability matrix, determine at least one target transmission link from multiple transmission links within the next preset scheduling time period, and use the target transmission link for data packet transmission.

[0179] Step 203 will be described in detail below.

[0180] In some embodiments, after obtaining the link selection probability matrix, the target transmission link corresponding to each time slot in the next preset scheduling time period is determined from multiple transmission links based on the link selection probability matrix, so as to use these target transmission links for data packet transmission in the next preset scheduling time period. The following will further describe how to determine the target transmission link based on the link selection probability matrix.

[0181] Reference Figure 12 The method involves determining at least one target transmission link from multiple transmission links within the next preset scheduling time period based on the link selection probability matrix, including the following steps 1201 to 1203.

[0182] Step 1201: Generate sampling method selection probability based on the link selection probability corresponding to multiple transmission links in each time slot.

[0183] Step 1202: Select the target sampling method from binary Bernoulli sampling and Monte Carlo sampling based on the sampling method selection probability.

[0184] Step 1203: Select the target transmission link corresponding to each time slot from multiple transmission links one by one based on the target sampling method and link selection probability.

[0185] Steps 1201 to 1203 are described in detail below.

[0186] In some embodiments, after obtaining the link selection probability matrix of multiple transmission links in multiple time slots within a preset scheduling time period, the link selection probability p corresponding to multiple transmission links in each time slot in the link selection probability matrix is ​​used. n The probability of selecting the sampling method is (1-p1)(1-p2)...(1-p n The sampling method selects probabilities (1-p1)(1-p2)...(1-p). n Let be the probability of selecting the target transmission link using Monte Carlo sampling. Then, the probability of selecting the target transmission link using binary Bernoulli sampling is 1 - (1 - p1)(1 - p2)...(1 - p...). n ).

[0187] In practical applications, a random value for method selection can be generated between zero and one. When this random value is less than the sampling method selection probability (1-p1)(1-p2)...(1-p...), the method selection probability is determined by the probability of the random value. n When the probability of selecting a sampling method is greater than (1-p1)(1-p2)...(1-p...), Monte Carlo sampling is used as the target sampling method; conversely, when the probability of selecting a sampling method is greater than ... (1-p1)(1-p2)...(1-p...), Monte Carlo sampling is used as the target sampling method. n When ), binary Bernoulli sampling is used as the target sampling method.

[0188] Then, based on the target sampling method and link selection probability, the target transmission link corresponding to each time slot is selected one by one from multiple transmission links.

[0189] When the target sampling method is binary Bernoulli sampling, the online scheduling algorithm first reads the current action probability distribution vector to obtain the execution probability of each action, and then performs Bernoulli sampling on each action based on these probability values. Its expression is shown in the following formula (11).

[0190] x i =p i k ×(1-p i ) 1-k (11)

[0191] Where, p i Let x be the probability of executing this action. iLet represent the sampled Boolean value of the i-th action in the action set. If the sampled value is 1, k represents the number of times the action has been selected. Then the agent executes the action. Assuming the action set contains n actions, after Bernoulli sampling, the action execution result of the agent should be X = (x1, x2, ..., x...). n Bernoulli sampling causes the agent to sample with probability p. i Perform action a i With probability 1-p i This action is not executed. Since each action is sampled individually, and actions with higher probabilities are more likely to be executed, this sampling method retains the principle of selecting actions with the highest expected returns, while also allowing the agent to execute multiple actions in parallel, i.e., distributing data packets to multiple links.

[0192] Since Bernoulli sampling has a certain probability of yielding all samples as zero, meaning the agent performs no action, this is clearly inconsistent with real-world logic. When this low-probability event occurs, the online scheduling algorithm switches to Monte Carlo sampling, randomly generating sequence numbers based on the current action probability distribution and executing the corresponding actions from the action set. The selection steps when the target sampling method is Monte Carlo will be further described below.

[0193] Reference Figure 13 The target transmission link corresponding to each time slot is selected one by one from multiple transmission links based on the target sampling method and the link selection probability, including the following steps 1301 to 1303.

[0194] Step 1301: Based on the multiple link selection probabilities within each time slot, generate multiple sampling sub-intervals corresponding to the transmission links and a sequence number identifier corresponding to each sampling sub-interval.

[0195] Step 1302: Generate random sampled values, and select the sampled sub-interval whose serial number matches the random sampled value from multiple sampled sub-intervals as the target sampled sub-interval.

[0196] Step 1303: Use the transmission link corresponding to the target sampling sub-interval as the target transmission link.

[0197] Steps 1301 to 1303 are described in detail below.

[0198] In some embodiments, after determining the use of the Monte Carlo method and obtaining the link selection probabilities of multiple transmission links, multiple sampling sub-intervals are generated based on these probabilities. Each sub-interval corresponds to a transmission link, and the index of each sampling sub-interval is generated by the link selection probability of that transmission link. Next, random sample values ​​are generated between zero and one, and then, based on these random sample values, the sampling sub-interval whose index matches the random sample value is selected from the multiple sampling sub-intervals as the target sampling sub-interval.

[0199] Taking three transmission links {A1, A2, A3} as an example, where the link selection probability of A1 is 0.5, the link selection probability of A2 is 0.3, and the link selection probability of A3 is 0.2, then the sequence number identifier corresponding to A1 is 0.5, the sequence number identifier corresponding to A2 is 0.5 + 0.3 = 0.8, and the sequence number identifier corresponding to A3 is 1. If the random sample value is 0.48, then A1 will be selected as the target transmission link for this time slot.

[0200] Through steps 1201 to 1203 and 1301 to 1303 above, a suitable target sampling method is selected by utilizing the link selection probability generated from the link state feature sequence in the real-time changing heterogeneous transmission network. This target sampling method, based on the link selection probability, selects a suitable target transmission link for each time slot within the next preset scheduling time period from multiple transmission links, thereby improving the reliability of data packet transmission. Unlike traditional reinforcement learning models, this application outputs a selection probability, which is then used for link selection. This approach has a certain degree of exploratory nature to avoid getting trapped in local optima, and can better refine the link scheduling model in practical applications to improve the reliability of data packet transmission.

[0201] Reference Figure 14 This is a schematic diagram illustrating the overall concept of a data packet transmission method for a heterogeneous transmission network provided in an embodiment of this application. Figure 14 As shown, a reinforcement learning architecture is used in advance to collect link state feature sequences of heterogeneous transmission networks in practical applications through random sampling scheduling as training samples input into an empirical sequence collector. This is used to train a link scheduling model (which includes updating the action probability distribution corresponding to different link state feature sequences). When responding to the communication sender's request to send data packets in the next scheduling time period, the link state feature sequence of the previous scheduling time period is used as input into the link scheduling model to obtain a suitable action probability distribution decision, thereby selecting a suitable target transmission link for data packet transmission and improving the reliability of data packet transmission.

[0202] The data packet transmission method and related equipment for heterogeneous transmission networks proposed in this application embodiment include multiple transmission links. The method includes: pre-acquiring link state feature parameters corresponding to link state feature sequences, acquiring link selection parameters for multiple transmission links, acquiring initial policy network parameters of the policy network and initial value function network parameters of the value function network, and accumulating all throughput parameters to obtain ideal aggregate throughput, obtaining a first throughput term based on the sum of the ideal aggregate throughput and the actual aggregate throughput, obtaining a second throughput term based on the difference between the ideal aggregate throughput and the actual aggregate throughput, obtaining an aggregate throughput difference based on the ratio of the first throughput term and the second throughput term, performing logarithmic processing on the aggregate throughput difference to obtain an ideal aggregate throughput difference, generating a state space based on the link state feature parameters, generating an action space based on link selection transmission, generating a reward space based on the ideal aggregate throughput difference, and generating an initial link schedule based on the initial policy network parameters, initial value function network parameters, state space, action space, and reward space. The model is developed, and the initial link scheduling model is trained multiple times to obtain a link scheduling model. Then, the link state information of each transmission link in each time slot is obtained, and the multiple link state information of each transmission link is concatenated to obtain a link state information sequence. The link state information sequence is then subjected to attention processing to obtain a link state feature sequence. Next, all link state feature sequences are input into the link scheduling model for data processing to obtain the link selection probability of each transmission link in each time slot. The multiple link selection probabilities of each transmission link are concatenated to obtain a link selection probability sequence. The multiple link selection probability sequences are then concatenated to obtain a link selection probability matrix. Finally, based on the link selection probabilities corresponding to multiple transmission links in each time slot, a sampling method selection probability is generated. Based on the sampling method selection probability, a target sampling method is selected from binary Bernoulli sampling and Monte Carlo sampling. Based on the target sampling method and the link selection probability, the target transmission link corresponding to each time slot is selected one by one from multiple transmission links.

[0203] This application embodiment concatenates all link state information of each time slot in the heterogeneous transmission network within the most recent preset scheduling time period to obtain a link state information sequence characterizing the dynamic changes of each transmission link. Then, a multi-attention mechanism is used to extract features from this link state information sequence to obtain a link state feature sequence more suitable for subsequent data processing, thereby improving the accuracy of target transmission link selection in the heterogeneous transmission network. Furthermore, the sum of throughput parameters of all transmission links within a time slot is used as a representation of the theoretically maximum aggregate throughput of multi-path transmission, and the actual aggregate throughput of all transmission links within the time slot is used to obtain an aggregate throughput difference value characterizing the difference between the actual and theoretical values. This aggregate throughput difference value can be used to describe the aggregation performance in the heterogeneous transmission network, facilitating the subsequent construction of a reward model for the initial link scheduling model. This ensures that the training process of the initial link scheduling model aims to improve aggregation performance, thereby enhancing... The reliability of data packet transmission is improved. Furthermore, by employing reinforcement learning, an initial link scheduling model corresponding to the actual data packet transmission process in a heterogeneous transmission network is constructed. The difference in aggregated throughput, representing the aggregation performance of the heterogeneous transmission network, is used as a training reward. Multiple training samples are used to train the decision network and value function network in the initial link scheduling model, improving the aggregation performance of the decisions obtained by the decision network. Therefore, when the decision network in the trained initial link scheduling model is used as the actual link scheduling model for target transmission link selection, the aggregation performance corresponding to the decisions can be improved, thereby enhancing the reliability of data packet transmission. In addition, by utilizing the link selection probability generated from the link state feature sequence in the real-time changing heterogeneous transmission network, a suitable target sampling method is selected. This target sampling method, based on the link selection probability, selects the appropriate target transmission link for each time slot within the next preset scheduling time period from multiple transmission links, thereby improving the reliability of data packet transmission.Unlike traditional reinforcement learning models, this application outputs selection probabilities, which are then used to select links. This approach allows for exploration and avoids getting trapped in local optima, enabling further refinement of the link scheduling model in practical applications to improve the reliability of data packet transmission. In summary, this method utilizes the dynamic link state feature sequence of multiple transmission links in a heterogeneous transmission network within a preset scheduling time period to characterize the real-time transmission capabilities of these links. Combined with a link scheduling model pre-constructed based on the data packet transmission process in the heterogeneous transmission network, a link selection probability matrix is ​​obtained, indicating which transmission links, matching the link state feature sequence, will be selected for data packet transmission in the next preset scheduling time period. This improves the reliability of data packet transmission. Based on this link selection probability matrix, at least one target transmission link is selected from multiple transmission links for data packet transmission in the next preset scheduling time period, thus avoiding local optima and further enhancing the success and reliability of data packet transmission in heterogeneous transmission networks.

[0204] This application also provides a data packet transmission apparatus for a heterogeneous transmission network, which can implement the above-mentioned data packet transmission method for a heterogeneous transmission network, as described above. Figure 15 The device 1500 includes:

[0205] The status acquisition module 1510 is used to acquire the link status feature sequence of each transmission link within a preset scheduling time period;

[0206] Data processing module 1520 is used to input all link status feature sequences into the link scheduling model for data processing to obtain the link selection probability matrix of multiple transmission links;

[0207] The target transmission link determination module 1530 is used to determine at least one target transmission link within the next preset scheduling time period from multiple transmission links based on the link selection probability matrix, and to use the target transmission link for data packet transmission.

[0208] In some embodiments, the status acquisition module 1510 is further configured to:

[0209] Obtain the link status information of each transmission link in each time slot;

[0210] Multiple link status information from each transmission link are concatenated to obtain a link status information sequence;

[0211] Attention processing is applied to the link state information sequence to obtain the link state feature sequence.

[0212] In some embodiments, the data processing module 1520 is further configured to:

[0213] All link state feature sequences are input into the link scheduling model for data processing to obtain the link selection probability of each transmission link in each time slot;

[0214] The link selection probabilities of each transmission link are concatenated to obtain a link selection probability sequence;

[0215] Multiple link selection probability sequences are concatenated to obtain a link selection probability matrix.

[0216] In some embodiments, the target transmission link determination module 1530 is further configured to:

[0217] Based on the link selection probabilities corresponding to multiple transmission links in each time slot, the sampling method selection probability is generated.

[0218] The target sampling method is selected from binary Bernoulli sampling and Monte Carlo sampling based on the sampling method selection probability.

[0219] Based on the target sampling method and link selection probability, the target transmission link corresponding to each time slot is selected one by one from multiple transmission links.

[0220] In some embodiments, the target transmission link determination module 1530 is further configured to:

[0221] Based on the multiple link selection probabilities within each time slot, multiple sampling sub-intervals corresponding to the transmission links and a sequence number identifier corresponding to each sampling sub-interval are generated.

[0222] Generate random sample values, and select the sample sub-interval whose serial number matches the random sample value from multiple sample sub-intervals as the target sample sub-interval;

[0223] The transmission link corresponding to the target sampling sub-interval is taken as the target transmission link.

[0224] In some embodiments, the data processing module 1520 is further configured to:

[0225] Obtain the link state feature parameters corresponding to the link state feature sequence, obtain the link selection parameters of multiple transmission links, obtain the initial policy network parameters of the policy network and the initial value function network parameters of the value function network, and generate the aggregate throughput ideal difference based on the link state feature parameters.

[0226] Generate state space based on link state characteristic parameters;

[0227] Action space is generated based on link selection and transmission;

[0228] The reward space is generated based on the ideal difference in aggregated throughput.

[0229] An initial link scheduling model is generated based on the initial policy network parameters, the initial value function network parameters, the state space, the action space, and the reward space. The initial link scheduling model is then trained multiple times, and the link scheduling model is obtained based on the trained initial link scheduling model.

[0230] In some embodiments, the data processing module 1520 is further configured to:

[0231] The ideal aggregate throughput is obtained by summing up all throughput parameters;

[0232] The first throughput term is obtained by summing the ideal aggregate throughput and the actual aggregate throughput, the second throughput term is obtained by the difference between the ideal aggregate throughput and the actual aggregate throughput, and the aggregate throughput difference is obtained by the ratio of the first throughput term and the second throughput term.

[0233] The aggregate throughput difference is logarithmically processed to obtain the ideal aggregate throughput difference. In some embodiments, the data processing module 1520 is further configured to:

[0234] Obtain the training state sequence, training action sequence, and training reward sequence;

[0235] The current policy state value, the subsequent policy state value, the current value function state value, and the subsequent value function state value are obtained based on the training state sequence.

[0236] The future policy reward is obtained based on the training reward sequence, the current policy state value, and the subsequent policy state value.

[0237] The initial policy network parameters are updated using gradient descent based on future policy rewards.

[0238] The updated state value of the value function is obtained based on the subsequent value function state value and the training reward sequence. The value function error is obtained based on the difference between the current value function state value and the updated state value of the value function. The initial value function network parameters are then updated using gradient descent based on the value function error.

[0239] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, the specific implementation of the data packet transmission device of the heterogeneous transmission network is basically the same as the specific implementation of the data packet transmission method of the heterogeneous transmission network, and will not be repeated here.

[0240] In this embodiment, the data packet transmission device of the heterogeneous transmission network concatenates all link state information of each time slot within the most recent preset scheduling time period of the heterogeneous transmission network to obtain a link state information sequence characterizing the dynamic changes of each transmission link. Then, a multi-attention mechanism is used to extract features from this link state information sequence to obtain a link state feature sequence more suitable for subsequent data processing, thereby improving the accuracy of target transmission link selection in the heterogeneous transmission network. Furthermore, the sum of the throughput parameters of all transmission links within a time slot is used as a representation of the theoretically maximum aggregate throughput of multi-path transmission, and the actual aggregate throughput of all transmission links within a time slot is used to obtain an aggregate throughput difference value to characterize the difference between the actual and theoretical throughput. This aggregate throughput difference value can be used to describe the aggregation performance in the heterogeneous transmission network, so that the reward model of the initial link scheduling model can be constructed using this aggregate throughput difference value, thereby improving the aggregation performance during the training process of the initial link scheduling model. To improve the reliability of data packet transmission, this paper utilizes reinforcement learning to construct an initial link scheduling model corresponding to the actual data packet transmission process in a heterogeneous transmission network. The difference in aggregated throughput, representing the aggregation performance of the heterogeneous transmission network, is used as a training reward. Multiple training samples are used to train the decision network and value function network in the initial link scheduling model, improving the aggregation performance of the decisions obtained by the decision network. Therefore, when the decision network in the trained initial link scheduling model is used as the actual link scheduling model for target transmission link selection, the aggregation performance corresponding to the decisions can be improved, thereby enhancing the reliability of data packet transmission. Furthermore, by using the link selection probability generated from the link state feature sequence in the real-time changing heterogeneous transmission network, a suitable target sampling method is selected. This target sampling method, based on the link selection probability, selects the appropriate target transmission link for each time slot within the next preset scheduling time period from multiple transmission links, thereby improving the reliability of data packet transmission.Unlike traditional reinforcement learning models, this application outputs selection probabilities, which are then used to select links. This approach allows for exploration and avoids getting trapped in local optima, enabling further refinement of the link scheduling model in practical applications to improve the reliability of data packet transmission. In summary, this method utilizes the dynamic link state feature sequence of multiple transmission links in a heterogeneous transmission network within a preset scheduling time period to characterize the real-time transmission capabilities of these links. Combined with a link scheduling model pre-constructed based on the data packet transmission process in the heterogeneous transmission network, a link selection probability matrix is ​​obtained, indicating which transmission links, matching the link state feature sequence, will be selected for data packet transmission in the next preset scheduling time period. This improves the reliability of data packet transmission. Based on this link selection probability matrix, at least one target transmission link is selected from multiple transmission links for data packet transmission in the next preset scheduling time period, thus avoiding local optima and further enhancing the success and reliability of data packet transmission in heterogeneous transmission networks.

[0241] This application also provides an electronic device, including:

[0242] At least one memory;

[0243] At least one processor;

[0244] At least one program;

[0245] The program is stored in a memory, and the processor executes the at least one program to implement the data packet transmission method for heterogeneous transmission networks described above. The electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0246] Please see Figure 16 , Figure 16 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0247] The processor 1601 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0248] The memory 1602 can be implemented in the form of ROM (Read-Only Memory), static storage device, dynamic storage device, or RAM (Random Access Memory). The memory 1602 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1602 and is called and executed by the processor 1601 to execute the data packet transmission method of the heterogeneous transmission network of the embodiments of this application.

[0249] The input / output interface 1603 is used to implement information input and output;

[0250] The communication interface 1604 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0251] Bus 1605 transmits information between various components of the device (e.g., processor 1601, memory 1602, input / output interface 1603, and communication interface 1604);

[0252] The processor 1601, memory 1602, input / output interface 1603 and communication interface 1604 are connected to each other within the device via bus 1605.

[0253] This application embodiment also provides a storage medium, which is a computer-readable storage medium, storing a computer program that, when executed by a processor, implements the above-described data packet transmission method for heterogeneous transmission networks.

[0254] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0255] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0256] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0257] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0258] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0259] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0260] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0261] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The coupling or direct coupling or communication connection between the shown or discussed units may be through some interfaces, or indirect coupling or communication connection between the apparatus or units, and may be electrical, mechanical, or other forms.

[0262] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0263] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0264] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0265] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A data packet transmission method for a heterogeneous transmission network, characterized in that, The heterogeneous transmission network includes multiple transmission links, and the method includes: Obtain the link status feature sequence of each transmission link within a preset scheduling time period; All the link state feature sequences are input into the link scheduling model for data processing to obtain the link selection probability matrix of multiple transmission links; Based on the link selection probability matrix, at least one target transmission link is determined from multiple transmission links for the next preset scheduling time period, and the target transmission link is used for data packet transmission. The preset scheduling time period includes multiple time slots, and determining at least one target transmission link in the next preset scheduling time period from multiple transmission links based on the link selection probability matrix includes: Based on the link selection probabilities corresponding to multiple transmission links within each time slot, a sampling method selection probability is generated; Based on the sampling method selection probability, a target sampling method is selected from binary Bernoulli sampling and Monte Carlo sampling; Based on the target sampling method and the link selection probability, the target transmission link corresponding to each time slot is selected one by one from the multiple transmission links.

2. The data packet transmission method for heterogeneous transmission networks according to claim 1, characterized in that, The preset scheduling time period includes multiple time slots, and obtaining the link status feature sequence of each transmission link within the preset scheduling time period includes: Obtain the link status information of each transmission link in each time slot; The link status information of each transmission link is concatenated to obtain a link status information sequence; The link state information sequence is subjected to attention processing to obtain the link state feature sequence.

3. The data packet transmission method for heterogeneous transmission networks according to claim 1, characterized in that, The step of inputting all the link state feature sequences into the link scheduling model for data processing to obtain a link selection probability matrix for multiple transmission links includes: All the link state feature sequences are input into the link scheduling model for data processing to obtain the link selection probability of each of the transmission links in each time slot; The multiple link selection probabilities of each transmission link are concatenated to obtain the link selection probability sequence; The link selection probability matrix is ​​obtained by concatenating multiple link selection probability sequences.

4. The data packet transmission method for heterogeneous transmission networks according to claim 1, characterized in that, When the target sampling method is Monte Carlo sampling, the step of selecting the target transmission link corresponding to each time slot from multiple transmission links one by one based on the target sampling method and the link selection probability includes: Based on the multiple link selection probabilities within each time slot, multiple sampling sub-intervals corresponding to the transmission links and a sequence number identifier corresponding to each sampling sub-interval are generated. Generate random sample values, and select the sample sub-interval whose sequence number matches the random sample value from the multiple sample sub-intervals as the target sample sub-interval; The transmission link corresponding to the target sampling sub-interval is taken as the target transmission link.

5. The data packet transmission method for heterogeneous transmission networks according to claim 1, characterized in that, The generation process of the link scheduling model includes: Obtain the link state feature parameters corresponding to the link state feature sequence, obtain the link selection parameters of multiple transmission links, obtain the initial policy network parameters of the policy network and the initial value function network parameters of the value function network, and generate the aggregate throughput ideal difference based on the link state feature parameters. A state space is generated based on the link state characteristic parameters; Based on the link selection, an action space is generated for transmission. A reward space is generated based on the ideal difference in aggregated throughput; An initial link scheduling model is generated based on the initial policy network parameters, the initial value function network parameters, the state space, the action space, and the reward space. The initial link scheduling model is then trained multiple times, and the link scheduling model is obtained based on the trained initial link scheduling model.

6. The data packet transmission method for heterogeneous transmission networks according to claim 5, characterized in that, The link state characteristic parameters include the throughput parameters of each transmission link and the actual aggregate throughput of the heterogeneous transmission network. Generating the ideal difference in aggregate throughput based on the link state characteristic parameters includes: The ideal aggregate throughput is obtained by summing all the aforementioned throughput parameters; A first throughput term is obtained based on the sum of the ideal aggregate throughput and the actual aggregate throughput; a second throughput term is obtained based on the difference between the ideal aggregate throughput and the actual aggregate throughput; and an aggregate throughput difference is obtained based on the ratio between the first throughput term and the second throughput term. The aggregate throughput difference is logarithmically processed to obtain the ideal aggregate throughput difference.

7. The data packet transmission method for heterogeneous transmission networks according to claim 5, characterized in that, The training process of the initial link scheduling model includes: Obtain the training state sequence, training action sequence, and training reward sequence; Based on the training state sequence, the current policy state value, the subsequent policy state value, the current value function state value, and the subsequent value function state value are obtained. The future policy reward is obtained based on the training reward sequence, the current policy state value, and the subsequent policy state value; The initial policy network parameters are updated using gradient descent based on the future policy reward. The updated state value of the value function is obtained based on the subsequent value function state value and the training reward sequence. The value function error is obtained based on the difference between the current value function state value and the updated state value of the value function. The initial value function network parameters are then updated using gradient descent based on the value function error.

8. A data packet transmission device for a heterogeneous transmission network, characterized in that, The heterogeneous transmission network includes multiple transmission links, and the device includes: The status acquisition module is used to acquire the link status feature sequence of each transmission link within a preset scheduling time period; The data processing module is used to input all the link state feature sequences into the link scheduling model for data processing to obtain the link selection probability matrix of multiple transmission links; The target transmission link determination module is used to determine at least one target transmission link in the next preset scheduling time period from multiple transmission links based on the link selection probability matrix, and to use the target transmission link for data packet transmission. The preset scheduling time period includes multiple time slots, and determining at least one target transmission link in the next preset scheduling time period from multiple transmission links based on the link selection probability matrix includes: Based on the link selection probabilities corresponding to multiple transmission links within each time slot, a sampling method selection probability is generated; Based on the sampling method selection probability, a target sampling method is selected from binary Bernoulli sampling and Monte Carlo sampling; Based on the target sampling method and the link selection probability, the target transmission link corresponding to each time slot is selected one by one from the multiple transmission links.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the data packet transmission method for heterogeneous transmission networks according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the data packet transmission method for heterogeneous transmission networks according to any one of claims 1 to 7.

Images