Data packet transmission method of heterogeneous transmission network and related equipment

Abstract

According to the data packet transmission method of the heterogeneous transmission network and the related equipment provided by the embodiment of the invention, the heterogeneous transmission network comprises a plurality of transmission links, and the method comprises the following steps: firstly, acquiring a link state feature sequence of each transmission link in a preset scheduling time period; then, inputting all the link state feature sequences into a link scheduling model for data processing to obtain a link selection probability matrix of a plurality of transmission links; and finally, determining at least one target transmission link in the next preset scheduling time period from the plurality of transmission links based on the link selection probability matrix, and performing data packet transmission by using the target transmission link, thereby further improving the success and reliability of data packet transmission in the heterogeneous transmission network.

Description

Technical Field

[0001] The present application relates to the field of data communication technology, and in particular to a data packet transmission method and related equipment in a heterogeneous transmission network. Background Art

[0002] With the rapid development of multi-interface technology, heterogeneous transmission networks with multi-link transmission have gradually become an important part of data communication. Transmitting data simultaneously through multiple interfaces and paths can significantly improve the bandwidth utilization and transmission reliability of the network. However, since the number of transmission links and transmission data packets in heterogeneous transmission networks is usually large, selecting a suitable transmission link for each transmission data packet in a heterogeneous transmission network is a relatively complex problem.

[0003] In the related art, usually, the corresponding transmission link is allocated in advance for each data packet with different transmission requirements according to the statistical transmission capacity of different transmission links, and the data packet is transmitted. However, since the real-time transmission capacity of the actual transmission link changes dynamically, it is easy for the transmission capacity of the actual transmission link to fail to meet the transmission requirements of the data packet when the original transmission link allocation is still used for data packet transmission, resulting in insufficient reliability of data packet transmission. Summary of the invention

[0004] The embodiments of the present application provide a method and related equipment for transmitting data packets in a heterogeneous transmission network, which can improve transmission reliability when transmitting data packets in a heterogeneous transmission network.

[0005] To achieve the above-mentioned object, a first aspect of an embodiment of the present application provides a method for transmitting data packets in a heterogeneous transmission network, wherein the heterogeneous transmission network includes multiple transmission links, and the method includes:

[0006] Acquire a link state characteristic sequence of each transmission link within a preset scheduling time period;

[0007] Inputting all the link state feature sequences into a link scheduling model for data processing to obtain a link selection probability matrix of multiple transmission links;

[0008] At least one target transmission link within the next preset scheduling time period is determined from the plurality of transmission links based on the link selection probability matrix, and the target transmission link is used to transmit data packets.

[0009] In some embodiments, the preset scheduling time period includes a plurality of time slots, and the acquiring of a link state characteristic sequence of each transmission link within the preset scheduling time period includes:

[0010] Obtaining link status information of each transmission link in each time slot;

[0011] splicing the plurality of link state information of each transmission link to obtain a link state information sequence;

[0012] The link state information sequence is subjected to attention processing to obtain the link state feature sequence.

[0013] In some embodiments, inputting all the link state feature sequences into a link scheduling model for data processing to obtain a link selection probability matrix of multiple transmission links includes:

[0014] Inputting all the link state feature sequences into a link scheduling model for data processing to obtain a link selection probability of each transmission link in each time slot;

[0015] splicing the multiple link selection probabilities of each transmission link to obtain the link selection probability sequence;

[0016] A plurality of the link selection probability sequences are concatenated to obtain the link selection probability matrix.

[0017] In some embodiments, the preset scheduling time period includes a plurality of time slots, and determining at least one target transmission link within the next preset scheduling time period from the plurality of transmission links based on the link selection probability matrix includes:

[0018] Generate a sampling method selection probability based on the link selection probabilities corresponding to the plurality of transmission links in each time slot;

[0019] Selecting a target sampling method from binary Bernoulli sampling and Monte Carlo sampling based on the sampling method selection probability;

[0020] The target transmission link corresponding to each time slot is selected one by one from the plurality of transmission links based on the target sampling method and the link selection probability.

[0021] In some embodiments, when the target sampling method is Monte Carlo sampling, selecting a target transmission link corresponding to each time slot from a plurality of transmission links one by one based on the target sampling method and the link selection probability includes:

[0022] Based on multiple link selection probabilities in each time slot, generate multiple sampling subintervals corresponding to the transmission links and a sequence number identifier corresponding to each sampling subinterval;

[0023] Generate a random sampling value, and select the sampling sub-interval whose sequence number identifier matches the random sampling value from the multiple sampling sub-intervals as the target sampling sub-interval;

[0024] The transmission link corresponding to the target sampling subinterval is used as a target transmission link.

[0025] In some embodiments, the link scheduling model generation process includes:

[0026] Acquire link state characteristic parameters corresponding to the link state characteristic sequence, acquire link selection parameters of the plurality of transmission links, and acquire initial policy network parameters of the policy network and initial value function network parameters of the value function network, and generate an ideal difference of aggregate throughput based on the link state characteristic parameters;

[0027] Generate a state space based on the link state characteristic parameters;

[0028] Selecting transmission based on the link generates an action space;

[0029] generating a reward space based on the aggregate throughput ideal difference;

[0030] An initial link scheduling model is generated based on the initial strategy network parameters, the initial value function network parameters, the state space, the action space and the reward space, and the initial link scheduling model is trained multiple times, and the link scheduling model is obtained based on the trained initial link scheduling model.

[0031] In some embodiments, the link state characteristic parameters include a throughput parameter of each of the transmission links and an actual aggregate throughput of the heterogeneous transmission network, and generating an ideal difference of aggregate throughput based on the link state characteristic parameters includes:

[0032] Accumulating all the throughput parameters to obtain an ideal aggregate throughput;

[0033] obtaining a first throughput item based on a sum of the ideal aggregate throughput and the actual aggregate throughput, obtaining a second throughput item based on a difference between the ideal aggregate throughput and the actual aggregate throughput, and obtaining an aggregate throughput difference based on a ratio of the first throughput item to the second throughput item;

[0034] The aggregate throughput difference is logarithmically processed to obtain the ideal aggregate throughput difference.

[0035] In some embodiments, the training process of the initial link scheduling model includes:

[0036] Obtain training state sequence, training action sequence and training reward sequence;

[0037] Based on the training state sequence, a current strategy state value, a subsequent strategy state value, a current value function state value, and a subsequent value function state value are obtained;

[0038] Obtaining a future policy reward based on the training reward sequence, the current policy state value, and the subsequent policy state value;

[0039] Performing gradient descent update on the initial policy network parameters based on the future policy reward;

[0040] A value function update state value is obtained based on the subsequent value function state value and the training reward sequence, a value function error is obtained based on the difference between the current value function state value and the value function update state value, and the initial value function network parameters are updated by gradient descent based on the value function error.

[0041] To achieve the above-mentioned object, a second aspect of an embodiment of the present application provides a data packet transmission device for a heterogeneous transmission network, wherein the heterogeneous transmission network includes multiple transmission links, and the device includes:

[0042] A state acquisition module, used to acquire a link state feature sequence of each transmission link within a preset scheduling time period;

[0043] A data processing module, used for inputting all the link state feature sequences into a link scheduling model for data processing to obtain a link selection probability matrix of multiple transmission links;

[0044] The target transmission link determination module is used to determine at least one target transmission link within the next preset scheduling time period from the multiple transmission links based on the link selection probability matrix, and use the target transmission link to transmit data packets.

[0045] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the data packet transmission method of the heterogeneous transmission network as described in the first aspect.

[0046] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium, and the storage medium stores a computer program. When the computer program is executed by a processor, it implements the data packet transmission method of the heterogeneous transmission network described in the first aspect above.

[0047] The embodiments of the present application propose a data packet transmission method and related equipment for a heterogeneous transmission network. The heterogeneous transmission network includes multiple transmission links. The method includes: first, obtaining a link state feature sequence of each transmission link within a preset scheduling time period; then, inputting all link state feature sequences into a link scheduling model for data processing to obtain a link selection probability matrix for multiple transmission links; finally, determining at least one target transmission link within the next preset scheduling time period from the multiple transmission links based on the link selection probability matrix, and using the target transmission link to transmit data packets. The embodiment of the present application utilizes the link state characteristic sequence of multiple transmission links that change in a heterogeneous transmission network and dynamically changes within a preset scheduling time period to characterize the real-time transmission capabilities of the current multiple transmission links, and combines it with a link scheduling model that is pre-constructed corresponding to the data packet transmission process performed in the heterogeneous transmission network, thereby obtaining a link selection probability matrix in which multiple transmission links that match the link state characteristic sequence are selected for data packet transmission within the next preset scheduling time period, so as to improve the reliability of data packet transmission, and select at least one target transmission link from the multiple transmission links for data packet transmission in the next preset scheduling time period according to the link selection probability matrix, thereby avoiding falling into local optimality and further improving the successful reliability of data packet transmission in the heterogeneous transmission network.

[0048] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a structural diagram of a heterogeneous transmission network provided in one embodiment of the present application.

[0050] Figure 2 This is a flowchart of a method for transmitting data packets in a heterogeneous transmission network provided by another embodiment of the present application.

[0051] Figure 3 yes Figure 2 Flow chart of step 201 in FIG.

[0052] Figure 4 This is a flowchart of attention feature extraction provided by another embodiment of the present application.

[0053] Figure 5 yes Figure 2 Flow chart of step 202 in FIG.

[0054] Figure 6 This is a flowchart of generating a link scheduling model provided by another embodiment of the present application.

[0055] Figure 7 This is a structural diagram of a strategy network provided by another embodiment of the present application.

[0056] Figure 8 This is a structural diagram of a value function network provided by another embodiment of the present application.

[0057] Fig. 9 yes Figure 6 Flowchart of step 601 in FIG.

[0058] Fig.10 This is a flowchart of training an initial link scheduling model provided by another embodiment of the present application.

[0059] Fig.11 This is a schematic diagram of a scheduling model based on a Markov decision process provided by another embodiment of the present application.

[0060] Fig.12 yes Figure 2 Flow chart of step 203 in FIG.

[0061] Fig.13 yes Fig.12 Flowchart of step 1203 in FIG.

[0062] Fig.14 It is a schematic diagram of the overall concept of a data packet transmission method for a heterogeneous transmission network provided in one embodiment of the present application.

[0063] Fig.15 It is a structural schematic diagram of a data packet transmission device for a heterogeneous transmission network provided in one embodiment of the present application.

[0064] Fig.16 It is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0066] It should be noted that although the functional modules are divided in the device schematic and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0068] First, some nouns involved in this application are analyzed:

[0069] Deep reinforcement learning is a branch of machine learning that combines the powerful representation learning capabilities of deep learning with the goal-oriented mechanism of reinforcement learning, allowing agents to learn how to make optimal decisions in an uncertain environment. Specifically, deep reinforcement learning uses deep neural networks as function approximators to estimate the value of actions or directly output action strategies, so that agents can learn the optimal path to achieve specific goals through trial and error in the process of interacting with the environment. This approach greatly improves the learning and generalization capabilities of the model, especially when dealing with high-dimensional input data such as images and natural language.

[0070] With the rapid development of multi-interface technology, heterogeneous transmission networks with multi-link transmission have gradually become an important part of data communication. Transmitting data simultaneously through multiple interfaces and paths can significantly improve the bandwidth utilization and transmission reliability of the network. However, since the number of transmission links and transmission data packets in heterogeneous transmission networks is usually large, selecting a suitable transmission link for each transmission data packet in a heterogeneous transmission network is a relatively complex problem.

[0071] In the related art, usually, the corresponding transmission link is allocated in advance for each data packet with different transmission requirements according to the statistical transmission capacity of different transmission links, and the data packet is transmitted. However, since the real-time transmission capacity of the actual transmission link changes dynamically, it is easy for the transmission capacity of the actual transmission link to fail to meet the transmission requirements of the data packet when the original transmission link allocation is still used for data packet transmission, resulting in insufficient reliability of data packet transmission.

[0072] In order to improve transmission reliability when performing data packet transmission in a heterogeneous transmission network, an embodiment of the present application utilizes a link state feature sequence of multiple transmission links that changes in a heterogeneous transmission network and that changes dynamically within a preset scheduling time period to characterize the real-time transmission capabilities of the current multiple transmission links, and combines it with a link scheduling model that is pre-constructed corresponding to the data packet transmission process performed in the heterogeneous transmission network, thereby obtaining a link selection probability matrix in which multiple transmission links that match the link state feature sequence are selected for data packet transmission within the next preset scheduling time period, so as to improve the reliability of data packet transmission, and select at least one target transmission link from the multiple transmission links for data packet transmission within the next preset scheduling time period according to the link selection probability matrix, thereby avoiding falling into local optimality and further improving the successful reliability of data packet transmission in the heterogeneous transmission network.

[0073] In order to better describe the data packet transmission method of the heterogeneous transmission network provided by the present application, the heterogeneous transmission network applied to the data packet transmission method of the heterogeneous transmission network is first described below. Figure 1 , is a schematic diagram of a heterogeneous transmission network provided in an embodiment of the present application. Figure 1 As shown in , the heterogeneous transmission network includes a transmitting device, a receiving device, a mobile router (SMR), an aggregation server (SAR), and multiple transmission links. The transmitting device sends data packet information to the receiving device through the mobile router and the aggregation server, and when the mobile router sends a data packet to the aggregation server, a target transmission link is selected from multiple transmission links to transmit the data packet.

[0074] In conjunction with the above-mentioned heterogeneous transmission network, the data packet transmission method and related devices of the heterogeneous transmission network provided in the embodiment of the present application will be further described below. The data packet transmission method of the heterogeneous transmission network provided in the embodiment of the present application can be applied to a transmitting terminal device, a mobile router or a processor device in a heterogeneous transmission network.

[0075] The following will specifically describe the data packet transmission method of the heterogeneous transmission network in the embodiment of the present application. Figure 2 , is an optional flow chart of a method for transmitting data packets in a heterogeneous transmission network provided in an embodiment of the present application, Figure 2 The method in the embodiment may include but is not limited to steps 201 to 203. Figure 2 The order of step 201 to step 203 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0076] Step 201: Obtain a link state feature sequence of each transmission link within a preset scheduling time period.

[0077] The following is a detailed description of step 201.

[0078] In some embodiments, in response to a transmission request for a data packet, in order to select a suitable target transmission link for the data packet in a heterogeneous transmission network so as to perform data transmission in the next preset scheduling time period to improve the reliability of data packet transmission, it is necessary to first obtain a link state characteristic sequence of all transmission links in the heterogeneous transmission network within the most recent preset scheduling time period, so as to use the link state characteristic sequence to characterize the real-time transmission capability of the transmission link, so as to facilitate the subsequent use of the link state characteristic sequence for target transmission link selection.

[0079] The data information in the link state characteristic sequence usually includes the number of data packets, the delay, throughput and congestion of each transmission link. The following will further describe how to obtain the link state characteristic sequence.

[0080] Reference Figure 3 , obtaining a link state feature sequence of each transmission link within a preset scheduling time period, including the following steps 301 to 303.

[0081] Step 301: Obtain link status information of each transmission link in each time slot.

[0082] Step 302: Multiple link state information of each transmission link are concatenated to obtain a link state information sequence.

[0083] Step 303: Perform attention processing on the link state information sequence to obtain a link state feature sequence.

[0084] Steps 301 to 303 are described in detail below.

[0085] In some embodiments, a detection management module is provided in the heterogeneous transmission network, which is used to obtain in real time the link status information of each transmission link in each time slot of a preset scheduling time period (which generally includes information such as the delay, throughput and congestion status of each transmission link).

[0086] Then, multiple link state information of each transmission link in all time slots of each transmission link are spliced ​​together to obtain a link state information sequence of all link state information of each transmission link in a preset scheduling time period.

[0087] After obtaining the link state information sequence, the Transformer encoder framework based on the multi-head self-attention mechanism is used to extract the feature representation of the state sequence to obtain the link state feature sequence corresponding to the link state information sequence. This encoding framework calculates the abstract representation of the current network state through position encoding and multi-layer self-attention encoding, captures the element features at each position in the input sequence, and enables the subsequent policy network to obtain more information about changes in the network environment. Figure 4 , is a flowchart of an attention feature extraction process provided by an embodiment of the present application. Figure 4 As shown in , the network first transforms the input sequence into an embedding vector through the embedding layer and adjusts its amplitude to control the gradient of the model. Secondly, the embedding vector is encoded through the position encoding layer, encoding layer, and mask encoding layer. Multi-layer encoding can extract richer features and generate abstract representations of data from multiple perspectives, which is suitable for network state spaces with multi-dimensional attributes. Finally, the linear layer is used to perform a linear transformation on the embedded vector after the encoding operation, and the vector is adjusted back to the dimension of the original state sequence for subsequent action reasoning by the policy network.

[0088] At the same time, the detection management module will also perform real-time management and data backup of the link state information and link state information sequence obtained by detection, and regularly deliver relevant information of the link state information to the online scheduling module to facilitate the online scheduling module to perform subsequent scheduling reasoning work, and the backed-up data files are used for training the policy network.

[0089] Through the above steps 301 to 303, all link state information of each time slot in the heterogeneous transmission network obtained in the most recent preset scheduling time period is spliced ​​to obtain a link state information sequence that characterizes the dynamic changes of each transmission link, and then the link state information sequence is subjected to feature extraction using a multi-attention mechanism to obtain a link state feature sequence that is more suitable for subsequent data processing, thereby improving the accuracy of selecting target transmission links in the heterogeneous transmission network.

[0090] Step 202: input all link state feature sequences into the link scheduling model for data processing to obtain a link selection probability matrix of multiple transmission links.

[0091] The following is a detailed description of step 202.

[0092] In some embodiments, after obtaining a link state feature sequence for characterizing the real-time transmission capacity of each transmission link, all link state feature sequences are input into a link scheduling model that is pre-constructed based on the data packet transmission process in a heterogeneous transmission network for data processing to obtain a link selection probability matrix for multiple transmission links that match the link state feature sequence and are selected for data packet transmission within the next preset scheduling time period, so as to facilitate the subsequent selection of a target transmission link from multiple transmission links based on the link selection probability matrix.

[0093] The following further describes how to obtain the link selection probability matrix of the corresponding multiple transmission links according to all link state feature sequences.

[0094] Reference Figure 5 , input all link state feature sequences into the link scheduling model for data processing to obtain the link selection probability matrix of multiple transmission links, including the following steps 501 to 503.

[0095] Step 501: All link state feature sequences are input into a link scheduling model for data processing to obtain the link selection probability of each transmission link in each time slot.

[0096] The following is a detailed description of step 501.

[0097] In some embodiments, after obtaining the link state feature sequence used to characterize the real-time transmission capability of each transmission link, all link state feature sequences are taken as input parameters and input into the link scheduling model for data processing to obtain the link selection probability corresponding to each transmission link being selected in each time slot within the preset scheduling time period, so as to select the corresponding target transmission link in each time slot from multiple transmission links based on the link selection probability.

[0098] The following will first describe how to generate a link scheduling model.

[0099] Reference Figure 6 , the link scheduling model generation process includes the following steps 601 to 605.

[0100] Step 601: Obtain link state characteristic parameters corresponding to the link state characteristic sequence, obtain link selection parameters of multiple transmission links, and obtain initial policy network parameters of the policy network and initial value function network parameters of the value function network, and generate an ideal difference in aggregate throughput based on the link state characteristic parameters.

[0101] Step 601 is described in detail below.

[0102] In some embodiments, in order to generate a suitable link scheduling model, it is necessary to first obtain the link state characteristic parameters corresponding to the link state characteristic sequence, which generally include the data volume of the data packet in each time slot, the delay of each transmission link, the packet loss rate, the number of unreply data packets, the number of unsent data packets and other information, which can fully characterize the transmission quality of each transmission link in the current heterogeneous transmission network, so that the intelligent agent used to select the target transmission link can more finely adjust the data packet allocation behavior.

[0103] In addition, it is also necessary to obtain link selection parameters of multiple transmission links in the heterogeneous transmission network, that is, a set of selectable transmission links in each time slot.

[0104] And, obtain the initial policy network parameters θ of the policy network and the initial value function network parameters ω of the value function network pre-generated according to the A2C algorithm in the deep reinforcement learning module.

[0105] It is understandable that the A2C network contains two key models, namely the policy network and the value function network. Among them, the policy network learns new scheduling strategies based on the environment state and feedback rewards, so that the online scheduling algorithm has network adaptability. The value function network evaluates the advantages of each action through the time difference error between the state values, and assists in updating the policy network.

[0106] The A2C algorithm reduces the variance of the policy gradient method by guiding the value function, thereby improving the stability of the algorithm and making it easier to converge to a better strategy. In the actual packet scheduling process, the flexible transmission mechanism needs to make frequent decisions for a large number of packets, and the immediacy requirement is high. A fast policy learning rate has greater advantages. At the same time, the aggregation performance of multipath transmission depends not only on the immediate reward of the current decision, but also needs to be considered from a more macroscopic long-term benefit perspective. The idea of ​​collaborative optimization of the policy network and the value function network helps to balance the impact of immediate performance rewards and long-term performance benefits on the scheduling algorithm. The policy network and the value function network will be further described below.

[0107] Reference Figure 7 , is a schematic diagram of a strategy network structure provided by an embodiment of the present application. Figure 7As shown in , the policy network sets the state feature extraction network as a submodule of the policy network. Two fully connected layers are set. The ReLU activation function is also used between the connection layers, and the action probability is output through the Softmax function. The network works in parallel with the online scheduling algorithm, responsible for training network parameters, optimizing scheduling strategies, and synchronizing parameters with the online scheduling module in real time. After the agent performs actions according to the online scheduling algorithm, the collected environmental states and the corresponding feedback rewards are input into the policy training module. This module uses the policy gradient method to maximize the reward expectation to guide the network optimization direction, so that the probability of high-reward actions being selected increases and the probability of low-reward actions decreases. After the training is completed, the policy training model copies the updated parameters to the policy network of the online scheduling algorithm to realize the function of the algorithm adjusting its own strategy as the network changes.

[0108] The value function estimation model is responsible for estimating the expected rewards that the agent may receive in the future starting from the current state, recorded as the state value V(s). This model actually evaluates the environmental quality of multiple transmission links. If the transmission environment is good, it has the potential to obtain more rewards, and the state value is higher. When reasoning about the actions of the agent, it is necessary not only to consider the immediate feedback rewards, but also to take future rewards into consideration. The value function estimation model evaluates the quality of the transmission environment in each time slot, quantifies the impact of the action on the long-term benefits, and assists in guiding the optimization direction of the policy network. Figure 8 , is a schematic diagram of the structure of a value function network provided in an embodiment of the present application. Figure 8 As shown in , the value function network is responsible for calculating the state according to the network state. It consists of two fully connected layers, and the nonlinear representation is introduced between the layers through the ReLU activation function. In order to learn the state characteristics in each time slot, the input of the value function network is actually the state feature vector obtained by deforming and reshaping the state sequence, and its dimension is Its output is the corresponding state value.

[0109] In addition, in order to generate a suitable reward model, an ideal difference of aggregate throughput is also generated based on the link state characteristic parameters, as described below.

[0110] Reference Fig. 9 , generating an ideal difference of aggregate throughput based on link state characteristic parameters, including the following steps 901 to 903.

[0111] Step 901: Accumulate all throughput parameters to obtain an ideal aggregate throughput.

[0112] Step 902: obtain a first throughput item based on the sum of the ideal aggregate throughput and the actual aggregate throughput, obtain a second throughput item based on the difference between the ideal aggregate throughput and the actual aggregate throughput, and obtain an aggregate throughput difference based on the ratio of the first throughput item to the second throughput item.

[0113] Step 903: Perform logarithmic processing on the aggregate throughput difference to obtain an ideal aggregate throughput difference.

[0114] Steps 901 to 903 are described in detail below.

[0115] In some embodiments, the throughput parameter of each transmission link is extracted from the link state characteristic parameter It is expressed as the throughput of the sth transmission link in the tth time slot, and then the throughput parameters of all transmission links in the tth time slot are accumulated. Get the ideal aggregate throughput

[0116] It is also necessary to extract the common throughput of all transmission links in each time slot (such as the tth time slot) from the link state characteristic parameters, that is, the actual aggregate throughput

[0117] Then, the first throughput term is obtained based on the sum of the ideal aggregate throughput and the actual aggregate throughput The second throughput term is obtained based on the difference between the ideal aggregate throughput and the actual aggregate throughput. Based on the ratio of the first throughput item to the second throughput item, the aggregate throughput difference is obtained as shown in the following formula (1).

[0118]

[0119] It can be seen that the actual aggregation throughput is close to the ideal aggregation throughput, and the aggregation effect is good; otherwise, it means that the actual aggregation bandwidth is small and the aggregation performance is poor.

[0120] Through the above steps 901 to 903, the sum of the throughput parameters of all transmission links in the time slot is used as a representation of the maximum aggregate throughput that can be achieved by multi-path transmission in theory, and the actual aggregate throughput of all transmission links as a whole in the time slot, to obtain an aggregate throughput difference used to characterize the difference between the actual and theoretical values, so that the aggregate throughput difference can be used to describe the aggregation performance in the heterogeneous transmission network, so as to facilitate the subsequent use of the aggregate throughput difference to construct a reward model of the initial link scheduling model, so that the training process of the initial link scheduling model aims to improve the aggregation performance, thereby improving the reliability of data packet transmission.

[0121] Step 602: Generate a state space based on link state characteristic parameters.

[0122] Step 603: Generate an action space based on link selection transmission.

[0123] Step 604: Generate a reward space based on the aggregated throughput ideal difference.

[0124] Step 605: Generate an initial link scheduling model based on the initial strategy network parameters, the initial value function network parameters, the state space, the action space, and the reward space, train the initial link scheduling model multiple times, and obtain a link scheduling model based on the trained initial link scheduling model.

[0125] Steps 602 to 605 are described in detail below.

[0126] In order to select a suitable target transmission link from multiple transmission links for data packet transmission, this embodiment transforms the data packet scheduling decision problem into a Markov decision process based on the state of the heterogeneous transmission network. The Markov decision process includes six key parameters: the agent, the state space, the action space of the agent, the environmental feedback (i.e., the reward space) obtained after the agent makes a decision, and the decision strategy.

[0127] Among them, in the scheduling decision-making process, the intelligent agent refers to the scheduling mechanism adopted in the heterogeneous transmission network (it can also be the execution body itself that executes the data packet transmission method of the heterogeneous transmission network in the embodiment of the present application), which is responsible for interacting with the heterogeneous transmission network and executing corresponding actions according to the strategy.

[0128] Then, after obtaining the link state characteristic parameters corresponding to the link state characteristic sequence, the link state characteristic parameters are used as the state space, which includes the amount of data packets in each time slot, the delay of each transmission link, the packet loss rate, the number of unreply packets, the number of unsent packets and other information, which can fully characterize the transmission quality of the current heterogeneous transmission network, so that the intelligent agent can adjust the packet allocation behavior more finely. After the detection management module completes the detection, the above n link information and global information are spliced ​​in sequence, and the state space obtained is S, and its sequence dimension is

[0129] For action decisions, the agent takes corresponding actions according to the current state and strategy. During the transmission process of multiple transmission links, the scheduling algorithm selects links for data packets based on the current environment state. Therefore, after obtaining the link selection parameters of multiple transmission links, the actual behavior of the agent is actually the link selection process, and its set is the set of links available for selection. Assuming that there are n transmission links in total, the action space of the agent is A = (a1, a2, ... a n ), the sequence dimension is

[0130] Regarding the reward space, after the agent performs an action, new data packets are allocated to each link. The current heterogeneous transmission network environment will definitely change, and the overall transmission performance will also be affected. The agent's behavior directly affects the aggregate transmission performance in this process. If a better link is selected for the data packet, the transmission performance will be improved. On the contrary, if the agent selects a worse link, the transmission performance may gradually decrease, or even cause serious performance degradation. In order to evaluate the pros and cons of the agent's current behavior, the agent uses the relevant algebraic expression of the multi-path transmission aggregation rate as the evaluation value of the immediate reward to guide the optimization direction of the decision-making strategy. Therefore, the reward space is determined based on the above-mentioned ideal difference in aggregate throughput, that is, the above formula (1). The larger the reward value, the closer the actual aggregate throughput is to the ideal aggregate throughput, and the better the aggregation effect; on the contrary, the smaller the actual aggregate bandwidth is, the worse the aggregation performance is. When the actual aggregation effect is much smaller than the ideal aggregate throughput, the reward value is approximately 0.

[0131] However, in practical applications, more attention needs to be paid to the performance during long-term transmission. Therefore, the evaluation of the transmission quality of multi-links does not only depend on the aggregation effect of a certain time slot, but also needs to consider the average aggregate transmission performance over a long period of time. Under this consideration, the concept of future rewards is introduced, which is the expectation of the weighted sum of future immediate rewards after the agent performs the current action, as shown in the following formula (2).

[0132]

[0133] Among them, G t represents the future accumulated reward after time t, R t+k+1 Represents the future immediate reward for subsequent time slots. γ k Represents the discount factor for the next k moments, which is responsible for balancing the importance of the current immediate reward and the future accumulated reward, so that the agent can more comprehensively evaluate the effect of its own behavior. A larger discount factor will make the agent pay more attention to the future accumulated reward, while a smaller discount factor can highlight the impact of the immediate reward.

[0134] Regarding the decision-making strategy (Policy), in the Markov decision process, the agent calculates the conditional probability distribution of the action according to the strategy based on the current state. That is, in the current state s, the probability of the agent performing action a is shown in the following formula (3).

[0135] π(a|s)=p(A=a|S=s) (3)

[0136] In the transmission process of multiple transmission links, the scheduling algorithm calculates the probability distribution of data packets assigned to n transmission links through the current environmental state of the heterogeneous transmission network. Assume that in the state of time slot t, the probability of the agent selecting transmission link i is pi , then the probability distribution of data packets assigned to each link is

[0137] In the actual multi-link transmission process, if the detection interval is too large, a lot of detailed network information will be lost, making it difficult for the scheduling algorithm to perceive the changes in the current network. However, if the online scheduling algorithm updates the strategy according to the feedback reward after each detection, the detection granularity is too small, and it is difficult to characterize the impact of the scheduling action on the link state from a global perspective. At the same time, the deep learning model used by the online scheduling algorithm needs to spend a certain amount of computing time when inferring the action probability distribution, and in general, it is greater than the detection interval. Therefore, in order to balance the network information detection frequency and the strategy update frequency, the actual input of the online scheduling algorithm is not the state vector of a single time slot defined in the state space, but a state sequence composed of state vectors detected within a preset scheduling time period (i.e., multiple time slots). Correspondingly, the actual output of the algorithm is an action probability distribution sequence of the same length as the sequence, and the feedback reward is the aggregate performance benefit after the agent executes a series of actions. In this way, the details of network changes can be retained and the detection frequency can be matched with the strategy update frequency. The specific sequence expression is shown in the following formula (4).

[0138]

[0139] After constructing the state space, action space, reward space, initial decision network parameters and initial value function parameters corresponding to the Markov decision, the initial link scheduling model will be constructed based on the idea of ​​the A2C algorithm, and then the initial link scheduling model will be trained multiple times using the training samples to obtain an actually usable link scheduling model using the trained initial link scheduling model.

[0140] The embodiment of the present application utilizes the dominant action-evaluation reinforcement learning framework (A2C) to train the policy network used in the initial link scheduling model. The A2C network combines the advantages of policy gradients and value function estimation, and aims to learn strategies that maximize expected rewards, which is consistent with the purpose of this application to improve transmission aggregation performance in multiple transmission links. Whenever the detection management module completes the staged collection of state-action-reward tuples, the offline training module stores the tuple data in the experience replay cache in turn, and performs preprocessing, which is used to train the policy network and the value function network, respectively. The following will detail how to train the initial link scheduling model.

[0141] Reference Fig.10 The training process of the initial link scheduling model includes the following steps 1001 to 1005.

[0142] Step 1001: Obtain a training state sequence, a training action sequence, and a training reward sequence.

[0143] Step 1002: Based on the training state sequence, obtain the current strategy state value, the subsequent strategy state value, the current value function state value and the subsequent value function state value.

[0144] Step 1003: Obtain future policy rewards based on the training reward sequence, the current policy state value, and the subsequent policy state value.

[0145] Step 1004: Perform gradient descent update on the initial policy network parameters based on the future policy rewards.

[0146] Step 1005: Obtain the value function update state value based on the subsequent value function state value and the training reward sequence, obtain the value function error based on the difference between the current value function state value and the value function update state value, and perform gradient descent update on the initial value function network parameters based on the value function error.

[0147] Steps 1001 to 1005 are described in detail below.

[0148] In some embodiments, the training of the policy network aims to maximize the accumulated reward, hoping to generate a better policy π θ (a t |s t ), so that the agent can choose the right action and get more rewards. Its training process uses the policy gradient method to calculate the action advantage, determine the gradient direction of the reward expectation, and update the parameters in the direction of maximizing the reward growth.

[0149] Therefore, firstly, the training samples of M preset scheduling time periods used for model training of the initial link scheduling model are obtained, which correspond to the training state sequence, training action sequence and training reward sequence, that is, τ′={(S1, A1, r1), (S2, A2, r2), ..., (S M , A M , r M )}. Then, based on the training state sequence, the current strategy state value V(S i ), the subsequent strategy state value V(S i+1 ).

[0150] Since the sampling selection of actions has a certain degree of randomness, the feedback rewards generated by different action sequences may vary greatly, making the variance of the expected gradient estimate of the accumulated reward high and difficult to converge to a stable state quickly. Therefore, the policy training module uses the state value predicted by the value function estimation module as the baseline for obtaining future rewards after executing the action, and calculates the future policy rewards of each action that exceed the baseline, as shown in the following formula (5).

[0151] ADV i =ri +γV(S i+1 )-V(S i ) (5)

[0152] Among them, S i and r i is the state sequence and feedback reward of the i-th time step, S i+1 is the state sequence of the next time step, γ is the discount factor, which is responsible for balancing the importance of immediate rewards and accumulated rewards, and V(·) is the state value calculated by the value function estimation module based on the state sequence, which represents the average level of future accumulated rewards obtained by the agent under this state sequence. Therefore, when the agent is in the state sequence S i Execute action sequence A i And get instant rewards i , the reward it can obtain from time step i is r i +γV(S i+1 ). This reward value is equal to the average future reward V(S i ) is the relative reward ADV for the agent to take action i , reflecting the potential of each action. The larger the relative reward, the more advantageous the action sequence is and the more rewards can be obtained. At the same time, the relative reward compares the advantages between each action, which helps to eliminate the influence of random noise, reduce variance, and make training more stable.

[0153] Through the above analysis, the training objective function expression of the policy network is shown in the following formula (6).

[0154] J(θ)=log(π θ (A i |S i ))·ADV i +β·H(π θ (Ai|S i )) (6)

[0155] This function consists of two parts: policy reward and entropy regularization. The former determines the direction of parameter update based on the advantage of the action, and maximizes the accumulated reward through training. The latter adds a certain degree of exploration to the training process, because when the entropy value is large, the action probability distribution is more even, and the action selection of the agent is more random.

[0156] Among them, θ is the policy network parameter, logπ θ (A i |S i ) is the log-likelihood probability of the action, ADV i is the relative future reward, β is the entropy coefficient, H(π θ (A i |Si )) is the entropy of the policy network. ADV i The sign of determines the update direction of the policy network parameters. If the value is negative, it means that the action receives a reward below the average level, and its probability should be reduced, and the parameters are updated along the negative gradient. Conversely, when the action has an advantage, the parameters are updated along the positive gradient to increase the probability of the action. Policy entropy represents the randomness of the policy network. If β is large, it means that the parameter update process tends to maintain the randomness of the policy.

[0157] The above process involves the calculation of action probabilities and relative future rewards. However, after the policy network infers the action probability distribution, it is still necessary to determine the actual action to be performed through Bernoulli sampling and Monte Carlo sampling, and there is a certain probability that multiple actions will be selected at the same time. Therefore, the log-likelihood probability shown in the following formula (7) is actually the sum of the log-likelihood probabilities of the selected actions. Assuming that k actions are selected, the log-likelihood probability of the action should be:

[0158]

[0159] In summary, the policy training model uses the policy gradient method based on the relative future reward of the action to update the parameters in the direction of increasing the reward. Therefore, the initial decision parameters are updated by gradient descent as shown in the following formula (8), where α is the learning rate and determines the update step size.

[0160]

[0161] The training process is shown in Algorithm 1 below. In actual network training, in order to simplify mathematical calculations, the training process of reinforcement learning is often expressed as a minimization optimization problem. To this end, Algorithm 1 introduces a negative sign in the objective function, transforming the problem of maximizing accumulated rewards into the problem of minimizing policy losses.

[0162]

[0163]

[0164] For the value function network V(S; ω), it is responsible for estimating the average level of rewards received by the agent in the current state. Its training goal is to minimize the error between the network's estimated rewards and the actual rewards. Therefore, after obtaining the training state sequence, the current strategy state value is obtained based on the training state sequence. Subsequent strategy status value

[0165] Among them, the value function network adopts the mean square error method to train the value function network parameters, and its objective function is shown in the following formula (9).

[0166]

[0167] During the training process, the initial value function network parameters are updated in the direction of the negative gradient of the loss function in order to reduce the error value. The initial value function network parameter update expression is shown in the following formula (10).

[0168]

[0169] To summarize, the training process of the value function network is shown in Algorithm 2 below.

[0170]

[0171]

[0172] Reference Fig.11 , is a schematic diagram of a scheduling model based on a Markov decision process provided in an embodiment of the present application. Fig.11 As shown in , the current environmental state of the heterogeneous transmission network is composed of network indicators such as the delay, packet loss rate, and congestion status of each transmission link, and the environmental states at different time slots constitute the state space, which is used to characterize the environmental change process of the heterogeneous transmission network. The scheduler proposed in this section is regarded as an intelligent agent that continuously interacts with the environment, and its main behavior is to distribute data packets on each link. Therefore, the behavior set of the agent is the set of available transmission links. The scheduler selects the best transmission link for the data packet from different sub-flows, which can be regarded as the decision-making process of the agent to choose different behaviors according to different system states, and the aggregate transmission performance of the transmission system is the feedback of the decision. In this framework, each action of the agent will have a corresponding environmental feedback reward, that is, the change of aggregate transmission performance. The agent adopts the action-feedback-learning strategy to learn the effects of different interactive behaviors with the network environment under different states, so as to adjust its subsequent behavior adaptively in order to obtain more rewards.

[0173] Through the above steps 601 to 605, and steps 1001 to 1005, the idea of ​​reinforcement learning is used to construct an initial link scheduling model corresponding to the actual data packet transmission process in the heterogeneous transmission network, and the aggregation throughput difference that characterizes the aggregation performance of the heterogeneous transmission network is used as a training reward, so that the decision network and the value function network in the initial link scheduling model are trained using multiple training samples to improve the aggregation performance of the decision obtained by the decision network, so that when the decision network in the trained initial link scheduling model is used as the actual link scheduling model for target transmission link selection, the aggregation performance corresponding to the decision can be improved, thereby improving the reliability of data packet transmission.

[0174] Step 502: Multiple link selection probabilities of each transmission link are concatenated to obtain a link selection probability sequence.

[0175] Step 503: multiple link selection probability sequences are concatenated to obtain a link selection probability matrix.

[0176] Steps 502 to 503 are described in detail below.

[0177] In some embodiments, all link state feature sequences are taken as input parameters and input into the link scheduling model for data processing to obtain the link selection probability corresponding to each transmission link being selected in each time slot within the preset scheduling time period. Afterwards, the link selection probabilities of each transmission link in multiple time slots of the preset scheduling time period are concatenated to obtain the link selection probability sequence Then, the link selection probability sequence p of multiple transmission links is further transformed into n The links are spliced ​​to obtain a link selection probability matrix of multiple transmission links in multiple time slots within a preset scheduling time period.

[0178] Step 203: Determine at least one target transmission link within the next preset scheduling time period from the multiple transmission links based on the link selection probability matrix, and use the target transmission link to transmit the data packet.

[0179] Step 203 is described in detail below.

[0180] In some embodiments, after obtaining the link selection probability matrix, the target transmission link corresponding to each time slot in the next preset scheduling time period will be determined from multiple transmission links based on the link selection probability matrix, so as to utilize these target transmission links for data packet transmission in the next preset scheduling time period. The following will further describe how to determine the target transmission link based on the link selection probability matrix.

[0181] Reference Fig.12 , determining at least one target transmission link within a next preset scheduling time period from multiple transmission links based on the link selection probability matrix, including the following steps 1201 to 1203.

[0182] Step 1201: Generate a sampling method selection probability based on the link selection probabilities corresponding to multiple transmission links in each time slot.

[0183] Step 1202: Select a target sampling method from binary Bernoulli sampling and Monte Carlo sampling based on the sampling method selection probability.

[0184] Step 1203: Select a target transmission link corresponding to each time slot from multiple transmission links one by one based on the target sampling method and the link selection probability.

[0185] Steps 1201 to 1203 are described in detail below.

[0186] In some embodiments, after obtaining the link selection probability matrix of multiple transmission links in multiple time slots in the next preset scheduling time period, based on the link selection probability matrix corresponding to the multiple transmission links in each time slot, n , the probability of selecting the generated sampling method is (1-p1)(1-p2)...(1-p n ). This sampling method selects with probability (1-p1)(1-p2)...(1-p n ) is the probability of selecting Monte Carlo sampling to select the target transmission link, then the probability of selecting binary Bernoulli sampling to continue the target transmission link selection is 1-(1-p1)(1-p2)...(1-p n ).

[0187] In practical applications, a method can be used to select a random value between zero and one. When the random value selected by the method is less than the sampling method selection probability (1-p1)(1-p2)...(1-p n ), Monte Carlo sampling is used as the target sampling method; conversely, when it is greater than the sampling method selection probability (1-p1)(1-p2)...(1-p n ), binary Bernoulli sampling is used as the target sampling method.

[0188] Then, based on the target sampling method and the link selection probability, the target transmission link corresponding to each time slot is selected one by one from the multiple transmission links.

[0189] When the target sampling method is binary Bernoulli sampling, the online scheduling algorithm first reads the current action probability distribution vector to obtain the execution probability of each action, and then performs Bernoulli sampling on each action based on these probability values. The expression is shown in the following formula (11).

[0190] x i =p i k ×(1-p i ) 1-k (11)

[0191] Among them, p i is the execution probability of the action, x irepresents the sampled Boolean value of the ith action in the action set. If the sampling result is 1, k is the number of times the action has been selected. Then the agent executes the action. Assuming that the action set contains n actions, after Bernoulli sampling, the action execution result of the agent should be X = (x1, x2, ..., x n ). Bernoulli sampling allows the agent to i Execute action a i , with probability 1-p i Do not execute this action. Since each action is sampled separately and actions with higher probability are more likely to be executed, this sampling method not only retains the principle of selecting actions with higher expected rewards as much as possible, but also enables the agent to execute multiple actions in parallel, that is, to assign data packets to multiple links.

[0192] Since Bernoulli sampling has a certain probability of the sampling results being all 0, that is, the agent does not perform any action, which is obviously inconsistent with real logic. When this low-probability event occurs, the online scheduling algorithm uses the Monte Carlo sampling method instead, randomly generates a sequence number based on the current action probability distribution and executes the corresponding action in the action set. The following will further describe the selection steps when the target sampling method is the Monte Carlo method.

[0193] Reference Fig.13 , based on the target sampling method and the link selection probability, the target transmission link corresponding to each time slot is selected one by one from multiple transmission links, including the following steps 1301 to 1303.

[0194] Step 1301: Based on multiple link selection probabilities in each time slot, generate multiple sampling sub-intervals corresponding to transmission links and a sequence number identifier corresponding to each sampling sub-interval.

[0195] Step 1302: Generate a random sampling value, and select a sampling sub-interval whose sequence number identifier matches the random sampling value from multiple sampling sub-intervals as a target sampling sub-interval.

[0196] Step 1303: The transmission link corresponding to the target sampling sub-interval is used as the target transmission link.

[0197] Steps 1301 to 1303 are described in detail below.

[0198] In some embodiments, after determining to use the Monte Carlo method and obtaining the link selection probabilities of multiple transmission links, multiple sampling subintervals are generated according to these probabilities, each subinterval corresponds to a transmission link, and the serial number of each sampling subinterval is generated by the link selection probability of the transmission link. Next, a random sampling value is generated within a range of zero to one, and then based on the random sampling value, a sampling subinterval whose serial number matches the random sampling value is selected from the multiple sampling subintervals as the target sampling subinterval.

[0199] Take three transmission links {A1, A2, A3} as an example, where the link selection probability of A1 is 0.5, the link selection probability of A2 is 0.3, and the link selection probability of A3 is 0.2, then the sequence number corresponding to A1 is 0.5, the sequence number corresponding to A2 is 0.5+0.3=0.8, and the sequence number corresponding to A3 is 1. The random sampling value is 0.48, so A1 is used as the target transmission link for this time slot.

[0200] Through the above steps 1201 to 1203, and the above steps 1301 to 1303, the link selection probability generated according to the link state feature sequence in the heterogeneous transmission network that changes in real time is used to select the corresponding appropriate target sampling method, and the target sampling method is used to select the appropriate target transmission link corresponding to each time slot in the next preset scheduling time period from multiple transmission links based on the link selection probability, thereby improving the reliability of data packet transmission. And unlike the traditional reinforcement learning model, the present application outputs the selection probability, and then uses the selection probability to select the link, so that it has a certain degree of exploration to avoid falling into local optimality, and then can better further correct the link scheduling model in practical applications to improve the reliability of data packet transmission.

[0201] Reference Fig.14 , is a schematic diagram of the overall concept of a data packet transmission method for a heterogeneous transmission network provided by an embodiment of the present application. Fig.14 As shown in the figure, a reinforcement learning architecture is used in advance to collect link state feature sequences of heterogeneous transmission networks in actual applications through random sampling scheduling as training samples and input into an experience sequence collector, and a link scheduling model is trained on this basis (which includes updating the action probability distribution corresponding to different link state feature sequences), and when responding to a communication sender's request to send a data packet in the next scheduling time period, the link state feature sequence obtained in the previous scheduling time period is input into the link scheduling model to obtain a suitable action probability distribution decision, thereby selecting a suitable target transmission link for data packet transmission, thereby improving the reliability of data packet transmission.

[0202] The embodiments of the present application propose a data packet transmission method and related equipment for a heterogeneous transmission network. The heterogeneous transmission network includes multiple transmission links. The method includes: in advance, obtaining link state characteristic parameters corresponding to a link state characteristic sequence, obtaining link selection parameters of multiple transmission links, and obtaining initial policy network parameters of a policy network and initial value function network parameters of a value function network, and accumulating all throughput parameters to obtain an ideal aggregate throughput, obtaining a first throughput item based on the sum of the ideal aggregate throughput and the actual aggregate throughput, obtaining a second throughput item based on the difference between the ideal aggregate throughput and the actual aggregate throughput, and obtaining an aggregate throughput difference based on the ratio of the first throughput item to the second throughput item, performing logarithmic processing on the aggregate throughput difference to obtain an ideal aggregate throughput difference, generating a state space based on the link state characteristic parameters, generating an action space based on link selection transmission, generating a reward space based on the ideal aggregate throughput difference, and generating an initial link schedule based on the initial policy network parameters, the initial value function network parameters, the state space, the action space, and the reward space. model, and train the initial link scheduling model multiple times, and obtain the link scheduling model based on the trained initial link scheduling model; then, obtain the link state information of each transmission link in each time slot, splice the multiple link state information of each transmission link to obtain the link state information sequence, and perform attention processing on the link state information sequence to obtain the link state feature sequence; secondly, input all link state feature sequences into the link scheduling model for data processing to obtain the link selection probability of each transmission link in each time slot, splice the multiple link selection probabilities of each transmission link to obtain the link selection probability sequence, and splice the multiple link selection probability sequences to obtain the link selection probability matrix; finally, based on the link selection probabilities corresponding to the multiple transmission links in each time slot, generate the sampling method selection probability, select the target sampling method from the binary Bernoulli sampling and the Monte Carlo sampling based on the sampling method selection probability, and select the target transmission link corresponding to each time slot from the multiple transmission links one by one based on the target sampling method and the link selection probability.

[0203] The embodiment of the present application concatenates all link status information of each time slot of the heterogeneous transmission network obtained within the most recent preset scheduling time period to obtain a link status information sequence that characterizes the dynamic changes of each transmission link, and then uses a multi-attention mechanism to extract features from the link status information sequence to obtain a link status feature sequence that is more suitable for subsequent data processing, thereby improving the accuracy of selecting target transmission links in the heterogeneous transmission network; and, uses the sum of the throughput parameters of all transmission links in the time slot as a representation of the maximum aggregation throughput that can be achieved in theory for multi-path transmission, as well as the actual aggregation throughput of all transmission links as a whole in the time slot, to obtain an aggregation throughput difference that characterizes the difference between the actual and theoretical values, so that the aggregation throughput difference can be used to describe the aggregation performance in the heterogeneous transmission network, so as to facilitate the subsequent use of the aggregation throughput difference to construct a reward model for the initial link scheduling model, so that the training process of the initial link scheduling model is aimed at improving the aggregation performance, thereby improving The reliability of data packet transmission is improved; and, by using the idea of ​​reinforcement learning, an initial link scheduling model corresponding to the actual data packet transmission process in the heterogeneous transmission network is constructed, and the aggregate throughput difference that characterizes the aggregation performance of the heterogeneous transmission network is used as a training reward, so that the decision network and the value function network in the initial link scheduling model are trained using multiple training samples to improve the aggregation performance of the decisions obtained by the decision network, so that when the decision network in the trained initial link scheduling model is used as the actual link scheduling model for target transmission link selection, the aggregation performance corresponding to the decision can be improved, thereby improving the reliability of data packet transmission; in addition, the link selection probability generated according to the link state feature sequence in the real-time changing heterogeneous transmission network is used to select the corresponding appropriate target sampling method, and the target sampling method is used to select the appropriate target transmission link corresponding to each time slot in the next preset scheduling time period from multiple transmission links based on the link selection probability, thereby improving the reliability of data packet transmission.And unlike the traditional reinforcement learning model, the present application outputs the selection probability, and then uses the selection probability to select the link, so that it has a certain degree of exploratory nature to avoid falling into local optimality, and then the link scheduling model can be further corrected in practical applications to improve the reliability of data packet transmission; in short, the link state feature sequence of multiple transmission links that change in the heterogeneous transmission network and dynamically changes within the preset scheduling time period is used to characterize the real-time transmission capacity of the current multiple transmission links, and combined with the link scheduling model pre-constructed based on the data packet transmission process in the heterogeneous transmission network, so as to obtain the link selection probability matrix of multiple transmission links corresponding to the link state feature sequence that are selected for data packet transmission in the next preset scheduling time period, so as to improve the reliability of data packet transmission, and select at least one target transmission link from multiple transmission links for data packet transmission in the next preset scheduling time period according to the link selection probability matrix, so as to avoid falling into local optimality and further improve the successful reliability of data packet transmission in the heterogeneous transmission network.

[0204] The embodiment of the present application also provides a data packet transmission device for a heterogeneous transmission network, which can implement the data packet transmission method of the heterogeneous transmission network, referring to Fig.15 , the device 1500 comprises:

[0205] The state acquisition module 1510 is used to obtain a link state feature sequence of each transmission link within a preset scheduling time period;

[0206] A data processing module 1520 is used to input all link state feature sequences into a link scheduling model for data processing to obtain a link selection probability matrix of multiple transmission links;

[0207] The target transmission link determination module 1530 is used to determine at least one target transmission link within the next preset scheduling time period from multiple transmission links based on the link selection probability matrix, and use the target transmission link to transmit data packets.

[0208] In some embodiments, the status acquisition module 1510 is further configured to:

[0209] Obtain link status information of each transmission link in each time slot;

[0210] splicing multiple link state information of each transmission link to obtain a link state information sequence;

[0211] The link state information sequence is subjected to attention processing to obtain a link state feature sequence.

[0212] In some embodiments, the data processing module 1520 is further configured to:

[0213] All link state feature sequences are input into the link scheduling model for data processing to obtain the link selection probability of each transmission link in each time slot;

[0214] Concatenate multiple link selection probabilities of each transmission link to obtain a link selection probability sequence;

[0215] Multiple link selection probability sequences are concatenated to obtain a link selection probability matrix.

[0216] In some embodiments, the target transmission link determination module 1530 is further configured to:

[0217] Generate a sampling method selection probability based on link selection probabilities corresponding to multiple transmission links in each time slot;

[0218] Select the target sampling method from binary Bernoulli sampling and Monte Carlo sampling based on the sampling method selection probability;

[0219] A target transmission link corresponding to each time slot is selected one by one from multiple transmission links based on a target sampling method and a link selection probability.

[0220] In some embodiments, the target transmission link determination module 1530 is further configured to:

[0221] Based on multiple link selection probabilities in each time slot, generate multiple sampling subintervals corresponding to the transmission links and a sequence number identifier corresponding to each sampling subinterval;

[0222] Generate a random sampling value, and select a sampling sub-interval whose serial number identifier matches the random sampling value from multiple sampling sub-intervals as a target sampling sub-interval;

[0223] The transmission link corresponding to the target sampling subinterval is used as the target transmission link.

[0224] In some embodiments, the data processing module 1520 is further configured to:

[0225] Acquire link state characteristic parameters corresponding to the link state characteristic sequence, acquire link selection parameters of multiple transmission links, and acquire initial policy network parameters of the policy network and initial value function network parameters of the value function network, and generate an ideal difference of aggregate throughput based on the link state characteristic parameters;

[0226] Generate a state space based on link state characteristic parameters;

[0227] Generate action space based on link selection transmission;

[0228] Generate a reward space based on the ideal difference in aggregated throughput;

[0229] An initial link scheduling model is generated based on initial strategy network parameters, initial value function network parameters, state space, action space and reward space, and the initial link scheduling model is trained multiple times, and a link scheduling model is obtained based on the trained initial link scheduling model.

[0230] In some embodiments, the data processing module 1520 is further configured to:

[0231] The ideal aggregate throughput is obtained by summing up all throughput parameters;

[0232] Obtaining a first throughput item based on a sum of an ideal aggregate throughput and an actual aggregate throughput, obtaining a second throughput item based on a difference between the ideal aggregate throughput and the actual aggregate throughput, and obtaining an aggregate throughput difference based on a ratio of the first throughput item to the second throughput item;

[0233] The aggregate throughput difference is logarithmically processed to obtain an ideal aggregate throughput difference. In some embodiments, the data processing module 1520 is further configured to:

[0234] Obtain training state sequence, training action sequence and training reward sequence;

[0235] Based on the training state sequence, the current strategy state value, the subsequent strategy state value, the current value function state value and the subsequent value function state value are obtained;

[0236] Get future policy rewards based on the training reward sequence, the current policy state value, and the subsequent policy state value;

[0237] Perform gradient descent updates on the initial policy network parameters based on future policy rewards;

[0238] The value function update state value is obtained based on the subsequent value function state value and the training reward sequence, the value function error is obtained based on the difference between the current value function state value and the value function update state value, and the initial value function network parameters are updated by gradient descent based on the value function error.

[0239] In the above embodiments, the description of each embodiment has its own emphasis. For the part that is not described in detail in a certain embodiment, the specific implementation method of the data packet transmission device of the heterogeneous transmission network is basically the same as the specific implementation method of the data packet transmission method of the above-mentioned heterogeneous transmission network, and will not be repeated here.

[0240] In an embodiment of the present application, a data packet transmission device of a heterogeneous transmission network splices all link state information of each time slot of the heterogeneous transmission network obtained within the most recent preset scheduling time period to obtain a link state information sequence that characterizes the dynamic changes of each transmission link, and then uses a multi-attention mechanism to extract features of the link state information sequence to obtain a link state feature sequence that is more suitable for subsequent data processing, thereby improving the accuracy of selecting target transmission links in the heterogeneous transmission network; and, using the sum of the throughput parameters of all transmission links in the time slot as a representation of the maximum aggregation throughput that can be achieved in theory by multi-path transmission, as well as the actual aggregation throughput of all transmission links as a whole in the time slot, to obtain an aggregation throughput difference used to characterize the difference between the actual and theoretical values, so that the aggregation throughput difference can be used to describe the aggregation performance in the heterogeneous transmission network, so as to facilitate the subsequent use of the aggregation throughput difference to construct a reward model for the initial link scheduling model, so that the aggregation performance can be improved during the training process of the initial link scheduling model. For the purpose, the reliability of data packet transmission is improved; and, by using the idea of ​​reinforcement learning, an initial link scheduling model corresponding to the actual data packet transmission process in the heterogeneous transmission network is constructed, and the aggregate throughput difference that characterizes the aggregation performance of the heterogeneous transmission network is used as a training reward, so that the decision network and the value function network in the initial link scheduling model are trained by using multiple training samples to improve the aggregation performance of the decision obtained by the decision network, so that when the decision network in the trained initial link scheduling model is used as the actual link scheduling model for target transmission link selection, the aggregation performance corresponding to the decision can be improved, thereby improving the reliability of data packet transmission; in addition, the link selection probability generated according to the link state feature sequence in the heterogeneous transmission network that changes in real time is used to select the corresponding appropriate target sampling method, and the target sampling method is used to select the appropriate target transmission link corresponding to each time slot in the next preset scheduling time period from multiple transmission links based on the link selection probability, thereby improving the reliability of data packet transmission.And unlike the traditional reinforcement learning model, the present application outputs the selection probability, and then uses the selection probability to select the link, so that it has a certain degree of exploratory nature to avoid falling into local optimality, and then the link scheduling model can be further corrected in practical applications to improve the reliability of data packet transmission; in short, the link state feature sequence of multiple transmission links that change in the heterogeneous transmission network and dynamically changes within the preset scheduling time period is used to characterize the real-time transmission capacity of the current multiple transmission links, and combined with the link scheduling model pre-constructed based on the data packet transmission process in the heterogeneous transmission network, so as to obtain the link selection probability matrix of multiple transmission links corresponding to the link state feature sequence that are selected for data packet transmission in the next preset scheduling time period, so as to improve the reliability of data packet transmission, and select at least one target transmission link from multiple transmission links for data packet transmission in the next preset scheduling time period according to the link selection probability matrix, so as to avoid falling into local optimality and further improve the successful reliability of data packet transmission in the heterogeneous transmission network.

[0241] The present application also provides an electronic device, including:

[0242] at least one memory;

[0243] at least one processor;

[0244] at least one program;

[0245] The program is stored in the memory, and the processor executes the at least one program to implement the data packet transmission method of the heterogeneous transmission network implemented in the present application. The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0246] See also Fig.16 , Fig.16 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0247] The processor 1601 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0248] The memory 1602 can be implemented in the form of ROM (Read Only Memory), static storage device, dynamic storage device or RAM (Random Access Memory). The memory 1602 can store operating systems and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1602, and the processor 1601 calls and executes the data packet transmission method for the heterogeneous transmission network of the embodiments of this application;

[0249] Input / output interface 1603, used to implement information input and output;

[0250] The communication interface 1604 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0251] A bus 1605 that transmits information between various components of the device (e.g., the processor 1601, the memory 1602, the input / output interface 1603, and the communication interface 1604);

[0252] The processor 1601 , the memory 1602 , the input / output interface 1603 and the communication interface 1604 are connected to each other in communication within the device via the bus 1605 .

[0253] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned data packet transmission method of the heterogeneous transmission network is implemented.

[0254] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0255] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0256] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0257] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0258] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0259] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0260] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0261] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0262] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0263] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0264] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0265] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for transmitting data packets in a heterogeneous transmission network, characterized in that: The heterogeneous transmission network includes multiple transmission links, and the method includes: Acquire a link state characteristic sequence of each transmission link within a preset scheduling time period; Inputting all the link state feature sequences into a link scheduling model for data processing to obtain a link selection probability matrix of multiple transmission links; At least one target transmission link within the next preset scheduling time period is determined from the plurality of transmission links based on the link selection probability matrix, and the target transmission link is used to transmit data packets.

2. The method for transmitting data packets in a heterogeneous transmission network according to claim 1, characterized in that: The preset scheduling time period includes a plurality of time slots, and the acquiring of a link state feature sequence of each transmission link within the preset scheduling time period includes: Obtaining link status information of each transmission link in each time slot; splicing the plurality of link state information of each transmission link to obtain a link state information sequence; The link state information sequence is subjected to attention processing to obtain the link state feature sequence.

3. The method for transmitting data packets in a heterogeneous transmission network according to claim 1, characterized in that: The step of inputting all the link state feature sequences into the link scheduling model for data processing to obtain a link selection probability matrix of multiple transmission links includes: Inputting all the link state feature sequences into a link scheduling model for data processing to obtain a link selection probability of each transmission link in each time slot; splicing the multiple link selection probabilities of each transmission link to obtain the link selection probability sequence; A plurality of the link selection probability sequences are concatenated to obtain the link selection probability matrix.

4. The method for transmitting data packets in a heterogeneous transmission network according to claim 3, characterized in that: The preset scheduling time period includes a plurality of time slots, and determining at least one target transmission link within the next preset scheduling time period from the plurality of transmission links based on the link selection probability matrix includes: Generate a sampling method selection probability based on the link selection probabilities corresponding to the plurality of transmission links in each time slot; Selecting a target sampling method from binary Bernoulli sampling and Monte Carlo sampling based on the sampling method selection probability; The target transmission link corresponding to each time slot is selected one by one from the plurality of transmission links based on the target sampling method and the link selection probability.

5. The method for transmitting data packets in a heterogeneous transmission network according to claim 4, characterized in that: When the target sampling method is Monte Carlo sampling, selecting a target transmission link corresponding to each time slot from the plurality of transmission links one by one based on the target sampling method and the link selection probability includes: Based on multiple link selection probabilities in each time slot, generate multiple sampling subintervals corresponding to the transmission links and a sequence number identifier corresponding to each sampling subinterval; Generate a random sampling value, and select the sampling sub-interval whose sequence number identifier matches the random sampling value from the multiple sampling sub-intervals as the target sampling sub-interval; The transmission link corresponding to the target sampling subinterval is used as a target transmission link.

6. The method for transmitting data packets in a heterogeneous transmission network according to claim 1, characterized in that: The generation process of the link scheduling model includes: Acquire link state characteristic parameters corresponding to the link state characteristic sequence, acquire link selection parameters of the plurality of transmission links, and acquire initial policy network parameters of the policy network and initial value function network parameters of the value function network, and generate an ideal difference of aggregate throughput based on the link state characteristic parameters; Generate a state space based on the link state characteristic parameters; Selecting transmission based on the link generates an action space; generating a reward space based on the aggregate throughput ideal difference; An initial link scheduling model is generated based on the initial strategy network parameters, the initial value function network parameters, the state space, the action space, and the reward space, and the initial link scheduling model is trained multiple times, and the link scheduling model is obtained based on the trained initial link scheduling model.

7. The method for transmitting data packets in a heterogeneous transmission network according to claim 6, characterized in that: The link state characteristic parameters include a throughput parameter of each of the transmission links and an actual aggregate throughput of the heterogeneous transmission network, and generating an ideal difference of aggregate throughput based on the link state characteristic parameters includes: Accumulating all the throughput parameters to obtain an ideal aggregate throughput; obtaining a first throughput item based on a sum of the ideal aggregate throughput and the actual aggregate throughput, obtaining a second throughput item based on a difference between the ideal aggregate throughput and the actual aggregate throughput, and obtaining an aggregate throughput difference based on a ratio of the first throughput item to the second throughput item; The aggregate throughput difference is logarithmically processed to obtain the ideal aggregate throughput difference.

8. The method for transmitting data packets in a heterogeneous transmission network according to claim 6, characterized in that: The training process of the initial link scheduling model includes: Obtain training state sequence, training action sequence and training reward sequence; Based on the training state sequence, a current strategy state value, a subsequent strategy state value, a current value function state value, and a subsequent value function state value are obtained; Obtaining a future policy reward based on the training reward sequence, the current policy state value, and the subsequent policy state value; Performing gradient descent update on the initial policy network parameters based on the future policy reward; A value function update state value is obtained based on the subsequent value function state value and the training reward sequence, a value function error is obtained based on the difference between the current value function state value and the value function update state value, and the initial value function network parameters are updated by gradient descent based on the value function error.

9. A data packet transmission device for a heterogeneous transmission network, characterized in that: The heterogeneous transmission network includes multiple transmission links, and the device includes: A state acquisition module, used to acquire a link state feature sequence of each transmission link within a preset scheduling time period; A data processing module, used for inputting all the link state feature sequences into a link scheduling model for data processing to obtain a link selection probability matrix of multiple transmission links; The target transmission link determination module is used to determine at least one target transmission link within the next preset scheduling time period from the multiple transmission links based on the link selection probability matrix, and use the target transmission link to transmit data packets.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method for transmitting data packets in a heterogeneous transmission network according to any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for transmitting data packets in a heterogeneous transmission network according to any one of claims 1 to 8 is implemented.

Images