Decentralized multi-communication link data transmission method based on deep reinforcement learning

Abstract

The invention relates to a decentralized multi-communication link data transmission method based on deep reinforcement learning. According to the system, the electronic equipment and the storage medium, various message communication modes applied to local, local area networks or cross-local area networks are realized through shared memory, P2P and MQTT, selection of message transmission channels is optimized through deep reinforcement learning, and data transmission is completed as soon as possible while load balancing is considered; the reliability of message communication is improved, the delay of message communication is reduced, the communication cost between applications is greatly reduced, and the communication efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data transmission, and in particular to a decentralized multi-communication link data transmission method, system, computer device and computer-readable storage medium based on deep reinforcement learning. Background Art

[0002] Data transmission refers to the process of sending and receiving data between different devices or systems. This process can be carried out in many ways and involves a variety of protocols and technologies.

[0003] In related technologies, if only a single communication link is relied upon, when the communication link fails or is interrupted, the entire data transmission will be affected, resulting in reduced reliability; at the same time, a single communication link is prone to congestion under high load conditions, resulting in increased latency and affecting data transmission speed; in addition, the bandwidth and processing power of a single communication link are also relatively limited, and cannot effectively meet the data communication needs of computing-intensive tasks.

[0004] In order to solve the above problems of single communication link transmission technology, multi-communication link technology is proposed. This technology uses multiple communication links for parallel transmission, and uses load balancing and redundancy technology to improve transmission performance and reliability; however, load balancing between different communication links may not be effective enough, resulting in some communication links being overloaded while other communication links are idle. In addition, the bandwidth and delay of different communication links may be different, resulting in uneven overall bandwidth utilization, affecting communication efficiency. Summary of the invention

[0005] The embodiments of the present application provide a decentralized multi-communication link data transmission method, system, computer device and computer-readable storage medium based on deep reinforcement learning, so as to at least solve the problem of uneven overall bandwidth utilization of multi-link data transmission in the related art.

[0006] In a first aspect, an embodiment of the present application provides a decentralized multi-communication link data transmission method based on deep reinforcement learning, characterized in that it includes:

[0007] Respond to the message sending request of the sending program and upload the message content to the target topic;

[0008] Acquire a receiving end program that subscribes to the target topic, and acquire a communication link supported by the receiving end program;

[0009] By selecting and scheduling a multi-communication link based on deep reinforcement learning, a scheduling strategy is dynamically output based on the network status and queue status of each communication link, wherein the scheduling strategy includes selecting a target communication link for transmitting the message content;

[0010] According to the scheduling strategy of the dynamic output, the message content is forwarded to the receiving end program in real time.

[0011] In some embodiments, the communication link includes a first communication link, a second communication link, and a third communication link, wherein:

[0012] The first communication link is an IPC channel constructed based on local shared memory when the receiving end program and the sending end program are located on the same device;

[0013] The second communication link is a Local channel constructed based on a P2P communication protocol in the local area network when the receiving end program and the sending end program are located in the same local area network;

[0014] The third communication link is a Remote channel constructed based on the MQTT protocol when the receiving end program and the sending end program are located in different network segments.

[0015] In some embodiments, by selecting a scheduling model through multiple communication links, based on the network status and queue status of each communication link, the dynamic output scheduling strategy includes:

[0016] In the initial state, according to the total amount of data of the message content, and the network state and queue state of each communication link, the minimum time required for each communication link to complete the transmission of the message content is obtained;

[0017] In a unit time window, the observation data of each communication link is obtained respectively, and a linear fitting is performed based on the observation data of each communication link by the least square method to obtain the delay variation trend of each communication link and the influence parameter of the delay variation trend on the subsequent transmission of data;

[0018] Through the multi-communication link selection scheduling model pre-trained based on the D3QN algorithm, the scheduling strategy at each time step is dynamically output based on the total amount of data in the message content, the minimum time consumption in the initial state, the delay change trend and the influencing parameters.

[0019] In some embodiments, according to the total amount of data of the message content, and the network status and queue status of each communication link, obtaining the minimum time required for each communication link to complete the transmission of the message content includes:

[0020] In the initial state, the total amount of data of the message content, as well as the established transmission rate of each communication link, the average delay and the amount of data to be processed in the queue of the channel are obtained;

[0021] According to the total amount of data and the established transmission rate of each communication link, the average delay and the amount of data to be processed in the queue, the transmission time required for each communication link to complete the task of forwarding the message content is obtained respectively;

[0022] The minimum transmission time required for each communication link to complete the task of forwarding the message content is obtained as the minimum time consumption.

[0023] In some embodiments, a linear fitting is performed based on the observed data of each communication link by the least square method to obtain the delay variation trend of each communication link, and the influence parameters of the delay variation trend on the subsequent transmission of data include:

[0024] In a unit time window, observation data of each communication link is obtained respectively, wherein the observation data includes the time when any data point is generated and the minimum time required for the data point to transmit a unit amount of data in its communication link;

[0025] According to the multiple observation points of each communication link, a least square method is used to perform linear fitting to obtain the delay change trend of each communication link in a future period;

[0026] According to the total amount of data and the delay variation trend of each communication link, the influencing parameter corresponding to each communication link is obtained.

[0027] In some embodiments, the multi-communication link selection scheduling model uses the delay variation trend of each communication link as state information, the degree of use of each communication link as transmission action information, and is trained with the goal of maximizing the reward function at each time step, wherein:

[0028] The reward function is expressed by the following formula:

[0029] R=-(X ipc I ipc +X local I local +X remote I remote )

[0030] Where R represents the reward function, x ipc 、x local 、x remote I respectively represent the transmission action information of the first communication link, the second communication link and the third communication link, ipc ,I local ,I remote Respectively represent the influencing parameters of the first communication link, the second communication link and the third communication link.

[0031] In some of the embodiments, the multi-communication link selection scheduling model includes an input layer, a hidden layer, and an output layer, wherein;

[0032] The state information is received through the input layer, and a multi-layer nonlinear transformation is performed on the state information through a plurality of hidden layers to obtain a high-level feature identifier;

[0033] Based on the high-level feature representation, the output layer dynamically outputs the action information of each communication link at the current time step under the constraint of the reward function.

[0034] In a second aspect, an embodiment of the present application provides a decentralized multi-communication link data transmission system based on deep reinforcement learning, characterized in that the system includes: an upload module, a scheduling module and a forwarding module, wherein:

[0035] The upload module is used to respond to the message sending request of the sending end program, upload the message content to the target topic, obtain the receiving end program subscribed to the target topic, and obtain the communication link supported by the receiving end program;

[0036] The scheduling module is used to dynamically output a scheduling strategy based on the network status and queue status of each communication link by selecting a scheduling model for multiple communication links obtained in advance based on deep reinforcement learning, wherein the scheduling strategy includes selecting a target communication link for transmitting the message content and transmitting action information of each target communication link;

[0037] The forwarding module is used to forward the message content to the receiving end program in real time according to the scheduling strategy of the dynamic output.

[0038] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect above when executing the computer program.

[0039] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the second aspect above.

[0040] Compared with the related technologies, the decentralized multi-communication link data transmission method based on deep reinforcement learning provided in the embodiment of the present application realizes various message communication modes between applications locally, in a local area network, or across local area networks through shared memory, P2P, and MQTT. At the same time, the selection of message transmission channels is optimized through deep reinforcement learning, and data transmission is completed as quickly as possible while considering load balancing; the reliability of message communication is increased, the message communication delay is reduced, the communication cost between applications is greatly reduced, and the communication efficiency is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0042] Figure 1 A decentralized multi-communication link data transmission method based on deep reinforcement learning according to an embodiment of the present application;

[0043] Figure 2 It is a schematic diagram of establishing a communication link between a sending end and a receiving end under different network environments according to an embodiment of the present application;

[0044] Figure 3 is a flow chart for making multi-channel scheduling decisions according to an embodiment of the present application;

[0045] Figure 4 is a flowchart of another decentralized multi-communication link data transmission method based on deep reinforcement learning according to an embodiment of the present application;

[0046] Figure 5 It is a schematic diagram of the overall interaction process between the sending end and the receiving end under different network environments according to an embodiment of the present application;

[0047] Figure 6 It is a structural block diagram of a decentralized multi-communication link data transmission system based on deep reinforcement learning according to an embodiment of the present application;

[0048] Figure 7 It is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0050] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0051] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0052] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0053] Faced with the growing number of computationally intensive tasks, edge computing is an emerging computing paradigm that moves computing resources and data storage functions to where data is generated and used to reduce data transmission delays and network congestion. In an edge computing environment, devices need to collaborate with each other, devices need to communicate with the center, and even the various processes within the device need to exchange data to meet the needs of existing edge computing and achieve more efficient computing and data processing. In order to ensure that a large amount of data in edge devices is processed efficiently and at the same time ensure the connection between edge devices and the center, edge device communication can meet different scenario requirements through different transmission methods. Among them, the publish-subscribe messaging paradigm decouples data producers and consumers in time and space, and has been widely used and verified in massive IoT devices for low-latency and high-reliability communication between devices.

[0054] In order to solve the problem of low reliability caused by the failure of a single data communication link and high latency in data communication of a large number of computing-intensive tasks, and at the same time to solve the problems existing in traditional multi-communication link transmission, the present application provides a decentralized intelligent multi-communication link data transmission method for multi-communication link, multi-channel, low-latency and high-reliability communication between edge devices and control centers, other local area network devices or other processes on the local machine.

[0055] Figure 1 A decentralized multi-communication link data transmission method based on deep reinforcement learning according to an embodiment of the present application, Figure 1 is a flowchart of a decentralized multi-communication link data transmission method for deep reinforcement learning according to an embodiment of the present application, such as Figure 1 As shown, the process includes the following steps:

[0056] S101, responding to a message sending request from a sending end program, uploading the message content to a target topic; obtaining a receiving end program subscribed to the target topic, and obtaining a communication link supported by the receiving end program;

[0057] Among them, the sending end program can be any process in any edge side device. In this embodiment, in the edge computing environment, the same local area network or different network segment devices need to collaborate, the device and the center need to communicate, and even the various processes in the device need to exchange data to meet the various complex requirements in the existing edge computing scenarios and achieve more efficient computing and data processing.

[0058] Furthermore, in order to ensure that large amounts of data in edge devices are processed efficiently while also ensuring the connection between edge devices and the center, the present application solution decouples senders and receivers based on a publish-subscribe messaging paradigm, which can be quickly applied to various devices to achieve low-cost message communication, thereby satisfying the edge devices' different scenario requirements through different transmission methods.

[0059] It should be noted that, through the Pub / Sub model, data producers and consumers no longer need to communicate directly, and producers do not care about the identity and specific status of the recipients, but interact indirectly through message brokers. In this way, producers only need to focus on the publication of data, while consumers focus on the subscription of data they are interested in, which greatly simplifies the system architecture; in this embodiment, the sender and the receiver are decoupled by using the publish-subscribe-based message paradigm, which can be quickly applied to various devices to achieve low-cost message communication, improving the maintainability and scalability of the system.

[0060] In the solution provided by this embodiment, after receiving a request to send a message, the message content to be transmitted is uploaded to a target topic (Topic); a subscription list needs to be maintained in the system to record which receiving end programs have subscribed to the topic, wherein different receiving end programs may be in different network environments from the sending end program and support different communication protocols and communication links; in this step, after determining the receiving end program that has subscribed to the target topic, the communication chain supported by each receiving end program can be obtained through a query or registration mechanism.

[0061] Specifically, Figure 2 Schematic diagram of establishing a communication link between a sending end and a receiving end under different network environments according to an embodiment of the present application, such as Figure 2 As shown, the communication link includes a first communication link, a second communication link and a third communication link, wherein: the first communication link is an IPC channel constructed based on local shared memory when the receiving program and the sending program are located in the same device; the second communication link is a Local channel constructed based on the P2P communication protocol in the local area network when the receiving program and the sending program are located in the same local area network; the third communication link is a Remote channel constructed based on the MQTT protocol when the receiving program and the sending program are located in different network segments.

[0062] S102, dynamically outputting a scheduling strategy based on the network status and queue status of each communication link through a multi-communication link selection scheduling model obtained in advance based on deep reinforcement learning, wherein the scheduling strategy includes selecting a target communication link for transmitting message content;

[0063] Among them, Deep Reinforcement Learning (DRL) is an artificial intelligence technology that combines deep learning and reinforcement learning, enabling computer systems to learn how to make the best decisions by interacting with the environment; its core idea is to use deep neural networks to represent state value functions (Value Function) and policies (Policy), thereby achieving autonomous learning and optimization.

[0064] Correspondingly, in the solution of this application, under the architecture based on the publish / subscribe message paradigm, the goal corresponding to the optimal strategy is to balance the load of each channel while minimizing the transmission time of the message content;

[0065] Among them, in order to avoid waste of resources and avoid data loss and increased delay caused by channel overload, in the present application scheme, it is necessary to additionally consider how to balance the pressure of each channel when determining the goal. Therefore, the goal based on deep reinforcement learning can be set to effectively balance the pressure of each channel while minimizing the transmission time of message content. Specifically: setting a reward function guided by this goal, training a multi-communication link selection scheduling model includes the following specific steps:

[0066] Step 1: According to the total amount of data in the message content, and the network status and queue status of each communication link, the minimum time required for each communication link to complete the transmission of the message content is obtained;

[0067] Specifically, this step includes the following specific steps:

[0068] Step 1.1, obtain the total amount of data in the message content, as well as the established transmission rate, average delay and amount of data to be processed in the channel transmission queue of each communication link;

[0069] Among them, the current data sender needs to transmit data amount N bits, assuming that the current local IPC channel based on shared memory transmission rate is S ipc , unit: bits / s; the transmission rate of the P2P-based Local channel in the LAN is S local , unit: bits / s; the transmission rate of the cross-segment MQTT-based Remote channel is S remote , unit: bits / s; in addition, the data preparation time is short and can be ignored.

[0070] In this embodiment, optionally, the communication link communication delay adopts the median value filtering method to calculate the average delay of each transmission channel in the 1S time window. If the communication link is unreachable, the value will be expressed as INF (infinity);

[0071] Among them, the IPC channel is T ipc , LAN channel is T local, Remote channel T remote , unit s. The length of data currently to be processed in each channel transmission queue, where L is the length of the IPC channel ipc , LAN channel is L local , Remote channel L remote , unit: bits;

[0072] Step 1.2, according to the total amount of data and the transmission rate of each communication link, the average delay and the amount of data to be processed in the channel transmission queue, the transmission time required for each communication link to complete the forwarding task of the message content is obtained respectively;

[0073] Specifically, when the receiving program is another program on the same device as the sending program, the time it takes to transmit to other programs on the local machine through the IPC queue can be expressed as:

[0074]

[0075] When the receiving program is other programs in the LAN, the time required to transmit to other programs in the LAN through the Local queue can be expressed as:

[0076]

[0077] When the receiving program is other programs on the Internet, the time required to transmit to other programs on the Internet through the Remote queue can be expressed as:

[0078]

[0079] Step 1.3, obtain the minimum transmission time required for each communication link to complete the message content forwarding task as the minimum time consumption.

[0080] Specifically: The minimum time required to send the data can be expressed as:

[0081] T min =min(t ipc , t local , t remote )

[0082] Step 2: Obtain the observed data of each communication link in the unit time window, perform linear fitting based on the observed data of each communication link by the least square method, and obtain the delay change trend of each communication link and the influence parameter of the delay change trend on the subsequent transmission of data;

[0083] Figure 3 is a flowchart of making a multi-channel scheduling decision according to an embodiment of the present application, such as Figure 3As shown, before calculating the above test trends and influencing parameters, it is necessary to first determine whether the current data volume can meet the data volume requirements of the trend calculation. If so, the test trend and influencing parameters are calculated based on the current network status and input into the multi-communication link optimization selection model for overall channel scheduling. If not, the solution of sending all three channels is output;

[0084] Specifically, this embodiment uses the least square method to perform linear fitting on the group of data based on multiple data points in a unit time window to calculate the delay change trend of each channel, which specifically includes the following steps:

[0085] Step 1, in a unit time window, respectively obtain the observation data of each communication link, where the observation point includes the time when the target data node is generated and the minimum time required for the target data node to transmit a unit amount of data in the communication link;

[0086] Taking the IPC channel as an example, suppose there are n observation data points P in the unit time window of the IPC transmission channel. ipc (t i , T min,i ), multiple observation data points constitute the above observation data; among them, t i Indicates the time when data i is generated, T min，i Indicates the minimum time required to complete the unit data volume of data i in this channel T min ;

[0087] Step 2: According to the observation points of each communication link, the least square method is used to perform linear fitting to obtain the delay change trend of each communication link in the future period;

[0088] In this embodiment, a linear model is obtained by fitting using a plurality of observed data points to represent how the data transmission delay changes with time on the channel.

[0089] For the IPC channel, the linear fitting equation is:

[0090] T=a ipc +k ipcl t

[0091] Among them, k ipc Indicates the trend (slope) of the delay over time, a ipc is a constant term, corresponding to the fixed part of the delay.

[0092] Specifically: Where t is the observation time, T is the observed completion time, and are the average values ​​of t and T respectively. Further,

[0093] In addition, based on the same method, the same calculation can be performed on other channels (such as Local channel and Remote channel) to obtain the corresponding k local , a local With k remote , a remote .

[0094] It can be understood that the slope k calculated by the least square method can be used to predict whether the delay of a certain channel will increase or decrease in the future.

[0095] When k>0, it means that the delay is increasing over time and the channel may gradually become more congested;

[0096] When k<0, it means that the delay is decreasing and the channel becomes smoother.

[0097] This trend information can provide an important reference for the intelligent scheduling algorithm, enabling the system to select the optimal channel for data transmission based on the predicted delay changes.

[0098] In addition, after determining the delay change trend, since the goal of intelligent scheduling is to send the above message data in the shortest possible time, the impact of the delay change trend on the subsequent sending of data must also be considered;

[0099] Step 3, according to the total amount of data and the delay change trend of each communication link, the corresponding influencing parameters of each communication link are obtained;

[0100] In an exemplary embodiment, when the total amount of data of the message content to be sent is N bits of data, the impact of the data in each communication link can be evaluated by the following formula:

[0101] I ipc =αNa ipc +βk ipc

[0102] Where N is the amount of data sent (in bits), a ipc is the constant term during IPC channel transmission (corresponding to the fixed part of the delay), k ipc is the delay change trend of the IPC channel (the part that changes with time), α and β are adjustable weight parameters, which are used to balance the influence of the fixed delay and the delay change trend. It can be adjusted according to the different network environments of each channel. Since how to adjust it has no effect on the core invention of this application, it will not be repeated in this embodiment.

[0103] Similarly, according to the above calculation logic, the influence parameters of the other two channels can be calculated:

[0104] I local =αNa local +βk local

[0105] I remote =αNa remote +βk remote

[0106] Step 3, through the multi-communication link selection scheduling model pre-trained based on the D3QN algorithm, the communication link selection is dynamically optimized based on the total data volume of the message content, the delay change trend and the influencing parameters, and the communication link currently used for data transmission and the action of each communication link are determined in real time.

[0107] In this embodiment, the D3QN deep reinforcement learning algorithm is selected, where D3QN stands for Dueling Double DeepQ-Network, which is an evolved form of a deep reinforcement learning algorithm. It combines the advantages of Dueling DQN and Double DQN, and uses Double Q-Learning to reduce the problem of overestimation. The model trained by D3QN will be used for channel scheduling decisions of data, which can provide more efficient and stable strategy learning capabilities.

[0108] Specifically, the deep reinforcement learning-based algorithm D3QN (Double Deep Q-Network) is used for the optimization selection problem of multi-channel data transmission, with the goal of minimizing latency while balancing the burden of each communication link.

[0109] First, define the state of the system as {A, K}, where: A = {a ipc , a local , a remote}, represents the fixed delay factor K of each channel = {k ipc , k local , k remote}, represents the delay trend of each channel, reflecting the change of delay over time. The definition of state is mainly to describe the communication link delay and delay change trend of the system at a certain moment;

[0110] Secondly, define the system action Action as {x ipc , x local , x remote}, x∈[0, 1], where 0 means not to use the channel for transmission and 1 means to select the channel for transmission, which means that at each decision step, the system will select one or more channels to perform data transmission at the same time.

[0111] Among them, the state is used to describe the environmental conditions of the system at a certain moment. In this scheme, the state S = {A, K} includes the fixed delay factor A of the channel and the delay change trend K, which reflects the transmission conditions of each channel at present and in the future, such as network delay, delay trend, etc.

[0112] State information is the basis for strategy selection, which determines how the system estimates the quality of an action under current conditions. The algorithm model will select the appropriate action based on the current state.

[0113] Furthermore, an action is a decision that the system can take. In this scenario, the action {x ipc , x local , x remote} is whether to perform data transmission on each channel. 0 means not selecting the channel, 1 means selecting the channel. Each action corresponds to selecting a different transmission channel for data transmission.

[0114] It can be understood that by adopting the above multi-communication link decision selection model, the system will select one or more actions according to the current state, that is, select the most suitable transmission channel at the moment and the action degree of these transmission channels. The reinforcement learning algorithm can learn the optimal action selection by selecting different actions, observing the impact of the actions on the state, and adjusting the strategy according to the feedback.

[0115] Furthermore, the reward function is used to measure the effect of an action. In this scheme, the goal of the reward function R is to minimize the transmission delay while balancing the load of each channel.

[0116] The system will calculate the corresponding delay for each action result and feed it back to the algorithm in the form of a negative value (because the smaller the delay, the better). The formula is:

[0117] R=-(X ipc I ipc +X local I local +X remote I remote )

[0118] Where R represents the reward function, x ipc 、x local 、x remote I respectively represent the transmission action information of the first communication link, the second communication link and the third communication link, ipc ,I local ,I remote Respectively represent the influencing parameters of the first communication link, the second communication link and the third communication link.

[0119] The feedback of the reward function directly affects the adjustment of the strategy. If the system selects a channel with lower latency, it will receive a higher reward, otherwise it will receive a lower reward. Through repeated attempts, the system will gradually learn to select channels that can obtain high rewards (i.e., low latency and load balance). The corresponding transmission action information under this condition is the optimized scheduling selection strategy output by the model.

[0120] In the process of training the above-mentioned multi-communication link selection, the state provides the delay and trend information of the current network, which is the basis for action selection; the action is the decision made by the system in a certain state, which channel is selected for transmission; the reward function is used to measure the closeness of the action to the expected goal, and feedback is given to the system based on the results caused by the action (such as delay); during the training process, through the state-action-reward cycle process, the action selection strategy is continuously learned and optimized, and finally a path selection and configuration scheme that can follow the state changes and minimize the delay is found.

[0121] In an exemplary embodiment, the multi-communication link selection scheduling model includes an input layer, multiple hidden layers, an output layer, and a target layer. The multi-communication link selection model implements a scheduling process through the following steps:

[0122] Step 1, receive the current state information {A, K} of each channel through the input layer;

[0123] Step 2: Process the state information transmitted from the input layer through multiple hidden layers, and extract high-level feature representations through multi-layer nonlinear transformations; the hidden layers can be either fully connected layers (Dense layers) or convolutional layers (Convolutional layers). Different numbers and configurations of hidden layers can be selected according to the complexity of the task and the characteristics of the input data;

[0124] Step 3: The output layer represents the corresponding action according to these high-level features, and judges the closeness of the action to the expected goal according to the reward function, and gradually adjusts the action to maximize the reward function. The action obtained in this case is the output scheduling strategy.

[0125] S103, forwarding the message content to the receiving end program in real time according to the dynamically selected target communication link and transmission action weight.

[0126] It is understandable that this process is not static, but is updated in real time according to the network conditions of each channel and the queue conditions within the channel. The algorithm model will dynamically select the most ideal scheduling strategy at each time step while sending data at the fastest speed and balancing complex constraints. The overall data transmission is completed according to the scheduling strategy.

[0127] Figure 4is a flowchart of another decentralized multi-communication link data transmission method based on deep reinforcement learning according to an embodiment of the present application, such as Figure 4 As shown, the process includes the following steps:

[0128] Step 1: Initialize each channel communication module;

[0129] Step 2: Set the processing callback after receiving the message;

[0130] Step 3: Choose whether to send a message, if yes, proceed to step 4, otherwise proceed to step 17;

[0131] Step 4: Enter the message content in the TOPIC you want to send;

[0132] Step 5: Pass the input data into the intelligent decision algorithm to obtain the channel decision;

[0133] Step 6: Whether it needs to be sent to the local channel, if yes, go to step 7, otherwise go to step 10;

[0134] Step 7: Send the message content to the shared memory-based message channel;

[0135] Step 8: Determine whether the message has received a reply, if yes, go back to step 3, otherwise go to step 10;

[0136] Step 9: Whether it needs to be sent to the LAN channel for review, if yes, go to step 10, otherwise go to step 13;

[0137] Step 10: Send the message content to the P2P-based message channel;

[0138] Step 11: Determine whether the message has received a reply, if yes, return to step 3, otherwise proceed to step 11;

[0139] Step 12: Whether it needs to be sent to the Internet channel, if yes, go to step 13, otherwise go to step 16

[0140] Step 13: Send the message content to the MQTT-based message channel;

[0141] Step 14: Determine whether the message has received a reply, if yes, go back to step 3, otherwise go to step 15;

[0142] Step 15: Determine whether the message has timed out, if so, go to step 16, otherwise go back to step 7;

[0143] Step 16: Discard the illegal message content and return to step 3;

[0144] Step 17: Disconnect all connections and exit;

[0145] By executing the above steps in a loop, the transmission channel can be dynamically selected and switched, and the transmission of the message content can be completed with the goal of minimum time consumption and load balancing.

[0146] also, Figure 5 It is a schematic diagram of the overall interaction process between the sending end and the receiving end under different network environments according to the embodiments of the present application.

[0147] Through the above steps S101 to S103, compared with the single-link transmission method in traditional technology, the present application implements multiple message communication methods between applications locally, in a local area network, or across local area networks through shared memory, P2P, and MQTT, which can avoid reducing the loss of device connection caused by a single channel failure; in addition, the selection of message transmission channels is optimized through deep reinforcement learning, and data transmission is completed as quickly as possible while considering load balancing; the reliability of message communication is increased, the message communication delay is reduced, the communication cost between applications is greatly reduced, and the communication efficiency is improved.

[0148] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0149] This application also provides a decentralized multi-communication link data transmission system based on deep reinforcement learning. Figure 6 is a structural block diagram of a decentralized multi-communication link data transmission system based on deep reinforcement learning according to an embodiment of the present application, such as Figure 6 As shown, the system includes an upload module 60, a scheduling module 61 and a forwarding module 62, wherein:

[0150] The upload module 60 is used to respond to the message sending request of the sending end program, upload the message content to the target topic, obtain the receiving end program subscribed to the target topic, and obtain the communication link supported by the receiving end program;

[0151] The scheduling module 61 is used to dynamically output a scheduling strategy based on the network status and queue status of each communication link through a multi-communication link selection scheduling model obtained in advance based on deep reinforcement learning, wherein the scheduling strategy includes selecting a target communication link for transmitting message content and transmission action information of each target communication link;

[0152] The forwarding module 62 is used to forward the message content to the receiving end program in real time according to the dynamically output scheduling strategy.

[0153] Through the above system, multiple message communication methods between applications locally, in a LAN, or across LANs are implemented through shared memory, P2P, and MQTT, which can avoid reducing the loss of device connection caused by a single channel failure; in addition, the selection of message transmission channels is optimized through deep reinforcement learning, and data transmission is completed as quickly as possible while considering load balancing; the reliability of message communication is increased, the message communication delay is reduced, the communication cost between applications is greatly reduced, and the communication efficiency is improved.

[0154] In one embodiment, Figure 7 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application, such as Figure 7 As shown, an electronic device is provided, which may be a server, and its internal structure diagram may be as shown in Figure 7 The electronic device includes a processor, a memory, a network interface and a database connected through a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the electronic device is used to store data. The network interface of the electronic device is used to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement a decentralized multi-communication link data transmission method based on deep reinforcement learning.

[0155] Those skilled in the art will understand that Figure 7 The structure shown in the figure is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0156] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0157] Those skilled in the art should understand that the technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0158] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A decentralized multi-communication link data transmission method based on deep reinforcement learning, characterized in that: include: Respond to the message sending request of the sending program and upload the message content to the target topic; Acquire a receiving end program that subscribes to the target topic, and acquire a communication link supported by the receiving end program; By selecting and scheduling a multi-communication link based on deep reinforcement learning, a scheduling strategy is dynamically output based on the network status and queue status of each communication link, wherein the scheduling strategy includes selecting a target communication link for transmitting the message content; According to the scheduling strategy of the dynamic output, the message content is forwarded to the receiving end program in real time.

2. The method according to claim 1, characterized in that The communication link comprises a first communication link, a second communication link and a third communication link, wherein: The first communication link is an IPC channel constructed based on local shared memory when the receiving end program and the sending end program are located on the same device; The second communication link is a Local channel constructed based on a P2P communication protocol in the local area network when the receiving end program and the sending end program are located in the same local area network; The third communication link is a Remote channel constructed based on the MQTT protocol when the receiving program and the sending program are located in different network segments.

3. The method according to claim 2, characterized in that Through the multi-communication link selection scheduling model, based on the network status and queue status of each communication link, the dynamic output scheduling strategy includes: In the initial state, according to the total amount of data of the message content, and the network state and queue state of each communication link, the minimum time required for each communication link to complete the transmission of the message content is obtained; In a unit time window, the observation data of each communication link is obtained respectively, and a linear fitting is performed based on the observation data of each communication link by the least square method to obtain the delay variation trend of each communication link and the influence parameter of the delay variation trend on the subsequent transmission of data; Through the multi-communication link selection scheduling model pre-trained based on the D3QN algorithm, the scheduling strategy at each time step is dynamically output based on the total amount of data in the message content, the minimum time consumption in the initial state, the delay change trend and the influencing parameters.

4. The method according to claim 3, characterized in that According to the total amount of data of the message content, and the network status and queue status of each communication link, obtaining the minimum time required for each communication link to complete the transmission of the message content includes: In the initial state, the total amount of data of the message content, as well as the established transmission rate of each communication link, the average delay and the amount of data to be processed in the queue of the channel are obtained; According to the total amount of data and the established transmission rate of each communication link, the average delay and the amount of data to be processed in the queue, the transmission time required for each communication link to complete the task of forwarding the message content is obtained respectively; The minimum transmission time required for each communication link to complete the task of forwarding the message content is obtained as the minimum time consumption.

5. The method according to claim 3, characterized in that: The delay variation trend of each communication link is obtained by performing linear fitting based on the observed data of each communication link by the least square method, and the influence parameters of the delay variation trend on the subsequent transmission of data include: In a unit time window, observation data of each communication link is obtained respectively, wherein the observation data includes the time when any data point is generated and the minimum time required for the data point to transmit a unit amount of data in its communication link; According to the multiple observation points of each communication link, a least square method is used to perform linear fitting to obtain the delay change trend of each communication link in a future period; According to the total amount of data and the delay variation trend of each communication link, the influencing parameter corresponding to each communication link is obtained.

6. The method according to claim 3, characterized in that The multi-communication link selection scheduling model uses the delay change trend of each communication link as state information, whether to use each communication link as action information, and is trained with the goal of maximizing the reward function at each time step, wherein: The reward function is expressed by the following formula: R=-(X ipc I ipc +X local I local +X remote I remote ) Where R represents the reward function, x ipc 、x local 、x remote I respectively represent the transmission action information of the first communication link, the second communication link and the third communication link, ipc ,I local ,I remote Respectively represent the influencing parameters of the first communication link, the second communication link and the third communication link.

7. The method according to claim 6, characterized in that The multi-communication link selection scheduling model includes an input layer, a hidden layer and an output layer, wherein; The state information is received through the input layer, and a multi-layer nonlinear transformation is performed on the state information through a plurality of hidden layers to obtain a high-level feature identifier; Based on the high-level feature representation, the output layer dynamically outputs the action information of each communication link at the current time step under the constraint of the reward function.

8. A decentralized multi-communication link data transmission system based on deep reinforcement learning, characterized in that: The system includes: an upload module, a scheduling module and a forwarding module, wherein: The upload module is used to respond to the message sending request of the sending end program, upload the message content to the target topic, obtain the receiving end program subscribed to the target topic, and obtain the communication link supported by the receiving end program; The scheduling module is used to dynamically output a scheduling strategy based on the network status and queue status of each communication link by selecting a scheduling model for multiple communication links obtained in advance based on deep reinforcement learning, wherein the scheduling strategy includes selecting a target communication link for transmitting the message content and transmission action information of each target communication link; The forwarding module is used to forward the message content to the receiving end program in real time according to the scheduling strategy of the dynamic output.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images