Heterogeneous service signal transmission method and system based on multi-agent reinforcement learning

Abstract

The application belongs to the technical field of communication and specifically relates to a heterogeneous service signal transmission method and system based on multi-agent reinforcement learning. The transmission process of data is modeled as a Markov decision process, and the current data packet forwarding task is mapped as a quadruple of state, action, state transition probability and reward. A multi-objective reward function is designed, which comprehensively considers the service priority, link reliability, delay and energy balance factors. The long-term cumulative reward expectation Q value is calculated through distributed value function iteration. Each node uses the local state, reward and Q value information of the neighbor node for joint optimization. The neighbor node with the maximum Q value is selected as the relay. The data packet is updated, packaged and transmitted. Compared with the prior art, the application can reduce the end-to-end delay of key data, balance the network load and energy consumption, and ensure the reliable transmission of heterogeneous service data under dynamic topology by constructing a multi-agent reinforcement learning framework to accelerate the learning of the optimal transmission path.

Description

Technical Field

[0001] This invention belongs to the field of communication technology, specifically relating to a heterogeneous service signal transmission method and system based on multi-agent reinforcement learning, which is applied to UAV network communication of heterogeneous service data, including anomaly alarms with high timeliness requirements, real-time perception with high throughput requirements, and periodic monitoring data that is not sensitive to latency. Background Technology

[0002] With the rapid development of 6G and the low-altitude economy, unmanned aerial vehicle (UAV) networks, as an important component, have shown great potential in fields such as environmental monitoring and disaster relief. Compared with traditional mobile networks, UAV networks operate in three-dimensional space, characterized by dramatic dynamic changes in topology and limited node energy. In complex monitoring tasks, the network needs to carry heterogeneous service data, including anomaly alarms with high timeliness requirements, real-time sensing with high throughput requirements, and periodic monitoring data that is not sensitive to latency.

[0003] Traditional transmission methods include location-based transmission and topology-aware transmission. Location-based transmission updates the UAV's position after network topology changes by broadcasting information based on node location information, or it uses methods such as directed particle swarm optimization and Kalman filtering to predict the UAV's position and select a transmission path by combining information such as the UAV's remaining energy and node degree. In addition, reinforcement learning-based transmission methods have been widely used. Through continuous interaction and learning with the environment, nodes can autonomously perceive the local topology and make transmission decisions, ultimately learning the optimal transmission path.

[0004] However, the above solutions do not consider the differences in business data. For example, networks have strict requirements for low latency and high reliability in transmitting abnormal alarm information, while networks tend to balance network load and energy consumption to extend network lifespan when transmitting periodic monitoring data. This means that existing solutions cannot meet the transmission needs of heterogeneous services. The main drawbacks of existing technologies include: 1. Ignoring the heterogeneous characteristics of business data: Most existing transmission methods adopt a single forwarding strategy, which fails to fully consider the heterogeneous characteristics of business data and the differences in their network resource requirements.

[0005] 2. Difficulty in guaranteeing the transmission performance of critical information: When facing mixed traffic, since all data packets are treated indiscriminately in the buffer, emergency alarm information triggered by sudden events may be overwhelmed by massive amounts of periodic monitoring data, increasing the queuing delay and packet loss risk of critical information.

[0006] 3. Traditional reinforcement learning frameworks are slow to learn: When calculating the cumulative expected reward, existing reinforcement learning frameworks only consider the local state and reward, without considering the rewards of neighboring nodes, which results in a slow speed in learning and obtaining the optimal transmission path. Summary of the Invention

[0007] To address the problems existing in the prior art, this invention provides a heterogeneous service signal transmission method based on multi-agent reinforcement learning, aiming to provide differentiated quality of service assurance for unmanned aerial vehicle (UAV) networks.

[0008] The technical solution adopted by this invention to solve its technical problem is as follows: A heterogeneous service signal transmission method based on multi-agent reinforcement learning, which models the data transmission process as a Markov decision process, and maps the current data packet forwarding task as a quadruple of state, action, state transition probability, and reward; designs a multi-objective reward function that comprehensively considers service priority, link reliability, latency, and energy balance factors; and calculates the long-term cumulative expected reward Q value through distributed value function iteration, with each node using local state, reward, and Q value information of neighboring nodes for joint optimization; including the following steps: S1. Receive data packets and obtain all information of the current data packet; S2. Prioritize heterogeneous services and mark their priority according to service type, and enter the corresponding level of cache queue; S3. Obtain the status information of neighboring nodes, calculate the immediate reward according to the multi-objective reward function, and perform Q-value iterative calculation according to the distributed value function; S4. Select the neighbor node with the largest Q value as the relay, and each node selects the neighbor node with the largest Q value as the next hop for data transmission. S5. Update the local Q table and status information; S6. Encapsulate the data packet and transmit it.

[0009] Preferably, the data packet structure includes: The packet ID is used for deduplication and tracking in multi-hop transmissions and serves as a unique identifier for the packet. Source node ID, which identifies the starting node from which the data was generated; The sending node ID and receiving node ID are used to record both parties in the current single-hop transmission; The V value represents the expected cumulative return of the current node; Business type, which identifies the business type of the data; Remaining energy, which represents the remaining battery capacity of the current node; Queue length, which reflects the number of data packets waiting to be transmitted in the current node's buffer; Payload data refers to heterogeneous business data that needs to be transmitted.

[0010] Preferably, in this method, each node maintains a heterogeneous service cache queue with different priorities to achieve classified storage of service data; during the data forwarding phase, the node dynamically sorts the data to be processed in the cache queue based on the importance of the service, and prioritizes scheduling high-priority data packets to the transmission channel for forwarding.

[0011] Preferably, the Markov decision process consists of quaternions. Description, where: the state set S is a finite set consisting of all nodes in the UAV network, and when the data packet is at node i, the system state is... Action set A is a finite set of nodes selected as the next-hop relay node by the current node. When node j is selected as the relay node for data transmission, the action is represented as follows: State transition probability When node i selects node j for data packet transmission, the action is completed. Successfully changed from state Transition to state The probability; R is the reward. To select an action by node i And the immediate reward obtained from successfully completing the state transition.

[0012] Preferably, the multi-objective reward function expression is: ; in, For fixed transmission overhead, As a priority factor, For reliability-related functions, For delay-dependent functions, The residual energy correlation function, These are the weighting coefficients corresponding to the reliability correlation function. These are the weighting coefficients corresponding to the time-delay correlation function. These are the weighting coefficients corresponding to the residual energy correlation function.

[0013] Preferably, the priority factor p is dynamically adjusted according to the service priority of the data packet, with higher priority services corresponding to higher p values, so as to prioritize the reliability and latency performance of high priority services.

[0014] Preferably, it is used for Q-value iterative calculation of a distributed value function: ; Expected cumulative return of the current node ; The immediate reward obtained for the current forwarding action is calculated by a multi-objective reward function. ; The learning rate is used to adjust the rate at which the Q-value is updated. Represents the set of state transitions. and These represent the weighting factors for the long-term returns of the next state and other states, respectively. It is the sum of the state values ​​of all neighboring nodes; The weighting factor for the impact of direct rewards at future moments on the current moment.

[0015] The heterogeneous service signal transmission system based on multi-agent reinforcement learning executes the heterogeneous service signal transmission method of this application and includes multiple UAV nodes, each UAV node containing the following modules: The data preprocessing module is used to maintain heterogeneous business cache queues with different priorities, realize the classified storage of business data, and dynamically sort the data to be processed in the cache queues based on business importance. The state awareness module is used to collect local node state information, neighbor node state information, and link state information as input states for reinforcement learning. The reinforcement learning decision module has a built-in multi-objective reward function and distributed value function iteration mechanism to calculate the Q value of the next hop node and select the neighbor node with the largest Q value as the next hop relay node. The data transmission module is used to prioritize high-priority data packets to the transmission channel for forwarding based on the output of the decision module.

[0016] Preferably, the reinforcement learning decision module supports dynamically adjusting weight coefficients to adapt to the optimization target requirements of different application scenarios.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention constructs a multi-agent reinforcement learning framework. By modeling UAV nodes as autonomous decision-making agents, a distributed framework is used to achieve local perception and independent decision-making, eliminating dependence on global information; and a distributed value function method is introduced, enabling nodes to jointly optimize by utilizing local state and reward while combining the Q-value information of neighboring agents.

[0018] 2. This invention designs a multi-objective reward function for heterogeneous service awareness. By introducing a priority factor and comprehensively considering factors such as link reliability, latency, congestion, and energy balance, the network can dynamically adjust forwarding priorities according to service importance, ensuring that urgent data receives priority forwarding and balancing network load and energy consumption.

[0019] 3. This invention proposes a data preprocessing mechanism. Each node maintains a priority-based heterogeneous service cache queue, and before executing forwarding tasks, the data to be processed is sorted according to service importance, ensuring that high-priority data packets are extracted to the transmission channel first.

[0020] In summary, this invention constructs a multi-agent reinforcement learning framework, which accelerates the learning of the optimal transmission path, reduces end-to-end latency of critical data, balances network load and energy consumption, and ensures reliable transmission of heterogeneous service data under dynamic topology. Attached Figure Description

[0021] Figure 1 This is a flowchart of the method for transmitting heterogeneous service signals in a UAV network in this application.

[0022] Figure 2 This is a diagram of the data packet structure in Example 1. Detailed Implementation

[0023] To facilitate understanding of the present invention, it will be described in more detail below with reference to the accompanying drawings and specific embodiments. However, the present invention can be implemented in many different forms and is not limited to the embodiments described in this specification. Rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of the present invention.

[0024] The heterogeneous service signal transmission method based on multi-agent reinforcement learning proposed in this invention has the following process: Figure 1 As shown. Includes the following steps: S1. Receive data packets and obtain all information of the current data packet; S2. Prioritize heterogeneous services and mark their priority according to service type, and enter the corresponding level of cache queue; S3. Obtain the status information of neighboring nodes, calculate the immediate reward according to the multi-objective reward function, and perform Q-value iterative calculation according to the distributed value function; S4. Select the neighbor node with the largest Q value as the relay, and each node selects the neighbor node with the largest Q value as the next hop for data transmission. S5. Update the local Q table and status information; S6. Encapsulate the data packet and transmit it.

[0025] This method models the data transmission process as a Markov decision process, mapping the current packet forwarding task to a quadruple of state, action, state transition probability, and reward. It designs a multi-objective reward function that comprehensively considers factors such as service priority, link reliability, latency, and energy balance. The method iteratively calculates the long-term cumulative expected reward Q-value through a distributed value function, with each node jointly optimizing using its local state, reward, and Q-value information from neighboring nodes. Each node maintains heterogeneous service cache queues with different priorities to achieve categorized storage of service data. During the data forwarding phase, nodes dynamically sort the data to be processed in the cache queues based on service importance, prioritizing high-priority packets for forwarding through the transmission channel.

[0026] In one embodiment, the data packet structure includes: The packet ID is used for deduplication and tracking in multi-hop transmissions and serves as a unique identifier for the packet. Source node ID, which identifies the starting node from which the data was generated; The sending node ID and receiving node ID are used to record both parties in the current single-hop transmission; The V value represents the expected cumulative return of the current node; Business type, which identifies the business type of the data; Remaining energy, which represents the remaining battery capacity of the current node; Queue length, which reflects the number of data packets waiting to be transmitted in the current node's buffer; Payload data refers to heterogeneous business data that needs to be transmitted.

[0027] Example 1: The heterogeneous service signal transmission method of this application is applied to UAV network communication. UAV networks operate in three-dimensional space and are characterized by drastic dynamic changes in topology and limited node energy. In complex monitoring tasks, the network needs to carry heterogeneous service data, including anomaly alarms with high timeliness requirements, real-time sensing with high throughput requirements, and periodic monitoring data that is not sensitive to latency.

[0028] This method models each UAV node in the network as an intelligent agent with autonomous decision-making capabilities, utilizing a reinforcement learning framework to achieve local perception and independent decision-making, eliminating dependence on global information. By introducing a distributed value function to achieve multi-agent collaboration, nodes not only utilize their local state and rewards but also the Q-value information of their neighbors for joint optimization, thereby achieving more global transmission decisions and accelerating the learning process. By comprehensively considering factors such as service priority, link reliability, latency, and energy balance, a multi-objective reward function is designed. Dynamically adjusting weight coefficients ensures reduced latency and improved reliability for high-priority services, while balancing network energy efficiency and load through low-priority services. Finally, through iterative approximate calculation of the Q-value, data packets can avoid nodes with poor link quality or those about to run out of energy, generating the optimal transmission path. The specific process is as follows.

[0029] The data transmission process in unmanned aerial vehicle (UAV) networks is modeled as a Markov decision process. This process can be represented by a quadruple. Description. Here, the state set S is defined as a finite set consisting of all nodes in the UAV network. When the data packet is at node i, the system state is... Define the action set A as a finite set of nodes selected by the current node as the next-hop relay node. When node j is selected as the relay node for data transmission, the action is represented as follows: Define the state transition probability. When node i selects node j for data packet transmission, the action is completed. Successfully changed from state Transition to state The probability of [the action chosen by node i]. Furthermore, the reward is defined as the action selected by node i. And the immediate reward obtained by successfully completing the state transition, namely The reward function is designed as follows: ; in, For fixed transmission overhead, As a priority factor, These are the correlation functions for reliability, latency, and remaining energy, respectively. These are the weighting coefficients corresponding to the relevant functions, and their values ​​vary depending on the specific scenario.

[0030] The direct return can then be expressed as: ; The Q-value can be calculated iteratively using a distributed value function: ; ; in, The learning rate is used to adjust the rate at which the Q-value is updated. Represents the set of state transitions. and These represent the weighting factors for the long-term returns of the next state and other states, respectively. The expected cumulative reward value of the current node is used for iterative calculation of the Q-value of neighboring nodes; The immediate reward obtained for the current forwarding action is calculated by a multi-objective reward function; It is the sum of the state values ​​of all neighboring nodes; The state-action value function; This is a weighting factor representing the impact of direct rewards from future moments on the current moment. The Q-value iteration mechanism of the distributed value function considers both the next state reward and the neighbor node state rewards during Q-value updates, eliminating dependence on global information and accelerating the convergence of multi-agent learning.

[0031] After calculation, each node selects the neighbor node with the largest Q-value as the next hop for data transmission. Furthermore, this method includes a data preprocessing mechanism. Each node maintains heterogeneous service cache queues with different priorities to achieve categorized storage of service data. During the data forwarding phase, nodes dynamically sort the data to be processed in the cache queues based on service importance, prioritizing high-priority data packets for forwarding onto the transmission channel. The data packet structure is as follows: Figure 2 As shown, it includes the information to be transmitted, data packet information, and routing information, and designs the packet header, wherein: The packet ID is used for deduplication and tracking in multi-hop transmission and serves as a unique identifier for the packet. The source node ID is used to identify the originating drone node that generated the data; The sending node ID is used to record the drone node that forwarded the data packet in the current single-hop transmission; The receiving node ID is used to record the drone node that received the data packet in the current single-hop transmission; The V value is used to represent the expected cumulative return of the current node and is used for iterative calculation of the Q value of neighboring nodes; The business type is used to identify the business type of the data and corresponds to the priority level; Remaining energy is used to represent the current battery balance of the drone node; The queue length reflects the number of data packets waiting to be transmitted in the current node's buffer, and is used for load awareness of neighboring nodes.

[0032] Payload data refers to heterogeneous business data that needs to be transmitted.

[0033] The heterogeneous service signal transmission system based on multi-agent reinforcement learning executes the heterogeneous service signal transmission method of this application and includes multiple UAV nodes, each UAV node containing the following modules: The data preprocessing module is used to maintain heterogeneous business cache queues with different priorities, realize the classified storage of business data, and dynamically sort the data to be processed in the cache queues based on business importance. The state awareness module is used to collect local node state information, neighbor node state information, and link state information as input states for reinforcement learning. The reinforcement learning decision module has a built-in multi-objective reward function and distributed value function iteration mechanism to calculate the Q value of the next hop node and select the neighbor node with the largest Q value as the next hop relay node. The data transmission module is used to prioritize high-priority data packets to the transmission channel for forwarding based on the output of the decision module.

[0034] In one embodiment, the reinforcement learning decision module supports dynamically adjusting weight coefficients to adapt to the optimization objectives of different application scenarios.

[0035] In one embodiment, each field uses a fixed encoding, and the total header length is 64 bytes, which is suitable for the low-overhead transmission requirements of drone networks.

[0036] This invention accelerates the network's learning rate for the optimal signal transmission path by constructing a multi-agent reinforcement learning framework, and dynamically adjusts forwarding priority and path selection according to the importance weight of the service. This reduces end-to-end latency of critical data while optimizing the balance between the transmission reliability of heterogeneous data and network energy efficiency.

[0037] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A heterogeneous service signal transmission method based on multi-agent reinforcement learning, characterized in that, The data transmission process is modeled as a Markov decision process, mapping the current packet forwarding task to a quadruple of state, action, state transition probability, and reward. A multi-objective reward function is designed, comprehensively considering factors such as service priority, link reliability, latency, and energy balance. The long-term cumulative expected reward Q-value is calculated iteratively using a distributed value function, and each node performs joint optimization using its local state, reward, and Q-value information from neighboring nodes. The process includes the following steps: S1. Receive data packets and obtain all information of the current data packet; S2. Prioritize heterogeneous services and mark their priority according to service type, and enter the corresponding level of cache queue; S3. Obtain the status information of neighboring nodes, calculate the immediate reward according to the multi-objective reward function, and perform Q-value iterative calculation according to the distributed value function; S4. Select the neighbor node with the largest Q value as the relay, and each node selects the neighbor node with the largest Q value as the next hop for data transmission. S5. Update the local Q table and status information; S6. Encapsulate the data packet and transmit it.

2. The heterogeneous service signal transmission method based on multi-agent reinforcement learning according to claim 1, characterized in that, The data packet structure includes: The packet ID is used for deduplication and tracking in multi-hop transmissions and serves as a unique identifier for the packet. Source node ID, which identifies the starting node from which the data was generated; The sending node ID and receiving node ID are used to record both parties in the current single-hop transmission; The V value represents the expected cumulative return of the current node; Business type, which identifies the business type of the data; Remaining energy, which represents the remaining battery capacity of the current node; Queue length, which reflects the number of data packets waiting to be transmitted in the current node's buffer; Payload data refers to heterogeneous business data that needs to be transmitted.

3. The heterogeneous service signal transmission method based on multi-agent reinforcement learning according to claim 1, characterized in that, In this method, each node maintains a heterogeneous business cache queue with different priorities to achieve classified storage of business data; During the data forwarding phase, nodes dynamically sort the data to be processed in the cache queue based on business importance, prioritizing high-priority data packets for forwarding to the transmission channel.

4. The heterogeneous service signal transmission method based on multi-agent reinforcement learning according to any one of claims 1-3, characterized in that, Markov decision processes, consisting of quadruples Description, where: the state set S is a finite set consisting of all nodes in the UAV network, and when the data packet is at node i, the system state is... Action set A is a finite set of nodes selected as the next-hop relay node by the current node. When node j is selected as the relay node for data transmission, the action is represented as follows: State transition probability When node i selects node j for data packet transmission, the action is completed. Successfully changed from state Transition to state The probability; R is the reward. To select an action by node i And the immediate reward obtained from successfully completing the state transition.

5. The heterogeneous service signal transmission method based on multi-agent reinforcement learning according to claim 4, characterized in that, The expression for the multi-objective reward function is: ； in, For fixed transmission overhead, As a priority factor, For reliability-related functions, For delay-dependent functions, The residual energy correlation function, These are the weighting coefficients corresponding to the reliability correlation function. These are the weighting coefficients corresponding to the time-delay correlation function. These are the weighting coefficients corresponding to the residual energy correlation function.

6. The heterogeneous service signal transmission method based on multi-agent reinforcement learning according to claim 5, characterized in that, The priority factor p is dynamically adjusted according to the service priority of the data packet. Higher priority services correspond to higher p values ​​in order to prioritize the reliability and latency performance of high priority services.

7. The heterogeneous service signal transmission method based on multi-agent reinforcement learning according to claim 5, characterized in that, Used for distributed value functions in Q-value iterative computation: ； Expected cumulative return of the current node ; The immediate reward obtained for the current forwarding action is calculated by a multi-objective reward function. ; The learning rate is used to adjust the rate at which the Q-value is updated. Represents the set of state transitions. and These represent the weighting factors for the long-term returns of the next state and other states, respectively. It is the sum of the state values ​​of all neighboring nodes; The weighting factor for the impact of direct rewards at future moments on the current moment.

8. A heterogeneous service signal transmission system based on multi-agent reinforcement learning, executing the heterogeneous service signal transmission method as described in claim 1, characterized in that, It includes multiple drone nodes, and each drone node contains the following modules: The data preprocessing module is used to maintain heterogeneous business cache queues with different priorities, realize the classified storage of business data, and dynamically sort the data to be processed in the cache queues based on business importance. The state awareness module is used to collect local node state information, neighbor node state information, and link state information as input states for reinforcement learning. The reinforcement learning decision module has a built-in multi-objective reward function and distributed value function iteration mechanism to calculate the Q value of the next hop node and select the neighbor node with the largest Q value as the next hop relay node. The data transmission module is used to prioritize high-priority data packets to the transmission channel for forwarding based on the output of the decision module.

9. The system according to claim 8, characterized in that, The reinforcement learning decision-making module supports dynamically adjusting weight coefficients to adapt to the optimization goals of different application scenarios.

Images