MAC protocol design method based on multi-objective optimization in underwater acoustic sensor networks

Abstract

The present invention discloses a MAC protocol design method based on multi-objective optimization in an underwater acoustic sensor network, and the specific steps are as follows: first, a network model of the underwater acoustic sensor network is established; second, a time slot model is established to convert the propagation delay between the underwater sensor node and the surface AP node into the number of time slots; on this basis, a multi-agent reinforcement learning model is established, and a centralized training and distributed execution architecture is adopted. By adopting a multi-agent reinforcement learning method based on multi-objective optimization, the present invention significantly improves the system throughput and energy efficiency in the underwater acoustic communication network, and effectively solves the problem of unfair resource allocation between nodes. By introducing a delayed reward and priority experience replay mechanism, the agent can accurately update the strategy in an environment with long propagation delay, thereby improving learning efficiency. The present invention not only improves the overall communication efficiency of the network, but also enhances the adaptability and stability of the system in a complex environment with high delay.

Description

Technical Field

[0001] The present invention belongs to the field of media access control (MAC) protocols for underwater acoustic sensor networks, and specifically relates to a MAC protocol design method based on multi-objective optimization in underwater acoustic sensor networks. Background Art

[0002] Underwater Acoustic Sensor Networks (UASNs) hold broad application prospects in areas such as marine environmental monitoring, resource exploration, and military applications. However, due to the unique characteristics of the underwater environment, including low channel bandwidth, high propagation delay, high bit error rate, and limited energy, underwater communication networks face numerous challenges in designing efficient MAC protocols. Traditional MAC protocols, such as ALOHA and TDMA, are unable to achieve efficient resource scheduling and node collaboration in such complex environments, resulting in high collision rates, low throughput, and unfair resource allocation.

[0003] In recent years, multi-agent reinforcement learning (MARL), as an effective method for achieving agent collaboration in complex dynamic environments, has gradually been applied to the design of MAC protocols. MARL can resolve conflicts and collaboration problems between nodes in the network through the autonomous learning of each agent. However, existing MARL-based MAC protocol designs often focus on optimizing a single objective, such as maximizing throughput, and fail to consider both energy efficiency and fairness in resource allocation. In underwater communication environments, due to the high transmission energy consumption and limited node energy, optimizing energy efficiency becomes an important goal in designing efficient MAC protocols. At the same time, the fairness issue, namely how to ensure that all nodes have equal access to communication resources, also directly affects the long-term stability and overall performance of the network. Summary of the Invention

[0004] To address the above problems, the present invention designs a MAC protocol method based on multi-objective optimization in underwater acoustic sensor networks. This method uses a centralized training and distributed execution architecture combined with a dual reward mechanism to achieve node energy efficiency optimization and resource allocation fairness while ensuring high system throughput, adapting to the special needs of underwater acoustic networks.

[0005] The technical solutions of the present invention are as follows:

[0006] A MAC protocol design method based on multi-objective optimization in an underwater acoustic sensor network includes the following steps:

[0007] Step 1: Establish a network model for an underwater acoustic sensor network. The underwater acoustic sensor network includes several underwater sensor nodes and surface AP nodes. The underwater sensor nodes use TDMA, ALOHA, and MOMA-MAC protocols, respectively, and are randomly distributed in the underwater area. They collect sensed data from the surrounding environment and transmit it to the surface AP nodes. Each sensor node sends a detection signal containing its own position to the surface AP node, and calculates the distance between each sensor node and the surface AP node after receiving the ACK packet from the surface AP node.

[0008] Step 2: Establish a time slot model and convert the distance between each underwater sensor node and the surface AP node into a propagation delay in units of time slots;

[0009] Step 3: Establish a multi-agent reinforcement learning model based on a partially observable Markov decision process (POMDP). Each underwater sensor node is modeled as an agent. Each underwater sensor node can only make decisions based on its local observation data. The CTDE architecture is used for centralized training and distributed execution. The global state is evaluated through a centralized critic network, and the joint actions of multiple agents are combined to optimize the strategy.

[0010] Step 4: Construct a multi-objective optimization problem and design a dual reward mechanism based on it, including global rewards and individual rewards. The global reward is used to improve the overall throughput of the system, while the individual reward takes into account energy efficiency and fairness goals.

[0011] Step 5: The underwater sensor nodes deploy a delayed reward multi-agent reinforcement learning algorithm, which can learn the unused idle time slots of TDMA nodes and ALOHA nodes for transmission and iteration. During the execution process, each underwater sensor node makes distributed decisions within the time slot according to the learned strategy and autonomously selects the appropriate time slot for data transmission. Through the collaboration and competition of multiple agents, the MOMA-MAC protocol can maximize the time slot utilization and effectively balance the node energy efficiency and resource allocation fairness while maintaining high throughput.

[0012] Preferably, the specific steps of step 2 above are as follows:

[0013] 2.1 In the time slot system, it is assumed that the data packets from different underwater sensor nodes have the same packet length, and the ACK packets from the surface AP node to different underwater sensor nodes also have the same packet length;

[0014] 2.2 Length of each time slot T slot Corresponds to the duration T of the data packet transmission prop Plus the duration T of ACK packet transmission ack and protection time t g; That is, each time slot is divided into two parts: the first part is for data packets, and the second part is for ACK packets;

[0015] 2.3 In the first part, in each time slot, if the MAC decision of an underwater sensor node is to send, it sends a data packet to the surface AP node, and the surface AP node waits for data packets from all underwater sensor nodes; in the second part, the underwater sensor node waits for the ACK packet from the surface AP node, and if a data packet has just been received in the first part of the time slot, the surface AP node broadcasts the ACK packet to all underwater sensor nodes;

[0016] 2.4 Propagation delay refers to the time it takes for a signal to be sent and received. For an underwater sensor node i (i∈{1,2,...,N}), assume that the distance between it and the surface AP node is d i , T slot is the length of a time slot, c is the speed of underwater sound propagation, and the propagation delay T prop Calculated by the following formula:

[0017]

[0018] Preferably, the specific steps of step 4 above are as follows:

[0019] 4.1 Construct a multi-objective optimization problem. The steps are as follows:

[0020] 4.1.1 Throughput Objective: Each agent's goal is to learn idle time slots and select appropriate time slots for data transmission without colliding with TDMA, ALOHA, or other agent nodes, thereby maximizing the time slot utilization of a frame. Through collaboration between agents, the overall system throughput is improved.

[0021] 4.1.2 Fairness Objective: To ensure the long-term stability of the system, agents need to consider past resource usage when selecting time slots. Agents that have not received transmission opportunities for a long time will be given priority to prevent individual agents from monopolizing resources. This fairness can be measured by the agents' historical transmission records to ensure balanced resource allocation.

[0022] 4.1.3 Energy Efficiency Objectives: During the transmission process, the energy consumption of each agent must be controlled. When deciding whether to transmit data, the agent must make decisions based on its current remaining energy. By avoiding excessive transmission, the node's working cycle can be extended, improving the energy efficiency of the entire network.

[0023] 4.2 Design a dual reward mechanism. The steps are as follows:

[0024] 4.2.1 Global Rewards: Global rewards are used to improve the overall throughput of the system. When an underwater sensor node successfully sends a data packet and the surface AP node receives and broadcasts an ACK packet, the system will allocate a global reward to the underwater sensor node. The goal of global rewards is to encourage individual agents to improve the system's time slot utilization and overall data transmission efficiency through collaborative work.

[0025] 4.2.2 Individual Rewards: Individual rewards take into account the local decision-making effect of each agent, encouraging the agent to maximize fairness and energy efficiency when selecting time slots. Fairness rewards are adjusted based on the time slot resources each underwater sensor node has obtained in the past. Underwater sensor nodes that have not obtained transmission opportunities for a long time will be given priority to receive more rewards, thereby ensuring fairness in resource allocation. Energy efficiency rewards are proportional to the remaining energy of the underwater sensor node, incentivizing underwater sensor nodes to transmit first when there is more remaining energy, thereby extending the life cycle of underwater sensor nodes.

[0026] 4.2.3 The final reward function is composed of a global reward and an individual reward. By adjusting the weights of these two rewards, the optimal balance between throughput, energy efficiency, and fairness is achieved according to the requirements of the application scenario.

[0027] Preferably, the specific steps for setting the reward function in step 4 are as follows:

[0028] The settings of the agent, action, state, and reward function are as follows:

[0029] Agent: Each underwater sensor node using the MOMA-MAC protocol is an agent;

[0030] Action: The action of agent i∈{1,2,...,n} in time slot t is defined as Decide whether to occupy the time slot for data transmission;

[0031] Local state: local observation of agent i∈{1,2,...,n} It consists of three parts. The first part is the transmission instruction. When agent i chooses an action at time slot t After that, it will be in time slot t+2D i get Indicates successful transmission, collision and channel idle respectively. Specifically, if And the underwater sensor node is in time slot t+2D i If the ACK packet returned by the surface AP node is received, the transmission is successful. Otherwise, a collision occurs during transmission. if And the underwater sensor node is in time slot t+2D iWhen receiving the ACK packets from other underwater sensor nodes returned by the surface AP node, the data packets sent by other underwater sensor nodes are successfully received. Otherwise the channel is idle, Part 2 Indicates the time from the beginning of the time slot to the time slot t-2D i , the total number of time slots occupied by agent i and the number of time slots occupied by agents other than agent i, specifically defined as

[0032]

[0033]

[0034] The last part is the remaining energy of agent i Therefore, the action-observation pair of agent i at time slot t is Expressed as

[0035]

[0036] in, After normalization, Specifically By connecting local observations in series, we can obtain the local state

[0037]

[0038] Where M is the length of the historical state. The larger M is, the more historical information the agent can obtain. However, as M increases, the state space will also expand accordingly. Therefore, it is necessary to choose an appropriate value to improve training efficiency.

[0039] Global state: The global observation is defined as Similar to the local state, the global state at time slot t is s t =[z t-M+1 ,z t-M+2 ,...,z t ];

[0040] Reward function: The global reward at time slot t is defined as

[0041]

[0042] When the transmission is successful, a positive reward is given. Individual rewards need to consider two factors: fairness and energy consumption. Assuming that the optimal action in the system is to select the agent node with the least throughput to occupy the channel, it is defined as

[0043]

[0044] Then, by comparing the actions of agent i Whether it is consistent with the optimal action to define the fairness of individual rewards

[0045]

[0046] The energy consumption of individual rewards can be directly obtained from local observations. In this way, the two can be combined to obtain the complete individual reward function

[0047]

[0048] Among them, λ is the weight coefficient used to adjust the trade-off between throughput fairness and energy consumption. Finally, the global reward and individual reward are combined together, and the reward at time slot t is expressed as

[0049]

[0050] Preferably, the specific steps of step 5 are as follows:

[0051] The training process of the delayed reward multi-agent reinforcement learning algorithm is as follows:

[0052] 5.1 Initialize each agent's policy network (Actor network) and value function network (Critic network). The policy network is used to select the agent's actions, while the value function network is used to estimate the value of each state;

[0053] 5.2 Each agent collects experience data through interaction with the environment. These data include state, action, reward, next state and its timestamp. Due to the long propagation delay in the underwater environment, the agent must wait twice the propagation delay to obtain reward feedback. During the delay, the agent caches the state and action information so that it can be jointly evaluated with the correct reward after the delay ends.

[0054]

[0055] 5.3 A centralized critic and distributed actor training approach is adopted. The critic network is responsible for evaluating the global state and joint actions, and performs weighted processing based on the propagation delay of different agents to ensure correct policy updates under the delayed reward mechanism. In the actor network, agents independently select actions based on local observations and delayed reward information, while sharing global information for policy optimization.

[0056] 5.4 At each time step, use the Critic network to calculate the advantage function of each agent

[0057] A π (s t,a t )=Q π (s i ,a t )-V π (s t ) (11)

[0058] Q π (s t ,a t )=E(G t |s t ,a t ) is the value function, V π (s t )=E(G t |s t )=∑ a π(a|s t )Q π (s t ,a) is the state value function;

[0059] 5.5 Use stochastic gradient descent to minimize the loss function to update the Actor network, maximize the advantage function while limiting the magnitude of the policy update to avoid large policy updates. The target loss function is as follows

[0060] L(θ)=E[min(r t (θ)A t ,clip(r t (θ),1-ε,1+ε)A t )] (12)

[0061] Among them, clip is the clipping function, which controls the change between the new and old strategies in the range of [1-ε,1+ε]. is the strategy update ratio, which indicates that the new strategy and the old strategy are updated at the same observation o t Next, select joint action a t ratio;

[0062] 5.6 Repeat steps 5.2 to 5.5 until the maximum number of iterations is reached.

[0063] The beneficial effects of the present invention are as follows:

[0064] This invention significantly improves the system throughput and energy efficiency of underwater acoustic communication networks by employing a multi-agent reinforcement learning method based on multi-objective optimization, while effectively addressing the issue of unfair resource allocation between nodes. By introducing a delayed reward mechanism, agents can accurately update their strategies in environments with long propagation delays, improving learning efficiency. This invention not only improves the overall communication efficiency of the network but also enhances the system's adaptability and stability in complex environments with high latency. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is the network model diagram of the underwater acoustic sensor network;

[0066] Figure 2 It is a time slot model diagram;

[0067] Figure 3 This is a diagram of the delayed reward multi-agent reinforcement learning algorithm framework. DETAILED DESCRIPTION

[0068] In order to help those skilled in the art better understand the technical solutions in this application, the technical solutions in the embodiments of this application are clearly and completely described below. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0069] Step 1: Establish a network model of the underwater acoustic sensor network;

[0070] like Figure 1 As shown in the figure, the underwater acoustic sensor network includes several underwater sensor nodes and surface AP nodes. The underwater sensor nodes use TDMA, ALOHA and MOMA-MAC protocols respectively and are randomly distributed in the underwater area. They collect sensed data from the surrounding environment and transmit it to the surface AP nodes. Each underwater sensor node sends a detection signal containing its own position to the surface AP node, and calculates the distance between itself and the AP after receiving the ACK packet from the surface AP node.

[0071] Step 2: Establish a time slot model, such as Figure 2 As shown, the distance from each underwater sensor node to the surface AP node is converted into a propagation delay in units of time slots;

[0072] Step 3: Establish a multi-agent reinforcement learning model based on a partially observable Markov decision process (POMDP). Each underwater sensor node is modeled as an agent. Each underwater sensor node can only make decisions based on its local observation data. The CTDE architecture is used for centralized training and distributed execution. The global state is evaluated through a centralized critic network, and the joint actions of multiple agents are combined to optimize the strategy.

[0073] Step 4: Construct a multi-objective optimization problem and design a dual reward mechanism based on it, including global rewards and individual rewards. The global reward is used to improve the overall throughput of the system, while the individual reward takes into account energy efficiency and fairness goals.

[0074] Step 5: The underwater sensor nodes deploy a delayed reward multi-agent reinforcement learning algorithm, which can learn to transmit and iterate in the idle time slots not used by TDMA nodes and ALOHA nodes. During the execution process, each underwater sensor node makes distributed decisions within the time slot based on the learned strategy and autonomously selects the appropriate time slot for data transmission. Through the collaboration and competition of multiple agents, the MOMA-MAC protocol can maximize the time slot utilization and effectively balance node energy efficiency and resource allocation fairness while maintaining high throughput.

[0075] Preferably, the specific steps of step 2 are as follows:

[0076] 2.1 In the time slot system, it is assumed that the data packets from different underwater sensor nodes have the same packet length, and the ACK packets from the surface AP node to different underwater sensor nodes also have the same packet length;

[0077] 2.2 Length of each time slot T slot Corresponds to the duration T of the data packet transmission prop Plus the duration T of ACK packet transmission ack and protection time t g ; That is, each time slot is divided into two parts: the first part is for data packets, and the second part is for ACK packets;

[0078] 2.3 In the first part, in each time slot, if the MAC decision of an underwater sensor node is to send, it sends a data packet to the surface AP node, and the surface AP node waits for data packets from all underwater sensor nodes; in the second part, the underwater sensor node waits for the ACK packet from the surface AP node, and if a data packet has just been received in the first part of the time slot, the surface AP node broadcasts the ACK packet to all underwater sensor nodes;

[0079] 2.4 Propagation delay refers to the time it takes for a signal to be sent and received. For an underwater sensor node i (i∈{1,2,...,N}), assume that the distance between it and the surface AP node is d i , T slot is the length of a time slot, c is the speed of underwater sound propagation, and the propagation delay T prop Calculated by the following formula:

[0080]

[0081] Preferably, the specific steps of step 4 above are as follows:

[0082] 4.1 Construct a multi-objective optimization problem. The steps are as follows:

[0083] 4.1.1 Throughput Objective: Each agent's goal is to learn idle time slots and select appropriate time slots for data transmission without colliding with TDMA, ALOHA, or other agent nodes, thereby maximizing the time slot utilization of a frame. Through collaboration between agents, the overall system throughput is improved.

[0084] 4.1.2 Fairness Objective: To ensure the long-term stability of the system, agents need to consider past resource usage when selecting time slots. Agents that have not received transmission opportunities for a long time will be given priority to prevent individual agents from monopolizing resources. This fairness can be measured by the agents' historical transmission records to ensure balanced resource allocation.

[0085] 4.1.3 Energy Efficiency Objectives: During the transmission process, the energy consumption of each agent must be controlled. When deciding whether to transmit data, the agent must make decisions based on its current remaining energy. By avoiding excessive transmission, the node's working cycle can be extended, improving the energy efficiency of the entire network.

[0086] 4.2 Design a dual reward mechanism. The steps are as follows:

[0087] 4.2.1 Global Rewards: Global rewards are used to improve the overall throughput of the system. When an underwater sensor node successfully sends a data packet and the surface AP node receives and broadcasts an ACK packet, the system will allocate a global reward to the underwater sensor node. The goal of global rewards is to encourage individual agents to improve the system's time slot utilization and overall data transmission efficiency through collaborative work.

[0088] 4.2.2 Individual Rewards: Individual rewards take into account the local decision-making effect of each agent, encouraging the agent to maximize fairness and energy efficiency when selecting time slots. Fairness rewards are adjusted based on the time slot resources each underwater sensor node has obtained in the past. Underwater sensor nodes that have not obtained transmission opportunities for a long time will be given priority to receive more rewards, thereby ensuring fairness in resource allocation. Energy efficiency rewards are proportional to the remaining energy of the underwater sensor node, incentivizing underwater sensor nodes to transmit first when there is more remaining energy, thereby extending the life cycle of underwater sensor nodes.

[0089] 4.2.3 The final reward function is composed of a global reward and an individual reward. By adjusting the weights of these two rewards, the optimal balance between throughput, energy efficiency, and fairness is achieved according to the requirements of the application scenario.

[0090] Preferably, the reward function in step 44 is set up in the following specific steps:

[0091] The settings of the agent, action, state, and reward function are as follows:

[0092] Agent: Each underwater sensor node using the MOMA-MAC protocol is an agent;

[0093] Action: The action of agent i∈{1,2,...,n} in time slot t is defined as Decide whether to occupy the time slot for data transmission;

[0094] Local state: local observation of agent i∈{1,2,...,n} It consists of three parts. The first part is the transmission instruction. When agent i chooses an action at time slot t After that, it will be in time slot t+2D i get Indicates successful transmission, collision and channel idle respectively. Specifically, if And the underwater sensor node is in time slot t+2D i If the ACK packet returned by the surface AP node is received, the transmission is successful. Otherwise, a collision occurs during transmission. if And the underwater sensor node is in time slot t+2D i When receiving the ACK packets from other underwater sensor nodes returned by the surface AP node, the data packets sent by other underwater sensor nodes are successfully received. Otherwise the channel is idle, Part 2 Indicates the time from the beginning of the time slot to the time slot t-2D i , the total number of time slots occupied by agent i and the number of time slots occupied by agents other than agent i, specifically defined as

[0095]

[0096]

[0097] The last part is the remaining energy of agent i Therefore, the action-observation pair of agent i at time slot t is Expressed as

[0098]

[0099] in, After normalization, Specifically By connecting local observations in series, we can obtain the local state

[0100]

[0101] Where M is the length of the historical state. The larger M is, the more historical information the agent can obtain. However, as M increases, the state space will also expand accordingly. Therefore, it is necessary to choose an appropriate value to improve training efficiency.

[0102] Global state: The global observation is defined as Similar to the local state, the global state at time slot t is s t =[z t-M+1 ,z t-M+2 ,...,z t ];

[0103] Reward function: The global reward at time slot t is defined as

[0104]

[0105] When the transmission is successful, a positive reward is given. Individual rewards need to consider two factors: fairness and energy consumption. Assuming that the optimal action in the system is to select the agent node with the least throughput to occupy the channel, it is defined as

[0106]

[0107] Then, by comparing the actions of agent i Whether it is consistent with the optimal action to define the fairness of individual rewards

[0108]

[0109] The energy consumption of individual rewards can be directly obtained from local observations. In this way, the two can be combined to obtain the complete individual reward function

[0110]

[0111] Among them, λ is the weight coefficient used to adjust the trade-off between throughput fairness and energy consumption. Finally, the global reward and individual reward are combined together, and the reward at time slot t is expressed as

[0112]

[0113] Preferably, the specific steps of step 5 are as follows:

[0114] like Figure 3 As shown in Figure 2, the training process of the delayed reward multi-agent reinforcement learning algorithm is as follows:

[0115] 5.1 Initialize each agent's policy network (Actor network) and value function network (Critic network). The policy network is used to select the agent's actions, while the value function network is used to estimate the value of each state;

[0116] 5.2 Each agent collects experience data through interaction with the environment. These data include state, action, reward, next state and its timestamp. Due to the long propagation delay in the underwater environment, the agent must wait twice the propagation delay to obtain reward feedback. During the delay, the agent caches the state and action information so that it can be jointly evaluated with the correct reward after the delay ends.

[0117]

[0118] 5.3 A centralized critic and distributed actor training approach is adopted. The critic network is responsible for evaluating the global state and joint actions, and performs weighted processing based on the propagation delay of different agents to ensure correct policy updates under the delayed reward mechanism. In the actor network, agents independently select actions based on local observations and delayed reward information, while sharing global information for policy optimization.

[0119] 5.4 At each time step, use the Critic network to calculate the advantage function of each agent

[0120] A π (s t ,a t )=Q π (s i ,a t )-V π (s t ) (11)

[0121] Q π (s t ,a t )=E(G t |s t ,a t ) is the value function, V π (s t )=E(G t |s t )=∑ a π(a|s t )Q π (s t ,a) is the state value function;

[0122] 5.5 Use stochastic gradient descent to minimize the loss function to update the Actor network, maximize the advantage function while limiting the magnitude of the policy update to avoid large policy updates. The target loss function is as follows

[0123] L(θ)=E[min(r t (θ)A t ,clip(r t(θ),1-ε,1+ε)A t )] (12)

[0124] Among them, clip is the clipping function, which controls the change between the new and old strategies in the range of [1-ε,1+ε]. is the strategy update ratio, which indicates that the new strategy and the old strategy are updated at the same observation o t Next, select joint action a t ratio;

[0125] 5.6 Repeat steps 5.2 to 5.5 until the maximum number of iterations is reached.

[0126] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A MAC protocol design method based on multi-objective optimization in an underwater acoustic sensor network, characterized in that: The following steps are involved: Step 1: Establish a network model for an underwater acoustic sensor network. The underwater acoustic sensor network includes several underwater sensor nodes and surface AP nodes. The underwater sensor nodes use TDMA, ALOHA, and MOMA-MAC protocols, respectively, and are randomly distributed in the underwater area. They collect sensed data from the surrounding environment and transmit it to the surface AP nodes. Each sensor node sends a detection signal containing its own position to the surface AP node, and calculates the distance between each sensor node and the surface AP node after receiving the ACK packet from the surface AP node. Step 2: Establish a time slot model and convert the distance between each underwater sensor node and the surface AP node into a propagation delay in units of time slots; Step 3: Establish a multi-agent reinforcement learning model based on a partially observable Markov decision process. Each underwater sensor node is modeled as an agent. Each underwater sensor node can only make decisions based on its local observation data. The CTDE architecture is used for centralized training and distributed execution. The global state is evaluated through a centralized critic network, and the joint actions of multiple agents are combined to optimize the strategy. Step 4: Construct a multi-objective optimization problem and design a dual reward mechanism based on it, including global rewards and individual rewards. The global reward is used to improve the overall throughput of the system, while the individual reward takes into account energy efficiency and fairness goals. Step 5: The underwater sensor nodes deploy a delayed reward multi-agent reinforcement learning algorithm, which can learn the unused idle time slots of TDMA nodes and ALOHA nodes for transmission and iteration. During the execution process, each underwater sensor node makes distributed decisions within the time slot according to the learned strategy and autonomously selects the appropriate time slot for data transmission. Through the collaboration and competition of multiple agents, the MOMA-MAC protocol can maximize the time slot utilization and effectively balance the node energy efficiency and resource allocation fairness while maintaining high throughput.

2. The MAC protocol design method based on multi-objective optimization in an underwater acoustic sensor network according to claim 1 is characterized in that: The specific steps of step 2 are as follows: 2.1 In the time slot system, it is assumed that the data packets from different underwater sensor nodes have the same packet length, and the ACK packets from the surface AP node to different underwater sensor nodes also have the same packet length; 2.2 Length of each time slot T slot Corresponds to the duration T of the data packet transmission prop Plus the duration T of ACK packet transmission ack and protection time t g ; That is, each time slot is divided into two parts: the first part is for data packets, and the second part is for ACK packets; 2.3 In the first part, in each time slot, if the MAC decision of an underwater sensor node is to send, it sends a data packet to the surface AP node, and the surface AP node waits for data packets from all underwater sensor nodes; in the second part, the underwater sensor node waits for the ACK packet from the surface AP node, and if a data packet has just been received in the first part of the time slot, the surface AP node broadcasts the ACK packet to all underwater sensor nodes; 2.4 Propagation delay refers to the time it takes for a signal to be sent and received. For an underwater sensor node i (i∈{1,2,...,N}), assume that the distance between it and the surface AP node is d i , T slot is the length of a time slot, c is the speed of underwater sound propagation, and the propagation delay T prop Calculated by the following formula:

3. The MAC protocol design method based on multi-objective optimization in an underwater acoustic sensor network according to claim 1 is characterized in that: The specific steps of step 4 are as follows: 4.1 Construct a multi-objective optimization problem. The steps are as follows: 4.1.1 Throughput Objective: Each agent's goal is to learn idle time slots and select appropriate time slots for data transmission without colliding with TDMA, ALOHA, or other agent nodes, thereby maximizing the time slot utilization of a frame. Through collaboration between agents, the overall system throughput is improved. 4.1.2 Fairness Objective: To ensure the long-term stability of the system, agents need to consider past resource usage when selecting time slots. Agents that have not received transmission opportunities for a long time will be given priority to prevent individual agents from monopolizing resources. This fairness can be measured by the agents' historical transmission records to ensure balanced resource allocation. 4.1.3 Energy Efficiency Objectives: During the transmission process, the energy consumption of each agent must be controlled. When deciding whether to transmit data, the agent must make decisions based on its current remaining energy. By avoiding excessive transmission, the node's working cycle can be extended, improving the energy efficiency of the entire network. 4.2 Design a dual reward mechanism. The steps are as follows: 4.2.1 Global Rewards: Global rewards are used to improve the overall throughput of the system. When an underwater sensor node successfully sends a data packet and the surface AP node receives and broadcasts an ACK packet, the system will allocate a global reward to the underwater sensor node. The goal of global rewards is to encourage individual agents to improve the system's time slot utilization and overall data transmission efficiency through collaborative work. 4.2.2 Individual Rewards: Individual rewards take into account the local decision-making effect of each agent, encouraging the agent to maximize fairness and energy efficiency when selecting time slots. Fairness rewards are adjusted based on the time slot resources each underwater sensor node has obtained in the past. Underwater sensor nodes that have not obtained transmission opportunities for a long time will be given priority to receive more rewards, thereby ensuring fairness in resource allocation. Energy efficiency rewards are proportional to the remaining energy of the underwater sensor node, incentivizing underwater sensor nodes to transmit first when there is more remaining energy, thereby extending the life cycle of underwater sensor nodes. 4.2.3 The final reward function is composed of a global reward and an individual reward. By adjusting the weights of these two rewards, the optimal balance between throughput, energy efficiency, and fairness is achieved according to the requirements of the application scenario.

4. The MAC protocol design method based on multi-objective optimization in an underwater acoustic sensor network according to claim 3 is characterized in that: The specific steps for setting the reward function in step 4 are as follows: The settings of the agent, action, state, and reward function are as follows: Agent: Each underwater sensor node using the MOMA-MAC protocol is an agent; Action: The action of agent i∈{1,2,...,n} in time slot t is defined as Decide whether to occupy the time slot for data transmission; Local state: local observation of agent i∈{1,2,...,n} It consists of three parts. The first part is the transmission instruction. When agent i chooses an action at time slot t After that, it will be in time slot t+2D i get Indicates successful transmission, collision and channel idle respectively. Specifically, if And the underwater sensor node is in time slot t+2D i If the ACK packet returned by the surface AP node is received, the transmission is successful. Otherwise, a collision occurs during transmission. if And the underwater sensor node is in time slot t+2D i When receiving the ACK packets from other underwater sensor nodes returned by the surface AP node, the data packets sent by other underwater sensor nodes are successfully received. Otherwise the channel is idle, Part 2 Indicates the time from the beginning of the time slot to the time slot t-2D i , the total number of time slots occupied by agent i and the number of time slots occupied by agents other than agent i, specifically defined as The last part is the remaining energy of agent i Therefore, the action-observation pair of agent i at time slot t is Expressed as in, After normalization, By connecting local observations in series, we can obtain the local state Among them, M is the length of the historical state. The larger M is, the more historical information the agent can obtain. However, as M increases, the state space will also expand accordingly. Therefore, it is necessary to choose a suitable value to improve training efficiency; Global state: The global observation is defined as Similar to the local state, the global state at time slot t is s t =[z t-M+1 ,z t-M+2 ,...,z t ]; Reward function: The global reward at time slot t is defined as When the transmission is successful, a positive reward is given. Individual rewards need to consider two factors: fairness and energy consumption. Assuming that the optimal action in the system is to select the agent node with the least throughput to occupy the channel, it is defined as Then, by comparing the actions of agent i Whether it is consistent with the optimal action to define the fairness of individual rewards The energy consumption of individual rewards can be directly obtained from local observations. In this way, the two can be combined to obtain the complete individual reward function Among them, λ is the weight coefficient used to adjust the trade-off between throughput fairness and energy consumption. Finally, the global reward and individual reward are combined together, and the reward at time slot t is expressed as 5. The MAC protocol design method based on multi-objective optimization in an underwater acoustic sensor network according to claim 1 is characterized in that: The specific steps of step 5 are as follows: The training process of the delayed reward multi-agent reinforcement learning algorithm is as follows: 5.1 Initialize the policy network and value function network of each agent. The policy network is used to select the action of the agent, while the value function network is used to estimate the value of each state. 5.2 Each agent collects experience data through interaction with the environment. These data include state, action, reward, next state and its timestamp. Due to the long propagation delay in the underwater environment, the agent must wait twice the propagation delay to obtain reward feedback. During the delay, the agent caches the state and action information so that it can be jointly evaluated with the correct reward after the delay ends. 5.3 A centralized critic and distributed actor training approach is adopted. The critic network is responsible for evaluating the global state and joint actions, and performs weighted processing based on the propagation delay of different agents to ensure correct policy updates under the delayed reward mechanism. In the actor network, agents independently select actions based on local observations and delayed reward information, while sharing global information for policy optimization. 5.4 At each time step, use the Critic network to calculate the advantage function of each agent; 5.5 Use stochastic gradient descent to minimize the loss function to update the Actor network, maximize the advantage function while limiting the magnitude of the policy update to avoid large policy updates; 5.6 Repeat steps 5.2 to 5.5 until the maximum number of iterations is reached.

Images