MAC protocol design method based on multi-objective optimization in underwater acoustic sensor network

Abstract

The invention discloses an MAC (Media Access Control) protocol design method based on multi-objective optimization in an underwater acoustic sensor network, which comprises the following specific steps of: firstly, establishing a network model of the underwater acoustic sensor network; secondly, establishing a time slot model, and converting the propagation delay of the underwater sensor node and the water surface AP node into a time slot number; on this basis, a multi-agent reinforcement learning model is established, and a centralized training and distributed execution architecture is adopted. By adopting the multi-agent reinforcement learning method based on multi-objective optimization, the system throughput and energy efficiency in the underwater acoustic communication network are remarkably improved, and meanwhile, the problem of unfair resource allocation among nodes is effectively solved. By introducing a delay reward and priority experience playback mechanism, the intelligent agent can accurately update the strategy in a long propagation delay environment, and the learning efficiency is improved. According to the invention, the overall communication efficiency of the network is improved, and the adaptability and stability of the system in a high-delay complex environment are enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of media access control (MAC) protocols in underwater acoustic sensor networks, and specifically relates to a MAC protocol design method based on multi-objective optimization in underwater acoustic sensor networks. Background Art

[0002] Underwater Acoustic Sensor Networks (UASNs) have broad application prospects in the fields of marine environment monitoring, resource exploration, military applications, etc. However, due to the particularity of the underwater environment, including the low bandwidth, high propagation delay, high bit error rate and limited energy of the channel, underwater communication networks face many challenges in designing efficient MAC protocols. Traditional MAC protocols, such as ALOHA and TDMA, cannot achieve efficient resource scheduling and node collaboration in such a complex environment, which easily leads to high conflict rate, low throughput and unfair resource allocation.

[0003] In recent years, multi-agent reinforcement learning (MARL) has been gradually applied to the design of MAC protocols as an effective method to achieve agent collaboration in complex dynamic environments. MARL can solve the conflict and collaboration problems between nodes in the network through the autonomous learning of each agent. However, the existing MARL-based MAC protocol designs mostly focus on the optimization of a single goal, such as maximizing throughput, and fail to consider energy efficiency and fairness in resource allocation at the same time. In underwater communication environments, due to the high transmission energy consumption and limited node energy, energy efficiency optimization becomes an important goal in designing efficient MAC protocols. At the same time, the fairness issue, that is, how to ensure that all nodes can obtain communication resources equally, also directly affects the long-term stability and overall performance of the network. Summary of the invention

[0004] In view of the above problems, the present invention designs a MAC protocol method based on multi-objective optimization in underwater acoustic sensor networks. This method combines the architecture of centralized training and distributed execution with a dual reward mechanism to achieve node energy efficiency optimization and fairness in resource allocation while ensuring high system throughput, thus meeting the special needs of underwater acoustic networks.

[0005] The technical solution of the present invention is as follows:

[0006] A MAC protocol design method based on multi-objective optimization in an underwater acoustic sensor network comprises the following steps:

[0007] Step 1: Establish a network model of an underwater acoustic sensor network; the underwater acoustic sensor network includes a number of underwater sensor nodes and surface AP nodes. The underwater sensor nodes use TDMA, ALOHA and MOMA-MAC protocols respectively and are randomly distributed in the underwater area. They collect sensed data from the surrounding environment and transmit it to the surface AP nodes. Each sensor node sends a detection signal containing its own position to the surface AP node, and calculates the distance between each sensor node and the surface AP node after receiving the ACK packet from the surface AP node.

[0008] Step 2: Establish a time slot model and convert the distance between each underwater sensor node and the surface AP node into a propagation delay in units of time slots;

[0009] Step 3: Establish a multi-agent reinforcement learning model based on the partially observable Markov decision process (POMDP). Each underwater sensor node is modeled as an agent. Each underwater sensor node can only make decisions based on its local observation data. The CTDE architecture is used for centralized training and distributed execution. The global state is evaluated through a centralized critic network, and the joint actions of multiple agents are combined to optimize the strategy.

[0010] Step 4: Construct a multi-objective optimization problem and design a dual reward mechanism based on it, including global reward and individual reward. The global reward is used to improve the overall throughput of the system, while the individual reward takes into account energy efficiency and fairness goals.

[0011] Step 5: The underwater sensor nodes deploy a delayed reward multi-agent reinforcement learning algorithm, which can learn the unused idle time slots of TDMA nodes and ALOHA nodes for transmission and iteration. During the execution process, each underwater sensor node makes distributed decisions within the time slot according to the learned strategy, and autonomously selects the appropriate time slot for data transmission. Through the collaboration and competition of multiple agents, the MOMA-MAC protocol can maximize the time slot utilization and effectively balance the node energy efficiency and fairness of resource allocation while maintaining high throughput.

[0012] Preferably, the specific steps of step 2 above are as follows:

[0013] 2.1 In the time slot system, it is assumed that the data packets from different underwater sensor nodes have the same packet length, and the ACK packets from the surface AP node to different underwater sensor nodes also have the same packet length;

[0014] 2.2 Length of each time slot T slot Corresponds to the duration T of the data packet transmission prop Plus the duration T of ACK packet transmission ack And the protection time t g; That is, each time slot is divided into two parts: the first part is for data packets, and the second part is for ACK packets;

[0015] 2.3 In the first part, in each time slot, if the MAC decision of an underwater sensor node is to send, a data packet is sent to the surface AP node, and the surface AP node waits for data packets from all underwater sensor nodes; in the second part, the underwater sensor node waits for the ACK packet from the surface AP node, and if a data packet has just been received in the first part of the time slot, the surface AP node broadcasts the ACK packet to all underwater sensor nodes;

[0016] 2.4 Propagation delay refers to the time taken for a signal to be sent and received. For an underwater sensor node i (i∈{1,2,...,N}), assume that the distance between it and the surface AP node is d i , T slot is the length of a time slot, c is the speed of underwater sound propagation, and the propagation delay T prop Calculated by the following formula:

[0017]

[0018] Preferably, the specific steps of step 4 above are as follows:

[0019] 4.1 Construct a multi-objective optimization problem. The steps are as follows:

[0020] 4.1.1 Throughput target: The target of each agent is to learn the idle time slots and select the appropriate time slots for data transmission without colliding with TDMA, ALOHA and other agent nodes, so as to maximize the time slot utilization of one frame. Through the collaboration between agents, the overall throughput of the system can be improved.

[0021] 4.1.2 Fairness goal: In order to ensure the long-term stability of the system, agents need to consider past resource usage when selecting time slots; agents that have not received transmission opportunities for a long time will be given priority to avoid individual agents monopolizing resources; this fairness can be measured through the historical transmission records of agents to ensure balanced allocation of resources;

[0022] 4.1.3 Energy efficiency goal: During the transmission process, the energy consumption of each intelligent agent must be controlled. When choosing whether to transmit data, the intelligent agent must make decisions based on its current remaining energy. By avoiding excessive transmission, the node working cycle can be extended and the energy efficiency of the overall network can be improved.

[0023] 4.2 Design a dual reward mechanism. The steps are as follows:

[0024] 4.2.1 Global Reward: The global reward is used to improve the overall throughput of the system. When an underwater sensor node successfully sends a data packet and the surface AP node receives and broadcasts an ACK packet, the system will allocate a global reward to the underwater sensor node. The goal of the global reward is to encourage each agent to improve the system's time slot utilization and overall data transmission efficiency through collaborative work.

[0025] 4.2.2 Individual rewards; Individual rewards take into account the local decision-making effect of each intelligent agent, and encourage the intelligent agent to maximize fairness and energy efficiency when selecting time slots. Fairness rewards will be adjusted according to the time slot resources obtained by each underwater sensor node in the past. Underwater sensor nodes that have not been able to obtain transmission opportunities for a long time will be given priority to obtain more rewards, thereby ensuring the fairness of resource allocation. Energy efficiency rewards are proportional to the remaining energy of underwater sensor nodes, which encourages underwater sensor nodes to give priority to transmission when there is more remaining energy, thereby extending the life cycle of underwater sensor nodes;

[0026] 4.2.3 The final reward function is composed of global rewards and individual rewards. By adjusting the weights of these two rewards, the optimal balance between throughput, energy efficiency and fairness is achieved according to the requirements of the application scenario.

[0027] Preferably, the specific steps of setting the reward function in step 4 above are as follows:

[0028] The settings of the agent, action, state, and reward function are as follows:

[0029] Agent: Each underwater sensor node using the MOMA-MAC protocol is an agent;

[0030] Action: The action of agent i∈{1,2,...,n} at time slot t is defined as Decide whether to occupy the time slot for data transmission;

[0031] Local state: local observation of agent i∈{1,2,...,n} It consists of three parts. The first part is the transmission instruction. When agent i chooses an action at time slot t After that, it will be in the time slot t+2D i get Respectively represent successful transmission, collision and channel idleness. Specifically, if And the underwater sensor node is in time slot t+2D i If the ACK packet returned by the surface AP node is received, the transmission is successful. Otherwise, a collision occurs during transmission. if And the underwater sensor node is in time slot t+2D iWhen receiving the ACK packets from other underwater sensor nodes returned by the surface AP node, the data packets sent by other underwater sensor nodes are successfully received. Otherwise the channel is idle. Part 2 Indicates from the beginning of the time slot to the time slot t-2D i , the total number of time slots occupied by agent i and the number of time slots occupied by agents other than agent i, specifically defined as

[0032]

[0033]

[0034] The last part is the remaining energy of agent i Therefore, the action-observation pair of agent i at time slot t is Expressed as

[0035]

[0036] in, After normalization, Specifically By concatenating local observations, we can obtain the local state

[0037]

[0038] Among them, M is the length of the historical state. The larger M is, the more historical information the agent can obtain. However, as M increases, the state space will also expand accordingly. Therefore, it is necessary to choose a suitable value to improve training efficiency.

[0039] Global state: The global observation is defined as Similar to the local state, the global state at time slot t is s t =[z t-M+1 ,z t-M+2 ,...,z t ];

[0040] Reward function: The global reward at time slot t is defined as

[0041]

[0042] When the transmission is successful, a positive reward is given. Individual rewards need to consider two factors: fairness and energy consumption. Assuming that the optimal action in the system is to select the intelligent node with the least throughput to occupy the channel, it is defined as

[0043]

[0044] Then, by comparing the actions of agent i Whether it is consistent with the optimal action to define the fairness of individual rewards

[0045]

[0046] The energy consumption of individual rewards can be directly obtained from local observations. In this way, the two can be combined to obtain the complete individual reward function

[0047]

[0048] Among them, λ is the weight coefficient, which is used to adjust the trade-off between throughput fairness and energy consumption. Finally, the global reward and individual reward are combined together, and the reward at time slot t is expressed as

[0049]

[0050] Preferably, the specific steps of step 5 above are as follows:

[0051] The training process of the delayed reward multi-agent reinforcement learning algorithm is as follows:

[0052] 5.1 Initialize the policy network (Actor network) and value function network (Critic network) of each agent. The policy network is used to select the action of the agent, while the value function network is used to estimate the value of each state;

[0053] 5.2 Each agent collects experience data through interaction with the environment. These data include state, action, reward, next state and its timestamp. Due to the long propagation delay in the underwater environment, the agent must wait twice the propagation delay to obtain reward feedback. During the delay, the agent caches the state and action information so that it can be jointly evaluated with the correct reward after the delay.

[0054]

[0055] 5.3 The training method of centralized critic and distributed actor is adopted. The critic network is responsible for the evaluation of global state and joint action, and performs weighted processing according to the propagation delay of different agents, so as to ensure the correct strategy update under the delayed reward mechanism. In the actor network, the agent independently selects actions based on local observations and delayed reward information, and shares global information for strategy optimization.

[0056] 5.4 At each time step, use the Critic network to calculate the advantage function of each agent

[0057] A π (s t,a t )=Q π (s i ,a t )-V π (s t ) (11)

[0058] Q π (s t ,a t )=E(G t |s t ,a t ) is the value function, V π (s t )=E(G t |s t )=∑ a π(a|s t )Q π (s t ,a) is the state value function;

[0059] 5.5 Use the stochastic gradient descent method to minimize the loss function to update the Actor network, maximize the advantage function while limiting the magnitude of the strategy update to avoid large-scale strategy updates. The target loss function is as follows

[0060] L(θ)=E[min(r t (θ)A t ,clip(r t (θ),1-ε,1+ε)A t )] (12)

[0061] Among them, clip is the clipping function, which controls the change between the new and old strategies in the range of [1-ε,1+ε]. is the strategy update ratio, which indicates that the new strategy and the old strategy are updated at the same observation o t Next select joint action a t ratio;

[0062] 5.6 Repeat steps 5.2 to 5.5 until the maximum number of iterations is reached.

[0063] The beneficial effects of the present invention are as follows:

[0064] The present invention significantly improves the system throughput and energy efficiency in the underwater acoustic communication network by adopting a multi-agent reinforcement learning method based on multi-objective optimization, and effectively solves the problem of unfair resource allocation between nodes. By introducing a delayed reward mechanism, the agent can accurately update the strategy in an environment with long propagation delays, thereby improving learning efficiency. The present invention not only improves the overall communication efficiency of the network, but also enhances the adaptability and stability of the system in a complex environment with high delays. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is the network model diagram of the underwater acoustic sensor network;

[0066] Figure 2 It is a time slot model diagram;

[0067] Figure 3 This is a diagram of the delayed reward multi-agent reinforcement learning algorithm framework. DETAILED DESCRIPTION

[0068] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application are clearly and completely described below. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0069] Step 1: Establish the network model of the underwater acoustic sensor network;

[0070] like Figure 1 As shown in the figure, the underwater acoustic sensor network includes several underwater sensor nodes and surface AP nodes. The underwater sensor nodes use TDMA, ALOHA and MOMA-MAC protocols respectively and are randomly distributed in the underwater area. They collect sensed data from the surrounding environment and transmit them to the surface AP nodes. Each underwater sensor node sends a detection signal containing its own position to the surface AP node, and calculates the distance between each of them and the AP after receiving the ACK packet from the surface AP node.

[0071] Step 2: Establish a time slot model, such as Figure 2 As shown, the distance from each underwater sensor node to the surface AP node is converted into a propagation delay in units of time slots;

[0072] Step 3: Establish a multi-agent reinforcement learning model based on the partially observable Markov decision process (POMDP). Each underwater sensor node is modeled as an agent. Each underwater sensor node can only make decisions based on its local observation data. The CTDE architecture is used for centralized training and distributed execution. The global state is evaluated through a centralized critic network, and the joint actions of multiple agents are combined to optimize the strategy.

[0073] Step 4: Construct a multi-objective optimization problem and design a dual reward mechanism based on it, including global reward and individual reward. The global reward is used to improve the overall throughput of the system, while the individual reward takes into account energy efficiency and fairness goals.

[0074] Step 5: The underwater sensor nodes deploy a delayed reward multi-agent reinforcement learning algorithm, which can learn the unused idle time slots of TDMA nodes and ALOHA nodes for transmission and iteration. During the execution process, each underwater sensor node makes distributed decisions within the time slot according to the learned strategy, and autonomously selects the appropriate time slot for data transmission. Through the cooperation and competition of multiple agents, the MOMA-MAC protocol can maximize the time slot utilization and effectively balance the node energy efficiency and resource allocation fairness while maintaining high throughput.

[0075] Preferably, the specific steps of step 2 are as follows:

[0076] 2.1 In the time slot system, it is assumed that the data packets from different underwater sensor nodes have the same packet length, and the ACK packets from the surface AP node to different underwater sensor nodes also have the same packet length;

[0077] 2.2 Length of each time slot T slot Corresponds to the duration T of the data packet transmission prop Plus the duration T of ACK packet transmission ack And the protection time t g ; That is, each time slot is divided into two parts: the first part is for data packets, and the second part is for ACK packets;

[0078] 2.3 In the first part, in each time slot, if the MAC decision of an underwater sensor node is to send, a data packet is sent to the surface AP node, and the surface AP node waits for data packets from all underwater sensor nodes; in the second part, the underwater sensor node waits for the ACK packet from the surface AP node, and if a data packet has just been received in the first part of the time slot, the surface AP node broadcasts the ACK packet to all underwater sensor nodes;

[0079] 2.4 Propagation delay refers to the time taken for a signal to be sent and received. For an underwater sensor node i (i∈{1,2,...,N}), assume that the distance between it and the surface AP node is d i , T slot is the length of a time slot, c is the speed of underwater sound propagation, and the propagation delay T prop Calculated by the following formula:

[0080]

[0081] Preferably, the specific steps of step 4 above are as follows:

[0082] 4.1 Construct a multi-objective optimization problem. The steps are as follows:

[0083] 4.1.1 Throughput target: The target of each agent is to learn the idle time slots and select the appropriate time slots for data transmission without colliding with TDMA, ALOHA and other agent nodes, so as to maximize the time slot utilization of one frame. Through the collaboration between agents, the overall throughput of the system can be improved.

[0084] 4.1.2 Fairness goal: In order to ensure the long-term stability of the system, agents need to consider past resource usage when selecting time slots; agents that have not received transmission opportunities for a long time will be given priority to avoid individual agents monopolizing resources; this fairness can be measured through the historical transmission records of agents to ensure balanced allocation of resources;

[0085] 4.1.3 Energy efficiency goal: During the transmission process, the energy consumption of each intelligent agent must be controlled. When choosing whether to transmit data, the intelligent agent must make decisions based on its current remaining energy. By avoiding excessive transmission, the node working cycle can be extended and the energy efficiency of the overall network can be improved.

[0086] 4.2 Design a dual reward mechanism. The steps are as follows:

[0087] 4.2.1 Global Reward: The global reward is used to improve the overall throughput of the system. When an underwater sensor node successfully sends a data packet and the surface AP node receives and broadcasts an ACK packet, the system will allocate a global reward to the underwater sensor node. The goal of the global reward is to encourage each agent to improve the system's time slot utilization and overall data transmission efficiency through collaborative work.

[0088] 4.2.2 Individual rewards; Individual rewards take into account the local decision-making effect of each intelligent agent, and encourage the intelligent agent to maximize fairness and energy efficiency when selecting time slots. Fairness rewards will be adjusted according to the time slot resources obtained by each underwater sensor node in the past. Underwater sensor nodes that have not been able to obtain transmission opportunities for a long time will be given priority to obtain more rewards, thereby ensuring the fairness of resource allocation. Energy efficiency rewards are proportional to the remaining energy of underwater sensor nodes, which encourages underwater sensor nodes to give priority to transmission when there is more remaining energy, thereby extending the life cycle of underwater sensor nodes;

[0089] 4.2.3 The final reward function is composed of global rewards and individual rewards. By adjusting the weights of these two rewards, the optimal balance between throughput, energy efficiency and fairness is achieved according to the requirements of the application scenario.

[0090] Preferably, the specific steps of setting the reward function in the above step 44 are as follows:

[0091] The settings of the agent, action, state, and reward function are as follows:

[0092] Agent: Each underwater sensor node using the MOMA-MAC protocol is an agent;

[0093] Action: The action of agent i∈{1,2,...,n} at time slot t is defined as Decide whether to occupy the time slot for data transmission;

[0094] Local state: local observation of agent i∈{1,2,...,n} It consists of three parts. The first part is the transmission instruction. When agent i chooses an action at time slot t After that, it will be in the time slot t+2D i get Respectively represent successful transmission, collision and channel idleness. Specifically, if And the underwater sensor node is in time slot t+2D i If the ACK packet returned by the surface AP node is received, the transmission is successful. Otherwise, a collision occurs during transmission. if And the underwater sensor node is in time slot t+2D i When receiving the ACK packets from other underwater sensor nodes returned by the surface AP node, the data packets sent by other underwater sensor nodes are successfully received. Otherwise the channel is idle. Part 2 Indicates from the beginning of the time slot to the time slot t-2D i , the total number of time slots occupied by agent i and the number of time slots occupied by agents other than agent i, specifically defined as

[0095]

[0096]

[0097] The last part is the remaining energy of agent i Therefore, the action-observation pair of agent i at time slot t is Expressed as

[0098]

[0099] in, After normalization, Specifically By concatenating local observations, we can obtain the local state

[0100]

[0101] Among them, M is the length of the historical state. The larger M is, the more historical information the agent can obtain. However, as M increases, the state space will also expand accordingly. Therefore, it is necessary to choose a suitable value to improve training efficiency.

[0102] Global state: The global observation is defined as Similar to the local state, the global state at time slot t is s t =[z t-M+1 ,z t-M+2 ,...,z t ];

[0103] Reward function: The global reward at time slot t is defined as

[0104]

[0105] When the transmission is successful, a positive reward is given. Individual rewards need to consider two factors: fairness and energy consumption. Assuming that the optimal action in the system is to select the intelligent node with the least throughput to occupy the channel, it is defined as

[0106]

[0107] Then, by comparing the actions of agent i Whether it is consistent with the optimal action to define the fairness of individual rewards

[0108]

[0109] The energy consumption of individual rewards can be directly obtained from local observations. In this way, the two can be combined to obtain the complete individual reward function

[0110]

[0111] Among them, λ is the weight coefficient, which is used to adjust the trade-off between throughput fairness and energy consumption. Finally, the global reward and individual reward are combined together, and the reward at time slot t is expressed as

[0112]

[0113] Preferably, the specific steps of step 5 above are as follows:

[0114] like Figure 3 As shown in Figure 2, the training process of the delayed reward multi-agent reinforcement learning algorithm is as follows:

[0115] 5.1 Initialize the policy network (Actor network) and value function network (Critic network) of each agent. The policy network is used to select the action of the agent, while the value function network is used to estimate the value of each state;

[0116] 5.2 Each agent collects experience data through interaction with the environment. These data include state, action, reward, next state and its timestamp. Due to the long propagation delay in the underwater environment, the agent must wait twice the propagation delay to obtain reward feedback. During the delay, the agent caches the state and action information so that it can be jointly evaluated with the correct reward after the delay.

[0117]

[0118] 5.3 The training method of centralized critic and distributed actor is adopted. The critic network is responsible for the evaluation of global state and joint action, and performs weighted processing according to the propagation delay of different agents, so as to ensure the correct strategy update under the delayed reward mechanism. In the actor network, the agent independently selects actions based on local observations and delayed reward information, and shares global information for strategy optimization.

[0119] 5.4 At each time step, use the Critic network to calculate the advantage function of each agent

[0120] A π (s t ,a t )=Q π (s i ,a t )-V π (s t ) (11)

[0121] Q π (s t ,a t )=E(G t |s t ,a t ) is the value function, V π (s t )=E(G t |s t )=∑ a π(a|s t )Q π (s t ,a) is the state value function;

[0122] 5.5 Use the stochastic gradient descent method to minimize the loss function to update the Actor network, maximize the advantage function while limiting the magnitude of the strategy update to avoid large-scale strategy updates. The target loss function is as follows

[0123] L(θ)=E[min(r t (θ)A t ,clip(r t(θ),1-ε,1+ε)A t )] (12)

[0124] Among them, clip is the clipping function, which controls the change between the new and old strategies in the range of [1-ε,1+ε]. is the strategy update ratio, which indicates that the new strategy and the old strategy are updated at the same observation o t Next select joint action a t ratio;

[0125] 5.6 Repeat steps 5.2 to 5.5 until the maximum number of iterations is reached.

[0126] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A MAC protocol design method based on multi-objective optimization in underwater acoustic sensor networks. It is characterized in that The following steps are involved: Step 1: Establish a network model of an underwater acoustic sensor network; the underwater acoustic sensor network includes a number of underwater sensor nodes and surface AP nodes. The underwater sensor nodes use TDMA, ALOHA and MOMA-MAC protocols respectively and are randomly distributed in the underwater area. They collect sensed data from the surrounding environment and transmit it to the surface AP nodes. Each sensor node sends a detection signal containing its own position to the surface AP node, and calculates the distance between each sensor node and the surface AP node after receiving the ACK packet from the surface AP node. Step 2: Establish a time slot model and convert the distance between each underwater sensor node and the surface AP node into a propagation delay in units of time slots; Step 3: Establish a multi-agent reinforcement learning model. Based on the partially observable Markov decision process, each underwater sensor node is modeled as an agent. Each underwater sensor node can only make decisions based on its local observation data. The CTDE architecture is used for centralized training and distributed execution. The global state is evaluated through a centralized critic network, and the joint actions of multiple agents are combined to optimize the strategy. Step 4: Construct a multi-objective optimization problem and design a dual reward mechanism based on it, including global reward and individual reward. The global reward is used to improve the overall throughput of the system, while the individual reward takes into account energy efficiency and fairness goals. Step 5: The underwater sensor nodes deploy a delayed reward multi-agent reinforcement learning algorithm, which can learn the unused idle time slots of TDMA nodes and ALOHA nodes for transmission and iteration. During the execution process, each underwater sensor node makes distributed decisions within the time slot according to the learned strategy, and autonomously selects the appropriate time slot for data transmission. Through the collaboration and competition of multiple agents, the MOMA-MAC protocol can maximize the time slot utilization and effectively balance the node energy efficiency and fairness of resource allocation while maintaining high throughput.

2. According to claim 1, the MAC protocol design method based on multi-objective optimization in the underwater acoustic sensor network, It is characterized in that The specific steps of step 2 are as follows: 2.1 In the time slot system, it is assumed that the data packets from different underwater sensor nodes have the same packet length, and the ACK packets from the surface AP node to different underwater sensor nodes also have the same packet length; 2.2 Length of each time slot T slot Corresponds to the duration T of the data packet transmission prop Plus the duration T of ACK packet transmission ack And the protection time t g ; That is, each time slot is divided into two parts: the first part is for data packets, and the second part is for ACK packets; 2.3 In the first part, in each time slot, if the MAC decision of an underwater sensor node is to send, a data packet is sent to the surface AP node, and the surface AP node waits for data packets from all underwater sensor nodes; in the second part, the underwater sensor node waits for the ACK packet from the surface AP node, and if a data packet has just been received in the first part of the time slot, the surface AP node broadcasts the ACK packet to all underwater sensor nodes; 2.4 Propagation delay refers to the time taken for a signal to be sent and received. For an underwater sensor node i (i∈{1,2,...,N}), assume that the distance between it and the surface AP node is d i , T slot is the length of a time slot, c is the speed of underwater sound propagation, and the propagation delay T prop Calculated by the following formula:

3. The MAC protocol design method based on multi-objective optimization in the underwater acoustic sensor network according to claim 1, It is characterized in that The specific steps of step 4 are as follows: 4.1 Construct a multi-objective optimization problem. The steps are as follows: 4.1.1 Throughput target: The target of each agent is to learn the idle time slots and select the appropriate time slots for data transmission without colliding with TDMA, ALOHA and other agent nodes, so as to maximize the time slot utilization of one frame. Through the collaboration between agents, the overall throughput of the system can be improved. 4.1.2 Fairness goal: In order to ensure the long-term stability of the system, agents need to consider past resource usage when selecting time slots; agents that have not received transmission opportunities for a long time will be given priority to avoid individual agents monopolizing resources; this fairness can be measured through the historical transmission records of agents to ensure balanced allocation of resources; 4.1.3 Energy efficiency goal: During the transmission process, the energy consumption of each intelligent agent must be controlled. When choosing whether to transmit data, the intelligent agent must make decisions based on its current remaining energy. By avoiding excessive transmission, the node working cycle can be extended and the energy efficiency of the overall network can be improved. 4.2 Design a dual reward mechanism. The steps are as follows: 4.2.1 Global Reward: The global reward is used to improve the overall throughput of the system. When an underwater sensor node successfully sends a data packet and the surface AP node receives and broadcasts an ACK packet, the system will allocate a global reward to the underwater sensor node. The goal of the global reward is to encourage each agent to improve the system's time slot utilization and overall data transmission efficiency through collaborative work. 4.2.2 Individual rewards; Individual rewards take into account the local decision-making effect of each intelligent agent, and encourage the intelligent agent to maximize fairness and energy efficiency when selecting time slots. Fairness rewards will be adjusted according to the time slot resources obtained by each underwater sensor node in the past. Underwater sensor nodes that have not been able to obtain transmission opportunities for a long time will be given priority to obtain more rewards, thereby ensuring the fairness of resource allocation. Energy efficiency rewards are proportional to the remaining energy of underwater sensor nodes, which encourages underwater sensor nodes to give priority to transmission when there is more remaining energy, thereby extending the life cycle of underwater sensor nodes; 4.2.3 The final reward function is composed of global rewards and individual rewards. By adjusting the weights of these two rewards, the optimal balance between throughput, energy efficiency and fairness is achieved according to the requirements of the application scenario.

4. The MAC protocol design method based on multi-objective optimization in an underwater acoustic sensor network according to claim 3, It is characterized in that The specific steps for setting the reward function in step 4 are as follows: The settings of the agent, action, state, and reward function are as follows: Agent: Each underwater sensor node using the MOMA-MAC protocol is an agent; Action: The action of agent i∈{1,2,...,n} at time slot t is defined as Decide whether to occupy the time slot for data transmission; Local state: local observation of agent i∈{1,2,...,n} It consists of three parts. The first part is the transmission instruction. When agent i chooses an action at time slot t After that, it will be in the time slot t+2D i get Respectively represent successful transmission, collision and channel idleness. Specifically, if And the underwater sensor node is in time slot t+2D i If the ACK packet returned by the surface AP node is received, the transmission is successful. Otherwise, a collision occurs during transmission. if And the underwater sensor node is in time slot t+2D i When receiving the ACK packets from other underwater sensor nodes returned by the surface AP node, the data packets sent by other underwater sensor nodes are successfully received. Otherwise the channel is idle. Part 2 Indicates from the beginning of the time slot to the time slot t-2D i , the total number of time slots occupied by agent i and the number of time slots occupied by agents other than agent i, specifically defined as The last part is the remaining energy of agent i Therefore, the action-observation pair of agent i at time slot t is Expressed as in, After normalization, Specifically By concatenating local observations, we can obtain the local state Among them, M is the length of the historical state. The larger M is, the more historical information the agent can obtain. However, as M increases, the state space will also expand accordingly. Therefore, it is necessary to choose a suitable value to improve training efficiency. Global state: The global observation is defined as Similar to the local state, the global state at time slot t is s t =[z t-M+1 ,z t-M+2 ,...,z t ]; Reward function: The global reward at time slot t is defined as When the transmission is successful, a positive reward is given. Individual rewards need to consider two factors: fairness and energy consumption. Assuming that the optimal action in the system is to select the intelligent node with the least throughput to occupy the channel, it is defined as Then, by comparing the actions of agent i Whether it is consistent with the optimal action to define the fairness of the individual reward The energy consumption of individual rewards can be directly obtained from local observations. In this way, the two can be combined to obtain the complete individual reward function Among them, λ is the weight coefficient, which is used to adjust the trade-off between throughput fairness and energy consumption. Finally, the global reward and individual reward are combined together, and the reward at time slot t is expressed as 5. The MAC protocol design method based on multi-objective optimization in an underwater acoustic sensor network according to claim 1, It is characterized in that The specific steps of step 5 are as follows: The training process of the delayed reward multi-agent reinforcement learning algorithm is as follows: 5.1 Initialize the policy network and value function network of each agent. The policy network is used to select the action of the agent, while the value function network is used to estimate the value of each state. 5.2 Each agent collects experience data through interaction with the environment. These data include state, action, reward, next state and its timestamp. Due to the long propagation delay in the underwater environment, the agent must wait twice the propagation delay to obtain reward feedback. During the delay, the agent caches the state and action information so that it can be jointly evaluated with the correct reward after the delay. 5.3 The training method of centralized critic and distributed actor is adopted. The critic network is responsible for the evaluation of global state and joint action, and performs weighted processing according to the propagation delay of different agents, so as to ensure the correct strategy update under the delayed reward mechanism. In the actor network, the agent independently selects actions based on local observations and delayed reward information, and shares global information for strategy optimization. 5.4 At each time step, use the Critic network to calculate the advantage function of each agent A π (s t ,a t )=Q π (s i ,a t )-V π (s t ) (11) Q π (s t ,a t )=E(G t |s t ,a t ) is the value function, V π (s t )=E(G t |s t )=∑ a π(a|s t )Q π (s t ,a) is the state value function; 5.5 Use the stochastic gradient descent method to minimize the loss function to update the Actor network, maximize the advantage function while limiting the magnitude of the strategy update to avoid large-scale strategy updates. The target loss function is as follows L(θ)=E[min(r t (i)A t ,clip(r t (θ),1-ε,1+ε)A t )] (12) Among them, clip is the clipping function, which controls the change between the new and old strategies in the range of [1-ε,1+ε]. is the strategy update ratio, which indicates that the new strategy and the old strategy are updated at the same observation o t Next select joint action a t ratio; 5.6 Repeat steps 5.2 to 5.5 until the maximum number of iterations is reached.

Images