Underwater wireless sensor network resource allocation method based on multi-agent reinforcement learning

Abstract

The invention relates to an underwater wireless sensor network resource allocation method based on multi-agent reinforcement learning, and aims to improve network fairness and long-term energy efficiency by optimizing power allocation. In consideration of the dynamic characteristics of the environment and the non-convexity of the problem, firstly, the problem is modeled as a Markov process, and a multi-agent resource scheduling scheme combined with priority experience playback is introduced. According to the method, the learning process is accelerated by efficiently utilizing key experience. Besides, in order to cope with credit distribution challenges in a multi-agent scene, the invention also provides a multi-agent resource scheduling scheme based on anti-factual rewards, which can evaluate contribution of each agent to global rewards, thereby improving learning efficiency and stability.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater wireless sensor network resource allocation, and in particular to an underwater wireless sensor network resource allocation method based on multi-agent reinforcement learning. Background Art

[0002] Underwater Wireless Sensor Networks (UWSNs) have attracted widespread attention from academia and industry due to their widespread applications in environmental data collection, maritime search and rescue, and collaborative navigation. In recent years, with the introduction of mobile platforms such as autonomous underwater vehicles (AUVs) and unmanned underwater vehicles (UUVs), the monitoring capabilities of UWSNs have been significantly enhanced. However, optimizing the performance of UWSNs for efficient underwater applications remains a key issue that needs to be addressed.

[0003] Resource allocation plays a key role in optimizing the overall performance of UWSNs, including network capacity and energy consumption. In an environment with limited energy supply and dynamically changing interference, existing research focuses on improving network efficiency through power allocation strategies to achieve more concurrent communications. Existing methods have some shortcomings in dealing with the dynamics and information incompleteness of underwater environments. Reinforcement learning (RL) methods can address the shortcomings of existing methods by enabling autonomous agents to iteratively learn optimal strategies through environmental interaction. Deep reinforcement learning (DRL) extends this capability by utilizing deep neural networks as universal function approximators, effectively solving the computational bottleneck inherent in traditional Q learning in high-dimensional state spaces. In underwater wireless sensor networks, RL has been used to balance energy consumption and packet transmission delay. Wang et al. (IEEE Internet Things Journal, 7(4): 2816-2827, 2020.) proposed a distributed resource allocation algorithm based on Q learning, aiming to maximize network capacity while ensuring communication link quality. Gou et al. proposed a reinforcement learning-based method (IEEE Internet of Things Journal, 10(7): 6507-6522, 2023.) to improve the performance of UWSN. However, this study focused on time division multiplexing and spatial multiplexing under the constraint of communication fairness. Zhang et al. (IEEE Internet of Things Journal, 9(10): 7196-7210, 2022.) introduced a distributed adaptive resource management framework (DARMF) to maximize local capacity and global spatial multiplexing efficiency. However, the DARMF model has significant limitations, especially the failure to fully consider the dynamic changes of mobile nodes and communication fairness issues, which may limit its practical application in dynamic underwater environments.

[0004] While traditional power control strategy optimization methods can address energy consumption issues to a certain extent, centralized approaches suffer from significant information feedback delays, heavy computational burdens on the central controller, and high communication overhead for gathering global information. While distributed approaches are more suitable for the needs of UWSNs, traditional reinforcement learning methods struggle to cope with the complex network environments of multiple transmitters and receivers, such as signal interference and increased spectrum competition. This makes it difficult to ensure the convergence and stability of cooperative reinforcement learning algorithms. Therefore, existing methods are unable to optimize UWSNs, and achieving the optimal trade-off between network capacity and communication fairness in underwater wireless sensor networks remains an urgent problem. Summary of the Invention

[0005] To address the above shortcomings, this paper proposes a resource allocation method for underwater wireless sensor networks based on Multi-Agent Reinforcement Learning (MARL). This method simultaneously considers communication fairness, efficiency, and reliability, finds the optimal balance between network capacity and communication fairness, and improves network performance by optimizing transmission power.

[0006] The technical solution of the present invention is a method for allocating resources in an underwater wireless sensor network based on multi-agent reinforcement learning, which comprises the following steps:

[0007] S1. Deploying an underwater wireless sensor network in a three-dimensional rectangular coordinate system, wherein the underwater wireless sensor network includes N pairs of fixed-position transmitters and receivers and M mobile entities, wherein each transmitter-receiver pair is displaced by water flow, and the mobile entities move randomly to perform tasks;

[0008] S2, obtaining the total communication link capacity of each transmitter-receiver pair and the energy consumption model of the transmitter in each transmitter-receiver pair;

[0009] S3, establishing a resource allocation optimization model with the fairness index, the total communication link capacity of each transmitter-receiver pair, and the energy consumption model of the transmitter in each transmitter-receiver pair as optimization targets;

[0010] S4. Construct a multi-agent Markov decision framework, where each transmitter acts as an independent agent and defines the agent's state space, action space, state transition probability, and reward function;

[0011] S5. Under the multi-agent Markov decision framework, based on the resource allocation optimization model, a multi-agent deep learning method is used to optimize the communication power between each pair of transmitters and receivers;

[0012] The specific process of the multi-agent deep learning method is as follows:

[0013] S51. Deploy an adversarial deep Q network and a PER buffer at each transmitter, where the adversarial deep Q network includes a main deep learning Q network and a target deep learning Q network;

[0014] S52. Obtain several experience samples for each transmitter and store them in the corresponding PER buffer, prioritize the several experience samples according to the time difference error, and perform iterative adversarial training on the adversarial deep Q network based on the priority experience replay mechanism to optimize the transmitter power allocation strategy.

[0015] The beneficial effects of the present invention are as follows: the above-mentioned underwater wireless sensor network resource allocation method based on multi-agent reinforcement learning adopts a fairness index to quantify the fairness of communication transmission, and constructs a communication optimization problem, which simultaneously considers communication fairness, efficiency and reliability. This problem explores finding the optimal balance between network capacity and communication fairness, and adopts a priority experience replay-based multi-agent resource scheduling scheme to optimize the transmission power of the transmitter to improve network energy efficiency, thereby providing a theoretical basis for resource allocation in UWSN.

[0016] Preferably, in step S2, the specific process of obtaining the total communication link capacity of each transmitter-receiver pair is as follows: defining a network communication constraint condition, and calculating the total communication link capacity of the i-th transmitter-receiver pair according to the network communication constraint condition; wherein the network communication constraint condition is specifically: requiring the signal-to-noise ratio plus interference-to-noise ratio value of each transmitter-receiver pair to exceed a preset threshold value γ th , otherwise the capacity of its communication link is recorded as 0.

[0017] Preferably, the total communication link capacity of the i-th transmitter-receiver pair is specifically expressed as: Where σ represents the length of each time slot t, represents the normalized communication capacity of the i-th transmitter-receiver pair in time slot t; when When , the network communication link is considered valid; otherwise, the link capacity is recorded as 0; represents the signal-to-noise ratio plus interference-noise ratio value of the i-th transmitter-receiver pair,

[0018] Among them, p i 、p k and p m Represents transmitter n i , adjacent transmitter n k and mobile entity n m The transmission power of the variable hij 、h kj and h mj Both represent channel gain; the first two terms in the denominator of the formula represent the receiver n j From adjacent transmitter n k and mobile entity n m where i, j∈{1, 2, 3, …, N}, i≠j, N represents the total number of transmitter-receiver pairs; N(f) represents the total noise generated in the underwater channel; d represents the signal transmission distance, f represents the communication frequency; Δf represents the frequency bandwidth.

[0019] Preferably, in step S2, the energy consumption model of the transmitter in each transmitter-receiver pair is specifically expressed as follows:

[0020] in, Indicates transmitter n i The energy consumption in time slot t corresponds to its transmission power σ represents the length of each time slot t, t∈{1, 2,…, K}, T represents the time for the transmitter to perform the service task, T>0.

[0021] Preferably, in step S3, the resource allocation optimization model is specifically expressed as:

[0022] Among them, p max and p min They represent the maximum and minimum transmit power values ​​of each transmitter respectively; P represents the matrix or vector of transmit power allocation; f represents the fairness index,

[0023] Preferably, in step S4, the multi-agent Markov decision framework is defined by a tuple (J, S, A, Γ, r), where J represents the multi-agent set, S represents the network state set, A represents the action set, Γ represents the state transition probability, and r represents the reward function; each transmitter is regarded as an agent i, and at the time interval t, agent i∈J, the current state of the agent And choose to perform actions according to its strategy π When the agent performs action a t When the agent switches to the next state s according to the state transition probability t+1 .

[0024] Preferably, in step S4, the specific process of the multi-agent Markov decision framework includes the following steps:

[0025] S41. In an underwater wireless sensor network, each transmitter is treated as an independent intelligent agent;

[0026] S42. Define the state space of the agent: define the local measurement value of each transmitter Expressed as: Among them, γ i (t-1) represents the signal-to-noise ratio plus interference noise ratio value of the receiver in the previous time slot, I i (t-1) represents the interference level; the local measurement value Each transmitter collects a set of local measurements from its neighboring transmitters, which are defined as Among them, N i represents the set of neighboring transmitters of the transmitter; then the state of the transmitter in the tth time slot is and the overall status of the transmitter during service execution t Specifically expressed as: Among them, h i (t) represents the transmission link gain, p i (t-1) represents the transmission power of the transmitter in the previous time slot;

[0027] S43. Define the action space of the agent: In the tth time slot, define the i-th transmitter according to its state s i tSelect an action Set the power selection range of the i-th transmitter to [p min ,p max ], the power level is discretized as: Where Y represents the number of discrete power levels; then a i ={y i} represents the action selection of the i-th transmitter, where y i ∈{0,…,Y};

[0028] S44. Define the state transition probability of the agent: the state transition probability Γ(s'|s,a) represents the probability of the transmitter transitioning from the current state s to the next state s' when performing action a;

[0029] S45. Design a reward function based on the resource allocation optimization model. The reward function is specifically expressed as: Among them, λ f ,λ p and λ c Both represent adjustment functions; f(t) represents fairness index; C i represents the total communication link capacity of the i-th transmitter-receiver pair; p i represents the transmission power of the i-th transmitter; represents the reward function of the i-th transmitter.

[0030] Preferably, the specific process of step S52 includes the following steps:

[0031] S521, set in the tth time slot, each transmitter according to the current network state s t Select action a t , after execution, you will get an immediate reward r t , and observe the next state s t+1 ; Get several experience samples (s) for each transmitter t ,a t ,r t ,s t+1 ), construct a sample set for each transmitter;

[0032] S522: Store the sample set of each transmitter into the corresponding PER buffer, with a buffer capacity of D max ;

[0033] S523. For each transmitter, in the multi-agent Markov decision framework, based on the resource allocation optimization model, a priority experience replay mechanism is used to perform multiple iterative adversarial training on the adversarial deep Q network to optimize the transmitter power allocation strategy;

[0034] The specific process of each iterative adversarial optimization training is as follows: each experience sample (s t ,a t ,r t ,s t+1 ) dynamic weight value; according to the obtained dynamic weight value, all experience samples are prioritized, and experience samples are extracted and replayed in order of priority for training;

[0035] The time difference error is expressed as: Where ξ represents the discount factor; r represents the state s t In action a t Down-convert to s t+1 The instant reward obtained when taking action a in the current state s is obtained; Q(s,a;θ,β,μ) represents the output of the main deep learning Q network, which estimates the expected return that can be obtained by taking action a in the current state s; θ represents the weight parameter of the main deep learning Q network, μ and β represent the weight parameters of the state value network and action advantage network in the main deep learning Q network respectively; Q tar (s',a';θ - ,β - ,μ -) represents the output of the target deep learning Q network, which estimates the expected reward that can be obtained by taking action a' in the next state s'; parameter θ - Represents the set of weights of the target deep learning Q network, μ - ,β - Represent the weight parameters of the target state value network and the target action advantage network respectively; wherein, the priority sorting of all experience samples is specifically expressed as: p c =|δ t |+ε; where ε represents a positive constant; where the probability of each experience sample being selected according to the priority order is expressed as: Where D represents the capacity of the experience pool and ν represents the priority index;

[0036] Among them, the dynamic weight value w of the jth experience sample j Expressed as: Among them, η represents the smoothing parameter, which is used to adjust the smoothness of the weight; the larger the η value, the smoother the distribution of the weight, that is, the smaller the penalty for low-frequency categories; the smaller the η value, the more biased the distribution of weight is towards high-frequency categories.

[0037] In step 5, a prioritized experience replay-based multi-agent resource scheduling scheme (PER-MA-RS) is proposed, in which each transmitter is modeled as an agent. Specifically, an adversarial deep Q-network (DDQN) is deployed at each transmitter to independently determine the transmission power and exchange local information with other neighboring transmitters. The learning framework is trained and implemented in a distributed manner, demonstrating high scalability and adaptability to dynamic and complex underwater environments, and enabling efficient learning of the agents.

[0038] Preferably, after step S52, the method further includes: S53, a multi-agent resource scheduling scheme based on counterfactual rewards optimizes the power allocation of the transmitter again, specifically: calculating the counterfactual rewards of each transmitter, adjusting the parameter update direction of the main deep learning Q network according to the counterfactual rewards, completing the parameter update of the main deep learning Q network by minimizing the loss function, and regularly copying the parameters of the main deep learning Q network to the target deep learning Q network.

[0039] Preferably, in step S53, the counterfactual reward c of each transmitter is t (s,a) is specifically expressed as: in, represents the immediate reward at the i-th time slot, a t represents the cooperative action of all transmitters at time slot t; It means that the i-th transmitter sends its original action a with uniform probability t Replace with alternative action situation; represents the expected reward when the i-th transmitter takes the alternative action while the actions of other transmitters remain unchanged.

[0040] Preferably, in step S53, the specific expression for adjusting the parameter update direction of the main deep learning Q network according to the counterfactual reward is: Among them, α represents the learning rate, γ represents the discount factor, represents the counterfactual reward obtained by agent i at time slot t; represents the action of agent i at time slot t, represents the state of agent i at time slot t, and θ represents the weight parameters of the main deep learning Q network.

[0041] Preferably, in step S53, the specific expression of the minimization loss function is:

[0042]

[0043] Among them, the parameter θ - Represents the set of weights of the target deep learning Q network; represents the experience tuple, where represents the neighboring state of agent i at time slot t in the target deep network, a i' represents the possible actions of agent i. D represents the experience replay buffer. Represents the expectation of multi-agent interaction from "state-action-reward-transition".

[0044] In the above steps, a multi-agent resource scheduling scheme based on counterfactual rewards (CR-MA-RS) was designed based on a priority experience replay mechanism. Each agent treats the other agents as an environment for learning. Within the multi-agent environment, it learns how to optimize its own behavior based on counterfactual rewards. Each agent's reward calculation takes into account the behavior of other agents. This method generates counterfactual reward feedback by evaluating the impact of the agent's current behavior and its alternative behavior on the global reward. This mechanism enables agents to optimize their strategies more precisely, alleviate issues such as uneven credit distribution or lazy behavior, and achieve efficient resource scheduling and optimization, thereby improving the overall system's collaborative efficiency and network performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Schematic diagram of the underwater wireless sensor network (UWSN) scenario in the present invention;

[0046] Figure 2This is a principle block diagram of the multi-agent resource scheduling solution based on priority experience playback in the present invention;

[0047] Figure 3 This is a principle block diagram of the multi-agent resource scheduling solution based on counterfactual rewards in the present invention;

[0048] Figure 4 Schematic diagram of fairness index performance comparison under different weight coefficients in the simulation experiment of the present invention;

[0049] Figure 5 Schematic diagram of network capacity performance comparison under different weight coefficients in the simulation experiment of the present invention;

[0050] Figure 6 Schematic diagram of energy efficiency comparison under different weight coefficients in the simulation experiment of the present invention;

[0051] Figure 7 Schematic diagram showing the reward-return analysis and comparison of four transmitter-receiver pairs using the CR-MA-RS scheme in the simulation experiment of the present invention;

[0052] Figure 8 Schematic diagram showing the comparison of the convergence analysis of SINR values ​​of four transmitter-receiver pairs using the CR-MA-RS scheme in the simulation experiment of the present invention;

[0053] Figure 9 This is a comparison chart of the fairness index changes during the training period using three different schemes in the simulation experiment of the present invention;

[0054] Figure 10 Comparison of the fairness index as a function of AUV position under three different schemes in the simulation experiment of the present invention;

[0055] Figure 11 The network throughput under three different schemes in the simulation experiment of the present invention varies with the AUV position;

[0056] Figure 12 Energy efficiency changes with AUV position under three different schemes in the simulation experiment of the present invention;

[0057] Figure 13 The fairness index under three different schemes in the simulation experiment of the present invention changes with the transmitter-receiver logarithm;

[0058] Figure 14 The network throughput under three different schemes in the simulation experiment of the present invention varies with the transmitter-receiver logarithm;

[0059] Figure 15 Energy efficiency changes with transmitter-receiver logarithm under three different schemes in the simulation experiment of the present invention. DETAILED DESCRIPTION

[0060] The present invention will be described in further detail below with reference to the accompanying drawings and embodiments.

[0061] The present invention relates to a resource allocation method for underwater wireless sensor networks based on multi-agent reinforcement learning. The method is to model each transmitter as an agent and deploy a confrontation deep Q network (DDQN) at each transmitter to independently determine the transmission power and exchange local information with other neighboring transmitters, such as Figure 1 The main implementation plans are as follows:

[0062] A three-dimensional rectangular coordinate system is established as a reference coordinate system in UWSNs, and it is assumed that there are N pairs of transmitters and receivers and M mobile entities in the network; each pair of transmitters and receivers is deployed at a fixed position, but may be displaced due to water flow; the mobile entities move randomly to complete short-term tasks or data collection tasks; each transmitter n i With a specific receiver n j Pairing, where i, j ∈ {1, 2, 3, …, N}, i ≠ j, forms a transmitter-receiver pair, which facilitates efficient data transmission; the receiver's signal-to-noise ratio plus interference-to-noise ratio value is described by: Among them, p i , p k and p m Represents transmitter n i , adjacent transmitter n k and mobile entity n m The transmission power of the channel is determined by the variable h ij , h kj and h mj The first two terms in the denominator correspond to the number of receivers n. j The interference received from adjacent transmitters and mobile entities; In addition, N(f) represents the total noise generated in the underwater channel, including turbulence noise, ship noise, thermal noise, wave noise, etc. The channel gain depends on the transmission distance d and the communication frequency f, that is, Where A0 is the normalization coefficient, k is the diffusion factor, usually set to 1.5, and α(f) is the absorption coefficient; the absorption coefficient is:

[0063]

[0064] Constructing a transmitter energy consumption model: Assume that the transmitter performs service tasks within time T>0, the network operates in a time slot manner, the length of each time slot t is σ, and the time domain is defined as t∈{1, 2, …, K}, where set up Indicates transmitter n iThe energy consumption in time slot t corresponds to its transmission power Then, the total energy consumption E in time T is i Modeled as:

[0065]

[0066] Network communication constraints: To ensure that the receiver can successfully receive and decode the signal, the signal-to-noise ratio plus interference-to-noise ratio of the receiver must exceed the specified threshold γ th ; For the i-th transmitter-receiver pair in time slot t, its normalized communication capacity It can be expressed as:

[0067] in, Represents the SNR plus INR value of time slot t; when , the communication link is considered valid; otherwise, the link capacity is recorded as 0; in addition, the total link capacity of the i-th transmitter-receiver pair is It can be defined as the cumulative data transmission capacity within time T, expressed as,

[0068] Construction of UWSNs resource allocation optimization model: In order to ensure fair multiplexing among numerous nodes, meet service quality requirements and reduce energy consumption of communication nodes. In the UWSNs scenario, when multiple pairs of transmitters and receivers coexist, maximizing network throughput may lead to unfairness between these transmitters and receivers. This is because high throughput goals usually require some nodes to continuously occupy more resources or adopt higher transmission power, thereby increasing interference and resource competition pressure on other nodes. This behavior not only prevents some nodes from meeting the minimum service quality requirements, but also aggravates the imbalance in energy consumption, and ultimately reduces the overall network performance and stability. Fair transmission schemes can enhance the stability and energy efficiency of the entire network and solve the diverse needs of complex underwater applications. Based on this, the present invention establishes a fairness index model as a quantitative indicator for evaluating fairness performance. The model is: Obviously, the fairness index ranges from 0 to 1, with higher values ​​indicating fairer reuse among nodes from the perspective of the system as a whole. Fair reuse strategies improve network resource utilization while optimizing the power allocation of each transmitter to meet the QoS requirements of communication nodes. This approach ensures the stability and reliability of concurrent multi-node communication. Therefore, the UWSNs resource allocation optimization model is described as:

[0069]

[0070] Among them, p max and p minThey represent the maximum and minimum transmit power values ​​of each transmitter respectively; P represents the matrix or vector of transmit power allocation.

[0071] Constructing a multi-agent Markov decision framework: For non-convex resource allocation optimization models, it is extremely difficult to directly solve their global optimal solutions. This paper adopts multi-agent reinforcement learning (MARL) to solve the modular optimization problem and designs a distributed resource allocation algorithm based on DRL. In this system, each transmitter is regarded as an agent, which gradually learns and optimizes its power allocation strategy by interacting with the dynamic underwater communication environment. Generally, the UWSN driven by MARL can be conceptualized as a cooperative and continuous game. To this end, we propose a Markov decision framework for cooperative games, in which all agents either share a unified reward feedback or follow a common potential function, aiming to improve the overall network performance. The multi-agent Markov decision framework is defined by a tuple (J, S, A, Γ, r), where J is the multi-agent set, S represents the state set, A is the action set, Γ represents the state transition probability, and r is the reward function. At time interval t, agent i∈J, the current network state of the agent And choose actions according to its policy π When the agent performs action a t When the state is converted to the next state s according to the state transition probability Γ information t+1 , to optimize the strategy of each agent. By Γ(s t+1 |s t ,a t ) and generates reward r(s t ,a t ), as repeated state and action interaction feedback, each agent gradually improves its strategy and learns the optimal action selection strategy to maximize the expected long-term discounted reward.

[0072] In the multi-agent Markov decision framework, each element of the multi-agent Markov decision process is specifically:

[0073] Agent: In UWSN, each transmitter is considered as an agent and actively participates in the learning process;

[0074] State space: Before each transmission phase, each transmitter is allowed to exchange some information. At the beginning of each time slot, each transmitter collects local measurement data from its neighbors via a wired or wireless backhaul link. Specifically, the local measurement value of the i-th transmitter is Contains two key parameters of the previous time slot: S INR Value γ i (t-1) and interference level I i (t-1), which can be expressed as Then, the local measurement The ith transmitter collects a set of local measurements from neighboring transmitters, defined as where N i represents the set of neighboring transmitters of the i-th transmitter. Therefore, combined with the estimated transmission link gain h i (t) and the transmitter power information p in the previous time slot i (t-1), the state of the i-th transmitter in the t-th time slot can be obtained and the entire network status s t , these states are described as:

[0075]

[0076] Action space: At the tth time slot, the i-th transmitter takes action according to its observed state Select an action Assume that the power selection range of the i-th transmitter is [p min ,p max ], the power level can be discretized as,

[0077] where Y represents the number of discrete power levels; the action selection of the i-th transmitter can be expressed as a i ={y i}, where y i ∈{0,…,Y}.

[0078] Transition probability: The transition probability Γ(s'|s,a) represents the probability of transitioning from the current state s to the next state s' by performing action a.

[0079] Reward function: The design of the reward function is crucial in the learning process, as it is often used to evaluate the quality of the derived decision-making policy. Our goal is to maximize the long-term energy efficiency of the UWSN while ensuring fair resource reuse among multiple nodes and meeting the QoS requirements of user device communications. This constitutes a multi-objective optimization problem. To solve this problem, we use the weighted coefficient method to transform the multi-objective problem into a single-objective problem, defining the optimization objective as the reward function:

[0080] Since the fairness index, transmission capacity, and power are of different magnitudes, they must be normalized to a common scale. In addition, in order to take into account the relative importance of these elements in the reward function, three adjustment coefficients λ are introduced f ,λ p and λ c .

[0081] Each agent i competitively allocates the optimal power pi To maximize the total utility. This cooperative game is expressed as G = [I, P, U], where I is the user set, P is the user's power allocation strategy, and P = {p i :0≤p i ≤p i,max}where i ≥ 1 and i ∈ I. U is the total utility function, which can be expressed as:

[0082] In a multi-agent Markov decision framework, based on the resource allocation optimization model, a priority-based experience replay multi-agent resource scheduling scheme is adopted to optimize the communication power between each pair of transmitters and receivers.

[0083] Priority experience replay-based multi-agent resource scheduling scheme: This scheme is mainly aimed at dynamic and complex underwater wireless sensor networks (UWSNs) environments, where channel conditions change frequently, node mobility is high, and resource competition is fierce. This environment requires the agent to be able to efficiently utilize key experience samples to accelerate the learning process, improve the policy convergence speed, and maintain stable performance in a dynamic environment. The scheme first quantifies the importance of experience samples by time difference error (TD-error) and prioritizes the samples accordingly. Samples with higher priority are sampled and replayed preferentially during the training process, which significantly improves learning efficiency and accelerates policy convergence. In addition, in order to enhance the adaptability of the agent to dynamic environments, the scheme introduces adversarial training, which helps the agent better cope with uncertainty and interference by simulating complex dynamic environments. At the same time, a dynamic weight adjustment mechanism is used to balance the deviation in the sampling process, further ensuring the stability and robustness of learning. Through these steps, the scheme not only optimizes the utilization of experience samples, but also maintains the adaptability and stability of the agent in a dynamic environment. The implementation framework of the scheme is as follows: Figure 2 . Figure 2 This paper presents a deep learning framework based on prioritized experience replay, which includes a master deep learning Q-network, a target deep learning Q-network, a PER buffer, a multi-agent system, and an environment. The dual-network architecture of a master deep learning Q-network and a target deep learning Q-network effectively reduces bias in Q-value estimation and enhances the stability of policy learning.

[0084] The specific process of optimizing the communication power between each pair of transmitters and receivers using the priority-based experience replay multi-agent resource scheduling scheme is as follows:

[0085] Deploy an adversarial deep Q network and a PER (priority experience replay) buffer at each transmitter. The adversarial deep Q network includes a main deep learning Q network and a target deep learning Q network. Set at the tth time slot, each transmitter is based on the current network state s tSelect action a t , after execution, you will get an immediate reward r t , and observe the next state s t+1 ; Get several experience samples (s) for each transmitter t ,a t ,r t ,s t+1 ), construct a sample set for each transmitter;

[0086] Store the sample set of each transmitter into the corresponding PER buffer with a buffer capacity of D max ;

[0087] For each transmitter, under the multi-agent Markov decision framework and based on the resource allocation optimization model, its sample set is input into the adversarial deep Q network for multiple iterative adversarial optimization training to obtain an optimized transmitter;

[0088] Among them, in each iterative adversarial optimization training process, the time difference error is used to compare each experience sample (s t ,a t ,r t ,s t+1 ) to obtain a quantified result; based on the obtained quantified result, all experience samples are prioritized, and experience samples are selected in order of priority for extraction and playback for training. Specifically, after each iterative adversarial optimization training, the priority of each experience sample in the PER buffer is updated by calculating a dynamic weight value for each experience sample based on the temporal difference error.

[0089] In deep network adversarial training, the training evaluates the representation of state value and the advantage of action in a given state separately. The state-action value function Q π (s,a) represents the expected return of selecting action a in state s under policy π, and the state value V π (s) represents the value of state s, which is defined as the expected value of all action values ​​generated by policy π in that state. The difference between these two quantities defines the advantage of choosing action a in state s: A π (s,a)=Q π (s,a)-V π(s), the adversarial network contains two parallel streams, one outputting the state value V(s; θ, β) and the other outputting the action advantage A(s, a; θ, μ). Here, μ and β represent the parameters of the two streams, respectively. The output of the adversarial strategy is then expressed as: Q(s, a; θ, β) = V(s; θ, β) + A(s, a; θ, μ). Since the network directly outputs the Q value, the state value V and the action advantage A cannot be clearly determined. To solve this problem, the action advantage estimate of the selected action is constrained to zero, resulting in a modified Q value representation: In practical applications, the maximum action advantage in Q-value calculation is usually replaced by the average action advantage. This replacement can enhance optimization stability while maintaining performance.

[0090] Priority experience replay mechanism: Time delay TD reflects the environment from state s in time interval t t Transition to its successor state s t+1 The learning experience at the decision time slot t is recorded as (s t ,a t ,r,s t+1 ) is also the cth training data in the experience pool. Its priority is defined as: p c =|δ t |+ε, where ε is a small positive constant to prevent the TD error from being zero, δ t Indicates that from state s t Convert to s t+1 The TD error of time slot t is defined as: where r is the state s t In action a t Down-convert to s t+1 The instant reward obtained when θ - Represents the parameters of the target network, which are updated regularly to maintain training stability. Since priority experience replay introduces bias, dynamic weights are introduced to balance sampling, and the dynamic weight w of learning experience j in the sample set is j Defined as: Where η increases linearly from 0 to 1, and P(c) represents the probability of learning experience c being selected, which is defined as: Where D is the capacity of the experience pool, ν is the priority index, which determines the priority p c The impact on the selection probability. When ν = 0, uniform random sampling is used. Based on this, the dynamic weight w j is integrated into the loss function to balance the sampling process. The loss function is defined as: Where B is the number of samples drawn from the experience replay at each priority level.

[0091] When optimizing the efficiency of multi-agent collaboration, particularly in environments with a large number of agents and complex behaviors, a multi-agent resource scheduling scheme based on counterfactual rewards addresses the credit allocation problem in multi-agent collaboration by quantifying each agent's contribution to the global reward. By calculating the differential impact of each agent's hypothetical alternative behavior on the global reward, this optimizes the efficiency of collaboration between agents, ensuring that each agent contributes to overall network performance while optimizing its own strategy. This approach is particularly well-suited for scenarios that require a balance between fairness and efficiency, effectively preventing some agents from adopting "lazy" strategies or over-relying on the behavior of other agents, thereby improving overall collaboration efficiency and network performance. Figure 3 We present a learning framework for deep Q-networks integrated with counterfactual rewards. Figure 2 The difference is that the counterfactual reward module is further introduced.

[0092] The scheme evaluates each agent’s contribution to the global reward by calculating the differential impact on the collective reward when the transmitter deviates from the action it performed and chooses an alternative action. The counterfactual reward of multi-agent i, denoted as i, is defined as: in, where represents the immediate (global) reward at time i, a t represents the cooperative actions of all agents at time i. It means that the i-th agent takes its original action a with uniform probability t Replace with alternative action situation. represents the expected reward when the ith agent takes an alternative action while the actions of other agents remain unchanged.

[0093] After optimizing the communication power between each pair of transmitters and receivers using a priority experience replay multi-agent resource scheduling scheme, the communication power between each pair of transmitters and receivers is further optimized using a multi-agent resource scheduling method based on counterfactual rewards. The specific process includes:

[0094] For each transmitter, initializing the trained adversarial deep Q network to obtain an initialized adversarial deep Q network;

[0095] Perform multiple iterative optimization training on the initialized adversarial deep Q network;

[0096] The specific process of each iterative optimization training includes:

[0097] For each transmitter, calculate its counterfactual reward c t (s,a); specifically expressed as: in, represents the immediate reward at the i-th time slot, a t represents the cooperative action of all transmitters at time slot t; It means that the i-th transmitter sends its original action a with uniform probability t Replace with alternative action situation; represents the expected reward when the i-th transmitter takes the alternative action while the actions of other transmitters remain unchanged;

[0098] Randomly sample a batch of experience samples from the PER buffer according to priority

[0099] Using a batch of sampled experience samples, the parameters of the main deep learning Q network are updated by minimizing the loss function;

[0100] Copies the parameters of the master deep learning Q network to the target deep learning Q network.

[0101] The multi-agent DQN based on counterfactual rewards extends the traditional deep Q learning algorithm by adopting a unique reward mechanism. Its update rule is expressed as:

[0102]

[0103] Among them, α represents the learning rate, which indicates the degree of influence of the newly calculated target (the combination of counterfactual rewards and future rewards) on the original Q value when the Q value is iteratively updated. The larger α is, the greater the update amplitude of the new experience on the Q value; the smaller α is, the slower the update is and the more dependent on historical experience. γ represents the discount factor, represents the counterfactual reward obtained by agent i at time slot t; represents the action of agent i at time slot t, represents the state of agent i at time slot t, and θ represents the weight parameters of the main deep learning Q network. In addition, the loss function of the i-th agent is expressed as:

[0104]

[0105] Among them, the parameter θ - Represents the set of weights of the target deep learning Q network; represents the experience tuple, where represents the neighboring state of agent i at time slot t in the target deep network, a i' represents the possible actions of agent i; D represents the experience replay buffer. The following simulation experiments are used to illustrate the advantages of the underwater wireless sensor network resource allocation method based on multi-agent reinforcement learning.

[0106] Simulation Analysis: In the simulation, a network consisting of N transmitter-receiver pairs and one mobile entity was considered. The transmitters and receivers were randomly deployed in an area with a depth of 2 km, a length of 4 km, and a width of 4 km. The mobile entity randomly entered the area and moved along a preset path, operating at a constant power of 1 W. To account for the random non-stationary characteristics of underwater signals, the variable ψ was introduced to simulate the impact of underwater uncertainty on the acoustic channel, where It obeys the Rayleigh distribution with a mean of 0.1. Therefore, in the simulation, the underwater acoustic channel gain is defined as h+ψ.

[0107] In order to comprehensively evaluate the performance of the algorithm, three experimental scenarios were designed:

[0108] (1) The convergence of the proposed scheme is evaluated in a scenario with four transmitter-receiver pairs and one AUV;

[0109] (2) Compare the performance of the three schemes during AUV movement in the same scenario and analyze their adaptability to dynamic environments;

[0110] (3) Evaluate the performance of the three schemes under different numbers of transmit-receive pairs and study their robustness when the network scale changes.

[0111] All agents use a fully connected adversarial deep Q-network (DDQN) architecture. The adversarial deep Q-network consists of two fully connected hidden layers, containing 256 and 128 neurons, respectively, while the number of neurons in the output layer is equal to the size of the action space. The input layer and the first two hidden layers use rectified linear units (ReLU) as the activation function, while the output layer does not use any activation function. The neural network weights are updated using the Adam optimizer with a learning rate of 0.0001. The experience replay buffer has a capacity of Dmax = 20,000, and training begins when the buffer contains more than 200 samples. The batch size is set to 64, and the target Q-network is updated every 64 steps. Action selection uses the ε-greedy algorithm with ε = 0.1. Other system parameters are shown in Table 1.

[0112] Table 1 System parameter values

[0113]

[0114] The weight coefficient λ in the reward function f ,λ c ,λ p To control the emphasis on multi-objective optimization, in order to evaluate the performance of the proposed scheme in terms of fairness, network capacity and energy efficiency, three different coefficient combinations are used, e.g. Figure 4 、 Figure 5 as well as Figure 6As shown in Figure 2, the performance of fairness, network capacity and energy efficiency under different coefficient combinations are evaluated respectively. Figure 4 、 Figure 5 as well as Figure 6 It can be seen that λ f = 0.8 significantly improves fairness, while λ c =0.8 gives priority to network capacity, λ p =0.8 significantly improves energy efficiency. The optimization goal of this study is to optimize network capacity and energy efficiency while maintaining network fairness. Therefore, the weight coefficient is selected as λ f =0.8,λ c =0.1,λ p =0.1.

[0115] The proposed scheme is compared with other learning schemes, including the existing PM-MA-RS method, the CR-MA-RS method proposed in this invention, and the PER-MARS method proposed based on the CR-MA-RS method. The names of these three methods are shown in Table 2.

[0116] Table 2 Simulation method names

[0117]

[0118]

[0119] The convergence behavior of the CR-MA-RS scheme proposed in the present invention is simulated in a scenario involving four transmitter-receiver pairs and a mobile node coexisting. Figure 7 As shown in Figure 3, the average reward of all transmitters increases with the length of training slots and eventually reaches a stable state. This indicates that the scheme enables transmitters to learn the optimal power allocation strategy to achieve a balance between multiple transmit-receive pairs. Figure 8 The figure shows how the SINR value of each transmitter changes as the number of training time slots increases. Figure 7 The SINR value gradually stabilizes over time, highlighting the scheme's strong convergence characteristics. Notably, the SINR value of transmitter 4 decreases, which is due to the scheme's consideration of network fairness. By sacrificing the performance of some transmitters, the scheme ensures higher SINR values ​​for other transmitters, particularly transmitter 1. The SINR value curve for transmitter 1 shows an upward trend and stabilizes above the threshold, demonstrating that the proposed CR-MA-RS scheme effectively balances fairness, transmission capacity, and energy consumption, achieving comprehensive network optimization.

[0120] Figure 9The paper shows the evolution of the fairness index of three different schemes during training in a scenario involving four transmitter-receiver pairs and a mobile node. Despite fluctuations due to dynamic environment characteristics and policy exploration, all three schemes gradually converge to a stable level.

[0121] PER-MA-RS achieves a significantly higher fairness index than PM-MA-RS, improving by approximately 22%. This improvement is attributed to its dual-network and prioritized experience replay mechanism, which optimizes learning efficiency. Compared to PER-MA-RS, the CR-MA-RS approach exhibits a slightly higher fairness index of approximately 5% and converges significantly faster, by approximately 50%. This performance is attributed to its introduction of a credit allocation mechanism and key experience learning within the multi-agent system. In contrast, the PM-MA-RS approach performs poorly because its sender focuses solely on maximizing its own received rewards and lacks the ability to identify and learn from key experiences.

[0122] Figure 10-12 The performance of three schemes was evaluated in a simulated underwater dynamic environment with four transmitter-receiver pairs and a mobile AUV, as the AUV moved from the edge to the center. As nodes approached, the fairness index, network capacity, and energy efficiency of all three schemes showed an overall downward trend. This is primarily due to the increased interference to each receiver as the AUV approaches, coupled with the limited power adjustment range of the transmitter. Figure 10 The comparison of fairness indices is shown. At position 1 of the AUV, the CR-MA-RS method improves by 31% compared to the PM-MA-RS method and the PER-MARS method improves by 26%. However, Figure 11 It shows that under all conditions, the PM-MA-RS method always outperforms the CR-MA-RS method and the PER-MA-RS method in terms of network capacity. Figure 12 The proposed scheme demonstrates superior energy efficiency. When the AUV is located between positions 3 and 5, the energy efficiency of the three schemes becomes comparable, as all schemes increase transmission power to varying degrees to improve communication capacity in response to rising interference. These results demonstrate that the proposed multi-agent resource scheduling scheme, combined with prioritized experience replay and counterfactual rewards, effectively learns distributed control strategies and achieves collaborative optimization of multiple transmitters.

[0123] In the following simulations, we evaluate the fairness index, network throughput, and energy efficiency of the three schemes with different numbers of transmitter-receiver pairs. The results show that all methods exhibit similar performance trends as the number of transmitter-receiver pairs increases.

[0124] like Figure 13As shown in the figure, the CR-MA-RS and PER-MA-RS methods achieve superior fairness indices compared to PM-MA-RS. Specifically, when the number of transmitter-receiver pairs is low, the proposed scheme shows a fairness index close to 1, reflecting its balanced resource allocation. However, as the number of pairs increases, the fairness index decreases in an irregular pattern, mainly due to the random deployment strategy and the limited adjustment range of the transmitter. Figure 14 and Figure 15 The PM-MA-RS method outperforms the CR-MA-RS and PER-MA-RS methods in terms of network throughput, but performs worse in terms of energy efficiency. This difference stems from the fact that the PM-MA-RS method sacrifices fairness in pursuit of high network throughput. Furthermore, as the number of transmitter-receiver pairs increases, the interference experienced by each receiver increases, prompting the transmitter to increase its power level to enhance network performance.

[0125] In summary, the solution proposed in this invention utilizes prioritized experience replay and counterfactual reward mechanism to more effectively learn resource allocation strategies.

Claims

1. A resource allocation method for underwater wireless sensor networks based on multi-agent reinforcement learning, characterized by: The method comprises the following steps: S1. Deploying an underwater wireless sensor network in a three-dimensional rectangular coordinate system, wherein the underwater wireless sensor network includes N pairs of fixed-position transmitters and receivers and M mobile entities, wherein each transmitter-receiver pair is displaced by water flow, and the mobile entities move randomly to perform tasks; S2, obtaining the total communication link capacity of each transmitter-receiver pair and the energy consumption model of the transmitter in each transmitter-receiver pair; S3, establishing a resource allocation optimization model with the fairness index, the total communication link capacity of each transmitter-receiver pair, and the energy consumption model of the transmitter in each transmitter-receiver pair as optimization targets; S4. Construct a multi-agent Markov decision framework, where each transmitter acts as an independent agent and defines the agent's state space, action space, state transition probability, and reward function; S5. Under the multi-agent Markov decision framework, based on the resource allocation optimization model, a multi-agent deep learning method is used to optimize the communication power between each pair of transmitters and receivers; The specific process of the multi-agent deep learning method is as follows: S51. Deploy an adversarial deep Q network and a PER buffer at each transmitter, where the adversarial deep Q network includes a main deep learning Q network and a target deep learning Q network; S52. Obtain several experience samples for each transmitter and store them in the corresponding priority experience replay PER buffer, prioritize the several experience samples according to the time difference error, and perform iterative adversarial training on the adversarial deep Q network based on the priority experience replay mechanism to optimize the transmitter power allocation strategy.

2. The underwater wireless sensor network resource allocation method based on multi-agent reinforcement learning according to claim 1 is characterized in that: In step S2, the specific process of obtaining the total communication link capacity of each transmitter-receiver pair is as follows: defining network communication constraints, and calculating the total communication link capacity of the i-th transmitter-receiver pair according to the network communication constraints; wherein the network communication constraints are specifically: requiring the signal-to-noise ratio plus interference-to-noise ratio value of each transmitter-receiver pair to exceed a preset threshold value γ th , otherwise the capacity of its communication link is recorded as 0.

3. The underwater wireless sensor network resource allocation method based on multi-agent reinforcement learning according to claim 2 is characterized in that: The total communication link capacity of the i-th transmitter-receiver pair is specifically expressed as: Where σ represents the length of each time slot t, represents the normalized communication capacity of the i-th transmitter-receiver pair in time slot t; when When , the network communication link is considered valid; otherwise, the link capacity is recorded as 0; represents the signal-to-noise ratio plus interference-noise ratio value of the i-th transmitter-receiver pair, Among them, p i 、p k and p m Represents transmitter n i , adjacent transmitter n k and mobile entity n m The transmission power of the variable h ij 、h kj and h mj Both represent channel gain; the first two terms in the denominator of the formula represent the receiver n j From adjacent transmitter n k and mobile entity n m where i, j∈{1, 2, 3, …, N}, i≠j, N represents the total number of transmitter-receiver pairs; N(f) represents the total noise generated in the underwater channel; d represents the signal transmission distance, f represents the communication frequency; Δf represents the frequency bandwidth.

4. The underwater wireless sensor network resource allocation method based on multi-agent reinforcement learning according to claim 1, 2 or 3, characterized in that: In step S2, the energy consumption model of the transmitter in each transmitter-receiver pair is specifically expressed as: in, Indicates transmitter n i The energy consumption in time slot t corresponds to its transmission power σ represents the length of each time slot t, t∈{1, 2,…, K}, T represents the time for the transmitter to perform the service task, T>

0.

5. The underwater wireless sensor network resource allocation method based on multi-agent reinforcement learning according to claim 4 is characterized in that: In step S3, the resource allocation optimization model is specifically expressed as: Among them, p max and p min They represent the maximum and minimum transmit power values ​​of each transmitter respectively; P represents the matrix or vector of transmit power allocation; f represents the fairness index, 6. The underwater wireless sensor network resource allocation method based on multi-agent reinforcement learning according to claim 5 is characterized in that: In step S4, the specific process of constructing the multi-agent Markov decision framework includes the following steps: S41. In an underwater wireless sensor network, each transmitter is treated as an independent intelligent agent; S42. Define the state space of the agent: define the local measurement value of each transmitter Expressed as: Among them, γ i (t-1) represents the signal-to-noise ratio plus interference noise ratio value of the receiver in the previous time slot, I i (t-1) represents the interference level; the local measurement value Each transmitter collects a set of local measurements from its neighbors via the backhaul link, which is defined as Among them, N i represents the set of neighboring transmitters of the transmitter; then the state of the transmitter in the tth time slot is and the overall status of the transmitter during service execution t Specifically expressed as: Among them, h i (t) represents the transmission link gain, p i (t-1) represents the transmission power of the transmitter in the previous time slot; S43. Define the action space of the agent: In the tth time slot, define the i-th transmitter according to its state Select an action Set the power selection range of the i-th transmitter to [p min ,p max ], the power level is discretized as: Where Y represents the number of discrete power levels; then a i ={y i } represents the action selection of the i-th transmitter, where y i ∈{0,…,Y}; S44. Define the state transition probability of the agent: the state transition probability Γ(s'|s,a) represents the probability of the transmitter transitioning from the current state s to the next state s' when performing action a; S45. Design a reward function based on the resource allocation optimization model. The reward function is specifically expressed as: Among them, λ f ,λ p and λ c Both represent adjustment functions; f(t) represents fairness index; C i represents the total communication link capacity of the i-th transmitter-receiver pair; p i represents the transmission power of the i-th transmitter; represents the reward function of the i-th transmitter.

7. The underwater wireless sensor network resource allocation method based on multi-agent reinforcement learning according to claim 6 is characterized in that: The specific process of step S52 includes the following steps: S521, set in the tth time slot, each transmitter according to the current network state s t Select action a t , after execution, you will get an immediate reward r t , and observe the next state s t+1 ; Get several experience samples (s) for each transmitter t ,a t ,r t ,s t+1 ), construct a sample set for each transmitter; S522: Store the sample set of each transmitter into the corresponding PER buffer, with a buffer capacity of D max ; S523. For each transmitter, in the multi-agent Markov decision framework, based on the resource allocation optimization model, a priority experience replay mechanism is used to perform multiple iterative adversarial training on the adversarial deep Q network to optimize the transmitter power allocation strategy; The specific process of each iterative adversarial optimization training is as follows: each experience sample (s t ,a t ,r t ,s t+1 ) dynamic weight value; according to the obtained dynamic weight value, all experience samples are prioritized, and experience samples are extracted and replayed in order of priority for training; The time difference error is expressed as: Where ξ represents the discount factor; r represents the state s t In action a t Down-convert to s t+1 The instant reward obtained when taking action a in the current state s is obtained; Q(s, a; θ, β, μ) represents the output of the main deep learning Q network, which estimates the expected return that can be obtained by taking action a in the current state s; θ represents the weight parameter of the main deep learning Q network, and μ and β represent the weight parameters of the state value network and action advantage network in the main deep learning Q network respectively; Q tar (s',a';θ - ,β - ,μ - ) represents the output of the target deep learning Q network, which estimates the expected reward that can be obtained by taking action a' in the next state s'; parameter θ - Represents the set of weights of the target deep learning Q network, μ - ,β - Represent the weight parameters of the target state value network and the target action advantage network respectively; wherein, the priority sorting of all experience samples is specifically expressed as: p c =|δ t |+ε; where ε represents a positive constant; where the probability of each experience sample being selected according to the priority order is expressed as: Where D represents the capacity of the experience pool and ν represents the priority index; Among them, the dynamic weight value w of the jth experience sample j Expressed as: Here, η represents the smoothing parameter.

8. The underwater wireless sensor network resource allocation method based on multi-agent reinforcement learning according to claim 1 or 7, characterized in that: After step S52, it also includes: S53, a multi-agent resource scheduling scheme based on counterfactual rewards optimizes the power allocation of the transmitter again, specifically: calculating the counterfactual rewards of each transmitter, adjusting the parameter update direction of the main deep learning Q network according to the counterfactual rewards, completing the parameter update of the main deep learning Q network by minimizing the loss function, and regularly copying the parameters of the main deep learning Q network to the target deep learning Q network.

9. The underwater wireless sensor network resource allocation method based on multi-agent reinforcement learning according to claim 8, characterized in that: In step S53, the counterfactual reward c of each transmitter is t (s,a) is specifically expressed as: in, represents the immediate reward at the i-th time slot, a t represents the cooperative action of all transmitters at time slot t; It means that the i-th transmitter sends its original action a with uniform probability t Replace with alternative action situation; represents the expected reward when the i-th transmitter takes the alternative action while the actions of other transmitters remain unchanged.

10. The underwater wireless sensor network resource allocation method based on multi-agent reinforcement learning according to claim 9, characterized in that: In step S53, the specific expression for adjusting the parameter update direction of the main deep learning Q network according to the counterfactual reward is: Among them, α represents the learning rate, γ represents the discount factor, represents the counterfactual reward obtained by agent i at time slot t; represents the action of agent i at time slot t, represents the state of agent i at time slot t, and θ represents the weight parameters of the main deep learning Q network; The specific expression of the minimization loss function is: Among them, the parameter θ - Represents the set of weights of the target deep learning Q network; represents the experience tuple, where represents the neighboring state of agent i at time slot t in the target deep network, a i' represents the possible actions of agent i; D represents the experience replay buffer; Represents the expectation of multi-agent interaction from "state-action-reward-transition".

Images