Underwater vehicle cooperative detection method based on multi-agent reinforcement learning

Abstract

The application discloses a kind of underwater submersible cooperative detection methods based on multi-agent reinforcement learning, comprising: obtaining the original observation flow of submersible, sea current field data and sound velocity profile data, generate the time series enhancement state containing submersible kinematics estimation and underwater acoustic link delay distribution;Based on time series enhancement state, call pre-trained strategy model, output constraint multiplier vector;According to constraint multiplier vector, and combining energy upper limit constraint, signal-to-noise ratio threshold constraint and minimum safety distance constraint, execute lower optimization, generate the feasible programming solution containing path set and connectivity configuration;Based on feasible programming solution, call pre-trained critic model, account communication value assessment;According to communication value assessment and communication trigger threshold, decision whether to execute underwater acoustic communication.The application solves the technical problems of representation failure under high delay asynchronous state, reinforcement learning and complex physical constraint decoupling difficulty and low efficiency of communication resources.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of detection, in particular to an underwater vehicle cooperative detection method based on multi-agent reinforcement learning. BACKGROUND

[0002] Multi-vehicle cooperative operation can achieve wide-area, efficient and robust detection and operation in complex sea areas through distributed sensing, task parallelization and functional complementation.

[0003] Currently, research on multi-agent cooperative control has made some progress. At the decision-making algorithm level, multi-agent reinforcement learning (MARL) methods, such as the MADDPG (Multi-Agent Deep Deterministic Policy Gradient) algorithm based on the Actor-Critic (Actor-Critic) framework, are used to implement end-to-end cooperative decision-making. At the path planning level, search algorithms such as A* and D*Lite or heuristic algorithms such as genetic algorithms and particle swarm algorithms are used to generate obstacle-avoiding navigation paths. At the communication coordination level, researchers usually use periodic broadcasting, master-slave polling or simple communication protocols triggered by events (such as distance exceeding) to maintain basic state synchronization among members of the formation. These technologies have achieved some degree of cooperation among multi-agents in ideal or simplified environments.

[0004] However, when existing cooperative technologies are directly applied to real underwater environments with high dynamics and strong constraints, there are serious technical challenges. These challenges mainly focus on state representation, decision-making coupling and communication efficiency. The existing reinforcement learning method faces the risk of failure in state representation under high delay and asynchronous underwater acoustic channels. The inherent high delay (seconds) and high packet loss rate of underwater acoustic communication make the neighbor states obtained by each agent necessarily outdated and asynchronous. If the traditional MARL method directly uses these outdated states, it will lead to serious decision-making mismatch; while simple kinematic extrapolation cannot effectively represent the uncertainty accumulation caused by increasing delay, leading to a disconnection between perception and reality. The end-to-end policy network of reinforcement learning is difficult to effectively couple with complex and nonlinear underwater physical constraints (such as signal-to-noise ratio threshold based on sound speed profile calculation, energy consumption upper limit affected by sea current, connectivity maintenance of topology change). Simply relying on sparse punishment rewards to guide network learning to meet constraints often leads to low training efficiency, convergence difficulty, or makes the strategy fall into a locally optimal conservative state, and cannot generate efficient feasible planning solutions. SUMMARY

[0005] The application aims to provide an underwater vehicle cooperative detection method based on multi-agent reinforcement learning, in order to solve the above problems of the prior art.

[0006] Technical solution, an underwater vehicle cooperative detection method based on multi-agent reinforcement learning, comprising:

[0007] Obtain the raw observation stream of the underwater vehicle, the sea current field data and the sound speed profile data, generate a time series enhanced state containing the kinematic estimation of the underwater vehicle and the delay distribution of the underwater acoustic link;

[0008] Based on the time series enhanced state, call the pre-trained strategy model to output a constraint multiplier vector;

[0009] According to the constraint multiplier vector, combined with the energy upper limit constraint, the signal-to-noise ratio threshold constraint and the minimum safety distance constraint, execute the lower layer optimization to generate a feasible planning solution containing a path set and connectivity configuration;

[0010] Based on the feasible planning solution, call the pre-trained critic model to calculate the communication value assessment;

[0011] According to the communication value assessment and the communication trigger threshold, decide whether to execute underwater acoustic communication.

[0012] Beneficial effects, the training method of the application adopts a differentiable double-layer joint optimization framework, calculates the shadow price through KKT system implicit differentiation, and constructs a dual reward backflow to the upper layer strategy network. The application solves the technical problems of representation failure in high-delay asynchronous state, decoupling difficulty of reinforcement learning and complex physical constraints, and low efficiency of communication resources. BRIEF DESCRIPTION OF DRAWINGS

[0013] Fig. 1 It is a step flowchart of a kind of underwater underwater vehicle cooperative detection method based on multi-agent reinforcement learning in the embodiment of the application.

[0014] Fig. 2 It is a step flowchart of pre-trained strategy model and critic model in the embodiment of the application.

[0015] Fig. 3 It is a step flowchart of executing double-layer joint optimization in the embodiment of the application.

[0016] Fig. 4 It is a step flowchart of calculating to generate shadow price set in the embodiment of the application. DETAILED DESCRIPTION

[0017] In order to better understand the present application, the present application will be described in detail below in conjunction with specific embodiments. It should be noted that these embodiments are only exemplary implementations of the present application and should not be understood as any limitation on the protection scope of the present application. Based on the content inspired by the present embodiments, those skilled in the art can make various modifications or variations to the present application without creative labor, and these modifications or variations all fall within the protection scope of the present application.

[0018] It should be noted that the perception failure and the decision coupling difficulty jointly lead to the inefficient use of communication resources. The communication strategy based on the fixed period or the simple threshold cannot evaluate the value of information according to the dynamic demand of the task and the tightness of the constraint, leading to the redundant broadcast (waste of energy and increase of exposure risk) when the communication is not needed, and the failure to initiate the high-value communication in time before the critical decision point (such as when the constraint is about to be violated), which restricts the overall coordination performance of the system. Therefore, the following embodiments are provided.

[0019] To solve these problems, in combination with Figs. 1 to 4 The present application is specifically illustrated by the following embodiments.

[0020] In this embodiment, the present method can be applied to a multi-agent system composed of multiple underwater autonomous underwater vehicles (UUVs). The environment in which the system operates is a complex underwater environment, which is characterized by high delay, high packet loss and strong time-varying of the communication channel.

[0021] Before the present method is executed, unified environment modeling and data initialization are required. The process specifically includes: obtaining geographical and hydrological data of the task area, such as seabed topographic data, known no-fly areas, and a set of task points; obtaining sound speed profile data, which reflects the propagation speed of sound waves at different water depths (such as the thermocline), and is the basis for modeling the underwater acoustic communication channel and tracking the sound line; obtaining sea current field data, which describes the flow velocity and direction at different positions and depths, and is the key basis for kinematic extrapolation and energy consumption calculation of the autonomous underwater vehicle.

[0022] The system also needs to load platform parameters, which specifically define the physical and performance limitations of each autonomous underwater vehicle. Exemplarily, they include the propulsion power-speed curve of the autonomous underwater vehicle, the maximum turning rate, the maximum climb rate, the sonar system parameters (such as the detection distance and angle), and the communication device parameters (such as the transmission power and operating frequency).

[0023] Based on the above data, the system constructs an integrated environment-platform scene description, which establishes a unified coordinate and time reference. It should be noted that, to solve the problem of multi-agent clock synchronization difficulty caused by the lack of global positioning system under water, the present embodiment can use a high-precision atomic clock, or use a mechanism combining relative time stamp and underwater acoustic communication handshake, to establish a unified time reference.

[0024] On this basis, the system generates a delay and packet loss distribution model using platform parameters and a statistical model of the underwater acoustic channel. The delay and packet loss distribution model is a statistical description of the time-varying characteristics of the underwater acoustic link. For example, it can be a probability distribution representing the probability of a transmitted data packet experiencing a specific delay or being lost under the current sea conditions. This model is an important input for subsequent embodiments of asynchronous timing compensation and communication decision-making. The system accesses the online sensing data of each underwater vehicle to form an original observation stream, and uses the original observation stream and ocean current field data to initialize the belief state of the system. The belief state can be a data structure used to store a set of triples of each agent's historical state of other neighbor agents, corresponding timestamps, and confidence levels.

[0025] Embodiment 2 describes the specific execution (or reasoning) process of the underwater vehicle cooperative detection method based on multi-agent reinforcement learning. This process assumes that the relevant policy model and critic model have been obtained in advance through the training process of subsequent embodiments.

[0026] The main process of this embodiment is as follows: obtaining the original observation stream of the underwater vehicle, ocean current field data, and sound speed profile data, generating a time-enhanced state containing kinematic estimation of the underwater vehicle and delay distribution of the underwater acoustic link; based on the time-enhanced state, calling the pre-trained policy model to output a constraint multiplier vector; based on the constraint multiplier vector, and combining the energy upper limit, signal-to-noise ratio threshold, and minimum safety distance constraints, executing lower-level optimization to generate a feasible planning solution containing a path set and connectivity configuration; based on the feasible planning solution, calling the pre-trained critic model to calculate the communication value assessment; based on the communication value assessment and the communication trigger threshold, deciding whether to perform underwater acoustic communication.

[0027] Specifically, the following steps are included:

[0028] Step 201, obtaining the original observation stream of the underwater vehicle, ocean current field data, and sound speed profile data, generating a time-enhanced state containing kinematic estimation of the underwater vehicle and delay distribution of the underwater acoustic link.

[0029] In this embodiment, the original observation stream refers to the real-time data stream obtained from the local sensors (such as sonar, inertial navigation system) of the underwater vehicle, which contains observations of its own state and (possibly outdated) observations of neighbor underwater vehicles. Ocean current field data and sound speed profile data.

[0030] The temporal augmented state is the core output of the perception side, which is a comprehensive representation of the current environment and system state. Its generation process aims to overcome the high delay and high packet loss characteristics of the underwater acoustic channel. The generation process specifically includes: performing probabilistic extrapolation on the outdated state information of the neighbor submarine in the original observation stream, combining the ocean current field data and the submarine dynamics model, estimating the kinematic state of the neighbor at the current time (i.e., submarine kinematic estimation). Combine the generated underwater acoustic link delay distribution to encode and confidence assess observation information from different sources and different delays. Fuse multi-source information such as local real-time observations, extrapolated neighbor estimates, and delay characteristics (e.g., through temporal correlation graph aggregation) to generate a unified temporal augmented state. This state will be used as the basis for decision-making in subsequent steps.

[0031] Step 202, based on the temporal augmented state, call the pre-trained policy model, output the constraint multiplier vector.

[0032] In this embodiment, the policy model is a deep neural network (e.g., Actor network) that has learned the mapping from the temporal augmented state to the optimal action through offline training. In this execution phase, the system feeds the generated temporal augmented state into the policy model as input.

[0033] The output of the policy model includes two parts: one part is the set of submarine motion and sensing actions (such as speed, attitude, and detection mode, which is implicitly used); the other part is the constraint multiplier vector. The constraint multiplier vector is a key feature of the two-layer optimization framework of the invention, which is a numerical vector generated by a specific output layer of the policy network and corresponds to each constraint (such as energy, communication, and safety distance) in the lower-level optimization problem. It can be understood that the policy model not only learns how to act, but also learns the tightness of each physical constraint or the estimated value of the shadow price under the current state.

[0034] Step 203, according to the constraint multiplier vector, and combining the energy upper limit constraint, the signal-to-noise ratio threshold constraint, and the minimum safety distance constraint, perform lower-level optimization to generate a feasible planning solution containing a path set and connectivity configuration.

[0035] In this embodiment, the system performs deterministic constrained optimization problem solving. The objective function of this optimization can be defined as minimizing the weighted sum of energy consumption, time, or exposure risk, etc. The constraint conditions include the physical limitations that the submarine must comply with.

[0036] The energy upper bound constraint ensures that the total energy consumption of the planned path does not exceed the remaining energy budget of the underwater vehicle. The signal-to-noise ratio threshold constraint is a key constraint for underwater coordination, which requires that the planned path must enable the underwater vehicles to maintain a minimum underwater acoustic communication quality (e.g., signal-to-noise ratio SNR greater than a certain threshold) at necessary moments, which needs to be evaluated in combination with sound speed profile data to assess acoustic loss. The minimum safety distance constraint prevents collisions between underwater vehicles or with obstacles.

[0037] The constraint multiplier vector is used here to guide the lower-level optimization solver (e.g., as the initial value of the multiplier in the augmented Lagrangian method, or to adjust the weight of the constraint), so that the preferences of the upper-level strategy (i.e., the multiplier vector) can be met by the lower-level optimizer. The output of the lower-level optimization is a feasible planning solution, which is a specific execution plan, exemplarily including a set of paths (e.g., a series of waypoints or B-spline curves) and the corresponding connectivity configuration (e.g., when, where, and with whom to maintain a communication link) for each underwater vehicle in the future time domain.

[0038] At step 204, based on the feasible planning solution, a pre-trained critic model is called to calculate the communication value assessment.

[0039] In this embodiment, the system needs to decide whether to perform underwater acoustic communication at the current moment (because communication consumes energy and may expose the location). In order to make a wise decision, the system uses a critic model (a pre-trained value network, such as a critic network) to assess the value of communication.

[0040] The communication value assessment is a calculated indicator for measuring the gain of the expected future return brought by performing communication compared to keeping silent, deducting the cost of communication itself. Its calculation process specifically includes: using the critic model to estimate the future value under the two scenarios of performing communication (assuming that new information is obtained and adjusted according to the feasible planning solution) and not performing communication respectively, and subtracting the calculated cost of communication energy consumption and time delay risk from the difference between the two, to obtain the final communication value assessment score.

[0041] At step 205, whether to perform underwater acoustic communication is decided according to the communication value assessment and a preset communication trigger threshold.

[0042] In this embodiment, the system compares the obtained communication value assessment score with the communication trigger threshold.

[0043] The communication trigger threshold is a dynamic threshold. In a preferred embodiment, the threshold is associated with the output constraint multiplier vector or the resulting shadow price. For example, when the constraints (especially the connectivity constraints) are tighter, the threshold is lower, so that the system is more inclined to trigger communication. If the communication value evaluation score is higher than the communication trigger threshold, the system decides to perform underwater acoustic communication, for example, to send a state update or a coordination instruction to a specific neighbor according to the connectivity configuration in the feasible planning solution. On the contrary, if the score is lower than the threshold, the system decides to remain silent in the current period, saving energy consumption and reducing exposure risk.

[0044] In summary, the embodiment provides a complete perception-decision-communication closed-loop execution process, which utilizes a pre-trained deep reinforcement learning model and combines a double-layer optimization and an event-triggered communication mechanism to achieve efficient cooperative detection in a complex underwater environment.

[0045] Embodiment 3 describes the training method of the pre-trained policy model and the critic model. A reinforcement learning training framework based on a differentiable double-layer optimization is provided. It can be understood that the training process described in the embodiment is usually performed offline or online in a training environment (such as a high-fidelity underwater simulation platform).

[0046] Step 301, in the training environment, obtain the sea current field data and the sound speed profile data, and generate a training time sequence state.

[0047] In the embodiment, the training environment simulates the generated environmental data and simulates the operation of the multi-agent system to produce an original observation stream containing high delay, packet loss and asynchrony. The process of generating the training time sequence state is consistent in methodology, and an asynchronous time sequence compensation perception mechanism is adopted to extract a consistent and information-rich state representation, i.e., the training time sequence state, from the chaotic and asynchronous observations. The time sequence state will be used as the input of the upper-layer policy network in the subsequent double-layer optimization.

[0048] In a preferred embodiment of the present application, the specific implementation of generating the training time sequence state includes a series of steps such as probability extrapolation, delay coding, confidence calibration, and time sequence association graph aggregation.

[0049] Step 302, based on the training time sequence state, perform double-layer joint optimization to generate a training feasible planning solution, and generate a dual reward reflecting the tightness of the energy upper bound constraint, the signal-to-noise ratio threshold constraint and the minimum safety distance constraint.

[0050] The embodiment describes a double-layer optimization structure for generating high-quality actions and structured feedback signals in training. The double-layer optimization framework is differentiable, i.e., the solution (specifically, the satisfaction of the constraints) of the lower-layer optimizer can be fed back to the upper-layer policy network in the form of a gradient signal.

[0051] This step can be decomposed into the following sub-steps:

[0052] The upper policy network, which is preset by the policy model, infers the training constraint multiplier vector according to the training time sequence state. The upper policy network is the policy model (for example, the Actor network) that needs to be trained. It receives the training time sequence state as input, and the network output layer thereof is specially designed (for example, a non-negative activation function such as ReLU or Softplus is used), and outputs a set of numerical values, that is, the training constraint multiplier vector. The vector corresponds to each physical constraint (such as energy, SNR, and safety distance) in the lower optimization problem one by one, and can be understood as a learned estimation of the policy network on the tension degree of each constraint under the current state.

[0053] The lower constraint optimization uses the training constraint multiplier vector to solve the training feasible planning solution and the Lagrange multiplier. The lower constraint optimization is a deterministic, model-based optimization solver. It uses the training constraint multiplier vector output by the upper layer as a parameter (for example, as the initial value or weight of the multiplier term in the augmented Lagrangian method), and strictly solves the optimization problem (for example, the goal is to minimize energy consumption and time) according to the defined environmental model (terrain, current, and sound speed profile) and platform constraints (energy, SNR, and safety distance).

[0054] The solver outputs two results: one is the training feasible planning solution, which is a specific execution plan that meets all the hard constraints; and the other is the Lagrange multiplier, which is the actual multiplier value calculated by the optimizer when converging, reflecting the sensitivity of each constraint.

[0055] In the preferred embodiment of the present application, the lower constraint optimization is solved by using a homotopy preserving topological-continuous hybrid representation (HPX-OT-SSP) method, which is particularly suitable for path planning in complex underwater environments.

[0056] Based on the training feasible planning solution and the Lagrange multiplier, a set of shadow prices is calculated. The shadow price is the dual variable of the constraint at the optimal point, which represents the revenue of the objective function when the constraint is relaxed by one unit in economics. In this embodiment, the shadow price is the key to realizing the differentiable connection. The calculation process is as follows: using the training feasible planning solution and the Lagrange multiplier obtained when the lower optimization converges, a KKT (Karush-Kuhn-Tucker) system is assembled. By performing implicit differentiation on the KKT system with respect to the key constraints (such as energy and SNR), the sensitivity of the objective function to the constraint change, that is, the set of shadow prices, can be calculated.

[0057] In the preferred embodiment of the present application, the specific implementation of calculating the set of shadow prices includes KKT system assembly, implicit differentiation, and sensitivity solving.

[0058] According to the constraint violation degree of the shadow price set corresponding to the training feasible planning solution, a dual reward is constructed. The dual reward is a structured reward signal, which does not care whether the task is completed (for example, whether the target is detected), but cares about the degree to which the generated training feasible planning solution violates or approaches each constraint. For example, it can be constructed as a dual reward = -∑(shadow price i * Constraint violation degree i ). The dual reward will serve as an important feedback signal for subsequent training steps.

[0059] Step 303, based on the training feasible planning solution and the dual reward, an event-triggered communication simulation is performed to generate an experience sample set containing state-action pairs and dual rewards.

[0060] In this embodiment, the system uses the output to simulate communication decisions in the training environment. Specifically, the system calculates a communication value assessment based on the training feasible planning solution and the dual reward, etc. Based on the shadow price set or the upper-level training constraint multiplier vector, the communication trigger threshold is dynamically set. The system makes a decision: if the communication value assessment exceeds the communication trigger threshold, the simulation performs a communication, and records the communication overhead, delay and success rate (into the communication execution log); if it does not exceed, the simulation skips the communication.

[0061] In the preferred embodiment of the present application, the specific implementation of the event-triggered communication simulation includes the calculation of the communication value assessment and the setting mechanism of the dynamic threshold. After the simulation decision is completed, the system packages the relevant data of this round into experience samples, i.e., an experience sample set containing state-action pairs and dual rewards. The sample at least contains: the current state, the executed action, the environmental reward (such as the detection income) and the key dual reward. These samples are stored in the experience replay buffer pool.

[0062] Step 304, using the experience sample set, integrated training is performed to update the policy model and the critic model.

[0063] In this embodiment, the trainer samples batches of samples from the experience replay buffer pool. The core of the training is to use the generated dual reward to guide the update of the upper-level policy model.

[0064] The trainer constructs a fusion training target. For the update of the critic model, the temporal difference (TD) target contains not only the environmental reward, but also the dual reward. For the update of the policy model, the dual reward is incorporated into the policy gradient or advantage function.

[0065] In this way, when the lower-level optimizer reports that a certain constraint (e.g. energy) is very tight (i.e. shadow price is high, dual reward is negative), the policy gradient will guide the upper-level policy network to adjust its output (i.e. train the constraint multiplier vector) so that next time in this state, the lower-level optimizer can generate a more energy-efficient feasible plan solution. This is the closed-loop learning mechanism of the proposed differentiable bi-level optimization, which enables the policy model to learn how to indirectly control and satisfy complex, non-differentiable lower-level physical constraints.

[0066] Embodiment 4, state construction for perception-side asynchronous timing compensation, this embodiment generates a detailed implementation scheme of the timing-enhanced state step. The core technical problem of this embodiment is to solve the problem of inconsistent observation information caused by high delay, high packet loss and asynchronous communication in the underwater environment.

[0067] Step 401, according to the original observation stream, the sea current field data and the platform parameters, the outdated state of the neighbor submarine is executed to obtain the forward-looking state before training.

[0068] When the submarine (main body) receives observation information from the neighbor (for example, neighbor j), the information must be outdated, for example, the information is sent by neighbor j d seconds ago (d is the delay). The purpose of this step is to estimate the forward-looking state of neighbor j at the current time.

[0069] This step is preferably implemented in the following way:

[0070] For the outdated pose in the original observation stream, the coupling of the submarine dynamics and the sea current integral of the sea current field data is executed to obtain the extrapolated pose prediction. It is assumed that the outdated pose (position, velocity, attitude) of neighbor j at t-d time is known. The system uses the submarine dynamics model (for example, kinematic equation based on velocity, acceleration and turning rate) in the platform parameters to predict its own motion, and uses the sea current field data to calculate the sea current integral (i.e. displacement caused by sea current) from t-d to t time. Superimpose the two to obtain the extrapolated pose prediction of neighbor j at the current time t. For example, for the x-axis coordinate, its prediction formula can be linearly expressed as: x pred (t)=x(t-d)+∫ td t (v j(τ) *cos(psi j(τ) ))d τ +∫ t-d t (current x(τ) )d τ . Wherein, x(t-d) is the known outdated position, v j and psi j are the speed and heading of neighbor j (calculated by the dynamics model), current xis the component of the ocean current in the x-axis.

[0071] The process noise covariance is constructed as an increasing function of the delay and maneuver score in the delay and drop distribution and platform parameters, to obtain the extrapolation uncertainty. This step is based on the fact that the confidence of the extrapolation prediction is limited, and the larger the delay d, the less accurate the prediction; the stronger the maneuver capability of the neighbor j, the less accurate the prediction. In implementation, the system calculates the maneuver score (maneuver score ) in the platform parameters according to the maximum turning rate and the maximum acceleration, and constructs the process noise covariance matrix Q score increasing with d and maneuver proc . Exemplarily, Q proc (d) = Q0*(1 + a*d + b 2 *d score *maneuver ). Wherein Q0 is the basic noise covariance, and a and b are growth coefficients. Q proc is the extrapolation uncertainty. The extrapolation pose prediction and the extrapolation uncertainty are combined to generate the training lookahead state. The training lookahead state is a probability distribution in mathematics, for example, a Gaussian distribution, whose mean value is given by the extrapolation pose prediction, and whose covariance is given by the extrapolation uncertainty Q proc .

[0072] In step 402, the delay feature and the delay confidence are generated according to the delay and drop distribution and the platform parameters.

[0073] This embodiment is based on the fact that the neural network is difficult to directly process the original delay value (such as d = 5s), and needs an explicit confidence index. This step is preferably implemented in the following way: for the delay of each observation in the delay and drop distribution, a multi-frequency sine and cosine time encoding vector is generated to form the delay feature. Specifically, for the delay d, the system calculates a vector, for example: [sin(w1*d), cos(w1*d), sin(w2*d), cos(w2*d), …, sin(w k *d), cos(w k *d)], wherein w k is a series of different frequencies. This encoding method enables the network to better understand the relative relationship of time, i.e. multi-frequency time encoding.

[0074] The delay confidence is calibrated as a nonlinear decay function of the delay and maneuver score in the delay and drop distribution and platform parameters. The delay confidence (conf(d)) is a scalar between 0 and 1, which is used to indicate the confidence of the extrapolation prediction. According to the supplementary requirements, its preferred calculation formula is: conf(d) = exp(-k1*d-k2*d 2 *maneuver score). Where κ1 and κ2 are calibrated attenuation coefficients, d is the delay, maneuver score is the computed maneuverability score. This formula shows that the larger the delay, or the stronger the maneuverability of the neighbor, the lower the confidence of this piece of information. The delay-weighted attention weight is computed, which is determined jointly by the delay corresponding to the delay feature and the delay confidence.

[0075] When the body needs to fuse information from multiple neighbors (j, k, l...), each piece of information needs to be assigned a weight. According to the supplementary requirements, the preferred calculation formula of attention weight (a j ) is: a j =softmax over_all_neighbors (-λ d *d j )*conf(d j ). Where d j is the delay of neighbor j, conf(d j ) is the confidence of neighbor j, λ d is a learnable parameter to balance the importance of delay and confidence. The softmax ensures that the sum of all weights is 1.

[0076] Step 403, fuse the training lookahead state, delay feature, delay confidence, and original observation stream, use the time correlation graph and aggregate to generate the training time series state.

[0077] Specifically, the system constructs the data structure of the time correlation graph. The nodes of the graph at least include:

[0078] Local real-time state node: represents the current, high-confidence state of the submarine itself.

[0079] Local historical summary node: represents the state sequence of the submarine itself in the past period of time.

[0080] A number of neighbor extrapolation belief nodes: each node corresponds to a neighbor (such as j), and the node feature is the generated lookahead state (mean and covariance) and the generated delay feature.

[0081] The edges of the graph represent information flow, for example, the edge from neighbor j to the body, and the weight is the computed delay-weighted attention weight a j . After constructing the graph, the system uses graph neural network (GNN) or graph attention network (GAT) aggregation method to perform information propagation and aggregation on the graph. The final output vector after aggregation is the training time series state (or time series enhanced state in the execution phase) required by the invention, which contains all the spatio-temporal and confidence information.

[0082] Step 404, model the lookahead target evaluation of the pre-roll.

[0083] This embodiment provides the key training objective for training. This step consists in that, since there is a delay (e.g. K seconds) in underwater communication, when the critic model evaluates the value of the state at time t, it should not only look at the current reward, but also see the reward K seconds later, matching the delay of the information.

[0084] In particular, the system uses an environment rollout model (e.g. a simplified dynamics and detection model) to predict K steps forward from the current state s t to get the K steps ahead state forward state (i.e. s t +K). The system uses the target critic network and the target policy network to compute the time-difference (TD) target y td . Its preferred formula is: y td =reward t +γ*Q target (forward state , policy target (forward state )). Where reward t is the immediate reward at time t (environment reward + dual reward), γ is the discount factor, Q target and policy target are the target networks. This y td value will be used to update the critic model, which makes the training process robustly handle the communication delay by the forward-looking way.

[0085] Embodiment 5, describes a process of path optimization that preserves the decision side - lower homotopy. A homotopy preserving topological-continuous hybrid representation (HPX-OT-SSP) method, which is specially used to solve the problem of generating a set of feasible and near-optimal paths for multi-agent in underwater environment with complex obstacles (such as submarine mountain, prohibited area) and time-varying environment (such as sea current, thermocline).

[0086] The lower optimization solving process of this embodiment specifically comprises the following steps:

[0087] Step 501, a preset B-spline control point and a sound speed profile data derived sound speed profile refraction section are combined to represent an underwater path.

[0088] This step defines the mathematical representation of the path. In this embodiment, a three-dimensional underwater path is not represented by a dense sequence of waypoints, but is parameterized and defined by a set of sparse B-spline control points. This representation is smooth and naturally satisfies the maximum curvature and other maneuverability constraints of the submarine.

[0089] Further, to precisely model the underwater acoustic characteristics, the present embodiment combines B-spline with sound speed profile refraction segment (SSP refraction segment). The SSP refraction segment is the sound wave propagation path calculated from the acquired sound speed profile data by ray tracing algorithm (e.g. Bellhop). This hybrid represents the bending (refraction) of underwater acoustic wave (especially for communication) when passing through thermocline. The traditional straight path assumption will cause severe distortion in the calculation of signal-to-noise ratio (SNR) constraint. By adopting the hybrid of B-spline (macro path) + SSP refraction segment (acoustic connectivity), the present invention can more accurately evaluate and satisfy the signal-to-noise ratio threshold constraint in path optimization.

[0090] Step 502, apply optimal transport re-alignment, match parent B-spline control points, generate initial path set.

[0091] This step is performed within a genetic algorithm or similar evolutionary optimization framework to produce offspring paths. In the evolutionary process, the B-spline control points of two parent paths (Parent1, Parent2) need to be crossed. However, the number or distribution of control points of different parent paths may be different, and direct crossing will result in invalid paths.

[0092] To solve this problem, the present embodiment preferably adopts optimal transport (OT) re-alignment. Specifically, the control point sets of the two parent paths are respectively regarded as the sampling points of two probability distributions. The system defines a cost function that not only considers the Euclidean distance between control points, but also preferably takes into account the work required to move the control points from position A to position B against the sea current (calculated according to the sea current field data).

[0093] Subsequently, the system solves the optimal transport problem (e.g. calculates the Wasserstein distance) to find the transmission scheme with the minimum total cost, i.e. how to match or move the control points of Parent1 to the control point positions of Parent2. It should be noted that, in view of the high computational complexity of OT matching in path space, the number of control points of the B-spline path in the present embodiment is limited to a finite number (e.g. 10 to 20), thereby ensuring the computational feasibility of the lower-level optimization solution. Through OT re-alignment, the system can generate aligned and structurally similar parents, laying the foundation for generating a high-quality initial path set for subsequent crossing operations.

[0094] Step 503, detect anchor points in the initial path set that bypass underwater topography or cross the thermocline, form a homotopy anchor set.

[0095] This step is used to identify the key topological features of the path. Homotopy refers to the topological class of a path in the obstacle space. For example, two paths are homotopic if they can be continuously deformed (without crossing obstacles) to coincide with each other. Anchor points refer to the key points that determine the homotopy class of a path. Exemplarily, anchor points can include: a chokepoint where the path passes between two seamounts, a transiency point where the path chooses to cross the thermocline from above or below, or a turning point where the path circumvents a no-go zone. This step detects such anchor points for each path in the initial path set by geometric analysis or topological algorithms, forming a homotopy anchor set.

[0096] Step 504, only exchange path segments at the alignment of the homotopy anchor set, perform homotopy preserving crossover, generate crossover candidate paths.

[0097] Traditional path crossover (e.g. single-point crossover) can illegally combine a path segment that circumvents the left side of an obstacle with a path segment that circumvents the right side of the obstacle, resulting in a child path that crosses the obstacle, which is infeasible. The homotopy preserving crossover (HPX) mechanism of the present embodiment avoids this problem. It aligns two parent paths topologically using the homotopy anchor set. It only exchanges B-spline control point sequences (i.e. path segments) between corresponding anchor points.

[0098] For example, if both Parent1 and Parent2 circumvent obstacle A and meet at chokepoint B. HPX allows exchanging their path segments before reaching B, because this preserves homotopy equivalence. However, if Parent1 circumvents the left and Parent2 circumvents the right, then HPX prohibits exchanging between their path segments that circumvent A. In this way, HPX ensures that the newly generated crossover candidate paths are topologically similar to their parents, thus improving the efficiency and feasibility of the optimization search.

[0099] Step 505, perform feasibility domain projection on the crossover candidate paths according to constraints including energy upper bound and signal-to-noise ratio threshold, to obtain a feasible candidate path set. The feasible candidate path set is used to train feasible planning solutions for the lower-level constrained optimization.

[0100] The generated crossover candidate paths, although topologically feasible, can still violate certain constraints in the continuous domain (e.g. energy, curvature, signal-to-noise ratio). Through a feasibility domain projection operator, these slightly infeasible paths are pulled back into the feasible domain. In a preferred embodiment, the projection is sequential. The system applies a series of projection operators in a fixed order:

[0101] Curvature projection: modifies the path to satisfy the maximum turning rate constraint.

[0102] Climb rate projection: modifies the path to satisfy the maximum pitch or climb constraint.

[0103] Collision projection: modify the path to maintain a minimum safety distance from terrain or other underwater vehicles.

[0104] SNR / bandwidth projection: adjust the path (e.g. fine-tune depth or relay location) to satisfy minimum signal-to-noise ratio and bandwidth constraints.

[0105] This fixed order of projection operations (e.g. first satisfy platform maneuverability, then satisfy environmental constraints) helps to guarantee the convergence of the projection operations. After projection, the resulting set of feasible candidate paths are submitted to a lower-level optimization solver (e.g. augmented Lagrangian method) for final numerical optimization, from which the optimal training feasible plan solution is selected for use.

[0106] Embodiment 6 describes the process of decision-side-dual back-filling integrated training. This embodiment serves as a detailed implementation scheme of performing integrated training. It describes how to implement the key feedback link in the differentiable bi-level optimization, i.e. how to back-propagate the gradient signal from the lower-level optimizer to the upper-level policy network.

[0107] The training update process of this embodiment specifically includes the following steps:

[0108] Step 601, based on the training feasible plan solution and the Lagrange multiplier, assemble the KKT system.

[0109] After the lower-level optimizer converges, it outputs the training feasible plan solution x* and the Lagrange multiplier λ.

[0110] This step substitutes these results into the standard Karush-Kuhn-Tucker (KKT) optimality conditions to form a large system of equations (i.e. the KKT system). This system mathematically describes the balance relationship (including first-order optimality, feasibility, and complementary slackness conditions) that must be satisfied between the gradient of the objective function, the gradient of the constraint function, and the Lagrange multiplier λ at the optimal point x*.

[0111] Step 602, perform implicit differentiation on the KKT system with respect to key constraints including energy upper bound, signal-to-noise ratio threshold, or minimum safety distance.

[0112] If the upper-level policy network changes its output (i.e. the training constraint multiplier vector θ, which can be treated as a parameter of the lower-level optimization), the gradient dJ / dθ of the lower-level optimization solution x* and the objective function value J needs to be calculated. It is extremely difficult or impossible to directly differentiate the lower-level optimization solver (e.g. lower-level genetic algorithm or SQP). Therefore, the present invention adopts implicit differentiation techniques.

[0113] Specifically, the system derives the analytical expression of dJ / dθ using the KKT system (which implicitly defines the relationship between x*and λ and θ) and the chain rule. This procedure bypasses the derivation of the optimization process and instead derives the optimization result (KKT conditions). Note that to ensure the numerical stability of the implicit differentiation of the KKT system in non-convex optimization problems (such as path planning), the lower-level optimization of the present embodiment can be performed in a locally convex region, or a continuous relaxation or smooth approximation function can be used for non-convex constraints (such as collision avoidance). In numerical solving, a conjugate gradient or preconditioning stabilization strategy can be used to improve robustness.

[0114] Step 603, solve the sensitivity index of the key constraint to the training constraint multiplier vector, and calculate the sensitivity index as a set of shadow prices.

[0115] By performing implicit differentiation, the system calculates the gradient sensitivity, i.e. the sensitivity of the objective function J and the solution x* to the upper-level parameter θ and the lower-level constraint threshold g (e.g. dJ / dθ, dx* / dθ, dJ / dg). These sensitivity indices are calculated as a set of shadow prices. In this context, the shadow price not only refers to the dual variable of the constraint threshold g (dJ / dg) in the traditional sense, but also more generally includes the influence of the upper-level strategy parameter θ on the lower-level solution and the objective (dJ / dθ). This set constitutes the structured gradient information from the lower level back to the upper level.

[0116] Step 604, sample a batch of samples from the experience sample set.

[0117] The system randomly selects a batch of experience samples from the generated experience replay buffer.

[0118] In a preferred embodiment, in order to improve the robustness of the model to the time-varying nature of the underwater acoustic channel, time-shifted playback and distribution robust training strategies are used. That is, instead of uniform sampling, the system performs weighted or hierarchical sampling according to the generated delay and packet loss distribution model. For example, high delay or high packet loss rate sea conditions (which occur less frequently in training) are given higher sampling weights to strengthen the training model's ability to cope with such extreme situations.

[0119] Step 605, obtain the time series difference objective corresponding to the batch of samples, and combine it with the dual reward to construct a fusion training objective.

[0120] This step is used to prepare the update target of the critic network. For each sample in the batch:

[0121] The system obtains its time series difference objective (TD objective). In the preferred embodiment, the TD objective is the calculated forward-looking target evaluation y td, which already incorporates the environmental return (e.g. exploration bonus) and the discount factor γ. The system also obtains the dual reward stored in the sample. The dual reward is constructed based on the shadow price and the violation of constraints. Its preferred formula is: reward = -∑ i (shadow price_i *violation i ), or simply dual reward = -λ T *g(x*). Where λ is the Lagrange multiplier and g(x*) is the vector of constraint violations. The system combines the two (e.g. simply adding or weighted adding), and constructs the fused training target: Target Value = y td + dual reward .

[0122] Step 606, incorporate the dual reward into the pre-set policy gradient or advantage function, and update the policy model and critic model using the fused training target.

[0123] This step performs backpropagation and parameter update.

[0124] For the update of critic model: calculate the loss (e.g. L2Loss) between the current output of critic network Q(s, a) and the calculated fused training target Target Value , and update the parameters of critic network accordingly.

[0125] For the update of policy model: the system calculates the advantage function A(s, a) = Q(s, a) - V(s). The key is that, at this moment, Q(s, a) has already indirectly contained the signal of dual reward through the update of critic model. Therefore, when the dual reward is negative (indicating violation of constraints), Q(s, a) will be pulled down, and the advantage function A(s, a) will also be correspondingly reduced, so as to punish the upper policy action that leads to this violation in the policy gradient calculation. In other words, the dual reward is incorporated into the advantage function and policy gradient through the critic network as a medium.

[0126] Step 607, update the adaptive law of training constraint multiplier vector simultaneously according to the fused training target.

[0127] While updating the parameters of the policy model, the system also uses the fused training target or the dual reward itself to update the adaptive law. This adaptive law is used to adjust the mean or range of the training constraint multiplier vector output by the upper policy network. For example, if the system continuously receives negative dual rewards about the upper limit of energy, the adaptive law will adjust the policy network so that it tends to output higher training constraint multipliers corresponding to the energy constraint, so as to prompt the lower optimizer to pay attention to energy consumption earlier.

[0128] Embodiment 7 describes a process of communication-side-value-of-communication-driven event-triggered communication. This embodiment is a detailed implementation scheme of performing event-triggered communication simulation. The mechanism described in this embodiment is adopted in both the training phase (simulation) and the execution phase, and its core is to make a decision on whether to perform expensive and risky underwater acoustic communication based on the value-of-communication (VoC) evaluation and dynamic threshold.

[0129] The communication decision and simulation process of this embodiment specifically includes the following steps:

[0130] Step 701, based on the training feasible plan solution, dual reward, and critic model, calculate the communication value evaluation.

[0131] The communication value evaluation (VoC) is a forward-looking indicator used to evaluate the net benefit that can be brought in the future period of time by performing one communication (i.e., sharing information) compared to keeping silent in the current state. It can be understood that this evaluation is forward-looking, i.e., it is estimated based on the generated forward-looking state (forward state ), rather than the current (possibly outdated) state.

[0132] The specific steps of calculating the communication value evaluation preferably include:

[0133] Under the assumption of no communication, estimate the no-communication return estimate using the critic model and the training time-series state (or forward state). This step calculates E[J|no comm , forward state ], i.e., what is the expected total return (including environmental return and dual reward) in the future if you keep silent now.

[0134] Under the assumption of communication, estimate the with-communication return estimate jointly using the critic model, the training time-series state, and the potential return of the dual reward. This step calculates E[J|with comm , forward state ], i.e., what is the expected total return in the future if you perform communication now (assuming that communication is successful, state uncertainty is reduced, and the cooperative strategy is adjusted according to the generated 'feasible plan solution'). It needs to be specifically pointed out that the with-communication return estimate estimated in this step explicitly takes into account the potential return of the dual reward brought by the execution of the cooperative plan solution (for example, the connectivity problem is solved through communication, and a positive dual reward is obtained).

[0135] According to the preset underwater acoustic channel statistical model and delay and packet loss distribution, the communication energy consumption, time delay and packet loss overhead are calculated to obtain the communication cost and risk. This step calculates CommCost and Risk. For example, CommCost includes the energy consumption required for performing one underwater acoustic communication (calculated according to the transmission power and time length); Risk includes the probability loss of communication failure (packet loss) and the exposure risk of communication signal being intercepted by the enemy.

[0136] The communication value evaluation is determined by combining the communication return estimate, the no-communication return estimate and the communication cost and risk. The preferred calculation formula is: VoC = E[J|with comm , forward state ] - E[J|no comm , forward state ] - CommCost - Risk.

[0137] In step 702, the communication trigger threshold is dynamically set based on the shadow price set and the training constraint multiplier vector.

[0138] The communication trigger threshold (τ comm ) is a dynamically adjusted threshold for comparison of the calculated VoC. The communication decision should not use a fixed threshold, but should be linked to the degree of constraint tension currently faced by the system. For example, when the connectivity between multiple agents is about to be disconnected (constraint tension), the system should be more inclined to communicate (i.e. the threshold should be lowered), even if the VoC is not high.

[0139] The step of dynamically setting the communication trigger threshold preferably includes:

[0140] The shadow price related to connectivity in the shadow price set is obtained. Specifically, the system extracts the shadow price component corresponding to the signal-to-noise ratio threshold constraint or the communication bandwidth constraint from the generated shadow price set, denoted as shadow price_conn . The multiplier related to connectivity in the training constraint multiplier vector is obtained. Specifically, the system extracts the connectivity multiplier component from the training constraint multiplier vector output by the upper policy network, denoted as λ conn .

[0141] The shadow price and the multiplier are mapped to the initial communication trigger threshold, wherein the more tense the constraint reflected by the shadow price or the multiplier, the lower the initial communication trigger threshold. The preferred calculation formula is: τ comm_initial = τ base - β1*λ conn - β2*shadow price_conn . Wherein τ baseis the base threshold, β1 and β2 are positive linkage coefficients. This formula explicitly embodies the linkage mechanism: when the connectivity constraint is more stressed (i.e. λ conn or shadow price_conn value is larger), the calculated τ comm_initial threshold is lower, thus making the system more likely to trigger communication.

[0142] Smooth or hysteresis processing is applied to the initial communication trigger threshold to generate the communication trigger threshold. For example, exponential moving average (EMA) is applied to smooth τ comm_initial , or a hysteresis interval is set to avoid frequent changes of the threshold value due to transient fluctuations of the shadow price, ensuring the stability of decision-making. In an optional implementation, this step also includes extreme sea state protection. That is, in extremely poor underwater acoustic channel conditions, the system will forcibly impose a protective threshold lower limit or upper limit to avoid invalid communication attempts on a doomed link.

[0143] Step 703, when the communication value evaluation exceeds the communication trigger threshold, a communication schedule is generated and a communication simulation (or actual communication) is performed.

[0144] This step is the execution of the decision. The system compares VoC and τ comm . If VoC > τ comm , it is determined to trigger communication. The system then generates a communication schedule, which specifies the physical layer parameters (such as frequency, power, coding rate) of this communication, the reserved time slot, the destination node, and the redundancy retransmission strategy.

[0145] In a preferred implementation, this step also includes dynamic relay role selection. When the system (especially the submerged vehicle that needs long-distance communication) decides to communicate, it uses the generated topo hint , which indicates the relay candidates in the current formation. The system dynamically assigns a relay role to a certain submerged vehicle by calculating a relay utility function (for example, evaluating which submerged vehicle as a relay can maximize the total throughput or reduce the total energy consumption of the network). The decision may also include predictive link establishment, that is, reserving relays and time slots in advance for the connectivity bottleneck moment on the future path. If VoC≤τ comm , it is determined to suppress communication, and no communication action is performed in this period.

[0146] Step 704, obtain the communication execution log, and use the communication execution log, the dual reward, and the corresponding state-action pair as an experience sample set.

[0147] This step is only executed in the training phase. When the simulation performs a communication (whether triggered or not), the system records a communication execution log, which includes whether the communication is successful, the actual round-trip time (RTT), the number of retransmissions, and the energy consumption. The system packages all the key information of this round, including the state, action, reward (environment reward), dual reward, and the communication execution log of this step, into a complete experience sample, and stores it in the experience replay buffer for use.

[0148] Example 8, the calculation process of the following three examples is described.

[0149] Example 8.1: Perception side - Delay confidence (conf(d)) calculation.

[0150] This example shows the calculation of delay confidence.

[0151] Assume that the system is calibrated with attenuation coefficients: K1 = 0.1, K2 = 0.05.

[0152] Case one (low maneuverability): Submarine A receives information from neighbor B with a delay d = 5 seconds. Neighbor B is cruising at low speed, and its maneuverability score is maneuver score = 0.2. Calculate conf(d):

[0153] conf(5) = exp(-0.1 * 5 - 0.05 * 5 2 * 0.2) = exp(-0.5 - 0.05 * 25 * 0.2) = exp(-0.5 - 0.25) = exp(-0.75) ≈ 0.472.

[0154] Case two (high maneuverability): Submarine A receives information from neighbor C with a delay d = 5 seconds. Neighbor C is performing a high-speed evasion maneuver, and its maneuverability score is maneuver score = 2.0. Calculate conf(d): conf(5) = exp(-0.1 * 5 - 0.05 * 5 2 * 2.0) = exp(-0.5 - 0.05 * 25 * 2.0) = exp(-0.5 - 2.5) = exp(-3.0) ≈ 0.049.

[0155] As can be seen, even if the delay is the same, the observation confidence of the high maneuverability neighbor (case two) is much lower than that of the low maneuverability neighbor (case one), which meets the actual needs of underwater cooperative detection.

[0156] Example 8.2: Decision side - Homotopy preserving intersection (HPX) path.

[0157] Assume in a two-dimensional underwater environment, there exists a circular underwater mountain (obstacle O). Parent path 1 (P1): its B-spline control point sequence makes it go around the left side of obstacle O. Parent path 2 (P2): its B-spline control point sequence makes it go around the right side of obstacle O. Parent path 3 (P3): its B-spline control point sequence makes it also go around the left side of obstacle O, but the path is closer to O. The system detects that P1 and P3 belong to the same homotopy class (go around left), P2 belongs to another homotopy class (go around right). Allowed crossover: a crossover between P1 and P3 can be performed. For example, exchange their path segments before going around O, the resulting child path is still go around left, topologically feasible. Forbidden crossover: a crossover between P1 and P2 (or P2 and P3) is forbidden. Because forcing to exchange their path segments during going around O will result in a child path crossing obstacle O, topologically infeasible.

[0158] Example 8.3: Procedure of communication-side VoC with dynamic threshold decision.

[0159] Assume the system parameters are set as: base threshold τ base = 20, linkage coefficients β1 = 2.0, β2 = 3.0.

[0160] At a certain time t, the system makes a communication decision: the critic model estimates the no-communication return estimate E[J|no comm ]= 100. The critic model estimates the with-communication return estimate (already accounting for the dual reward gain from synergy) E[J|with comm ]= 115. The system estimates the communication cost (energy consumption) CommCost = 5. The system estimates the communication risk (packet loss, exposure) Risk = 2. Compute VoC: VoC = 115 - 100 - 5 - 2 = 8.

[0161] Scenario one (constraint relaxed): the current formation is tight, connectivity is good. The upper policy network outputs λ conn = 0.1, the lower shadow price shadow price_conn = 0.2. Compute threshold: τ comm = 20 - 2.0 * 0.1 - 3.0 * 0.2 = 20 - 0.2 - 0.6 = 19.2. Decision: VoC (8) < τ comm (19.2). Conclusion: suppress communication.

[0162] Scenario two (constraint stressed): the formation is about to disperse, connectivity is at risk of breaking. The upper policy network perceives this state and outputs a higher multiplier λ conn = 1.5. The lower optimizer reports that the connectivity constraint has become a bottleneck and computes a high shadow price shadow price_conn = 2.0. Compute threshold: τ comm= 20 - 2.0*1.5 - 3.0*2.0 = 20 - 3.0 - 6.0 = 11.0. Decision: VoC(8) < τ comm (11.0). Conclusion: Still suppress communication (though the threshold has been greatly reduced).

[0163] Scenario three (constraint tight and VoC high): Assume on the basis of scenario two, the system predicts that after communication, a high-value target can be found, so that there is a communication return estimate E[J|with comm ]= 130. Calculate VoC: VoC = 130 - 100 - 5 - 2 = 23. Threshold: τ comm Still 11.0 (determined by constraint tightness).

[0164] Decision: VoC(23) > τ comm (11.0). Conclusion: Trigger communication.

[0165] This example shows that the decision mechanism of the present application is double-protected: communication must have information value (VoC high) or strategic necessity (threshold low) to be triggered.

[0166] To solve the risk of state representation failure under high-delay asynchronous channels, the present application fuses different asynchronous and different confidence observation information into a unified time-enhanced state through probability extrapolation, delay confidence modeling (such as the conf(d) formula), and time sequence correlation graph aggregation, overcoming the disconnection between perception and reality.

[0167] To solve the problem that reinforcement learning and complex underwater physical constraints (such as SNR, topology) are difficult to couple, the present application adopts a differentiable double-layer joint optimization framework. The shadow price is calculated by implicit differentiation of the KKT system, and is constructed as a dual reward backflow to the upper policy network, so that the policy can indirectly control the physical constraints handled by the lower optimizer (such as the HPX algorithm) efficiently and differentiably.

[0168] The present application solves the problem of inefficient use of communication resources. By calculating the communication value evaluation (VoC) to calculate the information income (which already contains the dual reward), the shadow price is used to dynamically set the communication trigger threshold (τ comm ) reflecting the constraint tightness, ensuring that communication is only triggered when it is of high value or high necessity.

[0169] So far, the main embodiments of the present application have been described. Those skilled in the art should understand that the above embodiments are only exemplary, and various combinations, modifications or equivalent replacements of the specific technical features and steps in the above embodiments can be made without departing from the spirit and scope of the present application.

Claims

1. A cooperative detection method for underwater vehicles based on multi-agent reinforcement learning, characterized in that, include: Acquire raw observation flow, ocean current field data and sound velocity profile data of the submersible, and generate a time-series augmented state including submersible kinematic estimation and underwater acoustic link delay distribution; Based on the temporal augmentation state, a pre-trained policy model is invoked to output a constraint multiplier vector; Based on the constraint multiplier vector, and combined with the energy upper limit constraint, signal-to-noise ratio threshold constraint and minimum safe distance constraint, the lower-level optimization is performed to generate a feasible planning solution that includes path set and connectivity configuration; Based on the feasible planning solution, a pre-trained critic model is invoked to calculate the communication value assessment. Based on the communication value assessment and the preset communication trigger threshold, a decision is made on whether to execute underwater acoustic communication; The pre-trained policy model and critic model, and the training steps, include: In the training environment, ocean current field data and sound velocity profile data are acquired to generate training time series states; Based on the training time sequence state, a two-layer joint optimization is performed to generate a training feasible planning solution and generate a dual reward that reflects the tightness of the energy upper limit constraint, the signal-to-noise ratio threshold constraint and the minimum safe distance constraint. Based on the training feasible planning solution and dual reward, perform event-triggered communication simulation to generate an empirical sample set containing state-action pairs and dual rewards; Using an empirical sample set, perform unified training to update the policy model and the critic model; Perform two-level joint optimization, including: The training constraint multiplier vector is inferred from the upper-level policy network pre-defined by the policy model based on the training time sequence state. The lower-level constrained optimization utilizes the training constraint multiplier vector to solve for and generate training feasible programming solutions and Lagrange multipliers. Based on the training feasible programming solution and Lagrange multipliers, the shadow price set is calculated and generated. Based on the constraint default degrees corresponding to the shadow price set and the training feasible programming solution, construct the generating dual reward; The calculation generates a set of shadow prices, including: Assemble a KKT system based on training feasible programming solutions and Lagrange multipliers; Implicit differentiation is performed on the KKT system to address key constraints, including energy limits, signal-to-noise ratio thresholds, or minimum safe distances. Based on the KKT system after implicit differentiation, the sensitivity index of key constraints to training constraint multiplier vectors is solved, and the sensitivity index is calculated as a set of shadow prices.

2. The method according to claim 1, characterized in that, Perform integrated training, including: Batch samples are sampled from the empirical sample set; the temporal difference objective corresponding to the batch sample is obtained and combined with the dual reward to construct a fusion training objective; The dual reward is incorporated into the preset policy gradient or advantage function, and the policy model and the critic model are updated using the fusion training objective. Based on the fusion training objective, the adaptive law of the training constraint multiplier vector is updated synchronously.

3. The method according to claim 1, characterized in that, The lower-level band-constrained optimization employs a homotopy-preserving topology-continuity hybrid representation, specifically including: The underwater path is represented by a combination of preset B-spline control points and sound velocity profile refraction segments derived from sound velocity profile data; optimal transmission reregistration is applied to match the parent B-spline T-spline control points to generate an initial path set. Detect anchor points in the initial path set that bypass underwater terrain or cross superstructures to form a homotopic anchor point set; Swap path segments only where the homotopy anchor set is aligned, perform homotopy-preserving intersection, and generate intersection candidate paths; Based on constraints including energy upper limit and signal-to-noise ratio threshold, feasible region projection is performed on the cross candidate paths to obtain a set of feasible candidate paths; The set of feasible candidate paths is used to train feasible planning solutions for the lower-level constrained optimization solution.

4. The method according to claim 2, characterized in that, The execution event triggers the communication simulation, specifically as follows: Based on training feasible programming solutions, dual rewards, and critic models, the value assessment of communication is calculated. Based on the shadow price set and the training constraint multiplier vector, the communication trigger threshold is dynamically set; When the communication value assessment exceeds the communication trigger threshold, a communication schedule is generated and a communication simulation is executed to obtain the communication execution log. The communication execution logs, dual rewards, and corresponding state-action pairs are used as the empirical sample set.

5. The method according to claim 4, characterized in that, The assessment of communication value includes: Under the assumption of no communication, the no-communication reward estimate is estimated using the critic model and the training time sequence. Under the assumption of communication, the potential benefits of the joint critic model, training time-series state, and dual reward are estimated to have a communication reward estimate. Based on the pre-defined underwater acoustic channel statistical model and the delay and packet loss distribution, the communication energy consumption, latency and packet loss overhead are calculated to obtain the communication cost and risk. Communication value assessment is determined by combining estimates of communication returns, estimates of communication returns without communication returns, and communication costs and risks.

6. The method according to claim 4, characterized in that, Dynamically set communication trigger thresholds, including: Retrieve the shadow prices related to connectivity from the shadow price set; Obtain the multipliers related to connectivity in the training constraint multiplier vector; The shadow price and the multiplier are mapped to the initial communication trigger threshold, where the tighter the constraint reflected by the shadow price or multiplier, the lower the initial communication trigger threshold. Apply smoothing or hysteresis processing to the initial communication trigger threshold to generate the communication trigger threshold.

7. The method according to claim 5, characterized in that, Generate training time-series states, including: Based on the original observed flow and ocean current field data and pre-stored platform parameters, probabilistic extrapolation is performed on the outdated state of neighboring underwater vehicles to obtain the training prospective state; Based on the latency and packet loss distribution and platform parameters, latency features and latency confidence are encoded and generated. The training prospective state, delayed features, delayed confidence, and original observation stream are integrated, and a time-series correlation graph is used to aggregate them to generate the training time-series state.

Images