A multi-uuv regional protection hierarchical cooperative decision-making method based on multi-agent reinforcement learning

Abstract

This invention specifically relates to a hierarchical collaborative decision-making method for multi-UUV area protection based on multi-agent reinforcement learning, comprising: first, establishing a multi-UUV collaborative area protection task scenario, describing the multi-UUV collaborative area protection process as a Markov game; then, designing a target allocation method based on an auction mechanism to achieve efficient and orderly division of interception tasks; finally, by designing a dual fusion mechanism, tightly integrating the multi-agent reinforcement learning method with the target allocation scheme to complete the learning and optimization of the optimal interception strategy. This invention can achieve efficient collaborative decision-making among multiple UUVs in complex adversarial environments, effectively improving strategy training efficiency and strategy balance, and increasing the execution efficiency and success rate of area protection tasks.

Description

Technical Field

[0001] This invention relates to the field of collaborative control and intelligent decision-making technology for multiple unmanned underwater vehicles (UUVs), specifically to a hierarchical collaborative decision-making method for regional protection of multiple UUVs based on multi-agent reinforcement learning. Background Technology

[0002] Unmanned Underwater Vehicles (UUVs), as miniaturized, multi-purpose, and intelligent unmanned marine transport platforms, are widely used in civilian fields such as marine hydrological observation and underwater target detection, as well as in underwater patrol and mine countermeasures operations. With the increasing complexity of missions, the limited payload deployment and execution capabilities of a single UUV mean that individual UUVs cannot meet the demands of modern civilian and military applications. Therefore, multiple UUVs can form a swarm system to jointly execute missions, either homogeneously or heterogeneously. A multi-UUV swarm not only has more comprehensive payload deployment capabilities but also a wider detection range, stronger mission execution capabilities, and higher fault tolerance, significantly improving the efficiency and success rate of diverse underwater operations. Based on this, multi-UUV swarms have gradually become a focus and hot topic in the development of developed countries in the marine field.

[0003] Underwater multi-UUV cooperative area protection missions, as a typical application of multi-UUV swarms, aim to utilize the coordinated efforts of multiple UUV swarms to prevent intruding targets from damaging high-value core areas. This mission typically exhibits significant strong adversarial nature and high dynamism, requiring UUVs to adjust their strategies in real time according to environmental changes. It also faces challenges such as limited perception, communication latency, and energy constraints. Among these, energy constraints are a crucial factor limiting the continuous execution of missions and overall interception effectiveness, especially in complex and dynamic environments where frequent path scheduling and target switching further exacerbate energy consumption.

[0004] In recent years, Multi-Agent Reinforcement Learning (MARL) has demonstrated significant advantages in solving complex multi-agent cooperative decision-making problems, such as target tracking, swarm control, and area protection. Compared to optimal control methods and differential game theory, MARL does not rely on precise system modeling, making it suitable for dynamic adversarial scenarios with complex target movements and uncertain environments, and it possesses real-time decision-making capabilities. Although research on MARL in cooperative defense tasks has made initial progress, several challenges and limitations remain. On the one hand, some work focuses on "many-to-one" scenarios, failing to fully reflect the cooperative and competitive relationships between agents in multi-target environments. On the other hand, although existing work addresses complex "many-to-many" scenarios, the analysis of coordination mechanisms among defensive agents is insufficient, failing to explicitly resolve target conflicts and still exhibiting resource allocation redundancy issues.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] This invention provides a multi-agent reinforcement learning-based hierarchical collaborative decision-making method for multi-UUV region protection, a computer-readable storage medium, and a computer program product, which can effectively overcome the defects existing in the prior art.

[0007] Other features and advantages of the invention will become apparent from the following detailed description, or may be learned in part by practice of the invention.

[0008] According to a first aspect of the present invention, a hierarchical collaborative decision-making method for multi-UUV region protection based on multi-agent reinforcement learning is provided, the method comprising: Acquire real-time situational information for multi-UUV area protection missions; the real-time situational information includes: protection area information, the movement status of multiple defensive UUVs, and the movement status of multiple intruding UUVs; A target allocation model is constructed between defensive UUVs and intruding UUVs, and the allocation benefits between defensive UUVs and intruding UUVs are calculated based on real-time situational information. The allocation benefits are combined with an auction mechanism to optimize the target allocation model and determine the task allocation result. The task allocation result is the intruding UUV that the defensive UUV needs to intercept at the current decision moment. Using task allocation results and real-time situational information, construct observation status information for each defending UUV; The observation status information of each defensive UUV is input into the pre-trained multi-UUV collaborative decision-making model to obtain the continuous interception actions of each defensive UUV. The continuous interception actions include the expected yaw rate and the expected forward velocity. The multi-UUV collaborative decision-making model is trained based on the reward function of the fused task allocation results. The reward function is used to evaluate the comprehensive benefits of each defensive UUV after performing continuous interception actions during the training process, so as to iteratively optimize the network parameters of the multi-UUV collaborative decision-making model.

[0009] In some exemplary embodiments, the calculation of the distribution of benefits between defensive UUVs and intrusive UUVs based on real-time situational information includes: The location vectors of each defending UUV and each intruding UUV are extracted from the real-time situation information. Take any defending UUV as the current defending UUV, and calculate the current position difference vector between the position vector of the current defending UUV and the position vectors of each intruding UUV; Calculate the L2 norm of each current position difference vector to obtain the distance cost of the current defending UUV relative to each intruding UUV; Based on the difference between the fixed revenue of each intruding UUV and the corresponding distance cost, the allocated revenue of the current defending UUV for each intruding UUV is obtained; Iterate through each defensive UUV to obtain the allocated benefits of each defensive UUV for each intruding UUV.

[0010] In some exemplary embodiments, the target allocation model includes:

[0011] In the formula, For assigning variables, Indicates intrusion into UUV Assigned to defend against UUVs ,otherwise ; For the profit function; This means that each invading UUV can be assigned to a maximum of two defensive UUVs; This means that at the current decision-making moment, each defensive UUV can be assigned at most one intrusion UUV as an interception target; To defend against UUV The remaining number of interceptions.

[0012] In some exemplary embodiments, the step of optimizing the target allocation model by combining the allocation of revenue with an auction mechanism to determine the task allocation result includes: Based on the remaining interception capabilities of each defensive UUV, candidate defensive UUVs with interception capabilities are identified from multiple defensive UUVs; Unassigned intrusion UUVs are identified as remaining targets to be assigned, and the current price of each remaining target to be assigned is obtained in the auction mechanism. Based on the difference between the allocation revenue of the candidate defense UUV for the remaining targets to be allocated and the corresponding current price, calculate the net revenue of the candidate defense UUV for the remaining targets to be allocated, and update the current price of the corresponding remaining targets to be allocated based on the net revenue. The net benefit of candidate defensive UUVs against the remaining unassigned targets is recalculated based on the updated current price; When the number of remaining targets to be assigned is not less than the number of candidate defense UUVs, one remaining target to be assigned is selected as the interception target for each candidate defense UUV based on the recalculated net benefit. When the number of remaining targets to be assigned is less than the number of candidate defense UUVs, cooperative interception targets are selected from the assigned targets for candidate defense UUVs based on the recalculated net benefit. Repeat the current price update and net profit calculation process until the target allocation matrix that maximizes the overall profit of the target allocation model is obtained, and use the target allocation matrix as the task allocation result.

[0013] In some exemplary embodiments, the step of constructing the observation status information of each defensive UUV using task allocation results and real-time situational information includes: Based on the task allocation results, determine the interception target corresponding to each defending UUV at the current decision moment; Taking any defensive UUV as the current defensive UUV, extract the motion state of the current defensive UUV from the real-time situation information; Based on the current motion state of the defending UUV and the motion states of other defending UUVs, generate the first relative motion state of other defending UUVs relative to the current defending UUV; By utilizing the current motion state of the defending UUV and the motion states of each intruding UUV, a second relative motion state of each intruding UUV relative to the current defending UUV is generated; Based on the current motion state of the defending UUV and the motion state of the corresponding interception target, a third relative motion state of each current defending UUV relative to the interception target is generated; By combining the third relative motion state and the remaining interception capability of the current UUV defense, the current UUV defense mission state is generated; By combining the current self-motion state, first relative motion state, second relative motion state, and mission state of the UUV defense, the observation space of the current UUV defense is obtained. By traversing all the defensive UUVs, the observation space of each defensive UUV is obtained.

[0014] In some exemplary embodiments, the step of inputting the observation status information of each defensive UUV into a pre-trained multi-UUV collaborative decision-making model to obtain the continuous interception actions of each defensive UUV includes: The observation status information of each defensive UUV is input into the policy network of the pre-trained multi-UUV collaborative decision-making model; the multi-UUV collaborative decision-making model is optimized based on the multi-agent dual-delay deep deterministic policy gradient algorithm. The action vectors for each UUV defense are calculated using the deterministic policy of the policy network; The expected forward velocity of each defensive UUV is determined based on the first motion component in the motion vector, and the expected yaw rate of each defensive UUV is determined based on the second motion component in the motion vector. The desired yaw rate and desired forward velocity are used as the continuous interception actions of each defensive UUV.

[0015] In some exemplary embodiments, the training process of the multi-UUV collaborative decision-making model includes: Initialize the network parameters, target network parameters, and experience buffer of the multi-UUV area protection task scenario and the multi-UUV collaborative decision-making model; and construct joint observation information for the training phase based on the task allocation matrix and the real-time situation information of the multi-UUV area protection task scenario. Based on the joint observation information and task allocation matrix during the training phase, the deterministic policy of the policy network in the multi-UUV collaborative decision-making model is used to generate continuous interception actions for each defensive UUV; and the continuous interception actions of each defensive UUV are spliced ​​into a joint action. Based on the joint action control, each defensive UUV performs actions in the multi-UUV area protection mission training scenario, obtaining mission execution status information and real-time situation information at the next training moment; and based on the mission allocation result, mission execution status information and reward function, calculate joint instant reward; and update the mission allocation matrix and the observation status information of each defensive UUV according to the real-time situation information at the next moment. The current joint observation information, joint actions, joint instant rewards, and the next time-in-time joint observation information, along with the updated task assignment matrix, are stored as experience data in the experience buffer. Empirical data is extracted from the empirical buffer, and the value network parameters and policy network parameters of the multi-UUV collaborative decision-making model are updated based on the empirical data and the network loss function. The target network parameters are softly updated according to a preset update interval, and a pre-trained multi-UUV collaborative decision-making model is obtained when the preset training termination condition is met.

[0016] In some exemplary embodiments, the reward function includes:

[0017] in, These are the weighting coefficients. For the first Rewards for continuous attacks on infiltrated UUVs. For the first A reward for a successful attack that infiltrates a UUV. For the first A defensive UUV relative to the assigned target Continuous defense rewards Rewards for the UUV defense team.

[0018] According to a second aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the storage medium is located to execute the above-described multi-UUV area protection hierarchical collaborative decision-making method based on multi-agent reinforcement learning.

[0019] According to a third aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the above-described multi-agent reinforcement learning-based hierarchical collaborative decision-making method for multi-UUV area protection.

[0020] According to a fourth aspect of the present invention, an electronic device is provided, comprising: Processor; and Memory for storing the executable instructions of the processor; The processor is configured to implement the above-described hierarchical collaborative decision-making method for multi-UUV region protection based on multi-agent reinforcement learning when executing the executable instructions.

[0021] The embodiments of this invention provide a hierarchical collaborative decision-making method for multi-UUV area protection based on multi-agent reinforcement learning. This method employs a hierarchical framework of "upper-layer target allocation—lower-layer action optimization," combining an auction mechanism with multi-agent reinforcement learning. At the upper layer, based on real-time situational information of the area protection task, the allocation benefits between defending UUVs and intruding UUVs are calculated. Combined with the remaining interception capabilities of the defending UUVs, the target allocation model is optimized to generate a target allocation scheme between the defending and intruding UUVs. At the lower layer, the target allocation scheme is integrated into the multi-agent reinforcement learning process. The trained policy network outputs the expected forward velocity and expected yaw rate of each defending UUV, generating continuous interception actions. Simultaneously, it enables the defending UUVs to make action decisions and learn strategies around the allocated interception targets. This improves the rationality of target allocation, the efficiency of collaborative interception, and the success rate of task execution in multi-UUV area protection tasks, thereby solving the technical problems of low interception efficiency and redundant resource allocation in existing multi-agent reinforcement learning methods for underwater multi-UUV area protection tasks.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0024] Figure 1 The flowchart illustrates an exemplary embodiment of the present invention: a hierarchical collaborative decision-making method for multi-UUV region protection based on multi-agent reinforcement learning. Figure 2 This schematic diagram illustrates a multi-UUV collaborative area protection task scenario according to an exemplary embodiment of the present invention. Figure 3 This schematic diagram illustrates the overall framework of a multi-UUV region protection hierarchical collaborative decision-making method based on multi-agent reinforcement learning, an exemplary embodiment of the present invention. Figure 4 This schematically illustrates a flowchart of the network parameter update process for a multi-agent dual-delay deep deterministic policy gradient algorithm, an exemplary embodiment of the present invention. Figure 5 The illustration shows a flowchart of a multi-UUV region protection hierarchical collaborative decision-making method based on multi-agent reinforcement learning, an exemplary embodiment of the present invention. Figure 6 The diagram illustrates the composition of an electronic device according to an exemplary embodiment of the present invention. Detailed Implementation

[0025] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the invention will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0026] Furthermore, the accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0027] To address the shortcomings and deficiencies of existing technologies, this example implementation provides a hierarchical collaborative decision-making method for multi-UUV area protection based on multi-agent reinforcement learning. (Reference) Figure 1 As shown, it can specifically include: Step S11: Obtain real-time situational information of the multi-UUV area protection mission; wherein, the real-time situational information includes: protection area information, movement status of multiple defensive UUVs and movement status of multiple intruding UUVs; Step S12: Construct a target allocation model between defensive UUVs and intruding UUVs, and calculate the allocation benefits between defensive UUVs and intruding UUVs based on real-time situational information; combine the allocation benefits with an auction mechanism to optimize the target allocation model and determine the task allocation result; wherein, the task allocation result is the intruding UUVs that the defensive UUV needs to intercept at the current decision moment. Step S13: Using the task allocation results and real-time situation information, construct the observation status information of each defensive UUV; Step S14: Input the observation space of each defensive UUV into the pre-trained multi-UUV collaborative decision-making model to obtain the continuous interception actions of each defensive UUV; wherein, the continuous interception actions include: expected yaw rate and expected forward velocity; the multi-UUV collaborative decision-making model is trained based on the reward function of the fused task allocation results. The reward function is used to evaluate the comprehensive benefits of each defensive UUV after performing continuous interception actions during the training process, so as to iteratively optimize the network parameters of the multi-UUV collaborative decision-making model; Step S15: Control the corresponding UUV to perform area protection tasks based on the continuous interception actions of each UUV.

[0028] The following will describe in more detail each step of a multi-agent reinforcement learning-based hierarchical collaborative decision-making method for multi-UUV area protection in this exemplary embodiment, with reference to the accompanying drawings and embodiments.

[0029] For example, refer to Figure 2 As shown, Figure 2 This is a schematic diagram of a multi-UUV collaborative area protection task scenario. The multi-UUV collaborative area protection task in step S11 consists of the core area... , Ships defending UUVs and The group consisted of three parties, including an intruding UUV. Defending against UUVs and intruding UUVs are represented as follows: and ,in, , .

[0030] The task establishes a global coordinate system with the core region on the same horizontal plane and a carrier coordinate system with the UUV itself; the underactuated three-degree-of-freedom motion model of each UUV on the horizontal plane is shown in expression (1).

[0031] (1) In the formula, Generalized coordinates in the global coordinate system For position coordinates, Yaw angle; For generalized velocity in the carrier coordinate system, Includes forward velocity and lateral speed , Yaw angular velocity, These are the maximum forward speed and the maximum yaw rate, respectively; the maximum forward speed of the defending UUV is set to be greater than the maximum forward speed of the intruding UUV; For the rotation matrix from the global system to the loaded system; These are the inertia matrix, the Coriolis force and centripetal force matrix, and the damping matrix, respectively. This is the sum of external forces and torques acting on the UUV.

[0032] Specifically, a multi-UUV collaborative regional protection task scenario is established, including scenario settings and UUV motion models, to provide a simulation environment for training the multi-UUV collaborative decision-making model, and to obtain real-time situational information of the multi-UUV regional protection task based on the multi-UUV collaborative regional protection task scenario.

[0033] For example, in step S12, calculating the distribution of benefits between defensive UUVs and intrusive UUVs based on real-time situational information includes: Step S121: Extract the position vectors of each defensive UUV and each intruding UUV from the real-time situation information. Step S122: Take any defensive UUV as the current defensive UUV, and calculate the current position difference vector between the position vector of the current defensive UUV and the position vector of each intruding UUV. Step S123: Calculate the L2 norm of each current position difference vector to obtain the distance cost of the current defending UUV relative to each intruding UUV; Step S124: Based on the difference between the fixed revenue of each intruding UUV and the corresponding distance cost, obtain the allocated revenue of the current defending UUV for each intruding UUV. Step S125: Iterate through each defensive UUV to obtain the allocated benefits of each defensive UUV for each intruding UUV.

[0034] Specifically, the distribution of benefits between defensive UUVs and intrusive UUVs is represented by the first... UUV defense Execute against the Invading UUV The allocation of benefits corresponding to the interception task is calculated as shown in expression (2).

[0035] (2) In the formula, for Fixed income, A constant value related to the size of the operational waters; For distance cost, It is represented as the 2-norm of a vector.

[0036] For example, in step S12, the optimization solution of the target allocation model by combining the allocation revenue with the auction mechanism to determine the task allocation result includes: Step S126: Based on the remaining interception capabilities of each defensive UUV, determine the candidate defensive UUVs with interception capabilities from multiple defensive UUVs. Step S127: Identify the unallocated intrusion UUVs as remaining targets to be allocated, and obtain the current price of each remaining target to be allocated in the auction mechanism; Step S128: Calculate the net benefit of the candidate defense UUV for the remaining targets to be allocated based on the difference between the allocation benefit of the candidate defense UUV for the remaining targets to be allocated and the corresponding current price, and update the current price of the corresponding remaining targets to be allocated based on the net benefit. Step S129: Recalculate the net benefit of the candidate defensive UUV for the remaining targets to be allocated based on the updated current price; Step S1210: When the number of remaining targets to be assigned is not less than the number of candidate defense UUVs, select one remaining target to be assigned as the interception target for each candidate defense UUV based on the recalculated net benefit. Step S1211: When the number of remaining targets to be assigned is less than the number of candidate defense UUVs, select cooperative interception targets for candidate defense UUVs from the assigned targets based on the recalculated net benefit. Step S1212: Repeat the current price update and net profit calculation process until the target allocation matrix that maximizes the overall profit of the target allocation model is obtained, and use the target allocation matrix as the task allocation result.

[0037] Specifically, the task objective allocation problem is modeled as a multi-task-single-robot-instantaneous allocation (MT-SR-IA) problem. MT-SR-IA is a typical nonlinear multi-constraint combinatorial optimization problem, so an objective allocation model that maximizes the overall benefit of defending against UUVs is established, as shown in expression (3).

[0038] (3) In the formula, For assigning variables, express Assigned to ,otherwise ; For the allocation matrix; For the profit function; for The remaining number of interceptions; This means that each invading UUV can be assigned to a maximum of two defensive UUVs; This means that at the current decision-making moment, each defensive UUV can be assigned a maximum of one intruding UUV as an interception target.

[0039] An optimization solution for the target allocation model is based on an auction mechanism. Intruding UUVs are considered "auction items," and all defending UUVs are considered "bidders." hour, Based on the net proceeds of the remaining targets to be allocated right Updated prices And there are two cases: (1) If the number of remaining intruding UUVs is not less than the number of defensive UUVs with interception capabilities, then all of Choose the highest price As the allocation target; (2) If the number of remaining intruding UUVs is less than the number of defensive UUVs with interception capabilities, there will be... of The case where no target has been assigned; in this case, the situation will be... Select the highest-priced intrusion UUV from the allocated set. As the assigned target, it cooperates with other defensive UUVs to complete the interception task and avoid... Idle.

[0040] in, This indicates the price of the intruding UUV in the auction mechanism, used to adjust the selection priority of the defending UUV over the intruding UUV; This indicates that UUV defense is taking current prices into account. The net benefit of subsequently choosing to invade UUVs.

[0041] For details, please refer to Figure 3 As shown, Figure 3 This is a diagram illustrating the overall framework of a multi-agent reinforcement learning-based hierarchical collaborative decision-making method for multi-UUV area protection, as proposed in this invention. The upper part of the diagram represents the target allocation method based on an auction mechanism, corresponding to the upper-level decision-making module in the entire hierarchical collaborative decision-making framework. Its main function is to determine the corresponding intrusion UUV interception targets for each defensive UUV based on the current area protection task situation and generate a target allocation scheme. Specifically, the upper layer models the target allocation problem as an MT-SR-IA problem, namely, a multi-task-single-robot-instant allocation problem. In this allocation process, defensive UUVs are treated as "bidders," and intrusion UUVs are treated as "auction items." Each defensive UUV generates a bid based on its own allocation benefits with each intrusion UUV and evaluates the bidding value of different intrusion UUVs. The upper-level process in the diagram can be understood as follows: after the defensive UUV generates a bid, the bid is evaluated, and the bidder with the best bid is selected; then, the target is allocated according to the auction results, the bid is updated, and the final target allocation scheme is obtained by optimizing the allocation process.

[0042] This process determines the interception target of each defending UUV at the current decision moment. Finally, the target allocation scheme output from the upper layer is passed to the lower-level multi-agent reinforcement learning module and further integrated into the lower-level reward function and observation space through the "dual fusion mechanism" shown in the diagram, guiding each defending UUV to generate continuous interception actions.

[0043] For example, in step S13, constructing the observation status information of each defensive UUV using the task allocation results and real-time situational information includes: Step S131: Based on the task allocation results, determine the interception target corresponding to each defending UUV at the current decision moment; Step S132: Taking any defensive UUV as the current defensive UUV, extract the motion state of the current defensive UUV from the real-time situation information. Step S133: Based on the current motion state of the defensive UUV itself and the motion states of other defensive UUVs, generate the first relative motion state of other defensive UUVs relative to the current defensive UUV. Step S134: Using the current motion state of the defending UUV and the motion state of each intruding UUV, generate the second relative motion state of each intruding UUV relative to the current defending UUV. Step S135: Based on the current motion state of the defending UUV and the motion state of the corresponding intercepted target, generate the third relative motion state of each current defending UUV relative to the intercepted target; Step S136: Combine the third relative motion state and the remaining interception capability of the current UUV defense to generate the current UUV defense mission state; Step S137: Combine the current self-motion state, first relative motion state, second relative motion state and mission state of the UUV defense to obtain the current observation state information of the UUV defense. Step S138: Traverse each defensive UUV to obtain the observation status information of each defensive UUV.

[0044] Specifically, in conjunction with the target allocation scheme, defense against UUVs The observation state information is derived from the motion state. and task status It consists of two parts, namely .

[0045] motion state information The description is in three parts: 1) Personal information ; 2) Other UUV defenses Relative position and velocity information ; 3) With intrusion UUV Relative position and velocity information .

[0046] but , in,

[0047] .

[0048] Task status information The description is in two parts: 1) Combining the target allocation matrix, the allocated... and Relative position and velocity information Included in task status; 2) Remaining interception attempts ; but .

[0049] For example, in step S14, inputting the observation status information of each defensive UUV into a pre-trained multi-UUV collaborative decision-making model to obtain the continuous interception actions of each defensive UUV includes: Step S141: Input the observation status information of each defensive UUV into the policy network of the pre-trained multi-UUV collaborative decision-making model; wherein, the multi-UUV collaborative decision-making model is optimized based on the multi-agent dual-delay deep deterministic policy gradient algorithm. Step S142: Calculate the action vector for each UUV defense using the deterministic policy of the policy network; Step S143: Determine the expected forward velocity of each defensive UUV based on the first motion component in the motion vector, and determine the expected yaw rate of each defensive UUV based on the second motion component in the motion vector. Step S144: The desired yaw rate and desired forward velocity are used as the continuous interception actions of each defensive UUV.

[0050] Specifically, the multi-UUV collaborative regional protection process is modeled as a Markov game, denoted as... .in, It is a state space; It is a joint action space. It was a joint operation. for The action; Let be the state transition probability function; For the joint reward function, for Instant reward value; For joint observation space; Let be the observation probability function; For joint observation, for Observed values; This is the discount factor. (Definition) .

[0051] The motion space of the UUV is set as two consecutive desired motion signals. ,in, For the desired yaw rate, The desired forward velocity.

[0052] Furthermore, the multi-UUV regional protection task is modeled as a Markov game model. Based on the Markov game model, the observation space and action space of the defending UUVs are determined. Then, using a trained multi-UUV collaborative decision-making model, the deterministic actions of each defending UUV under the current observation information are solved. (Reference) Figure 3As shown, the target allocation scheme output from the upper layer is integrated into the reward function and the observation space through a dual fusion mechanism. On one hand, the target allocation scheme is used to determine the interception target of each defending UUV and participates in the construction of the reward function, enabling the reward function to calculate the reward value of each defending UUV based on the relative state between the defending UUV and the corresponding interception target, the state of the intruding UUV relative to the protected area, and the overall area protection mission state. On the other hand, the target allocation scheme is introduced into the observation space, so that the observation state information of each defending UUV includes not only its own motion state, the relative motion state of other defending UUVs and the relative motion state of the intruding UUV, but also the mission state information determined by the target allocation scheme.

[0053] Specifically, for the m-th defensive UUV Based on the dual fusion mechanism, corresponding observation state information is generated. and reward value Combine the observation status information of multiple UUV defense vessels into a joint observation value. The reward value from multiple defensive UUVs is combined into a joint reward value. After inputting the observation information of each defensive UUV into the trained multi-UUV collaborative decision-making model, the policy network outputs the corresponding actions for each defensive UUV. Multiple actions together form a combined action. .

[0054] Among them, actions To defend against UUV The continuous interception actions include the desired forward velocity and desired yaw rate. The area protection mission scenario is based on joint actions. Update the motion status of the defensive UUV and feed back the joint observations for the next moment. and joint reward value .

[0055] During the model training phase, The experience data is stored in the experience replay buffer, and the MATD3 model is optimized for strategy and value based on the experience data in the experience replay buffer. During the model usage phase, the trained policy network is directly used to output continuous interception actions of each defending UUV based on the current joint observations, thereby realizing collaborative action decision-making in multi-UUV area protection tasks.

[0056] For example, in step S15, the corresponding UUV is controlled to perform an area protection task based on the continuous interception actions of each UUV.

[0057] At the current decision-making moment, the pre-trained multi-agent reinforcement learning model outputs continuous interception actions for each defensive UUV and sends these actions to the corresponding defensive UUVs. This controls each defensive UUV to adjust its navigation state according to the desired forward velocity and desired yaw rate. After each defensive UUV executes its continuous interception actions, the area protection mission scenario undergoes a state transition. The position, velocity, and heading of both the defensive and intruding UUVs are updated accordingly, thus obtaining the real-time situational information for the next moment. The next moment refers to the subsequent decision-making moment formed after the defensive UUVs execute their continuous interception actions at the current decision-making moment, resulting in an updated area protection mission scenario.

[0058] Based on the real-time situational information of the next moment, the task allocation results and observation information are regenerated, and the trained multi-agent reinforcement learning model is called again to output the continuous interception actions of the next moment, thus forming a cyclical decision-making process of "situational update - task allocation - observation construction - action output - task execution".

[0059] For example, in step S21, the multi-UUV collaborative decision-making model training process includes: Step S211: Initialize the network parameters, target network parameters, and experience buffer of the multi-UUV area protection task scenario and the multi-UUV collaborative decision-making model; and construct joint observation information for the training phase based on the task allocation matrix and the real-time situation information of the multi-UUV area protection task scenario. Step S212: Based on the joint observation information and task allocation matrix during the training phase, the deterministic policy of the policy network in the multi-UUV collaborative decision-making model is used to generate continuous interception actions for each defensive UUV; and the continuous interception actions of each defensive UUV are spliced ​​into a joint action. Step S213: According to the joint action control, each defensive UUV performs actions in the multi-UUV area protection task training scenario to obtain task execution status information and real-time situation information at the next training moment; and calculate the joint instant reward based on the task allocation result, task execution status information and reward function; and update the task allocation matrix and the observation status information of each defensive UUV according to the real-time situation information at the next moment. Step S214: Store the current joint observation information, joint actions, joint instant rewards, and joint observation information at the next time step as experience data in the experience buffer. Step S215: Extract empirical data from the empirical buffer, and update the value network parameters and policy network parameters of the multi-UUV collaborative decision-making model based on the empirical data and the network loss function. Step S216: Soft update the target network parameters according to the preset update interval, and obtain the pre-trained multi-UUV collaborative decision model when the preset training termination condition is met.

[0060] Specifically, the reward function is shown in expression (4).

[0061] (4) in, These are the weighting coefficients. Rewards for continued attacks, For the first A reward for a successful attack that infiltrates a UUV. For the first A defensive UUV relative to the assigned target Continuous defense rewards Rewards for the UUV defense team.

[0062] This is used to guide the defense UUV to complete the area protection task as soon as possible, which helps to avoid reward sparsity. The specific calculation method is shown in expression (5).

[0063] (5) express A one-time penalty for a successful attack, i.e., when... hour, ;otherwise , The radius of the intrusion.

[0064] Used for guiding defense against UUVs Intercepting the assigned target ( ),encourage continuously towards Approach until the interception threshold is reached.

[0065] when hour, Depend on and interception target Distance Rewards and angle bonus items It consists of two parts. For interception radius, distance to reward item and angle bonus items As shown in expression (6).

[0066] (6) when When, it means Successfully intercepted the assigned target You will receive a one-time reward. .

[0067] This is a one-time reward given to each UUV in the defensive UUV team at the end of the round. When the area protection mission is successful, ;otherwise, .

[0068] A successful area protection mission is indicated by either all intruding UUVs meeting the interception success criteria during mission execution, or none meeting the intrusion success criteria within the maximum mission time.

[0069] The condition for successful interception is expressed as: if and only if for any ,exist satisfy ,but Successfully intercepted ; The condition for a successful intrusion is expressed as: if and only if for any satisfy ,but Successfully infiltrated the core area.

[0070] Specifically, in policy learning based on multi-agent reinforcement learning, the Multi-Agent Dual-Delay Deep Deterministic Policy Gradient Algorithm (MATD3) is used to optimize interception actions. The UUV collaborative decision-making model includes a policy network, two value networks, and corresponding target policy and target value networks. The policy network outputs continuous interception actions based on local observations of UUV defense, while the value networks estimate action values ​​based on joint observations and joint actions.

[0071] For the UUV defense The deterministic policy fit by the policy network is expressed as: ,in, It is a deterministic policy that the policy network fits. These are policy network parameters. The actions of multiple UUV defense vessels together constitute a coordinated action. .

[0072] Value networks are used to estimate action values. ,in, For value network parameters, For the deterministic policy fitted to the policy network, During the value network update process, the value network loss function is shown in expression (7).

[0073] (7) In the formula, the target value Represented as: ; and These represent the joint observation and joint action at the next moment, respectively. These are the parameters of the target value network.

[0074] To ensure the smoothness of the target strategy, Gaussian noise is sheared. Add it to the actions generated by all target policy networks, as shown in expression (8).

[0075] (8) In the formula, It is the action output by the target policy network. These are the network parameters of the target policy; It has a mean of 0 and a variance of Gaussian white noise; It is a constant value; This is the shearing function.

[0076] During the policy network update process, the policy network utilizes local observations. Select Action The value of the action output by the policy network is evaluated through the value network. The loss function of the policy network under the deterministic policy is shown in expression (9).

[0077] (9) in, It is a deterministic policy that the policy network fits. These are the policy network parameters. The corresponding deterministic policy gradient is shown in expression (10): (10) Thus, the value network uses target values The policy network learns to estimate the value of joint actions, and updates its parameters based on the evaluation results of the value network, thereby gradually learning the mapping relationship from local observations to continuous interception actions.

[0078] Specifically, the multi-UUV collaborative decision-making model is iteratively trained in a simulation scenario. The training process for the collaborative decision-making model is designed, and the total number of training rounds is denoted as... The maximum task time is recorded as experience buffer Minimum training batch The training batch size is denoted as Update interval The training process includes the following steps: Step 1: Initialization: Experience Buffer Allocation matrix Policy network parameters Policy network parameters Value network parameters Target value network parameters and make ; Step 2: Iterate through all rounds ,implement: (1) Initialize the environment and obtain joint observations Target allocation matrix ; (2) If the conditions for ending the regional protection task are not met, the current time is ,implement: Deterministic policy fitted by policy network Calculate actions To obtain joint action Perform joint actions to earn joint instant rewards. Update joint observations Target allocation matrix The sampled data is stored in an empirical buffer. , If the current experience buffer The number of samples in the batch is greater than the minimum training batch. ,from random sampling Given a set of empirical data, perform the following steps: calculate the value network loss function and update the value network parameters; calculate the policy network loss function and update the policy network parameters; if the update interval is met, perform a soft update of the target policy network parameters and the target value network parameters. (3) If the conditions for the termination of the regional protection task are met, the current cycle is terminated and step (1) is executed.

[0079] The conditions for the termination of a regional conservation mission are expressed as follows: From The mission ends when the maximum mission time is reached or when all intruding UUVs meet the conditions for successful interception or successful intrusion.

[0080] Through the iterative training process described above, the policy network gradually learns the output rules for continuous interception actions of each defensive UUV under different observation information, while the value network gradually learns the ability to evaluate the value of joint actions. After training, the trained policy network is used as an online decision network to output the desired forward velocity and desired yaw rate based on the observation space of each defensive UUV.

[0081] Further, refer to Figure 4 As shown, Figure 4This is a flowchart of the network parameter update process for a multi-agent, dual-delay deep deterministic policy gradient algorithm. The process involves extracting training samples from the experience replay buffer and updating the parameters of the policy network, value network, and target network based on these extracted training samples.

[0082] Specifically, the Actor network, Critic1 network, Critic2 network, and their corresponding target networks are first initialized. The Actor network outputs continuous interception actions based on observation information about UUV defense; the Critic1 and Critic2 networks estimate action values ​​based on joint observations and joint actions; and the target Actor network, target Critic1 network, and target Critic2 network calculate target actions and their values ​​during training.

[0083] During network training, batches of experience data are randomly sampled from the experience replay buffer. For the sampled next-time observation information, the target Actor network is used to calculate the target action at the next time step. Then, the joint observation information and target action at the next time step are input into the target Critic1 network and the target Critic2 network, respectively, to obtain the target Q1 value and the target Q2 value. The target value is then determined based on the target Q1 value and the target Q2 value. At the same time, the current joint observation information and the current joint action are input into the Critic1 network and the Critic2 network, respectively, to obtain the current Q1 value and the current Q2 value.

[0084] Based on the error between the target value and the current Q1 and Q2 values, the value network loss is calculated, and the parameters of the Critic1 and Critic2 networks are updated. Subsequently, the updated value network is used to evaluate the value of the actions output by the Actor network, the Actor network loss is calculated, and the Actor network parameters are updated with a preset delay. After updating the parameters of the Actor and Critic networks, a soft update is performed on the target Actor network, target Critic1 network, and target Critic2 network, so that the target network parameters smoothly approach the current network parameters according to a preset ratio.

[0085] Through the above process, the Actor network gradually learns the mapping relationship between UUV defense observation information and continuous interception actions, while the Critic1 and Critic2 networks gradually learn the ability to estimate the value of joint actions. After training, the trained Actor network is retained as the policy network for online decision-making, used to output corresponding continuous interception actions based on the observation status information of each UUV being defended.

[0086] The method provided in the embodiments of the present invention is referred to Figure 5This is a flowchart illustrating a hierarchical collaborative decision-making method for multi-UUV area protection based on multi-agent reinforcement learning. The method employs a hierarchical framework, combining an auction mechanism with multi-agent reinforcement learning to generate expected multi-UUV collaborative interception actions at the guidance level. The upper layer is a target allocation method based on an auction mechanism, and the lower layer is an interception action optimization method based on multi-agent reinforcement learning. A dual fusion mechanism is designed to tightly integrate the upper and lower layers, realizing a hierarchical collaborative interception strategy for multi-UUV area protection. Specifically, it includes the following steps: Step 1: Establish a multi-UUV collaborative regional protection task scenario, including: motion models and Markov game modeling of the core area, defending UUVs, and intruding UUVs.

[0087] Step 2: Design a target allocation method based on an auction mechanism: Generate a target allocation scheme based on the allocation benefits between defending UUVs and intruding UUVs, the remaining interception capacity, and the auction price.

[0088] Step 3: Design an optimization method for interception actions that integrates target allocation results.

[0089] (1) Design an observation space fusion mechanism: integrate the target allocation scheme into the observation information of each defense UUV; (2) Design a reward function fusion mechanism: Integrate the target allocation scheme into the reward function to calculate the reward value of each defending UUV.

[0090] (3) Optimize interception actions based on MATD3.

[0091] Step 4: Design the iterative training process for the multi-UUV collaborative decision-making model: Use experience replay, policy network, value network and target network updates to train the collaborative decision-making model.

[0092] The beneficial effects of this invention are as follows: (1) The present invention is a multi-agent reinforcement learning-based hierarchical collaborative decision-making method for multi-UUV region protection. Through a hierarchical framework, the upper layer is based on an auction mechanism to achieve reasonable resource scheduling and target allocation; the lower layer combines a dual fusion mechanism with the target allocation scheme and MATD3 to optimize the collaborative interception action of UUVs. Through the close combination of the upper and lower layers, the policy search space is effectively reduced and the transmission of gradient signals is improved, thereby improving the policy training efficiency and policy balance.

[0093] (2) Compared with traditional multi-agent reinforcement learning methods, this invention can achieve efficient collaborative decision-making among multiple UUVs in a strong adversarial environment, optimize resource allocation, and thus improve interception efficiency, interception success rate and mission success rate.

[0094] It should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0095] It should be noted that although several modules or units of the device for performing actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0096] Figure 6 A schematic diagram of an electronic device suitable for implementing embodiments of the present invention is shown.

[0097] It should be noted that, Figure 6 The electronic device 1000 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0098] like Figure 6 As shown, the electronic device 1000 includes a Central Processing Unit (CPU) 1001, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 1002 or programs loaded from storage section 1008 into Random Access Memory (RAM) 1003. The RAM 1003 also stores various programs and data required for system operation. The CPU 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An Input / Output (I / O) interface 1005 is also connected to the bus 1004. Furthermore, the electronic device 1000 also includes an FPGA device and a System-on-a-Chip (SoC) device.

[0099] The following components are connected to I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to I / O interface 1005 as needed. Removable media 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1010 as needed so that computer programs read from them can be installed into storage section 1008 as needed.

[0100] In particular, according to embodiments of the present invention, the processes described below with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1009, and / or installed from removable medium 1011. When the computer program is executed by central processing unit (CPU) 1001, it performs various functions defined in the system of this application.

[0101] Specifically, the aforementioned electronic devices can be airborne intelligent electronic devices.

[0102] It should be noted that the storage medium shown in the embodiments of the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, wherein computer-readable program code is carried. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any storage medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the storage medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0103] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0104] The units described in the embodiments of the present invention can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0105] It should be noted that, as another aspect, this application also provides a storage medium, which may be included in an electronic device or may exist independently without being assembled into the electronic device. The aforementioned storage medium carries one or more programs, which, when executed by an electronic device, cause the electronic device to perform the methods described in the following embodiments. For example, the electronic device may perform... Figure 1 The steps of the method shown.

[0106] In one embodiment, this application provides a computer program product including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0107] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0108] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.

[0109] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A hierarchical collaborative decision-making method for multi-UUV area protection based on multi-agent reinforcement learning, characterized in that, The method includes: Acquire real-time situational information for multi-UUV area protection missions; the real-time situational information includes: protection area information, the movement status of multiple defensive UUVs, and the movement status of multiple intruding UUVs; A target allocation model is constructed between defensive UUVs and intruding UUVs, and the allocation benefits between defensive UUVs and intruding UUVs are calculated based on real-time situational information. The allocation benefits are combined with an auction mechanism to optimize the target allocation model and determine the task allocation result. The task allocation result is the intruding UUV that the defensive UUV needs to intercept at the current decision moment. Using task allocation results and real-time situational information, construct observation status information for each defending UUV; The observation status information of each defensive UUV is input into the pre-trained multi-UUV collaborative decision-making model to obtain the continuous interception actions of each defensive UUV. The continuous interception actions include the expected yaw rate and the expected forward velocity. The multi-UUV collaborative decision-making model is trained based on the reward function of the fused task allocation results. The reward function is used to evaluate the comprehensive benefits of each defensive UUV after performing continuous interception actions during the training process, so as to iteratively optimize the network parameters of the multi-UUV collaborative decision-making model. Based on the continuous interception actions of each defensive UUV, the corresponding defensive UUV is controlled to perform area protection tasks.

2. The method according to claim 1, characterized in that, The calculation of the distribution of benefits between defensive UUVs and intrusive UUVs based on real-time situational information includes: The location vectors of each defending UUV and each intruding UUV are extracted from the real-time situation information. Take any defending UUV as the current defending UUV, and calculate the current position difference vector between the position vector of the current defending UUV and the position vectors of each intruding UUV; Calculate the L2 norm of each current position difference vector to obtain the distance cost of the current defending UUV relative to each intruding UUV; Based on the difference between the fixed revenue of each intruding UUV and the corresponding distance cost, the allocated revenue of the current defending UUV for each intruding UUV is obtained; Iterate through each defensive UUV to obtain the allocated benefits of each defensive UUV for each intruding UUV.

3. The method according to claim 2, characterized in that, The target allocation model includes: In the formula, For assigning variables, Indicates intrusion into UUV Assigned to defend against UUVs ,otherwise ; For the profit function; This means that each invading UUV can be assigned to a maximum of two defensive UUVs; This means that at the current decision-making moment, each defensive UUV can be assigned at most one intrusion UUV as an interception target; To defend against UUV The remaining number of interceptions.

4. The method according to claim 2, characterized in that, The step of optimizing the target allocation model by combining the distribution of revenue with an auction mechanism to determine the task allocation result includes: Based on the remaining interception capabilities of each defensive UUV, candidate defensive UUVs with interception capabilities are identified from multiple defensive UUVs; Unassigned intrusion UUVs are identified as remaining targets to be assigned, and the current price of each remaining target to be assigned is obtained in the auction mechanism. Based on the difference between the allocation revenue of the candidate defense UUV for the remaining targets to be allocated and the corresponding current price, calculate the net revenue of the candidate defense UUV for the remaining targets to be allocated, and update the current price of the corresponding remaining targets to be allocated based on the net revenue. The net benefit of candidate defensive UUVs against the remaining unassigned targets is recalculated based on the updated current price; When the number of remaining targets to be assigned is not less than the number of candidate defense UUVs, one remaining target to be assigned is selected as the interception target for each candidate defense UUV based on the recalculated net benefit. When the number of remaining targets to be assigned is less than the number of candidate defense UUVs, cooperative interception targets are selected from the assigned targets for candidate defense UUVs based on the recalculated net benefit. Repeat the current price update and net profit calculation process until the target allocation matrix that maximizes the overall profit of the target allocation model is obtained, and use the target allocation matrix as the task allocation result.

5. The method according to claim 4, characterized in that, The process of constructing observation status information for each defending UUV using task allocation results and real-time situational information includes: Based on the task allocation results, determine the interception target corresponding to each defending UUV at the current decision moment; Taking any defensive UUV as the current defensive UUV, extract the motion state of the current defensive UUV from the real-time situation information; Based on the current motion state of the defending UUV and the motion states of other defending UUVs, generate the first relative motion state of other defending UUVs relative to the current defending UUV; By utilizing the current motion state of the defending UUV and the motion states of each intruding UUV, a second relative motion state of each intruding UUV relative to the current defending UUV is generated; Based on the current motion state of the defending UUV and the motion state of the corresponding interception target, a third relative motion state of each current defending UUV relative to the interception target is generated; By combining the third relative motion state and the remaining interception capability of the current UUV defense, the current UUV defense mission state is generated; The observation status information of the current UUV defense is obtained by combining the current self-motion state, first relative motion state, second relative motion state and mission state. By iterating through each defensive UUV, the observation status information of each defensive UUV is obtained.

6. The method according to claim 1, characterized in that, The step of inputting the observation status information of each defensive UUV into a pre-trained multi-UUV collaborative decision-making model to obtain the continuous interception actions of each defensive UUV includes: The observation status information of each defensive UUV is input into the policy network of the pre-trained multi-UUV collaborative decision-making model; the multi-UUV collaborative decision-making model is optimized based on the multi-agent dual-delay deep deterministic policy gradient algorithm. The action vectors for each UUV defense are calculated using the deterministic policy of the policy network; The expected forward velocity of each defensive UUV is determined based on the first motion component in the motion vector, and the expected yaw rate of each defensive UUV is determined based on the second motion component in the motion vector. The desired yaw rate and desired forward velocity are used as the continuous interception actions of each defensive UUV.

7. The method according to claim 1, characterized in that, The training process of the multi-UUV collaborative decision-making model includes: Initialize the network parameters, target network parameters, and experience buffer of the multi-UUV area protection task scenario and the multi-UUV collaborative decision-making model; and construct joint observation information for the training phase based on the task allocation matrix and the real-time situation information of the multi-UUV area protection task scenario. Based on joint observation information and task allocation matrix, the deterministic policy of the policy network in the multi-UUV collaborative decision-making model is used to generate continuous interception actions of each defensive UUV; and the continuous interception actions of each defensive UUV are spliced ​​into a joint action. Based on the joint action control, each defensive UUV performs actions in the multi-UUV area protection mission training scenario, obtaining mission execution status information and real-time situation information at the next training moment; and based on the mission allocation result, mission execution status information and reward function, calculate joint instant reward; and update the mission allocation matrix and the observation status information of each defensive UUV according to the real-time situation information at the next moment. The current joint observation information, joint actions, joint instant rewards, and the next time-in joint observation information are stored as experience data in the experience buffer. Empirical data is extracted from the empirical buffer, and the value network parameters and policy network parameters of the multi-UUV collaborative decision-making model are updated based on the empirical data and the network loss function. The target network parameters are softly updated according to a preset update interval, and a pre-trained multi-UUV collaborative decision-making model is obtained when the preset training termination condition is met.

8. The method according to claim 1, characterized in that, The reward function includes: in, These are the weighting coefficients. For the first Rewards for continuous attacks on infiltrated UUVs. For the first A reward for a successful attack that infiltrates a UUV. For the first A defensive UUV relative to the assigned target Continuous defense rewards Rewards for the UUV defense team.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the storage medium is located to perform the method according to any one of claims 1 to 8.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images