Multi-platform cooperative underwater target search reinforcement learning method and system

Abstract

The application provides a multi-platform cooperative underwater target search reinforcement learning method and system, comprising: constructing a target belief graph, a communication graph and an information freshness index; taking the communication link quality, information freshness and local target belief graph of each platform as a part of the state of multi-agent reinforcement learning, taking the motion control, search area selection, sensor working mode, communication action and relay role selection as a part of the action, and constructing a comprehensive reward function; in the training stage, the multi-agent strategy network is centrally trained by using global information, and in the execution stage, each platform only outputs actions according to local observation, the latest received compressed situation message and the local strategy network; each platform generates a compressed situation message according to the belief graph change, information freshness, link quality and task conflict degree, and sends the compressed situation message to adjacent platforms, and the receiving platform updates the target belief graph and the local task state in an incremental fusion manner.

Description

Technical Field

[0001] This invention relates to the field of underwater platform control technology, and in particular to a multi-platform collaborative underwater target search reinforcement learning method and system. Background Technology

[0002] Deep-sea underwater target searches typically require the collaboration of multiple platforms. Surface vessels provide continuous search and command and control, unmanned underwater platforms can conduct close-range underwater detection, aerial platforms can rapidly relocate within the search area, and sonobuoys can provide localized detection coverage and communication relay. During the search process, these platforms must simultaneously determine the search area, flight path, sensor operating modes, communication transmission timing, and relay role.

[0003] In existing technologies, multi-platform collaborative search often employs methods such as fixed partition search, grid coverage search, matrix game theory, heuristic task allocation, or single-platform deep reinforcement learning path planning. Regarding communication systems, situational information transmission is typically achieved through underwater acoustic communication, satellite communication, short-range communication, or buoy relay. While these methods can accomplish search or communication functions to a certain extent, search decisions and communication status are often disconnected, making it difficult to address issues of target uncertainty, platform heterogeneity, and limited links within a unified framework.

[0004] The existing technology has at least the following shortcomings: 1) Insufficient consideration of communication constraints. Most cooperative search methods assume that each platform can share the global situation in a timely manner. However, in actual underwater missions, underwater acoustic communication suffers from problems such as low bandwidth, high latency, packet loss, intermittent interruptions, and communication exposure costs, resulting in a significant lag in the situation information obtained by each platform.

[0005] 2) The search mission and communication scheduling lack closed-loop linkage. Existing buoy communication or communication capability assessment schemes usually focus on link transmission, wake-up, or capability prediction, and lack a mechanism to feed back link quality, information freshness, message compression, and relay role selection to search area allocation, route planning, and sensor mode control.

[0006] 3) Single-platform or centralized planning is difficult to adapt to heterogeneous multi-platform execution. Existing deep reinforcement learning search planning is mostly geared towards single-agent or centralized control scenarios. When there are differences in platform type, sensor coverage, energy status, and communication capabilities, centralized strategies are highly dependent on real-time communication, and distributed execution lacks robustness.

[0007] 4) Fixed strategy spaces struggle to adapt to dynamic task changes. Matrix games or pre-set tactical approaches typically rely on fixed strategy sets and pre-set payoff matrices. When the target maneuvers, buoy drifts, platform malfunctions, link failures, or sudden changes in local discovery information occur, it is difficult to adjust the collaborative search strategy in a timely manner.

[0008] Therefore, a technical solution is needed that can combine target belief graphs, communication graphs, information freshness, event-triggered communication, and multi-agent reinforcement learning, so that multiple platforms can still form an executable, interpretable, and continuously updated collaborative underwater target search strategy under communication-limited conditions. Summary of the Invention

[0009] This invention provides a multi-platform collaborative underwater target search reinforcement learning method and system to address the shortcomings of existing technologies, such as reliance on real-time global communication, communication link status not entering the search decision loop, information expiration affecting task allocation, and difficulty in distributed execution on heterogeneous platforms.

[0010] In a first aspect, the present invention provides a multi-platform collaborative underwater target search reinforcement learning method, comprising: Construct a target belief graph to characterize the probability of target existence, and a communication graph and information freshness index to characterize the status of inter-platform communication links and the timeliness of situational information; The communication link quality, information freshness, and local target belief graph of each platform are used as state components in multi-agent reinforcement learning, and motion control, search area selection, sensor working mode, communication action, and relay role selection are used as action components. A comprehensive reward function containing communication cost and information freshness penalty is constructed. During the training phase, global information is used to train the multi-agent policy network in a centralized manner. During the execution phase, each platform autonomously outputs actions based only on local observations, the most recently received compressed situational information, and the local policy network. Each platform conducts event-triggered communication based on changes in the belief map, information freshness, link quality, and the degree of task conflict, generates compressed situational messages and sends them to neighboring platforms, and the receiving platform performs incremental fusion updates on the target belief map and local task status.

[0011] According to the multi-platform collaborative underwater target search reinforcement learning method provided by the present invention, the target belief graph is constructed and updated in the following manner: Based on the target's prior distribution, target motion model, historical detection results, and platform sensor coverage, the search area is discretized into a grid or a set of particles. The prior existence probability is obtained by predicting and propagating the belief map of the previous moment, and then the prior existence probability is corrected based on the detection probability and false alarm probability observed by the sensors of each platform. Each platform maintains a local target belief map that is jointly updated by observations of the platform, the most recently received compressed situational information, and mission priors.

[0012] According to the multi-platform collaborative underwater target search reinforcement learning method provided by the present invention, the communication graph is constructed in the following manner: A communication edge is constructed based on the underwater acoustic communication, satellite communication, short-range communication, and buoy relay relationships between the platforms; The weights of each communication edge include the probability of successful communication, expected latency, available bandwidth, communication exposure cost, and communication energy consumption.

[0013] According to the reinforcement learning method for multi-platform collaborative underwater target search provided by the present invention, the information freshness index is the difference between the current time and the last effective update time of the corresponding situation information; When the freshness of the information exceeds a preset threshold, the system increases the communication priority of the corresponding information.

[0014] According to the reinforcement learning method for multi-platform collaborative underwater target search provided by the present invention, the conditions for event-triggered communication include at least one of the following: The change in the local target belief map exceeds the threshold; the information freshness exceeds the threshold; the probability of successful communication is higher than the sending threshold and the communication cost is lower than the cost threshold; a conflict occurs in the search areas of multiple platforms; the platform detects a suspected target.

[0015] According to the reinforcement learning method for multi-platform collaborative underwater target search provided by the present invention, the compressed situation message includes at least a message timestamp, sending platform identifier, key high-probability target area, low-confidence search area, communication link status, platform remaining energy, and message validity period. For the target belief graph, only send the regions of significant change or the low-rank compressed representation.

[0016] According to the multi-platform collaborative underwater target search reinforcement learning method provided by the present invention, the incremental fusion update includes: The receiving platform performs weighted fusion of the received compressed situational information based on message timestamp, source credibility, link latency, and local observation consistency. When a message has expired or conflicts with a local high-confidence observation, its fusion weight is reduced or a verification search is triggered.

[0017] According to the reinforcement learning method for multi-platform collaborative underwater target search provided by the present invention, when communication is interrupted, each platform continues to independently perform the search based on the local target belief map; When communication is restored, each platform incrementally synchronizes the observation results during the offline period using timestamps, message credibility, and information freshness to avoid old information overwriting new observations.

[0018] According to the multi-platform collaborative underwater target search reinforcement learning method provided by the present invention, the comprehensive reward function includes at least one of the following: The benefits include target discovery, effective coverage, communication costs, information freshness penalties, energy consumption costs, task risk or communication exposure costs, and platform task conflict or duplicate coverage penalties.

[0019] Secondly, the present invention also provides a multi-platform collaborative underwater target search reinforcement learning system, comprising: The module is used to construct a target belief graph to represent the probability of the existence of a target, as well as a communication graph and information freshness index to represent the status of inter-platform communication links and the timeliness of situational information. The modeling module is used to incorporate the communication link quality, information freshness, and local target belief graph of each platform as state components of multi-agent reinforcement learning, and to incorporate motion control, search area selection, sensor working mode, communication action, and relay role selection as action components, and to construct a comprehensive reward function that includes communication cost and information freshness penalty. The training module is used to centrally train the multi-agent policy network using global information during the training phase. During the execution phase, each platform autonomously outputs actions based solely on local observations, the most recently received compressed situational information, and the local policy network. The fusion module is used by each platform to conduct event-triggered communication based on changes in the belief map, information freshness, link quality, and the degree of task conflict, generate compressed situational messages and send them to neighboring platforms, and receive platforms to perform incremental fusion updates on the target belief map and local task status.

[0020] The multi-platform collaborative underwater target search reinforcement learning method and system provided by this invention have the following beneficial effects: 1) By explicitly writing the communication link quality and information freshness into the multi-agent state space, this invention enables the search strategy to perceive the risks of link interruption, delay and information expiration, thus avoiding over-reliance on real-time global situation under the condition of limited communication in deep sea.

[0021] 2) By using a target belief graph to express the probability and confidence of the target's existence, this invention can unify historical detection, missed detection areas, target motion models, and compressed situational information into the same decision object, making the platform's search area allocation based on probability.

[0022] 3) Due to the adoption of a centralized training and distributed execution framework, this invention can learn collaborative behavior by utilizing global information during the training phase, and allows each platform to make autonomous decisions based solely on local observations and limited communication messages during the execution phase, thereby improving executability in communication interruption scenarios.

[0023] 4) Due to the use of an event-triggered communication mechanism, this invention can communicate when the belief map changes significantly, information is too fresh, task conflicts occur, or a suspected target is discovered, while remaining silent under normal conditions, thereby reducing invalid communication, lowering energy consumption, and reducing the cost of communication exposure.

[0024] 5) By combining motion control, search area selection, sensor mode, communication action, and relay role selection as joint actions, the present invention can simultaneously optimize the perception, communication, and action processes, reduce redundant coverage, and increase the effective search benefit per unit time.

[0025] 6) Because it outputs communication trigger records, situation fusion records, coverage heatmaps, information freshness curves, and energy consumption results, this invention can provide interpretable evidence for task review, manual confirmation, and subsequent search adjustments. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating the multi-platform collaborative underwater target search reinforcement learning method provided by the present invention; Figure 2 This is a schematic diagram of the multi-platform collaborative search system structure provided by the present invention; Figure 3 This is a flowchart of centralized training distributed execution under communication-constrained conditions provided by the present invention; Figure 4 This is a schematic diagram of the overlay of the target belief diagram and the communication diagram provided by the present invention; Figure 5 This is a flowchart of event-triggered communication and situational awareness fusion provided by the present invention; Figure 6 This is a schematic diagram of the multi-platform collaborative underwater target search results provided by the present invention; Figure 7 This is a schematic diagram of the multi-platform collaborative underwater target search reinforcement learning system provided by the present invention; Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0029] To address the problems in existing technologies, such as reliance on real-time global communication for collaborative search, communication link status not being included in the search decision loop, information expiration affecting task allocation, and difficulty in distributed execution on heterogeneous platforms, the present invention aims to provide a reinforcement learning method for multi-platform collaborative underwater target search under communication-constrained conditions.

[0030] The core technical problem that this invention aims to solve is: how to construct a target belief graph and a communication graph under the conditions of low data rate, high latency, intermittent interruption, heterogeneous platform sensors, and uncertain target state in underwater acoustic communication, and how to enable each platform to autonomously generate search areas, motion control, sensor modes, communication timing, and relay role selection based on a multi-agent reinforcement learning framework with centralized training and distributed execution, based on local observations, limited communication messages, and information freshness.

[0031] Furthermore, this invention also addresses the situational synchronization problem when communication resources are limited. Specifically, it triggers communication when the target belief graph changes, the information freshness exceeds a threshold, the link quality meets the transmission conditions, or a search task conflicts. It updates the local belief graphs of each platform by compressing situational messages and using incremental fusion, thereby reducing invalid communication, redundant coverage, and energy consumption.

[0032] Figure 1 This is a flowchart illustrating the multi-platform collaborative underwater target search reinforcement learning method provided in this embodiment of the invention, as shown below. Figure 1 As shown, it includes: Step 100: Construct a target belief map to characterize the probability of target existence, and a communication map and information freshness index to characterize the status of inter-platform communication links and the timeliness of situational information; Step 200: The communication link quality, information freshness, and local target belief graph of each platform are used as state components of multi-agent reinforcement learning, and motion control, search area selection, sensor working mode, communication action, and relay role selection are used as action components. A comprehensive reward function including communication cost and information freshness penalty is constructed. Step 300: During the training phase, the multi-agent policy network is trained centrally using global information. During the execution phase, each platform autonomously outputs actions based solely on local observations, the most recently received compressed situational information, and the local policy network. Step 400: Each platform conducts event-triggered communication based on changes in the belief map, information freshness, link quality, and the degree of task conflict, generates compressed situational messages and sends them to neighboring platforms, and the receiving platform performs incremental fusion updates on the target belief map and local task status.

[0033] Specifically, embodiments of the present invention include, for example... Figure 2 As shown, it includes the following steps: Step S1: Search Task and Multi-Platform Status Initialization. Obtain the search area range, target prior information, platform set, initial platform position, platform motion constraints, sensor parameters, remaining energy, health status, sonar buoy position, communication method, link availability constraints, and task termination conditions. The platform set includes at least two types of vessels, unmanned underwater platforms, aerial search platforms, sonar buoys, and shore-based nodes.

[0034] Step S2: Target Belief Map Construction. Based on the target prior distribution, target motion model, historical detection results, missed detection areas, platform sensor coverage, and environmental detectability conditions, the search sea area is discretized into two-dimensional grids, three-dimensional grids, or particle sets to generate a target belief map that characterizes the probability and confidence level of target presence.

[0035] Step S3: Construction of Communication Graph and Information Freshness. Based on the underwater acoustic communication, satellite communication, short-range communication, and buoy relay relationships between platforms, a communication graph is constructed. The communication success probability, expected latency, available bandwidth, communication exposure cost, and communication energy consumption of each communication edge are calculated. The information freshness of situational information for each platform or grid is also calculated.

[0036] Step S4: Multi-agent reinforcement learning modeling of states, actions, and rewards. Local states are constructed from each platform's local observations, local target belief graphs, communication link quality vectors, information freshness, remaining energy, health status, historical actions, and task progress. Actions are constructed from motion control, search area selection, sensor patterns, communication actions, and relay role selection. A comprehensive reward is constructed based on target discovery probability, search coverage, communication cost, information freshness, energy consumption, task risk, and platform conflict.

[0037] Step S5: Centralized Training and Distributed Execution Policy Training. During the training phase, a centralized training environment is constructed using the global goal belief graph, the complete communication graph, the platform's global state, and task feedback to train the multi-agent policy network. During the execution phase, each platform only uses local observations, the most recently received compressed situational information, and the local policy network's output actions, without requiring continuous acquisition of the complete global situational awareness.

[0038] Step S6: Event-triggered communication and situational awareness fusion. Each platform determines whether to trigger communication based on changes in the belief map, information freshness, link quality, task conflict level, and suspected target discovery events. When triggered, a compressed situational awareness message is generated and sent to neighboring platforms via available links or relay platforms. The receiving platform incrementally merges and updates the target belief map, communication map, and local task status.

[0039] Step S7: Distributed cooperative search execution. Each platform, based on the trained policy network and updated local state, outputs the search area, motion control action, sensor operating mode, communication transmission timing, and whether it assumes a relay role for the next moment. If communication is interrupted, the search continues based on the local belief graph, and situational synchronization is performed when the link is restored.

[0040] Step S8: Output and Rolling Update of Results. Output multi-platform search trajectory, area coverage results, target discovery probability, target confirmation location, communication trigger record, situational message fusion record, energy consumption, information freshness curve, and task evaluation results; when the target status, platform status, or communication status changes, the target belief map and communication map are updated on a rolling basis, and steps S4 to S7 are continued.

[0041] In one embodiment, the construction of the target belief map in step S2 specifically includes the following: S2.1 Discretization of the search area. The search area is divided into a set of grids or a set of particles. Each grid or particle records the probability of the target's existence, confidence level, most recent update time, nearest observation platform, and searchable identifier. For three-dimensional underwater search scenarios, the depth layer, acoustic propagation conditions, and platform reachability constraints are also recorded.

[0042] S2.2 Prior Prediction Update. Based on the target's historical position, possible speed, possible course, maneuver model, and environmental constraints, the target belief map from the previous moment is propagated to obtain the prior probability of the target's existence at the current moment.

[0043] S2.3 Observation Correction Update. Based on sensor observations from each platform, missed detection areas where no target was detected, sensor detection probability, false alarm probability, and coverage area, the probability of prior target existence is corrected to increase the probability of confirmed target areas and decrease the probability of areas that are effectively covered but where no target was detected.

[0044] S2.4 Local Belief Maintenance. Each platform maintains a local target belief map, which is updated jointly by the platform's observations, the most recently received compressed situational awareness messages, and mission priors. When communication is interrupted, each platform continues to maintain the local belief map based on local observations and the target motion model.

[0045] In a preferred implementation, the grid At any moment Target belief value Represented as:

[0046] in, This represents a search grid or particle unit formed after the search area is discretized. This represents the predicted belief derived from the target motion model. Indicates the current observational contribution. This indicates the fusion contribution from receiving situational information. This indicates a deduction item that was effectively covered but did not result in the target being identified. These are the weighting coefficients. This is the normalization function.

[0047] Furthermore, the target belief map can be updated using predictive correction. Let the search grid set be... The target is a grid Transfer to grid The probability is , observation When the target is located in the grid The likelihood of time is ,but:

[0048]

[0049] in, Predictive beliefs before observational correction.

[0050] In one implementation, the observation likelihood is determined by the detection probability and the false alarm probability:

[0051] in, For the platform For grid The detection probability, This represents the probability of a false alarm. Indicates whether a suspected target has been detected. This indicates whether a false alarm has occurred. The local belief graph entropy can be represented as:

[0052] in, Indicates the first Each platform at any time For grid Maintained local goal belief values, It represents a small positive constant to prevent the logarithmic term from reaching zero, and is used to evaluate the uncertainty of the target distribution and the necessity of communication triggering.

[0053] In one embodiment, the communication-constrained state modeling and reward construction in steps S3 to S4 specifically include the following: Figure 4 As shown: S3.1 Constructing the communication graph. Let the platform set be... The communication diagram is represented as ,in This represents the set of currently available communication edges. Represents the set of communication edge weights; the edge weights include at least the probability of successful communication, expected latency, available bandwidth, communication exposure cost, and communication energy consumption.

[0054] S3.2, Calculate link quality. Platform With the platform Link quality between It can be represented as:

[0055]

[0056] in, For the link signal-to-noise ratio, For the platform The transmission power, For the platform transmit gain, For the platform Receiver gain For underwater sound propagation loss, Noise level, For bandwidth, The signal-to-noise ratio threshold for communication. For smoothing scale, This represents the Sigmoid normalization function. This represents the input variables of the function. The probability of successful communication. For normalized bandwidth, For normalized delay, To normalize communication energy consumption, The cost of exposing communication.

[0057] S3.3 Calculate information freshness. For the platform Holding from the platform or region The situational information, and the information freshness AoI are defined as the difference between the current time and the last valid update time of the information; for inter-platform messages, updates can be made as follows:

[0058] in, This represents the time interval between adjacent decision-making moments or information freshness update cycles. When the AoI exceeds the threshold, the system increases the communication priority of the corresponding message, platform, or region.

[0059] S4.1, Constructing Local States. Each platform at any time The local state is represented as:

[0060] in, Indicates position and attitude. Indicates speed, Indicates remaining energy. Indicates health status. Indicates local observation. Represents a local goal belief map. Represents the communication link quality vector. Indicators representing the freshness of information.

[0061] S4.2, Constructing the Action Space. The actions of each platform are represented as follows:

[0062] in, This indicates motion control or selection of a search area. Indicates the sensor's operating mode. Indicates a communication action. This indicates whether the user assumes a relay, collaborative guidance, or key search role.

[0063] S4.3 Construct the comprehensive reward function. The global training reward is represented as:

[0064] in, This indicates that the target has been discovered or its revenue has been recognized. This indicates effective coverage of revenue. Indicates the communication cost. This indicates a penalty for information freshness. This indicates the cost of energy consumption. This indicates the risk or cost of mission exposure or communication breaches. This indicates a platform penalty for task conflicts or duplicate overwriting. These are the weighting coefficients.

[0065] The target discovery benefit, effective coverage benefit, and conflict penalty can be further expressed as:

[0066]

[0067]

[0068] in, For the platform At any moment The planned search area. This expression enables the reward function to simultaneously encourage coverage of high-belief regions and suppress duplicate searches across multiple platforms.

[0069] In one embodiment, the training, communication, and execution in steps S5 to S8 specifically include the following: Figure 3 As shown: S5.1 Centralized Training. A multi-agent reinforcement learning algorithm is used to train the policy network. This algorithm includes MAPPO, QMIX, MADDPG, graph neural network reinforcement strategies, or combinations thereof. The centralized trainer has access to the global goal belief graph, the complete communication graph, the platform's global state, and global rewards to learn inter-platform cooperative behavior.

[0070] S5.1.1 Training Objective. The joint policy and cumulative discount reward objective can be expressed as:

[0071]

[0072] in, This is a set of strategy parameters for each platform. As a discount factor, Indicating a joint strategy The distribution of state-action trajectories generated by interaction with the training environment is taken as the mathematical expectation. This indicates the duration of a single training iteration. During the training phase, a centralized commentator or value function can be used to decompose the network; during the execution phase, only platform-specific policies are retained.

[0073] When the task requires limiting the number of communications, energy consumption, or exposure risks, the training objective can take the form of constraints:

[0074] in, To constrain the multipliers, These are budgets for communications, energy, and risk, respectively.

[0075] S5.2 Distributed Execution. After training, the policy parameters are distributed to each platform; during the execution phase, each platform executes based on its local state. Output Action The feasibility of actions is screened based on task constraints, communication constraints, and platform security constraints.

[0076] S6.1 Event trigger judgment, such as Figure 5As shown, communication is triggered if one of the following conditions is met: the entropy of the local target belief map decreases or changes by more than a threshold; the freshness of information in a certain area or on a certain platform exceeds a threshold; the probability of successful communication is higher than the sending threshold and the communication cost is lower than the cost threshold; a conflict occurs in the search areas of multiple platforms; or the platform detects a suspected target or the platform's health status is abnormal.

[0077] In one implementation, the first Event triggering variables for each platform Represented as:

[0078] in, For the change in the belief diagram, The change in entropy of the belief graph. To maximize information freshness, For the best available link quality, This is an indicator of task conflict. This is a suspected target detection marker. To trigger weights, The platform-adaptive trigger threshold.

[0079] S6.2, Compressed Situation Message Generation. A compressed situation message includes at least a message timestamp, sending platform identifier, key high-probability target areas, low-confidence search areas, communication link status, remaining platform energy, suggested collaborative actions, and message validity period. For the target belief graph, only significantly changed areas, summary vectors, or low-rank compressed representations are sent.

[0080] S6.3 Situational Fusion Update. The receiving platform performs weighted fusion of received messages based on message timestamp, source credibility, link latency, and local observation consistency. If a message has expired or conflicts with local high-confidence observations, its fusion weight is reduced or a verification search is triggered.

[0081] In one implementation, the receiving platform For neighboring platforms Set fusion weights for compressed situational messages:

[0082] And update the local belief map:

[0083] in, To ensure the credibility of the source, Indicates platform At any moment The set of communicable neighbors This indicates the index platform in the neighbor set. This is the information expiration decay coefficient. To prevent zero constant, The belief value recovered based on the neighbor's compressed situation message.

[0084] S7.1 Rolling Search Execution. Each platform executes search actions according to the strategy output and writes the detection results, communication events, and energy changes into the local task status; if the link is interrupted, the local strategy continues to search based on the local target belief map; if the link is restored, incremental synchronization is performed.

[0085] S8.1, Search result output, such as Figure 6 As shown, the system outputs the target confirmation location, search trajectories on each platform, coverage heatmap, communication event sequence, situational message fusion record, energy consumption, average information freshness, duplicate coverage ratio, and task success criteria, providing a basis for subsequent review, manual confirmation, or re-search.

[0086] The multi-platform cooperative underwater target search reinforcement learning system provided by the present invention is described below. The multi-platform cooperative underwater target search reinforcement learning system described below can be referred to in correspondence with the multi-platform cooperative underwater target search reinforcement learning method described above.

[0087] Figure 7 This is a schematic diagram of the structure of the multi-platform collaborative underwater target search reinforcement learning system provided in an embodiment of the present invention, as shown below. Figure 7 As shown, it includes: a construction module 71, a modeling module 72, a training module 73, and a fusion module 74, wherein: The construction module 71 is used to construct a target belief map to represent the probability of target existence, and a communication map and information freshness index to represent the communication link status and situation information timeliness between platforms. The modeling module 72 is used to incorporate the communication link quality, information freshness and local target belief map of each platform as state components of multi-agent reinforcement learning, and to incorporate motion control, search area selection, sensor working mode, communication action and relay role selection as action components, and to construct a comprehensive reward function that includes communication cost and information freshness penalty. The training module 73 is used to centrally train the multi-agent policy network using global information during the training phase, and during the execution phase, each platform autonomously outputs actions based only on local observations, the most recently received compressed situation messages and the local policy network. The fusion module 74 is used for each platform to conduct event-triggered communication based on belief map changes, information freshness, link quality and task conflict degree, generate compressed situation messages and send them to neighboring platforms, and receive platforms to incrementally fuse and update the target belief map and local task status.

[0088] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a multi-platform cooperative underwater target search reinforcement learning method. This method includes: constructing a target belief map to characterize the probability of target existence, and a communication map and information freshness index to characterize the communication link status and situational information timeliness between platforms; using the communication link quality, information freshness, and local target belief map of each platform as state components of the multi-agent reinforcement learning, and using motion control, search area selection, sensor operating mode, communication actions, and relay role selection as action components; constructing a comprehensive reward function that includes communication cost and information freshness penalty; in the training phase, using global information to centrally train the multi-agent policy network; in the execution phase, each platform autonomously outputs actions based solely on local observations, the most recently received compressed situational information, and the local policy network; each platform performs event-triggered communication based on changes in the belief map, information freshness, link quality, and task conflict degree, generating compressed situational information and sending it to neighboring platforms; and receiving platforms incrementally fusion-updating the target belief map and local task status.

[0089] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0090] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0091] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-platform collaborative underwater target search reinforcement learning method, characterized in that, include: Construct a target belief graph to characterize the probability of target existence, and a communication graph and information freshness index to characterize the status of inter-platform communication links and the timeliness of situational information; The communication link quality, information freshness, and local target belief graph of each platform are used as state components in multi-agent reinforcement learning, and motion control, search area selection, sensor working mode, communication action, and relay role selection are used as action components. A comprehensive reward function including communication cost and information freshness penalty is constructed. During the training phase, global information is used to train the multi-agent policy network in a centralized manner. During the execution phase, each platform autonomously outputs actions based only on local observations, the most recently received compressed situational information, and the local policy network. Each platform conducts event-triggered communication based on changes in the belief map, information freshness, link quality, and the degree of task conflict, generates compressed situational messages and sends them to neighboring platforms, and the receiving platform performs incremental fusion updates on the target belief map and local task status.

2. The multi-platform collaborative underwater target search reinforcement learning method according to claim 1, characterized in that, The target belief map is constructed and updated in the following ways: Based on the target's prior distribution, target motion model, historical detection results, and platform sensor coverage, the search area is discretized into a grid or a set of particles. The prior existence probability is obtained by predicting and propagating the belief map of the previous moment, and then the prior existence probability is corrected based on the detection probability and false alarm probability observed by the sensors of each platform. Each platform maintains a local target belief map that is jointly updated by observations of the platform, the most recently received compressed situational information, and mission priors.

3. The multi-platform collaborative underwater target search reinforcement learning method according to claim 1, characterized in that, The communication graph is constructed in the following way: A communication edge is constructed based on the underwater acoustic communication, satellite communication, short-range communication, and buoy relay relationships between the platforms; The weights of each communication edge include the probability of successful communication, expected latency, available bandwidth, communication exposure cost, and communication energy consumption.

4. The multi-platform collaborative underwater target search reinforcement learning method according to claim 1, characterized in that, The information freshness index is the difference between the current moment and the last valid update time of the corresponding situation information; When the freshness of the information exceeds a preset threshold, the system increases the communication priority of the corresponding information.

5. The multi-platform collaborative underwater target search reinforcement learning method according to claim 1, characterized in that, The conditions for the event to trigger communication include at least one of the following: The change in the local target belief map exceeds the threshold; the information freshness exceeds the threshold; the probability of successful communication is higher than the sending threshold and the communication cost is lower than the cost threshold; a conflict occurs in the search areas of multiple platforms; the platform detects a suspected target.

6. The multi-platform collaborative underwater target search reinforcement learning method according to claim 1, characterized in that, The compressed situational message includes at least a message timestamp, sending platform identifier, key high-probability target area, low-confidence search area, communication link status, platform remaining energy, and message validity period. For the target belief graph, only send the regions of significant change or the low-rank compressed representation.

7. The multi-platform collaborative underwater target search reinforcement learning method according to claim 1, characterized in that, The incremental fusion update includes: The receiving platform performs weighted fusion of the received compressed situational information based on message timestamp, source credibility, link latency, and local observation consistency. When a message has expired or conflicts with a local high-confidence observation, its fusion weight is reduced or a verification search is triggered.

8. The multi-platform collaborative underwater target search reinforcement learning method according to any one of claims 1 to 7, characterized in that, When communication is interrupted, each platform continues to perform the search independently based on the local target belief map; When communication is restored, each platform incrementally synchronizes the observation results during the offline period using timestamps, message credibility, and information freshness to avoid old information overwriting new observations.

9. The multi-platform cooperative underwater target search reinforcement learning method according to any one of claims 1 to 7, characterized in that, The comprehensive reward function includes at least one of the following: The benefits include target discovery, effective coverage, communication costs, information freshness penalties, energy consumption costs, task risk or communication exposure costs, and platform task conflict or duplicate coverage penalties.

10. A multi-platform collaborative underwater target search reinforcement learning system, characterized in that, include: The module is used to construct a target belief graph to represent the probability of the existence of a target, as well as a communication graph and information freshness index to represent the status of inter-platform communication links and the timeliness of situational information. The modeling module is used to incorporate the communication link quality, information freshness, and local target belief graph of each platform as state components of multi-agent reinforcement learning, and to incorporate motion control, search area selection, sensor working mode, communication action, and relay role selection as action components, and to construct a comprehensive reward function that includes communication cost and information freshness penalty. The training module is used to centrally train the multi-agent policy network using global information during the training phase. During the execution phase, each platform autonomously outputs actions based solely on local observations, the most recently received compressed situational information, and the local policy network. The fusion module is used by each platform to conduct event-triggered communication based on changes in the belief map, information freshness, link quality, and the degree of task conflict, generate compressed situational messages and send them to neighboring platforms, and receive platforms to perform incremental fusion updates on the target belief map and local task status.

Images