Maritime cruise instant deduction method and system based on reinforcement learning and war game deduction

Abstract

This application provides a real-time simulation method and system for maritime patrols based on reinforcement learning and wargaming, belonging to the field of intelligent scheduling and simulation technology. This application obtains the oil spill location coordinates, ocean current velocity vectors, and the position coordinates and remaining endurance of each patrol vessel to form an initial state set. Using a simulation sandbox as the simulation environment, the agent submits control commands at each time step. The simulation sandbox synchronously updates the edge position coordinates, simulated position coordinates, and remaining endurance. The initial scheduling strategy is iterated until convergence, using spatial coverage and response time as rewards, and a target scheduling strategy is output. After the target scheduling strategy is converted into scheduling commands and issued, the difference between the transmitted position and the simulated position coordinates is accumulated to obtain the cumulative offset. The maximum offset is converted into a trigger interval duration through an inverse proportional mapping. After the timeout, the sandbox is updated and the target scheduling strategy is regenerated. This application achieves high coverage, rapid response, and closed-loop update at the oil spill diffusion front.

Description

Technical Field

[0001] This application belongs to the field of intelligent scheduling and simulation technology, and in particular relates to a real-time simulation method and system for maritime patrol based on reinforcement learning and wargaming. Background Technology

[0002] Following a marine oil spill, the oil continues to spread with ocean currents. Emergency patrol vessels, limited by their range, only need to conduct rapid coverage patrols of the oil spill's leading edge. The quality of patrol scheduling directly determines the timeliness of monitoring the oil spill's spread and is a core element of maritime emergency response. Combining simulation and intelligent scheduling holds promise for generating adaptive patrol paths to address dynamic spread patterns, demonstrating significant application value in the field of marine oil spill emergency response.

[0003] Existing research has introduced reinforcement learning methods into the scenario of multi-ship cooperative mission planning at sea. By constructing a simulation environment to model the multi-ship path allocation problem, and using coverage or arrival time as optimization objectives, an agent is trained in the simulation environment to generate multi-ship cooperative control strategies. Some studies further combine simulation sandboxes to simulate and model the motion process of dynamic targets, enabling the agent to iteratively generate scheduling strategies in scenarios where targets are constantly changing.

[0004] However, the aforementioned methods typically execute the scheduling strategy in a fixed manner after its generation, failing to incorporate the actual positional deviation of the vessel during execution into the triggering criteria for strategy correction. This results in the generated scheduling strategy being unable to respond promptly to changes in the accumulated deviation when the vessel's actual position deviates from the predetermined projected position due to sea state disturbances. As the deviation continues to accumulate, the strategy's coverage effect continuously declines without correction. Therefore, existing technologies suffer from technical problems such as insufficient real-time projection during maritime patrols due to the inability of the patrol scheduling strategy to adaptively trigger corrections based on the actual positional deviation during execution and the continuous degradation of coverage effect in dynamic scenarios. Summary of the Invention

[0005] The purpose of this application is to provide a method and system for real-time simulation of maritime patrols based on reinforcement learning and wargaming, so as to solve the problem of insufficient real-time simulation of maritime patrols in the prior art.

[0006] To address the aforementioned technical problems, in a first aspect, this application provides a real-time simulation method for maritime patrols based on reinforcement learning and wargaming, comprising: Obtain the oil spill location coordinates, ocean current velocity vector, and the location coordinates and remaining range of each patrol vessel to form an initial state set; Using the constructed simulation sandbox as the simulation environment, based on the initial state set and the initial scheduling strategy, in each time step, the agent submits the control commands of each patrol vessel to the simulation sandbox. The simulation sandbox updates the edge position coordinates of the oil spill diffusion area and the simulation position coordinates and remaining endurance of each patrol vessel, which are then returned to the agent as the observation vector for the next time step. Within each time step, the initial scheduling strategy is iteratively updated with the spatial coverage of each patrol vessel to the edge location coordinates and the response time of each patrol vessel to the edge of the oil spill diffusion area as reward values ​​until convergence to obtain the target scheduling strategy, which includes the control command sequence of the patrol vessels. After the target scheduling strategy is converted into scheduling instructions, they are sent to each patrol vessel and the actual position is received back. The difference between the actual position and the simulated position coordinates of the corresponding patrol vessel in the simulation sand table is calculated to obtain the spatial offset of each patrol vessel. The spatial offsets are accumulated within a preset time window to obtain the cumulative offset of each patrol vessel. The maximum cumulative offset is converted into the trigger interval duration through an inverse proportional mapping relationship. When the timer reaches the trigger interval duration, the simulation sand table is updated with the actual positions returned by each patrol vessel to regenerate a new target scheduling strategy.

[0007] Optionally, using the constructed simulation sandbox as the simulation environment, based on the initial state set and initial scheduling strategy, within each time step, the agent submits control commands for each patrol vessel to the simulation sandbox, including: Load the oil spill location coordinates and ocean current velocity vector from the initial state set into the oil spill diffusion object in the simulation sandbox, and load the location coordinates and remaining range of each patrol vessel into the vessel object in the simulation sandbox. Set the time step of the simulation sand table. Within each time step, extract the set of edge position coordinates of the oil spill diffusion area, the simulation position coordinates of each patrol vessel, and the remaining endurance, and stitch them together to form the observation vector of the intelligent agent. The agent processes the observation vectors according to the initial scheduling strategy, generates control commands for each patrol vessel, including speed and heading angle, and submits them to the simulation sand table.

[0008] Optionally, the initial scheduling strategy includes a sequence of initial control commands generated based on preset speed and direction allocation rules; The agent processes the observation vectors according to the initial scheduling strategy, generates control commands for each patrol vessel, including speed and heading angles, and submits them to the simulation sandbox, including: Extract the set of edge position coordinates and the projected position coordinates of each patrol vessel from the observation vector, and calculate the distribution range of the direction from the oil spill position coordinates to the set of edge position coordinates; According to the direction allocation rules, the direction distribution range is divided into multiple coverage sectors. The angle value of the simulated position coordinates of each patrol vessel pointing to the center position of the corresponding coverage sector is calculated to obtain the heading angle. Using a preset speed as the speed value, the speed value and heading angle are used as control commands for each patrol vessel and submitted to the simulation sand table.

[0009] Optionally, the simulation sand table updates the edge coordinates of the oil spill diffusion area and the simulated position coordinates and remaining range of each patrol vessel, which are then returned to the agent as the observation vector for the next time step, including: Based on the oil spill location coordinates and ocean current velocity vector, the oil spill diffusion object in the simulation sand table is calculated. The set of edge location coordinates of the oil spill diffusion area after its outward expansion at the end of each time step is obtained, and the updated set of edge location coordinates is obtained. Based on position coordinates and control commands, the ship objects in the simulation sand table calculate the simulation position coordinates of each patrol ship at the end of each time step under the constraints of maximum speed and remaining range of each patrol ship, and obtain the updated simulation position coordinates. The consumed range is calculated based on the speed value and time step, and the difference between the remaining range and the remaining range is calculated to obtain the updated remaining range. The updated set of edge position coordinates, the inferred position coordinates, and the remaining endurance are concatenated and returned to the agent as the observation vector for the next time step.

[0010] Optionally, within each time step, the initial scheduling strategy is iteratively updated using the spatial coverage of each patrol vessel's edge location coordinates and the response time of each patrol vessel to the edge of the oil spill diffusion area as reward values ​​until the target scheduling strategy is converged, including: Within each time step, the distance between the projected position coordinates of each patrol vessel and each coordinate point in the edge position coordinate set is calculated. The spatial coverage rate is obtained based on the ratio of the number of coordinate points with a distance less than the preset coverage threshold to the total number of coordinate points in the edge position coordinate set. The ratio of the minimum distance to the speed value in the control command is calculated to obtain the response time of each patrol vessel to reach the edge of the oil spill spread area. The reward value for each time step is obtained by weighted summation of spatial coverage and response time. The initial scheduling policy is iteratively updated using the reward value until convergence to obtain the target scheduling policy.

[0011] Optionally, the initial scheduling policy is iteratively updated using the reward value until convergence is obtained to obtain the target scheduling policy, including: Calculate the reward difference between the reward value of the current time step and the reward value of the previous time step. When the reward difference is less than or equal to zero, the angle range of each coverage sector is re-divided using a preset adjustment step size. The direction allocation rules are updated with the angle range of each coverage sector after re-division. Based on the updated direction allocation rules, the heading angle and control commands of each patrol vessel are regenerated to recalculate the reward value until the reward difference is greater than zero and the absolute value is less than the preset convergence threshold. The control command sequence of each patrol vessel generated by the direction allocation rules at the time of convergence is output as the target scheduling strategy.

[0012] Optionally, the maximum cumulative offset is converted into a trigger interval duration through an inverse proportional mapping relationship. When the timer reaches the trigger interval duration, the simulation sand table is updated with the actual positions returned by each patrol vessel to regenerate a new target scheduling strategy, including: Extract the maximum cumulative offset from all cumulative offsets, and obtain the trigger interval duration based on the ratio of the preset inverse proportional coefficient to the maximum cumulative offset; When the timeout period reaches the trigger interval, the latest actual position transmitted back by each patrol vessel is extracted, and the simulated position coordinates of each patrol vessel in the simulation sand table are updated using the actual position. Based on the updated simulated position coordinates, a revised target scheduling strategy is generated.

[0013] Secondly, this application provides a real-time simulation system for maritime patrols based on reinforcement learning and wargaming, comprising: The acquisition module is used to acquire the oil spill location coordinates, ocean current velocity vector, and the location coordinates and remaining range of each patrol vessel to form an initial state set. The update module is used to simulate the oil spill spread area using the constructed simulation sandbox as the simulation environment. Based on the initial state set and the initial scheduling strategy, the agent submits control commands for each patrol vessel to the simulation sandbox at each time step. The simulation sandbox updates the edge position coordinates of the oil spill spread area and the simulation position coordinates and remaining range of each patrol vessel, which are then returned to the agent as the observation vector for the next time step. The update module is also used to iteratively update the initial scheduling strategy at each time step, using the spatial coverage of each patrol vessel to the edge location coordinates and the response time of each patrol vessel to the edge of the oil spill diffusion area as reward values, until the target scheduling strategy is converged. The target scheduling strategy includes the control command sequence of the patrol vessels. The calculation module is used to convert the target scheduling strategy into scheduling instructions and send them to each patrol vessel. It also receives the actual position returned by the patrol vessel. The actual position is calculated by the difference between the actual position and the simulated position coordinates of the corresponding patrol vessel in the simulation sand table to obtain the spatial offset of each patrol vessel. The spatial offset is accumulated within a preset time window to obtain the cumulative offset of each patrol vessel. The generation module is used to convert the maximum cumulative offset into the trigger interval duration through an inverse proportional mapping relationship. When the timer reaches the trigger interval duration, the simulation sand table is updated with the actual positions returned by each patrol vessel to regenerate a new target scheduling strategy.

[0014] Thirdly, this application provides an electronic device, comprising: Memory, used to store computer programs; A processor, configured to implement the steps of the real-time simulation method for maritime patrol based on reinforcement learning and wargaming as described in the first aspect above, when executing the computer program.

[0015] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps of the real-time simulation method for maritime patrol based on reinforcement learning and wargaming as described in the first aspect above.

[0016] The real-time simulation method for maritime patrol based on reinforcement learning and wargaming provided in this application uses a simulation sand table as the simulation environment. It integrates the edge coordinates of the oil spill diffusion area with the simulation coordinates of each patrol vessel and their remaining endurance into an observation vector. The initial scheduling strategy is continuously iterated and converged with spatial coverage and response time as reward values, so that the generated target scheduling strategy can achieve a high coverage and rapid response to the oil spill diffusion front under the constraint of endurance.

[0017] During the strategy execution phase, the difference between the actual and projected location coordinates is accumulated within a preset time window. The trigger interval is dynamically determined using an inverse proportional mapping relationship based on the maximum accumulated offset. The larger the deviation, the more timely the trigger, effectively preventing the continuous accumulation of deviations from deteriorating coverage. This solves the technical problem in existing technologies where patrol scheduling strategies cannot adaptively trigger corrections based on the degree of actual location deviation, and where coverage continuously deteriorates in dynamic scenarios.

[0018] Furthermore, this application loads the oil spill location coordinates and ocean current velocity vectors onto the oil spill diffusion object, and loads the location coordinates and remaining range of each patrol vessel onto the vessel object. Within each time step, the edge location coordinate set, the projected location coordinates, and the remaining range are stitched together into an observation vector. This enables the agent to perceive the real-time dynamics of the oil spill diffusion front and the current status of each patrol vessel within each time step, providing a complete state basis for generating control commands including speed values ​​and heading angles. This ensures the continuous alignment of scheduling decisions with the oil spill diffusion situation during the projection process. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating a real-time simulation method for maritime patrol based on reinforcement learning and wargaming, provided for an embodiment of this application; Figure 2 A schematic diagram illustrating the initial coverage sector division and course generation process for maritime patrols, provided in this application embodiment; Figure 3 A schematic diagram illustrating a control strategy adjustment process provided in an embodiment of this application; Figure 4 This is a schematic diagram of an iterative optimization process for reinforcement learning provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a real-time simulation system for maritime patrol based on reinforcement learning and wargaming, provided as an embodiment of this application. Detailed Implementation

[0021] Existing methods for multi-ship scheduling at sea that combine reinforcement learning with simulation sandboxes typically iterate strategies in a simulation environment before issuing and executing fixed scheduling strategies. Strategy corrections rely entirely on manual judgment or fixed-period triggering, failing to establish a quantitative mechanism for the deviation between the actual and simulated positions of ships. When ships experience positional deviations due to sea state disturbances during navigation, there is a lack of correlation between the degree of deviation and the timing of strategy corrections. This leads to frequent corrections for minor deviations, resulting in unnecessary resource consumption, while severe deviations, due to the failure to trigger corrections in a timely manner, cause the scheduling strategy to remain inaccurate. Consequently, the coverage effect of the oil spill front deteriorates continuously as deviations accumulate.

[0022] To address the aforementioned issues, this application proposes a real-time simulation method for maritime patrols based on reinforcement learning and wargaming. Its core lies in combining the iterative generation of a target scheduling strategy using a simulation sandbox with adaptive closed-loop correction based on actual positional deviations. Specifically, this application uses the spatial coverage and response time of each patrol vessel at the oil spill spread edge as reward values ​​to drive the initial scheduling strategy to continuously iterate and converge in the simulation sandbox, generating a target scheduling strategy covering the front line.

[0023] During the strategy execution phase, the difference between the actual position and the simulated position coordinates transmitted by the ships is continuously accumulated within a preset time window. The trigger interval is dynamically calculated using the largest cumulative offset among all ships through an inverse proportional mapping relationship. The larger the deviation, the shorter the trigger interval. This reduces unnecessary corrections when the deviation is slight and promptly drives the simulation sandbox to regenerate the target scheduling strategy based on the actual position when the deviation is severe.

[0024] This method directly applies the quantified results of actual execution deviation to the triggering time of strategy correction, abandoning the indiscriminate processing method of fixed-period correction, and solving the technical problem of continuous degradation of coverage effect in dynamic scenarios due to the lack of deviation perception mechanism in the existing technology.

[0025] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] To address the problems of existing technologies, embodiments of this application provide a method, apparatus, device, computer storage medium, and computer program product for real-time maritime patrol simulation based on reinforcement learning and wargaming. The method for real-time maritime patrol simulation based on reinforcement learning and wargaming provided in this application embodiment will be described below first.

[0027] Figure 1 This diagram illustrates a flowchart of a real-time maritime patrol simulation method based on reinforcement learning and wargaming, provided in one embodiment of this application. Figure 1 As shown, the method includes: S101. Obtain the oil spill location coordinates, ocean current velocity vector, and the location coordinates and remaining range of each patrol vessel to form an initial state set.

[0028] The initial state set refers to the data set formed at the moment of the oil spill, which integrates the data describing the starting location of the oil spill spread, the state of ocean current movement, and the current position and available endurance of each patrol vessel. It includes four types of data: oil spill location coordinates, ocean current velocity vector, position coordinates of each patrol vessel, and remaining endurance.

[0029] Oil spill location coordinates refer to the geographical coordinates of the oil spill location, expressed in latitude and longitude. Ocean current velocity vector refers to the combination of the direction and magnitude of the current in the current sea area, expressed as a two-dimensional vector including both the direction angle and the velocity value. Remaining range refers to the maximum distance each patrol vessel can continue to travel under its current fuel conditions, expressed in nautical miles.

[0030] Specifically, after an oil spill occurs, the first step is to obtain the geographical coordinates of the oil spill point through a marine environmental monitoring network, thus obtaining the oil spill location coordinates. This represents the longitude and latitude of the oil spill point. Simultaneously, the ocean current velocity vector and sea surface temperature at the moment of the accident are obtained to obtain the ocean current velocity vector. ,in The direction angle of the ocean current. For the magnitude of the current and the sea surface temperature The unit is Celsius.

[0031] Next, the current location coordinates and remaining range of each patrol vessel participating in the emergency response are obtained, assuming a total of patrol vessel, number The coordinates of the ships are as follows The corresponding remaining battery life is denoted as The set of position coordinates of each patrol vessel is then given. The remaining battery life is Ultimately, it will , , , and Integrate to form an initial state set .

[0032] S102. Using the constructed simulation sandbox as the simulation environment, based on the initial state set and the initial scheduling strategy, within each time step, the agent submits control commands for each patrol vessel to the simulation sandbox. The simulation sandbox updates the edge position coordinates of the oil spill diffusion area and the simulation position coordinates and remaining range of each patrol vessel, which are then returned to the agent as the observation vector for the next time step.

[0033] Optionally, step S102, using the constructed simulation sandbox as the simulation environment, and based on the initial state set and initial scheduling strategy, the process of the agent submitting control commands for each patrol vessel to the simulation sandbox at each time step can specifically include: S1021. Load the oil spill location coordinates and ocean current velocity vector from the initial state set into the oil spill diffusion object in the simulation sand table, and load the location coordinates and remaining range of each patrol vessel into the vessel object in the simulation sand table.

[0034] A simulation sandbox is a modeling environment created by discretizing the marine area affected by an oil spill. The sandbox maintains two types of simulation entities: oil spill diffusion objects and vessel objects. Oil spill diffusion objects are the simulation entities in the sandbox responsible for simulating the continuous outward expansion of the oil spill under the influence of ocean currents, including the initial location of the oil spill and the ocean current velocity vector required for diffusion calculations. Vessel objects are the simulation entities in the sandbox responsible for simulating the navigation status of each patrol vessel, including the current simulated position coordinates and remaining range of each patrol vessel.

[0035] Specifically, firstly, the initial state set oil spill location coordinates and ocean current velocity vector The oil spill diffusion objects are loaded into the simulation sandbox, and the oil spill diffusion objects are... As the starting point of diffusion, with Direction angle in and flow rate This serves as a parameter driving the oil spill to expand outwards. Next, the position coordinates of each patrol vessel are collected. and remaining battery life Ship objects loaded into the simulation sandbox, of which the first The ship objects corresponding to each ship are As the initial calculated position coordinates, with This represents the initial remaining battery life.

[0036] S1022. Set the time step of the simulation sand table. Within each time step, extract the set of edge position coordinates of the oil spill diffusion area, the simulation position coordinates of each patrol vessel, and the remaining range, and stitch them together to form the observation vector of the intelligent agent.

[0037] Time step This refers to the fixed time interval, measured in minutes, between each state update of the simulation sandbox, determining the frequency of state updates for oil spill propagation objects and vessel objects. The edge position coordinate set refers to the set of coordinates of the leading edge position of the oil spill propagation area after its outward expansion at the end of the current time step, denoted as... and For the first One edge coordinate point, This represents the total number of edge coordinate points.

[0038] The simulated position coordinates refer to the current position coordinates of each patrol vessel in the simulation sand table after being updated according to control commands, and are denoted as the first... The projected position coordinates of the ship at the current time step are: Observation vector It refers to a one-dimensional numerical sequence formed by sequentially splicing together the set of edge position coordinates within the current time step, the projected position coordinates of each patrol vessel, and the remaining range.

[0039] Specifically, the time step of the simulation sand table is first set. like Minutes, simulation sand table according to The state of the oil spill dispersion object and the vessel object is updated alternately at intervals. Within each time step, the current edge position coordinates are extracted from the oil spill dispersion object. Extract the inferred position coordinates from each ship object. and remaining battery life Next, , and By concatenating these elements sequentially, the observation vector for the current time step is formed. .

[0040] S1023. The agent processes the observation vectors according to the initial scheduling strategy, generates control commands for each patrol vessel, including speed and heading angle, and submits them to the simulation sand table.

[0041] The initial scheduling strategy refers to the initial decision rules configured for the agent at the start of the simulation cycle, including the initial control command sequence generated based on preset speed and direction allocation rules. The preset speed refers to the navigation speed pre-set for each patrol vessel in the initial scheduling strategy, and the direction allocation rule refers to the allocation rule in the initial scheduling strategy that specifies the proportion of the angle interval within the directional distribution range to be covered by each patrol vessel, including the corresponding allocation proportion for each patrol vessel.

[0042] Control commands refer to the combinations of navigation parameters generated by the agent for each patrol vessel at each time step, including speed and heading angle. It refers to the first The ship's speed and heading angle within the current time step. It refers to the first The heading angle of a ship within the current time step.

[0043] Table 1 shows the specific parameter settings for the preset speed and direction allocation rules included in the initial scheduling strategy in this embodiment: Table 1: Initial Scheduling Strategy Parameter Comparison Table

[0044] As shown in Table 1, the preset speed of each patrol vessel in Table 1 is used as the speed value of each time step in the initial control command sequence. The direction allocation ratio determines the size of the angle range that each patrol vessel is responsible for covering in the direction distribution range. The sum of the direction allocation ratios of all patrol vessels is 100%, covering the entire direction distribution range.

[0045] Specifically, the intelligent agent receives the observation vector. Then, according to the initial scheduling strategy, The process involves three sub-steps, S10231, S10232, and S10233, which sequentially generate control commands for each patrol vessel and submit them to the simulation sand table. For details, please refer to the sub-step descriptions below.

[0046] In this process, the initial coverage sector division and course generation for maritime patrols are specifically as follows: Figure 2 As shown. Figure 2 The diagram shows the polygonal boundary 220 of the oil spill diffusion area, which evolves from the oil spill center point 210. The diffusion range is divided into multiple non-overlapping coverage sectors by the coverage sector boundary 230 generated through direction assignment rules. For each independent coverage sector, the corresponding center position ray is solved and intersected with the polygonal boundary to obtain the corresponding center position 240.

[0047] At this time, the simulated position 250 of the patrol vessels distributed at different coordinate points in the sea area is used as the starting node. By calculating the absolute great circle azimuth of the patrol vessel simulated position 250 pointing to the center position 240 of the covered sector, a unique heading angle 260 is assigned to each physical entity object as a control parameter.

[0048] Optionally, the initial scheduling strategy in step S1023 includes an initial control command sequence generated based on preset speed and direction allocation rules.

[0049] Step S1023, where the intelligent agent processes the observation vectors according to the initial scheduling strategy, generates control commands for each patrol vessel including speed and heading angles, and submits them to the simulation sand table, can specifically include: S10231. Extract the set of edge position coordinates and the projected position coordinates of each patrol vessel from the observation vector, and calculate the distribution range of the direction from the oil spill position coordinates to the set of edge position coordinates.

[0050] Directional distribution range refers to the range based on the coordinates of the oil spill location. Using the reference point as the reference point, the set of coordinates pointing to the edge position. The angular interval covered by the orientation angles of all edge coordinate points in the range is denoted as . and It is the minimum value among all direction angles. This is the maximum value among all direction angles, which increase clockwise with true north as 0°.

[0051] Specifically, first from the observation vector Extract the set of edge position coordinates and the projected position coordinates of each patrol vessel Next, using the coordinates of the oil spill location... Using the reference point, through and The difference between latitude and longitude was calculated. point to Each edge coordinate point Direction angle For all acquired After sorting in ascending order, calculate the angular gap between adjacent angles. Extract the angular gap with the largest value, and define the complement interval of this largest gap as the directional distribution range. .

[0052] S10232. Divide the direction distribution range into multiple coverage sectors according to the direction allocation rules, calculate the angle value of the predicted position coordinates of each patrol vessel pointing to the center position of the corresponding coverage sector, and obtain the heading angle.

[0053] Coverage sector refers to the range of sectors distributed according to directional allocation rules. After being divided, the angle sub-intervals to be covered by each patrol vessel are assigned and the first... The coverage sector corresponding to each ship is denoted as The center of the covered sector Refers to the coverage sector The midpoint direction angle corresponding to the angle sub-interval. Heading angle. This refers to the projected position coordinates of each patrol vessel. Points to the center of the corresponding covered sector The angle value of the corresponding sea area target point and through Along The difference in latitude and longitude of the target point in the direction is calculated.

[0054] Specifically, firstly, based on the directional allocation ratio of each patrol vessel in Table 1, the directional distribution range is determined. Divided proportionally into A continuous coverage sector. The total span of the directional distribution range is... , No. The coverage sector corresponding to each ship The angular span is the sum of the total span and the distribution ratio. The product of.

[0055] Next, calculate the center direction angle of each coverage sector, the first... Each ship corresponds to a coverage sector Central direction angle The starting angle of this sector and the ending angle The average value is Finally, adjust the center direction angle. The set of coordinates of the ray and the edge of the oil spill. The geometric intersections of the resulting polygonal boundaries are determined, and the nearest intersection is taken as the target point in the sea area. The calculated position coordinates of each patrol vessel are then determined using the great circle course azimuth formula. The angle value pointing to this absolute geographic target point yields the heading angle. .

[0056] S10233. Using the preset speed as the speed value, the speed value and heading angle are used as control commands for each patrol vessel and submitted to the simulation sand table.

[0057] Speed ​​value This refers to the execution speed of each patrol vessel at this time step, which is directly obtained from the preset speed of the initial scheduling strategy in Table 1.

[0058] Specifically, firstly, the preset speeds of each patrol vessel are extracted sequentially from Table 1 as speed values, resulting in... , , , Next, the speed values ​​of each patrol vessel were recorded. and heading angle Encapsulated as control commands Obtain the set of control commands Finally, the control command set... Submitted to the simulation sandbox, each vessel in the simulation sandbox receives the corresponding control commands. Under the constraints of maximum speed and remaining range, according to and Update the projected position coordinates of each patrol vessel. and remaining battery life Complete the status update for the current time step.

[0059] This embodiment enables each patrol vessel to cover the front of the oil spill in zones according to the real-time situation of the oil spill spread, ensuring continuous alignment between dispatch decisions and the oil spill spread boundary.

[0060] Optionally, the process in step S102 of updating the edge coordinates of the oil spill diffusion area and the estimated position coordinates and remaining range of each patrol vessel using the simulation sand table as the observation vector for the next time step and returning it to the agent can specifically include: S1024. Based on the oil spill location coordinates and ocean current velocity vector, calculate the set of edge position coordinates of the oil spill diffusion area after it expands outward at the end of each time step, and obtain the updated set of edge position coordinates.

[0061] The updated set of edge position coordinates refers to the oil spill diffusion object at the current time step. At the end, based on the ocean current velocity vector The set of coordinates of the leading edge position obtained by recalculation after the oil spill area expands outward is denoted as... and For the updated number One edge coordinate point, This represents the total number of updated edge coordinate points.

[0062] Specifically, the oil spill dispersion target first extracts the ocean current velocity vector at the current time step from the simulation sand table. , where the direction angle The dominant direction of oil film diffusion is determined by the flow velocity. Determine the diffusion rate. Then calculate the diffusion rate at the time step. Internal oil film along direction angle drift distance Drift distance equal to flow rate With time step The product of and . Simultaneously calculate the radius increment caused by the diffusion of the oil film itself. radius increment The diffusion calculation is obtained using a diffusion model based on environmental parameters such as oil film viscosity and sea surface temperature. The specific diffusion calculation formula is as follows:

[0063] in The empirical diffusion coefficient was determined by a pool experiment. and These are the densities of seawater and oil spill, respectively. It is the acceleration due to gravity. This represents the initial volume of the oil spill. For time step; Regarding sea surface temperature The viscosity correction function is defined as follows: ,in For standard reference temperature, This is the temperature decay coefficient based on a specific oil product.

[0064] Then use the current edge position coordinate set coordinate points in the middle Using this as a reference, the coordinate points are aligned along the direction angle. Translational drift distance Then expand the radius increment outwards. The updated edge position coordinates are obtained. Finally, the update calculation is completed by traversing all edge coordinate points, resulting in the updated set of edge position coordinates. .

[0065] S1025. Based on the position coordinates and control commands, the ship objects in the simulation sand table calculate the simulation position coordinates of each patrol ship at the end of each time step under the constraints of the maximum speed and remaining range of each patrol ship, and obtain the updated simulation position coordinates. The consumed range is calculated according to the speed value and time step, and the difference between the remaining range and the remaining range is calculated to obtain the updated remaining range.

[0066] Maximum speed constraint This refers to the maximum executable speed set by each patrol vessel in the simulation sand table, when the speed value in the control command... Exceed The actual execution speed of the ship object is taken as The remaining range constraint refers to the current remaining range of each patrol vessel. The limit on the navigable distance within this time step applies when the calculated navigable distance exceeds... The actual sailing distance of the ship object at that time is taken .

[0067] Updated projection coordinates This refers to the time step after each patrol vessel executes a control command. New coordinates reached at the end. Range. This refers to the distance covered by fuel consumed by each patrol vessel within the current time step, expressed in nautical miles. Updated remaining range. This refers to the current remaining battery life. Deducting the driving range The remaining usable battery life obtained afterwards.

[0068] Specifically, each ship object first receives control commands submitted by the intelligent agent. Extract speed value and heading angle Next, set the speed value. Constraints on the ship's maximum speed If a comparison is made, The actual speed taken is Otherwise, the actual speed will be taken as... Record the actual speed as .

[0069] Then, by calculating the actual speed of flight... and The product of these two products yields the product at the time step. Internal speed based on actual operating speed Theoretical sailing distance Then the theoretical sailing distance With current remaining battery life If a comparison is made, The actual distance the ship traveled was... Pick Otherwise, the actual sailing distance Pick .

[0070] Next, using the current calculated position coordinates Starting from the heading angle Actual distance traveled in the direction of travel The updated projected position coordinates are obtained through latitude and longitude offset calculations. At the same time, the actual sailing distance will be... The remaining range consumed within this time step Calculate the current remaining battery life and driving range The difference is used to obtain the updated remaining battery life. .

[0071] Specifically, check after each state update. If the remaining battery life after the update If the ship is marked as out of fuel in the simulation sandbox, its simulation position coordinates will remain unchanged in subsequent time steps. It will no longer receive new control commands and will no longer be included in the effective calculation nodes for spatial coverage and response time.

[0072] S1026. The updated set of edge position coordinates, the inferred position coordinates, and the remaining endurance are concatenated and returned to the agent as the observation vector for the next time step.

[0073] The observation vector at the next time step This refers to the set of edge position coordinates updated at the end of the current time step. Updated simulated position coordinates of each patrol vessel and the updated remaining battery life The one-dimensional numerical sequence formed by sequentially piecing together the data.

[0074] Specifically, the updated set of edge position coordinates is first extracted from the oil spill diffusion object. Extract the updated projected position coordinates from each ship object. and the updated remaining battery life Next, , and The vectors are concatenated sequentially to form the observation vector for the next time step. Finally, the observation vectors Returning to the agent, the agent uses As input, the control instruction generation process is re-executed according to the initial scheduling strategy, and the deduction loop for the next time step begins.

[0075] This embodiment combines the updated set of edge position coordinates, the inferred position coordinates, and the remaining endurance into an observation vector for the next time step and returns it to the agent, enabling the agent to perceive the synchronous evolution of the oil spill spread situation and the vessel status within each time step.

[0076] S103. Within each time step, the initial scheduling strategy is iteratively updated with the spatial coverage of each patrol vessel to the edge location coordinates and the response time of each patrol vessel to the edge of the oil spill diffusion area as reward values ​​until the target scheduling strategy is obtained. The target scheduling strategy includes the control command sequence of the patrol vessels.

[0077] Optionally, step S103, which iteratively updates the initial scheduling strategy within each time step using the spatial coverage of each patrol vessel to the edge location coordinates and the response time of each patrol vessel to the edge of the oil spill diffusion area as reward values ​​until convergence to obtain the target scheduling strategy, may specifically include: S1031. Within each time step, calculate the distance between the projected position coordinates of each patrol vessel and each coordinate point in the edge position coordinate set. Based on the ratio of the number of coordinate points with a distance less than the preset coverage threshold to the total number of coordinate points in the edge position coordinate set, obtain the spatial coverage rate.

[0078] spatial coverage This refers to the proportion of coordinate points covered by each patrol vessel within the current time step, relative to the total number of edge coordinate points. The value ranges from 0 to 1 and is used to measure the coverage degree of each patrol vessel on the oil spill front. Preset coverage threshold. This refers to the critical distance, expressed in nautical miles, used to determine whether an edge coordinate point is covered by a patrol vessel. When the distance between the patrol vessel's projected position coordinates and the edge coordinate point is less than... At that time, the coordinate point was considered to be covered.

[0079] Table 2 shows the specific preset coverage threshold values ​​under different sea state conditions in this embodiment: Table 2: Preset Coverage Threshold Comparison Table

[0080] As shown in Table 2, the preset coverage threshold The sea state is classified according to its level, which is determined by the current ocean current velocity vector. Medium flow velocity The decision is that the higher the current velocity, the higher the sea state level, and the larger the corresponding preset coverage threshold.

[0081] The effective coverage duration represents the time window during which the patrol vessel maintains effective coverage of the edge coordinates under the given sea conditions, and the applicable diffusion radius represents the radius of the oil spill diffusion area to which this threshold applies. In actual simulations, this is determined based on the current ocean current speed. Select the preset coverage threshold corresponding to the sea state level from Table 2. Used for coverage determination.

[0082] Preset coverage threshold Its effectiveness assessment is dynamically determined strictly according to the lookup logic in Table 2. Specifically, firstly, within each time step, the ocean current velocity vector... Extracting flow rate Combined with the actual diffusion radius of the oil spill target, the current sea state and diffusion stage level are determined based on the dual conditions of the flow velocity range and applicable diffusion radius defined in Table 2, and then the corresponding preset coverage threshold is extracted. And the effective duration of coverage.

[0083] If the time a point is marked as covered exceeds the effective coverage duration obtained from the lookup table, the coverage mark becomes invalid, and the patrol vessel must reapproach the point to re-determine coverage at the current time step. Then, the calculated position coordinates of each patrol vessel are traversed. Regarding the first ships, calculation With edge position coordinate set Each edge coordinate point Distance between ,distance pass and The difference in latitude and longitude is calculated using the great circle distance formula.

[0084] Then for each edge coordinate point Determine whether at least one patrol vessel meets the requirements. If the conditions are met, then the coordinate point will be... Marked as covered. Then, the set of edge location coordinates is calculated. The number of coordinate points marked as covered Finally, calculate the number of covered coordinate points. Total number of edge coordinate points The ratio of the two values ​​gives the spatial coverage. .

[0085] S1032. Calculate the ratio of the minimum distance to the speed value in the control command to obtain the response time of each patrol vessel to reach the edge of the oil spill diffusion area.

[0086] Response time It refers to the first The patrol vessels are located at their current coordinates. The speed value in the control command Set of coordinates of the edge location The time required to reach the nearest edge coordinates at medium distance, in minutes, is used to measure the rapid response capability of each patrol vessel to the oil spill front.

[0087] Specifically, firstly regarding the first Patrol vessels, from the simulated location coordinates With edge position coordinate set coordinates of each edge point Distance set between Extract the distance with the smallest value. Then, from the control commands corresponding to the patrol vessel... Extracting speed value Then calculate the minimum distance. With speed value Multiplying the ratio by 60 gives the response time of the patrol vessel. Finally, iterate through all... The response time of each of the patrol vessels was calculated. .

[0088] S1033. The reward value for each time step is obtained by weighted summation of spatial coverage and response time. The initial scheduling policy is iteratively updated using the reward value until convergence is obtained to obtain the target scheduling policy.

[0089] Reward Value This refers to the spatial coverage within each time step. and response time of each patrol vessel The scalar value obtained by weighted summation of the normalized reciprocals is used to measure the comprehensive performance of the current control commands on the coverage effect and rapid response capability of the oil spill diffusion front. The target scheduling strategy refers to the optimal scheduling strategy output after multiple rounds of deduction, iterative updates, and convergence of the initial scheduling strategy, and includes the sequence of control commands for each patrol vessel over multiple time steps in the future. .

[0090] The core operational architecture of this strategy's iterative optimization is based on a reinforcement learning closed-loop system that facilitates interaction between the agent and the simulation environment. The specific optimization process is as follows: Figure 4As shown. In each time-series simulation, the simulation environment 420 is in a specific environmental state. The oil spill diffusion objects within it dynamically calculate the oil spill evolution state at the end of the current time step, while the vessel objects receive parameterized control commands from the previous time step to update their entity coordinates and remaining fuel. The simulation environment 420 concatenates the physical state sequences of both into a one-dimensional matrix and encapsulates it into a state observation vector, which is then fed back to the agent 410 across the boundary.

[0091] The built-in parameterized action value function of agent 410 extracts features from the state observation vector, evaluates the expected cumulative reward of the state, and generates a future-oriented control command sequence through the policy update network, which is then re-inputted into the simulation environment 420. Simultaneously, the simulation environment 420 calculates a weighted comprehensive reward value based on spatial coverage and rapid response indicators, and inputs this scalar value back into the policy update network of agent 410. Through a gradient update rule that maximizes the expected reward, the initial scheduling policy is driven to continuously evolve until full convergence.

[0092] Specifically, the first step is to assess the response time of each patrol vessel. Perform normalization and calculate the average of all response times. Then, the normalized response index is obtained by taking the reciprocal of the average response time. Next, spatial coverage will be... With normalized response index Perform a weighted summation to obtain the reward value. ,in and The weighting coefficients for spatial coverage and normalized response index are respectively, and satisfy the following conditions: In this embodiment, we take , .

[0093] Then, the reward value is calculated using the current time step. The algorithm framework of temporal difference reinforcement learning is used to iteratively update parameters such as the direction allocation rules in the initial scheduling strategy. Specifically, the agent maintains a parameterized action value function. After calculating the reward value Then, the TD error is calculated by evaluating the maximum expected return for the next time step:

[0094] in, This represents the TD error generated at the current time step. This is the reward value; It serves as a discount factor and is used to measure the attenuation weight of future projected states on the current decision; and These represent the observation vector formed by concatenating the current time step and the issued control command, respectively. To deduce the new observation vector returned at the next time step in the simulation of the sand table evolution; This represents all candidate actions that may be taken in the next time step; Indicates the state Next action The expected cumulative return that can be assessed.

[0095] The agent then uses the learning rate The network parameters are updated according to the gradient ascent rule to guide policy evolution. Details of the specific policy parameterization adjustments in this reinforcement learning process are explained in sub-steps S10331 and S10332 below.

[0096] Optionally, the process of iteratively updating the initial scheduling policy using the reward value until convergence to obtain the target scheduling policy in step S1033 may specifically include: S10331. Calculate the reward difference between the reward value of the current time step and the reward value of the previous time step. When the reward difference is less than or equal to zero, redivide the angle range of each coverage sector with a preset adjustment step size.

[0097] Reward Difference This refers to the reward value calculated at the current time step. The reward value calculated with the previous time step The difference between them is used to determine whether the initial scheduling strategy iteration direction causes the reward value to continue to increase.

[0098] Preset adjustment step size This refers to the difference in rewards. The angle increment used to fine-tune the angular range boundary of each coverage sector is in degrees, and the preset adjustment step size dynamically changes with the simulation cycle to achieve an adaptive transition from coarse to fine adjustment.

[0099] Table 3 shows the specific settings of the preset adjustment step size and its attenuation coefficient for different simulation cycle intervals in this embodiment: Table 3: Preset Adjustment Step Size Comparison Table

[0100] As shown in Table 3, the preset adjustment step size The simulation cycle is divided into segments based on the current cycle interval. In the early stage of the simulation, a larger adjustment step size is used to explore the angle range of the covered sector. As the simulation cycle increases, the adjustment step size is gradually reduced to achieve fine adjustment.

[0101] Attenuation coefficient Used to decay the adjustment step size after each adjustment. The current adjustment step size is decayed after each use within the current round interval. Updates are performed until the minimum step size limit is reached, at which point the changes remain unchanged. The adjustment mode indicates the intensity of the strategy adjustment used in different rounds; the coarse adjustment mode allows for larger changes in sector boundaries, while the fine adjustment mode only allows for small-scale adjustments.

[0102] Specifically, first record the number of rounds that the current simulation loop has been executed. Based on the deduced cycle intervals in Table 3, determine the current mode and select the corresponding preset adjustment step size. attenuation coefficient and minimum step size limit Next, calculate the reward value for the current time step. Reward value compared to the previous time step The difference is used to obtain the reward difference. .

[0103] Then determine the reward difference. Is it less than or equal to zero? This indicates that the current initial scheduling strategy has not increased the reward value or has caused the reward value to decrease, and it is necessary to re-divide the angle range of each coverage sector in the direction allocation rule.

[0104] Then, for each coverage sector Fine-tune the angular range boundary, adjusting it by adjusting the first... Each ship corresponds to a coverage sector starting angle Subtract the current preset adjustment step size Ending angle Plus preset adjustment step size At the same time, the angular span of adjacent coverage sectors is reduced accordingly, so that the sum of the angular spans of all coverage sectors is equal to the total span of the directional distribution range. Finally, the current preset adjustment step size is updated by decay, and the calculation is performed. ,like Then the adjustment step size will be updated to Otherwise, maintain the adjustment step size. This is used for the next sector angle adjustment.

[0105] S10332. Update the direction allocation rules with the angle range of each re-divided coverage sector, regenerate the heading angle and control commands of each patrol vessel based on the updated direction allocation rules, and recalculate the reward value until the reward difference is greater than zero and the absolute value is less than the preset convergence threshold. Output the control command sequence of each patrol vessel generated by the direction allocation rules at the time of convergence as the target scheduling strategy.

[0106] The updated direction allocation rule refers to converting the angle ranges of each newly divided coverage sector into new allocation ratios for each patrol vessel. The resulting new allocation rules and new allocation ratios This is equivalent to re-partitioning and covering sectors. The proportion of the angular span to the total span of the directional distribution range.

[0107] Preset convergence threshold This refers to the critical value of the absolute value of the reward difference used to determine whether the initial scheduling strategy has converged. Greater than zero and its absolute value Less than The strategy is considered converged at this point. (Control instruction sequence) This refers to the ordered set of control commands that each patrol vessel in the target scheduling strategy will execute sequentially over multiple future time steps, denoted as and For the first The set of control commands for each patrol vessel within a time step. This represents the total number of time steps for the preset strategy planning.

[0108] Specifically, the first step is to calculate the coverage sectors after the re-division. The angular span is calculated by dividing the angular span of each coverage sector by the total span of the directional distribution range. The new allocation ratio for each patrol vessel was obtained. According to the new allocation ratio Update the allocation ratio parameters of the direction allocation rules in Table 1.

[0109] Next, based on the updated direction allocation rules, step S10232 is re-executed to calculate the center direction angle of each coverage sector. and the course angle of each patrol vessel The process generates a new heading angle. Combined with the preset speed in Table 1 Encapsulated as a new set of control instructions .

[0110] Then the new set of control commands Submitted to the simulation sand table, the simulation sand table is based on Update the projected and edge position coordinate sets for each patrol vessel, return the new observation vector, and re-execute the calculation processes for spatial coverage, response time, and reward value in steps S1031, S1032, and S1033 to obtain the new reward value. .

[0111] After that As the reward value for the current time step, it will be used to offset the previous round's rewards. As the reward value of the previous time step Recalculate the reward difference Finally, determine the difference in rewards. Is it greater than zero and its absolute value? Less than the preset convergence threshold If the conditions are met, the initial scheduling strategy is considered to have converged. The control commands for each patrol vessel generated according to the current direction allocation rule are extracted within each time step to form a control command sequence. The target scheduling strategy output is used as the output; if it is not satisfied, continue to execute step S10331 to readjust the coverage sector angle range.

[0112] This embodiment enables the initial scheduling strategy to be continuously iterated and updated in the simulation cycle to improve coverage and shorten response time, and finally converges to the optimal scheduling strategy for high coverage and rapid response to the oil spill front.

[0113] S104. After converting the target scheduling strategy into scheduling instructions, send them to each patrol vessel and receive the actual position returned. Calculate the difference between the actual position and the simulated position coordinates of the corresponding patrol vessel in the simulation sand table to obtain the spatial offset of each patrol vessel. Accumulate the spatial offsets within a preset time window to obtain the cumulative offset of each patrol vessel.

[0114] Dispatch instructions This refers to the sequence of control instructions in the target scheduling strategy. This is converted into navigation commands that can be received and executed by the shipborne terminal, including the speed and heading angle of each patrol vessel at each time step. Actual position It refers to the first The patrol vessels use onboard AIS equipment to actually measure and transmit the real geographical coordinates of their locations back to the shore, expressed in latitude and longitude.

[0115] Spatial offset It refers to the first The actual position of a patrol vessel at a certain moment Coordinates of the simulated position at the corresponding moment in the simulation sand table The spatial distance between them, measured in nautical miles, is used to measure the degree of deviation between the actual sailing state of a ship and the predicted state.

[0116] Preset time window This refers to the length of a fixed time interval, measured in minutes, used to accumulate the spatial offset of each patrol vessel. Cumulative Offset It refers to the first Patrol vessels in the preset time window Spatial offset at each time step The cumulative sum, expressed in nautical miles, is used to assess the cumulative degree of deviation of a vessel over a period of time.

[0117] Specifically, the control instruction sequence is first extracted from the target scheduling strategy output in step S103. , of which A set of control instructions for each time step Including the speed values ​​of each patrol vessel and heading angle .

[0118] Next, the control command sequence Control instructions at each time step are sequentially converted into scheduling instructions. Dispatch instructions This includes a sequence of speed and heading angles that the vessel needs to execute at each future time step, along with timestamps of the execution times. Then, the corresponding dispatch instructions for each patrol vessel are transmitted via a VHF wireless communication channel. The instructions are sent to the shipboard terminals of the corresponding vessels. After receiving the dispatch instructions, each patrol vessel begins to carry out its navigation mission according to the speed value and heading angle sequence in the instructions.

[0119] Subsequently, the shipborne AIS equipment of each patrol vessel transmitted data back at the preset frequency. It periodically transmits its actual location coordinates back to the shore-based command center, with a preset transmission frequency. Set to send data back every few minutes, for example Minutes. In the... Next return time , No. The actual position coordinates returned by the patrol vessels are as follows This indicates that the ship was at a certain time. The actual latitude and longitude. The shore-based command center receives the actual position coordinates transmitted back by each patrol vessel. .

[0120] Next, regarding the first Patrol vessels at all times The actual location returned Extract the ship's position at the same time from the simulation sand table. Deduced position coordinates Then calculate the actual position. with the calculated position coordinates The spatial distance between the two points is calculated using the great circle distance formula by taking the difference in latitude and longitude between the two points to obtain the spatial offset. ,in This represents the function for calculating the distance to a great circle. It iterates through all... The patrol vessels calculated the time of each vessel. Spatial offset .

[0121] Then set the preset time window. ,For example Minutes, within the time window The spatial offset of each patrol vessel is continuously accumulated within a time window. shared within Next return time For the first The patrol vessel, and its spatial offset at each transmission time. Summing yields the cumulative offset. Traverse all The patrol vessels were counted separately within a preset time window. Cumulative offset within .

[0122] This embodiment establishes a deviation quantification mechanism between the actual execution state and the predicted state, avoiding the problems of frequent corrections under slight deviations and lagging corrections under severe deviations caused by fixed-period corrections, and providing a quantitative basis for realizing adaptive closed-loop correction based on the actual degree of deviation.

[0123] The entire adaptive difference determination and inverse proportional mapping control strategy adjustment is as follows: Figure 3 As shown, within the current time step, the actual positions reported by each cooperating vessel are periodically acquired through the shipboard system. After receiving the data, the calculation module retrieves the high-precision vessel position coordinates synchronously maintained in the simulation sand table, and calculates the instantaneous great circle distance using the latitude and longitude values ​​of both. This spatial offset is accumulated in real time within a preset time window to obtain a cumulative offset that accurately reflects the strength of the varying levels of external sea state interference.

[0124] If, after inverse proportional mapping and maximum / minimum boundary constraint checks, the maximum cumulative offset is found to have a timer duration that has not yet reached the dynamically calculated trigger interval, then the current vessel's execution trajectory is determined to be under control, and the original target scheduling strategy branch is executed. If the timer duration reaches the trigger interval, the adaptive closure correction process is forcibly activated, immediately extracting the latest actual position of the current node and forcibly overwriting the outdated vessel position coordinates in the simulation sandbox. Simultaneously, the preset inverse proportional coefficients in the environmental parameter comparison table, such as the ocean current velocity vector, are reset based on the latest dynamic characteristics.

[0125] Based on this, the agent perceives discontinuous abrupt changes in the environmental state, reinitializes the state observation vector, restarts the policy iteration convergence calculation, and outputs the corrected target scheduling policy to generate new scheduling instructions to be issued to the physical layer for execution. After clearing the current accumulated offset and resetting the timer, the logical endpoint determines whether the overall job termination condition is met; if not, it re-enters the loop monitoring flow; if met, the process officially reaches the end mark, terminating the current deduction step.

[0126] S105. The maximum cumulative offset is converted into the trigger interval duration through an inverse proportional mapping relationship. When the timer reaches the trigger interval duration, the simulation sand table is updated with the actual position returned by each patrol vessel to regenerate a new target scheduling strategy.

[0127] Optionally, step S105 converts the maximum cumulative offset into a trigger interval duration through an inverse proportional mapping relationship. When the timer reaches the trigger interval duration, the simulation sand table is updated with the actual positions returned by each patrol vessel to regenerate a new target scheduling strategy. This process may specifically include: S1051. Extract the maximum cumulative offset from all cumulative offsets, and obtain the trigger interval duration based on the ratio of the preset inverse proportional coefficient to the maximum cumulative offset.

[0128] Maximum cumulative offset This refers to the cumulative offset set from all patrol vessels. The maximum cumulative offset extracted from the data, expressed in nautical miles, is used to represent the deviation status of the patrol vessel with the most severe cumulative deviation.

[0129] Preset inverse proportional coefficient This refers to the mapping parameter used to convert the maximum cumulative offset into the trigger interval duration. It is measured in nautical miles per minute, and the magnitude of the inverse proportionality coefficient determines the sensitivity of the trigger interval duration to changes in the cumulative offset. Trigger Interval Duration It refers to the time required from the start of the current time to triggering the policy correction, and is expressed in minutes. The trigger interval is inversely proportional to the maximum cumulative offset, that is, the larger the cumulative offset, the shorter the trigger interval.

[0130] Table 4 shows the specific settings of the preset inverse proportional coefficients for different sea state conditions and cumulative offset ranges in this embodiment: Table 4: Comparison Table of Inverse Proportional Coefficients

[0131] As shown in Table 4, the preset inverse proportional coefficient Based on sea state class and maximum cumulative offset The area is classified into different levels, with the sea state level determined by the current ocean current speed. The decision is as follows: The maximum trigger interval represents the longest allowable waiting time under the given sea state and offset range; the minimum trigger interval represents the shortest allowable waiting time; and the correction priority represents the urgency of strategy correction within the given offset range. When the cumulative offset is small, a larger inverse proportional coefficient is used to extend the trigger interval and avoid frequent corrections; when the cumulative offset is large, a smaller inverse proportional coefficient is used to shorten the trigger interval and achieve timely corrections.

[0132] Specifically, firstly, the cumulative offset set of each patrol vessel calculated in step S104 is used. Iterate through all cumulative offset values, extract the cumulative offset with the largest value, and denote it as . Next, from the ocean current velocity vector... Extracting flow rate The current sea state level is determined based on the current velocity range in Table 2, combined with the maximum cumulative offset. The corresponding preset inverse proportional coefficient is selected from Table 4 based on the given numerical range. .

[0133] Then calculate the preset inverse proportional coefficient. With maximum cumulative offset The ratio of the two values ​​gives the trigger interval duration. Then, the calculated trigger interval duration was... Perform boundary constraint checks and extract the maximum trigger interval for the corresponding sea state level from Table 4. and minimum trigger interval ,like Then Revised to ,like Then Revised to This ensures that the trigger interval is within a reasonable range.

[0134] Finally, start the timer and record the current moment as the starting point of the timer. The timer starts accumulating the duration in minutes. When the timer duration Duration of reaching the trigger interval When the policy correction process is triggered, proceed to step S1052.

[0135] S1052. When the timing duration reaches the trigger interval duration, extract the latest actual position returned by each patrol vessel, update the simulation position coordinates of each patrol vessel in the simulation sand table using the actual position, and generate a corrected target scheduling strategy based on the updated simulation position coordinates.

[0136] Revised target scheduling strategy This refers to the new target scheduling strategy output by re-executing the simulation cycle and strategy iteration process in steps S102 and S103 after updating the simulation sand table with the latest actual positions returned by each patrol vessel, and includes the corrected control command sequence for each patrol vessel in multiple future time steps.

[0137] Specifically, firstly, the accumulated timing duration is continuously monitored. ,when The system determines when the timer has reached the trigger interval and initiates a strategy correction process. Next, it extracts the latest actual position coordinates transmitted by each patrol vessel at the trigger time and records them as follows. ,in For the first The latest actual position reported by each ship. If a ship has not reported a new actual position by the trigger time, the position within the preset time window will be used. The last transmitted position is used as the latest position.

[0138] Then, using the latest actual positions transmitted back by each patrol vessel Update the simulation position coordinates of the corresponding ship object in the simulation sandbox, and update the simulation position coordinates of the first ship object in the simulation sandbox. The projected position coordinates of the ships Replace with actual location The simulation sand table is synchronized with its current state. Simultaneously, the latest ocean current velocity vector at the trigger moment is obtained from the marine environmental monitoring network. The ocean current velocity vector of the oil spill diffusion target in the sand table will be used to deduce the oil spill diffusion target. Updated to .

[0139] Then, based on the updated simulation sandbox, the process of the agent submitting control commands to the simulation sandbox in steps S1021 to S1023 is re-executed, using the updated simulation position coordinates. and the latest ocean current velocity vector As a new initial state, it constitutes a new observation vector. The agent schedules according to the initial scheduling strategy. The process generates a new set of control instructions.

[0140] Next, the process of updating the edge position coordinate set of the simulation sand table and the simulation position coordinates of each patrol vessel in steps S1024 to S1026 is repeated, as well as the complete process of calculating spatial coverage, response time, reward value and iteratively updating the strategy in steps S1031 to S10332 until convergence, and the corrected target scheduling strategy is output. .

[0141] Finally, the revised target scheduling strategy will be implemented. Transform into new scheduling instructions The new dispatch instructions are transmitted via VHF wireless communication to the onboard terminals of each patrol vessel. Upon receiving the new instructions, each vessel ceases executing the original instructions and switches to executing the revised speed and heading angle sequences to continue its patrol mission. Simultaneously, the timer is reset and the accumulated offset set is cleared. Then, restart the calculation process of spatial offset and cumulative offset in step S104 to prepare for the next policy correction trigger.

[0142] This embodiment establishes an adaptive triggering mechanism based on the actual degree of deviation accumulation, which dynamically links the timing of strategy correction with the severity of deviation, and realizes closed-loop dynamic correction of the oil spill front patrol and scheduling strategy in the actual execution process.

[0143] Figure 5 A schematic diagram illustrating a specific implementation of a real-time maritime patrol simulation system based on reinforcement learning and wargaming, provided in this application embodiment, is shown below. Figure 5 The system may include: The acquisition module 510 is used to acquire the oil spill location coordinates, ocean current velocity vector, and the location coordinates and remaining range of each patrol vessel to form an initial state set. The update module 520 is used to use the constructed simulation sandbox as the simulation environment. Based on the initial state set and the initial scheduling strategy, in each time step, the agent submits the control commands of each patrol vessel to the simulation sandbox. The simulation sandbox updates the edge position coordinates of the oil spill diffusion area and the simulation position coordinates and remaining endurance of each patrol vessel, which are returned to the agent as the observation vector for the next time step. The update module 520 is also used to iteratively update the initial scheduling strategy at each time step, using the spatial coverage of each patrol vessel to the edge position coordinates and the response time of each patrol vessel to the edge of the oil spill diffusion area as reward values, until the target scheduling strategy is converged. The target scheduling strategy includes the control command sequence of the patrol vessels. The calculation module 530 is used to convert the target scheduling strategy into scheduling instructions and send them to each patrol vessel and receive the actual position returned. It calculates the difference between the actual position and the simulation position coordinates of the corresponding patrol vessel in the simulation sand table to obtain the spatial offset of each patrol vessel. It accumulates the spatial offset within a preset time window to obtain the cumulative offset of each patrol vessel. The generation module 540 is used to convert the maximum cumulative offset into the trigger interval duration through an inverse proportional mapping relationship. When the timer reaches the trigger interval duration, the simulation sand table is updated with the actual position returned by each patrol vessel to regenerate a new target scheduling strategy.

[0144] The real-time maritime patrol simulation system based on reinforcement learning and wargaming in this application is used to implement the aforementioned real-time maritime patrol simulation method based on reinforcement learning and wargaming. Therefore, the specific implementation of the real-time maritime patrol simulation system based on reinforcement learning and wargaming can be found in the embodiment section of the real-time maritime patrol simulation method based on reinforcement learning and wargaming mentioned above. The specific implementation can be referred to the description of the corresponding embodiments, which will not be repeated here.

[0145] Electronic devices may include processors and memory storing computer program instructions.

[0146] Specifically, the processor may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0147] The memory may include a large-capacity storage device for data or instructions. For example, and not limitingly, the memory may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory may include removable or non-removable (or fixed) media. Where appropriate, the memory may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, the memory is non-volatile solid-state memory.

[0148] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to the first aspect of this disclosure.

[0149] The processor reads and executes computer program instructions stored in memory to implement any of the real-time simulation methods for maritime patrols based on reinforcement learning and wargaming in the above embodiments.

[0150] In one example, the electronic device may also include a communication interface and a bus. The processor, memory, and communication interface are connected via the bus and communicate with each other.

[0151] The communication interface is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0152] A bus, including hardware, software, or both, couples components of an online data traffic metering device together. For example, and not limitingly, a bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, a bus may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0153] The electronic device can execute the real-time simulation method for maritime patrol based on reinforcement learning and wargaming as described in the embodiments of this application, thereby realizing the real-time simulation method for maritime patrol based on reinforcement learning and wargaming as described in conjunction with the accompanying drawings.

[0154] Furthermore, in conjunction with the real-time maritime patrol simulation method based on reinforcement learning and wargaming in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; when executed by a processor, these computer program instructions implement any of the real-time maritime patrol simulation methods based on reinforcement learning and wargaming in the above embodiments.

[0155] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0156] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0157] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0158] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0159] The foregoing has provided a detailed description of a real-time maritime patrol simulation method and system based on reinforcement learning and wargaming, as provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A real-time simulation method for maritime patrols based on reinforcement learning and wargaming, characterized in that, include: Obtain the oil spill location coordinates, ocean current velocity vector, and the location coordinates and remaining range of each patrol vessel to form an initial state set; Using the constructed simulation sandbox as the simulation environment, based on the initial state set and the initial scheduling strategy, in each time step, the agent submits control commands for each patrol vessel to the simulation sandbox. The simulation sandbox updates the edge position coordinates of the oil spill diffusion area and the simulation position coordinates and remaining range of each patrol vessel, which are then returned to the agent as the observation vector for the next time step. Within each time step, the initial scheduling strategy is iteratively updated with the spatial coverage of each patrol vessel to the edge location coordinates and the response time of each patrol vessel to the edge of the oil spill diffusion area as reward values ​​until the target scheduling strategy is converged. The target scheduling strategy includes a sequence of control commands for the patrol vessels. After the target scheduling strategy is converted into a scheduling instruction, it is sent to each patrol vessel and the actual position is received back. The difference between the actual position and the simulation position coordinates of the corresponding patrol vessel in the simulation sand table is calculated to obtain the spatial offset of each patrol vessel. The spatial offsets are accumulated within a preset time window to obtain the cumulative offset of each patrol vessel. The maximum cumulative offset is converted into a trigger interval duration through an inverse proportional mapping relationship. When the timer reaches the trigger interval duration, the simulation sand table is updated with the actual positions returned by each patrol vessel to regenerate the new target scheduling strategy.

2. The method according to claim 1, characterized in that, Using the constructed simulation sandbox as the simulation environment, and based on the initial state set and initial scheduling strategy, the agent submits control commands for each patrol vessel to the simulation sandbox at each time step, including: The oil spill location coordinates and ocean current velocity vectors in the initial state set are loaded into the oil spill diffusion object in the simulation sand table, and the location coordinates and remaining range of each patrol vessel are loaded into the vessel object in the simulation sand table. Set the time step of the simulation sand table. Within each time step, extract the set of edge position coordinates of the oil spill diffusion area, the simulation position coordinates of each patrol vessel, and the remaining endurance, and stitch them together to form the observation vector of the intelligent agent. The agent processes the observation vectors according to the initial scheduling strategy, generates control commands for each patrol vessel, including speed and heading angle, and submits them to the simulation sand table.

3. The method according to claim 2, characterized in that, The initial scheduling strategy includes a sequence of initial control commands generated based on preset speed and direction allocation rules; The intelligent agent processes the observation vectors according to the initial scheduling strategy, generates control commands for each patrol vessel including speed and heading angles, and submits them to the simulation sand table, including: Extract the set of edge position coordinates and the projected position coordinates of each patrol vessel from the observation vector, and calculate the distribution range of the direction from the oil spill position coordinates to the set of edge position coordinates; According to the direction allocation rule, the direction distribution range is divided into multiple coverage sectors. The angle value of the projected position coordinates of each patrol vessel pointing to the center position of the corresponding coverage sector is calculated to obtain the heading angle. Using the preset speed as the speed value, the speed value and the heading angle are used as control commands for each patrol vessel and submitted to the simulation sand table.

4. The method according to claim 3, characterized in that, The simulation sand table updates the edge coordinates of the oil spill diffusion area and the simulated position coordinates and remaining range of each patrol vessel, which are then returned to the agent as the observation vector for the next time step, including: Based on the oil spill location coordinates and the ocean current velocity vector, the set of edge location coordinates of the oil spill diffusion area after the outward expansion of the oil spill diffusion area at the end of each time step is calculated to obtain the updated set of edge location coordinates. Based on the position coordinates and control commands, the ship objects in the simulation sand table calculate the simulation position coordinates of each patrol ship at the end of each time step under the constraints of maximum speed and remaining range of each patrol ship, and obtain the updated simulation position coordinates. The consumed range is calculated according to the speed value and the time step, and the difference between the remaining range and the remaining range is calculated to obtain the updated remaining range. The updated set of edge position coordinates, the inferred position coordinates, and the remaining endurance are concatenated and returned to the agent as the observation vector for the next time step.

5. The method according to claim 4, characterized in that, Within each time step, the initial scheduling strategy is iteratively updated until the target scheduling strategy is converged, using the spatial coverage of the edge location coordinates by each patrol vessel and the response time of each patrol vessel to the edge of the oil spill diffusion area as reward values. This includes: Within each time step, the distance between the projected position coordinates of each patrol vessel and each coordinate point in the edge position coordinate set is calculated. The spatial coverage rate is obtained based on the ratio of the number of coordinate points whose distance is less than a preset coverage threshold to the total number of coordinate points in the edge position coordinate set. The ratio of the minimum distance to the speed value in the control command is calculated to obtain the response time of each patrol vessel to reach the edge of the oil spill diffusion area. The reward value for each time step is obtained by weighted summation of the spatial coverage and the response time. The initial scheduling strategy is iteratively updated using the reward value until convergence is obtained to obtain the target scheduling strategy.

6. The method according to claim 5, characterized in that, The step of iteratively updating the initial scheduling policy using the reward value until convergence is achieved to obtain the target scheduling policy includes: Calculate the reward difference between the reward value at the current time step and the reward value at the previous time step. When the reward difference is less than or equal to zero, redivide the angle range of each coverage sector with a preset adjustment step size. The direction allocation rule is updated with the angle range of each re-divided coverage sector. Based on the updated direction allocation rule, the heading angle and control command of each patrol vessel are regenerated to recalculate the reward value until the reward difference is greater than zero and the absolute value is less than a preset convergence threshold. The control command sequence of each patrol vessel generated by the direction allocation rule at the time of convergence is output as the target scheduling strategy.

7. The method according to claim 6, characterized in that, The process of converting the maximum cumulative offset into a trigger interval duration using an inverse proportional mapping relationship, and updating the simulation sand table with the actual positions returned by each patrol vessel when the timer reaches the trigger interval duration to regenerate the new target scheduling strategy, includes: Extract the maximum cumulative offset from all the cumulative offsets, and obtain the trigger interval duration based on the ratio of the maximum cumulative offset to a preset inverse proportional coefficient; When the timing reaches the trigger interval, the latest actual position transmitted by each patrol vessel is extracted, and the simulated position coordinates of each patrol vessel in the simulation sand table are updated using the actual position. Based on the updated simulated position coordinates, the corrected target scheduling strategy is generated.

8. A real-time simulation system for maritime patrols based on reinforcement learning and wargaming, characterized in that, include: The acquisition module is used to acquire the oil spill location coordinates, ocean current velocity vector, and the location coordinates and remaining range of each patrol vessel to form an initial state set. The update module is used to use the constructed simulation sandbox as the simulation environment. Based on the initial state set and the initial scheduling strategy, in each time step, the agent submits the control commands of each patrol vessel to the simulation sandbox. The simulation sandbox updates the edge position coordinates of the oil spill diffusion area and the simulation position coordinates and remaining range of each patrol vessel, so as to return the observation vector to the agent for the next time step. The update module is also used to iteratively update the initial scheduling strategy at each time step, using the spatial coverage of each patrol vessel to the edge location coordinates and the response time of each patrol vessel to the edge of the oil spill diffusion area as reward values, until the target scheduling strategy is converged. The target scheduling strategy includes a sequence of control commands for the patrol vessels. The calculation module is used to convert the target scheduling strategy into scheduling instructions and send them to each patrol vessel, and receive the actual position returned. It calculates the difference between the actual position and the simulated position coordinates of the corresponding patrol vessel in the simulation sand table to obtain the spatial offset of each patrol vessel. It accumulates the spatial offsets within a preset time window to obtain the cumulative offset of each patrol vessel. The generation module is used to convert the maximum cumulative offset into a trigger interval duration through an inverse proportional mapping relationship. When the timer reaches the trigger interval duration, the simulation sand table is updated with the actual positions returned by each patrol vessel to regenerate the new target scheduling strategy.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the real-time simulation method for maritime patrol based on reinforcement learning and wargaming as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the real-time simulation method for maritime patrol based on reinforcement learning and wargaming as described in any one of claims 1 to 7.

Images