A ship abnormal behavior identification method and system based on dynamic and static fusion and deep reinforcement learning

Abstract

The application discloses a ship abnormal behavior identification method and system based on dynamic and static fusion and deep reinforcement learning. In the method, static electronic chart and ship AIS dynamic data are acquired; static environment element data is generated based on the chart, and AIS data is cleaned and verified to build dynamic environment elements; after fusion, spatial features are extracted by using CNN, and time sequence features are extracted by using LSTM; the above features are fused through a gate attention mechanism to obtain a joint state representation; and finally, a behavior category is output through a decision network based on deep reinforcement learning. The application effectively fuses dynamic and static environment information, and improves the accuracy and generalization ability of ship abnormal behavior identification in complex sea areas.

Description

Technical Field

[0001] This invention relates to the field of ship navigation safety monitoring technology, and in particular to a method and system for identifying abnormal ship behavior based on dynamic and static fusion and deep reinforcement learning. Background Technology

[0002] With the rapid development of the shipping industry, the number of ships at sea has surged, making navigation safety issues increasingly prominent. Abnormal ship behaviors (such as deviating from the route, illegally entering restricted areas, speeding, and abnormal stops) are important causes of maritime accidents, marine pollution, and waterway congestion. Therefore, achieving accurate and real-time identification of abnormal ship behaviors is of great practical significance.

[0003] Existing technologies for identifying abnormal ship behavior are mainly based on a single data source: one type uses only Automatic Identification System (AIS) data to analyze dynamic parameters such as the ship's position, heading, speed, and turning rate to determine whether the ship's behavior is in accordance with the norm; the other type is based on static information from electronic nautical charts to delineate navigable waters, prohibited areas, shoals, and other dangerous areas to conduct preliminary verification of the ship's position.

[0004] However, existing technologies have significant drawbacks: relying solely on AIS data lacks consideration of static marine environmental information, making it impossible to distinguish the rationality of vessel behavior in different sea areas (such as navigable zones and restricted zones), and easily leading to misjudgments or omissions; relying solely on electronic charts cannot capture the dynamic operational characteristics of vessels, making it difficult to identify dynamic abnormal behaviors such as speeding and abnormal turning within navigable waters. Furthermore, AIS data may suffer from transmission delays, missing data, or errors, and the reliability of a single data source is insufficient, further affecting the accuracy of the identification results.

[0005] Therefore, how to effectively integrate electronic chart data with AIS dynamic data, give full play to the complementary advantages of the two types of data, and improve the accuracy, real-time performance, and reliability of identifying abnormal ship behavior is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0006] This invention provides a method and system for identifying abnormal ship behavior based on dynamic and static fusion and deep reinforcement learning, which can solve the technical problems of low accuracy and poor reliability in identifying abnormal ship behavior caused by reliance on a single data source in the prior art.

[0007] The first aspect of this invention provides a method for identifying abnormal ship behavior based on dynamic and static fusion and deep reinforcement learning, comprising:

[0008] Acquire static electronic nautical chart data of the target sea area and AIS dynamic data of the target vessel;

[0009] Based on static electronic chart data, static environmental elements are generated to represent at least one of the following: navigation marks, warning zones, obstructions, anchorages, channel boundaries, and separation zones.

[0010] Clean and verify the ship's AIS dynamic data to construct the ship's dynamic environmental elements;

[0011] Static and dynamic environmental elements are integrated to form a fused state input;

[0012] Convolutional neural networks are used to extract static spatial features from the fused state input, and long short-term memory networks are used to extract dynamic temporal features from the fused state input.

[0013] A joint state representation is obtained by fusing static spatial features and dynamic temporal features through a gating attention fusion mechanism.

[0014] Based on deep reinforcement learning algorithms, the joint state representation is input into the decision network, and the ship behavior category is output to complete the identification of abnormal behavior.

[0015] Optionally, based on static electronic chart data, static environmental elements representing at least one of the following elements—navigation marks, warning zones, obstructions, anchorages, channel boundaries, and separation zones—are generated, including:

[0016] The static electronic nautical chart data is analyzed to extract the geometric and attribute information of navigation marks, warning zones, obstructions, anchorages, channel boundaries and separation zones;

[0017] Based on the extracted geometric and attribute information, a corresponding mathematical model is constructed, and the mathematical model is converted into a machine-processable data format to generate static environmental elements.

[0018] Optionally, the ship's AIS dynamic data is cleaned and verified to construct the ship's dynamic environmental elements, including:

[0019] Threshold checks are performed on the position, speed, and heading in AIS dynamic data to remove invalid data;

[0020] Differentiated geometric representation models are used to model water targets with different navigation characteristics in order to obtain dynamic environmental elements.

[0021] Optionally, the static spatial features in the fused state input are extracted using a convolutional neural network, and the dynamic temporal features in the fused state input are extracted using a long short-term memory network, including:

[0022] Organize static environmental elements into multi-channel raster images;

[0023] Multi-channel raster images are input into a convolutional neural network containing convolutional layers, pooling layers, and attention gating mechanisms to extract static spatial features;

[0024] Organize dynamic environmental elements into a time-series matrix;

[0025] The temporal matrix is ​​input into a long short-term memory network containing bidirectional long short-term memory units and an adaptive attention mechanism to extract dynamic temporal features.

[0026] Optionally, fusing static spatial features with dynamic temporal features through a gated attention fusion mechanism includes:

[0027] Project static spatial features onto the same feature dimension as dynamic temporal features;

[0028] The fusion weights of static spatial features and dynamic temporal features are dynamically calculated using a learnable weight matrix.

[0029] The static spatial features and dynamic temporal features are weighted and fused based on the fusion weight.

[0030] Optionally, the method also includes training convolutional neural networks, long short-term memory networks, and decision networks;

[0031] The training adopts a strategy that combines course learning with multi-task assisted learning, and uses a framework based on a dual-delay deep deterministic policy gradient algorithm.

[0032] The course includes dividing training samples into different difficulty levels based on trajectory complexity, and training the model in stages from low to high difficulty.

[0033] Multi-task-assisted learning includes training auxiliary tasks such as ship type classification, ship next position prediction, and sea area risk level estimation while training the main task of abnormal behavior identification.

[0034] When training the decision network, a composite reward function is used to calculate the reward value. The composite reward function includes at least an accuracy reward, a risk-weighted reward, and a safe zone penalty.

[0035] Optionally, the decision-making network includes an Actor strategy network and a Critic value network; the output ship behavior categories include:

[0036] The Actor policy network outputs continuous action values ​​based on the joint state representation. When the continuous action value is greater than or equal to 0, it is judged as abnormal behavior; when the continuous action value is less than 0, it is judged as normal behavior.

[0037] A second aspect of this invention provides a ship anomaly behavior identification system based on dynamic and static fusion and deep reinforcement learning, comprising a data layer, a service layer, and an application layer:

[0038] The data layer is used to acquire and store static electronic nautical chart data of the target sea area and AIS dynamic data of the target vessels;

[0039] The service layer, which connects to the data layer, includes:

[0040] The static data processing module is used to generate static environmental elements based on static electronic nautical chart data, representing at least one of the following elements: navigation marks, warning zones, navigational obstructions, anchorages, channel boundaries, and separation zones.

[0041] The dynamic data processing module is used to clean and verify AIS dynamic data and construct the dynamic environmental elements of the ship.

[0042] The feature extraction and fusion module is used to fuse static environmental elements and dynamic environmental elements to form a fused state input. It uses a convolutional neural network to extract static spatial features, uses a long short-term memory network to extract dynamic temporal features, and uses a gated attention fusion mechanism to fuse static spatial features and dynamic temporal features to obtain a joint state representation.

[0043] The decision module is used to input the joint state representation into the decision network based on the deep reinforcement learning algorithm, and output the ship behavior category to complete the abnormal behavior identification.

[0044] The application layer, connected to the service layer, is used to receive and display the results of ship behavior category identification.

[0045] A third aspect of the present invention provides a ship abnormal behavior identification device based on dynamic and static fusion and deep reinforcement learning, comprising:

[0046] One or more processors;

[0047] A memory on which one or more programs are stored;

[0048] When the one or more programs are executed by the one or more processors, the one or more processors implement the ship abnormal behavior identification method based on dynamic and static fusion and deep reinforcement learning as described in any of the above.

[0049] A fourth aspect of the present invention provides a computer storage medium for storing a program, which, when executed, is used to implement the ship abnormal behavior identification method based on dynamic-static fusion and deep reinforcement learning as described in any of the preceding claims.

[0050] Beneficial effects:

[0051] With improved accuracy, this invention overcomes the limitations of a single data source by integrating the static environmental constraints of electronic charts with the dynamic behavioral characteristics of AIS, significantly improving the accuracy of identifying behaviors such as violations of navigation restrictions and abnormal turns.

[0052] With strong generalization ability, this invention adopts a deep reinforcement learning framework, which uses a composite reward function and a course learning strategy to enable the model to learn autonomously and adapt to new and complex abnormal behavior patterns. Its generalization ability is superior to traditional supervised learning methods.

[0053] With high real-time performance and reliability, the system of this invention adopts a three-layer architecture, which can realize real-time data processing and instant early warning of abnormal behavior, and reduces the risk of false alarms caused by errors or missing data sources through multi-source data fusion. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 A flowchart illustrating a method for identifying abnormal ship behavior based on dynamic-static fusion and deep reinforcement learning, provided for an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of an AIS data processing flow provided in an embodiment of the present invention;

[0057] Figure 3 A schematic diagram illustrating a process for identifying abnormal ship behavior according to an embodiment of the present invention;

[0058] Figure 4 A schematic diagram of the structure of a ship abnormal behavior identification system based on dynamic and static fusion and deep reinforcement learning provided in an embodiment of the present invention;

[0059] Figure 5 This is a schematic diagram of the structure of a device provided in an embodiment of the present invention. Detailed Implementation

[0060] This invention provides a method and system for identifying abnormal ship behavior based on dynamic and static fusion and deep reinforcement learning, which can solve the technical problems of low accuracy and poor reliability in identifying abnormal ship behavior caused by reliance on a single data source in the prior art.

[0061] See Figure 1 This figure is a flowchart illustrating a method for identifying abnormal ship behavior based on dynamic-static fusion and deep reinforcement learning, provided by an embodiment of the present invention. The method for identifying abnormal ship behavior based on dynamic-static fusion and deep reinforcement learning provided by this embodiment of the present invention can be implemented, for example, through the following steps S101-107.

[0062] S101: Acquire static electronic nautical chart data of the target sea area and AIS dynamic data of the target vessel.

[0063] In this embodiment of the invention, ship navigation data is collected and stored, including: static environmental data, electronic nautical chart raster images (multi-channel, including geographical information such as waterways, restricted areas, and port boundaries), with a size of H×W×Cstatic; dynamic environmental data, ship AIS trajectory sequences (including information such as timestamps, latitude and longitude, speed, and heading) and meteorological data, organized as a time-series matrix of T×Ddynamic; and knowledge graph data, structured knowledge such as port scheduling plans and permitted fishing vessel operation areas, used to assist in determining "whether the berthing is legal".

[0064] S102: Based on static electronic chart data, generate static environmental elements that characterize at least one of the following: navigation marks, warning zones, obstructions, anchorages, channel boundaries, and separation zones.

[0065] In this embodiment of the invention, static electronic nautical chart data is parsed to extract geometric and attribute information of navigation marks, warning zones, obstructions, anchorages, channel boundaries, and separation zones; based on the extracted geometric and attribute information, a corresponding mathematical model is constructed, and the mathematical model is converted into a machine-processable data format to generate static environmental elements.

[0066] Specifically, the elements obtain static electronic nautical chart data of the target sea area through maritime database interfaces (such as the National Maritime Safety Administration's electronic nautical chart database and the IHO official nautical chart database). The data format is the S-57 standard format (vector electronic nautical chart format), containing core data layers such as basic chart information, geographic feature information, and attribute constraint information. S-57 standard parsing tools (such as the GDAL / OGR geographic data processing library and the dedicated chart parsing software CARIS HIPS and SIPS) are used to verify the format validity of the input electronic nautical chart data, focusing on checking the integrity of the data header file, the standardization of feature coding, and the rationality of topological relationships, eliminating invalid nautical chart data with corrupted formats or missing features. Based on the geographic feature classification system defined by the S-57 standard, the electronic nautical chart is parsed hierarchically to extract core static information related to ship navigation safety and abnormal behavior identification. Based on the motion state (static / dynamic) and morphological characteristics (point / line / area) of environmental elements, the dynamic and static environmental elements of the study waters are classified. Based on the environmental element classification results, mathematical models (geometric feature parameterized expression) of various environmental elements are constructed. According to the input format of the mathematical model of each environmental element, the ECDIS data of the study water area is extracted and processed, and the processed data is imported into the constructed static environmental element mathematical model to realize the data-driven digitalization of the static traffic environment for static environmental perception.

[0067] In this embodiment of the invention, the acquisition of static element information is based on the S-57 standard electronic nautical chart and implemented using the GDAL geospatial data processing library in Python. This satisfies the machine-resolvable requirements of navigation environment elements, providing reliable environmental perception input for autonomous navigation systems in shallow waters. The S-57 standard, developed by the International Hydrographic Organization, is an internationally accepted standard for digital hydrographic data. Its core application is in the generation and exchange of electronic nautical charts, aiming to ensure cross-system data compatibility and consistency.

[0068] This standard describes entity attributes and geometric relationships through feature-based and spatial markers, respectively, abstracting and modeling real-world geographic entities. It provides underlying data support for marine geographic information systems and ECDIS, and promotes the fusion and analysis of multi-source heterogeneous data. Its technical implementation is based on the ISO / IEC 8211 data encapsulation specification, transforming markers into independent records. It covers six major categories of spatial entities: points, lines, surfaces, volumes, lattices, and topological networks, and defines over 180 marker types, including LIGHTS (navigation marks), OBSTRN (obstacles), TSSLPT (channel points), and ACHARE (anchorages). Through a six-bit feature code encoding rule, it achieves machine-resolvable markers and supports dynamic organization and hierarchical storage of geometric shapes such as points, lines, and polygons, providing an efficient framework for the structured expression and visualization of complex marine environmental information. The information acquisition process is as follows.

[0069] 1) Loading the file. In the `extract_obstruction_data` function, the S-57 file is loaded using `gdal.OpenEx`. If file loading fails, an error message is output and the process terminates; if successful, proceed to the next step.

[0070] 2) Connect to the database. Use sqlite3.connect to establish a connection with the SQLite database and create a cursor object to execute SQL commands.

[0071] 3) Traverse the layers. Obtain the number of layers (including points, lines, polygons, etc.) in the S-57 file using dataset.GetLayerCount, and then traverse each layer sequentially. For each layer, call the create_sqlite_table function to create the corresponding database table. The table structure includes geometry type (geometry_type), layer field, and longitude and latitude fields.

[0072] 4) Process geometric data. For each feature in a layer, extract its geometric object and determine its type (point, line, or polygon), then perform the corresponding operation based on the geometric type. For a point, extract latitude and longitude; for a line, iterate through each point of the line and extract latitude and longitude; for a polygon, iterate through each ring of the polygon and each point within it and extract latitude and longitude. Finally, insert the extracted latitude and longitude data into the SQLite database.

[0073] 5) Commit the transaction and close the connection. After all data insertion operations are complete, commit the transaction using `conn.commit` to ensure data persistence. Finally, close the database connection and output a success message. If an exception occurs during processing, catch the exception and perform a transaction rollback (`conn.rollback`), then close the database connection.

[0074] In this embodiment of the invention, the construction of the mathematical model for static environmental elements is as follows:

[0075] Mathematical model:

[0076] Static elements affecting ship navigation include virtual static environmental elements that regulate ship behavior, such as navigation lanes, waterways, medians, warning zones, and anchorages, as well as real natural static environmental elements such as navigation marks, shipwrecks, and reefs. Based on the mapping relationship between static environmental components and ship navigation safety, this paper summarizes their categories, analyzes the typical physical characteristics of each element, and proposes a highly abstract mathematical expression model.

[0077] (1) Navigation mark

[0078] Navigational aids are crucial navigational facilities for maritime traffic. Through standardized designs of color, shape, and light, they provide vessels with multiple functions, including indicating channel boundaries, water depth, hazard warnings, and anchorage boundaries. This study adopts the International Association of Lighthouse Authorities (IALA) Area A standard, and key information on navigational aids in a shallow waterway is shown in Table 1. During autonomous navigation, the location and meaning of navigational aids, combined with water depth data, provide important guidance for route planning, ensuring navigational safety and compliance with local regulations. Real-time positioning and directional guidance indicate the vessel's location within the channel and its distance from boundaries, facilitating dynamic navigation and course adjustments. Furthermore, as physical obstacles in the water, navigational aids require vessels to adjust their course or reduce speed to avoid collision risks, further ensuring navigational safety.

[0079] Table 1 Key Information on Navigation Markers in Shallow Waters

[0080]

[0081] The navigation mark is represented by the following formula:

[0082] ;

[0083] In the formula, Location of navigation mark; This is a set of navigational beacon attributes. Color code (International Association of Lighthouse Authorities standard color code); These are geometric feature parameters; For the functional logic expression, taking the green starboard marker as an example, the importing vessel should keep it on starboard, which can be determined by observing the marker's relative bearing from the center of the vessel. To determine, that is, within the current flight segment, .

[0084] In digital transportation environment modeling, it is uniformly modeled as a circular area. As shown in the formula below:

[0085] ;

[0086] In the formula, For the radius of the navigation mark, and These are the longitude and latitude coordinates of any point within the sea area.

[0087] (2) Warning Zone

[0088] A navigational warning zone is a specific body of water designated for special attention and cautious navigation due to high traffic density, complex hydrological conditions, or frequent vessel encounters. Its geographical location and extent define the water boundaries that vessels cannot cross or must traverse with caution, and it provides a basis for route planning through constraints such as navigation restrictions or speed limits. The characteristics of a warning zone clearly define avoidance requirements, such as requiring complete detours from prohibited areas, or avoiding or prohibiting overtaking in certain areas.

[0089] The restricted areas are mostly circular or polygonal, with circular areas being the most common. The model is shown below:

[0090] ;

[0091] In the formula, The center of the restricted area, The radius of the warning zone.

[0092] polygonal region The model is shown below:

[0093] ;

[0094] In the formula, These are the vertices that form the quadrilateral or polygon of the warning zone; This is the first vertex that forms the quadrilateral or polygon of the restricted area. For completely restricted areas, the ship's position must be satisfied. ,in, and These are the longitude and latitude coordinates of the ship.

[0095] (3) Obstructions to navigation

[0096] Navigational obstructions are objects or areas in waterways that may pose a threat to the safe navigation of ships, such as shipwrecks, reefs, bridge piers, and floating debris. The location and size of these obstructions can define areas that are unnavigable or require detours, providing constraints for route planning. The course or speed can be adjusted in real time based on the distance and relative bearing to the obstruction to maintain safe navigation. Fixed obstructions require complete detours, while dynamic or semi-fixed obstructions require speed reduction, course adjustments, or emergency avoidance to ensure a safe distance.

[0097] Circular obstruction model As shown below:

[0098] ;

[0099] In the formula, To obstruct the center position of the navigational object, Radius of the obstruction.

[0100] Polygonal navigational obstruction model As shown below:

[0101] ;

[0102] In the formula, The vertices that form the quadrilateral or polygonal shape of the obstruction. It is the first vertex that forms the quadrilateral or polygonal shape of the obstruction.

[0103] (4) Anchorage

[0104] Anchorages are bodies of water that provide temporary berthing for ships, typically used for waiting to berth, seeking shelter from wind, or for loading and unloading operations. During autonomous navigation, anchorages define berthing points and restrict areas that cannot be arbitrarily crossed, providing a basis for route planning. Simultaneously, ships adjust their navigation status based on anchorage boundaries and occupancy to achieve safe navigation, avoid interfering with anchored vessels, and for stationary anchored vessels and their anchor chains, it is necessary to detour or reduce speed and increase safe distance nearby to avoid the risk of collision.

[0105] Its shape is usually circular or polygonal, mathematical model and They are shown below:

[0106] ;

[0107] In the formula, The center of the circular anchorage. The radius of the circular anchorage.

[0108] ;

[0109] In the formula, These are the vertices that form the anchorage quadrilateral or polygon; It is the first vertex that forms the anchorage quadrilateral or polygon.

[0110] (5) Channel boundary

[0111] A channel boundary refers to the waterway area designated for the safe navigation of ships. In electronic nautical charts, the channel boundary is geometrically defined using discrete channel points, forming a digital spatial reference for ship navigation. In actual navigation, the navigator needs to combine the channel point data from the electronic chart with the positional information of navigation marks to comprehensively judge the relative relationship between the ship's position and the channel boundary. Therefore, in the channel boundary model, navigation marks and channel points together constitute the spatial reference of the navigation area, providing constraints for path planning. At the same time, the channel boundary acts as a virtual obstacle, guiding the ship to adjust its course to avoid deviating from the channel.

[0112] At the original navigation point The navigation mark attribute field has been added to the system, and the expanded waypoints are defined as follows:

[0113] ;

[0114] In the formula, Latitude and longitude coordinates; The point type distinguishes between channel points and navigational aid points.

[0115] use Indicates the first The first of the separate navigation system The line is shown below:

[0116] ;

[0117] In the formula, For the first The first of the separate navigation system The first line One point.

[0118] Indicates the first The first of the separate navigation system The navigation channels are as follows:

[0119] ;

[0120] In the formula, , The first The first of the separate navigation system The two boundary lines of the navigation lanes.

[0121] The planned route, formed by the centerline of the two boundary lines of the air traffic separation channel, is shown below:

[0122] .

[0123] (6) Separator

[0124] A median strip is a specific navigation control area established to regulate vessel traffic flow and prevent collisions between oncoming vessels. Based on its location and extent, it strictly separates oncoming or different-purpose vessel traffic flows, ensuring that vessels strictly adhere to designated navigation lanes and providing crucial information for route planning. By calculating the distance and relative position of vessels to the median strip boundary in real time, it supports vessels in adjusting their course or speed for safe navigation, particularly suitable for densely populated or complex confluence waters. As a virtual boundary or restricted area, the median strip requires vessels to adjust their navigation strategies according to their characteristics, increasing safe distances to effectively reduce the risk of encounter conflicts and collisions.

[0125] Its shape is usually quadrilateral or polygonal, and its mathematical model is shown in the following formula:

[0126] ;

[0127] in, The nth vertex that forms the quadrilateral or polygon of the dividing strip; It is the first vertex that forms the quadrilateral or polygon of the dividing strip.

[0128] The build process is as follows:

[0129] Through a systematic process of data filtering, mathematical modeling, and database integration, an orderly transformation from S-57 chart data to a digital twin environment can be achieved, providing data support for autonomous navigation decisions. Based on the needs analysis for navigation decisions, static environmental element data is extracted from S-57 standard electronic charts to construct datasets for each element. To ensure efficient data management and application, the lightweight and efficient characteristics of the SQLite database are utilized to achieve centralized data storage and rapid retrieval, ensuring the data integrity and real-time accessibility of the digital twin model.

[0130] After extracting the element data, its spatial attributes are combined with mathematical modeling methods for abstract representation to construct a geometric expression of the digital twin environment. Specifically, the spatial characteristics of the elements are simplified into geometric models such as points, lines, polygons, or circles, and presented in a concise and efficient mathematical description to meet computational needs.

[0131] Finally, using the traffic environment dataset as input, the corresponding mathematical model is driven to realize the digitization of the static traffic environment.

[0132] S103: Clean and verify the ship's AIS dynamic data to construct the ship's dynamic environmental elements.

[0133] In this embodiment of the invention, the position, speed and heading in the AIS dynamic data are subjected to threshold tests to remove invalid data; and different geometric representation models are used to model water targets with different navigation characteristics to obtain dynamic environmental elements.

[0134] Specifically, the AIS data is first processed. AIS data provides dynamic information such as a vessel's real-time position, speed, and heading, as well as static data such as vessel name, call sign, draft, and cargo information. This data enables the simulation system to update the vessel's position and status in real time, accurately reflecting actual traffic flow. Furthermore, AIS data is large in volume, has a wide coverage area, is unaffected by waves, storms, or obstacles, and offers high accuracy within its effective range. This provides reliable data support for simulating dynamic traffic environments; therefore, AIS data is selected as the data source for driving the target vessel. See also... Figure 2 , Figure 2 This is a schematic diagram of an AIS data processing flow provided by an embodiment of the present invention. After extracting AIS data, the AIS data collected from multiple sources is preprocessed, including cleaning, noise reduction, and format unification. The cleaned AIS information is then stored in the backend database in text form.

[0135] Then, the target ship's dynamic information is obtained based on AIS data. Since data processing is not the focus of this study, only a brief overview of the relevant methods is provided here. In the data preprocessing stage, to address interference factors such as signal loss, position drift, and transmission errors, speed threshold filtering and heading continuity verification methods are employed to rigorously screen dynamic parameters such as ship speed, heading, and position. Cubic spline trajectory interpolation is then performed, and the processed data is stored in an SQLite database. This data contains multi-dimensional static and dynamic information about the ship at every moment, including MMSI number, ship type, length, beam, position, heading, track direction, speed, and turning angular velocity.

[0136] ① Verify the location data. Coordinate points outside the valid range are removed by setting latitude and longitude boundary values ​​for the water area, as shown in the following formula:

[0137] ;

[0138] In the formula, , , , These are the minimum and maximum latitude and longitude values ​​of the studied water area, respectively. and The first The longitude and latitude coordinates of each trajectory point are used. Records that do not meet the above conditions are removed to avoid invalid data interference caused by location errors or drift.

[0139] ② Verify the speed data. Abnormal speed data is eliminated by setting a normal range for ship speed values, as shown below:

[0140] ;

[0141] In the formula, This is the historical minimum speed of ships in this waterway (usually taken as 0 m / s). For the first The real-time speed of the vessel , The number of samples; This is the highest historical speed for ships in this waterway.

[0142] ③ Verify the heading data. Any heading exceeding... All AIS trajectory points are considered noise data and must be removed. Furthermore, by combining this with the continuity check of the heading rate of change, heading jumps caused by signal loss or transmission errors can be further identified and corrected.

[0143] Dynamic environmental element model construction. Differentiated geometric representation methods are employed for maritime targets with varying navigation characteristics. For conventional motorized vessels, an elliptical model is established with the vessel's center of gravity as the reference, as shown in the following formula, with the semi-major axis of the ellipse being 0.5. The minor semi-axis is 0.3. ( (For the captain).

[0144] ;

[0145] In the formula, This refers to the target ship's location in the AIS data.

[0146] Non-motorized boats and sailboats, due to their relatively small geometric shape, are typically modeled as point targets; while boats with limited maneuverability or those out of control, being larger in size, are modeled as circular targets. The mathematical models for point and circular targets are shown in the following equations:

[0147] ;

[0148] In the formula, The radius of the target ship model.

[0149] Due to the presence of impassable physical connections, such as towlines and fishing nets, trawler convoys and similar formations need to be defined as linear targets. Their endpoints are determined by the position of the outermost vessel in the convoy, and their mathematical model is shown below:

[0150] ;

[0151] In the formula, , These are the two endpoints of the linear target.

[0152] S104: Integrate static environmental elements with dynamic environmental elements to form a fused state input.

[0153] In this embodiment of the invention, the static environmental elements and dynamic environmental elements are input as a fused state.

[0154] S105: Use a convolutional neural network to extract static spatial features from the fused state input, and use a long short-term memory network to extract dynamic temporal features from the fused state input.

[0155] In this embodiment of the invention, static environmental elements are organized into multi-channel raster images; the multi-channel raster images are input into a convolutional neural network containing convolutional layers, pooling layers, and attention gating mechanisms to extract static spatial features; dynamic environmental elements are organized into temporal matrices; the temporal matrices are input into a long short-term memory network containing bidirectional long short-term memory units and adaptive attention mechanisms to extract dynamic temporal features.

[0156] S106: Static spatial features and dynamic temporal features are fused through a gating attention fusion mechanism to obtain a joint state representation.

[0157] In this embodiment of the invention, static spatial features are projected onto the same feature dimension as dynamic temporal features; the fusion weights of static spatial features and dynamic temporal features are dynamically calculated using a learnable weight matrix; and the static spatial features and dynamic temporal features are weighted and fused according to the fusion weights.

[0158] S107: Based on a deep reinforcement learning algorithm, the joint state representation is input into the decision network, and the ship behavior category is output to complete the identification of abnormal behavior.

[0159] In this embodiment of the invention, the decision network includes an Actor policy network and a Critic value network; the Actor policy network outputs continuous action values ​​based on the joint state representation, and when the continuous action value is greater than or equal to 0, it is determined to be abnormal behavior, and when the continuous action value is less than 0, it is determined to be normal behavior.

[0160] In this embodiment of the invention, before determining what constitutes abnormal ship behavior, it is necessary to first clarify what ship behavior is. Ship behavior is defined as the behavioral patterns and behaviors adopted by crew members while operating a ship, primarily for navigation and collision avoidance. In the field of waterway transportation, abnormal ship behavior is a pattern that deviates from the normal navigation behavior of a ship. This includes specific behaviors that differ from the behavior of most ships while the ship is underway, at anchor, or in a certain state; ship behaviors that endanger ship safety by not complying with maritime traffic management regulations (i.e., not conforming to the 1972 International Regulations for Preventing Collisions at Sea) are also considered abnormal ship behavior. Regarding the rule-driven identification method, the identification process can be divided into two parts. The first part is establishing a ship behavior identification model. The navigation rules, management regulations, and common practices of seafarers in the target waters are quantified to establish a ship behavior identification model. The second part is identifying abnormal ship behavior. Using the ship behavior identification model, the target ship's track data and ship motion parameters are analyzed to identify ships exhibiting abnormal behavior.

[0161] See Figure 3 This demonstrates the data sources, judgment criteria, and core functional modules of the ship abnormal behavior identification system. The four types of abnormal ship behavior to be identified include dimensional anomalies, position anomalies, heading anomalies, and speed anomalies. Figure 3 The document specifies three dimensions upon which behavioral judgments are based: navigation regulations, bridge safety, and navigation experience. It also lists four specific judgment modules: conditions for judging abnormal dimensions, conditions for judging abnormal position, conditions for judging abnormal course, and conditions for judging abnormal speed.

[0162] Figure 3 The data acquisition process is also described. The system obtains the ship's AIS coded message from the AIS system. After AIS data decoding and processing, it forms behavior identification data. At the same time, the system obtains environmental data from sources such as sensors and relevant websites, and extracts information such as flow velocity, flow direction, wind speed, and wind direction from them, which are used together for the final judgment of abnormal ship behavior.

[0163] Abnormal ship behavior is categorized into three main types: abnormal ship speed, abnormal ship course, and abnormal ship position. Abnormal ship speed includes abnormally high speeds, abnormally low speeds, and stop-and-go traffic. Abnormal ship course includes abnormal course and location intrusion. Abnormal ship position includes exceeding historical path limits, area mismatch, regulatory violations, encroachment on restricted areas, and threats to infrastructure.

[0164] In one implementation of this invention, a convolutional neural network, a long short-term memory network, and a decision network are trained. The training employs a strategy combining curriculum learning and multi-task-assisted learning, and uses a framework based on a dual-delay deep deterministic policy gradient algorithm. The curriculum learning includes dividing training samples into different difficulty levels according to trajectory complexity, and training the model in stages from low to high difficulty. The multi-task-assisted learning includes training auxiliary tasks such as ship type classification, ship next position prediction, and sea area risk level estimation simultaneously with the main task of abnormal behavior identification. When training the decision network, a composite reward function is used to calculate the reward value, which includes at least an accuracy reward, a risk-weighted reward, and a safe zone penalty.

[0165] Specifically, a training strategy combining course learning with multi-task-assisted learning is adopted, and the specific process is as follows:

[0166] 1. Data Preprocessing

[0167] Data augmentation: Dynamic data augmentation, applying reasonable perturbations to the AIS trajectory: time jitter, random offset ±5 seconds; position Gaussian noise, (Complies with AIS positioning error ±30 meters); random point loss, analog signal loss, point loss probability 5%; heading change rate constraint, generated trajectory heading change rate ≤ 5° / s (refer to ship dynamics characteristics).

[0168] Static data augmentation: elastic deformation or rotation of the chart grid: random rotation: ±15°; random scaling: 0.8-1.2 times; salt and pepper noise: apply salt and pepper noise to the radar echo channel, intensity <5%.

[0169] Track segmentation: The track is segmented using the sliding window method, with a window length of 2 hours and a step size of 1 hour. Key points are extracted from the segmented track, and key waypoints with a heading change rate > 2° / min or a speed change rate > 1 knot / min are retained.

[0170] 2. Course Learning Strategies

[0171] The difficulty grading criteria include low-difficulty, medium-difficulty, and high-difficulty samples. Low-difficulty samples are characterized by stable speed (within ±1 knot), heading change rate ≤2° / min, and KL divergence <0.5 from the normal trajectory distribution; medium-difficulty samples are characterized by speed fluctuation of 1-3 knots, heading change rate of 2-5° / min, and KL divergence 0.5-1.0 from the normal trajectory distribution; high-difficulty samples are characterized by speed fluctuation >3 knots, heading change rate >5° / min, KL divergence >1.0 from the normal trajectory distribution, or boundary tracks entering restricted areas.

[0172] Phased training:

[0173] Phase 1 (Basic Learning) uses low-difficulty samples to pre-train the CNN-LSTM encoder for a total of 50,000 steps with a learning rate of 1e-3;

[0174] Phase 2 (Medium Difficulty) introduces medium difficulty samples, continues to train the CNN-LSTM encoder and begins training the DDPG policy network, for a total of 100,000 steps, with a learning rate of 5e-4;

[0175] Phase 3 (High Difficulty) introduces high-difficulty samples and jointly trains the CNN-LSTM encoder and DDPG policy network for a total of 150,000 steps with a learning rate of 1e-4.

[0176] In this embodiment of the invention, the course learning adopts a difficulty level based on trajectory complexity. By calculating the KL divergence between the trajectory and the normal trajectory distribution, simple and complex trajectories are automatically distinguished, enabling the model to gradually transition from simple scenarios to complex scenarios and improving the model's ability to identify new abnormal behaviors.

[0177] 3. Multi-task-assisted learning

[0178] The auxiliary task design includes: Task A, ship type classification (cross-entropy loss); Task B, ship next position prediction (mean squared error loss); Task C, sea area risk level estimation (mean squared error loss).

[0179] Loss function design includes:

[0180] The main task loss and the binary classification cross-entropy loss are calculated using the following formulas:

[0181] ;

[0182] Where y is the true label and p is the predicted probability.

[0183] The multi-task loss function uses a weighted summation method, and the formula is as follows:

[0184] ;

[0185] in, These are the weighting coefficients. For location-based loss prediction, For type classification loss, Risk level estimates loss.

[0186] In this embodiment of the invention, multi-task learning adopts a ship navigation state feature sharing mechanism. By sharing the CNN-LSTM encoder, the model can learn ship behavior features from multiple perspectives, thereby improving the ability to identify abnormal ship behavior.

[0187] A composite reward function is used to guide the agent in learning the optimal strategy for identifying abnormal ship behavior. The specific design is as follows:

[0188] 1. Reward Function Structure

[0189] The composite reward function consists of three parts, and the formula is as follows:

[0190] ;

[0191] Among them, w1, w2, and w3 are weighting coefficients with initial values ​​of 1.0, 0.8, and 1.2, respectively.

[0192] Detailed definitions of each part:

[0193] Accuracy Bonus (AccRew):

[0194] When the agent's action a≥0 and the true label y=1 (correctly identified anomaly), AccRew=+1;

[0195] When the agent's action a < 0 and the true label y = 0 (correct recognition is normal), AccRew = +1;

[0196] In other cases, AccRew=0.

[0197] Risk-weighted reward (RiskRew):

[0198] The risk weight for calculating the ship's current position is given by the following formula:

[0199] ;

[0200] Among them, the accident density is the historical accident density of the sea area where the ship is located, and the average density is the average accident density of the entire sea area.

[0201] If the vessel is in a restricted area Add an extra 0.5;

[0202] Risk-weighted reward is:

[0203] ;

[0204] Where T represents the time (in hours) the ship spends in the current sea area.

[0205] Safe area penalty (SafePen):

[0206] When the agent's action a≥0 (judged as abnormal) and the actual label y=0 (actually normal), the safe zone penalty is triggered;

[0207] The penalty coefficient λ varies depending on the type of sea area in which the vessel is located.

[0208] No-navigation zone: λ=2;

[0209] For general water bodies: λ=1;

[0210] Safe zone: λ=0.5.

[0211] The penalty for safe zones is:

[0212] .

[0213] 2. Dynamic weight adjustment

[0214] To balance accuracy, security, and risk perception, an adaptive weighting engine is introduced to adjust weighting coefficients in real time based on validation set metrics.

[0215]

[0216] The reward function employs legal compliance constraints, directly quantifying maritime regulations (such as no-navigation zone regulations) as penalties to ensure that the model output meets maritime safety requirements. Simultaneously, through adaptive weight adjustment based on validation set metrics, the model achieves an optimal balance between accuracy, recall, and false positive rate.

[0217] Beneficial effects: The ship behavior recognition method based on the fusion of dynamic and static data disclosed in this embodiment of the invention proposes a ship behavior recognition method that fuses electronic nautical chart raster images with ship AIS trajectory sequences, which solves the problem that traditional methods only consider the ship's motion state and ignore static environmental constraints;

[0218] The gated attention CNN-LSTM feature fusion architecture disclosed in this invention has a gated attention mechanism that dynamically adjusts the contribution ratio of spatial and temporal features, enabling the model to automatically optimize feature representation based on the ship's environment.

[0219] This invention provides a reinforcement learning framework for identifying abnormal ship behavior based on TD3. It replaces the traditional DDPG with the TD3 algorithm and solves the problems of DDPG being prone to oscillation in high-dimensional state space and sensitive to hyperparameters through improvements such as dual-Q network, delayed policy update and target policy smoothing.

[0220] The composite reward function-guided ship behavior recognition strategy disclosed in this invention design a composite reward function that comprehensively considers behavioral safety, recognition accuracy, and marine risk weights. Through adaptive weight adjustment, the model can achieve an optimal balance among different indicators.

[0221] The course learning and multi-task assisted ship behavior recognition training strategy disclosed in this invention adopts a course learning strategy based on trajectory complexity and multi-task assisted learning based on the sharing of ship navigation state features, which improves the model's ability to identify new abnormal behaviors and its feature representation ability.

[0222] Based on the methods provided in the above embodiments, this invention also provides a ship abnormal behavior identification system based on dynamic and static fusion and deep reinforcement learning. The following describes the ship abnormal behavior identification system based on dynamic and static fusion and deep reinforcement learning in conjunction with the accompanying drawings.

[0223] See Figure 4 This figure is a schematic diagram of the structure of a ship abnormal behavior identification system based on dynamic and static fusion and deep reinforcement learning, provided by an embodiment of the present invention. The system adopts a three-layer architecture design, including a data layer 401, a service layer 402, and an application layer 403. The data layer 401 is responsible for collecting and storing ship navigation data. The service layer 402 provides core processing capabilities, including: data services, which perform preprocessing such as cleaning, denoising, and format unification on AIS data collected from multiple sources; algorithm services, which implement CNN spatial feature extraction, LSTM temporal feature extraction, feature fusion, and DDPG decision output functions; interface services, which provide query interfaces for real-time or historical behavior and status throughout the ship's entire lifecycle; and verification services, which realize online verification and evaluation of model performance, including real-time calculation of indicators such as false positive rate and recall rate. Application layer 403 provides user interaction interface and business functions, including: visualization display, which displays the ship's navigation trajectory, abnormal behavior detection results and alarm information in real time through electronic charts; alarm classification management, which classifies alarms according to risk level (low / medium / high), and high-risk alarms are immediately pushed to management personnel; data analysis and reporting, which provides functions such as ship behavior statistical analysis and abnormal behavior distribution heat map to assist maritime management decision-making.

[0224] The ship abnormal behavior identification system 400 based on dynamic and static fusion and deep reinforcement learning provided in this embodiment of the invention includes: a data layer 401, a service layer 402 and an application layer 403.

[0225] Data layer 401 is used to acquire and store static electronic nautical chart data of the target sea area and AIS dynamic data of the target vessel;

[0226] Service layer 402, connected to the data layer, includes:

[0227] The static data processing module is used to generate static environmental elements based on static electronic nautical chart data, representing at least one of the following elements: navigation marks, warning zones, navigational obstructions, anchorages, channel boundaries, and separation zones.

[0228] The dynamic data processing module is used to clean and verify AIS dynamic data and construct the dynamic environmental elements of the ship.

[0229] The feature extraction and fusion module is used to fuse static environmental elements and dynamic environmental elements to form a fused state input. It uses a convolutional neural network to extract static spatial features, uses a long short-term memory network to extract dynamic temporal features, and uses a gated attention fusion mechanism to fuse static spatial features and dynamic temporal features to obtain a joint state representation.

[0230] The decision module is used to input the joint state representation into the decision network based on the deep reinforcement learning algorithm, and output the ship behavior category to complete the abnormal behavior identification.

[0231] Application layer 403 connects to the service layer and is used to receive and display the ship behavior category identification results.

[0232] In one possible implementation, the static data processing module is specifically used for:

[0233] The static electronic nautical chart data is analyzed to extract the geometric and attribute information of navigation marks, warning zones, obstructions, anchorages, channel boundaries and separation zones;

[0234] Based on the extracted geometric and attribute information, a corresponding mathematical model is constructed, and the mathematical model is converted into a machine-processable data format to generate static environmental elements.

[0235] In one possible implementation, the dynamic data processing module is specifically used for:

[0236] Threshold checks are performed on the position, speed, and heading in AIS dynamic data to remove invalid data;

[0237] Differentiated geometric representation models are used to model water targets with different navigation characteristics in order to obtain dynamic environmental elements.

[0238] In one possible implementation, the feature extraction and fusion module has the following functions:

[0239] Organize static environmental elements into multi-channel raster images;

[0240] Multi-channel raster images are input into a convolutional neural network containing convolutional layers, pooling layers, and attention gating mechanisms to extract static spatial features;

[0241] Organize dynamic environmental elements into a time-series matrix;

[0242] The temporal matrix is ​​input into a long short-term memory network containing bidirectional long short-term memory units and an adaptive attention mechanism to extract dynamic temporal features.

[0243] In one possible implementation, the feature extraction and fusion module has the following functions:

[0244] Project static spatial features onto the same feature dimension as dynamic temporal features;

[0245] The fusion weights of static spatial features and dynamic temporal features are dynamically calculated using a learnable weight matrix.

[0246] The static spatial features and dynamic temporal features are weighted and fused based on the fusion weight.

[0247] In one possible implementation, a training module is also included, which has the following functions:

[0248] The training adopts a strategy that combines course learning with multi-task assisted learning, and uses a framework based on a dual-delay deep deterministic policy gradient algorithm.

[0249] The course includes dividing training samples into different difficulty levels based on trajectory complexity, and training the model in stages from low to high difficulty.

[0250] Multi-task-assisted learning includes training auxiliary tasks such as ship type classification, ship next position prediction, and sea area risk level estimation while training the main task of abnormal behavior identification.

[0251] When training the decision network, a composite reward function is used to calculate the reward value. The composite reward function includes at least an accuracy reward, a risk-weighted reward, and a safe zone penalty.

[0252] In one possible implementation, the decision module has features for:

[0253] The Actor policy network outputs continuous action values ​​based on the joint state representation. When the continuous action value is greater than or equal to 0, it is judged as abnormal behavior; when the continuous action value is less than 0, it is judged as normal behavior.

[0254] The core model adopts a hybrid CNN-LSTM-DDPG architecture, mainly consisting of a fusion state input layer, a CNN spatial feature extraction layer, an LSTM temporal feature extraction layer, a feature fusion module, and a DDPG decision output layer.

[0255] First, the fusion state input layer receives the normalized fusion state tensor S', which is composed of static and dynamic environmental elements:

[0256] Static environmental elements include electronic nautical chart raster images containing multi-channel geographic information (such as channels, restricted areas, and port boundaries), with dimensions of H×W×Cstatic, where Cstatic represents the number of channels. Specifically, it includes: a water depth raster map (1 channel): obtained by converting an S57 vector chart to a raster, with a resolution of 256×256 and a value range of 0-30 meters; a channel boundary map (1 channel): represented in binary form, with 1 inside the channel and 0 outside; a restricted area map (1 channel): represented in binary form, with 1 for the restricted area and 0 for the navigable area; and a navigation mark distribution map (1 channel): a heat map showing the location of navigation marks, with a value range of 0-1.

[0257] The dynamic environmental elements are time-aligned AIS trajectory sequences and meteorological data, organized as a T×Ddynamic time series matrix, where T is the time step, set to 120 minutes, with a time interval of 5 minutes; Ddynamic is the feature dimension, including features such as ship speed, heading, position, turning rate (rate of change of heading / minute), rate of change of speed (rate of change of speed / minute), and density of surrounding ships.

[0258] Second, the CNN spatial feature extraction layer is responsible for extracting static spatial features from electronic charts.

[0259] The first convolutional layer uses 32 5×5 convolutional kernels with a stride of 1, the same padding method, and ReLU activation function, with an output feature map size of H×W×32; the second convolutional layer uses 64 3×3 convolutional kernels with a stride of 1, the same padding method, and ReLU activation function, with an output feature map size of H×W×64; the max pooling layer uses a 2×2 window with a stride of 2, max pooling, with an output feature map size of (H / 2)×(W / 2)×64;

[0260] Feature flattening flattens the pooled feature map into a vector of length . ;

[0261] The output is a static space feature vector. Where H is the image height and W is the image width. It represents the set of real numbers.

[0262] The CNN spatial feature extraction layer employs an improved attention gating mechanism, dynamically weighting the contribution of different geographical elements to anomaly detection. Specifically, an SE (Squeeze-and-Excitation) block is inserted after the second convolutional layer. Channel weights are obtained through global average pooling, then compressed and expanded through a fully connected layer, and finally, channel attention weights are obtained through a sigmoid function. The feature map x is then weighted according to its channel dimension, as shown in the formula:

[0263] ;

[0264] ;

[0265] Where GAP represents global average pooling, W1 and W2 are the parameters of the fully connected layer, δ is the channel attention weight, and x' is the weighted feature map.

[0266] Third, the LSTM temporal feature extraction layer is responsible for extracting the temporal features of the ship's trajectory.

[0267] The bidirectional LSTM encoder consists of two stacked layers of bidirectional LSTM units, with forward and backward hidden states concatenated, resulting in an output dimension of 2 × 128 = 256. The first LSTM layer contains 128 hidden units, retaining intermediate sequence outputs; the second LSTM layer contains 128 hidden units, outputting only the final hidden state. The attention mechanism applies an adaptive attention mechanism to the output of the bidirectional LSTM, assigning higher weights to key points in the ship's trajectory (such as abrupt changes in heading and speed), as shown in the formula:

[0268] ;

[0269] ;

[0270] in, For LSTM at time steps The output, The weighted output, For attention weights, , , These are learnable parameters.

[0271] The output is a dynamic temporal feature vector ftemporal∈R^256.

[0272] The LSTM layer employs chart-constrained trajectory prediction, using chart grid information as a mask input to prevent the predicted trajectory from deviating from restricted areas or shoals. Specifically, this is achieved by overlaying a chart constraint mask onto the LSTM input, using the following formula:

[0273] ;

[0274] in, The chart constraint mask represents the ship's current status. It is generated based on the ship's current position and safety status on the chart. The mask value for safe areas is 0, and the mask value for restricted areas is 1, which constrains the trajectory prediction.

[0275] Fourth, the feature fusion module is responsible for fusing spatial features with temporal features.

[0276] Perform linear projection, projecting the spatial feature vector fspecial output by the CNN through a linear layer to the same dimension (256 dimensions) as the LSTM output.

[0277] The contribution of spatial and temporal features is dynamically adjusted through learnable gating weights, as shown in the formula:

[0278] ;

[0279] in, For the Sigmoid function, and For learnable parameters, This indicates element-wise multiplication, and z represents the fused joint state with a dimension of 256.

[0280] The feature fusion module adopts a gated attention mechanism, which dynamically adjusts the contribution ratio of spatial and temporal features through a learnable weight matrix, enabling the model to automatically adjust feature weights according to the current environment of the ship, thereby improving its adaptability to different scenarios.

[0281] Fifth, the DDPG decision output layer consists of an Actor network and a Critic network.

[0282] The input to the Actor network is the fused joint state. The network structure is a three-layer fully connected network (256→128→64→1), with ReLU activation function used between layers; the output is a continuous action value. , indicating the category of ship behavior; when a≥0, it is judged as abnormal; when a<0, it is judged as normal.

[0283] The target network is the Target Actor network, and its parameters are updated synchronously with those of the Actor network.

[0284] The input to the Critic network is the fused joint state. and actions ;

[0285] The network structure is a four-layer fully connected network (257→128→64→32→1), with ReLU activation function used between layers; the output is the state-action value Q(z, a). The target network is the Target Critic network, and its parameters are updated synchronously with the Critic network.

[0286] The DDPG layer uses the TD3 (Twin Delayed DDPG) algorithm instead of the traditional DDPG, improving model stability and convergence speed through the following improvements:

[0287] Dual Q network: The Critic network uses two independent Q networks and takes the minimum value as the state-action value estimate to alleviate the problem of Q-value overestimation;

[0288] Delayed update strategy: The Actor network updates only once every 2 steps of Critic update, improving training stability;

[0289] Target policy smoothing: Add Gaussian noise to the target action, as shown in the formula:

[0290] ;

[0291] in, For the target Actor network, It is Gaussian noise. This represents the standard deviation of noise.

[0292] Since the ship abnormal behavior identification system 400 based on dynamic-static fusion and deep reinforcement learning is a system corresponding to the ship abnormal behavior identification method based on dynamic-static fusion and deep reinforcement learning provided in the above method embodiments, the specific implementation of each unit of the ship abnormal behavior identification system 400 based on dynamic-static fusion and deep reinforcement learning is based on the same concept as in the above method embodiments. Therefore, for the specific implementation of each unit of the ship abnormal behavior identification system 400 based on dynamic-static fusion and deep reinforcement learning, please refer to the description of the ship abnormal behavior identification method based on dynamic-static fusion and deep reinforcement learning in the above method embodiments, and it will not be repeated here.

[0293] This invention also provides a ship abnormal behavior identification device based on dynamic and static fusion and deep reinforcement learning, the device comprising: a processor and a memory;

[0294] The memory is used to store instructions;

[0295] The processor is used to execute the instructions in the memory to perform the ship abnormal behavior identification method based on dynamic and static fusion and deep reinforcement learning mentioned in the above embodiments.

[0296] It should be noted that the hardware structure of the ship abnormal behavior identification device based on dynamic and static fusion and deep reinforcement learning provided in the embodiments of the present invention can be as follows: Figure 5 The structure shown, Figure 5 This is a schematic diagram of the structure of a device provided in an embodiment of the present invention.

[0297] Please see Figure 5 As shown, device 500 includes: processor 510, communication interface 520, and memory 530. The number of processors 510 in device 500 can be one or more. Figure 5Taking a processor as an example, in this embodiment of the invention, the processor 510, communication interface 520, and memory 530 can be connected via a bus system or other means. Figure 5 Taking the connection between China and Israel via the 540 bus system as an example.

[0298] Processor 510 may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP. Processor 510 may further include hardware chips. These hardware chips may be application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0299] Memory 530 may include volatile memory, such as random-access memory (RAM); memory 530 may also include non-volatile memory, such as flash memory, hard disk drive (HDD) or solid-state drive (SSD); memory 530 may also include a combination of the above types of memory.

[0300] Optionally, the memory 530 stores an operating system and programs, executable modules, or data structures, or subsets thereof, or extended sets thereof. The programs may include various operation instructions for implementing various operations. The operating system may include various system programs for implementing various basic business processes and handling hardware-based tasks. The processor 510 can read the programs in the memory 530 to implement the ship anomaly behavior identification method based on dynamic-static fusion and deep reinforcement learning provided in this embodiment of the invention.

[0301] The bus system 540 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus system 540 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0302] This invention also provides a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the ship abnormal behavior identification method based on dynamic-static fusion and deep reinforcement learning mentioned in the above embodiments.

[0303] This invention also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute the ship abnormal behavior identification method based on dynamic and static fusion and deep reinforcement learning mentioned in the above embodiments.

[0304] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.

Claims

1. A method for identifying abnormal ship behavior based on dynamic and static fusion and deep reinforcement learning, characterized in that, The method includes: Acquire static electronic nautical chart data of the target sea area and AIS dynamic data of the target vessel; Based on the static electronic chart data, static environmental elements representing at least one of the following elements are generated: parsing the static electronic chart data to extract the geometric and attribute information of the navigation marks, warning zones, navigation obstructions, anchorages, channel boundaries, and separation zones; constructing corresponding mathematical models based on the extracted geometric and attribute information; and converting the mathematical models into a machine-processable data format to generate the static environmental elements. The AIS dynamic data of the target vessel is cleaned and verified to construct the vessel's dynamic environmental elements; The static environmental elements and the dynamic environmental elements are merged to form a merged state input; Static spatial features in the fused state input are extracted using a convolutional neural network, and dynamic temporal features in the fused state input are extracted using a long short-term memory network. The static spatial features and the dynamic temporal features are fused through a gating attention fusion mechanism to obtain a joint state representation; Based on a deep reinforcement learning algorithm, the joint state representation is input into a decision network, and the ship behavior category is output to complete the identification of abnormal behavior. The training of the convolutional neural network, the long short-term memory network, and the decision network includes: the training employs a strategy combining curriculum learning and multi-task assisted learning, and adopts a framework based on a dual-delay deep deterministic policy gradient algorithm; the curriculum learning includes dividing training samples into different difficulty levels according to trajectory complexity, and training the model in stages from low difficulty to high difficulty; the multi-task assisted learning includes simultaneously training auxiliary tasks such as ship type classification, ship next position prediction, and sea area risk level estimation while training the main task of abnormal behavior identification; when training the decision network, a composite reward function is used to calculate the reward value, and the composite reward function includes at least an accuracy reward, a risk-weighted reward, and a safe zone penalty.

2. The method according to claim 1, characterized in that, The process of cleaning and verifying the AIS dynamic data of the target vessel to construct the vessel's dynamic environmental elements includes: Threshold checks are performed on the position, speed, and heading in AIS dynamic data to remove invalid data; Differentiated geometric representation models are used to model water targets with different navigation characteristics in order to obtain the dynamic environmental elements.

3. The method according to claim 1, characterized in that, The step of extracting static spatial features from the fused state input using a convolutional neural network and extracting dynamic temporal features from the fused state input using a long short-term memory network includes: The static environmental elements are organized into a multi-channel raster image; The multi-channel raster image is input into a convolutional neural network containing convolutional layers, pooling layers, and an attention gating mechanism to extract the static spatial features; The dynamic environmental elements are organized into a time-series matrix; The time series matrix is ​​input into a long short-term memory network containing bidirectional long short-term memory units and an adaptive attention mechanism to extract the dynamic time series features.

4. The method according to claim 1, characterized in that, The process of fusing the static spatial features with the dynamic temporal features through a gated attention fusion mechanism includes: The static spatial features are projected onto the same feature dimension as the dynamic temporal features; The fusion weights of the static spatial features and the dynamic temporal features are dynamically calculated using a learnable weight matrix. The static spatial features and the dynamic temporal features are weighted and fused according to the fusion weight.

5. The method according to claim 1, characterized in that, The decision-making network includes an Actor strategy network and a Critic value network; the output ship behavior categories include: The Actor policy network outputs continuous action values ​​based on the joint state representation. When the continuous action value is greater than or equal to 0, it is determined to be abnormal behavior; when the continuous action value is less than 0, it is determined to be normal behavior.

6. A ship abnormal behavior identification system based on dynamic and static fusion and deep reinforcement learning, characterized in that, It includes the data layer, service layer, and application layer: The data layer is used to acquire and store static electronic nautical chart data of the target sea area and AIS dynamic data of the target vessel; The service layer, connected to the data layer, includes: The static data processing module is used to generate static environmental elements representing at least one of the following elements based on the static electronic nautical chart data: navigation marks, warning zones, navigational obstructions, anchorages, channel boundaries, and separation zones. The static data processing module is specifically used to parse the static electronic nautical chart data, extract the geometric and attribute information of the navigation marks, warning zones, navigational obstructions, anchorages, channel boundaries and separation zones; based on the extracted geometric and attribute information, construct the corresponding mathematical model, and convert the mathematical model into a machine-processable data format to generate the static environmental elements; The dynamic data processing module is used to clean and verify the AIS dynamic data and construct the dynamic environmental elements of the ship. The feature extraction and fusion module is used to fuse the static environmental elements and the dynamic environmental elements to form a fused state input. It uses a convolutional neural network to extract static spatial features, uses a long short-term memory network to extract dynamic temporal features, and uses a gated attention fusion mechanism to fuse the static spatial features and the dynamic temporal features to obtain a joint state representation. The decision module is used to input the joint state representation into the decision network based on a deep reinforcement learning algorithm, and output the ship behavior category to complete the abnormal behavior identification. The application layer, connected to the service layer, is used to receive and display the ship behavior category identification results; The training module is used to train the convolutional neural network, the long short-term memory network, and the decision network. The training employs a strategy combining curriculum learning and multi-task-assisted learning, and uses a framework based on a dual-delay deep deterministic policy gradient algorithm. The curriculum learning includes dividing training samples into different difficulty levels based on trajectory complexity and training the model in stages from low to high difficulty. The multi-task-assisted learning includes simultaneously training auxiliary tasks such as ship type classification, ship next position prediction, and sea area risk level estimation while training the main task of abnormal behavior identification. When training the decision network, a composite reward function is used to calculate the reward value, which includes at least an accuracy reward, a risk-weighted reward, and a safe zone penalty.

7. An electronic device, characterized in that, The device includes: a processor and a memory; The memory is used to store instructions; The processor is configured to execute the instructions in the memory to perform the method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, Including instructions that, when run on a computer, cause the computer to perform the method described in any one of claims 1-5 above.

Images