Inland ship abnormal behavior sensing method and device based on incremental graph convolutional network

Abstract

The invention provides an inland ship abnormal behavior sensing method and device fusing multi-modal data based on an incremental graph convolutional network. Comprising the following steps: acquiring laser point cloud data, video image data, AIS data and navigable water level data of an inland ship; the obtained four types of data are preprocessed; video and laser point cloud data fusion: based on a two-stage front and back fusion algorithm, fusing the preprocessed data to obtain a ship image target with accurate three-dimensional information; video and AIS data fusion: fusing the preprocessed video image data and AIS data to obtain a ship target containing ship attribute data; based on the ship feature data, constructing a semantic clustering incremental graph model S-GCN; constructing an incremental graph model M-GCN of multi-modal information association fusion; constructing an incremental graph convolution framework graph based on multi-modal information fusion abnormal behavior perception of an incremental graph model; and matching the feature data to obtain fused multi-modal feature data, inputting the fused multi-modal feature data into a detection framework network, and showing relatively high detection performance, detection accuracy and effectiveness in ship yaw early warning detection, ship bridge crossing early warning detection and ship collision early warning detection tasks.

Description

Technical Field

[0001] The present invention relates to the technical field of abnormal behavior perception of ships, and particularly to a method and device for perceiving abnormal behavior of inland river ships based on an incremental graph convolutional network. Background Technique

[0002] The supervision of inland river ships relies on various types of sensor perception data, such as AIS data, video surveillance data, radar data, water-related business management data, navigable environment element data, etc. The perception of inland river ship behavior, especially the perception of abnormal behavior events, is the key content of inland river ship supervision. In the field of ship abnormal event perception, current research mainly focuses on using AIS data for abnormal detection and early warning, covering the optimization of classification and clustering algorithms, geometric feature extraction, and the innovative application of machine learning and deep learning models.

[0003] In terms of classification and clustering methods, a prediction framework of clustering and integrated machine learning algorithms is used to predict the risky encounters between ships; in the analysis method based on geometric features, by analyzing the redundancy and curvature of the AIS trajectories of ships and combining with the DBSCAN clustering algorithm, ship trajectories are divided into normal and abnormal trajectories; in the application of machine learning methods, through the AIS ship trajectories, the motion parameter curves of ship trajectories are generated, and the sliding window algorithm is used to detect abnormal ship behaviors. In the application of the feedforward random neural network model, the model based on random vector functional link (RVFL) has significant advantages in prediction accuracy and computational efficiency, especially showing strong generalization ability in short-term trajectory prediction tasks. In addition, the detection method based on visual data also opens up a new direction for ship abnormal event detection and early warning, identifying abnormal behaviors that deviate significantly from the normal route. This method is particularly effective for narrow coastal areas, especially those areas where AIS data is insufficient.

[0004] Existing Problems:

[0005] In the field of ship abnormal event detection and early warning, existing technical means have their own advantages and disadvantages. Classification and clustering technologies can effectively process large-scale data and are particularly suitable for quickly identifying risky encounters between ships, but they have high requirements for data quality and are easily affected by noise; geometric feature analysis identifies abnormal behaviors through trajectory morphology and is suitable for real-time monitoring, but it is unstable when dealing with irregular routes; machine learning and deep learning methods perform well in dealing with complex non-linear problems and can identify various abnormal behaviors, but they rely on a large amount of labeled data and the training process is time-consuming; the feedforward random neural network model has high computational performance in short-term prediction tasks, but improper model design may lead to overfitting; the detection method based on visual data provides an effective supplement in the case of insufficient AIS data and is particularly suitable for abnormal detection in complex environments, but it is sensitive to hardware and environmental factors.

[0006] In summary, classification and clustering techniques, geometric feature analysis, machine learning and deep learning models, as well as feedforward stochastic neural networks and vision data-based detection methods have all demonstrated important application values in the field of ship supervision. Existing technical means have their own advantages in the detection and early warning of ship abnormal events, but there are also certain limitations. These technical means can be combined to make up for their respective deficiencies, providing a more comprehensive solution for the detection and early warning of ship abnormal events. By comprehensively leveraging their respective advantages, more intelligent and diversified ship abnormal detection methods can be constructed. For example, classification and clustering techniques can be combined with machine learning models to improve the detection accuracy and robustness by fusing information at different levels. At the same time, with the development of multi-source data fusion technology, integrating various data sources such as AIS data, vision data, and lidar point cloud data can provide richer feature expressions for abnormal detection, making up for the deficiencies of single data sources.

[0007] (1) The system stability of abnormal event perception is insufficient. This is mainly reflected in the relatively single current data source and the insufficient ability of the system to resist single risks.

[0008] (2) The environmental adaptability of abnormal event perception needs to be further improved. Video detection has poor adaptability in low visibility environments; lidar data is relatively accurate, but the effective range is limited; the influence of navigable environments such as water levels on abnormal event detection is not considered; existing systems mainly use AIS and cannot achieve active perception, and cannot fully utilize the advantages of multi-modal data.

[0009] (3) The accuracy of abnormal event perception needs to be further improved. The accuracy of abnormal behavior detection of the current system needs to be improved. Summary of the Invention

[0010] The present invention aims to solve at least one of the technical problems existing in the prior art, and provides an inland river ship abnormal behavior perception method and device based on an incremental graph convolutional network.

[0011] In a first aspect, the present invention provides an inland river ship abnormal behavior perception method based on an incremental graph convolutional network, including:

[0012] Obtain lidar point cloud data, video image data, AIS data, and navigable water level data of inland river ships;

[0013] Preprocess the obtained lidar point cloud data, video image data, AIS data, and navigable water level data;

[0014] Fuse the laser point cloud data and video data. Based on the two-stage front and back fusion algorithm, fuse the preprocessed video image data and laser point cloud data to obtain a ship image target with three-dimensional depth information;

[0015] Fuse the video data and AIS data. Based on the multi-feature parameter global optimal association algorithm, fuse the preprocessed video image data and AIS data to obtain a ship target containing ship attribute data;

[0016] Construct a semantic clustering incremental graph model S-GCN based on laser point cloud feature data, video image feature data, AIS feature data, and water level feature data;

[0017] Construct an incremental graph model M-GCN for multi-modal information association and fusion;

[0018] Construct an incremental graph convolutional framework network for multi-modal information fusion and abnormal behavior perception based on the incremental graph model;

[0019] Match the laser point cloud feature data, video image feature data, and AIS feature data with the water level feature data to obtain fused multi-modal feature data;

[0020] Input the fused multi-modal feature data into the incremental graph convolutional framework network to conduct ship abnormal behavior perception detection.

[0021] In some possible embodiments, the step of fusing the preprocessed video image data and laser point cloud data based on the two-stage front and back fusion algorithm to obtain a ship image target with three-dimensional depth information includes:

[0022] For the front fusion, perform cross-overlap calculation on the target candidate regions extracted from the top view features of the laser point cloud data and the feature candidate regions extracted from the image, and recommend the top K regions of interest (ROI) according to the target box selection coverage rate of the overlapping part. Send the fused ROI into the region of interest alignment layer (ROI Align) to be integrated into the same dimension and then send it to the subsequent network for fusion analysis;

[0023] For the back fusion, perform target fusion on the three-dimensional bounding boxes and two-dimensional target boxes extracted from the laser point cloud data and video image data respectively. Transform the three-dimensional bounding box information to the image coordinate system through time synchronization and spatial transformation matrix to obtain the ship image target with three-dimensional depth information.

[0024] In some possible embodiments, the step of fusing the preprocessed video image data and AIS data based on the multi-feature parameter global optimal association algorithm to obtain a ship target containing ship attribute data includes:

[0025] Extract ship features based on the video image data; wherein, the ship features include at least one of position, size, speed, and gray-scale statistics;

[0026] Extract ship information based on the AIS data; wherein, the ship information includes at least one of ship name, position, size, speed, and course;

[0027] Associate the ship features and the ship information belonging to the same ship target to obtain a ship target containing ship attribute data.

[0028] In some possible embodiments, constructing a semantic clustering incremental graph model S-GCN based on the laser point cloud feature data, video image feature data, AIS feature data, and water level feature data includes:

[0029] Cluster the laser point cloud feature data, video image feature data, AIS feature data, and water level feature data, generate a new graph and then perform graph learning operations to obtain new vertex features considering the information at the semantic level of the sensors;

[0030] Use mean pooling to complete the fusion of the same sensor feature vertices. The edges between the fused vertices and the new vertices are calculated according to the similarity of the vertex features, and the cosine distance is used to determine the weights of the edges in the new graph to obtain the semantic clustering incremental graph model S-GCN.

[0031] In some possible embodiments, constructing a multi-modal information association and fusion incremental graph model M-GCN includes:

[0032] Use the multi-head attention mechanism to process the graph at time t, find the potential relationships between different modal data, generate several new graphs after processing, and update the features of the nodes through graph learning; fuse these features together to obtain multiple features with the same number as the nodes in the graph to obtain the multi-modal information association and fusion incremental graph model M-GCN.

[0033] In some possible embodiments, constructing an incremental graph convolution framework network for multi-modal information fusion of abnormal events based on the incremental graph model includes:

[0034] Connect the features obtained from the semantic clustering incremental graph model S-GCN and the features obtained from the multi-modal information association and fusion incremental graph model M-GCN, and then construct an incremental graph convolution framework network through a fully connected layer and softmax.

[0035] In some possible embodiments, matching the laser point cloud feature data, video image feature data, and AIS feature data with the water level feature data to obtain fused multi-modal feature data includes:

[0036] Match the laser point cloud feature data, video image feature data, and AIS feature data with the water level information according to the corresponding timestamps and geographical location information to obtain the fused multi-modal feature data.

[0037] In a second aspect, the present invention provides an abnormal behavior perception device for inland river ships based on an incremental graph convolutional network, including:

[0038] An acquisition module for acquiring the laser point cloud data, video image data, AIS data, and navigable water level data of inland river ships;

[0039] A preprocessing module for preprocessing the acquired laser point cloud data, video image data, AIS data, and navigable water level data;

[0040] A first fusion module for fusing the preprocessed video image data and laser point cloud data based on a two-stage front and back fusion algorithm to obtain a ship image target with three-dimensional depth information;

[0041] A second fusion module for fusing the preprocessed video image data and AIS data based on a multi-feature parameter global optimal association algorithm to obtain a ship target containing ship attribute data;

[0042] A first construction module for constructing a semantic clustering incremental graph model S-GCN based on laser point cloud feature data, video image feature data, AIS feature data, and water level feature data;

[0043] A second construction module for constructing an incremental graph model M-GCN for multi-modal information association and fusion;

[0044] A third construction module for constructing an incremental graph convolutional framework network for multi-modal information fusion and abnormal behavior perception based on an incremental graph model;

[0045] A matching module for matching the laser point cloud feature data, video image feature data, and AIS feature data with the water level feature data to obtain the fused multi-modal feature data;

[0046] A perception module for inputting the fused multi-modal feature data into the incremental graph convolutional framework network to perform ship abnormal behavior perception and detection.

[0047] In a third aspect, the present invention provides an electronic device, characterized by including:

[0048] One or more processors;

[0049] A storage unit for storing one or more programs, which, when executed by the one or more processors, enable the one or more processors to implement the method for perceiving abnormal behaviors of inland river ships based on the incremental graph convolutional network described above.

[0050] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, it can implement the method for perceiving abnormal behaviors of inland river ships based on the incremental graph convolutional network described above.

[0051] The beneficial effects of the present invention are as follows:

[0052] The method and device for perceiving abnormal behaviors of inland river ships based on the incremental graph convolutional network according to the embodiments of the present invention are used for detecting abnormal behavior events of ships. The incremental graph multi-modal event detection algorithm realizes the temporal dynamics and multi-modal association of video images, lidar point clouds, water levels of waterways, and AIS data by combining the S-GCN based on semantic information clustering of ship information and water level information and the M-GCN at the multi-modal information fusion level. The S-GCN focuses on perceiving the changes in data at the temporal level and captures dynamic features through clustering and mean pooling; the M-GCN uses the multi-head attention mechanism to associate the fusion relationships of multi-modal information in the graph structure, generates multiple fully connected graphs and performs graph learning, so as to capture the potential associations between different perception data. The incremental graph multi-modal event detection algorithm shows high detection performance in tasks such as ship yaw warning detection, ship bridge-crossing warning detection, and ship collision warning detection, especially improving the detection accuracy and effectiveness of the algorithm in complex scenarios. Description of the Drawings

[0053] Figure 1 It is a schematic structural diagram of an example electronic device for the method for perceiving abnormal behaviors of inland river ships based on the incremental graph convolutional network according to an embodiment of the present invention;

[0054] Figure 2 It is a flowchart of the method for perceiving abnormal behaviors of inland river ships based on the incremental graph convolutional network according to another embodiment of the present invention;

[0055] Figure 3 It is a schematic diagram of the GCN process based on semantic information clustering of ship information and water level information according to another embodiment of the present invention;

[0056] Figure 4 It is a schematic diagram of the GCN process for multi-information association and fusion according to another embodiment of the present invention;

[0057] Figure 5 It is a framework diagram of the incremental graph convolutional network according to another embodiment of the present invention

[0058] Figure 6Schematic structural diagram of an abnormal behavior perception device for inland river ships based on an incremental graph convolutional network according to another embodiment of the present invention. Detailed implementation manners

[0059] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Apparently, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0060] In the description of the present invention, it should be noted that the orientation or positional relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. is based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation to the present invention. In addition, the terms "first", "second", and "third" are only used for descriptive purposes and should not be construed as indicating or implying relative importance.

[0061] In the description of the present invention, it should be noted that unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0062] Figure 1 Schematic structural diagram of an exemplary electronic device for implementing an abnormal behavior perception method for inland river ships based on an incremental graph convolutional network according to an embodiment of the present invention. As Figure 1 shown, the electronic device 100 includes one or more processors 110, one or more storage devices 120, one or more input devices 130, one or more output devices 140, etc., and these components are interconnected through a bus system 150 and / or other forms of connection mechanisms. It should be noted that Figure 1 the components and structures of the shown electronic device are only exemplary and not restrictive. According to needs, the electronic device can also have other components and structures.

[0063] The processor 110 can be a central processing unit (CPU), or can be composed of multiple processing cores, or have other forms of processing units with data processing capabilities and / or instruction execution capabilities, and can control other components in the electronic device 100 to perform desired functions.

[0064] The storage device 120 may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage media, and the processor may run the program instructions to implement the client functions (implemented by the processor) in the embodiments of the present disclosure described below and / or other desired functions. Various application programs and various data may also be stored in the computer-readable storage media. For example, various data used and / or generated by the application programs, etc.

[0065] The input device 130 may be a device used by a user to input instructions, and may include one or more of a keyboard, a mouse, a microphone, a touch screen, etc.

[0066] The output device 140 may output various information (such as images or sounds) to the outside (such as the user), and may include one or more of a display, a speaker, etc.

[0067] Figure 2 It is a flowchart of a method for perceiving abnormal behaviors of inland river ships based on an incremental graph convolutional network according to another embodiment of the present invention. As Figure 2 shown, a method for perceiving abnormal behaviors of inland river ships based on an incremental graph convolutional network includes the following steps S210 to step S280:

[0068] Step S210, obtain the laser point cloud data, video image data, AIS data, and navigable water level data of the inland river ship.

[0069] Specifically, in this step, a laser scanner (such as LiDAR) installed on the ship can be used to collect detailed three-dimensional environmental information. Such equipment can generate accurate point cloud data for describing the ship and its surrounding environment. Alternatively, laser scanning and point cloud data can also be obtained from a geographic information service company. For video image data, surveillance cameras can be set up at key positions in the inland waterway to record the video data of passing ships in real time. Or, an unmanned aerial vehicle equipped with a high-definition camera can be used for aerial photography to obtain high-quality ship video image data. For AIS data, it can be obtained through the Automatic Identification System (AIS), and these information can be obtained by subscribing to the AIS data service provided by a third party. For navigable water level data, most important inland waterways have official hydrological stations that regularly release water level data. These data can be queried through the local water conservancy department or the official website.

[0070] Step S220: Preprocess the obtained laser point cloud data, video image data, AIS data, and navigable water level data.

[0071] Specifically, in this step, the preprocessing of the obtained laser point cloud data, video image data, AIS data, and navigable water level data may specifically include eliminating false data, eliminating duplicate data, data normalization, data completion, and related clustering algorithms, etc.

[0072] Step S230: Fuse the laser point cloud data and video data. Based on a two-stage front and back fusion algorithm, fuse the preprocessed video image data and laser point cloud data to obtain a ship image target with three-dimensional depth information.

[0073] Specifically, in this step, for the front fusion, cross-overlap calculation is performed on the target candidate regions extracted from the top view features of the laser point cloud data and the feature candidate regions extracted from the image, and the top K Regions of Interest (ROIs) are recommended according to the target box selection coverage rate of the overlapping part. The fused ROIs are sent to the Region of Interest Alignment layer ROIAlign for integration into the same dimension and then sent to the subsequent network for fusion analysis. This part can effectively focus on the regions with relatively high target probabilities, thereby reducing the computational complexity of the algorithm in the later stage. At the same time, region fusion can utilize feature points of different modalities to improve the recall rate of the target. For the back fusion, target fusion is performed on the three-dimensional bounding boxes and two-dimensional target boxes extracted from the laser point cloud data and the video image data respectively. The three-dimensional bounding box information is transformed to the image coordinate system through time synchronization and a spatial transformation matrix to obtain the ship image target with three-dimensional depth information.

[0074] Step S240: Fuse video data and AIS data. Based on the global optimal association algorithm for multi-feature parameters, fuse the preprocessed video image data and AIS data to obtain a ship target containing ship attribute data.

[0075] Specifically, in this step, extract ship features based on the video image data; wherein, the ship features include at least one of position, size, speed, and gray-scale statistics; extract ship information based on the AIS data; wherein, the ship information includes at least one of ship name, position, size, speed, and course; associate the ship features and the ship information belonging to the same ship target to obtain a ship target containing ship attribute data.

[0076] In this step, for the association problem between AIS data and video image data, a global optimal association algorithm for multi-feature parameters is proposed. The JVC algorithm is used to solve the optimal matching resources, thereby improving the accuracy of the association between AIS and video image ship targets in a dense environment. Ship features such as position, size, speed, and gray-scale statistics can be extracted from the video image; the information that can be extracted by AIS includes information such as ship name, position, size, speed, and course. Therefore, the ship position, size, and speed that can be jointly detected by the two are used as the feature information for determining whether they are the same ship target, and the associated ship target has richer attribute information.

[0077] Step S250: Based on the laser point cloud feature data, video image feature data, AIS feature data, and water level feature data, construct a semantic clustering incremental graph model S-GCN.

[0078] Specifically, in this step, refer to Figure 3 , Figure 3 which is a schematic diagram of the GCN process for semantic information clustering based on ship information and water level information in another embodiment of the present invention. Cluster the laser point cloud feature data, video image feature data, AIS feature data, and water level feature data, generate a new graph and then perform graph learning operations to obtain new vertex features considering the information at the semantic level of the sensors; use mean pooling to complete the fusion of the same sensor feature vertices, and calculate the edges between the fused vertices and the new vertices according to the similarity of the vertex features, and use the cosine distance to determine the weights of the edges in the new graph to obtain the semantic clustering incremental graph model S-GCN.

[0079] In this step, focus on the impact of sensors on the learning of new ship features, cluster the previous ship features according to the corresponding sensors, generate a new graph and then perform graph learning operations, so that the obtained new vertex features consider the information at the semantic level of the sensors. The specific process is as Figure 3As shown in the figure. Mean pooling is used to complete the fusion of the same sensor feature vertices. The edges between the fused vertices and the new vertices are calculated based on the similarity of the vertex features, and the cosine distance is used to determine the weights of the edges in the new graph.

[0080] Step S260: Construct an incremental graph model M - GCN for multi - modal information association and fusion.

[0081] Specifically, in this step, refer to Figure 4 , Figure 4 which is a schematic diagram of the GCN process for multi - information association and fusion in another embodiment of the present invention. As Figure 4 shown, the graph at time t is processed using the multi - head attention mechanism to find the potential relationships between different - modality data. After processing, several new graphs are generated, and the features of the nodes are updated through graph learning; these features are fused together to obtain multiple features with the same number as the nodes in the graph, thus obtaining the incremental graph model M - GCN for multi - modal information association and fusion.

[0082] In this step, as the perception data is continuously collected, the number of nodes in the incremental graph convolutional model also increases accordingly. At any given moment, the M - GCN at the multi - information association and fusion level is used to mine the potential associations between data. The specific method is to process the graph at time t using the multi - head attention mechanism to find the potential relationships between different - modality data. After processing, several new graphs are generated, and then the features of the nodes are updated through graph learning. Finally, these features are fused together to obtain multiple features with the same number as the nodes in the graph, and the specific process is as Figure 4 shown.

[0083] Step S270: Construct an incremental graph convolutional framework network for multi - modal information fusion and abnormal behavior perception based on the incremental graph model.

[0084] Specifically, in this step, refer to Figure 5 , Figure 5 which is an incremental graph convolutional framework network in another embodiment of the present invention. The features obtained from the semantic clustering incremental graph model S - GCN and the features obtained from the incremental graph model M - GCN for multi - modal information association and fusion are connected, and then an incremental graph convolutional framework network is constructed through a fully - connected layer and softmax.

[0085] Step S280: Match the laser point cloud feature data, video image feature data, and AIS feature data with the water level feature data to obtain the fused multi - modal feature data.

[0086] Specifically, in this step, the laser point cloud feature data, video image feature data, and AIS feature data are matched with the water level information according to the corresponding timestamps and geographical location information to obtain the fused multi-modal feature data.

[0087] In this step, the laser point cloud feature data, video image feature data, and AIS feature data are matched with the water level information according to the corresponding timestamps and geographical location information to ensure the consistency of various types of data in terms of time and space, thereby generating the fused multi-modal feature data. These multi-modal feature data integrate key factors such as the three-dimensional spatial information, movement trajectory, real-time dynamics, and environmental water level of the ship.

[0088] Step S290: Input the fused multi-modal feature data into the incremental graph convolutional framework network to perform ship abnormal behavior perception detection.

[0089] Specifically, in this step, these fused multi-modal feature data are input into the incremental graph convolutional framework network constructed in step S270 to complete the ship abnormal event detection task.

[0090] The method for perceiving abnormal behaviors of inland river ships based on the incremental graph convolutional network in the embodiments of the present invention is used for detecting ship abnormal behavior events. The incremental graph multi-modal event detection algorithm realizes the temporal dynamics and multi-modal association of video images, lidar point clouds, water levels of waterways, and AIS data by combining S-GCN based on semantic information clustering of ship information and water level information and M-GCN at the multi-modal information fusion level. S-GCN focuses on perceiving the changes in data at the temporal level and captures dynamic features through clustering and mean pooling; M-GCN uses the multi-head attention mechanism to associate the fusion relationships of multi-modal information in the graph structure, generates multiple fully connected graphs, and performs graph learning to capture the potential associations between different perception data. The incremental graph multi-modal event detection algorithm shows high detection performance in tasks such as ship yaw warning detection, ship bridge crossing warning detection, and ship collision warning detection, especially improving the detection accuracy and effectiveness of the algorithm in complex scenarios.

[0091] Based on the same inventive concept, the embodiments of the present invention also provide an apparatus for perceiving abnormal behaviors of inland river ships based on the incremental graph convolutional network. This apparatus can be applied to the method for perceiving abnormal behaviors of inland river ships based on the incremental graph convolutional network described above. For specific references, please refer to the relevant descriptions above and will not be elaborated here.

[0092] Figure 6 This is a schematic structural diagram of an apparatus for perceiving abnormal behaviors of inland river ships based on the incremental graph convolutional network according to another embodiment of the present invention. As Figure 6As shown in the figure, the inland river ship abnormal behavior perception device based on the incremental graph convolutional network includes: an acquisition module 510, a preprocessing module 520, a first fusion module 530, a second fusion module 540, a first construction module 550, a second construction module 560, a third construction module 570, a matching module 580, and a perception module 590.

[0093] Specifically, the acquisition module 510 is used to acquire the laser point cloud data, video image data, AIS data, and navigable water level data of the inland river ship. The preprocessing module 520 is used to preprocess the acquired laser point cloud data, video image data, AIS data, and navigable water level data. The first fusion module 530 is used to fuse the preprocessed video image data and laser point cloud data based on a two-stage front and back fusion algorithm to obtain a ship image target with three-dimensional depth information. The second fusion module 540 is used to fuse the preprocessed video image data and AIS data based on a multi-feature parameter global optimal association algorithm to obtain a ship target containing ship attribute data. The first construction module 550 is used to construct a semantic clustering incremental graph model S-GCN based on laser point cloud feature data, video image feature data, AIS feature data, and water level feature data. The second construction module 560 is used to construct an incremental graph model M-GCN for multi-modal information association and fusion. The third construction module 570 is used to construct an incremental graph convolutional framework network for multi-modal information fusion abnormal behavior perception based on the incremental graph model. The matching module 580 is used to match the laser point cloud feature data, video image feature data, and AIS feature data with the water level feature data to obtain fused multi-modal feature data. The perception module 590 is used to input the fused multi-modal feature data into the incremental graph convolutional framework network to perform ship abnormal behavior perception detection.

[0094] The inland river ship abnormal behavior perception device based on the incremental graph convolutional network in the embodiment of the present invention is used for ship abnormal behavior event detection. The incremental graph multi-modal event detection algorithm realizes the temporal dynamics and multi-modal association of video images, lidar point clouds, water levels of the waterway, and AIS data by combining S-GCN for semantic information clustering based on ship information and water level information and M-GCN at the multi-modal information fusion level. S-GCN focuses on perceiving the changes in data at the temporal level and captures dynamic features through clustering and mean pooling; M-GCN uses the multi-head attention mechanism to associate the fusion relationships of multi-modal information in the graph structure, generates multiple fully connected graphs and performs graph learning, so as to capture the potential associations between different perception data. The incremental graph multi-modal event detection algorithm shows high detection performance in ship yaw warning detection, ship bridge crossing warning detection, and ship collision warning detection tasks, especially improving the detection accuracy and effectiveness of the algorithm in complex scenarios.

[0095] Another aspect of the embodiments of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it can implement the method for perceiving abnormal behaviors of inland river ships based on the incremental graph convolutional network described above.

[0096] Among them, the computer-readable medium may be included in the devices, equipment, and systems of the present disclosure, or may exist independently.

[0097] Among them, the computer-readable storage medium can be any tangible medium that contains or stores a program. It can be an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment. More specific examples include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, an optical fiber, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0098] Among them, the computer-readable storage medium may also include a data signal propagated in a baseband or as part of a carrier wave, on which computer-readable program code is carried. Specific examples include, but are not limited to, electromagnetic signals, optical signals, or any suitable combination thereof.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for sensing abnormal behavior of inland vessels based on incremental graph convolutional network, characterized in that: include: Obtain laser point cloud data, video image data, AIS data and navigation water level data of inland vessels; Preprocessing the acquired laser point cloud data, video image data, AIS data and navigation water level data; The laser point cloud data is fused with the video data, and based on a two-stage front-to-back fusion algorithm, the pre-processed video image data and the laser point cloud data are fused to obtain a ship image target with three-dimensional depth information; The video data is fused with the AIS data, and based on the global optimal association algorithm of multiple feature parameters, the pre-processed video image data and the AIS data are fused to obtain a ship target containing ship attribute data; Based on laser point cloud feature data, video image feature data, AIS feature data and water level feature data, a semantic clustering incremental graph model S-GCN is constructed; Construct an incremental graph model M-GCN for multimodal information association fusion; Construct an incremental graph convolution framework network for multimodal information fusion and abnormal behavior perception based on incremental graph models; Matching the laser point cloud feature data, the video image feature data and the AIS feature data with the water level feature data to obtain fused multimodal feature data; The fused multimodal feature data is input into the incremental graph convolution framework network to carry out perception and detection of abnormal ship behavior.

2. According to claim 1, the method for sensing abnormal behavior of inland vessels based on incremental graph convolutional network is characterized in that: The two-stage front-to-back fusion algorithm is based on which the pre-processed video image data and the laser point cloud data are fused to obtain a ship image target with three-dimensional depth information, including: The front fusion performs cross-overlap calculation on the target candidate area extracted from the overhead view feature of the laser point cloud data and the feature candidate area extracted from the image, and recommends the first K regions of interest ROIs according to the target frame selection coverage of the intersection part, and sends the fused ROIs to the region of interest alignment layer ROI Align to be integrated into the same dimension, so as to be sent to the subsequent network for fusion analysis; The post-fusion performs target fusion on the three-dimensional bounding box and the two-dimensional target frame extracted from the laser point cloud data and the video image data respectively, transforms the three-dimensional bounding box information into the image coordinate system through time synchronization and space transformation matrix, and obtains the ship image target with three-dimensional depth information.

3. According to claim 1, the method for sensing abnormal behavior of inland vessels based on incremental graph convolutional network is characterized in that: The pre-processed video image data and AIS data are fused based on the multi-feature parameter global optimal association algorithm to obtain a ship target containing ship attribute data, including: Extracting ship features based on the video image data; wherein the ship features include at least one of position, size, speed and grayscale statistics; Extracting ship information based on the AIS data; wherein the ship information includes at least one of a ship name, a position, a size, a speed, and a heading; The ship characteristics and the ship information belonging to the same ship target are associated to obtain a ship target containing ship attribute data.

4. The method for sensing abnormal behavior of inland vessels based on incremental graph convolutional network according to any one of claims 1 to 3, characterized in that: The semantic clustering incremental graph model S-GCN is constructed based on the laser point cloud feature data, the video image feature data, the AIS feature data and the water level feature data, including: Clustering the laser point cloud feature data, the video image feature data, the AIS feature data and the water level feature data, generating a new graph and then performing a graph learning operation to obtain new vertex features that take into account information of the sensor at the semantic level; Mean pooling is used to complete the fusion of feature vertices of the same sensor. The edges between the fused vertices and the new vertices are calculated according to the similarity of the vertex features. The cosine distance is used to determine the weights of the edges in the new graph, and the semantic clustering incremental graph model S-GCN is obtained.

5. The method for sensing abnormal behavior of inland vessels based on incremental graph convolutional network according to any one of claims 1 to 3, characterized in that: The incremental graph model M-GCN for building multimodal information association fusion includes: A multi-head attention mechanism is used to process the graph of time t to find the potential relationship between different modal data. After processing, several new graphs are generated, and the features of the nodes are updated through graph learning. These features are fused together to obtain multiple features with the same number of nodes in the graph, and the incremental graph model M-GCN of multimodal information association fusion is obtained.

6. The method for sensing abnormal behavior of inland vessels based on incremental graph convolutional network according to any one of claims 1 to 3, characterized in that: The incremental graph convolution framework network for multimodal information fusion abnormal behavior perception based on the incremental graph model is constructed, including: The features obtained based on the semantic clustering incremental graph model S-GCN and the features obtained by the incremental graph model M-GCN based on multimodal information association fusion are connected, and then an incremental graph convolution framework network is obtained through a fully connected layer and softmax.

7. The method for sensing abnormal behavior of inland vessels based on incremental graph convolutional network according to any one of claims 1 to 3, characterized in that: The step of matching the laser point cloud feature data, the video image feature data and the AIS feature data with the water level feature data to obtain fused multimodal feature data includes: The laser point cloud feature data, video image feature data and AIS feature data are matched with the water level information according to corresponding timestamps and geographic location information to obtain the fused multimodal feature data.

8. An abnormal behavior perception device for inland vessels based on incremental graph convolutional network, characterized in that: include: Acquisition module, used to acquire laser point cloud data, video image data, AIS data and navigation water level data of inland vessels; A preprocessing module, used for preprocessing the acquired laser point cloud data, video image data, AIS data and navigation water level data; A first fusion module is used to fuse the pre-processed video image data and the laser point cloud data based on a two-stage front-to-back fusion algorithm to obtain a ship image target with three-dimensional depth information; A second fusion module is used to fuse the pre-processed video image data and AIS data based on a multi-feature parameter global optimal association algorithm to obtain a ship target containing ship attribute data; The first construction module is used to construct a semantic clustering incremental graph model S-GCN based on laser point cloud feature data, video image feature data, AIS feature data and water level feature data; The second building module is used to build an incremental graph model M-GCN for multimodal information association fusion; The third building module is used to build an incremental graph convolution framework network for multimodal information fusion abnormal behavior perception based on the incremental graph model; A matching module, used for matching the laser point cloud feature data, the video image feature data and the AIS feature data with the water level feature data to obtain fused multi-modal feature data; The perception module is used to input the fused multimodal feature data into the incremental graph convolution framework network to carry out perception and detection of abnormal ship behavior.

9. An electronic device, characterized in that: include: one or more processors; A storage unit, used to store one or more programs, which, when executed by the one or more processors, enable the one or more processors to implement the method for sensing abnormal behavior of inland vessels based on an incremental graph convolutional network according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it can implement the method for sensing abnormal behavior of inland vessels based on an incremental graph convolutional network according to any one of claims 1 to 7.

Images