Intelligent ship anti-collision decision-making method based on multi-heterogeneous modal information fusion

Through multi-heterogeneous mode information fusion and reinforcement learning technology, an intelligent ship collision prevention decision model is built, which solves the limitations of data processing of a single mode sensor in complex environments, and achieves more accurate and reliable collision avoidance decisions, reducing the Collective risks of maritime traffic.

CN119066612APending Publication Date: 2024-12-03DALIAN MARITIME UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411151775.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-21
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In maritime traffic, the data of a single modal sensor has limitations and instability, which makes it difficult for ship collision avoidance decisions to achieve accurate and reliable information processing in complex and dynamic environments, increasing the risk of collision.

Method used

Using an intelligent ship collision prevention decision-making method based on multi-heterogeneous mode information fusion, by obtaining proprioceptive information, visual observation information and point cloud information, use Transformer encoder and fully connected network for feature extraction and fusion, and build a Marco based on The multi-heterogeneous mode anti-collision decision model of the decision-making process combines reinforcement learning and attention mechanisms to design reward functions to optimize decision-making.

Benefits of technology

It significantly improves the information processing capabilities and decision-making accuracy of smart ships in complex environments, reduces collision risks, ensures the safety and smoothness of maritime traffic, and enhances the adaptability and robustness of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119066612A_ABST
    Figure CN119066612A_ABST
Patent Text Reader

Abstract

The invention discloses an intelligent ship anti-collision decision-making method based on multi-heterogeneous modal information fusion. The method comprises the following steps: acquiring ontology feeling information, visual observation information and point cloud information of an intelligent ship; encoding the ontology feeling information, the visual observation information and the point cloud information so as to extract corresponding modal features, and converting the generated three kinds of heterogeneous modal features into Transform encoded tokens; layer stacking processing is carried out on a Transform encoder, tokens information of the three heterogeneous modes is fused on multiple levels, and after feature vectors of the three heterogeneous modes are spliced, the vectors connected in series are projected to a final output vector through a full-connection network; and inputting the output vector into a trained multi-heterogeneous modal anti-collision decision model to carry out anti-collision decision, wherein the multi-heterogeneous modal anti-collision decision model is constructed based on a Markov decision process. The research result of the invention not only provides an effective technical means for collision avoidance of the intelligent ship, but also lays a solid foundation for the development of an intelligent shipping system in the future.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of autonomous unmanned systems, artificial intelligence, navigation, and more particularly to an intelligent ship collision avoidance decision method based on multi-heterogeneous modal information fusion. Background Art

[0002] With the increase in ship density and the interweaving of routes, maritime traffic has become increasingly complex. This complexity has made maritime traffic face more potential dangerous encounter scenarios, highlighting the importance of ship collision avoidance decisions in ensuring maritime traffic safety. Especially in scenarios where dynamic and static obstacle environments are intertwined, ship collision avoidance decisions are particularly critical. In the current maritime traffic management system, ship collision avoidance decisions rely on a variety of sensors and information sources, including but not limited to AIS (automatic identification system), radar, GPS, and visual sensors. These sensors provide rich environmental perception data, allowing ships to understand the surrounding dynamic and static obstacles in real time. However, single-mode information often has limitations. For example, radar signals may be affected by weather conditions, AIS data may be delayed or erroneous, and visual sensors may fail in low visibility conditions. Therefore, the fusion of multi-heterogeneous modal information has become an important means to improve the effectiveness of ship collision avoidance decisions. Multi-heterogeneous modal information fusion technology can provide more comprehensive and accurate environmental perception by integrating data from different sensors. The fused information can not only make up for the shortcomings of a single sensor, but also improve the reliability and accuracy of perception through information complementarity and redundancy.

[0003] However, multimodal information fusion technology also faces a series of challenges in its application. First, the data formats and characteristics of different sensors are different, and how to effectively standardize and synchronize them is a key issue. Second, multi-source data often contains a large amount of redundant and noisy information. How to filter out irrelevant information and extract useful features is another difficult problem in achieving efficient collision avoidance decisions. Especially in complex and dynamically changing marine environments, fast and accurate information processing is essential for real-time collision avoidance decisions. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides an intelligent ship collision avoidance decision-making method based on multi-heterogeneous modal information fusion, which can not only improve the accuracy and reliability of collision avoidance decisions, but also provide stable performance support under various complex sea conditions, thereby effectively ensuring the safety and smoothness of maritime traffic.

[0005] The technical means adopted by the present invention are as follows:

[0006] An intelligent ship collision avoidance decision method based on multi-heterogeneous modal information fusion includes the following steps:

[0007] S1, obtaining proprioceptive information of the intelligent ship and visual observation information and point cloud information extracted by external sensors;

[0008] S2, respectively encoding the proprioceptive information, visual observation information and point cloud information to extract corresponding modal features, and converting the generated three heterogeneous modal features into Transformer-encoded tokens;

[0009] S3, stacking the Transformer encoder, fusing the token information of the three heterogeneous modalities at multiple levels, concatenating the feature vectors of the three heterogeneous modalities, and using a fully connected network to project the concatenated vectors into the final output vector;

[0010] S4. Inputting the output vector into the trained multi-heterogeneous modal collision avoidance decision model to make a collision avoidance decision. The multi-heterogeneous modal collision avoidance decision model is constructed based on a Markov decision process. The construction process includes:

[0011] (1) Observation space design: The intelligent ship receives proprioceptive information, visual observation information, and point cloud information.

[0012] (2) Action space design: The action output of the intelligent ship in the collision avoidance decision-making environment is heading information and speed information;

[0013] (3) Reward function design: The reward function shown in the following formula is used to complete the intelligent ship collision avoidance decision-making task:

[0014] R = αR forward +βR energy +ηR alive +KR sphere

[0015] Among them, α, β, η are the weights of the corresponding rewards, K is the reward weight under the dynamic collision avoidance decision task, R sphere For smart ship collision penalty, R forward Represents the reward for moving forward, R energy Minimum energy usage for smart ships, R alive For positive rewards.

[0016] Furthermore, the proprioceptive information, visual observation information and point cloud information are encoded respectively to extract corresponding modal features, including:

[0017] A multi-layer perception mechanism is used to encode the features of proprioceptive information;

[0018] ConvNet encoder is used to encode the features of visual observation information;

[0019] PointNet encoder is used to encode the features of point cloud information.

[0020] The three generated heterogeneous modal features are converted into Transformer-encoded tokens, including:

[0021] Furthermore, as the input data of the Transformer model, the three heterogeneous modal features are segmented into tokens;

[0022] These tokens are converted into vector representations through the embedding layer. The conversion process is shown in the following formula:

[0023]

[0024] Among them, t prop Represents proprioceptive characteristics, represents the visual observation feature, Represents point cloud features, W prop ,E prop are the parameter matrices at the spatial position (i, j), b prop are the weights and biases of the linear projection of the proprioceptive tokens embedding, and T0 is the input token sequence.

[0025] Furthermore, the proprioceptive data includes a vector consisting of heading, speed, body position, and historical action sequence; the visual information includes a 3×64×64 shape depth image collected by the smart ship through the camera every second; and the point cloud information includes the distance measurement laser data obtained by the radar in a continuous time.

[0026] Furthermore, the heading information includes left turn, right turn and straight ahead, and the speed information includes acceleration and deceleration.

[0027] Furthermore, the following formula is used to find the minimum energy used by the smart ship:

[0028] R energy =-||τ|| 2

[0029] Among them, τ is the servo torque.

[0030] Compared with the prior art, the present invention has the following advantages:

[0031] (1) The present invention improves the accuracy and reliability of smart ship collision avoidance decisions: By constructing a multi-heterogeneous modal neural network model and introducing the Transformer layer and attention mechanism, the present invention can effectively integrate and analyze data from multiple sensors, selectively focus on relevant information and filter out irrelevant information. This method significantly improves the information processing capability and decision-making accuracy of smart ships in complex dynamic environments, thereby greatly reducing the risk of collision.

[0032] (2) The present invention designs a multi-heterogeneous modal collision avoidance decision model based on reinforcement learning, which not only enhances the adaptability and robustness of the model in different scenarios, but also optimizes the performance of intelligent ships in actual collision avoidance tasks, ensuring that fast and accurate decisions can still be made in complex sea conditions, effectively ensuring the safety and smoothness of maritime traffic. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0034] Figure 1 The present invention is a flow chart of an intelligent ship collision avoidance decision-making method based on multi-heterogeneous modal information fusion.

[0035] Figure 2 This is a network structure diagram for making intelligent ship anti-collision decisions in an embodiment of the present invention.

[0036] Figure 3 The figure is a general framework diagram for making intelligent ship collision avoidance decisions in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0038] like Figure 1 As shown, the present invention provides an intelligent ship collision avoidance decision method based on multi-heterogeneous modal information fusion, comprising the following steps:

[0039] S1. Acquire the proprioceptive information of the intelligent ship and the visual observation information and point cloud information extracted by external sensors.

[0040] Specifically, in the intelligent ship collision avoidance decision-making environment, the intelligent ship uses proprioceptive state and range sensor observation to make decisions. Proprioceptive state and range sensor observation are completely different modalities, where proprioceptive input is a vector, including heading, speed, proprioceptive position, and historical action sequence. Range sensor observation refers to visual observation information and point cloud information. Among them, visual observation information needs to be additionally encoded and processed, which provides semantic information and external environment information; point cloud information is used to obtain more stable and accurate structured information.

[0041] S2. Encode the proprioceptive information, visual observation information and point cloud information respectively to extract corresponding modal features, and convert the generated three heterogeneous modal features into Transformer-encoded tokens.

[0042] Specifically, in order to extract effective information from three heterogeneous modalities and fuse them, the present invention uses three separate encoders to process proprioceptive, visual, and point cloud data, and uniformly aligns and represents them in the latent space. Therefore, the neural network architecture constructed by the present invention includes: encoding the three heterogeneous modalities of proprioceptive state information, visual information, and point cloud information at the input end of the neural network to obtain features. A multi-layer perception mechanism is used to feature encode the proprioceptive input, and a ConvNet encoder is used to feature encode the visual information, and a PointNet encoder is used to feature encode the point cloud information.

[0043] These three heterogeneous modal features are then converted into tokens encoded by the Transformer. Specifically, in the Transformer model, the input data is first segmented into tokens, and then these tokens are converted into vector representations through the embedding layer for subsequent processing and encoding. Therefore, the conversion process is shown in the following formula:

[0044]

[0045] Among them, t prop , They are ontology, vision, and point cloud features, W prop ,E prop are the parameter matrices at the spatial position (i, j), b propare the weights and biases of the linear projection of the proprioceptive tokens embedding, and T0 is the input token sequence, i.e., the transformed Transformer encoded tokens output. Wprop is a parameter matrix associated with the spatial position (i, j), which plays a role in capturing the spatial relationship in the data during the transformation process. Eprop is another parameter matrix that is used together with Wprop to perform a linear projection of the tokens. It is used to encode different modal information (such as proprioceptive, visual, and lidar data) into a unified representation. Although both matrices are used in the projection and transformation process, they play different roles in encoding spatial and modality-specific information.

[0046] The Transformer model is used to share the three tokens information to generate the ontology, vision, and point cloud features. The Transformer encoder layers are stacked to fuse the tokens information of the three heterogeneous modalities at multiple levels. Finally, the feature vectors of the three heterogeneous modalities are concatenated and projected into the final output vector using a fully connected network. The network structure design is as follows: Figure 1 shown.

[0047] S3. Perform layer stacking on the Transformer encoder to fuse the token information of the three heterogeneous modalities at multiple levels, concatenate the feature vectors of the three heterogeneous modalities, and use a fully connected network to project the concatenated vectors into the final output vector.

[0048] Specifically, fusing the token information of these three heterogeneous modalities at multiple levels can be understood as an internal transformation process, where the embedded tokens (i.e., T0) are input into a multi-layer stacked Transformer encoder, each of which includes a multi-head self-attention mechanism and a feedforward neural network. In each layer of the network, the input tokens are first processed by the multi-head self-attention mechanism to obtain a new representation, which is then passed to the feedforward neural network for further transformation. These processes are serial, that is, self-attention calculations are performed first, and then the feedforward network is processed. The output of each layer is passed to the next layer through residual connections and layer normalization. Finally, the fused tokens vectors are concatenated to form a comprehensive feature vector, which is projected into the final output vector using a fully connected network, thereby obtaining a comprehensive feature representation containing rich multimodal information.

[0049] S4. Inputting the output vector into the trained multi-heterogeneous modal collision avoidance decision model to make a collision avoidance decision. The multi-heterogeneous modal collision avoidance decision model is constructed based on a Markov decision process. The construction process includes:

[0050] (1) Observation space design: The smart ship receives proprioceptive state information, visual information, and point cloud information. The proprioceptive data includes vectors composed of heading, speed, body position, and historical action sequence; the visual information includes the 3×64×64 shape depth image collected by the smart ship through the camera every second, which provides spatial and temporal visual information for the smart ship; the point cloud information includes the distance measurement laser data obtained by the radar in a continuous time.

[0051] (2) Action space design: The action output of an intelligent ship in a collision avoidance decision-making environment is information such as heading and speed. The heading includes left turn, right turn and straight ahead; the speed includes acceleration and deceleration.

[0052] (3) Reward function design: Use the reward function of Formula 1 to complete the intelligent ship collision avoidance decision-making task.

[0053] R = αR forward +βR energy +ηR alive +KR sphere (1)

[0054] Among them, α, β, η are the weights of the corresponding rewards, K is the reward weight under the dynamic collision avoidance decision task, R sphere For smart ship collision penalty, R forward Represents the reward for moving forward, R energy Using the minimum energy for smart ships can improve the performance of smart ship motion and increase the energy consumption ratio. Specifically, the Euclidean norm is used to penalize actions that produce high motion torque, as shown in Formula 2.

[0055] R energy =-||τ|| 2 (2)

[0056] Where τ is the servo torque, R alive To encourage smart ships to stay longer, it gives a positive reward of 1.0 at each time step until termination. Dangerous behaviors such as collision will be judged as termination.

[0057] The present invention uses the Markov decision process model to construct a multi-heterogeneous modal collision avoidance decision model based on reinforcement learning, and uses the Actor-Critic architecture and PPO algorithm to train smart ships for collision avoidance decisions. The overall invention patent framework is as follows: Figure 2 shown.

[0058] The present invention constructs a multi-heterogeneous modal neural network model, which enables the intelligent ship to learn from its own sensor data and external sensor data; the introduction of the Transformer layer and the attention mechanism enables the intelligent ship to selectively focus on relevant sensor information and filter irrelevant information. The present invention also designs a multi-heterogeneous modal collision avoidance decision model based on reinforcement learning, which is trained by combining supervised and unsupervised learning techniques, thereby optimizing the performance of intelligent ships in collision avoidance decision tasks.

[0059] In summary, the present invention proposes an intelligent ship collision avoidance decision method based on multi-heterogeneous modal information fusion, which can demonstrate collision avoidance decisions in encounter scenarios of different complexities and improve performance. The research results of the present invention not only provide an effective technical means for intelligent ship collision avoidance, but also lay a solid foundation for the development of future intelligent shipping systems. The combination of multi-heterogeneous modal information fusion and advanced deep learning technology shows great application potential and is expected to play an important role in practical applications and improve the level of intelligence in maritime traffic management.

[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An intelligent ship collision avoidance decision method based on multi-heterogeneous modal information fusion, characterized in that: The following steps are involved: S1, obtaining proprioceptive information of the intelligent ship and visual observation information and point cloud information extracted by external sensors; S2, respectively encoding the proprioceptive information, visual observation information and point cloud information to extract corresponding modal features, and converting the generated three heterogeneous modal features into Transformer-encoded tokens; S3, stacking the Transformer encoder, fusing the token information of the three heterogeneous modalities at multiple levels, concatenating the feature vectors of the three heterogeneous modalities, and using a fully connected network to project the concatenated vectors into the final output vector; S4. Inputting the output vector into the trained multi-heterogeneous modal collision avoidance decision model to make a collision avoidance decision. The multi-heterogeneous modal collision avoidance decision model is constructed based on a Markov decision process. The construction process includes: (1) Observation space design: The intelligent ship receives proprioceptive information, visual observation information, and point cloud information. (2) Action space design: The action output of the intelligent ship in the collision avoidance decision-making environment is heading information and speed information; (3) Reward function design: The reward function shown in the following formula is used to complete the intelligent ship collision avoidance decision-making task: R=αR forward +βR energy +ηR alive +KR sphere Among them, α, β, η are the weights of the corresponding rewards, K is the reward weight under the dynamic collision avoidance decision task, R sphere For smart ship collision penalty, R forward Represents the reward for moving forward, R energy Minimum energy usage for smart ships, R alive For positive rewards.

2. According to claim 1, an intelligent ship collision avoidance decision method based on multi-heterogeneous modal information fusion is characterized in that: The proprioceptive information, visual observation information and point cloud information are respectively encoded to extract corresponding modal features, including: A multi-layer perception mechanism is used to encode the features of proprioceptive information; ConvNet encoder is used to encode the features of visual observation information; PointNet encoder is used to encode the features of point cloud information.

3. According to claim 1, the intelligent ship collision avoidance decision method based on multi-heterogeneous modal information fusion is characterized in that: The three generated heterogeneous modal features are converted into Transformer-encoded tokens, including: As the input data of the Transformer model, the three heterogeneous modal features are segmented into tokens; These tokens are converted into vector representations through the embedding layer. The conversion process is shown in the following formula: t prop =W prop (E prop )+b prop ,t prop ∈R C Among them, t prop Represents proprioceptive characteristics, represents the visual observation feature, Represents point cloud features, W prop ,E prop are the parameter matrices at the spatial position (i, j), W prop Used to capture spatial relationships of data, E prop Used to perform linear projection on tokens, b prop are the weights and biases of the linear projection of the proprioceptive tokens embedding, and T0 is the input token sequence.

4. According to claim 1, the intelligent ship collision avoidance decision method based on multi-heterogeneous modal information fusion is characterized in that: The proprioceptive data includes vectors composed of heading, speed, body position, and historical action sequences; the visual information includes 3×64×64 shape depth images collected by the smart ship through the camera every second; and the point cloud information includes distance measurement laser data obtained by the radar in a continuous time.

5. According to claim 1, the intelligent ship collision avoidance decision method based on multi-heterogeneous modal information fusion is characterized in that: The heading information includes left turn, right turn and straight ahead, and the speed information includes acceleration and deceleration.

6. According to claim 1, the intelligent ship collision avoidance decision method based on multi-heterogeneous modal information fusion is characterized in that: Use the following formula to find the minimum energy used by smart ships: R energy =-||τ|| 2 Among them, τ is the servo torque.