A city inland river cruise ship driving dynamic safety monitoring method and system based on multi-modal sensing

Abstract

The application relates to the technical field of ship safety monitoring, in particular to a city inland river cruise ship driving dynamic safety monitoring method and system based on multi-modal sensing, which comprises the following steps: based on multi-source heterogeneous data collected by a multi-modal sensor network in real time, performing data preprocessing and synchronization, and outputting time-space aligned multi-source heterogeneous data; based on the time-space aligned multi-source heterogeneous data, performing cross-modal attention model feature fusion, and outputting fused multi-dimensional state features; based on the fused multi-dimensional state features, performing dynamic risk assessment, and outputting a final risk level; based on the final risk level and historical risk data, judging a risk type and whether to activate an emergency response; when the emergency response is activated, performing a risk avoidance operation and generating a feedback report. The application has the effect of effectively preventing navigation risks and reducing potential accident losses.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of ship safety monitoring, in particular to a city inland river cruise ship driving dynamic safety monitoring method and system based on multi-modal sensing. BACKGROUND

[0002] Traditional city inland river cruise ship safety monitoring technology mainly relies on single sensor (such as GPS, gyroscope or radar) to collect ship dynamic or environmental data, which has the defects of single data dimension and insufficient scene coverage. For example, the attitude monitoring based on inertial sensor cannot perceive external obstacles, and although radar ranging can detect surrounding targets, it is difficult to identify obstacle types or assess the internal safety state of the ship body (such as fire, water leakage). In addition, the existing system usually adopts independent data processing module, lacks the ability of spatio-temporal alignment and fusion of multi-modal data, resulting in insufficient real-time risk prediction accuracy for complex navigation scenes (such as bad weather, sudden obstacles), and most schemes only provide passive alarm, which cannot be linked with the ship control system to realize active risk avoidance. Although some researches try to combine vision and inertial sensors, their data processing framework is still limited to static models, and the conflict resolution and collaborative decision-making problems of multi-source heterogeneous data in dynamic environment have not been solved, which limits the reliability and practicality of the system.

[0003] Therefore, it is necessary to improve. SUMMARY

[0004] To solve the above technical problems, the present application provides a city inland river cruise ship driving dynamic safety monitoring method and system based on multi-modal sensing.

[0005] The purpose of the present application is achieved by the following technical solutions:

[0006] A city inland river cruise ship driving dynamic safety monitoring method based on multi-modal sensing, comprising the steps of:

[0007] Based on the multi-source heterogeneous data collected by the multi-modal sensor network in real time, data preprocessing and synchronization are performed, and spatio-temporally aligned multi-source heterogeneous data is outputted;

[0008] Based on the spatio-temporally aligned multi-source heterogeneous data, cross-modal attention model feature fusion is performed, and the fused multi-dimensional state features are outputted;

[0009] Based on the fused multi-dimensional state features, dynamic risk assessment is performed, and the final risk level is outputted;

[0010] Based on the final risk level and historical risk data, the risk type is judged, and it is judged whether to activate the emergency response;

[0011] When the emergency response is activated, when the emergency response is activated, the risk avoidance operation is performed, and the feedback report is generated.

[0012] In a preferred embodiment, the step of collecting multi-source heterogeneous data based on the multi-modal sensor network in real time, performing data preprocessing and synchronization, and outputting spatio-temporally aligned multi-source heterogeneous data comprises:

[0013] The multi-source heterogeneous data includes ship dynamic data, environmental perception data, and cabin safety data.

[0014] The ship dynamic data includes ship data and power data.

[0015] The ship data includes ship attitude angle data, speed data, heading data, ship load data, heading deviation angle data, and ship braking distance data.

[0016] The power data includes propeller speed data, temperature data, and torque data.

[0017] The environmental perception data includes obstacle identification data and environmental state data.

[0018] The obstacle identification data includes relative distance data, relative speed data, classification data, relative bearing data, and dynamic trajectory data of the obstacle.

[0019] The environmental state data includes real-time wave height data, wind speed data, and visibility data.

[0020] The cabin safety data includes passenger distribution density data, water leakage area location information data, and smoke concentration data.

[0021] The multi-source heterogeneous data is timestamped and aligned to the same coordinate system.

[0022] The unified multi-source heterogeneous data is cleaned, including removing noise, outliers, and redundant data.

[0023] In a preferred embodiment, the step of performing preset cross-modal attention model feature fusion based on the spatio-temporally aligned multi-source heterogeneous data and outputting the fused multi-dimensional state features comprises:

[0024] Based on the preset cross-modal attention model, the multi-source heterogeneous data is fused to generate a multi-dimensional state vector containing spatio-temporal correlation.

[0025] Based on Kalman filtering, the multi-dimensional state vector is dynamically noise suppressed to output the fused multi-dimensional state features.

[0026] In a preferred embodiment, the step of performing dynamic risk assessment based on the fused multi-dimensional state features and outputting the final risk level comprises:

[0027] The dynamic risk assessment model comprises a short-term risk assessment and a long-term risk assessment.

[0028] The short-term risk assessment: based on a preset coupled fluid dynamics simulation model and the ship attitude angle data, a ship capsizing risk coefficient K is calculated at a preset period;

[0029] Based on the obstacle relative speed data, the heading deviation angle data, and the ship braking distance, a collision probability Bayesian network model is constructed, and a collision probability P is output;

[0030] The long-term risk assessment: based on a vibration spectrum analysis method, a ship structure resonance frequency is extracted, and in combination with historical fatigue data of a preset ship maintenance database, a residual life T of a key part of the ship body is predicted.

[0031] Based on a time window integration method, the heading deviation angle cumulative value Z in a preset time period is calculated.

[0032] In a preferred embodiment, the step of performing dynamic risk assessment based on the fused multi-dimensional state features and outputting a final risk level further comprises:

[0033] When K≥ε1, P<δ1, ε1 is a preset first capsizing threshold, and δ1 is a preset first collision threshold, the final risk level is a short-term low risk level.

[0034] When ε2≤K<ε1, δ1≤P<δ2, ε2 is a preset second capsizing threshold, and δ2 is a preset second collision threshold, the final risk level is a short-term medium risk level.

[0035] When K<ε2, P≥δ2, it is a short-term high risk level.

[0036] When T≥0.8·μ, μ is a preset life threshold, is a preset heading deviation threshold, the final risk level is a long-term low risk level.

[0037] When 0.5·μ≤T<0.8·μ, the final risk level is a long-term medium risk level.

[0038] When T<0.5·μ, the final risk level is a long-term high risk level.

[0039] When the short-term risk level and the long-term risk level are inconsistent, the higher risk level of the two is taken as the final risk level.

[0040] When the short-term risk level and the long-term risk level are consistent, the risk level is directly taken as the final risk level.

[0041] In a preferred embodiment, the step of determining the risk type and determining whether to activate the emergency response based on the final risk level and historical risk data comprises:

[0042] integrating the final risk level and historical risk event data to generate a complete data set;

[0043] extracting key features from the integrated complete data set, the key features including a risk level value R and a corresponding change rate V, a type T of historical risk event h and a corresponding frequency F;

[0044] comparing the key features with preset threshold values based on a preset risk classification model to determine the risk type T f ;

[0045] when R≥R1, V≥V1, T h = capsizing, and F≥F1, T f is a capsizing risk;

[0046] when R≥R2, V≥V2, T h = collision, and F≥F2, T f is a collision risk;

[0047] when R≥R3, V≥V3, T h = structural fatigue, and F≥F3, T f is a structural fatigue risk;

[0048] wherein R1 is a preset first risk level threshold value, R2 is a preset second risk level threshold value, and R3 is a preset third risk level threshold value;

[0049] wherein V1 is a preset first risk level change rate threshold value, V2 is a preset second risk level change rate threshold value, and V3 is a preset third risk level change rate threshold value;

[0050] wherein F1 is a preset first historical risk event frequency threshold value, F2 is a preset second historical risk event frequency threshold value, and F3 is a preset third historical risk event frequency threshold value.

[0051] In a preferred embodiment, the step of determining the risk type and determining whether to activate the emergency response based on the final risk level and historical risk data further comprises:

[0052] the emergency response includes a capsizing emergency response, a collision emergency response, and a structural fatigue emergency response;

[0053] when R>R'1 and T f is a capsizing risk, the capsizing emergency response is activated;

[0054] when R > R'2, T f activate a collision emergency response when there is a collision risk;

[0055] when R > R'3, T f activate a structural fatigue emergency response when there is a structural fatigue risk;

[0056] wherein R'1 is a preset first activation threshold, R'2 is a preset second activation threshold, and R'3 is a preset third activation threshold.

[0057] In a preferred embodiment, when the emergency response is activated, the steps of performing risk avoidance operations and generating a feedback report include:

[0058] generating corresponding risk warning information based on the final risk level and the risk type;

[0059] The risk warning information includes risk description, recommended risk avoidance measures, and emergency contact information.

[0060] Based on the risk type, match the corresponding risk avoidance operation instruction from the preset risk avoidance operation library;

[0061] Perform risk avoidance operations based on the risk warning information and the risk avoidance operation instruction;

[0062] Real-time monitoring of the execution effect of the risk avoidance operations, including changes in the state of the ship and changes in the risk level;

[0063] Generate a feedback report including risk warning information and the execution effect of the risk avoidance operations.

[0064] The second object of the application is achieved through the following technical solutions:

[0065] A city inland river cruise ship driving dynamic safety monitoring system based on multi-modal sensing, comprising:

[0066] A first output module: based on multi-source heterogeneous data collected in real time by a multi-modal sensor network, perform data preprocessing and synchronization, and output spatio-temporally aligned multi-source heterogeneous data;

[0067] A second output module: based on spatio-temporally aligned multi-source heterogeneous data, perform cross-modal attention model feature fusion, and output fused multi-dimensional state features;

[0068] A third output module: based on the fused multi-dimensional state features, perform dynamic risk assessment, and output a final risk level;

[0069] A judgment module: based on the final risk level and historical risk data, determine the risk type and whether to activate an emergency response;

[0070] Generating module: when the emergency response is activated, performing the risk-avoiding operation, generating the feedback report.

[0071] The third application object of the present application is achieved by the following technical solution:

[0072] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the above-mentioned city inland river cruise ship driving dynamic safety monitoring method based on multi-modal sensing.

[0073] The fourth application object of the present application is achieved by the following technical solution:

[0074] A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the above-mentioned city inland river cruise ship driving dynamic safety monitoring method based on multi-modal sensing.

[0075] In summary, the present application includes at least one of the following beneficial technical effects:

[0076] Through the multi-modal sensor network, the multi-source heterogeneous data of the city inland river cruise ship is collected in real time, and after data preprocessing and synchronization (S10), the spatio-temporal alignment of the data is ensured. Then, the cross-modal attention model is used to fuse the features of the spatio-temporal aligned data (S20), effectively integrating the information of different sensors and generating multi-dimensional state features that comprehensively reflect the state of the cruise ship. Based on these features, the system performs dynamic risk assessment (S30) and outputs the final risk level. Combined with the final risk level and historical risk data, the system determines the risk type and decides whether to activate the emergency response (S40). Once the emergency response is activated, the system immediately performs the risk-avoiding operation and generates the feedback report (S50), which records the risk warning information, the risk-avoiding measures and the execution effect in detail. This method realizes real-time and accurate monitoring of the dynamic safety of city inland river cruise ships, effectively improving the safety of cruise ship driving and the efficiency of emergency response. BRIEF DESCRIPTION OF DRAWINGS

[0077] Figure 1 is an implementation flowchart of an embodiment of the city inland river cruise ship driving dynamic safety monitoring method based on multi-modal sensing of the present application;

[0078] Figure 2 is an implementation flowchart of step S20 in an embodiment of the city inland river cruise ship driving dynamic safety monitoring method based on multi-modal sensing of the present application;

[0079] Figure 3is an implementation flowchart of step S30 in an embodiment of a city inland river cruise ship driving dynamic safety monitoring method based on multi-modal sensing of the present application.

[0080] Figure 4 is another implementation flowchart of step S30 in an embodiment of a city inland river cruise ship driving dynamic safety monitoring method based on multi-modal sensing of the present application.

[0081] Figure 5 is an implementation flowchart of step S40 in an embodiment of a city inland river cruise ship driving dynamic safety monitoring method based on multi-modal sensing of the present application.

[0082] Figure 6 is a principle block diagram of a computer device of the present application. DETAILED DESCRIPTION

[0083] The following will be described in combination with the accompanying drawings. Figures 1-6 The present application will be further described in detail.

[0084] In an embodiment, as shown in the accompanying drawings, Figure 1 The present application discloses a city inland river cruise ship driving dynamic safety monitoring method based on multi-modal sensing, which specifically comprises the following steps:

[0085] S10: Based on the multi-source heterogeneous data collected by the multi-modal sensor network in real time, data preprocessing and synchronization are performed, and the spatio-temporally aligned multi-source heterogeneous data is outputted;

[0086] S20: Based on the spatio-temporally aligned multi-source heterogeneous data, cross-modal attention model feature fusion is performed, and the fused multi-dimensional state features are outputted;

[0087] S30: Based on the fused multi-dimensional state features, dynamic risk assessment is performed, and the final risk level is outputted;

[0088] S40: Based on the final risk level and the historical risk data, the risk type is judged, and it is judged whether to activate the emergency response;

[0089] S50: When the emergency response is activated, when the emergency response is activated, the risk avoidance operation is performed, and the feedback report is generated.

[0090] In this embodiment, multi-source heterogeneous data of city inland cruise ships is collected in real time by a multi-modal sensor network, and data preprocessing and synchronization (S10) is performed to ensure the spatio-temporal alignment of the data. Then, a cross-modal attention model is used to fuse the features of the spatio-temporal aligned data (S20), effectively integrating the information from different sensors and generating multi-dimensional state features that comprehensively reflect the state of the cruise ship. Based on these features, the system performs dynamic risk assessment (S30) and outputs the final risk level. Combined with the final risk level and historical risk data, the system determines the risk type and decides whether to activate the emergency response (S40). Once the emergency response is activated, the system immediately performs risk avoidance operations and generates a feedback report (S50), detailing the risk warning information, risk avoidance measures, and execution effect. This method realizes real-time and accurate monitoring of the dynamic safety of city inland cruise ships, effectively improving the safety of cruise ship travel and the efficiency of emergency response.

[0091] The S10 step includes:

[0092] S101: The multi-source heterogeneous data includes ship dynamic data, environmental perception data, and cabin safety data.

[0093] S102: The ship dynamic data includes ship data and power data.

[0094] S103: The ship data includes ship attitude angle data, speed data, heading data, ship load data, heading deviation angle data, and ship braking distance data.

[0095] S104: The power data includes propeller speed data, temperature data, and torque data.

[0096] S105: The environmental perception data includes obstacle recognition data and environmental state data.

[0097] S106: The obstacle recognition data includes obstacle relative distance data, obstacle relative speed data, obstacle classification data, obstacle relative bearing data, and obstacle dynamic trajectory data.

[0098] S107: The environmental state data includes real-time wave height data, wind speed data, and visibility data.

[0099] S108: The cabin safety data includes passenger distribution density data, water leakage area location information data, and smoke concentration data.

[0100] S109: Time stamp alignment is performed on the multi-source heterogeneous data, and the data is unified in the same coordinate system.

[0101] S110: Data cleaning is performed on the unified multi-source heterogeneous data, including removing noise, outliers, and redundant data.

[0102] In this embodiment, the S10 step comprehensively collects the ship dynamic data, environmental perception data and cabin safety data of the cruise ship through the multi-modal sensor network, ensuring the comprehensiveness and real-time of the monitoring. Among them, the ship dynamic data is subdivided into ship data (such as attitude angle, speed, heading, etc.) and power data (such as propeller speed, temperature, torque, etc.), the environmental perception data includes obstacle identification data (such as relative distance, speed, classification, etc.) and environmental state data (such as wave height, wind speed, visibility, etc.), and the cabin safety data focuses on key safety indicators such as passenger distribution, water leakage position and smoke concentration. Through time stamp alignment and coordinate unification, the consistency and accuracy of the data are ensured. Further data cleaning steps effectively remove noise, outliers and redundant data, providing a high-quality data foundation for subsequent feature fusion and risk assessment. This series of data processing measures ensures the accuracy and reliability of the monitoring system, providing a strong guarantee for the driving safety of urban inland river cruise ships.

[0103] Figure 2 The S20 step includes:

[0104] S201: Based on the preset cross-modal attention model, the multi-source heterogeneous data is fused to generate a multi-dimensional state vector containing spatio-temporal correlation;

[0105] S202: Based on Kalman filtering, the multi-dimensional state vector is dynamically noise suppressed to output the fused multi-dimensional state feature.

[0106] In this embodiment, an advanced cross-modal attention model is used to deeply fuse the preprocessed multi-source heterogeneous data. This model can effectively capture the spatio-temporal correlation between different data sources and generate a multi-dimensional state vector containing rich information. In order to further improve the data quality, the system further applies Kalman filtering technology to dynamically suppress the noise of the multi-dimensional state vector, effectively filtering out random interference and uncertainty in the data, and finally outputting more accurate and reliable fused multi-dimensional state features. This step not only improves the accuracy and efficiency of data processing, but also provides a solid data foundation for subsequent dynamic risk assessment, ensuring the accuracy and effectiveness of the dynamic safety monitoring of urban inland river cruise ships.

[0107] Figure 3 The S30 step includes:

[0108] S301: The dynamic risk assessment model includes short-term risk assessment and long-term risk assessment;

[0109] S302: The short-term risk assessment: based on the preset coupled fluid mechanics simulation model and the ship attitude angle data, the ship capsizing risk coefficient K is calculated at a preset period;

[0110] S303: constructing a collision probability Bayesian network model based on the obstacle relative speed data, the heading deviation angle data, and the ship braking distance, and outputting a collision probability P;

[0111] S304: long-term risk assessment: extracting a ship body structure resonance frequency based on a vibration spectrum analysis method, combining historical fatigue data of a preset ship maintenance database, and predicting a residual life T of a key part of the ship body;

[0112] S305: calculating a heading deviation angle cumulative value Z in a preset time period based on a time window integration method.

[0113] In the embodiment, a comprehensive dynamic risk assessment model is adopted, which is divided into short-term and long-term risk assessment parts. The short-term risk assessment calculates a capsizing risk coefficient K of the ship regularly through a preset coupled fluid mechanics simulation model and ship attitude angle data, and simultaneously constructs a collision probability Bayesian network model based on obstacle relative speed data, heading deviation angle data, and ship braking distance to output a collision probability P in real time, so as to realize accurate assessment of the short-term risk. The long-term risk assessment extracts a resonance frequency of the ship body structure by using a vibration spectrum analysis method, and predicts a residual life T of a key part of the ship body in combination with historical fatigue data, and simultaneously calculates a cumulative value Z of the heading deviation angle by using a time window integration method, to provide prediction and early warning for long-term safety. The hierarchical and multi-angle risk assessment method not only improves the comprehensiveness and accuracy of risk identification, but also enhances the safety and predictability of the cruise ship, and provides scientific and effective decision support for dynamic safety monitoring of the urban river cruise ship.

[0114] Figure 4 The S30 step further comprises:

[0115] SB1: when K≥ε1, P<δ1, ε1 is a preset first capsizing threshold value, and δ1 is a preset first collision threshold value, the final risk level is a short-term low risk level;

[0116] SB2: when ε2≤K<ε1, δ1≤P<δ2, ε2 is a preset second capsizing threshold value, and δ2 is a preset second collision threshold value, the final risk level is a short-term medium risk level;

[0117] SB3: when K<ε2, P≥δ2, it is a short-term high risk level;

[0118] SB4: when T≥0.8·μ, μ is a preset life threshold value, is a preset heading deviation threshold value, the final risk level is a long-term low risk level;

[0119] SB5: When 0.5 μ ≤ T < 0.8 μ, the final risk level is medium-term risk level;

[0120] SB6: When T < 0.5 μ, the final risk level is long-term high risk level;

[0121] SB7: When the short-term risk level and the long-term risk level are inconsistent, take the higher risk level of the two as the final risk level;

[0122] SB8: When the short-term risk level and the long-term risk level are consistent, directly adopt the risk level as the final risk level.

[0123] In this embodiment, the logic of dynamic risk assessment is further refined by setting multiple threshold values to distinguish different levels of risk. In the short-term risk assessment, according to the comparison of the capsizing risk coefficient K and the collision probability P with the preset first capsizing threshold ε1, the preset second capsizing threshold ε2, the preset first collision threshold δ1, and the preset second collision threshold δ2, the risk is divided into three levels of low, medium, and high (SB1, SB2, SB3). The long-term risk assessment is based on the comparison of the ship's remaining life T and the heading deviation angle cumulative value Z with the preset life threshold μ and the preset heading deviation threshold , and the risk is also divided into three levels of low, medium, and high (SB4, SB5, SB6). In addition, this embodiment also considers the case that the short-term and long-term risk levels may not be consistent, by taking the higher risk level of the two as the final risk level (SB7), ensuring the conservatism and safety of risk assessment. When the short-term and long-term risk levels are consistent, directly adopt the risk level as the final risk level (SB8), simplifying the decision-making process. This hierarchical and graded risk assessment mechanism improves the fineness and accuracy of risk judgment, and provides a more scientific and reasonable risk warning and decision-making basis for the driving safety of urban inland river cruise ships.

[0124] Figure 5 , S40 step, comprising:

[0125] S401: integrate the final risk level and the historical risk event data to generate a complete data set;

[0126] S402: extract key features from the integrated complete data set, the key features including risk level value R and corresponding change rate V, type T h of historical risk event and corresponding frequency F;

[0127] S403: based on the preset risk classification model, compare the key features with the preset threshold to determine the risk type T f ;

[0128] S404: When R≥R1, V≥V1, T h = capsizing, F≥F1, T f is capsizing risk;

[0129] S405: When R≥R2, V≥V2, T h = collision, F≥F2, T f is collision risk;

[0130] S406: When R≥R3, V≥V3, T h = structural fatigue, F≥F3, T f is structural fatigue risk;

[0131] S407: Wherein R1 is a preset first risk level threshold, R2 is a preset second risk level threshold, and R3 is a preset third risk level threshold.

[0132] S408: Wherein V1 is a preset first risk level change rate threshold, V2 is a preset second risk level change rate threshold, and V3 is a preset third risk level change rate threshold.

[0133] S409: Wherein F1 is a preset first historical risk event frequency threshold, F2 is a preset second historical risk event frequency threshold, and F3 is a preset third historical risk event frequency threshold.

[0134] In this embodiment, by integrating the final risk level and historical risk event data, a complete data set is generated, providing a comprehensive information basis for accurate risk type judgment. Then, key features are extracted from the integrated data set, including risk level value R and its change rate V, type T h of historical risk event and its frequency F, which can effectively reflect the dynamic changes and historical trends of risks. Based on the preset risk classification model, the extracted key features are compared with the preset multi-level thresholds, so as to judge the specific risk type T f , such as capsizing risk, collision risk and structural fatigue risk. By setting different levels of risk level threshold, risk level change rate threshold and historical risk event frequency threshold (such as preset first, second and third thresholds), this embodiment realizes fine division and accurate identification of risk type. This risk type judgment mechanism not only improves the accuracy and pertinence of risk identification, but also provides scientific and reasonable basis for further emergency response and risk avoidance operation, significantly enhancing the effectiveness and practicality of the city inland river cruise ship driving dynamic safety monitoring system.

[0135] S40 step, further comprising:

[0136] SE1: the emergency response comprises a capsizing emergency response, a collision emergency response, and a structure fatigue emergency response;

[0137] SE2: when R > R'1, T f is a capsizing risk, activating the capsizing emergency response;

[0138] SE3: when R > R'2, T f is a collision risk, activating the collision emergency response;

[0139] SE4: when R > R'3, T f is a structure fatigue risk, activating the structure fatigue emergency response;

[0140] SE5: wherein R'1 is a preset first activation threshold, R'2 is a preset second activation threshold, and R'3 is a preset third activation threshold.

[0141] In this embodiment, the emergency response mechanism is further refined, and corresponding emergency response strategies are set for different types of risks. Through the preset activation thresholds, the system can automatically activate the corresponding emergency response according to the judged risk type, including the capsizing emergency response, the collision emergency response, and the structure fatigue emergency response. Specifically, when the risk level reaches the preset capsizing risk activation threshold, the system will activate the capsizing emergency response (SE2); similarly, when the risk level reaches the preset collision risk activation threshold, the system will activate the collision emergency response (SE3); and when the risk level reaches the preset structure fatigue risk activation threshold, the system will activate the structure fatigue emergency response (SE4). This type-based and level-based emergency response mechanism ensures that appropriate risk avoidance measures can be taken quickly and accurately when facing different risks, effectively reducing the likelihood of accidents and losses. Through the preset first, second, and third activation thresholds (SE5), this embodiment realizes the flexibility and pertinence of emergency response, further improving the emergency handling capability and overall safety of the urban inland river cruise ship driving dynamic safety monitoring system.

[0142] S50 step, comprising:

[0143] S501: generating corresponding risk warning information based on the final risk level and the risk type;

[0144] S502: the risk warning information comprises risk description, recommended risk avoidance measures, and emergency contact information;

[0145] S503: based on the risk type, matching the corresponding risk avoidance operation instruction from the preset risk avoidance operation library;

[0146] S504: based on the risk warning information and the risk avoidance operation instruction, performing risk avoidance operation;

[0147] S505: Real-time monitoring of the execution effect of the risk avoidance operation, including ship state changes and risk level changes;

[0148] S506: Generating a feedback report including risk warning information and execution effect of the risk avoidance operation.

[0149] In this embodiment, a complete closed-loop management from risk warning to risk avoidance operation execution and effect monitoring is achieved. First, the system generates detailed risk warning information based on the final risk level and risk type, including risk description, recommended risk avoidance measures and emergency contact information, providing comprehensive risk awareness and response guidance for relevant personnel (S501, S502). Then, according to the risk type, the corresponding risk avoidance operation instruction is matched from the preset risk avoidance operation library, ensuring the pertinence and effectiveness of the risk avoidance operation (S503). Subsequently, the system executes the risk avoidance operation and monitors the execution effect of the risk avoidance operation in real time, including ship state changes and risk level changes, so as to timely evaluate the effect of the risk avoidance measures and make necessary adjustments (S504, S505). Finally, a feedback report is generated, summarizing the risk warning information and the execution effect of the risk avoidance operation, providing important reference for subsequent risk management and decision-making (S506). This series of steps constitutes a dynamic and efficient safety monitoring and emergency response system, significantly improving the response capability and safety of urban river cruise ships in the face of potential risks, ensuring the smooth operation of the cruise ships and the safety of passengers.

[0150] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0151] In an embodiment, a dynamic safety monitoring system for urban river cruise ship travel based on multi-modal sensing is provided, which corresponds to the dynamic safety monitoring method for urban river cruise ship travel based on multi-modal sensing in the above embodiment. The dynamic safety monitoring system for urban river cruise ship travel based on multi-modal sensing comprises:

[0152] The first output module: based on the multi-source heterogeneous data collected by the multi-modal sensor network in real time, performing data preprocessing and synchronization, outputting the spatio-temporally aligned multi-source heterogeneous data;

[0153] The second output module: based on the spatio-temporally aligned multi-source heterogeneous data, performing cross-modal attention model feature fusion, outputting the fused multi-dimensional state features;

[0154] The third output module: based on the fused multi-dimensional state features, performing dynamic risk assessment, outputting the final risk level;

[0155] determining module: determining the risk type based on the final risk level and historical risk data, and determining whether to activate the emergency response;

[0156] generating module: performing a hedging operation and generating a feedback report when the emergency response is activated.

[0157] Optionally, it further comprises:

[0158] The first comprising module: the multi-source heterogeneous data comprises ship dynamic data, environmental perception data, and cabin safety data.

[0159] The second comprising module: the ship dynamic data comprises ship data and power data.

[0160] The third comprising module: the ship data comprises ship attitude angle data, speed data, heading data, ship load data, heading deviation angle data, and ship braking distance data.

[0161] The fourth comprising module: the power data comprises propeller speed data, temperature data, and torque data.

[0162] The fifth comprising module: the environmental perception data comprises obstacle recognition data and environmental state data.

[0163] The sixth comprising module: the obstacle recognition data comprises obstacle relative distance data, obstacle relative speed data, obstacle classification data, obstacle relative bearing data, and obstacle dynamic trajectory data.

[0164] The seventh comprising module: the environmental state data comprises real-time wave height data, wind speed data, and visibility data.

[0165] The eighth comprising module: the cabin safety data comprises passenger distribution density data, water leakage area location information data, and smoke concentration data.

[0166] The unifying module: time stamp alignment is performed on the multi-source heterogeneous data, and the multi-source heterogeneous data is unified into the same coordinate system.

[0167] The cleaning module: data cleaning is performed on the unified multi-source heterogeneous data, and the data cleaning comprises removing noise, outliers, and redundant data.

[0168] Optionally, it further comprises:

[0169] The first generating module: based on a preset cross-modal attention model, the multi-source heterogeneous data is subjected to feature fusion to generate a multi-dimensional state vector containing spatio-temporal correlation.

[0170] The fourth output module: based on Kalman filtering, the multi-dimensional state vector is dynamically noise suppressed, and a fused multi-dimensional state feature is output.

[0171] Optionally, further comprising:

[0172] The first module: the dynamic risk assessment model includes short-term risk assessment and long-term risk assessment;

[0173] The first calculation module: the short-term risk assessment: based on a preset coupled fluid dynamics simulation model and the ship attitude angle data, a ship capsizing risk coefficient K is calculated at a preset period;

[0174] The fifth output module: based on the obstacle relative speed data, the heading deviation angle data, and the ship braking distance, a collision probability Bayesian network model is constructed, and a collision probability P is output;

[0175] The prediction module: the long-term risk assessment: based on the vibration spectrum analysis method, the ship structure resonance frequency is extracted, and the historical fatigue data of the preset ship maintenance database are combined to predict the residual life T of the key parts of the ship body;

[0176] The second calculation module: based on the time window integration method, the heading deviation angle cumulative value Z in a preset time period is calculated.

[0177] Optionally, further comprising:

[0178] The first judgment module: when K≥ε1, P<δ1, ε1 is a preset first capsizing threshold, and δ1 is a preset first collision threshold, the final risk level is a short-term low risk level;

[0179] The second judgment module: when ε2≤K<ε1, δ1≤P<δ2, ε2 is a preset second capsizing threshold, and δ2 is a preset second collision threshold, the final risk level is a short-term medium risk level;

[0180] The third judgment module: when K<ε2, P≥δ2, it is a short-term high risk level;

[0181] The fourth judgment module: when T≥0.8·μ, μ is a preset life threshold, is a preset heading deviation threshold, the final risk level is a long-term low risk level;

[0182] The fifth judgment module: when 0.5·μ≤T<0.8·μ, the final risk level is a long-term medium risk level;

[0183] The sixth judgment module: when T<0.5·μ, the final risk level is a long-term high risk level;

[0184] The seventh judging module: when the short-term risk level and the long-term risk level are inconsistent, taking the higher risk level of the two as the final risk level;

[0185] The eighth judging module: when the short-term risk level and the long-term risk level are consistent, directly using the risk level as the final risk level.

[0186] Optionally, further comprising:

[0187] The integration module: integrating the final risk level and the historical risk event data to generate a complete data set;

[0188] The extraction module: extracting key features from the integrated complete data set, the key features including a risk level value R and a corresponding change rate V, a type T of historical risk event and a corresponding frequency F; h

[0189] The comparison module: based on a preset risk classification model, comparing the key features with preset threshold values to determine the risk type T; f

[0190] The second module: when R≥R1, V≥V1, T h = capsizing, and F≥F1, T f is a capsizing risk;

[0191] The third module: when R≥R2, V≥V2, T h = collision, and F≥F2, T f is a collision risk;

[0192] The fourth module: when R≥R3, V≥V3, T h = structural fatigue, and F≥F3, T f is a structural fatigue risk;

[0193] The fifth module: wherein R1 is a preset first risk level threshold, R2 is a preset second risk level threshold, and R3 is a preset third risk level threshold;

[0194] The sixth module: wherein V1 is a preset first risk level change rate threshold, V2 is a preset second risk level change rate threshold, and V3 is a preset third risk level change rate threshold;

[0195] The seventh module: wherein F1 is a preset first historical risk event frequency threshold, F2 is a preset second historical risk event frequency threshold, and F3 is a preset third historical risk event frequency threshold.

[0196] Optionally, further comprising:

[0197] ​​The eighth module: the emergency response includes a capsizing emergency response, a collision emergency response, and a structure fatigue emergency response.

[0198] The first activation module: when R>R'1 and T f The capsizing emergency response is activated when the capsizing risk occurs.

[0199] The second activation module: when R>R'2 and T f The collision emergency response is activated when the collision risk occurs.

[0200] The third activation module: when R>R'3 and T f The structure fatigue emergency response is activated when the structure fatigue risk occurs.

[0201] The ninth module: wherein R'1 is a preset first activation threshold, R'2 is a preset second activation threshold, and R'3 is a preset third activation threshold.

[0202] Optionally, the system further comprises:

[0203] The early warning information module: based on the final risk level and the risk type, corresponding risk early warning information is generated.

[0204] The risk module: the risk early warning information includes risk description, recommended risk avoidance measures, and emergency contact information.

[0205] The matching module: based on the risk type, corresponding risk avoidance operation instructions are matched from a preset risk avoidance operation library.

[0206] The execution module: based on the risk early warning information and the risk avoidance operation instructions, risk avoidance operations are executed.

[0207] The monitoring module: the execution effect of the risk avoidance operations is monitored in real time, and the execution effect includes ship state changes and risk level changes.

[0208] The feedback report module: a feedback report is generated, and the feedback report includes the risk early warning information and the execution effect of the risk avoidance operations.

[0209] The specific limitations of the city river cruise ship driving dynamic safety monitoring system based on multi-modal sensing can be referred to the limitations of the city river cruise ship driving dynamic safety monitoring method based on multi-modal sensing in the above, which will not be repeated here. Each module in the city river cruise ship driving dynamic safety monitoring system based on multi-modal sensing can be realized by software, hardware, and their combinations. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0210] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides the environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores ship dynamic data. The network interface communicates with external terminals via a network. When executed by the processor, the computer program implements a method for dynamic safety monitoring of urban inland waterway cruise ships based on multimodal sensing.

[0211] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a method for dynamic safety monitoring of urban inland waterway cruise ships based on multimodal sensing.

[0212] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, is a method for dynamic safety monitoring of urban inland waterway cruise vessels based on multimodal sensing.

[0213] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0214] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

Claims

1. A multi-modal sensor-based urban inland river cruise ship running dynamic safety monitoring method, characterized in that, The method comprises the steps of: Based on the multi-source heterogeneous data collected by the multi-modal sensor network in real time, data preprocessing and synchronization are performed, and multi-source heterogeneous data aligned in time and space are outputted; The multi-source heterogeneous data includes ship dynamic data, environmental perception data, and cabin safety data; The ship dynamic data includes ship data and power data; The ship data includes ship attitude angle data, speed data, heading data, ship load data, heading deviation angle data, and ship braking distance data; The power data includes propeller speed data, temperature data, and torque data; The environmental perception data includes obstacle identification data and environmental state data; The obstacle identification data includes obstacle relative distance data, obstacle relative speed data, obstacle classification data, obstacle relative bearing data, and obstacle dynamic trajectory data; The environmental state data includes real-time wave height data, wind speed data, and visibility data; The cabin safety data includes passenger distribution density data, water leakage area location information data, and smoke concentration data; The multi-source heterogeneous data is timestamped and unified in the same coordinate system; The unified multi-source heterogeneous data is cleaned, including removing noise, outliers, and redundant data; Based on the multi-source heterogeneous data aligned in time and space, cross-modal attention model feature fusion is performed, and the fused multi-dimensional state features are outputted; Based on the preset cross-modal attention model, the multi-source heterogeneous data is fused to generate a multi-dimensional state vector containing time and space correlation; Based on Kalman filtering, the multi-dimensional state vector is dynamically noise suppressed, and the fused multi-dimensional state features are outputted; Based on the fused multi-dimensional state features, dynamic risk assessment is performed, and the final risk level is outputted; The dynamic risk assessment model includes short-term risk assessment and long-term risk assessment; The short-term risk assessment: based on the preset coupled fluid mechanics simulation model and the ship attitude angle data, the ship capsizing risk coefficient K is calculated at a preset period; Based on the obstacle relative speed data, the heading deviation angle data, and the ship braking distance, a collision probability Bayesian network model is constructed, and the collision probability P is outputted; The long-term risk assessment: based on the vibration spectrum analysis method, the ship structure resonance frequency is extracted, and combined with the historical fatigue data of the preset ship maintenance database, the residual life T of the key parts of the ship body is predicted; Based on the time window integration method, the heading deviation angle cumulative value Z in the preset time period is calculated; Based on the final risk level and the historical risk data, the risk type is judged, and it is judged whether to activate the emergency response; When the emergency response is activated, the risk avoidance operation is performed, and the feedback report is generated.

2. The urban inland river cruise ship running dynamic safety monitoring method based on multi-modal sensing according to claim 1, characterized in that, The step of performing dynamic risk assessment based on the fused multi-dimensional state features and outputting the final risk level further comprises: When K≥ε1, P<δ1, ε1 is a preset first capsizing threshold, and δ1 is a preset first collision threshold, the final risk level is a short-term low risk level; When ε2≤K<ε1, δ1≤P<δ2, ε2 is a preset second capsizing threshold, and δ2 is a preset second collision threshold, the final risk level is a short-term medium risk level. When K < ε2, P ≥ δ2, it is a short-term high-risk level; when T ≥ 0.8 μ, μ is a preset lifetime threshold, is a preset heading deviation threshold, the final risk level is a long-term low risk level. When 0.5 μ < T < 0.8 μ, the final risk level is a long-term medium risk level; when T < 0.5 · μ, the final risk level is a long-term high risk level; When the short-term risk level and the long-term risk level are inconsistent, the higher risk level of the two is taken as the final risk level; When the short-term risk level and the long-term risk level are consistent, the risk level is directly taken as the final risk level.

3. The method according to claim 1, wherein, The step of judging the risk type and judging whether to activate the emergency response based on the final risk level and the historical risk data comprises: integrating the final risk level and the historical risk event data to generate a complete data set; extracting key features from the integrated complete dataset, said key features including a risk level value R and a corresponding rate of change V, a type T of historical risk event h and a corresponding frequency F; Based on the preset risk classification model, the key features are compared with the preset threshold to determine the risk type T f ; When R ≥ R1, V ≥ V1, T h = capsizing, F ≥ F1, T f is the capsizing risk; When R ≥ R2, V ≥ V2, T h = collision, F ≥ F2, T f is the risk of collision. When R ≥ R3, V ≥ V3, T h = structural fatigue, F ≥ F3, T f is the risk of structural fatigue. Wherein, R1 is a preset first risk level threshold, R2 is a preset second risk level threshold, and R3 is a preset third risk level threshold. Wherein, V1 is a preset first risk level change rate threshold, V2 is a preset second risk level change rate threshold, and V3 is a preset third risk level change rate threshold. Wherein, F1 is a preset first historical risk event frequency threshold, F2 is a preset second historical risk event frequency threshold, and F3 is a preset third historical risk event frequency threshold.

4. The urban inland river cruise ship running dynamic safety monitoring method based on multi-modal sensing according to claim 1, characterized in that, The step of judging the risk type and judging whether to activate the emergency response based on the final risk level and the historical risk data further comprises: The emergency response comprises a capsizing emergency response, a collision emergency response, and a structure fatigue emergency response. When R > R1', T f activating a roll-over emergency response when R > R1', T When R > R2', T f activate a collision emergency response; when R > R3', T f activate the structural fatigue emergency response when the structure is at risk of structural fatigue; Wherein, R1' is a preset first activation threshold, R2' is a preset second activation threshold, and R3' is a preset third activation threshold.

5. The method of claim 1, wherein the method is characterized by: The step of executing the risk avoidance operation and generating a feedback report when the emergency response is activated comprises: generating corresponding risk warning information based on the final risk level and the risk type; The risk warning information comprises risk description, recommended risk avoidance measures, and emergency contact information; Based on the risk type, matching corresponding risk avoidance operation instructions from a preset risk avoidance operation library; Based on the risk warning information and the risk avoidance operation instructions, executing the risk avoidance operation; Real-time monitoring of the execution effect of the risk avoidance operation, the execution effect comprising ship state changes and risk level changes; Generating a feedback report, the feedback report comprising the risk warning information and the execution effect of the risk avoidance operation.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method for monitoring the dynamic safety of an urban inland river cruise ship based on multi-modal sensing according to any one of claims 1-5.

7. A computer readable storage medium storing a computer program, wherein the computer program is executable by a processor to implement the steps of the method for monitoring the dynamic safety of an urban inland river cruise ship based on multi-modal sensing according to any one of claims 1-5.

Images