Urban inland river pleasure boat driving dynamic safety monitoring method and system based on multi-modal sensing

Abstract

The invention relates to the technical field of ship safety monitoring, in particular to an urban inland river pleasure boat driving dynamic safety monitoring method and system based on multi-modal sensing, and the method comprises the steps: carrying out the data preprocessing and synchronization based on multi-source heterogeneous data collected by a multi-modal sensor network in real time, and outputting the multi-source heterogeneous data with time-space alignment; performing cross-modal attention model feature fusion based on the multi-source heterogeneous data of space-time alignment, and outputting fused multi-dimensional state features; based on the fused multi-dimensional state features, dynamic risk assessment is executed, and a final risk level is output; based on the final risk level and the historical risk data, the risk type is judged, and whether emergency response is activated or not is judged; when the emergency response is activated, risk avoiding operation is executed, and a feedback report is generated. The method has the effects of effectively preventing navigation risks and reducing potential accident loss.

Description

Technical Field

[0001] The present application relates to the technical field of ship safety monitoring, and in particular to a method and system for dynamic safety monitoring of urban inland river cruise ships based on multimodal sensing. Background Art

[0002] Traditional urban inland river cruise ship safety monitoring technology mainly relies on a single sensor (such as GPS, gyroscope or radar) to collect ship dynamic or environmental data, which has the defects of single data dimension and insufficient scene coverage. For example, attitude monitoring based on inertial sensors cannot perceive external obstacles, and although radar ranging can detect surrounding targets, it is difficult to identify the type of obstacles or evaluate the internal safety status of the hull (such as fire, water leakage). In addition, existing systems usually use independent data processing modules and lack the ability to align and fusion multimodal data in time and space, resulting in insufficient accuracy in real-time risk prediction for complex navigation scenarios (such as bad weather and sudden obstacles), and most solutions only provide passive alarms and cannot link the ship control system to achieve active risk avoidance. Although some studies have attempted to combine vision and inertial sensors, their data processing framework is still limited to static models, and the conflict resolution and collaborative decision-making of multi-source heterogeneous data in dynamic environments have not been solved, limiting the reliability and practicality of the system.

[0003] Therefore, improvements are needed. Summary of the invention

[0004] In order to solve the above technical problems, the present application provides a method and system for dynamic safety monitoring of urban inland river cruise ships based on multimodal sensing.

[0005] The first object of the invention of this application is achieved through the following technical solutions:

[0006] A method for monitoring the driving dynamic safety of an urban inland river cruise ship based on multimodal sensing comprises the following steps:

[0007] Based on the multi-source heterogeneous data collected in real time by the multi-modal sensor network, data preprocessing and synchronization are performed to output multi-source heterogeneous data aligned in time and space;

[0008] Based on the multi-source heterogeneous data aligned in time and space, perform cross-modal attention model feature fusion and output the fused multi-dimensional state features;

[0009] Based on the fused multi-dimensional status features, dynamic risk assessment is performed to output the final risk level;

[0010] Based on the final risk level and historical risk data, determine the risk type and whether to activate the emergency response;

[0011] When an emergency response is activated, avoidance operations are performed and a feedback report is generated.

[0012] In a preferred embodiment, the step of performing data preprocessing and synchronization on the multi-source heterogeneous data collected in real time by the multi-modal sensor network and outputting the spatio-temporally aligned multi-source heterogeneous data includes:

[0013] The multi-source heterogeneous data includes ship dynamic data, environmental perception data, and in-cabin safety data;

[0014] The ship dynamic data includes ship data and power data;

[0015] The ship data includes ship attitude angle data, speed data, heading data, ship load data, heading deviation angle data, and ship braking distance data;

[0016] The power data includes propeller rotation speed data, temperature data, and torque data;

[0017] The environmental perception data includes obstacle recognition data and environmental state data;

[0018] The obstacle recognition data includes obstacle relative distance data, obstacle relative speed data, obstacle classification data, relative azimuth data of the obstacle, and obstacle dynamic trajectory data;

[0019] The environmental state data includes real-time wave height data, wind speed data, and visibility data;

[0020] The in-cabin safety data includes passenger distribution density data, water leakage area location information data, and smoke concentration data;

[0021] Perform timestamp alignment on the multi-source heterogeneous data and unify it to the same coordinate system;

[0022] Perform data cleaning on the unified multi-source heterogeneous data, and the data cleaning includes removing noise, outliers, and redundant data.

[0023] In a preferred embodiment, the step of performing preset cross-modal attention model feature fusion on the spatio-temporally aligned multi-source heterogeneous data and outputting the fused multi-dimensional state features includes:

[0024] Based on a preset cross-modal attention model, perform feature fusion on the multi-source heterogeneous data to generate a multi-dimensional state vector containing spatio-temporal correlation;

[0025] Based on Kalman filtering, perform dynamic noise suppression on the multi-dimensional state vector and output the fused multi-dimensional state features.

[0026] In a preferred embodiment, the step of performing dynamic risk assessment on the fused multi-dimensional state features and outputting the final risk level includes:

[0027] The dynamic risk assessment model includes short-term risk assessment and long-term risk assessment;

[0028] The short-term risk assessment: Based on a preset coupled hydrodynamics simulation model and the ship attitude angle data, calculate the ship capsizing risk coefficient K at a preset period;

[0029] Based on the obstacle relative velocity data, the course deviation angle data, and the ship braking distance, construct a collision probability Bayesian network model and output the collision probability P;

[0030] The long-term risk assessment: Based on the vibration spectrum analysis method, extract the hull structure resonance frequency, and combine with the historical fatigue data in a preset ship maintenance database to predict the remaining life T of the key parts of the hull;

[0031] Based on the time window integration method, calculate the cumulative value Z of the course deviation angle within a preset time period.

[0032] In a preferred embodiment, the step of performing dynamic risk assessment based on the fused multi-dimensional state features and outputting the final risk level further includes:

[0033] When K≥ε1, P<δ1, ε1 is a preset first capsizing threshold, and δ1 is a preset first collision threshold, the final risk level is the short-term low risk level;

[0034] When ε2≤K<ε1, δ1≤P<δ2, ε2 is a preset second capsizing threshold, and δ2 is a preset second collision threshold, the final risk level is the short-term medium risk level;

[0035] When K<ε2, P≥δ2, it is the short-term high risk level;

[0036] When T≥0.8·μ, μ is a preset life threshold, is a preset course deviation threshold, the final risk level is the long-term low risk level;

[0037] When 0.5·μ≤T<0.8·μ, the final risk level is the long-term medium risk level;

[0038] When T<0.5·μ, the final risk level is the long-term high risk level;

[0039] When the short-term risk level is inconsistent with the long-term risk level, take the higher risk level of the two as the final risk level;

[0040] When the short-term risk level is consistent with the long-term risk level, directly adopt this risk level as the final risk level.

[0041] In a preferred embodiment, the step of determining the risk type and determining whether to activate the emergency response based on the final risk level and historical risk data includes:

[0042] Integrate the final risk level and historical risk event data to generate a complete data set;

[0043] Extract key features from the integrated complete data set, where the key features include the risk level value R and the corresponding change rate V, and the type T of historical risk events h and the corresponding frequency F;

[0044] Based on a preset risk classification model, compare the key features with preset thresholds to determine the risk type T f ;

[0045] When R≥R1, V≥V1, T h = Overturning, F≥F1, then T f is an overturning risk;

[0046] When R≥R2, V≥V2, T h = Collision, F≥F3, then T f is a collision risk;

[0047] When R≥R3, V≥V3, T h = Structural fatigue, F≥F3, then T f is a structural fatigue risk;

[0048] Among them, R1 is a preset first risk level threshold, R2 is a preset second risk level threshold, and R3 is a preset third risk level threshold;

[0049] Among them, V1 is a preset first risk level change rate threshold, V2 is a preset second risk level change rate threshold, and V3 is a preset third risk level change rate threshold;

[0050] Among them, F1 is a preset first historical risk event frequency threshold, F2 is a preset second historical risk event frequency threshold, and F3 is a preset third historical risk event frequency threshold.

[0051] In a preferred embodiment, the step of determining the risk type and determining whether to activate the emergency response based on the final risk level and historical risk data further includes:

[0052] The emergency response includes an overturning emergency response, a collision emergency response, and a structural fatigue emergency response;

[0053] When R>R′1, T f is an overturning risk, activate the overturning emergency response;

[0054] When R > R′2, T f is a collision risk, activate the collision emergency response;

[0055] When R > R′3, T f is a structural fatigue risk, activate the structural fatigue emergency response;

[0056] Among them, R′1 is the preset first activation threshold, R′2 is the preset second activation threshold, and R′3 is the preset third activation threshold.

[0057] In a preferred embodiment, the steps of performing an evasive action and generating a feedback report when activating the emergency response include:

[0058] Generate corresponding risk warning information based on the final risk level and risk type;

[0059] The risk warning information includes risk description, recommended evasive measures, and emergency contact information;

[0060] Match the corresponding evasive action instructions from the preset evasive action library based on the risk type;

[0061] Perform an evasive action based on the risk warning information and evasive action instructions;

[0062] Real-time monitor the execution effect of the evasive action, and the execution effect includes changes in ship status and risk level;

[0063] Generate a feedback report, and the feedback report includes risk warning information and the execution effect of the evasive action.

[0064] The second invention object of this application is achieved through the following technical solutions:

[0065] A dynamic safety monitoring system for urban inland river cruise ships based on multi-modal sensing, comprising:

[0066] The first output module: Based on the multi-source heterogeneous data collected in real time by the multi-modal sensor network, perform data preprocessing and synchronization, and output the spatio-temporally aligned multi-source heterogeneous data;

[0067] The second output module: Based on the spatio-temporally aligned multi-source heterogeneous data, perform cross-modal attention model feature fusion, and output the fused multi-dimensional state features;

[0068] The third output module: Based on the fused multi-dimensional state features, perform dynamic risk assessment, and output the final risk level;

[0069] The judgment module: Based on the final risk level and historical risk data, judge the risk type and whether to activate the emergency response;

[0070] Generation module: When the emergency response is activated, perform an evasion operation and generate a feedback report.

[0071] The third object of the invention of this application is achieved through the following technical solutions:

[0072] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned method for dynamically monitoring the safety of urban inland river cruise ship driving based on multi-modal sensing are implemented.

[0073] The fourth object of the invention of this application is achieved through the following technical solutions:

[0074] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the steps of the above-mentioned method for dynamically monitoring the safety of urban inland river cruise ship driving based on multi-modal sensing are implemented.

[0075] In summary, this application includes at least one of the following beneficial technical effects:

[0076] By using a multi-modal sensor network to collect multi-source heterogeneous data of urban inland river cruise ships in real time, and through data preprocessing and synchronization (S10), the spatio-temporal alignment of the data is ensured. Then, a cross-modal attention model is used to perform feature fusion on the spatio-temporally aligned data (S20), effectively integrating information from different sensors and generating multi-dimensional state features that comprehensively reflect the state of the cruise ship. Based on these features, the system performs dynamic risk assessment (S30) and outputs the final risk level. Combining the final risk level and historical risk data, the system determines the risk type and decides whether to activate the emergency response (S40). Once the emergency response is activated, the system immediately performs an evasion operation and generates a feedback report (S50), which details the risk warning information, evasion measures, and implementation effects. This method realizes the real-time and accurate monitoring of the dynamic safety of urban inland river cruise ship driving, effectively improving the safety of cruise ship driving and the efficiency of emergency response. Description of the Drawings

[0077] Figure 1 is a flowchart of an implementation of an embodiment of a method for dynamically monitoring the safety of urban inland river cruise ship driving based on multi-modal sensing in this application;

[0078] Figure 2 is a flowchart of an implementation of step S20 in an embodiment of a method for dynamically monitoring the safety of urban inland river cruise ship driving based on multi-modal sensing in this application;

[0079] Figure 3It is a flowchart of the implementation of step S30 in an embodiment of a method for dynamically monitoring the safety of urban inland river cruise ship navigation based on multimodal sensing in this application;

[0080] Figure 4 It is another flowchart of the implementation of step S30 in an embodiment of a method for dynamically monitoring the safety of urban inland river cruise ship navigation based on multimodal sensing in this application;

[0081] Figure 5 It is a flowchart of the implementation of step S40 in an embodiment of a method for dynamically monitoring the safety of urban inland river cruise ship navigation based on multimodal sensing in this application;

[0082] Figure 6 It is a principle block diagram of a computer device in this application. Detailed implementation manners

[0083] The following further describes this application in detail with reference to the appended Figure 1-6 drawings.

[0084] In one embodiment, as Figure 1 shown, this application discloses a method for dynamically monitoring the safety of urban inland river cruise ship navigation based on multimodal sensing, which specifically includes the following steps:

[0085] S10: Based on the multi-source heterogeneous data collected in real time by the multimodal sensor network, perform data preprocessing and synchronization, and output the spatio-temporally aligned multi-source heterogeneous data;

[0086] S20: Based on the spatio-temporally aligned multi-source heterogeneous data, perform cross-modal attention model feature fusion, and output the fused multi-dimensional state features;

[0087] S30: Based on the fused multi-dimensional state features, perform dynamic risk assessment, and output the final risk level;

[0088] S40: Based on the final risk level and historical risk data, judge the risk type and determine whether to activate the emergency response;

[0089] S50: When the emergency response is activated, perform evasive actions and generate a feedback report.

[0090] In this embodiment, multi-source heterogeneous data of urban inland river cruise ships is collected in real time through a multi-modal sensor network, and through data preprocessing and synchronization (S10), the spatio-temporal alignment of the data is ensured. Then, a cross-modal attention model is used to perform feature fusion on the spatio-temporally aligned data (S20), effectively integrating information from different sensors and generating multi-dimensional state features that comprehensively reflect the state of the cruise ship. Based on these features, the system performs dynamic risk assessment (S30) and outputs the final risk level. Combining the final risk level and historical risk data, the system determines the risk type and decides whether to activate the emergency response (S40). Once the emergency response is activated, the system immediately performs evasive actions and generates a feedback report (S50), detailing the risk warning information, evasive measures, and implementation effects. This method realizes real-time and accurate monitoring of the dynamic safety of urban inland river cruise ship navigation, effectively improving the safety of cruise ship navigation and the efficiency of emergency response.

[0091] Step S10 includes:

[0092] S101: The multi-source heterogeneous data includes ship dynamic data, environmental perception data, and in-cabin safety data;

[0093] S102: The ship dynamic data includes ship data and power data;

[0094] S103: The ship data includes ship attitude angle data, speed data, heading data, ship load data, heading deviation angle data, and ship braking distance data;

[0095] S104: The power data includes propeller rotation speed data, temperature data, and torque data;

[0096] S105: The environmental perception data includes obstacle recognition data and environmental state data;

[0097] S106: The obstacle recognition data includes relative distance data of obstacles, relative speed data of obstacles, obstacle classification data, relative azimuth data of obstacles, and dynamic trajectory data of obstacles;

[0098] S107: The environmental state data includes real-time wave height data, wind speed data, and visibility data;

[0099] S108: The in-cabin safety data includes passenger distribution density data, location information data of leakage areas, and smoke concentration data;

[0100] S109: Align the timestamps of the multi-source heterogeneous data and unify them to the same coordinate system;

[0101] S110: Clean the unified multi-source heterogeneous data, and the data cleaning includes removing noise, outliers, and redundant data.

[0102] In this embodiment, in step S10, the multi-modal sensor network comprehensively collects the ship dynamic data, environmental perception data, and in-cabin safety data of the cruise ship, ensuring the comprehensiveness and real-time nature of the monitoring. Among them, the ship dynamic data is subdivided into ship data (such as attitude angle, speed, course, etc.) and power data (such as propeller speed, temperature, torque, etc.), the environmental perception data includes obstacle recognition data (such as relative distance, speed, classification, etc.) and environmental state data (such as wave height, wind speed, visibility, etc.), and the in-cabin safety data focuses on key safety indicators such as passenger distribution, leakage location, and smoke concentration. Through timestamp alignment and coordinate unification, the consistency and accuracy of the data are ensured. Further data cleaning steps effectively remove noise, outliers, and redundant data, providing a high-quality data basis for subsequent feature fusion and risk assessment. This series of data processing measures guarantees the accuracy and reliability of the monitoring system, providing strong protection for the driving safety of urban inland river cruise ships.

[0103] Figure 2 , step S20 includes:

[0104] S201: Based on a preset cross-modal attention model, perform feature fusion on the multi-source heterogeneous data to generate a multi-dimensional state vector containing spatio-temporal correlation;

[0105] S202: Based on Kalman filtering, perform dynamic noise suppression on the multi-dimensional state vector and output the fused multi-dimensional state features.

[0106] In this embodiment, an advanced cross-modal attention model is adopted to perform in-depth feature fusion on the preprocessed multi-source heterogeneous data. This model can effectively capture the spatio-temporal correlation between different data sources and generate a multi-dimensional state vector containing rich information. To further improve the data quality, the system further applies Kalman filtering technology to perform dynamic noise suppression on the multi-dimensional state vector, effectively filtering out the random interference and uncertainty in the data, and finally outputting more accurate and reliable fused multi-dimensional state features. This step not only improves the accuracy and efficiency of data processing, but also provides a solid data basis for subsequent dynamic risk assessment, ensuring the accuracy and effectiveness of the dynamic safety monitoring of urban inland river cruise ships.

[0107] Figure 3 , step S30 includes:

[0108] S301: The dynamic risk assessment model includes short-term risk assessment and long-term risk assessment;

[0109] S302: The short-term risk assessment: Based on a preset coupled hydrodynamics simulation model and the ship attitude angle data, calculate the ship capsizing risk coefficient K at a preset period;

[0110] S303: Based on the relative velocity data of the obstacle, the course deviation angle data, and the ship's braking distance, construct a Bayesian network model for collision probability and output the collision probability P;

[0111] S304: The long-term risk assessment: Based on the vibration spectrum analysis method, extract the resonance frequency of the hull structure, and combine with the historical fatigue data in the preset ship maintenance database to predict the remaining life T of the key parts of the hull;

[0112] S305: Based on the time window integration method, calculate the cumulative value Z of the course deviation angle within a preset time period.

[0113] In this embodiment, a comprehensive dynamic risk assessment model is adopted. This model is divided into two parts: short-term and long-term risk assessments. The short-term risk assessment regularly calculates the capsizing risk coefficient K of the ship through a preset coupled hydrodynamics simulation model and ship attitude angle data. At the same time, a Bayesian network model for collision probability is constructed using the relative velocity data of the obstacle, the course deviation angle data, and the ship's braking distance, and the collision probability P is output in real time, so as to achieve accurate assessment of short-term risks. The long-term risk assessment extracts the resonance frequency of the hull structure by using the vibration spectrum analysis method, and combines with the historical fatigue data to predict the remaining life T of the key parts of the hull. At the same time, the cumulative value Z of the course deviation angle is calculated by the time window integration method to provide prediction and early warning for long-term safety. This hierarchical and multi-angle risk assessment method not only improves the comprehensiveness and accuracy of risk identification, but also enhances the safety and predictability of the cruise ship's operation, providing scientific and effective decision-making support for the dynamic safety monitoring of urban inland river cruise ships.

[0114] Figure 4 , step S30 also includes:

[0115] SB1: When K≥ε1, P<δ1, where ε1 is the preset first capsizing threshold and δ1 is the preset first collision threshold, the final risk level is the short-term low risk level;

[0116] SB2: When ε2≤K<ε1, δ1≤P<δ2, where ε2 is the preset second capsizing threshold and δ2 is the preset second collision threshold, the final risk level is the short-term medium risk level;

[0117] SB3: When K<ε2, P≥δ2, it is the short-term high risk level;

[0118] SB4: When T≥0.8·μ, where μ is the preset life threshold, is the preset course deviation threshold, the final risk level is the long-term low risk level;

[0119] SB5: When 0.5·μ ≤ T < 0.8·μ, the final risk level is the long - term medium risk level;

[0120] SB6: When T < 0.5·μ, the final risk level is the long - term high risk level;

[0121] SB7: When the short - term risk level is inconsistent with the long - term risk level, take the higher risk level of the two as the final risk level;

[0122] SB8: When the short - term risk level is consistent with the long - term risk level, directly adopt this risk level as the final risk level.

[0123] In this embodiment, the logic of dynamic risk assessment is further refined by setting multi - level thresholds to distinguish different levels of risk. In short - term risk assessment, according to the comparison of the capsizing risk coefficient K and the collision probability P with the preset first capsizing threshold ε1, the preset second capsizing threshold ε2, the preset first collision threshold δ1, and the preset second collision threshold δ2, the risk is divided into three levels: low, medium, and high (SB1, SB2, SB3). Long - term risk assessment is based on the comparison of the remaining hull life T and the cumulative value Z of the course deviation angle with the preset life threshold μ and the preset course deviation threshold Similarly, the risk is divided into three levels: low, medium, and high (SB4, SB5, SB6). In addition, this embodiment also considers the situation where the short - term and long - term risk levels may be inconsistent. By taking the higher risk level of the two as the final risk level (SB7), the conservativeness and safety of risk assessment are ensured. When the short - term and long - term risk levels are consistent, directly adopt this risk level as the final risk level (SB8), which simplifies the decision - making process. This hierarchical and classified risk assessment mechanism improves the fineness and accuracy of risk judgment, providing a more scientific and reasonable risk warning and decision - making basis for the safe operation of urban inland river cruise ships.

[0124] Figure 5 , step S40 includes:

[0125] S401: Integrate the final risk level and historical risk event data to generate a complete data set;

[0126] S402: Extract key features from the integrated complete data set. The key features include the risk level value R and the corresponding change rate V, the type T h of historical risk events and the corresponding frequency F;

[0127] S403: Based on a preset risk classification model, compare the key features with preset thresholds to determine the risk type T f ;

[0128] S404: When R ≥ R1, V ≥ V1, T h = capsizing, F ≥ F1, then T f is the capsizing risk;

[0129] S405: When R ≥ R2, V ≥ V2, T h = collision, F ≥ F3, then T f is the collision risk;

[0130] S406: When R ≥ R3, V ≥ V3, T h = structural fatigue, F ≥ F3, then T f is the structural fatigue risk;

[0131] S407: Among them, R1 is the preset first risk level threshold, R2 is the preset second risk level threshold, and R3 is the preset third risk level threshold;

[0132] S408: Among them, V1 is the preset first risk level change rate threshold, V2 is the preset second risk level change rate threshold, and V3 is the preset third risk level change rate threshold;

[0133] S409: Among them, F1 is the preset first historical risk event frequency threshold, F2 is the preset second historical risk event frequency threshold, and F3 is the preset third historical risk event frequency threshold.

[0134] In this embodiment, by integrating the final risk level and historical risk event data, a complete data set is generated, providing a comprehensive information basis for the accurate judgment of risk types. Then, key features are extracted from the integrated data set, including the risk level value R and its change rate V, the type T h of historical risk events and their frequency F. These features can effectively reflect the dynamic changes and historical trends of risks. Based on the preset risk classification model, the extracted key features are compared with the preset multi-level thresholds to determine the specific risk type T f , such as capsizing risk, collision risk, and structural fatigue risk, etc. By setting different levels of risk level thresholds, risk level change rate thresholds, and historical risk event frequency thresholds (such as the preset first, second, and third thresholds), this embodiment realizes the fine division and accurate identification of risk types. This risk type judgment mechanism not only improves the accuracy and pertinence of risk identification, but also provides a scientific and reasonable basis for further emergency response and risk avoidance operations, significantly enhancing the effectiveness and practicality of the dynamic safety monitoring system for urban inland river cruise ships.

[0135] The S40 step also includes:

[0136] SE1: The emergency response includes capsizing emergency response, collision emergency response, and structural fatigue emergency response;

[0137] SE2: When R > R′1, T f is the capsizing risk, activate the capsizing emergency response;

[0138] SE3: When R > R′2, T f is the collision risk, activate the collision emergency response;

[0139] SE4: When R > R′3, T f is the structural fatigue risk, activate the structural fatigue emergency response;

[0140] SE5: Among them, R′1 is the preset first activation threshold, R′2 is the preset second activation threshold, and R′3 is the preset third activation threshold.

[0141] In this embodiment, the emergency response mechanism is further refined, and corresponding emergency response strategies are set for different types of risks. Through the preset activation thresholds, the system can automatically activate the corresponding emergency response according to the judged risk type, including capsizing emergency response, collision emergency response, and structural fatigue emergency response. Specifically, when the risk level reaches the preset capsizing risk activation threshold, the system will activate the capsizing emergency response (SE2); similarly, when the risk level reaches the preset collision risk activation threshold, the system will activate the collision emergency response (SE3); and when the risk level reaches the preset structural fatigue risk activation threshold, the system will activate the structural fatigue emergency response (SE4). This type-based and level-based emergency response mechanism ensures that corresponding risk avoidance measures can be taken quickly and accurately when facing different risks, effectively reducing the possibility and loss of accidents. Through the preset first, second, and third activation thresholds (SE5), this embodiment realizes the flexibility and pertinence of the emergency response, and further improves the emergency handling ability and overall safety of the urban inland river cruise ship dynamic safety monitoring system.

[0142] Step S50 includes:

[0143] S501: Generate corresponding risk warning information based on the final risk level and risk type;

[0144] S502: The risk warning information includes risk description, recommended risk avoidance measures, and emergency contact information;

[0145] S503: Match the corresponding risk avoidance operation instructions from the preset risk avoidance operation library based on the risk type;

[0146] S504: Execute the risk avoidance operation based on the risk warning information and risk avoidance operation instructions;

[0147] S505: Monitor the execution effect of the risk avoidance operation in real time, where the execution effect includes changes in the ship's state and changes in the risk level.

[0148] S506: Generate a feedback report, where the feedback report includes risk warning information and the execution effect of the risk avoidance operation.

[0149] In this embodiment, a complete closed-loop management from risk warning to the execution of risk avoidance operations and then to effect monitoring is achieved. First, the system generates detailed risk warning information based on the final risk level and risk type, including risk descriptions, recommended risk avoidance measures, and emergency contact information, providing comprehensive risk awareness and response guidance for relevant personnel (S501, S502). Then, according to the risk type, corresponding risk avoidance operation instructions are matched from the preset risk avoidance operation library to ensure the pertinence and effectiveness of the risk avoidance operations (S503). Subsequently, the system executes the risk avoidance operation and monitors the execution effect of the risk avoidance operation in real time, including changes in the ship's state and changes in the risk level, so as to timely evaluate the effect of the risk avoidance measures and make necessary adjustments (S504, S505). Finally, a feedback report is generated, summarizing the risk warning information and the execution effect of the risk avoidance operation, providing important references for subsequent risk management and decision-making (S506). This series of steps constitutes a dynamic and efficient safety monitoring and emergency response system, significantly improving the response ability and safety of urban inland river cruise ships in the face of potential risks, ensuring the smooth progress of cruise ship travel and the safety of passengers.

[0150] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0151] In one embodiment, a dynamic safety monitoring system for urban inland river cruise ship travel based on multi-modal sensing is provided. This dynamic safety monitoring system for urban inland river cruise ship travel based on multi-modal sensing corresponds to the above-mentioned method for dynamic safety monitoring of urban inland river cruise ship travel based on multi-modal sensing. This dynamic safety monitoring system for urban inland river cruise ship travel based on multi-modal sensing includes:

[0152] The first output module: Based on the multi-source heterogeneous data collected in real time by the multi-modal sensor network, perform data preprocessing and synchronization, and output spatio-temporally aligned multi-source heterogeneous data.

[0153] The second output module: Based on the spatio-temporally aligned multi-source heterogeneous data, perform cross-modal attention model feature fusion, and output the fused multi-dimensional state features.

[0154] The third output module: Based on the fused multi-dimensional state features, perform dynamic risk assessment, and output the final risk level.

[0155] Judgment module: Based on the final risk level and historical risk data, judge the risk type and whether to activate the emergency response;

[0156] Generation module: When the emergency response is activated, perform hazard avoidance operations and generate a feedback report.

[0157] Optionally, it further includes:

[0158] First inclusion module: The multi-source heterogeneous data includes ship dynamic data, environmental perception data, and in-cabin safety data;

[0159] Second inclusion module: The ship dynamic data includes ship data and power data;

[0160] Third inclusion module: The ship data includes ship attitude angle data, speed data, course data, ship load data, course deviation angle data, and ship braking distance data;

[0161] Fourth inclusion module: The power data includes propeller rotation speed data, temperature data, and torque data;

[0162] Fifth inclusion module: The environmental perception data includes obstacle recognition data and environmental state data;

[0163] Sixth inclusion module: The obstacle recognition data includes obstacle relative distance data, obstacle relative speed data, obstacle classification data, relative azimuth data of the obstacle, and obstacle dynamic trajectory data;

[0164] Seventh inclusion module: The environmental state data includes real-time wave height data, wind speed data, and visibility data;

[0165] Eighth inclusion module: The in-cabin safety data includes passenger distribution density data, water leakage area location information data, and smoke concentration data;

[0166] Unification module: Align the time stamps of the multi-source heterogeneous data and unify them to the same coordinate system;

[0167] Cleaning module: Clean the unified multi-source heterogeneous data, and the data cleaning includes removing noise, outliers, and redundant data.

[0168] Optionally, it further includes:

[0169] First generation module: Based on a preset cross-modal attention model, perform feature fusion on the multi-source heterogeneous data to generate a multi-dimensional state vector containing spatio-temporal correlation;

[0170] Fourth output module: Based on Kalman filtering, perform dynamic noise suppression on the multi-dimensional state vector and output the fused multi-dimensional state features.

[0171] Optionally, it further includes:

[0172] First module: The dynamic risk assessment model includes short-term risk assessment and long-term risk assessment;

[0173] First calculation module: The short-term risk assessment: Based on a preset coupled hydrodynamics simulation model and the ship attitude angle data, calculate the ship capsizing risk coefficient K at a preset period;

[0174] Fifth output module: Based on the obstacle relative velocity data, the course deviation angle data, and the ship braking distance, construct a collision probability Bayesian network model and output the collision probability P;

[0175] Prediction module: The long-term risk assessment: Based on the vibration spectrum analysis method, extract the hull structure resonance frequency, and combine with the historical fatigue data in a preset ship maintenance database to predict the remaining life T of the key parts of the hull;

[0176] Second calculation module: Based on the time window integration method, calculate the cumulative value Z of the course deviation angle within a preset time period.

[0177] Optionally, it further includes:

[0178] First judgment module: When K≥ε1, P<δ1, ε1 is a preset first capsizing threshold, and δ1 is a preset first collision threshold, the final risk level is the short-term low risk level;

[0179] Second judgment module: When ε2≤K<ε1, δ1≤P<δ2, ε2 is a preset second capsizing threshold, and δ2 is a preset second collision threshold, the final risk level is the short-term medium risk level;

[0180] Third judgment module: When K<ε2, P≥δ2, it is the short-term high risk level;

[0181] Fourth judgment module: When T≥0.8·μ, μ is a preset life threshold, is a preset course deviation threshold, the final risk level is the long-term low risk level;

[0182] Fifth judgment module: When 0.5·μ≤T<0.8·μ, the final risk level is the long-term medium risk level;

[0183] Sixth judgment module: When T<0.5·μ, the final risk level is the long-term high risk level;

[0184] The seventh judgment module: when the short-term risk level is inconsistent with the long-term risk level, take the higher risk level of the two as the final risk level;

[0185] The eighth judgment module: when the short-term risk level is consistent with the long-term risk level, directly adopt this risk level as the final risk level.

[0186] Optionally, it further includes:

[0187] The integration module: integrate the final risk level and historical risk event data to generate a complete data set;

[0188] The extraction module: extract key features from the integrated complete data set, and the key features include the risk level value R and the corresponding change rate V, the type T of historical risk events h and the corresponding frequency F;

[0189] The comparison module: based on a preset risk classification model, compare the key features with preset thresholds to judge the risk type T f ;

[0190] The second module: when R≥R1, V≥V1, T h = capsizing, F≥F1, T f is the capsizing risk;

[0191] The third module: when R≥R2, V≥V2, T h = collision, F≥F3, T f is the collision risk;

[0192] The fourth module: when R≥R3, V≥V3, T h = structural fatigue, F≥F3, T f is the structural fatigue risk;

[0193] The fifth module: where R1 is the preset first risk level threshold, R2 is the preset second risk level threshold, and R3 is the preset third risk level threshold;

[0194] The sixth module: where V1 is the preset first risk level change rate threshold, V2 is the preset second risk level change rate threshold, and V3 is the preset third risk level change rate threshold;

[0195] The seventh module: where F1 is the preset first historical risk event frequency threshold, F2 is the preset second historical risk event frequency threshold, and F3 is the preset third historical risk event frequency threshold.

[0196] Optionally, it further includes:

[0197] Eighth Module: The emergency response includes capsizing emergency response, collision emergency response, and structural fatigue emergency response;

[0198] First Activation Module: When R > R′1, T f is the capsizing risk, activate the capsizing emergency response;

[0199] Second Activation Module: When R > R′2, T f is the collision risk, activate the collision emergency response;

[0200] Third Activation Module: When R > R′3, T f is the structural fatigue risk, activate the structural fatigue emergency response;

[0201] Ninth Module: Among them, R′1 is the preset first activation threshold, R′2 is the preset second activation threshold, and R′3 is the preset third activation threshold.

[0202] Optionally, it further includes:

[0203] Early Warning Information Module: Generate corresponding risk early warning information based on the final risk level and risk type;

[0204] Risk Module: The risk early warning information includes risk description, recommended risk avoidance measures, and emergency contact information;

[0205] Matching Module: Match corresponding risk avoidance operation instructions from the preset risk avoidance operation library based on the risk type;

[0206] Execution Module: Execute risk avoidance operations based on the risk early warning information and risk avoidance operation instructions;

[0207] Monitoring Module: Monitor the execution effect of risk avoidance operations in real time. The execution effect includes changes in ship status and changes in risk level;

[0208] Feedback Report Module: Generate a feedback report, which includes risk early warning information and the execution effect of risk avoidance operations.

[0209] For the specific limitations of a dynamic safety monitoring system for urban inland river cruise ships based on multi-modal sensing, reference can be made to the limitations of a dynamic safety monitoring method for urban inland river cruise ships based on multi-modal sensing in the above text, which will not be elaborated here. Each module in the above-mentioned dynamic safety monitoring system for urban inland river cruise ships based on multi-modal sensing can be implemented in whole or in part through software, hardware, and their combinations. The above-mentioned modules can be embedded in the processor of a computer device in hardware form or be independent of it, or be stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

[0210] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in Figure 6 . The computer device includes a processor, a memory, a network interface, and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store ship dynamic data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a method for dynamically monitoring the safety of the operation of urban inland river cruise ships based on multi-modal sensing.

[0211] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements a method for dynamically monitoring the safety of the operation of urban inland river cruise ships based on multi-modal sensing.

[0212] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, it is a method for dynamically monitoring the safety of the operation of urban inland river cruise ships based on multi-modal sensing.

[0213] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0214] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

Claims

1. A method for dynamically monitoring the driving safety of urban inland river cruise ships based on multimodal sensing, characterized in that, Including the steps: Based on the multi-source heterogeneous data collected in real time by the multi-modal sensor network, perform data preprocessing and synchronization, and output the multi-source heterogeneous data with spatio-temporal alignment; Based on the multi-source heterogeneous data with spatio-temporal alignment, perform cross-modal attention model feature fusion, and output the fused multi-dimensional state features; Based on the fused multi-dimensional state features, perform dynamic risk assessment, and output the final risk level; Based on the final risk level and historical risk data, judge the risk type and determine whether to activate the emergency response; When the emergency response is activated, perform evasive actions and generate a feedback report.

2. The dynamic safety monitoring method for the operation of urban inland river cruise ships based on multi-modal sensing according to claim 1, characterized in that The step of performing data preprocessing and synchronization based on the multi-source heterogeneous data collected in real time by the multi-modal sensor network and outputting the multi-source heterogeneous data with spatio-temporal alignment includes: The multi-source heterogeneous data includes ship dynamic data, environmental perception data, and in-cabin safety data; The ship dynamic data includes ship data and power data; The ship data includes ship attitude angle data, ship speed data, course data, ship load data, course deviation angle data, and ship braking distance data; The power data includes propeller rotation speed data, temperature data, and torque data; The environmental perception data includes obstacle recognition data and environmental state data; The obstacle recognition data includes obstacle relative distance data, obstacle relative speed data, obstacle classification data, relative azimuth data of the obstacle, and obstacle dynamic trajectory data; The environmental state data includes real-time wave height data, wind speed data, and visibility data; The in-cabin safety data includes passenger distribution density data, leak area location information data, and smoke concentration data; Perform timestamp alignment on the multi-source heterogeneous data and unify it to the same coordinate system; Perform data cleaning on the unified multi-source heterogeneous data. The data cleaning includes removing noise, outliers, and redundant data.

3. A dynamic safety monitoring method for the operation of urban inland river cruise ships based on multi-modal sensing according to claim 1, characterized in that The step of performing preset cross-modal attention model feature fusion based on the multi-source heterogeneous data with spatio-temporal alignment and outputting the fused multi-dimensional state features includes: Based on a preset cross-modal attention model, perform feature fusion on the multi-source heterogeneous data to generate a multi-dimensional state vector containing spatio-temporal correlation; Based on the Kalman filter, perform dynamic noise suppression on the multi-dimensional state vector and output the fused multi-dimensional state features.

4. A method for dynamically monitoring the safety of an urban inland river cruise ship based on multimodal sensing according to claim 1, characterized in that, The step of performing dynamic risk assessment based on the fused multi-dimensional state features and outputting the final risk level includes: The dynamic risk assessment model includes short-term risk assessment and long-term risk assessment; The short-term risk assessment: Based on a preset coupled hydrodynamics simulation model and the ship attitude angle data, calculate the ship capsizing risk coefficient K at a preset period; Based on the obstacle relative speed data, the course deviation angle data, and the ship braking distance, construct a collision probability Bayesian network model and output the collision probability P; The long-term risk assessment: Based on the vibration spectrum analysis method, extract the hull structure resonance frequency, and combine the historical fatigue data in a preset ship maintenance database to predict the remaining life T of the key parts of the hull; Based on the time window integration method, calculate the cumulative value Z of the course deviation angle within a preset time period.

5. A dynamic safety monitoring method for the operation of urban inland river cruise ships based on multi-modal sensing according to claim 1, characterized in that, The step of performing dynamic risk assessment based on the fused multi-dimensional state features and outputting the final risk level further includes: When K≥ε1, P<δ1, where ε1 is a preset first overturning threshold and δ1 is a preset first collision threshold, the final risk level is a short-term low risk level; When ε2≤K<ε1, δ1≤P<δ2, where ε2 is a preset second overturning threshold and δ2 is a preset second collision threshold, the final risk level is a short-term medium risk level; When K<ε2, P≥δ2, it is a short-term high risk level; When T ≥ 0.8·μ, where μ is a preset life threshold, and is a preset course deviation threshold, the final risk level is a long-term low risk level; When 0.5·μ ≤ T < 0.8·μ, the final risk level is a medium-term long-term risk level; When T < 0.5·μ, the final risk level is the long-term high-risk level; When the short-term risk level is inconsistent with the long-term risk level, the higher risk level of the two is taken as the final risk level; When the short-term risk level is consistent with the long-term risk level, directly adopt this risk level as the final risk level.

6. The dynamic safety monitoring method for the operation of urban inland river cruise ships based on multi-modal sensing according to claim 1, characterized in that, The step of judging the risk type and judging whether to activate the emergency response based on the final risk level and historical risk data includes: Integrate the final risk level and historical risk event data to generate a complete data set; Extract key features from the integrated complete data set, where the key features include the risk level value R and the corresponding change rate V, the type T of historical risk events h and the corresponding frequency F; Based on a preset risk classification model, compare the key features with the preset threshold to determine the risk type T f ; When R≥R1, V≥V1, T h = overturning, F≥F1, then T f is the risk of overturning; When R ≥ R2, V ≥ V2, T h = collision, F ≥ F3, then T f is the collision risk; When R≥R3, V≥V3, T h = structural fatigue, when F≥F3, T f is the risk of structural fatigue; Wherein, R1 is a preset first risk level threshold, R2 is a preset second risk level threshold, and R3 is a preset third risk level threshold; Wherein, V1 is a preset first risk level change rate threshold, V2 is a preset second risk level change rate threshold, and V3 is a preset third risk level change rate threshold; Wherein, F1 is a preset first historical risk event frequency threshold, F2 is a preset second historical risk event frequency threshold, and F3 is a preset third historical risk event frequency threshold.

7. A dynamic safety monitoring method for the operation of urban inland river cruise ships based on multimodal sensing according to claim 1, characterized in that The step of judging the risk type and judging whether to activate the emergency response based on the final risk level and historical risk data further includes: The emergency response includes an overturning emergency response, a collision emergency response, and a structural fatigue emergency response; When R > R1′, T f When there is a risk of capsizing, activate the capsizing emergency response; When R > R2′, T f is a collision risk, activate the collision emergency response; When R > R3′, T f When it is a structural fatigue risk, activate the structural fatigue emergency response; Wherein, R1′ is a preset first activation threshold, R2′ is a preset second activation threshold, and R3′ is a preset third activation threshold.

8. A dynamic safety monitoring method for the operation of urban inland river cruise ships based on multi-modal sensing according to claim 1, characterized in that, The step of performing a risk avoidance operation and generating a feedback report when the emergency response is activated includes: Generate corresponding risk warning information based on the final risk level and risk type; The risk warning information includes risk description, recommended risk avoidance measures, and emergency contact information; Match corresponding risk avoidance operation instructions from a preset risk avoidance operation library based on the risk type; Perform a risk avoidance operation based on the risk warning information and risk avoidance operation instructions; Real-time monitor the execution effect of the risk avoidance operation, and the execution effect includes changes in ship status and risk level; Generate a feedback report, and the feedback report includes risk warning information and the execution effect of the risk avoidance operation.

9. A computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of a method for dynamically monitoring the safety of urban inland river cruise ship navigation based on multi-modal sensing as claimed in claims 1-8.

10. A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps of a method for dynamically monitoring the safety of urban inland river cruise ship navigation based on multi-modal sensing as claimed in claims 1-8.

Images