A ship safety risk monitoring method, device and storage medium

Abstract

The application discloses a kind of ship safety risk monitoring method, device, storage medium and computer program product, belong to ship safety risk monitoring technical field, the method includes: obtaining the multimodal monitoring data of ship and using flow parallel mode to carry out pre-processing and feature extraction, obtain the feature data of each mode;Using the detection framework of multistage progression to each mode feature data carries out risk determination;If the confidence degree of the determination result output by the current detection level meets the preset condition, then directly output monitoring result;If the confidence degree of the determination result output by the current detection level does not meet the preset condition, then enter next detection level;Wherein, according to real-time environmental context, the fusion weight corresponding to each mode is dynamically generated in the last detection level, after fusion processing is carried out based on the fusion weight to multimodal feature data, output final monitoring result.The scheme improves the discrimination accuracy of ship safety risk, reduces false alarm rate, shortens the end-to-end response delay of risk determination.

Description

Technical Field

[0001] This invention belongs to the field of ship safety risk monitoring technology, specifically relating to a ship safety risk monitoring method, device, storage medium, and computer program product. Background Technology

[0002] With the digital development of smart shipping and maritime traffic safety supervision, crew behavior and status monitoring have become core aspects of vessel navigation safety management. Early monitoring methods mainly relied on shore-based personnel manually checking shipboard video footage, which was inefficient and lacked coverage. With advancements in computer vision and sensing technologies, the industry has gradually evolved into automated monitoring solutions that integrate multi-source data. These solutions typically employ a three-layer architecture: data acquisition, independent single-modal analysis, and rule-based engine decision-making. The perception layer collects video streams, audio streams, ship sensor data, and electronic map vector data. The analysis layer independently analyzes crew visual posture, voice keywords, and ship operational status. The decision-making layer combines the results from each modality using preset rules, relying on multimodal cross-validation to reduce the false alarm rate of single video surveillance.

[0003] However, existing technologies essentially employ a serial fusion architecture that extracts single-modal features independently and makes decisions based on fixed rules. The inherent flaws of this architecture are the core problem that current ship crew abnormal behavior monitoring cannot overcome. This directly results in the monitoring scheme being unable to simultaneously achieve high discrimination accuracy and low response latency in complex ship navigation scenarios, specifically manifested as follows:

[0004] First, the fixed-weight fusion strategy cannot dynamically adapt to the complex and ever-changing navigation scenarios and environmental conditions of ships, resulting in insufficient accuracy in abnormal behavior identification and a high false alarm rate. In existing solutions, the fusion weights and judgment logic of each modality are pre-set, and cannot dynamically adjust the modality confidence based on contextual information such as real-time light intensity, environmental noise level, and ship navigation status. For example, during nighttime navigation, the lighting in the bridge is extremely low, the visible light video acquisition effect is poor, and the recognition accuracy drops significantly, but the solution still relies on video data with fixed weights for judgment, which easily leads to a large number of false alarms. At the same time, it cannot combine ship speed and the type of water area to distinguish semantically ambiguous scenarios such as "drowsiness during navigation" and "normal rest at anchor," further increasing the false alarm rate and reducing the system's practicality.

[0005] Secondly, the serial full-scale computation processing mode results in excessive end-to-end response latency, which cannot meet the second-level handling requirements of sudden emergencies. Existing solutions require waiting for the analysis results of all modalities to be output before executing fusion decisions. Each step is executed serially without an early exit mechanism, requiring full-link computation in both simple and complex scenarios. For example, in emergency scenarios such as a crew member suddenly falling or detecting an open flame in a cabin, the end-to-end latency of existing solutions can typically reach 2.5 to 5 seconds, easily missing the optimal intervention and rescue opportunity, posing serious safety hazards.

[0006] Third, spatial region determination relies on real-time geometric calculations, which incur high computational costs and further restrict the real-time processing capabilities of edge devices. Existing electronic fences and water functional area matching require sequential spatial geometric calculations, making it impossible to quickly query scene types. This not only exacerbates the consumption of device computing power but also lengthens the overall determination process.

[0007] In summary, the aforementioned serial fixed rule multimodal fusion architecture makes it difficult for existing monitoring technologies to simultaneously meet the dual requirements of high precision and low latency, and it cannot adapt to the complex and ever-changing navigation scenarios of inland waterway and maritime vessels.

[0008] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0009] The purpose of this invention is to provide a method, device, storage medium, and computer program product for monitoring ship safety risks, in order to solve the problems of existing multimodal monitoring of abnormal behavior of ship crew members, which adopts a fusion architecture based on serial fixed rules, and cannot simultaneously meet the dual requirements of high discrimination accuracy and low response latency in complex navigation scenarios, resulting in high false alarm rates and delayed handling of sudden emergencies. The invention effectively improves the discrimination accuracy of ship safety risks in complex navigation scenarios, reduces the false alarm rate, and significantly shortens the end-to-end response latency of risk assessment.

[0010] This invention provides a method for monitoring ship safety risks, comprising: acquiring multimodal monitoring data of a ship; preprocessing and extracting features from each modal monitoring data using a streaming parallel approach to obtain feature data corresponding to each modality; using a multi-level progressive detection architecture to determine the risk of each modal feature data; if the confidence level of the determination result output by the current detection level meets a preset condition, the monitoring result is directly output, and the execution of subsequent detection levels is terminated; if the confidence level of the determination result output by the current detection level does not meet the preset condition, the next detection level is entered; wherein, the final detection level dynamically generates fusion weights corresponding to each modality based on the real-time environmental context, and after fusing the multimodal feature data based on the fusion weights, the final monitoring result is output.

[0011] In some implementations, the multimodal monitoring data includes: visual modal data, acoustic modal data, ship state modal data, and spatial rule modal data.

[0012] In some implementations, the monitoring data for each modality are preprocessed and feature extracted using a streaming parallel approach, including: dynamically adjusting the window size of the streaming processing based on the temporal characteristics of the risk type to be monitored corresponding to each modality; the window size is used to limit the length of the time-series data in a single round of processing to match the time scale of the corresponding risk behavior to be monitored; within each streaming window, preprocessing operations are performed independently on the original monitoring data of the corresponding modality, including data cleaning, data standardization, and format normalization; multi-level feature extraction is performed on the preprocessed modality data to obtain the feature data of the corresponding modality; wherein the preprocessing and feature extraction processes of each modality are independent and run asynchronously in parallel.

[0013] In some implementations, the multi-level progressive detection architecture includes a first detection level, a second detection level, and a third detection level executed sequentially; the first detection level is used to perform initial screening of each modal feature data based on lightweight rules; the second detection level is used to perform single-modal independent analysis and risk assessment on feature data whose confidence level does not meet preset conditions after being determined by the first detection level; and the third detection level is used to perform fusion processing of the modal feature data.

[0014] In some implementations, the fusion weights corresponding to each modality are dynamically generated based on the real-time environmental context, including: encoding the real-time environmental context information to obtain a context feature vector; inputting the context feature vector into a gating network to output the initial weight parameters corresponding to each modality; and normalizing the initial weight parameters to obtain the final fusion weights of each modality.

[0015] In some implementations, the fusion processing of multimodal feature data based on the fusion weights includes: weighting each modal feature data with its corresponding fusion weight; performing cross-modal attention interaction on the weighted multimodal features to generate a fusion feature vector; and determining security risks based on the fusion feature vectors.

[0016] In some implementations, during the risk assessment of personnel status anomalies, the assessment results are filtered by combining the ship's motion state with the type of water area it is in; when the ship is in a preset stationary state and located in a preset legal mooring area, the monitoring results of the corresponding personnel status anomalies are blocked.

[0017] In conjunction with the above method, another aspect of the present invention provides a ship safety risk monitoring device, comprising: an acquisition unit configured to acquire multimodal monitoring data of a ship; a feature extraction unit configured to preprocess and extract features from each modal monitoring data using a streaming parallel approach to obtain feature data corresponding to each modality; a monitoring unit configured to perform risk assessment on the feature data of each modality using a multi-level progressive detection architecture; the monitoring unit is further configured to directly output the monitoring result and terminate the execution of subsequent detection levels if the confidence level of the assessment result output by the current detection level meets a preset condition; the monitoring unit is further configured to proceed to the next detection level if the confidence level of the assessment result output by the current detection level does not meet the preset condition; wherein, the final detection level dynamically generates fusion weights corresponding to each modality based on the real-time environmental context, performs fusion processing on the multimodal feature data based on the fusion weights, and outputs the final monitoring result.

[0018] In conjunction with the above method, the present invention further provides a storage medium comprising a stored program, wherein, when the program is executed, the device on which the storage medium is located executes the above-described ship safety risk monitoring method.

[0019] In conjunction with the above method, the present invention further provides a computer program product comprising a computer program that, when processed and executed, implements the steps of the above-described ship safety risk monitoring method.

[0020] The present invention acquires multimodal monitoring data of ships, performs preprocessing and feature extraction on each modality's monitoring data using a streaming parallel approach, and then conducts risk assessment through a multi-level progressive detection architecture. If the confidence level of the assessment result output by the current detection level meets preset conditions, the monitoring result is directly output and the execution of subsequent detection levels is terminated. If the preset conditions are not met, the next detection level is entered. The final detection level dynamically generates fusion weights corresponding to each modality based on the real-time environmental context, performs fusion processing on the multimodal feature data, and outputs the final monitoring result. This approach improves the accuracy of risk assessment in complex navigation scenarios and reduces the false alarm rate through a dynamic adaptive fusion mechanism, while significantly shortening the response latency of risk assessment through streaming parallel processing and multi-level early termination. This solves the problem that traditional serial fixed-rule fusion architectures cannot simultaneously achieve high assessment accuracy and low response latency.

[0021] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention.

[0022] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating an embodiment of the ship safety risk monitoring method of the present invention;

[0024] Figure 2 This is a schematic diagram of an embodiment of the ship safety risk monitoring device of the present invention;

[0025] Figure 3 A structural diagram of a ship safety risk monitoring terminal;

[0026] Figure 4 This is a schematic diagram of a multi-source, multi-modal crew abnormal behavior analysis system.

[0027] Referring to the accompanying drawings, the reference numerals in the embodiments of the present invention are as follows:

[0028] 101 - Acquisition unit; 102 - Feature extraction unit; 103 - Monitoring unit. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0030] According to embodiments of the present invention, a method for monitoring ship safety risks is provided, such as... Figure 1 The flowchart of an embodiment of the method of the present invention is shown. The ship safety risk monitoring method may include steps S110 to S150.

[0031] In step S110, multimodal monitoring data of the ship is acquired. The multimodal monitoring data includes: visual modal data, acoustic modal data, ship state modal data, and spatial rule modal data.

[0032] Multimodal monitoring data comprises a collection of ship safety-related data from various information collection channels, exhibiting diverse data representations and semantic dimensions. These data types reflect the ship's safety status from different perspectives, including personnel status, environmental information, equipment status, and scenario rules. Specifically, visual modal data, acquired through imaging equipment, provides visual information about the appearance, posture, and location of personnel and objects within the monitoring area. Acoustic modal data, acquired through sound acquisition equipment, provides audio data to identify speech content, abnormal sounds, and other acoustic features, supplementing information not covered by visual data. Ship status modal data, collected through shipboard sensors and navigation equipment, consists of structured time-series data representing the ship's own operational parameters, reflecting its motion state and equipment operating conditions, providing contextual basis for determining compliance. Spatial rule modal data, containing prior data on waterway geography and navigation regulations, clarifies the spatial attributes and corresponding regulatory requirements of the ship's environment, serving as a rule benchmark for distinguishing scenarios and determining compliance.

[0033] Relying on a single modality is insufficient for comprehensive and accurate risk assessment. By acquiring multi-dimensional heterogeneous data, cross-validation and complementary advantages of different modal information can be achieved, providing a complete data foundation for subsequent accurate risk assessment and effectively supporting low false alarm and highly reliable ship safety risk monitoring.

[0034] In some embodiments, after the data for each modality is collected, an independent feature extraction chain is established: the visual modality extracts fine-grained features such as facial features, human posture, target detection, and area intrusion detection; the acoustic modality extracts voiceprint features, keyword features, and abnormal sound features; the ship status modality extracts motion statistics features, abnormal fluctuation features, and equipment operating condition features; and the spatial rule modality extracts features such as area type, rule level, and fence triggering status. For spatial rule modality data, the coordinate information of all areas can be converted into a spatial hash index during the system initialization phase. This hash mapping enables fast lookup of ship position to area type, with a constant-time complexity. This eliminates the need for repeated complex geometric calculations such as polygon inclusion determination, significantly improving the processing efficiency of spatial determination. The four modalities do not necessarily need to be enabled simultaneously; they can be configured as needed based on the actual monitoring scenario. For example, for open flame monitoring only in the engine room, only the visual modality and the spatial rule modality can be enabled; for navigation watch monitoring, all four modalities can be enabled.

[0035] In some embodiments, the types of modalities can be expanded according to the type of vessel and regulatory requirements. For example, a gas concentration detection modal can be added to dangerous goods vessels, and a passenger flow statistics modal can be added to passenger areas on passenger ships. External data such as meteorological and traffic control instructions issued from shore can also be used as supplementary modalities.

[0036] In step S120, the monitoring data of each modality are preprocessed and feature extracted using a streaming parallel method to obtain the feature data corresponding to each modality.

[0037] Streaming parallel processing is a real-time data processing architecture in which data from each modality is processed independently and synchronously in the form of a continuous data stream, without waiting for data from other modalities to arrive or be processed. The processing continues in a sliding time window as the basic unit.

[0038] Traditional serial processing requires waiting for all modal data to be collected before processing, resulting in latency due to differences in processing speed between different modalities. Adopting a streaming parallel architecture eliminates the waiting overhead between modalities, while the sliding window processing mode adapts to continuous real-time monitoring scenarios, enabling data acquisition and processing simultaneously and ensuring real-time monitoring. Specifically, the raw monitoring data for each modality is input into the processing chain as a continuous data stream, and processing units are divided according to the sliding time window. Each modality independently completes preprocessing operations such as data cleaning and format standardization, and then extracts multi-level features that can be used for risk assessment from the preprocessed data. The processing of each modality proceeds synchronously without waiting for each other.

[0039] In some implementations, step S120 involves preprocessing and feature extraction of the monitoring data for each modality using a streaming parallel approach. This includes: dynamically adjusting the window size of the streaming processing based on the temporal characteristics of the risk type to be monitored corresponding to each modality; the window size is used to limit the length of the time-series data in a single round of processing to match the time scale of the corresponding risk behavior to be monitored; within each streaming window, preprocessing operations are performed independently on the original monitoring data of the corresponding modality, including data cleaning, data standardization, and format normalization; multi-level feature extraction is performed on the preprocessed modality data to obtain the feature data of the corresponding modality; wherein the preprocessing and feature extraction processes of each modality are independent and run asynchronously in parallel.

[0040] In streaming data processing, the window is a defined unit of time, dividing a continuous data stream into data segments of corresponding durations. These segments serve as the basic unit for a single round of data processing, and the window's duration directly determines the amount of data processed in a single round and the overall response latency. In preprocessing, data cleaning corrects and removes invalid values, anomalous jumps, and missing content from the raw data, aiming to remove noise and interference and improve the validity and reliability of the basic data. Data standardization maps data with different dimensions and numerical ranges to a fixed numerical interval, eliminating the impact of data magnitude differences on subsequent feature extraction and judgment, and improving the stability of the algorithm. Format normalization converts raw data with different structures and encoding formats into a unified data structure and encoding format. Multi-level feature extraction is a process of extracting features of different granularities from the raw data layer by layer, from the underlying signal to the higher semantic level. Lower-level features focus on the basic attributes of the original signal, while higher-level features focus on semantic behavioral and state attributes. Multiple processing links run asynchronously and in parallel, each advancing its processing progress independently without waiting for the processing results of other links or for progress alignment. Each link can autonomously advance according to its own data volume and processing complexity, eliminating the time overhead of waiting for each other.

[0041] Specifically, the system pre-calibrates the temporal characteristics of various monitored risks, clarifying the typical duration and observation period corresponding to different risks. During operation, the system automatically matches an appropriate window size for the corresponding modality based on the currently activated monitoring task and risk type. Each window contains continuous time-series data of the corresponding duration, serving as the basic unit for single-round data processing. Each modality corresponds to an independent preprocessing link, performing corresponding preprocessing operations based on the data characteristics of that modality within the data range of a single streaming window. The data cleaning stage removes abnormal jump values, completes missing sampling points, and filters environmental noise; the data standardization stage unifies the raw data of different dimensions into a standard numerical range; the format normalization stage converts the raw data into a unified data structure compatible with subsequent feature extraction modules. Each modality is configured with an independent feature extraction link, extracting features of different granularities from the preprocessed data sequentially from the bottom layer to the top layer, forming a multi-dimensional feature dataset. The preprocessing and feature extraction processes of all modalities proceed independently without waiting for the processing progress of other modalities. After each modality completes feature extraction, the feature data is stored in a shared cache.

[0042] By dynamically adjusting the streaming window size, it can adapt to the risk monitoring needs of different time scales while taking into account both detection accuracy and response speed; through modal-independent preprocessing operations, it can specifically improve the quality of various types of data and adapt to the data characteristics of different modalities; through asynchronous parallel processing mode, it can eliminate waiting overhead between modalities and improve overall processing efficiency.

[0043] In some embodiments, the streaming processing architecture employs an asynchronous, non-blocking fusion mechanism. Subsequent cross-modal fusion calculations do not require waiting for all modalities to output feature data within the same time window; fusion decisions can be performed directly based on the latest available features generated by each modality, further reducing overall latency. For spatial rule-based data, the preprocessing is completed in one go during system initialization. Spatial indexing transforms regional coordinate information into a fast-queryable index structure. During runtime, regional attributes are directly retrieved via location queries, eliminating the need for repeated spatial calculations.

[0044] In step S130, a multi-level progressive detection architecture is used to determine the risk of each modal feature data.

[0045] In step S140, if the confidence level of the judgment result output by the current detection level meets the preset condition, the monitoring result is directly output and the execution of subsequent detection levels is terminated.

[0046] In step S150, if the confidence level of the judgment result output by the current detection level does not meet the preset condition, then proceed to the next detection level.

[0047] The multi-level progressive detection architecture is a hierarchical risk assessment architecture. All levels are arranged sequentially according to computational resource consumption from low to high and judgment granularity from coarse to fine. The detection process progresses progressively along each level. When the judgment result of a certain level meets the confidence criteria, the calculation of all subsequent levels can be terminated early. Confidence level is a numerical indicator used to quantify the reliability of the risk assessment result; a higher value indicates stronger reliability.

[0048] In real-world monitoring scenarios, most common scenarios are characterized by clear features and simple judgments, requiring no complex full-scale computation. Traditional full-scale computation performs complete deep analysis on all scenarios, resulting in significant wasted computing power and prolonged overall response time. By employing a multi-level, progressive, and early-termination mechanism, most simple scenarios can be quickly judged at a low-computing-power level, with only a few ambiguous scenarios requiring deep-level computation. This significantly reduces average processing latency while maintaining judgment accuracy. Specifically, multiple detection levels are set up in order of increasing computational cost and coarse-to-fine granularity. Feature data first enters the first detection level for rapid judgment, outputting the judgment result and corresponding confidence score. If the confidence score meets the preset credibility condition, the monitoring result for that level is directly output, and subsequent detection levels are not executed. If the confidence score does not meet the condition, the feature data is passed to the next detection level for more refined judgment, repeating the confidence score judgment and progressive level logic. For example, when a ship is in a legal berthing area and its main engine is completely shut down, a conclusion that "crew members leaving their posts to rest is not a violation" can be reached with high confidence simply by using lightweight rules to determine the ship's status and position. In this case, the normal result can be output directly without having to call complex calculation modules such as visual analysis and voice analysis.

[0049] The final detection layer dynamically generates fusion weights for each modality based on the real-time environmental context. After fusing the multimodal feature data based on the fusion weights, the final monitoring result is output.

[0050] Specifically, when feature data enters the final detection layer, the current environmental and scene parameters are first collected to generate environmental context information. Then, based on this context information, the fusion weight corresponding to each modality is calculated. The feature data of each modality are weighted and combined according to the weights, and cross-modal information interaction is performed to generate a unified fusion feature. Finally, risk assessment is completed based on the fusion feature, and the final monitoring result is output. Here, real-time environmental context refers to environmental and scene parameters that affect the quality of data collection and the effectiveness of assessment for each modality at the current moment, covering multiple dimensions such as environmental conditions, ship operating status, and waterway regulatory level. The fusion weight is used to measure the contribution ratio of different modal feature data in the final risk assessment; a higher weight indicates greater reference value for the corresponding modality's assessment.

[0051] This invention effectively shortens the risk assessment delay in conventional scenarios and improves the monitoring response speed by using a streaming parallel processing architecture and a multi-level progressive and early termination detection mechanism. At the same time, by using a dynamic weight fusion mechanism based on real-time environmental context, it solves the problems of poor adaptability of fixed fusion strategies and high false alarm rate in complex scenarios, achieving a dual improvement in detection accuracy and response speed, and can adapt to the complex and ever-changing environment and scenario requirements during ship navigation.

[0052] In some implementations, the multi-level progressive detection architecture includes a first detection level, a second detection level, and a third detection level executed sequentially; the first detection level is used to perform initial screening of each modal feature data based on lightweight rules; the second detection level is used to perform single-modal independent analysis and risk assessment on feature data whose confidence level does not meet preset conditions after being determined by the first detection level; and the third detection level is used to perform fusion processing of the modal feature data.

[0053] The first detection layer is the first decision layer in the multi-level progressive detection architecture. It achieves the fastest initial scene screening with the lowest computational cost, relying solely on extremely simple rules and basic features to complete the decision. It is used to quickly filter common scenarios with clear features and simple decision-making. Lightweight rules are a decision-making method that quickly verifies basic features based on preset simple decision rules and low-complexity computational logic. It does not require complex model calculations, has extremely low computational overhead, and the fastest decision-making speed.

[0054] In real-world ship monitoring scenarios, most scenarios have clear characteristics and simple judgment logic, eliminating the need for complex analysis models. Using lightweight rules as the first detection layer allows for rapid filtering of numerous common scenarios with minimal computational overhead, avoiding ineffective deep calculations and significantly reducing average judgment time and system computational load. Specifically, the system inputs the basic features of all modalities into the lightweight rule engine, performs rapid verification according to preset basic judgment rules, covering three core rule categories: ship operational status verification, water area attribute verification, and basic motion feature verification. After verification, it outputs preliminary judgment results and corresponding confidence scores. If the confidence score meets preset conditions, it directly outputs the monitoring result, terminating the execution of subsequent layers. For example, when a ship is in a legal anchorage area and its speed is zero, lightweight rule verification based solely on ship status features and spatial rule features can determine with high confidence that "crew member on duty status abnormality is not established," directly outputting a normal result without calling visual analysis, voice analysis, or other models. When the frame difference method detects no moving targets and the ship is sailing, it can quickly screen out personnel absence risks, directly triggering subsequent processes.

[0055] The second detection layer is an intermediate decision layer in the multi-level progressive detection architecture. For scenarios where the initial screening cannot provide a clear determination, it performs medium-granularity risk assessment using a single-modal independent analysis approach. Its computational cost is higher than the first layer but lower than the deep fusion layer. Single-modal independent analysis refers to the process of using a lightweight analysis model to independently complete risk identification and assessment for each modality of data, relying solely on the feature output of the single modality itself, without requiring cross-modal information interaction.

[0056] While some scenarios cannot be directly determined using basic rules, sufficiently reliable judgment results can be obtained based on the features of a single modality, eliminating the need for high-computational operations such as multimodal fusion. Setting up an intermediate level for independent single-modal analysis allows for further filtering of moderately complex scenarios beyond the initial screening. This reduces the proportion of scenarios entering the highest computational level while maintaining judgment accuracy, achieving a balance between accuracy and efficiency. Specifically, for scenarios where the judgment confidence level of the first detection level does not meet preset conditions, the system activates lightweight analysis models for each modality, performing independent risk identification and judgment on the feature data of the corresponding modality. Each modality outputs its own judgment result and confidence level. If the judgment confidence level of any modality or multimodal combination meets preset conditions, the monitoring result is directly output, terminating the execution of subsequent levels. For example, when the view from the cockpit is clear and the lighting is sufficient, the visual modality can identify open flame features through a lightweight target detection model and determine that the confidence level is high. At this time, the open flame alarm result can be directly output without the need to combine it with acoustic, ship status and other modalities for fusion analysis. When the acoustic modality clearly captures distress keywords and there is no obvious environmental noise, it can be directly determined that there is an emergency abnormality of personnel and trigger the alarm process.

[0057] The third detection layer is the final decision layer in the multi-level progressive detection architecture. It is only used for ambiguous scenarios that cannot be determined with high confidence by the first two levels. It conducts in-depth analysis through multimodal feature fusion, which has the highest computational cost but the best decision accuracy.

[0058] In some complex scenarios, interference factors such as insufficient lighting, high environmental noise, and unclear features exist, making reliable judgments impossible using only single-modal or simple rules. Cross-validation and information complementarity of multimodal data are essential to ensure accuracy. By setting deep fusion as the final level, high-power computation is only used for a very small percentage of ambiguous scenarios. This ensures accuracy in complex scenarios without significantly increasing the overall computational cost and average latency of the system. Specifically, for ambiguous scenarios where the first two detection levels fail to reach the preset confidence level, the system activates the third detection level. This level integrates fine-grained feature data from all modalities, dynamically generates fusion weights based on real-time environmental context, completes cross-validation and deep fusion of multimodal features, performs the final risk assessment based on the fused features, and outputs the monitoring results. For example, during nighttime navigation, the lighting in the bridge is extremely low, and the confidence level of visual modal recognition of crew members with their eyes closed is insufficient. At the same time, the presence of equipment operating noise in the environment reduces the reliability of acoustic modal judgment. In this case, the third detection level is entered, which combines the ship's navigation status and the contextual information of the water area to dynamically adjust the weights of each modality. Through multi-feature cross-validation, "fatigue and drowsiness" are distinguished from "normal observation of equipment," and an accurate judgment result is output.

[0059] The confidence thresholds for the three detection levels are adjusted progressively, with the highest threshold set for the first detection level to ensure high reliability of the initial screening results and prevent premature termination of the process due to misjudgments in simple scenarios. The confidence thresholds for the intermediate detection levels are slightly lower than those for the first level. The final detection level outputs the final judgment result, eliminating the need for further threshold adjustments. In actual operation, most common scenarios can be judged within the first two levels, with only a small number of ambiguous scenarios reaching the third level. The overall computational power consumption is far lower than that of a solution that performs full-scale computation across all scenarios.

[0060] Each detection level corresponds one-to-one with the multi-level feature extraction results from the preceding layers. The first detection level directly calls the underlying basic features, the second detection level calls the mid-level single-modal semantic features, and the third detection level calls the high-level fine-grained features, avoiding repeated feature extraction at different levels and further reducing redundant computation. For different monitoring areas and risk types, the system can be configured with differentiated level activation strategies. For example, in high-risk scenarios such as open flames or people falling into water, the confidence threshold of the second level can be appropriately lowered, allowing more scenarios to quickly enter deep judgment and ensuring the accuracy of high-risk identification.

[0061] In some implementations, the fusion weights corresponding to each modality are dynamically generated based on the real-time environmental context, including: encoding the real-time environmental context information to obtain a context feature vector; inputting the context feature vector into a gating network to output the initial weight parameters corresponding to each modality; and normalizing the initial weight parameters to obtain the final fusion weights of each modality.

[0062] Specifically, the system collects various environmental and scene parameters that affect the quality of modality data and the validity of judgments in real time. All parameters are input into a lightweight encoding network, which, through feature mapping and dimensionality transformation, outputs a fixed-dimensional context feature vector. This vector fully carries the current environmental state information. The encoded context feature vector is then input into a pre-trained gating network. The gating network, through its internal weight mapping logic, calculates the initial weight parameters for each modality. The magnitude of these initial weight parameters corresponds to the reliability of the judgment for that modality in the current environment. The initial weight parameters of all modalities output by the gating network are converted into proportional weights using a normalization algorithm, ensuring that the sum of the fusion weights for all modalities is a fixed value. Finally, fusion weights for each modality that can be directly used for feature fusion calculations are obtained.

[0063] By using environmental context encoding, gating network weight generation, and normalization processing, the multimodal fusion weights are dynamically and adaptively adjusted according to the real-time environment and scenario. This replaces the traditional fixed-weight fusion strategy and can proactively adapt to changes in lighting, noise, and scene conditions during ship navigation. It automatically reduces the contribution of the interference mode to the judgment and increases the role of the reliable mode. This mechanism reduces the false alarm rate in complex environments and improves the accuracy and environmental adaptability of multimodal fusion judgment.

[0064] In some embodiments, the gated network employs a lightweight single-hidden-layer feedforward network structure, resulting in a small number of model parameters, fast computation speed, and efficient operation on shipboard edge devices without significantly increasing decision latency. Environmental context information covers the following core dimensions: first, illumination conditions, used to evaluate the data acquisition quality of visual modalities; second, environmental noise levels, used to evaluate the recognition reliability of acoustic modalities; third, ship motion state, used to evaluate the discriminative power of motion characteristics in ship state modalities; fourth, water area type, used to adjust the balance between overall alarm sensitivity and modal weights; and fifth, sensor signal quality, used to evaluate the data reliability of each sensor type modality.

[0065] The dynamic adjustment of weights follows a clear adaptation rule: under low light conditions, the weight of the visual modality automatically decreases, while the weight of the ship state and spatial rule modality increases; under high noise conditions, the weight of the acoustic modality automatically decreases, while the weight of the visual and ship state modalities increases; under stationary conditions, the weight of the motion characteristics of the ship state modality decreases, while the weight of the visual and spatial rule modality increases; and the weight differences of each modality within key monitored waters are reduced, strengthening the rigor of multimodal cross-validation.

[0066] In some implementations, the fusion processing of multimodal feature data based on the fusion weights includes: weighting each modal feature data with its corresponding fusion weight; performing cross-modal attention interaction on the weighted multimodal features to generate a fusion feature vector; and determining security risks based on the fusion feature vectors.

[0067] Specifically, for the feature data of each modality, the feature values ​​in the feature vector are correlated with the corresponding fusion weights of that modality to obtain weighted modal features. Modal features with higher weights contribute more to subsequent fusion, while those with lower weights have their contributions suppressed. The weighted features from all modalities are concatenated and input into the cross-attention calculation module. This module assigns attention weights to the features of each modality, enabling them to automatically focus on semantically relevant features from other modalities, strengthening the association of complementary information between modalities and weakening the interference of irrelevant noise. After multiple rounds of attention interaction, a unified fusion feature vector is output. The generated fusion feature vector is input into a pre-trained risk assessment model or rule assessment module, outputting the final risk category, risk level, and corresponding confidence level as the final monitoring result. Cross-modal attention interaction is used to enable features from different modalities to automatically capture semantic associations and correspondences, strengthening complementary information, weakening redundant interference information, and achieving deep information complementarity between modalities.

[0068] Through a deep fusion mechanism of feature weighting and cross-modal attention interaction, the fusion weights are dynamically adapted to the environment, and the semantic association and information complementarity value between multimodal features are fully explored. Compared with traditional fixed-rule decision-level fusion, it can significantly improve the accuracy of risk judgment in complex and ambiguous scenarios, reduce the false alarm rate caused by environmental interference, and maintain the flexibility and adaptability of fusion computation.

[0069] In some embodiments, the focusing logic of cross-modal attention differs for different types of security risks. For risks related to personnel physiological states, the attention mechanism focuses on associating visual behavioral features with ship motion features; for risks related to emergency events, the attention mechanism focuses on associating visual posture features with acoustic anomaly features; and for risks related to compliance operations, the attention mechanism focuses on associating spatial region features, speech semantic features, and visual action features. This fusion method is a feature-level fusion, which, compared to the decision-level fusion of traditional solutions where "each modality outputs results independently and then the results are combined and judged by a rule engine," can retain more fine-grained feature information, reduce information loss in layer-by-layer judgment, and achieve better judgment accuracy. In actual operation, only the final detection layer performs complete cross-attention deep fusion; the preceding detection layers do not involve this computationally intensive operation, ensuring overall operational efficiency. In addition, a temporal smoothing mechanism can be added during the fusion process, combining the fusion features of the current window and historical windows for temporal-dimensional association judgment, avoiding single misjudgments caused by instantaneous interference, and further improving the stability of the results.

[0070] In some implementations, during the risk assessment of personnel status anomalies, the assessment results are filtered by combining the ship's motion state with the type of water area it is in; when the ship is in a preset stationary state and located in a preset legal mooring area, the monitoring results of the corresponding personnel status anomalies are blocked.

[0071] Abnormalities in personnel status refer to abnormal behaviors directly related to crew members' on-duty status and physiological condition. These mainly include fatigue, inattention, unauthorized absence from duty, and sleeping on duty. These are the most common monitoring targets in ship navigation safety supervision, and their compliance determination is strongly correlated with the ship's operational scenario. Ship motion status characterizes the ship's current operating condition, determined by a combination of parameters such as speed, main engine operating status, and hull acceleration. It is typically categorized into navigation status and stationary status. Water area types are classifications of water area functional attributes based on electronic navigation charts or electronic nautical charts, including channels, anchorages, berthing areas, bridge areas, prohibited anchorage areas, and traffic control reporting areas. Different types of water areas correspond to differentiated navigation supervision rules and personnel duty requirements. Pre-defined legal berthing areas are water areas marked in the spatial rule system that allow ships to berth and where personnel duty requirements can be relaxed. These typically include legal anchorages, port berthing areas, and dock operation berths that comply with maritime regulatory rules.

[0072] Specifically, after the system completes the preliminary judgment of personnel status anomalies, it does not directly output the final result. Instead, it simultaneously retrieves the current vessel's motion status parameters and the area type of the current waterway obtained through spatial matching. These two scenario contexts are used as verification conditions to perform compliance checks on the preliminary judgment results. Anomalies that comply with navigation scenario regulatory rules are retained and output normally, while results that do not conform to scenario logic enter the subsequent shielding process. The system is set up with a shielding rule triggered by two conditions. The shielding operation is only performed on personnel status anomaly judgment results when both conditions—"the vessel is in a preset stationary state" and "the vessel is located in a preset legal mooring area"—are met simultaneously. If either condition is not met, the shielding mechanism is not triggered, and the anomaly monitoring results are output according to the normal process.

[0073] By employing a dual-condition scenario filtering mechanism based on vessel motion status and water area type, the system effectively eliminates scenario ambiguity in determining personnel status anomalies. It distinguishes between navigation violations and normal berthing behavior from the perspective of judgment logic, significantly reducing invalid false alarms in legal berthing scenarios and improving the practicality and reliability of the monitoring system. At the same time, the dual-condition joint verification design avoids the risk of anomaly underreporting, ensuring the rigor of navigation safety supervision while reducing false alarms, and is adaptable to the differentiated supervision needs of different vessel navigation scenarios.

[0074] In some embodiments, personnel status anomalies specifically include anomalies directly related to personnel duty status, such as fatigued driving, inattention, unauthorized absence from duty, sleeping on duty, and failure to wear protective equipment as required. For high-risk anomalies unrelated to the parking scenario, such as open flames, sudden falls, and violations of regulations, this filtering and shielding mechanism is not applied, and normal alarm output is always maintained to avoid missing high-risk events.

[0075] The preset stationary state is not determined solely by zero speed, but rather by a comprehensive assessment using multiple parameters such as speed threshold, main engine speed, and hull acceleration. This avoids misjudging scenarios like ship drifting with currents or slow mooring as stationary, ensuring the accuracy of shielding triggers. Pre-defined legal mooring areas are pre-marked using geofencing, and area boundaries and regulatory rules can be dynamically updated according to maritime management requirements. Area determination can be quickly queried using spatial indexing technology, eliminating the need for repeated complex geometric calculations and preventing additional determination delays.

[0076] In addition to filtering out berthing scenarios, the system also supports a reverse scenario enhancement mechanism: when a vessel enters key monitored waters such as bridge areas, narrow waterways, and VTS traffic control areas, the system will automatically increase the sensitivity of detecting abnormal personnel status, lower the judgment threshold, strengthen the monitoring and supervision of key waters, and adapt to the safety management needs of high-risk navigation segments.

[0077] The blocked abnormal results are not directly discarded, but stored in the form of ordinary operation logs on the ship's local terminal, supporting subsequent backtracking queries and statistical analysis. This avoids invalid alarms interfering with regulatory personnel, while retaining complete behavioral data for ship safety management and behavior analysis.

[0078] Figure 3This is a structural diagram of a ship safety risk monitoring terminal. As the core computing and alarm carrier on the ship, this terminal interfaces with four types of multimodal data input sources and two types of alarm output peripherals. Internally, it achieves a complete data access, local analysis, alarm triggering, data storage, and remote transmission function through five functional modules. Sensor data, map data, video streams, and audio streams are uniformly accessed through an interface module. This interface module performs protocol conversion and data aggregation before forwarding to the core computing module. The computing module executes all risk monitoring algorithms, including streaming preprocessing, multi-level progressive detection, and dynamic weight fusion, and outputs risk assessment results and control commands. The storage module is connected to the computing module. The ship is responsible for local storage of raw multimodal data, algorithm models, judgment logs, and alarm records, supporting historical backtracking queries. The alarm module receives alarm commands from the computing module and drives external alarm lights and speakers to provide on-site audio-visual alerts and voice broadcasts, enabling local early warnings as soon as risks occur. The network transmission module carries ship-to-shore communication functions, supporting multiple network standards to transmit alarm information, equipment status, and key evidence data back to the shore-based monitoring platform. It also receives configuration update commands issued by the shore-based platform. The entire architecture adopts an edge local computing mode, with all core analysis completed on the ship, reducing dependence on ship-to-shore network bandwidth and ensuring real-time response capabilities for risk alarms.

[0079] Figure 4This is a schematic diagram of a multi-source, multi-modal crew abnormal behavior analysis system. The system is divided into a multi-modal perception layer, a terminal edge processing layer, and an application and alarm layer. All core calculations are deployed on the shipborne edge terminal, balancing real-time monitoring and judgment accuracy. The multi-modal perception layer provides the system with raw data input. Data sources include the ship's real-time positioning, speed, and heading data; the ship's real-time water depth data; AIS dynamic data of surrounding vessels; electronic nautical chart data covering areas such as bridges, wharves, anchorages, berths, and reefs; and voice and video data from the bridge crew. The above data is categorized by modality into four types: real-time sensor data, map data, voice stream data, and video stream data, and is uniformly transmitted to the terminal edge processing layer. The terminal edge processing layer is the core computing unit of the system. Raw data first enters the streaming preprocessing engine, where parallel preprocessing and cleaning of data from various modalities are completed using a sliding window approach. The preprocessed data is then sent to a cascaded detector for hierarchical judgment. For simple scenarios with clear features, judgment results can be quickly output and audible / visual alarms triggered directly through first- and second-level detection without requiring deep computation. Only for complex scenarios where the judgment is questionable will the third-level processing flow be initiated. This involves calling sensor feature extractors, map feature extractors, voice stream feature extractors, and video stream feature extractors to complete fine-grained feature extraction. Then, the context-gated fusion module dynamically allocates the weights of each modality based on the real-time environment, completing multimodal deep fusion judgment and outputting fused alarm information, which then triggers audible / visual alarms. The topmost application and alarm layer is responsible for alarm delivery and management. The audible / visual alarm output is divided into two paths: one pushes on-site alarm information to the crew for real-time local intervention on the ship; the other synchronizes remote alarm information to the crew abnormal behavior management system via a network transmission module, pushing alarm information to shore-based backend users to support remote monitoring, verification, and statistical analysis.

[0080] The technical solution adopted in this embodiment first collects multimodal monitoring data of the ship, completes the preprocessing and feature extraction of each modality data in a streaming parallel mode, and then conducts risk assessment based on a multi-level progressive detection architecture. If the confidence level of the current detection level assessment result meets the standard, the result is directly output and the subsequent detection is terminated. If the standard is not met, the next level is entered. The final level dynamically generates the fusion weight of each modality in combination with the real-time environmental context, and outputs the final monitoring result after completing the fusion of multimodal features. It can not only improve the risk identification accuracy and reduce the false alarm rate in complex navigation scenarios through dynamic adaptive fusion, but also significantly compress the risk assessment response time by means of streaming parallel processing and hierarchical early termination mechanism.

[0081] According to an embodiment of the present invention, a ship safety risk monitoring device corresponding to a ship safety risk monitoring method is also provided. See also Figure 2 The diagram shows a structural schematic of an embodiment of the device of the present invention. The ship safety risk monitoring device may include: an acquisition unit 101, a feature extraction unit 102, and a monitoring unit 103.

[0082] The acquisition unit 101 is configured to acquire multimodal monitoring data of the ship. The multimodal monitoring data includes: visual modal data, acoustic modal data, ship state modal data, and spatial rule modal data.

[0083] Multimodal monitoring data comprises a collection of ship safety-related data from various information collection channels, exhibiting diverse data representations and semantic dimensions. These data types reflect the ship's safety status from different perspectives, including personnel status, environmental information, equipment status, and scenario rules. Specifically, visual modal data, acquired through imaging equipment, provides visual information about the appearance, posture, and location of personnel and objects within the monitoring area. Acoustic modal data, acquired through sound acquisition equipment, provides audio data to identify speech content, abnormal sounds, and other acoustic features, supplementing information not covered by visual data. Ship status modal data, collected through shipboard sensors and navigation equipment, consists of structured time-series data representing the ship's own operational parameters, reflecting its motion state and equipment operating conditions, providing contextual basis for determining compliance. Spatial rule modal data, containing prior data on waterway geography and navigation regulations, clarifies the spatial attributes and corresponding regulatory requirements of the ship's environment, serving as a rule benchmark for distinguishing scenarios and determining compliance.

[0084] Relying on a single modality is insufficient for comprehensive and accurate risk assessment. By acquiring multi-dimensional heterogeneous data, cross-validation and complementary advantages of different modal information can be achieved, providing a complete data foundation for subsequent accurate risk assessment and effectively supporting low false alarm and highly reliable ship safety risk monitoring.

[0085] The feature extraction unit 102 is configured to preprocess and extract features from each modality monitoring data in a streaming parallel manner to obtain feature data corresponding to each modality.

[0086] Streaming parallel processing is a real-time data processing architecture in which data from each modality is processed independently and synchronously in the form of a continuous data stream, without waiting for data from other modalities to arrive or be processed. The processing continues in a sliding time window as the basic unit.

[0087] Traditional serial processing requires waiting for all modal data to be collected before processing, resulting in latency due to differences in processing speed between different modalities. Adopting a streaming parallel architecture eliminates the waiting overhead between modalities, while the sliding window processing mode adapts to continuous real-time monitoring scenarios, enabling data acquisition and processing simultaneously and ensuring real-time monitoring. Specifically, the raw monitoring data for each modality is input into the processing chain as a continuous data stream, and processing units are divided according to the sliding time window. Each modality independently completes preprocessing operations such as data cleaning and format standardization, and then extracts multi-level features that can be used for risk assessment from the preprocessed data. The processing of each modality proceeds synchronously without waiting for each other.

[0088] In some implementations, the feature extraction unit 102 performs preprocessing and feature extraction on the monitoring data of each modality using a streaming parallel approach, including: dynamically adjusting the window size of the streaming processing based on the temporal characteristics of the risk type to be monitored corresponding to each modality; the window size is used to limit the length of the time-series data in a single round of processing to match the time scale of the corresponding risk behavior to be monitored; within each streaming window, preprocessing operations are performed independently on the original monitoring data of the corresponding modality, the preprocessing operations including data cleaning, data standardization and format normalization; multi-level feature extraction is performed on the preprocessed modality data to obtain the feature data of the corresponding modality; wherein, the preprocessing and feature extraction processes of each modality are independent and run asynchronously in parallel.

[0089] In streaming data processing, the window is a defined unit of time, dividing a continuous data stream into data segments of corresponding durations. These segments serve as the basic unit for a single round of data processing, and the window's duration directly determines the amount of data processed in a single round and the overall response latency. In preprocessing, data cleaning corrects and removes invalid values, anomalous jumps, and missing content from the raw data, aiming to remove noise and interference and improve the validity and reliability of the basic data. Data standardization maps data with different dimensions and numerical ranges to a fixed numerical interval, eliminating the impact of data magnitude differences on subsequent feature extraction and judgment, and improving the stability of the algorithm. Format normalization converts raw data with different structures and encoding formats into a unified data structure and encoding format. Multi-level feature extraction is a process of extracting features of different granularities from the raw data layer by layer, from the underlying signal to the higher semantic level. Lower-level features focus on the basic attributes of the original signal, while higher-level features focus on semantic behavioral and state attributes. Multiple processing links run asynchronously and in parallel, each advancing its processing progress independently without waiting for the processing results of other links or for progress alignment. Each link can autonomously advance according to its own data volume and processing complexity, eliminating the time overhead of waiting for each other.

[0090] Specifically, the system pre-calibrates the temporal characteristics of various monitored risks, clarifying the typical duration and observation period corresponding to different risks. During operation, the system automatically matches an appropriate window size for the corresponding modality based on the currently activated monitoring task and risk type. Each window contains continuous time-series data of the corresponding duration, serving as the basic unit for single-round data processing. Each modality corresponds to an independent preprocessing link, performing corresponding preprocessing operations based on the data characteristics of that modality within the data range of a single streaming window. The data cleaning stage removes abnormal jump values, completes missing sampling points, and filters environmental noise; the data standardization stage unifies the raw data of different dimensions into a standard numerical range; the format normalization stage converts the raw data into a unified data structure compatible with subsequent feature extraction modules. Each modality is configured with an independent feature extraction link, extracting features of different granularities from the preprocessed data sequentially from the bottom layer to the top layer, forming a multi-dimensional feature dataset. The preprocessing and feature extraction processes of all modalities proceed independently without waiting for the processing progress of other modalities. After each modality completes feature extraction, the feature data is stored in a shared cache.

[0091] By dynamically adjusting the streaming window size, it can adapt to the risk monitoring needs of different time scales while taking into account both detection accuracy and response speed; through modal-independent preprocessing operations, it can specifically improve the quality of various types of data and adapt to the data characteristics of different modalities; through asynchronous parallel processing mode, it can eliminate waiting overhead between modalities and improve overall processing efficiency.

[0092] The monitoring unit 103 is configured to use a multi-level progressive detection architecture to determine the risk of each modal feature data.

[0093] The monitoring unit 103 is further configured to directly output the monitoring result and terminate the execution of subsequent detection levels if the confidence level of the judgment result output by the current detection level meets the preset conditions.

[0094] The monitoring unit 103 is also configured to proceed to the next detection level if the confidence level of the judgment result output by the current detection level does not meet the preset conditions.

[0095] The multi-level progressive detection architecture is a hierarchical risk assessment architecture. All levels are arranged sequentially according to computational resource consumption from low to high and judgment granularity from coarse to fine. The detection process progresses progressively along each level. When the judgment result of a certain level meets the confidence criteria, the calculation of all subsequent levels can be terminated early. Confidence level is a numerical indicator used to quantify the reliability of the risk assessment result; a higher value indicates stronger reliability.

[0096] In real-world monitoring scenarios, most common scenarios are characterized by clear features and simple judgments, requiring no complex full-scale computation. Traditional full-scale computation performs complete deep analysis on all scenarios, resulting in significant wasted computing power and prolonged overall response time. By employing a multi-level, progressive, and early-termination mechanism, most simple scenarios can be quickly judged at a low-computing-power level, with only a few ambiguous scenarios requiring deep-level computation. This significantly reduces average processing latency while maintaining judgment accuracy. Specifically, multiple detection levels are set up in order of increasing computational cost and coarse-to-fine granularity. Feature data first enters the first detection level for rapid judgment, outputting the judgment result and corresponding confidence score. If the confidence score meets the preset credibility condition, the monitoring result for that level is directly output, and subsequent detection levels are not executed. If the confidence score does not meet the condition, the feature data is passed to the next detection level for more refined judgment, repeating the confidence score judgment and progressive level logic. For example, when a ship is in a legal berthing area and its main engine is completely shut down, a conclusion that "crew members leaving their posts to rest is not a violation" can be reached with high confidence simply by using lightweight rules to determine the ship's status and position. In this case, the normal result can be output directly without having to call complex calculation modules such as visual analysis and voice analysis.

[0097] The monitoring unit 103 is further configured to: dynamically generate fusion weights corresponding to each modality based on the real-time environmental context at the final detection level; perform fusion processing on the multimodal feature data based on the fusion weights; and output the final monitoring result.

[0098] Specifically, when feature data enters the final detection layer, the current environmental and scene parameters are first collected to generate environmental context information. Then, based on this context information, the fusion weight corresponding to each modality is calculated. The feature data of each modality are weighted and combined according to the weights, and cross-modal information interaction is performed to generate a unified fusion feature. Finally, risk assessment is completed based on the fusion feature, and the final monitoring result is output. Here, real-time environmental context refers to environmental and scene parameters that affect the quality of data collection and the effectiveness of assessment for each modality at the current moment, covering multiple dimensions such as environmental conditions, ship operating status, and waterway regulatory level. The fusion weight is used to measure the contribution ratio of different modal feature data in the final risk assessment; a higher weight indicates greater reference value for the corresponding modality's assessment.

[0099] This invention effectively shortens the risk assessment delay in conventional scenarios and improves the monitoring response speed by using a streaming parallel processing architecture and a multi-level progressive and early termination detection mechanism. At the same time, by using a dynamic weight fusion mechanism based on real-time environmental context, it solves the problems of poor adaptability of fixed fusion strategies and high false alarm rate in complex scenarios, achieving a dual improvement in detection accuracy and response speed, and can adapt to the complex and ever-changing environment and scenario requirements during ship navigation.

[0100] In some implementations, the multi-level progressive detection architecture includes a first detection level, a second detection level, and a third detection level executed sequentially; the first detection level is used to perform initial screening of each modal feature data based on lightweight rules; the second detection level is used to perform single-modal independent analysis and risk assessment on feature data whose confidence level does not meet preset conditions after being determined by the first detection level; and the third detection level is used to perform fusion processing of the modal feature data.

[0101] The first detection layer is the first decision layer in the multi-level progressive detection architecture. It achieves the fastest initial scene screening with the lowest computational cost, relying solely on extremely simple rules and basic features to complete the decision. It is used to quickly filter common scenarios with clear features and simple decision-making. Lightweight rules are a decision-making method that quickly verifies basic features based on preset simple decision rules and low-complexity computational logic. It does not require complex model calculations, has extremely low computational overhead, and the fastest decision-making speed.

[0102] In real-world ship monitoring scenarios, most scenarios have clear characteristics and simple judgment logic, eliminating the need for complex analysis models. Using lightweight rules as the first detection layer allows for rapid filtering of numerous common scenarios with minimal computational overhead, avoiding ineffective deep calculations and significantly reducing average judgment time and system computational load. Specifically, the system inputs the basic features of all modalities into the lightweight rule engine, performs rapid verification according to preset basic judgment rules, covering three core rule categories: ship operational status verification, water area attribute verification, and basic motion feature verification. After verification, it outputs preliminary judgment results and corresponding confidence scores. If the confidence score meets preset conditions, it directly outputs the monitoring result, terminating the execution of subsequent layers. For example, when a ship is in a legal anchorage area and its speed is zero, lightweight rule verification based solely on ship status features and spatial rule features can determine with high confidence that "crew member on duty status abnormality is not established," directly outputting a normal result without calling visual analysis, voice analysis, or other models. When the frame difference method detects no moving targets and the ship is sailing, it can quickly screen out personnel absence risks, directly triggering subsequent processes.

[0103] The second detection layer is an intermediate decision layer in the multi-level progressive detection architecture. For scenarios where the initial screening cannot provide a clear determination, it performs medium-granularity risk assessment using a single-modal independent analysis approach. Its computational cost is higher than the first layer but lower than the deep fusion layer. Single-modal independent analysis refers to the process of using a lightweight analysis model to independently complete risk identification and assessment for each modality of data, relying solely on the feature output of the single modality itself, without requiring cross-modal information interaction.

[0104] While some scenarios cannot be directly determined using basic rules, sufficiently reliable judgment results can be obtained based on the features of a single modality, eliminating the need for high-computational operations such as multimodal fusion. Setting up an intermediate level for independent single-modal analysis allows for further filtering of moderately complex scenarios beyond the initial screening. This reduces the proportion of scenarios entering the highest computational level while maintaining judgment accuracy, achieving a balance between accuracy and efficiency. Specifically, for scenarios where the judgment confidence level of the first detection level does not meet preset conditions, the system activates lightweight analysis models for each modality, performing independent risk identification and judgment on the feature data of the corresponding modality. Each modality outputs its own judgment result and confidence level. If the judgment confidence level of any modality or multimodal combination meets preset conditions, the monitoring result is directly output, terminating the execution of subsequent levels. For example, when the view from the cockpit is clear and the lighting is sufficient, the visual modality can identify open flame features through a lightweight target detection model and determine that the confidence level is high. At this time, the open flame alarm result can be directly output without the need to combine it with acoustic, ship status and other modalities for fusion analysis. When the acoustic modality clearly captures distress keywords and there is no obvious environmental noise, it can be directly determined that there is an emergency abnormality of personnel and trigger the alarm process.

[0105] The third detection layer is the final decision layer in the multi-level progressive detection architecture. It is only used for ambiguous scenarios that cannot be determined with high confidence by the first two levels. It conducts in-depth analysis through multimodal feature fusion, which has the highest computational cost but the best decision accuracy.

[0106] In some complex scenarios, interference factors such as insufficient lighting, high environmental noise, and unclear features exist, making reliable judgments impossible using only single-modal or simple rules. Cross-validation and information complementarity of multimodal data are essential to ensure accuracy. By setting deep fusion as the final level, high-power computation is only used for a very small percentage of ambiguous scenarios. This ensures accuracy in complex scenarios without significantly increasing the overall computational cost and average latency of the system. Specifically, for ambiguous scenarios where the first two detection levels fail to reach the preset confidence level, the system activates the third detection level. This level integrates fine-grained feature data from all modalities, dynamically generates fusion weights based on real-time environmental context, completes cross-validation and deep fusion of multimodal features, performs the final risk assessment based on the fused features, and outputs the monitoring results. For example, during nighttime navigation, the lighting in the bridge is extremely low, and the confidence level of visual modal recognition of crew members with their eyes closed is insufficient. At the same time, the presence of equipment operating noise in the environment reduces the reliability of acoustic modal judgment. In this case, the third detection level is entered, which combines the ship's navigation status and the contextual information of the water area to dynamically adjust the weights of each modality. Through multi-feature cross-validation, "fatigue and drowsiness" are distinguished from "normal observation of equipment," and an accurate judgment result is output.

[0107] The confidence thresholds for the three detection levels are adjusted progressively, with the highest threshold set for the first detection level to ensure high reliability of the initial screening results and prevent premature termination of the process due to misjudgments in simple scenarios. The confidence thresholds for the intermediate detection levels are slightly lower than those for the first level. The final detection level outputs the final judgment result, eliminating the need for further threshold adjustments. In actual operation, most common scenarios can be judged within the first two levels, with only a small number of ambiguous scenarios reaching the third level. The overall computational power consumption is far lower than that of a solution that performs full-scale computation across all scenarios.

[0108] Each detection level corresponds one-to-one with the multi-level feature extraction results from the preceding layers. The first detection level directly calls the underlying basic features, the second detection level calls the mid-level single-modal semantic features, and the third detection level calls the high-level fine-grained features, avoiding repeated feature extraction at different levels and further reducing redundant computation. For different monitoring areas and risk types, the system can be configured with differentiated level activation strategies. For example, in high-risk scenarios such as open flames or people falling into water, the confidence threshold of the second level can be appropriately lowered, allowing more scenarios to quickly enter deep judgment and ensuring the accuracy of high-risk identification.

[0109] In some implementations, the monitoring unit 103 dynamically generates fusion weights corresponding to each modality based on the real-time environmental context, including: encoding the real-time environmental context information to obtain a context feature vector; inputting the context feature vector into a gating network to output initial weight parameters corresponding to each modality; and normalizing the initial weight parameters to obtain the final fusion weights of each modality.

[0110] Specifically, the system collects various environmental and scene parameters that affect the quality of modality data and the validity of judgments in real time. All parameters are input into a lightweight encoding network, which, through feature mapping and dimensionality transformation, outputs a fixed-dimensional context feature vector. This vector fully carries the current environmental state information. The encoded context feature vector is then input into a pre-trained gating network. The gating network, through its internal weight mapping logic, calculates the initial weight parameters for each modality. The magnitude of these initial weight parameters corresponds to the reliability of the judgment for that modality in the current environment. The initial weight parameters of all modalities output by the gating network are converted into proportional weights using a normalization algorithm, ensuring that the sum of the fusion weights for all modalities is a fixed value. Finally, fusion weights for each modality that can be directly used for feature fusion calculations are obtained.

[0111] By using environmental context encoding, gating network weight generation, and normalization processing, the multimodal fusion weights are dynamically and adaptively adjusted according to the real-time environment and scenario. This replaces the traditional fixed-weight fusion strategy and can proactively adapt to changes in lighting, noise, and scene conditions during ship navigation. It automatically reduces the contribution of the interference mode to the judgment and increases the role of the reliable mode. This mechanism reduces the false alarm rate in complex environments and improves the accuracy and environmental adaptability of multimodal fusion judgment.

[0112] In some implementations, the monitoring unit 103 performs fusion processing on the multimodal feature data based on the fusion weights, including: weighting each modal feature data with its corresponding fusion weight; performing cross-modal attention interaction on the weighted multimodal features to generate a fusion feature vector; and completing a security risk determination based on the fusion feature vector.

[0113] Specifically, for the feature data of each modality, the feature values ​​in the feature vector are correlated with the corresponding fusion weights of that modality to obtain weighted modal features. Modal features with higher weights contribute more to subsequent fusion, while those with lower weights have their contributions suppressed. The weighted features from all modalities are concatenated and input into the cross-attention calculation module. This module assigns attention weights to the features of each modality, enabling them to automatically focus on semantically relevant features from other modalities, strengthening the association of complementary information between modalities and weakening the interference of irrelevant noise. After multiple rounds of attention interaction, a unified fusion feature vector is output. The generated fusion feature vector is input into a pre-trained risk assessment model or rule assessment module, outputting the final risk category, risk level, and corresponding confidence level as the final monitoring result. Cross-modal attention interaction is used to enable features from different modalities to automatically capture semantic associations and correspondences, strengthening complementary information, weakening redundant interference information, and achieving deep information complementarity between modalities.

[0114] Through a deep fusion mechanism of feature weighting and cross-modal attention interaction, the fusion weights are dynamically adapted to the environment, and the semantic association and information complementarity value between multimodal features are fully explored. Compared with traditional fixed-rule decision-level fusion, it can significantly improve the accuracy of risk judgment in complex and ambiguous scenarios, reduce the false alarm rate caused by environmental interference, and maintain the flexibility and adaptability of fusion computation.

[0115] In some implementations, the monitoring unit 103 is also configured to: filter the judgment results of personnel status anomalies in the process of risk determination of personnel status anomalies by combining the ship's motion state and the type of water area where it is located; and block the output of the monitoring results of the corresponding personnel status anomalies when the ship is in a preset stationary state and located in a preset legal mooring area.

[0116] Abnormalities in personnel status refer to abnormal behaviors directly related to crew members' on-duty status and physiological condition. These mainly include fatigue, inattention, unauthorized absence from duty, and sleeping on duty. These are the most common monitoring targets in ship navigation safety supervision, and their compliance determination is strongly correlated with the ship's operational scenario. Ship motion status characterizes the ship's current operating condition, determined by a combination of parameters such as speed, main engine operating status, and hull acceleration. It is typically categorized into navigation status and stationary status. Water area types are classifications of water area functional attributes based on electronic navigation charts or electronic nautical charts, including channels, anchorages, berthing areas, bridge areas, prohibited anchorage areas, and traffic control reporting areas. Different types of water areas correspond to differentiated navigation supervision rules and personnel duty requirements. Pre-defined legal berthing areas are water areas marked in the spatial rule system that allow ships to berth and where personnel duty requirements can be relaxed. These typically include legal anchorages, port berthing areas, and dock operation berths that comply with maritime regulatory rules.

[0117] Specifically, after the system completes the preliminary judgment of personnel status anomalies, it does not directly output the final result. Instead, it simultaneously retrieves the current vessel's motion status parameters and the area type of the current waterway obtained through spatial matching. These two scenario contexts are used as verification conditions to perform compliance checks on the preliminary judgment results. Anomalies that comply with navigation scenario regulatory rules are retained and output normally, while results that do not conform to scenario logic enter the subsequent shielding process. The system is set up with a shielding rule triggered by two conditions. The shielding operation is only performed on personnel status anomaly judgment results when both conditions—"the vessel is in a preset stationary state" and "the vessel is located in a preset legal mooring area"—are met simultaneously. If either condition is not met, the shielding mechanism is not triggered, and the anomaly monitoring results are output according to the normal process.

[0118] By employing a dual-condition scenario filtering mechanism based on vessel motion status and water area type, the system effectively eliminates scenario ambiguity in determining personnel status anomalies. It distinguishes between navigation violations and normal berthing behavior from the perspective of judgment logic, significantly reducing invalid false alarms in legal berthing scenarios and improving the practicality and reliability of the monitoring system. At the same time, the dual-condition joint verification design avoids the risk of anomaly underreporting, ensuring the rigor of navigation safety supervision while reducing false alarms, and is adaptable to the differentiated supervision needs of different vessel navigation scenarios.

[0119] Since the processing and functions implemented by the device in this embodiment are basically the same as the embodiments, principles and examples of the aforementioned methods, any details not covered in the description of this embodiment can be found in the relevant descriptions in the aforementioned embodiments, and will not be repeated here.

[0120] The technical solution of this invention, targeting ship safety risk monitoring scenarios, first acquires multi-dimensional ship monitoring data, and uses a streaming parallel independent processing link to complete the preprocessing and feature extraction of each modality's data. Through a multi-level detection architecture with computational costs ranging from low to high, risk judgment is advanced layer by layer. The monitoring results are directly output at the level that meets the confidence requirements, while those that do not meet the requirements enter a deeper level of analysis. The final detection level dynamically adapts the fusion weights of each modality based on the real-time environmental context, and outputs the final result after deep feature fusion. This significantly accelerates the risk response speed while improving the accuracy of judgment in complex scenarios and reducing invalid false alarms, thus making up for the shortcomings of traditional serial fixed rule architectures that cannot achieve both accuracy and efficiency.

[0121] According to an embodiment of the present invention, a storage medium corresponding to the ship safety risk monitoring method is also provided, the storage medium including a stored program, wherein the program controls the device where the storage medium is located to execute the ship safety risk monitoring method described above when it is running.

[0122] Since the processing and functions implemented by the storage medium in this embodiment are basically the same as the embodiments, principles and examples of the aforementioned methods, any details not covered in the description of this embodiment can be found in the relevant descriptions in the aforementioned embodiments, and will not be repeated here.

[0123] By adopting the technical solution of this invention, when monitoring ship safety risks, after collecting multimodal ship monitoring data, the preprocessing and feature extraction of each modality data are completed through streaming parallel processing. A multi-level progressive judgment mechanism is used for risk identification. When the judgment confidence meets the preset requirements, the result is directly output; otherwise, it is advanced to the next detection level. The final level dynamically adjusts the fusion weight of each modality according to the real-time environmental context. After completing the fusion of multimodal features, the final monitoring result is output. This effectively improves the risk discrimination accuracy in complex navigation scenarios, reduces the false alarm probability, and significantly shortens the response time of risk judgment. It solves the problem that the traditional serial fixed rule fusion architecture cannot simultaneously guarantee judgment accuracy and response speed.

[0124] According to an embodiment of the present invention, a computer program product corresponding to the ship safety risk monitoring method is also provided. The computer program product includes a computer program that, when processed and executed, implements the steps of the above-described ship safety risk monitoring method.

[0125] Since the processing and functions implemented by the computer program product in this embodiment are basically corresponding to the embodiments, principles and examples of the aforementioned methods, any details not covered in the description of this embodiment can be found in the relevant descriptions in the aforementioned embodiments, and will not be repeated here.

[0126] The technical solution of this invention not only improves the accuracy of risk assessment in complex navigation scenarios and reduces false alarms, but also significantly reduces the delay of risk assessment, thus overcoming the shortcomings of traditional serial fixed rule fusion architecture in achieving both high accuracy and low latency.

[0127] In summary, it is readily understood by those skilled in the art that, without conflict, the aforementioned advantageous methods can be freely combined and superimposed.

[0128] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A method for monitoring ship safety risks, characterized in that, include: Acquire multimodal monitoring data of ships; The monitoring data for each modality were preprocessed and feature extracted using a streaming parallel approach to obtain the feature data corresponding to each modality. A multi-level progressive detection architecture is used to assess the risk of each modal feature data; If the confidence level of the judgment result output by the current detection level meets the preset conditions, the monitoring result will be output directly, and the execution of subsequent detection levels will be terminated. If the confidence level of the judgment result output by the current detection level does not meet the preset conditions, then proceed to the next detection level; The final detection layer dynamically generates fusion weights for each modality based on the real-time environmental context. After fusing the multimodal feature data based on the fusion weights, the final monitoring result is output.

2. The ship safety risk monitoring method according to claim 1, characterized in that, The multimodal monitoring data includes: visual modal data, acoustic modal data, ship state modal data, and spatial rule modal data.

3. The ship safety risk monitoring method according to claim 1 or 2, characterized in that, The monitoring data for each modality were preprocessed and feature extracted using a streaming parallel approach, including: Based on the temporal characteristics of the risk type to be monitored corresponding to each modality, the window size of the streaming processing is dynamically adjusted; the window size is used to limit the length of the time series data in a single round of processing and match the time scale of the corresponding risk behavior to be monitored. Within each streaming window, preprocessing operations are performed independently on the raw monitoring data of the corresponding modality. These preprocessing operations include data cleaning, data standardization, and format normalization. Multi-level feature extraction is performed on each modal data after preprocessing to obtain the feature data of the corresponding modality; the preprocessing and feature extraction processes of each modality are independent and run asynchronously in parallel.

4. The ship safety risk monitoring method according to claim 1, characterized in that, The multi-level progressive detection architecture includes a first detection level, a second detection level, and a third detection level executed sequentially. The first detection level is used to perform initial screening of each modal feature data based on lightweight rules. The second detection level is used to perform single-modal independent analysis and risk assessment on feature data whose confidence level does not meet the preset conditions after being judged by the first detection level. The third detection layer is used to perform the fusion processing of the modal feature data.

5. The ship safety risk monitoring method according to claim 1, characterized in that, The fusion weights for each modality are dynamically generated based on the real-time environmental context, including: Encode the real-time environment context information to obtain a context feature vector; The context feature vector is input into a gating network, and the initial weight parameters corresponding to each modality are output. The initial weight parameters are normalized to obtain the final fusion weights for each mode.

6. The ship safety risk monitoring method according to claim 1 or 5, characterized in that, The multimodal feature data is fused based on the fusion weights, including: Each modal feature data is weighted and calculated with its corresponding fusion weight; Perform cross-modal attention interaction on the weighted multimodal features to generate a fused feature vector; Security risk assessment is completed based on the fused feature vector.

7. The ship safety risk monitoring method according to claim 1, characterized in that, In the risk assessment process for personnel status anomalies, the assessment results are filtered by combining the ship's motion status and the type of water area it is in; when the ship is in a preset stationary state and located in a preset legal mooring area, the monitoring results of the corresponding personnel status anomalies are blocked.

8. A ship safety risk monitoring device, characterized in that, include: The acquisition unit is configured to acquire multimodal monitoring data of the ship; The feature extraction unit is configured to preprocess and extract features from the monitoring data of each modality in a streaming parallel manner to obtain the feature data corresponding to each modality. The monitoring unit is configured to use a multi-level progressive detection architecture to determine the risk of each modal feature data; The monitoring unit is further configured to directly output the monitoring result and terminate the execution of subsequent detection levels if the confidence level of the judgment result output by the current detection level meets the preset conditions. The monitoring unit is also configured to proceed to the next detection level if the confidence level of the judgment result output by the current detection level does not meet the preset conditions. The final detection layer dynamically generates fusion weights for each modality based on the real-time environmental context. After fusing the multimodal feature data based on the fusion weights, the final monitoring result is output.

9. A storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, it controls the device containing the storage medium to perform the ship safety risk monitoring method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the ship safety risk monitoring method according to any one of claims 1 to 7.

Images