Intelligent method and system for safety monitoring of water transportation engineering construction ship

Abstract

The invention discloses an intelligent method and system for safety monitoring of a water transportation engineering construction ship. The system comprises a multi-mode sensing device, an on-board server, an alarm notification module and a cloud platform server. The intelligent method comprises the following steps: S1, collecting videos, infrared images, temperature and humidity, smog, gas concentration, vibration, position and equipment state data of different parts of a ship through multi-mode sensing equipment, and storing the data in a server on the ship; s2, performing deep integration on the video monitoring data from different sources and various sensor data by adopting a multi-modal data fusion method; carrying out labeling, data enhancement and balancing on the fused multi-modal data, and constructing a training set; and S3, training the constructed alarm model by adopting the training set, and storing the optimal alarm model. According to the method, video data and sensor data are combined, a multi-modal fusion algorithm is adopted, and the accuracy and reliability of ship safety monitoring can be improved.

Description

Technical Field

[0001] This invention relates to the field of safety monitoring technology for construction vessels, and in particular to an intelligent method and system for safety monitoring of construction vessels in waterway engineering. Background Technology

[0002] Traditional ship safety monitoring relies primarily on manual inspections and reviewing data from monitoring equipment. Safety personnel need to regularly inspect various parts of the ship, such as the engine room and deck, to identify potential safety hazards and issue early warnings. However, while these methods can identify some unsafe behaviors, their accuracy is difficult to guarantee in the complex construction environment of waterway engineering (e.g., changes in lighting, ship vibration, severe weather). Furthermore, existing systems lack the ability to integrate multi-source data, failing to comprehensively reflect the ship's safety status. Manual inspections cannot achieve 24 / 7 coverage, and their efficiency is typically low, consuming significant manpower and resources while failing to meet the demands of modern intelligent ship safety management. This is especially true in large-scale hydraulic engineering projects, where the number and types of ships are numerous and diverse, involving various large and heavy equipment. Safety personnel cannot guarantee 24 / 7 manual inspections of the ship, engine room, deck, etc., nor can they maintain high mobility and standby 24 / 7. In the event of danger, timely reporting and emergency response are difficult, posing significant challenges to safety management.

[0003] The development and widespread adoption of artificial intelligence (AI) technology have driven the application of monitoring systems and equipment in the construction and management of waterway engineering projects. To a certain extent, this has helped to obtain production data from construction vessels and assist in safety management and inspection. However, waterway engineering involves numerous facilities and is extremely complex. The operating conditions of vessels at the construction site are varied and the operating environment is harsh, resulting in significant gaps in intelligent vessel monitoring. Furthermore, the lack of standardized procedures presents many challenges in the deployment of AI video equipment and the timeliness of data transmission. Summary of the Invention

[0004] Purpose of the invention: The purpose of this invention is to provide an intelligent method and system for safety monitoring of construction vessels in water transport engineering. By combining image recognition, sensor data fusion and edge computing technologies, it can achieve comprehensive safety monitoring and early warning of construction vessels, thereby improving the efficiency and accuracy of vessel safety management.

[0005] Technical solution: An intelligent system for safety monitoring of waterway engineering construction vessels, including multimodal sensing equipment, onboard server, alarm notification module and cloud platform server; The multimodal sensing device is used to realize data acquisition functions, including acquiring video of the ship, the environment, and equipment status. The onboard server is used to process multimodal data in real time; it fuses video data and sensor data in a multimodal manner, establishes an alarm model, and generates safety alarm information based on the fusion results. The alarm notification module categorizes alarm information and notifies crew members and ship management personnel based on the information level; simultaneously, it interacts with regulatory agencies through the cloud platform server to upload the ship's safety status and alarm information in real time. The cloud platform server is used to exchange ship safety production monitoring data with owners, regulatory agencies, and ship managers.

[0006] An intelligent method for safety monitoring of construction vessels in water transport engineering, based on the aforementioned intelligent system, includes the following steps: S1 collects video, infrared images, temperature and humidity, smoke, gas concentration, vibration, location and equipment status data of different parts of the ship through multimodal sensing devices and stores them on the ship's server. S2 employs a multimodal data fusion method to deeply integrate video surveillance data from different sources with data from multiple sensors; the fused multimodal data is then labeled, augmented, and balanced to construct a training set. S3, use the training set to train the constructed alarm model and save the best alarm model.

[0007] Furthermore, the multimodal data fusion method includes data preprocessing and spatiotemporal alignment, feature-level fusion, decision-level fusion, and fusion optimization in its data processing. The data preprocessing and spatiotemporal alignment include data cleaning, time synchronization, and spatial registration. The data cleaning process involves video stabilization and image enhancement of the raw video stream and sensor data, as well as smoothing filtering and outlier removal of the sensor data. The time synchronization uses the NTP protocol or hardware clock synchronization to give all input data a precise timestamp, ensuring that the analyzed events occur on the same time reference. The spatial registration involves establishing a spatial coordinate system for the modalities involving location information, including: mapping and aligning the target location in the video frame with the AIS target location, and mapping the sensor alarm location onto the video monitoring frame or electronic chart. The feature-level fusion includes multimodal feature extraction, feature association, and fusion; The multimodal feature extraction enables the extraction of video features and sensor features; The video features refer to the target objects, behaviors, and scene states extracted using deep learning models; The sensor characteristics refer to the time-domain characteristics, frequency-domain characteristics, and event characteristics of the sensor data; The feature association and fusion, through the design of association rules, associates features extracted from different modalities that describe the same event or target. The decision-level fusion integrates the preliminary judgment results of each modality and designs rule-based logical judgments.

[0008] Furthermore, the rule-based logical judgment includes: If the video analysis report shows "open flame" and the infrared temperature sensor reports "local temperature > preset fire threshold" -> confirm "fire alarm"; If the video analysis report shows "person falls and remains still" & there is no violent vibration in the area & duration > T seconds -> confirm "person falls / injuries alarm"; If the AIS report states "High Collision Risk" and video surveillance confirms the presence of the target vessel and its abnormal course, then an "Imminent Collision Risk Alert" will be issued.

[0009] Furthermore, when contradictions arise between different modal data or preliminary judgments, the system initiates a conflict resolution process: checking data quality and synchronization status; calling higher-precision models or adding other sensor data for auxiliary judgment; making a decision based on preset priority rules or historical statistical data; if the conflict cannot be resolved immediately, generating an alarm that "manual review is required" and displaying conflict information.

[0010] Furthermore, the association rules include spatiotemporal association rules, semantic association rules, causal logic association rules, and statistical association rules; Spatiotemporal association rules require that the features of different modalities be highly consistent in time and space; Semantic association rules use the semantic information or attributes of features for matching, making up for the uncertainty or ambiguity of spatiotemporal information; Causal logic association rules establish causal relationships or logical chains between features based on industry knowledge; Statistical association rules use historical data or online statistical characteristics to determine the association between features.

[0011] Furthermore, in step S2, the annotation of multimodal data includes annotation of targets, events, multimodal associations, and alarm levels; Target annotation involves defining and classifying bounding boxes for ships, personnel, equipment, flames, smoke, and hazardous areas in the image. Event labeling: Labeling behaviors / events on consecutive video frames, including the start frame, end frame, and event type; Multimodal association annotation associates and annotates keyframes in the video with synchronized sensor data to form cross-modal labels; Alarm level labeling: Each event is labeled with an alert level, which serves as the output target of the alarm model; the alert levels include urgent, important, and general. The data augmentation and balancing include spatial augmentation, environmental augmentation, category balancing, and multimodal data augmentation; Spatial enhancement simulates changes in camera viewpoint by randomly rotating, scaling, cropping, flipping, and perspective transforming video data. Environmental enhancements include adding simulated rain and fog, lens smudges, dynamic shadows, and low-light noise to the image data. Category balancing is implemented, with SMOTE oversampling or copy-paste enhancement applied to rare categories. Multimodal data enhancement adds Gaussian noise and temporal offset to sensor data and applies it synchronously with video enhancement to maintain modal consistency.

[0012] Furthermore, the alarm model includes: Alarm models for water-falling and smoking scenarios are established by using computer vision and deep learning algorithms to detect and analyze the characteristics of water-falling, smoke, and flames, and to build corresponding models to achieve automatic identification and alarm for water-falling and smoking behaviors. The alarm model for open flame scenarios uses infrared thermal imaging algorithms and flame detection algorithms to monitor and identify open flames in real time. The alarm model for not wearing a life jacket uses deep learning algorithms to analyze images in surveillance videos to identify the differences between those wearing life jackets and those not wearing them, and combines posture analysis methods to identify human posture.

[0013] Compared with the prior art, the significant advantages of this invention are as follows: 1. The method of the present invention, by combining video data and sensor data and employing a multimodal fusion algorithm, significantly improves the accuracy and reliability of ship safety monitoring; 2. The system of the present invention can automatically generate early warning information of different levels based on the results of multimodal data fusion, and notify relevant personnel through various means to ensure that unsafe behaviors are dealt with quickly; it can realize safety monitoring of the construction behavior of personnel on board, and provide management and decision-making basis for ship managers and owners; 3. The system of the present invention provides a standard data interface for data interaction with regulatory agencies, and uploads the safety status and early warning information of ships in real time, which facilitates data statistics and analysis by regulatory agencies. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the system architecture of the present invention; Figure 2 This is a schematic diagram of multimodal data fusion. Detailed Implementation

[0015] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0016] like Figure 1 As shown, this embodiment provides an intelligent system for safety monitoring of waterway engineering construction vessels, including multimodal sensing devices, onboard servers, alarm notification modules, etc. Multimodal sensing devices are used to achieve data acquisition functions, including collecting multimodal data such as ship video, environment, and equipment status; Multimodal sensing devices include cameras, infrared sensors, temperature and humidity sensors, vibration sensors, etc. The shipboard server is used to process multimodal data in real time; it fuses video data with sensor data in a multimodal manner, establishes an alarm model, and generates safety alarm information based on the fusion results. The alarm notification module categorizes alarm information and notifies crew members and ship management personnel based on the information level; at the same time, it interacts with regulatory agencies through the cloud platform server to upload the ship's safety status and alarm information in real time. The cloud platform server is used to exchange ship safety production monitoring data with owners, regulatory agencies, and ship managers.

[0017] like Figure 2 As shown in the figure, this embodiment provides an intelligent method for safety monitoring of construction vessels in water transport engineering. The method includes the following steps: Step 1, Data Acquisition; Multimodal sensing devices are used to collect video, infrared images, temperature and humidity, smoke, gas concentration, vibration, location, and equipment status data from different parts of the ship, and these data are stored on the ship's server. High-definition cameras, infrared sensors, temperature and humidity sensors, smoke and gas concentration sensors, vibration sensors, location sensors, and equipment status sensors are installed in key areas of the ship, such as the deck, engine room, and bridge. The cameras are used to monitor the behavior of the crew and the status of the equipment in real time. The infrared sensors are used to detect open flames and high-temperature areas. The temperature and humidity sensors are used to monitor environmental changes. The smoke and gas concentration sensors are used to monitor harmful gases and smoke generated in the cabin. The vibration sensors are used to detect abnormal vibrations of the ship's equipment.

[0018] Step 2: Use a multimodal data fusion method to process video surveillance data from different sources and data from multiple sensors; Multimodal data fusion methods deeply integrate video surveillance data from different sources with data from various sensors (such as infrared sensors, temperature sensors, smoke sensors, gas concentration sensors, vibration sensors, position sensors (AIS (Automatic Identification System) / GNSS (Global Navigation Satellite System)), equipment status sensors, etc.); its core implementation methods include data preprocessing and spatiotemporal alignment, feature-level fusion, decision-level fusion, and fusion optimization. Data preprocessing and spatiotemporal alignment include data cleaning, time synchronization, and spatial registration; Data cleaning performs noise filtering and image enhancement on the raw video stream and sensor data, including video decluttering and image enhancement; and performs smoothing filtering and outlier removal on the sensor data to improve data quality. Time synchronization uses the NTP protocol or hardware clock synchronization to accurately timestamp all input data, ensuring that analyzed events occur on the same time base; the system has dynamic time calibration capabilities to compensate for data transmission delays between different devices; Spatial registration involves unifying the spatial coordinate system for modalities involving location information (such as video, AIS / GNSS); for example, mapping and aligning the target location in a video frame with the target location in an AIS frame, or mapping the sensor alarm location onto a video surveillance screen or electronic nautical chart. Feature-level fusion includes multimodal feature extraction, feature association, and fusion; Multimodal feature extraction extracts video features and sensor features; Video features refer to high-dimensional semantic features extracted using deep learning models (such as convolutional neural networks (CNN), object detection models (YOLO / Faster R-CNN), and behavior recognition models), including target objects (people, equipment, ships, cargo), behaviors (climbing, falling, illegal operations, open flames, smoke), and scene states (visibility, water accumulation). Deep learning models are trained and optimized using massive amounts of data from specific scenes (such as ports, ship cabins, and decks). Sensor features refer to the time-domain features (such as mean, variance, peak value, trend), frequency-domain features (such as FFT spectrum, characteristic frequency energy), and event features (such as temperature exceedance indicator, smoke concentration threshold trigger, vibration intensity alarm) of the extracted sensor data. Feature association and fusion associate features extracted from different modalities that describe the same event or target by designing association rules; for example, associating a “suspected flame area” detected in a video with a “high temperature anomaly point” from an infrared sensor in the same spatial location.

[0019] Association rules include spatiotemporal association rules, semantic association rules, causal logic association rules, and statistical association rules; Spatiotemporal correlation requires that the features of different modalities are highly consistent in time and space. For example, if the video detects a "suspected smoke area" near the driver's side window and the smoke sensor installed in the driver's side alarms at the same time, the two occur almost simultaneously in time, and the location of the smoke sensor is within the "suspected smoke area" detected by the video, then there is a strong correlation. Semantic association rules use the semantic information or attributes of features for matching to make up for the uncertainty or ambiguity of spatiotemporal information. For example, video analysis may find that the surface color of a device is abnormal (reddish), and at the same time, an infrared sensor near the device detects a "local high temperature point". Although the surface color change and the internal / local high temperature are not exactly the same point in space, the semantic knowledge that "high temperature causes surface discoloration" can support the association. Causal logic association rules establish causal relationships or logical chains between features based on industry knowledge (ship safety, equipment working principles, etc.). For example, a position sensor shows that the ship is sending abnormal displacement (such as severe drift), while a vibration sensor detects a violent collision of the hull, which is logically strongly related. Statistical association rules use historical data or online statistical characteristics to determine the correlation between features. For example, if historical data analysis reveals that when a certain pattern appears in the vibration spectrum of a specific area, the probability of bolt loosening (originating from equipment status sensors or visual inspection features) in that area increases significantly, then a statistical association rule can be established.

[0020] Feature-level fusion uses a weighted fusion algorithm to assign dynamic weights to features of different modalities. The weights are dynamically calculated based on feature confidence (such as the confidence score of the target detection box and the signal-to-noise ratio of the sensor data), modal reliability (such as the decrease in video confidence and the increase in infrared weight under low light conditions) and historical performance (the false alarm rate of a sensor under specific conditions). Feature-level fusion uses a deep learning fusion model (such as feature concatenation + fully connected layer, attention mechanism model) to input the extracted multimodal feature vectors into the fusion neural network. The network automatically learns the complex relationships between different features and their contribution to the final decision, and outputs a unified feature representation after fusion.

[0021] Decision-level fusion integrates the preliminary judgment results from various modalities to design rule-based decision trees or logical judgments: If the video analysis report shows "open flame" and the infrared temperature sensor reports "local temperature > preset fire threshold" -> confirm "fire alarm"; If the video analysis report shows "person falls and remains still" & there is no severe vibration in the area (vibration sensor is normal) & duration > T seconds -> confirm "person falls / injuries alarm"; If the AIS reports "high collision risk" and video surveillance confirms the presence of the target vessel and its abnormal course, then an "imminent collision risk warning" will be issued. The fusion optimization assigns a confidence score to the preliminary judgment result of each modality; the Dempster-Shafer evidence theory method is used for fusion to output the comprehensive confidence score and the final conclusion. Furthermore, when conflicting data or preliminary judgments arise between different modalities (e.g., a video shows a "flame" but the temperature is normal), the system initiates a conflict resolution process: checking data quality and synchronization status; calling a higher-precision model or adding other sensor data for auxiliary judgment (e.g., calling smoke sensor data); making a decision based on preset priority rules (e.g., in safety-critical events, sensor alarms have higher priority than video misidentification) or historical statistical data; if the conflict cannot be resolved immediately, an alarm "requires manual review" is generated, along with conflict information.

[0022] Step 3, alarm model construction; A ship construction safety alarm model is built using a deep convolutional neural network framework to provide safety alarms for scenarios such as accidental fall into the water, smoking, open flame, not wearing a life jacket, and engine hood not being closed. The alarm model for water-falling and smoking scenarios uses computer vision and deep learning algorithms to detect and analyze features such as water-falling, smoke, and flames, establishing a corresponding model to automatically identify and trigger alarms for water-falling and smoking behaviors. Furthermore, different warning levels can be set for different areas: high warning levels for smoking in areas such as the engine room, material storage, and paint shop; and low warning levels for smoking on the deck. The alarm model for open flame scenarios uses infrared thermal imaging algorithms, flame detection algorithms, and other technologies to monitor and identify open flames in real time. The alarm model for not wearing a life jacket uses deep learning algorithms to analyze images in surveillance videos to identify the differences between wearing and not wearing life jackets. It also combines posture analysis technology to identify human posture, thereby accurately judging the behavior of not wearing a life jacket. At the same time, through multimodal data fusion and combined with data from other sensors, the accuracy of identifying the behavior of not wearing a life jacket is improved.

[0023] Step 4, alarm model training; A training set is constructed by labeling, augmenting, and balancing the collected multimodal data; Multimodal data source: Real-time monitoring video stream: covering typical areas of the ship (deck, engine room, cargo hold, bridge, dock loading and unloading area), including different time periods (day / night), weather (sunny / rainy / foggy) and operation scenarios (navigation / berthing / loading and unloading). Historical alarm event database: Collects verified real alarm cases (fire, people falling into water, collision risk, equipment failure, etc.) and their associated multimodal data (video clips + sensor readings). Simulated Scene Library: Synthetic data of extreme scenes (such as strong light interference, occlusion, and low visibility) are generated by the 3D simulation engine to make up for the lack of real-world data.

[0024] Multimodal data annotation includes annotations for targets, events, multimodal associations, and alarm levels; Target annotation involves labeling and classifying targets in the image, such as ships, personnel, equipment, flames, smoke, and hazardous areas, using bounding boxes (e.g., "open flame", "welding sparks", "steam"). Event labeling: Label consecutive video frames with actions / events (such as "person falls", "goods slips", "illegal smoking", "equipment leaks"), label the start frame, end frame and event type; Multimodal association annotation associates keyframes of video with synchronized sensor data (temperature, smoke concentration, vibration amplitude) to form cross-modal labels (e.g., "video flame + temperature > 300°C = real fire"). Alarm level labeling: Label each event with an alert level (urgent / important / general), which serves as the output target of the alarm model; Data augmentation and balancing include spatial augmentation, environmental augmentation, category balancing, and multimodal data augmentation; Spatial enhancement simulates changes in camera viewpoint by randomly rotating (±30°), scaling (0.8~1.2x), cropping, flipping, and perspective transforming video data. Environmental enhancement involves adding simulated rain and fog, lens smudges, dynamic shadows, and low-light noise to the image data (using GAN to generate adversarial examples) to improve the robustness of the model. Category balancing is implemented, and SMOTE oversampling or copy-paste enhancement is used for rare categories (such as "people falling into the water") (pasting small targets to different backgrounds). Multimodal data enhancement adds Gaussian noise and temporal offset to sensor data and applies it synchronously with video enhancement to maintain modal consistency.

[0025] Step 5: Perform real-time data processing based on the optimal alarm model; When it is determined that the current model does not require further optimization, the collected multimodal data is processed in real time, and the optimized alarm model is used directly to carry out intelligent monitoring. The ship's server performs real-time analysis of the collected multimodal data to identify unsafe behaviors of crew members (such as accidental falls into the water, smoking, open flames, not wearing life jackets, and leaving the engine room cover open), and generates a comprehensive ship safety status report.

[0026] Step 6, Intelligent Early Warning and Feedback; Based on the ship's safety status report, early warning information is automatically generated and released in a tiered manner, while relevant personnel are notified through multiple communication channels. The core mechanism is as follows: A1, Warning Information Classification: Early warning information is classified into three levels based on the potential harm, urgency, and scope of impact of the alarm event: emergency alarm, important alarm, and general alarm. Emergency alarms refer to immediate threats involving personal safety, significant property damage, or major environmental risks (such as collision risk, fire, people falling into water, or serious equipment failure leading to loss of control). Such alarms require an immediate response. Critical alerts refer to events that may affect navigation safety, operational efficiency, or violate important regulations, but have not yet posed an immediate major threat (such as: abnormal performance of critical equipment, deviation from the planned route threshold, moderate severe weather warning, or overdue important maintenance tasks). Such alerts require a rapid response. General alarms refer to routine reminders, status notifications, or low-risk anomalies (such as routine maintenance reminders, minor anomalies in non-critical equipment, and information update notifications). These alarms require timely attention and handling. A2, Differentiated Notification Strategy: Notification methods include shore-based notification and ship-based notification; Shore-based notifications are primarily targeted at shore-based management personnel, dispatch centers, and emergency command personnel; they are sent via network platforms (dedicated monitoring systems / Web Portal / App push notifications), SMS, and email; emergency alarms utilize multiple channels (SMS + platform pop-ups + email) for strong alerts (e.g., continuous ringing, flashing); important alarms use SMS + platform messages; and general alarms are primarily notified via network platforms or email. Shipboard notifications are sent to the crew and operators on board; voice announcements are made via the ship's public address system; emergency alarms trigger the highest priority broadcast (which may interrupt the current broadcast), are broadcast repeatedly, and are accompanied by a specific warning sound; important alarms trigger the standard priority broadcast; general alarms may be announced silently via broadcast or onboard terminals (such as displays); The notification recipients vary depending on the warning level. Emergency alarms immediately activate the emergency response mechanism, mandating notification to all relevant management personnel (such as the captain, chief engineer, shore-based emergency duty manager, safety supervisor, etc.), and the notification scope can be expanded according to preset rules (such as nearby vessels, search and rescue centers); important alarms push messages to directly responsible personnel (such as the on-duty driver, equipment chief engineer, dispatcher), and copy their superiors; general alarms push messages to the corresponding personnel or relevant working groups. Step 7, Data Interaction and Supervision; The intelligent system of this invention provides a standard data interface for data interaction with regulatory agencies such as maritime authorities and local administrations, and uploads vessel safety status and early warning information in real time. Regulatory agencies can monitor vessel safety conditions in real time through the platform and perform data statistics and analysis.

Claims

1. An intelligent system for safety monitoring of construction vessels in water transport engineering projects, characterized in that, This includes multimodal sensing devices, onboard servers, alarm notification modules, and cloud platform servers; The multimodal sensing device is used to realize data acquisition functions, including acquiring video of the ship, the environment, and equipment status. The onboard server is used to process multimodal data in real time; it fuses video data and sensor data in a multimodal manner, establishes an alarm model, and generates safety alarm information based on the fusion results. The alarm notification module categorizes alarm information and notifies crew members and ship management personnel based on the information level; simultaneously, it interacts with regulatory agencies through the cloud platform server to upload the ship's safety status and alarm information in real time. The cloud platform server is used to exchange ship safety production monitoring data with owners, regulatory agencies, and ship managers.

2. An intelligent method for safety monitoring of construction vessels in water transport engineering, implemented based on the intelligent system as described in claim 1, characterized in that, Includes the following steps: S1 collects video, infrared images, temperature and humidity, smoke, gas concentration, vibration, location and equipment status data of different parts of the ship through multimodal sensing devices and stores them on the ship's server. S2 employs a multimodal data fusion method to deeply integrate video surveillance data from different sources with data from multiple sensors; The fused multimodal data is labeled, augmented, and balanced to construct a training set; S3, use the training set to train the constructed alarm model and save the best alarm model.

3. The intelligent method for safety monitoring of waterway engineering construction vessels according to claim 2, characterized in that, The multimodal data fusion method includes data preprocessing and spatiotemporal alignment, feature-level fusion, decision-level fusion, and fusion optimization. The data preprocessing and spatiotemporal alignment include data cleaning, time synchronization, and spatial registration. The data cleaning process involves video stabilization and image enhancement of the raw video stream and sensor data, as well as smoothing filtering and outlier removal of the sensor data. The time synchronization uses the NTP protocol or hardware clock synchronization to give all input data a precise timestamp, ensuring that the analyzed events occur on the same time reference. The spatial registration involves establishing a spatial coordinate system for the modalities involving location information, including: mapping and aligning the target location in the video frame with the AIS target location, and mapping the sensor alarm location onto the video monitoring frame or electronic chart. The feature-level fusion includes multimodal feature extraction, feature association, and fusion; The multimodal feature extraction enables the extraction of video features and sensor features; The video features refer to the target objects, behaviors, and scene states extracted using deep learning models; The sensor characteristics refer to the time-domain characteristics, frequency-domain characteristics, and event characteristics of the sensor data; The feature association and fusion, through the design of association rules, associates features extracted from different modalities that describe the same event or target. The decision-level fusion integrates the preliminary judgment results of each modality and designs rule-based logical judgments.

4. The intelligent method for safety monitoring of construction vessels in water transport engineering according to claim 3, characterized in that, The rule-based logical judgment includes: If the video analysis report shows "open flame" and the infrared temperature sensor reports "local temperature > preset fire threshold" -> confirm "fire alarm"; If the video analysis report shows "person falls and remains still" and there is no severe vibration in the area, and the duration is greater than T seconds, then confirm "person falls / injuries alarm"; If the AIS report states "High Collision Risk" and video surveillance confirms the presence of the target vessel and its abnormal course, then an "Imminent Collision Risk Alert" will be issued.

5. The intelligent method for safety monitoring of construction vessels in water transport engineering according to claim 3, characterized in that, When conflicting data from different modalities or preliminary judgments arise, the system initiates a conflict resolution process: checking data quality and synchronization status; calling a higher-precision model or adding other sensor data for auxiliary judgment; and making a decision based on preset priority rules or historical statistical data. If the issue cannot be resolved immediately, an alert will be generated stating "Manual review required," along with conflict information.

6. The intelligent method for safety monitoring of construction vessels in waterway engineering according to claim 3, characterized in that, The association rules include spatiotemporal association rules, semantic association rules, causal logic association rules, and statistical association rules; Spatiotemporal association rules require that the features of different modalities be highly consistent in time and space; Semantic association rules use the semantic information or attributes of features for matching, making up for the uncertainty or ambiguity of spatiotemporal information; Causal logic association rules establish causal relationships or logical chains between features based on industry knowledge; Statistical association rules use historical data or online statistical characteristics to determine the association between features.

7. The intelligent method for safety monitoring of construction vessels in water transport engineering according to claim 2, characterized in that, In step S2, the annotation of multimodal data includes annotation of targets, events, multimodal associations, and alarm levels; Target annotation involves defining and classifying bounding boxes for ships, personnel, equipment, flames, smoke, and hazardous areas in the image. Event labeling: Labeling behaviors / events on consecutive video frames, including the start frame, end frame, and event type; Multimodal association annotation associates and annotates keyframes in the video with synchronized sensor data to form cross-modal labels; Alarm level labeling: Each event is labeled with an alert level, which serves as the output target of the alarm model; the alert levels include urgent, important, and general. The data augmentation and balancing include spatial augmentation, environmental augmentation, category balancing, and multimodal data augmentation; Spatial enhancement simulates changes in camera viewpoint by randomly rotating, scaling, cropping, flipping, and perspective transforming video data. Environmental enhancements include adding simulated rain and fog, lens smudges, dynamic shadows, and low-light noise to the image data. Category balancing is implemented, with SMOTE oversampling or copy-paste enhancement applied to rare categories. Multimodal data enhancement adds Gaussian noise and temporal offset to sensor data and applies it synchronously with video enhancement to maintain modal consistency.

8. The intelligent method for safety monitoring of construction vessels in water transport engineering according to claim 2, characterized in that, The alarm model includes: Alarm models for water-falling and smoking scenarios are established by using computer vision and deep learning algorithms to detect and analyze the characteristics of water-falling, smoke, and flames, and to build corresponding models to achieve automatic identification and alarm for water-falling and smoking behaviors. The alarm model for open flame scenarios uses infrared thermal imaging algorithms and flame detection algorithms to monitor and identify open flames in real time. The alarm model for not wearing a life jacket uses deep learning algorithms to analyze images in surveillance videos to identify the differences between those wearing life jackets and those not wearing them, and combines posture analysis methods to identify human posture.

Images