Intelligent video monitoring system for ship

Abstract

The invention discloses a ship intelligent video monitoring system, and belongs to the technical field of computers. The system comprises a data acquisition unit used for acquiring various modal data of a ship; the data fusion unit is used for determining reference information, adaptively determining to-be-fused target modal data in the multiple modal data based on the reference information, and fusing the target modal data to obtain multi-modal fusion data; the data processing unit is used for determining abnormal information based on the multi-modal fusion data, a pre-constructed ship safety rule base and an abnormal detection model, and the abnormal information comprises ship safety equipment abnormal information, ship personnel action abnormal information, sensor abnormal information and ship traveling track abnormal information; and the alarm unit is used for processing the abnormal information to obtain the abnormal level of the abnormal information, and determining alarm information by using the abnormal level and the abnormal information. According to the invention, the safety and standardization of the ship in the sailing process can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to an intelligent video surveillance system for ships. Background Art

[0002] With the rapid development of the shipping industry, the number of ships continues to increase, and the navigation environment is becoming increasingly complex. Vessels navigating this complex ocean environment face a variety of potential safety threats, such as severe weather, maritime collisions, equipment failures, human error, and the transport of dangerous goods. These factors can lead to serious safety incidents, resulting in not only significant economic losses but also significant threats to the marine ecosystem and human life. Therefore, how to effectively ensure navigational safety and achieve real-time, accurate, and comprehensive vessel status monitoring has become a critical issue that needs to be addressed. Summary of the Invention

[0003] This application provides an intelligent video surveillance system for ships, which can improve the safety and standardization of ship navigation. The technical solution is as follows: In one aspect, the present application provides a ship intelligent video surveillance system, the system comprising: A data acquisition unit, configured to acquire multiple modal data of the vessel, wherein the multiple modal data include at least two of position modal data, monitoring modal data, environmental modal data, or sensor modal data; a data fusion unit, configured to determine reference information, adaptively determine target modal data to be fused from the multiple modal data based on the reference information, and fuse the target modal data to obtain multimodal fusion data, wherein the reference information includes at least one of an environmental risk level, confidence levels of the multiple modal data, or mission requirements at different stages of a ship's voyage; a data processing unit, configured to determine abnormal information based on the multimodal fusion data, a pre-built ship safety rule base, and an abnormality detection model, wherein the abnormal information includes abnormal information of ship safety equipment, abnormal information of ship personnel actions, abnormal information of sensors, and abnormal information of ship driving trajectory; The alarm unit is used to process the abnormal information to obtain the abnormal level of the abnormal information, and determine the alarm information using the abnormal level and the abnormal information.

[0004] On the other hand, the present application provides a method for intelligent video surveillance of a ship, the method comprising: Acquiring multiple modal data of the vessel, the multiple modal data including at least two of position modal data, monitoring modal data, environmental modal data, or sensor modal data; determining reference information, adaptively determining target modal data to be fused from the multiple modal data based on the reference information, and fusing the target modal data to obtain multimodal fusion data, wherein the reference information includes at least one of an environmental risk level, confidence levels of the multiple modal data, or mission requirements at different stages of ship navigation; Determining abnormal information based on the multimodal fusion data, a pre-built ship safety rule base, and an abnormality detection model, wherein the abnormal information includes abnormal information of ship safety equipment, abnormal information of ship personnel actions, abnormal information of sensors, and abnormal information of ship driving trajectory; The abnormal information is processed to obtain an abnormality level of the abnormal information, and alarm information is determined using the abnormality level and the abnormal information.

[0005] On the other hand, an embodiment of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the ship intelligent video surveillance method as described in any of the above embodiments is implemented.

[0006] On the other hand, an embodiment of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the ship intelligent video surveillance method as described in any of the above embodiments.

[0007] The technical solution provided by this application brings at least the following beneficial effects: This application adaptively selects target modal data to be fused based on reference information such as environmental risk levels, the confidence level of multimodal data, or mission requirements at different stages of a ship's voyage. This makes the fused data more targeted and effective, avoids interference from irrelevant or low-value data in the analysis, improves data fusion quality, reduces the amount of fused data, and increases data fusion efficiency. Furthermore, the multimodal fusion data includes multiple modal data such as location, monitoring, environment, and sensors, enabling the multimodal fusion data to more comprehensively and deeply reflect the ship's operating status. Anomaly information is determined using a pre-built ship safety rule library and anomaly detection model, and multimodal data is cross-validated to improve the accuracy of anomaly information determination. Anomaly information is processed to determine anomaly levels, and alarm information is determined using the anomaly level and anomaly information, enabling hierarchical management of anomalies. Alarms issued based on both anomaly levels and specific anomaly information can be more accurate and targeted, making it easier for managers to quickly understand the situation and take appropriate measures, thereby improving the safety and standardization of the ship's navigation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 Schematic diagram of a ship intelligent video surveillance system provided by an embodiment of the present application; Figure 2 Schematic diagram of a method for intelligent video surveillance of ships provided in an embodiment of the present application; Figure 3 It is a structural diagram of a computer system of a terminal device or server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0009] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0010] It should be noted that the terms "first," "second," and the like in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0011] Figure 1 Schematic diagram of a ship intelligent video surveillance system provided by an embodiment of the present application. Figure 1 As shown, the ship intelligent video monitoring system includes a data acquisition unit 101 , a data fusion unit 102 , a data processing unit 103 and an alarm unit 104 .

[0012] In an exemplary embodiment of the present application, the data acquisition unit 101 is used to acquire multiple modal data of the ship, and the multiple modal data include at least two of position modal data, monitoring modal data, environmental modal data or sensor modal data.

[0013] Among them, position modal data refers to information used to describe the specific location of a ship in geographic space and its related motion status. It can be obtained by the ship's Automatic Identification System (AIS) or the Beidou positioning system, including the ship's latitude and longitude, speed, heading, detection of surrounding ships, obstacle distances, and other data.

[0014] Surveillance modal data refers to data generated by video surveillance equipment installed in key areas of a vessel. This data records real-time scenes inside and outside the vessel in the form of images and videos, intuitively reflecting the activities of personnel, the operating status of equipment, cargo loading and unloading, and changes in the surrounding environment. For example, surveillance modal data includes video data from cameras monitoring areas such as the deck, engine room, and bridge in real time, as well as abnormal sounds (such as collisions and explosions) detected by microphone arrays.

[0015] Environmental modal data refers to various physical and meteorological parameters of the waters and surrounding environment in which the ship is located. Environmental modal data includes wind speed, visibility, wave height measured by meteorological sensors; water depth, water flow velocity and other data measured by hydrological sensors.

[0016] Sensor modal data refers to data related to the operating status of ship equipment and system performance, collected in real time by various sensors installed on ships. This data includes data such as smoke, temperature, and gas concentrations (such as CO2) measured by cabin sensors, and engine speed, oil pressure, and power system status measured by equipment status sensors.

[0017] In an exemplary embodiment of the present application, the data fusion unit 102 is used to determine reference information, adaptively determine target modal data to be fused among multiple modal data based on the reference information, fuse the target modal data to obtain multimodal fusion data, and the reference information includes at least one of the environmental risk level, the confidence of multiple modal data, or the task requirements of different stages of ship navigation.

[0018] The environmental risk level in the reference information can be calculated using formula (1): Formula (1) In the above formula (1), is the environmental risk level parameter at time t. N is the number of types of environmental parameters, for example, environmental parameters include wind speed, wave height, visibility, salt spray concentration, etc. is the weight coefficient of the i-th environmental parameter, which indicates the contribution of this parameter to the overall risk and can be obtained through expert evaluation, historical data analysis or machine learning training. is the vector of the i-th environmental parameter at time t. For example, if the environmental parameter is wind speed, then is a vector containing wind speed values. It is the environmental parameter vector under safe conditions. The environmental parameters under safe conditions are determined by ship design standards, historical safety data or industry specifications. is the Euclidean distance. is the variance of the environmental parameter of the i-th type. is the chaos weight factor. It is the maximum Lyapunov exponent, which is used to quantify the degree of chaos in the ship's motion trajectory and predict sudden risks (such as trajectory instability).

[0019] After determining the environmental risk level parameters, multiple level thresholds can be set and compared with the environmental risk level parameters to obtain the environmental risk level. For example, the environmental risk level can be divided into high risk level, medium risk level, and low risk level.

[0020] The confidence of multiple modal data in the reference information can be calculated by formula (2): Formula (2) In the above formula (2), is the confidence score of the i-th modal data. is a normalization function that converts the result into a probability distribution, is the total number of time steps. It is a bidirectional long short-term memory network used to process time series data. It is a spatial attention module that processes the spatial distribution characteristics of sensors. The data of the i-th mode at time t. Represents element-wise multiplication (Hadamard product). is the time decay function, is the time interval.

[0021] In an exemplary embodiment of the present application, the different stages of a ship's navigation include, but are not limited to, the departure stage, the berthing stage, the navigation stage, and the night or foggy navigation stage. Different navigation stages correspond to different mission requirements. Exemplarily, the mission requirements for the departure stage are obstacle detection, personnel safety, and equipment monitoring. The mission requirements for the departure stage are mapped into a multidimensional vector, i.e., X = [obstacle detection, personnel safety, equipment monitoring]. For example, T = [0.6, 0.3, 0.1]. Exemplarily, the mission requirements for the berthing stage are centimeter-level positioning, cable control, and personnel specifications. The mission requirements for the berthing stage are mapped into a multidimensional vector, i.e., X = [centimeter-level positioning, cable control, personnel specifications]. For example, T = [0.8, 0.1, 0.1].

[0022] In an exemplary embodiment of the present application, the process of adaptively determining target modality data to be fused among multiple modality data based on reference information includes steps A1 to A5.

[0023] In step A1 , the data level of each modal data is determined based on the environmental risk level and the mission requirements at different stages of the ship's voyage.

[0024] For example, after determining the multidimensional vectors corresponding to the environmental risk levels and the mission requirements at different stages of the ship's voyage, the data levels of different modal data are determined by the formula of the dynamic scoring model (i.e., formula (3)): Formula (3) In the above formula (3), is the score of the i-th modal data; The balance factor between environmental risk level and task requirements at different stages is used to adjust the weight of environmental risk and task requirements in the scoring; is the environmental risk level; is the effectiveness score of the i-th mode in historical high-risk scenarios. This score may be based on the performance of the mode in similar high-risk environments in historical data. For example, a radar effectiveness score of 0.9 in typhoons indicates that radars perform well in typhoon conditions in historical data and have high effectiveness. is the priority of task requirements, i.e., a multidimensional vector; is the fitness score of the i-th modal data for the current task.

[0025] In step A2, modal data having a data level greater than or equal to a level threshold is determined as first modal data.

[0026] For example, after determining the data level, the data is compared with a level threshold, and modal data having a data level greater than or equal to the level threshold is screened out as the first modal data. The level threshold can be determined based on the actual conditions of the vessel, and this application does not impose any restrictions thereon.

[0027] In step A3, the confidence level of the first modal data is determined, and the modal data whose confidence level of the first modal data is greater than or equal to a confidence threshold is determined as the target modal data.

[0028] For example, after determining the first modal data, the confidence of the first modal data can be further determined. The confidence of the first modal data can be determined based on formula (2). For details, see the relevant description of formula (2) and will not be repeated here. The confidence threshold is adaptively adjusted according to the environmental risk level and the mission requirements of the navigation phase. In high-risk environments or critical navigation phases, the confidence threshold can be appropriately increased to ensure that only the most reliable data is selected; in low-risk environments or routine navigation phases, the confidence threshold can be appropriately lowered to obtain more information. The first modal data is further screened using the confidence threshold to obtain the first modal data with a confidence greater than or equal to the confidence threshold, i.e., the target modal data.

[0029] In step A4, the fusion weight of the target modality data is determined based on the data level, the confidence level and the reference weight, and the reference weight is determined based on the historical navigation data of the ship system.

[0030] For example, the reference weight can be determined based on the historical navigation data of the ship system. For example, if the AIS signal is frequently lost in a certain sea area in the historical navigation data, its weight will be automatically reduced. The fusion weight of the target modal data = k1 × data level + k2 × confidence + k3 × reference weight, where k1, k2, and K3 are correction parameters that can be set based on actual conditions.

[0031] In step A5, the target modality data is fused based on the fusion weight to obtain multimodal fusion data.

[0032] In one embodiment, all target modal data are time-synchronized and spatially aligned to ensure data consistency. A weighted average is then calculated for each data point to obtain multimodal fusion data. It should be noted that the method for fusing target modal data in this application is an exemplary description. Kalman filter fusion, neural network fusion, etc. can also be used to fuse target modal data, and the application itself does not impose any restrictions on this.

[0033] This application determines data levels based on environmental risk levels and mission requirements during the vessel's voyage phase, screens the most valuable modal data, and fuses them using confidence levels and reference weights. This effectively reduces the impact of noise and errors, improving the accuracy and reliability of multimodal fusion data. Furthermore, environmental risk levels and mission requirements change dynamically during voyage phases, and adaptively adjusting data screening and fusion strategies based on these changes makes the system more adaptable.

[0034] In an exemplary embodiment of the present application, the ship intelligent video surveillance system also includes a data preprocessing unit, which performs image segmentation processing and target object recognition processing on the modal data in video format when the format of the modal data is a video format, to obtain first preprocessed data; the data fusion unit adaptively determines the target modal data to be fused in the first preprocessed data based on reference information.

[0035] For example, when the modal data format is video, keyframes are first selected from the video and image segmentation is performed on the keyframes. Image segmentation includes semantic segmentation, instance segmentation, and pixel-level segmentation. Semantic segmentation can be performed using the DeepLabV3+ model, trained using a ship-specific dataset, to obtain pixel-level classification labels (e.g., ocean, sky, ship, coastline). Instance segmentation can be performed using Mask R-CNN to separate multiple crew members or equipment targets. Pixel-level segmentation can be performed using the U-Net model to mark hazardous material leak areas. Target recognition is then performed based on the segmentation results and a target recognition model to identify human behavior, navigation equipment, marine obstacles, etc., to obtain first preprocessed data. The target recognition model can include YOLOv8, CRNN, etc. It should be noted that the process by which the data fusion unit adaptively determines the target modal data to be fused from the first preprocessed data based on reference information is similar to the process by which the data fusion unit adaptively determines the target modal data to be fused from multiple modal data based on reference information, and will not be further described here.

[0036] This application adds image segmentation and target recognition preprocessing steps to the video modality data, which can effectively filter out key target information such as ship personnel and equipment in the monitoring data, reduce interference from irrelevant information, and make the fused data more focused on factors related to ship prevention and control, thereby improving the data processing efficiency of the entire system.

[0037] In an exemplary embodiment of the present application, the data preprocessing unit can also perform integrity detection on the target object in the modal data in the video format to obtain a detection result, where the target object includes ship safety equipment and ship personnel; when the detection result indicates that the target object is occluded, the supplementary features are determined; the target object is completed based on the supplementary features to obtain second preprocessed data; the data fusion unit adaptively determines the target modal data to be fused in the second preprocessed data based on the reference information.

[0038] Specifically, based on the first preprocessed data, a suitable integrity detection model, such as the YOLO-OCC model, is selected to perform target integrity detection. If there is no occlusion, the first preprocessed data is used; if there is occlusion, an occlusion threshold can be set to determine whether it is partial occlusion or severe occlusion. For example, when the visibility of the key feature is greater than 30%, it is considered to be partial occlusion, and when it is less than or equal to 30%, it is considered to be severe occlusion. For partial occlusion, a repair method can be used to repair the occluded target, such as using the EdgeConnect network to restore the occluded texture, or using the Point Completion Network (PCN) to generate the missing part of the 3D point cloud. For severe occlusion, other data can be used for supplementation. For example, safety equipment occlusion can be coordinated across cameras to perform multi-view geometric consistency verification. After supplementing the features of the occluded target object, the second preprocessed data is obtained. It should be noted that the process in which the data fusion unit adaptively determines the target modal data to be fused in the second preprocessed data based on the reference information is similar to the process in which the data fusion unit adaptively determines the target modal data to be fused in multiple modal data based on the reference information, and will not be repeated here.

[0039] By performing integrity detection on the target object, this application can timely discover occlusion problems in the data and take corresponding measures to complete the data according to different occlusion degrees, which can effectively improve the integrity and quality of the target modal data to be fused. In the subsequent process, when the target modal data to be fused is used for fusion and abnormal information is determined, the accuracy and reliability of the abnormal information judgment can be improved and the false alarm rate of abnormal information can be reduced.

[0040] In an exemplary embodiment of the present application, the data preprocessing unit can also cross-validate multiple modal data to obtain cross-validation results; determine abnormal data of the multiple modal data in the cross-validation results; delete the abnormal data in the multimodal data to obtain third preprocessed data; and the data fusion unit adaptively determines the target modal data to be fused in the third preprocessed data based on the reference information.

[0041] For example, multiple modal data sets are grouped according to certain rules, such as by time series or data source. For each data set, the performance of different modal data sets under the same conditions is compared. For example, within the same time period, the ship's position data, speed data, and heading data can be compared.

[0042] Cross-validation is performed on multiple modal data to check for significant conflicts or inconsistencies between them, generating cross-validation results. For example, if the position data indicates a ship is stationary in a port, but the speed data indicates a higher speed, this could indicate a possible data anomaly. Data consistency can be determined by setting thresholds or rules. For example, if the speed data exceeds a reasonable range and does not match the position data, an anomaly is considered present.

[0043] In one embodiment, a reasonable threshold range is set for each modal data type based on cross-validation results and the statistical characteristics of historical data. For example, for ship speed data, a normal speed range is set based on the ship type and navigation environment. If the speed data exceeds this range, it is determined to be abnormal data.

[0044] In another embodiment, data variation patterns are observed to identify data that does not conform to normal patterns. For example, ship location data typically exhibits continuous, smooth variations. Sudden jumps or discontinuous variations may indicate abnormal data. Data mining algorithms, such as cluster analysis and time series analysis, can be used to identify data patterns.

[0045] A comprehensive assessment of the data is conducted by combining multiple methods. For example, not only is the data considered to determine whether it exceeds the threshold, but factors such as the data's changing trend and its correlation with other modal data are also considered. If a data point does not exceed the threshold, but its changing trend is significantly different from other data points and its correlation with other modal data is also abnormal, it can also be identified as an outlier.

[0046] Select an appropriate correlation metric based on the type and characteristics of the modal data. For continuous data, commonly used correlation metrics include the Pearson correlation coefficient and the Spearman correlation coefficient. For example, for ship position and speed data, the Pearson correlation coefficient can be used to measure the linear correlation between them.

[0047] Optionally, before calculating the correlation, the data needs to be preprocessed to ensure data quality and consistency, for example, by performing missing value processing, outlier processing, and normalization on the data.

[0048] Using the selected correlation metric, calculate the correlation coefficient between the different modal data. Correlation coefficients typically range from -1 to 1, with values ​​closer to 1 indicating stronger correlation and values ​​closer to 0 indicating weaker correlation. For example, if the Pearson correlation coefficient for position and velocity data is close to 1, it indicates a strong positive correlation between them.

[0049] A simple linear regression model is built based on the correlation to predict the value of one modality based on the data of another modality. For example, if position data is used to predict speed data, if the actual speed data deviates significantly from the predicted value, it may be an anomaly and thus identified as abnormal data.

[0050] After determining the abnormal data, the abnormal data is deleted from the multimodal data to obtain third preprocessed data. It should be noted that the process by which the data fusion unit adaptively determines the target modal data to be fused from the third preprocessed data based on the reference information is similar to the process by which the data fusion unit adaptively determines the target modal data to be fused from the multiple modal data based on the reference information, and will not be further described here.

[0051] This application identifies abnormal data through cross-validation and multiple judgment methods, avoiding the limitations of single-modal data detection and enabling more comprehensive and accurate identification of anomalies in the data. Calculating correlations between different modal data and establishing linear regression models can reveal the inherent connections between the data, providing a basis for further data analysis and decision-making. For ship monitoring systems, comprehensive analysis of multiple modal data can provide a more comprehensive understanding of the ship's operating status and promptly identify potential safety hazards.

[0052] In an exemplary embodiment of the present application, the data processing unit 103 is used to determine abnormal information based on multimodal fusion data, a pre-built ship safety rule base and an anomaly detection model. The abnormal information includes abnormal information of ship safety equipment, abnormal information of ship personnel actions, abnormal information of sensors and abnormal information of ship driving trajectory.

[0053] For example, the pre-built ship safety rule library includes but is not limited to equipment inspection rules, personnel behavior rules, navigation rules, sensor calibration rules, etc. For example, the equipment inspection rule requires that the number of life jackets is greater than or equal to the number of crew members. If it is detected that the number of life jackets is less than the number of crew members, the ship safety equipment abnormality information is determined. Personnel behavior rules, such as smoking is prohibited in hazardous chemical cabins, can be used to determine abnormal information on the ship personnel's actions based on the hot spots detected by the infrared camera and the excessive CO2 detected by the smoke sensor in the multimodal fusion data. Navigation rules, such as the angle of the heading deviation from the planned route is less than or equal to 5 degrees, can be used to obtain whether the current heading angle is within the rule range through trajectory information. If not, the ship's trajectory abnormality information is determined. Sensor calibration rules, such as the difference in position measured by the radar and AIS is greater than 100m and lasts for more than 30s, the ship's trajectory abnormality information is determined.

[0054] It should be noted that the above rules in the pre-built ship safety rule library can be static rules or dynamic rules. For example, when sailing at night, the safety equipment detection threshold corresponding to the equipment inspection rule is reduced from 0.8 to 0.7 to reduce the missed detection rate.

[0055] In an exemplary embodiment of the present application, the process of determining abnormal information may include: using a pre-built ship safety rule library to perform anomaly detection on multimodal fusion data to obtain first abnormal information; using an anomaly detection model to perform anomaly detection on multimodal fusion data to obtain second abnormal information and abnormal probability; determining abnormal information based on the first abnormal information and the second abnormal information.

[0056] For example, any rule in the pre-built ship safety rule library can have a parameter threshold or parameter range. The multimodal fused data is parsed to extract data fields relevant to each rule in the ship safety rule library. For example, data such as the ship's current speed, lifeboat engine operating time, liferaft positioning device power level, personnel operation command records, sensor readings (such as gyroscope and accelerometer data), and ship position and heading can be extracted from the fused data.

[0057] The extracted data is matched against the rules in the ship safety rule library one by one. If the data does not meet the conditions of a particular rule, the corresponding safety item is considered abnormal. The abnormal rule number, abnormal data value, and abnormal description are recorded to form the first abnormality information. For example, if the ship's current speed exceeds the speed limit in a certain water area, the first abnormality information will record "Rule number: XX, abnormal item: ship speed, abnormal value: current speed A knots (speed limit B knots)".

[0058] The multimodal fusion data to be detected is input into a trained anomaly detection model. The model outputs the reconstructed data and the error probability. Based on a preset reconstruction error threshold, the input data is determined to be anomaly data. If the error probability is greater than the preset reconstruction error threshold, the data is considered anomaly. The features and anomaly probability of the anomaly data are recorded to obtain the second anomaly information.

[0059] Align and correlate the first and second anomaly information to identify any corresponding anomalies. For example, if the first anomaly information records a "low battery life raft positioning device" anomaly, and the second anomaly information records "abnormal data from a sensor related to a safety device," and further analysis reveals that the sensor is related to battery life raft positioning device detection, then these two anomaly information are correlated.

[0060] The associated anomaly information is integrated, removing duplicate anomaly descriptions to form a comprehensive anomaly information list. For the same anomaly, if the two detection methods provide different details, these details are merged. For different anomalies, if the second anomaly information corresponds to a low probability of anomaly, auxiliary information (such as visual inspection and sending a message to the corresponding crew member to confirm the existence of an anomaly) is obtained to determine whether it is an anomaly. If the first and second anomaly information differ, both are determined to be anomalies.

[0061] Then, according to the predefined abnormality types (abnormal information of ship safety equipment, abnormal information of ship personnel actions, abnormal information of sensors, and abnormal information of ship driving trajectory), the integrated abnormal information is classified to obtain the final abnormal information.

[0062] The ship safety rule library proposed in this application is based on clear industry standards and can quickly detect anomalies that conform to known rules, with high interpretability and reliability. The anomaly detection model, by learning the characteristic distribution of large amounts of data, can discover potential anomaly patterns that are difficult to describe with rules. The two, when used together, complement each other, improving the accuracy of anomaly detection, reducing missed detections and false positives, ensuring full coverage of anomaly detection, and enhancing safety and reliability during navigation.

[0063] In an exemplary embodiment of the present application, historical maintenance information and equipment parameter information of the ship can also be obtained through a data acquisition unit (not shown in the figure); the data processing unit 103 can determine abnormal information based on historical maintenance information, equipment parameter information, multimodal fusion data, a pre-built ship safety rule base and anomaly detection model.

[0064] Exemplarily, the data acquisition unit interfaces with the vessel's maintenance management system, equipment management platform, and paper maintenance record digitization system to obtain historical vessel maintenance information and equipment parameter information. For example, structured data such as vessel scheduled maintenance records, equipment repair records, and parts replacement records can be retrieved from the maintenance management system through an application programming interface (API). Alternatively, optical character recognition (OCR) technology can be used to scan and recognize paper maintenance records, converting image data into processable text data to obtain historical maintenance information and equipment parameter information.

[0065] The process of identifying abnormal information using the pre-built ship safety rule base and anomaly detection model has been described above and will not be repeated here. Anomalies are further screened using historical maintenance information and equipment parameter information to obtain filtered abnormal information.

[0066] The historical maintenance information and equipment parameter information of the ship in this application are used to judge abnormal information, which helps to more accurately identify potential problems such as aging failures of ship safety equipment and abnormal equipment performance, and can more deeply explore potential safety hazards of the ship, enriching the functions and accuracy of the ship's prevention and control system.

[0067] In an exemplary embodiment of the present application, the multimodal fusion data may also be processed based on a photoelectric attenuation factor to obtain abnormal information, wherein the light attenuation factor and the electrical signal attenuation factor corresponding to the multimodal fusion data.

[0068] In one embodiment, multimodal fusion data is processed based on a photoelectric attenuation factor to obtain abnormal information, including: calculating the illumination attenuation factor of the multimodal fusion data through a nonlinear illumination attenuation model; calculating the electrical signal attenuation factor of the multimodal fusion data through a frequency domain response model; compensating the multimodal fusion data by the illumination attenuation factor and the electrical signal attenuation factor, respectively, to obtain compensated multimodal fusion data; calculating a comprehensive weight of the compensated multimodal fusion data based on the illumination attenuation factor, the electrical signal attenuation factor, the confidence of the multimodal fusion data, and the environmental risk level; generating an anti-attenuation multimodal fusion feature matrix based on the comprehensive weight and the compensated multimodal fusion data; and inputting the anti-attenuation multimodal fusion feature matrix into a pre-trained 3D convolutional anomaly detection network to obtain abnormal information.

[0069] Among them, the comprehensive weight of the compensated multimodal fusion data The calculation formula is as follows: Formula (4) In the above formula (4), Represents the confidence of multimodal fusion data, Indicates the environmental risk level, represents the light attenuation factor, represents the electrical signal attenuation factor, Represents the sum of the light attenuation factor and the electrical signal attenuation factor.

[0070] This embodiment specifically compensates for data attenuation and improves data quality. Specifically, it uses a nonlinear light attenuation model and a frequency domain response model to calculate the light attenuation factor and the electrical signal attenuation factor, respectively. This approach specifically addresses two core attenuation issues in ship monitoring. Regarding light attenuation, ships experience complex and variable lighting conditions in various navigation environments (e.g., at night, during heavy rain, and in tunnels and bridges). The nonlinear model more accurately fits the attenuation patterns of light intensity with distance and the medium (e.g., fog and seawater splash), thus avoiding issues such as blurry surveillance videos and feature loss caused by insufficient or overexposure. Regarding electrical signal attenuation, the electrical signals from sensors (e.g., cameras and radars) are susceptible to electromagnetic interference and noise during transmission. The frequency domain response model analyzes the signal attenuation characteristics in the frequency domain (e.g., high-frequency signals attenuate faster) to effectively filter out noise, restore signal integrity, and enhance the reliability of sensor data. Compensating for raw multimodal data (e.g., video frames and sensor signals) based on the attenuation factor directly corrects distortion during data acquisition, providing clearer and more accurate input data for subsequent anomaly detection.

[0071] Furthermore, this embodiment employs dynamic weighted fusion, adapting to different environments and data credibility. Data confidence reflects the reliability of the multimodal data itself (e.g., whether the camera focus is normal or the sensor calibration status), preventing errors introduced by fusing low-credibility data. Weights are dynamically adjusted based on the environmental risk level (e.g., port calls, complex sea conditions), increasing the weight of key data (e.g., ship position, personnel movements) in high-risk scenarios to ensure that anomaly detection priorities match actual needs. The attenuation factor directly correlates to the degree of data damage; higher attenuation results in lower weights, avoiding overreliance on damaged data. This achieves an adaptive fusion strategy of "heavy weighting of high-quality data and light weighting of low-quality data." This mechanism enables the system to dynamically balance the contributions of each modality in complex environments (e.g., inclement weather or areas with strong electromagnetic interference), enhancing the robustness of multimodal fusion.

[0072] Furthermore, this embodiment constructs an attenuation-resistant feature matrix to enhance the generalization capabilities of the anomaly detection model. By weightedly fusing the compensated data using comprehensive weights, this generates an attenuation-resistant multimodal fusion feature matrix. This matrix integrates spatiotemporal information (such as the temporal continuity of video frame sequences and the spatial distribution of sensor data) and suppresses the effects of noise and distortion through attenuation compensation and dynamic weighting. When input into a 3D convolutional anomaly detection network, the attenuation-resistant feature matrix more clearly presents anomaly patterns (such as the spatiotemporal trajectory of abnormal human actions and the mutation trend of sensor data), reduces feature ambiguity or misjudgment caused by data attenuation, and improves the model's sensitivity and detection accuracy for subtle anomalies (such as early equipment failures and illegal human operation).

[0073] Furthermore, this embodiment enhances the safety and reliability of the ship monitoring system. Addressing common issues in ship monitoring, such as illumination variations (e.g., day-night cycles, inclement weather) and electrical signal interference (e.g., complex electromagnetic environments at sea), this solution improves the availability of multimodal data from the source through data compensation and dynamic weighting, thereby reducing the missed and false alarm rates of anomaly detection. By combining weighting based on environmental risk levels and data confidence, the system can more accurately focus on critical data in high-risk scenarios, meeting the stringent real-time and accuracy requirements of ship safety monitoring, and providing more reliable technical support for ship navigation safety, equipment operation and maintenance, and personnel management.

[0074] In an exemplary embodiment of the present application, the alarm unit 104 is configured to process the abnormal information, obtain the abnormal level of the abnormal information, and determine the alarm information using the abnormal level and the abnormal information.

[0075] Exemplarily, the process of determining the alarm information includes but is not limited to steps B1 to B3.

[0076] Step B1: Process the abnormal information and determine the evaluation parameters, which include at least one of the risk diffusion speed, potential loss level or handling urgency.

[0077] Risk diffusion rates include, but are not limited to, hazardous chemical leakage rates and the time to collision (TCPA). For example, data from hazardous chemical leakage sensors can be used to calculate leakage rates using fluid dynamics models, combining the physical properties of the hazardous chemical (such as density and viscosity) and the size and shape of the leak. For example, for small-hole leaks, the Bernoulli equation can be used to calculate the leakage rate, taking into account the impact of changes in pressure, temperature, and other factors during the leak process.

[0078] Another example is using the Automatic Identification System (AIS) to obtain information such as the position, speed, and heading of the own ship and surrounding ships, and using geometric algorithms to calculate the relative motion trajectory and collision time of the two ships. For example, using vector operations, based on the position vectors and velocity vectors of the two ships, the rate of change of their distance in the collision direction is calculated, thereby obtaining the TCPA.

[0079] Potential loss levels include, but are not limited to, casualty risk, cargo value loss, and environmental pollution. For example, a risk assessment model (such as a Monte Carlo simulation-based casualty risk model) can be used to predict potential casualties, combining ship personnel distribution data (such as the number of crew members and passenger distribution by area) with the location and type of anomaly. For example, the impact of toxic gas diffusion on personnel under different leakage scenarios can be simulated, and the probability of casualty calculated based on exposure time and concentration can be converted into a casualty risk level.

[0080] For example, detailed information about the ship's cargo (such as cargo type, quantity, and value) is obtained to analyze potential damage to the cargo caused by anomalies (such as damage caused by fire, flooding, or collision). Based on the extent of damage and market value, the cargo's loss is estimated and categorized into different loss levels.

[0081] For example, environmental diffusion models (such as oil and chemical diffusion models) are used to predict the spread and concentration distribution of pollutants in the ocean, taking into account the type of hazardous chemical, the amount leaked, the leak location, and the surrounding marine environment (such as currents, tides, and marine ecologically sensitive areas). Based on the predicted results and in conjunction with environmental protection standards, the degree of environmental pollution is assessed.

[0082] The urgency of the response can be used to determine the remaining safe response time window. For example, based on the type and development trend of the anomaly, combined with the ship's emergency response capabilities (such as the availability of firefighting equipment and the response time of emergency rescue personnel), the upper limit of the time from the discovery of the anomaly to the implementation of effective response measures is determined, i.e., the remaining safe response time window. For example, in the case of a ship fire, the fire spread rate and the distribution of combustible materials, combined with the firefighting capabilities of the ship's firefighting equipment, can be used to calculate the time required for firefighting operations to prevent the fire from getting out of control.

[0083] Step B2: Determine the abnormality level using the evaluation parameters.

[0084] In one embodiment, determining the abnormality level using the evaluation parameters may include the following: Construct a hierarchical model: Decompose the risk assessment problem into a target layer (determining the anomaly level), a criterion layer (risk diffusion rate, potential loss level, and urgency of action), and a solution layer (different anomaly scenarios). Invite ship safety experts, risk management experts, and others to compare each criterion layer parameter pairwise using a 1-9 scale based on their impact on the anomaly level, constructing a judgment matrix. For example, if the experts believe that the risk diffusion rate has a slightly greater impact on the anomaly level than the potential loss level, the corresponding position in the judgment matrix will be assigned a value of 3.

[0085] The maximum eigenvalue and corresponding eigenvector of the judgment matrix are calculated using the eigenvalue method. The eigenvector is normalized to obtain the weight vector of each evaluation parameter. A consistency check is then performed to ensure that the consistency ratio (CR) of the judgment matrix is ​​less than 0.1 to ensure the rationality of the weight distribution.

[0086] A comprehensive risk value is calculated based on the quantitative values ​​and weight vectors of each assessment parameter. For example, the comprehensive risk value = 0.6 × the quantitative value of the risk diffusion rate + 0.3 × the quantitative value of the potential loss level + 0.1 × the quantitative value of the urgency of action. Based on the size of the comprehensive risk value, anomalies are classified into different levels, such as Level 1 (low risk), Level 2 (medium-low risk), Level 3 (medium risk), Level 4 (medium-high risk), and Level 5 (high risk).

[0087] In another embodiment, determining anomaly levels using evaluation parameters may include utilizing historical anomaly data, including evaluation parameters (such as risk diffusion rate, potential loss level, and urgency of action) and corresponding anomaly level labels. Feature extraction and selection are performed on this data to construct a multimodal feature vector. For example, in addition to evaluation parameters, sensor data trends (such as the rate of change of parameters like temperature and pressure) and crew location distribution may also be considered.

[0088] Select the random forest classifier as the machine learning model and train it using the prepared dataset. Adjust the model's hyperparameters (such as the number of trees and maximum depth) through methods such as cross-validation to improve the model's classification accuracy and generalization ability.

[0089] The evaluation parameters and multimodal features of the anomaly to be evaluated are input into the trained random forest classifier, and the model outputs the corresponding risk level (Level 1-5).

[0090] Optionally, the alarm unit 104 may also modify the abnormality level based on the time and environment of the abnormality information, wherein the modification includes time factor modification and environmental factor modification.

[0091] Correction for time factors: The system obtains current time and environmental information from the ship's clock system and meteorological sensors. When it is nighttime (e.g., based on sunset and sunrise times) or inclement weather (e.g., wind speeds reaching a certain level, visibility falling below a certain threshold, or wave heights exceeding a certain value), the system adjusts the severity of specific anomalies. For example, the risk threshold for falls might be set to Level 2 under normal conditions, but automatically lowered to Level 1 at night or in inclement weather. This is because at night or in inclement weather, visibility is limited, mobility is impaired, and the likelihood of falls increases, necessitating earlier alerts.

[0092] Environmental correction: Utilizing a vessel's navigation system and geographic information system (GIS), the system identifies whether the vessel is currently in sensitive waters, such as ecologically protected areas, military restricted areas, or busy shipping lanes. In sensitive waters, the vessel's track deviation tolerance is reduced by 50%. For example, in normal waters, the permitted track deviation range is ±100 meters, while in sensitive waters, it is adjusted to ±50 meters. If a vessel's track deviation exceeds the adjusted tolerance, the anomaly level is raised accordingly, allowing timely action to prevent impacts on sensitive areas.

[0093] Step B3: determining alarm information based on one of the abnormality level, the occurrence time of the abnormality information, the occurrence location of the abnormality information, or the affected area of ​​the abnormality information. The alarm information includes an alarm mode and an alarm area.

[0094] For example, a mapping relationship is established between the modified abnormality level and the alert method. For example, for a Level 1 abnormality, a gentle prompt is issued on the system interface; for a Level 2 abnormality, an audible alert is added; for a Level 3 abnormality, a text message is sent to relevant personnel; for a Level 4 abnormality, an email is sent and the ship's broadcast system is activated; for a Level 5 abnormality, in addition to the above methods, an emergency distress signal is sent to the nearby maritime rescue center and ship management department.

[0095] Furthermore, personalized alert methods are set up for different positions, taking into account the responsibilities and permissions of different personnel. For example, for the captain and the officer on duty, in addition to the conventional alert method, real-time push notifications are also sent via handheld terminals to ensure that they can receive important warning information immediately.

[0096] For example, the alarm area is determined based on the location of the abnormal information and its possible impact range. For example, if the abnormality is a fire in a certain area of ​​the ship, the alarm area should cover the fire area and the surrounding cabins, passages, and equipment areas that may be affected.

[0097] The impact range is then optimized based on environmental factors. For example, the warning area is optimized by taking into account factors such as the ship's structural layout, ventilation system, and evacuation routes. For example, when determining a fire warning area, the warning range in the downwind area is appropriately expanded based on wind and ventilation direction to ensure timely evacuation of personnel to a safe area. Furthermore, when an abnormality occurs in a sensitive sea area, the warning area should also include surrounding waters that may have an impact on the sensitive area, so as to alert nearby ships to avoid it.

[0098] This application comprehensively considers multiple evaluation parameters such as the speed of risk diffusion, potential loss level and urgency of handling, and can more comprehensively and accurately assess the severity of the anomaly, avoiding the problem of inaccurate alarms caused by relying solely on a single factor. In addition, the anomaly level is corrected in combination with the time and environment in which the anomaly information occurs, which can adapt to changes in risks under different circumstances and ensure the timeliness of the alarm. Determining different alarm methods based on the corrected anomaly level can enable relevant personnel to take appropriate actions based on the urgency of the alarm, thereby improving the pertinence of the emergency response. For example, for high-risk anomalies, using multiple alarm methods to notify relevant personnel and management departments at the same time can quickly mobilize resources from all parties for handling and reduce accident losses. Determining the alarm area based on the location of the anomaly and the scope of impact, and optimizing it in combination with environmental factors can enable the alarm information to be more accurately conveyed to areas and personnel that may be affected, avoiding unnecessary panic and waste of resources.

[0099] In an exemplary embodiment of the present application, the ship intelligent video surveillance system further includes a storage unit (not shown in the figure), which is used to format the multimodal data and store the formatted multimodal data.

[0100] Exemplarily, the storage unit first formats the multimodal data to ensure that the data can be efficiently stored and retrieved. The main operations include adding a unified timestamp (UTC time, millisecond accuracy) to all data to solve the problem of clock asynchrony among multiple devices; unifying the spatial coordinates of the data, such as converting the location data of the Beidou positioning system, AIS, radar, etc. into the WGS-84 standard coordinate system; and then storing the data in layers. The original data of the last 72 hours can be stored by a solid-state drive (SSD), and the compressed structured data within a certain period, such as data within 30 days, can be stored with the database. Data for longer periods can be stored using Alibaba Cloud and blockchain.

[0101] This application improves the availability of data by formatting multimodal data through storage units, and builds a data management system of "real-time efficiency - medium-term availability - long-term trustworthiness" through unified time and space benchmarks and hierarchical storage, thereby optimizing storage efficiency.

[0102] In an exemplary embodiment of the present application, the storage unit is also used to encode the abnormal modal data corresponding to the abnormal information, obtain a content identifier corresponding to the abnormal modal data, associate the content identifier with the reference modal data, obtain associated data, and store the associated data. The reference modal data is data with a different data modality from the abnormal modal data and generated at the same time.

[0103] Exemplarily, after the data processing unit determines the abnormal information, the storage unit first encodes the abnormal modal data corresponding to the abnormal information. For example, if the abnormal information is discovered by monitoring modal data (such as a camera capturing abnormal operational behavior of a crew member), then the monitoring modal data becomes abnormal modal data. Encoding is the process of converting the abnormal modal data into a unique content identifier using a specific algorithm or rule. This content identifier can be a string of numbers, letters, or a combination thereof, used to uniquely identify the content characteristics of the abnormal modal data. For example, for video data (monitoring modal data) of a specific crew member's abnormal operational behavior, a specific identifier is generated through an encoding algorithm. This identifier can reflect the key characteristic information of the operational behavior. The encoding algorithm can optionally use a hash algorithm. The hash algorithm can convert input data of any length into a fixed-length hash value through a specific hash function. This hash value can then be used as a content identifier.

[0104] Reference modal data is data that has a different data modality than the abnormal modal data and is generated at the same time. For example, if the abnormal modal data is monitoring modal data, then the reference modal data may be position modal data (such as the ship's latitude and longitude, speed, etc. at the time), environmental modal data (such as the wind speed and direction at the time), or sensor modal data (such as the engine's oil pressure and speed at the time) acquired at the same time. The storage unit associates the generated content identifier with the corresponding reference modal data to form associated data. This association is based on time synchronization, ensuring that the abnormal modal data and other related modal data correspond in the time dimension, facilitating subsequent comprehensive analysis. For example, the content identifier representing the crew's abnormal operational behavior is associated with reference modal data such as the ship's position, environmental conditions, and equipment operating parameters at the same time.

[0105] The storage unit stores the obtained associated data. This storage method can be a database storage method, where the associated data is stored in a certain structure and format in the database to facilitate subsequent query, retrieval, and analysis; cloud storage and blockchain storage can also be used.

[0106] This application associates and stores the content identifier of abnormal modal data with the reference modal data. When analyzing the abnormal information later, multiple modal data related to the abnormality can be quickly obtained, which facilitates the comprehensive analysis of the abnormal information. It organically combines data of different modalities but related to time to form a complete data set about the abnormal event. This storage method ensures the integrity and systematicness of the data. For abnormal events during the operation of the ship, the storage of associated data provides good traceability, and the introduction of content identifiers makes data retrieval more convenient.

[0107] This application adaptively selects target modal data to be fused based on reference information such as environmental risk levels, the confidence level of multimodal data, or mission requirements at different stages of a ship's voyage. This makes the fused data more targeted and effective, avoids interference from irrelevant or low-value data in the analysis, improves data fusion quality, reduces the amount of fused data, and increases data fusion efficiency. Furthermore, the multimodal fusion data includes multiple modal data such as location, monitoring, environment, and sensors, enabling the multimodal fusion data to more comprehensively and deeply reflect the ship's operating status. Anomaly information is determined using a pre-built ship safety rule library and anomaly detection model, and multimodal data is cross-validated to improve the accuracy of anomaly information determination. Anomaly information is processed to determine anomaly levels, and alarm information is determined using the anomaly level and anomaly information, enabling hierarchical management of anomalies. Alarms issued based on both anomaly levels and specific anomaly information can be more accurate and targeted, making it easier for managers to quickly understand the situation and take appropriate measures, thereby improving the safety and standardization of the ship's navigation process.

[0108] Based on the above Figure 1 The system shown, an embodiment of the present application provides a method for intelligent video monitoring of ships. Figure 2 201 to 204.

[0109] Step 201: Acquire multiple modal data of a ship, where the multiple modal data include at least two of position modal data, monitoring modal data, environmental modal data, or sensor modal data.

[0110] Step 202: Determine reference information, adaptively determine target modal data to be fused from multiple modal data based on the reference information, fuse the target modal data to obtain multimodal fusion data, and the reference information includes at least one of the environmental risk level, the confidence of multiple modal data, or the mission requirements of different stages of ship navigation.

[0111] Step 203: Determine abnormal information based on the multimodal fusion data, the pre-built ship safety rule base, and the abnormality detection model. The abnormal information includes abnormal information of ship safety equipment, abnormal information of ship personnel actions, abnormal information of sensors, and abnormal information of ship driving trajectory.

[0112] Step 204: Process the abnormal information to obtain the abnormal level of the abnormal information, and determine the alarm information using the abnormal level and the abnormal information.

[0113] It should be noted that step 201 has been described in detail in the relevant content of the data acquisition unit 101, step 202 has been described in detail in the relevant content of the data fusion unit 102, the content of step 203 has been described in detail in the relevant content of the data processing unit 103, and the content of step 204 has been described in detail in the relevant content of the alarm unit 104, and will not be repeated here.

[0114] This application adaptively selects target modal data to be fused based on reference information such as environmental risk levels, the confidence level of multimodal data, or mission requirements at different stages of a ship's voyage. This makes the fused data more targeted and effective, avoids interference from irrelevant or low-value data in the analysis, improves data fusion quality, reduces the amount of fused data, and increases data fusion efficiency. Furthermore, the multimodal fusion data includes multiple modal data such as location, monitoring, environment, and sensors, enabling the multimodal fusion data to more comprehensively and deeply reflect the ship's operating status. Anomaly information is determined using a pre-built ship safety rule library and anomaly detection model, and multimodal data is cross-validated to improve the accuracy of anomaly information determination. Anomaly information is processed to determine anomaly levels, and alarm information is determined using the anomaly level and anomaly information, enabling hierarchical management of anomalies. Alarms issued based on both anomaly levels and specific anomaly information can be more accurate and targeted, making it easier for managers to quickly understand the situation and take appropriate measures, thereby improving the safety and standardization of the ship's navigation process.

[0115] Figure 3 This is a schematic diagram of the structure of a client provided in an embodiment of the present application. In an exemplary embodiment of the present application, Figure 3 As shown, the client may include: at least one processor 410, at least one network interface 420, a user interface 430, a bus 440, and a memory 450. The various components in the client are coupled together via bus 440. It will be appreciated that bus 440 is used to enable communication between these components. In addition to a data bus, bus 440 also includes a power bus, a control bus, and a status signal bus.

[0116] Exemplarily, the processor 410 may be an integrated circuit chip having signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc., wherein the general-purpose processor may be a microprocessor or any conventional processor, etc.

[0117] The user interface 430 may include one or more output devices 431, including one or more speakers and / or one or more visual display screens. The user interface 430 may also include one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0118] Memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, and the like. Memory 450 may optionally include one or more storage devices physically remote from processor 410. Memory 450 may include volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory may be read-only memory (ROM), and volatile memory may be random access memory (RAM). Memory 450 described in the embodiments of the present application is intended to include any suitable type of memory. In some embodiments, memory 450 is capable of storing data to support various operations. Examples of such data include programs, modules, and data structures, or subsets or supersets thereof. The memory 450 may include an operating system, a system program for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks; the memory 450 may include a network communication module, for reaching other determined devices via one or more (wired or wireless) network interfaces 420. Exemplarily, the network interface 420 includes: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.; the memory 450 may include a presentation module, for enabling information to be presented (for example, a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 associated with the user interface 430 (for example, a display screen, a speaker, etc.).

[0119] In some exemplary embodiments, the ship intelligent video surveillance device provided in this application can be implemented in software. Figure 1The intelligent video surveillance device for ships stored in memory 450 is shown. This device, which may be software in the form of a program or plug-in, includes one or more of the following software processing units (modules): a data acquisition unit 101, a data fusion unit 102, a data processing unit 103, and an alarm unit 104. These modules are logical and can be arbitrarily combined or further separated based on the functions they implement. The functions and effects of each module have been described in detail above and will not be repeated here.

[0120] In other exemplary embodiments, the ship intelligent video surveillance device provided in this application may be implemented in hardware. As an example, the ship intelligent video surveillance device provided in this application may be a processor in the form of a hardware decoding processor, which is programmed to execute the ship intelligent video surveillance method provided in the embodiments of this application. For example, the processor in the form of a hardware decoding processor may be one or more application-specific integrated circuits (ASICs), digital signal processing (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components. The ship intelligent video surveillance method provided in the exemplary embodiments of this application is primarily executed by a client, and specifically, the client may be implemented by running the various computer programs described above.

[0121] The above-described specific embodiments further illustrate the objectives, technical solutions, and beneficial effects of the present invention. It should be understood that the above description is merely a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A ship intelligent video monitoring system, characterized in that: The system comprises: A data acquisition unit, configured to acquire multiple modal data of the vessel, wherein the multiple modal data include at least two of position modal data, monitoring modal data, environmental modal data, or sensor modal data; a data fusion unit, configured to determine reference information, adaptively determine target modal data to be fused from the multiple modal data based on the reference information, and fuse the target modal data to obtain multimodal fusion data, wherein the reference information includes at least one of an environmental risk level, confidence levels of the multiple modal data, or mission requirements at different stages of a ship's voyage; a data processing unit, configured to determine abnormal information based on the multimodal fusion data, a pre-built ship safety rule base, and an abnormality detection model, wherein the abnormal information includes abnormal information of ship safety equipment, abnormal information of ship personnel actions, abnormal information of sensors, and abnormal information of ship driving trajectory; The alarm unit is used to process the abnormal information to obtain the abnormal level of the abnormal information, and determine the alarm information using the abnormal level and the abnormal information.

2. The system according to claim 1, wherein: The data fusion unit is used to: Determining the data level of each modal data based on the environmental risk level and the mission requirements at different stages of the ship's voyage; Determining the modal data whose data level is greater than or equal to the level threshold as first modal data; Determining the confidence level of the first modal data, and determining modal data whose confidence level of the first modal data is greater than or equal to a confidence threshold as target modal data; determining a fusion weight of the target modality data based on the data level, the confidence level, and a reference weight, wherein the reference weight is determined based on historical navigation data of a ship system; The target modality data is fused based on the fusion weight to obtain the multimodal fusion data.

3. The system according to claim 1, wherein: The data processing unit is used to: Performing anomaly detection on the multimodal fusion data using the pre-built ship safety rule library to obtain first anomaly information; Performing anomaly detection on the multimodal fusion data using the anomaly detection model to obtain second anomaly information and anomaly probability; The abnormality information is determined based on the first abnormality information and the second abnormality information.

4. The system according to claim 1, wherein: The system further comprises a data acquisition unit, The data acquisition unit is used to acquire historical maintenance information and equipment parameter information of the ship; The data processing unit is used to determine abnormal information based on the historical maintenance information, the equipment parameter information, the multimodal fusion data, a pre-built ship safety rule base and an abnormality detection model.

5. The system according to any one of claims 1 to 4, characterized in that: The system further includes a data preprocessing unit, wherein the data preprocessing unit is configured to perform image segmentation processing and target object recognition processing on the modal data in the video format to obtain first preprocessed data when the format of the modal data is a video format; The data fusion unit is further configured to adaptively determine target modality data to be fused in the first preprocessed data based on the reference information.

6. The system according to claim 5, characterized in that The data preprocessing unit is further used for: performing integrity detection on target objects in the modal data in the video format to obtain a detection result, wherein the target objects include ship safety equipment and ship personnel; In a case where the detection result indicates that the target object is occluded, determining a supplementary feature; Completing the target object based on the supplementary features to obtain second preprocessed data; The data fusion unit is further configured to adaptively determine target modality data to be fused in the second preprocessed data based on the reference information.

7. The system according to any one of claims 1 to 4, characterized in that: The system further includes a data pre-processing unit, wherein the data pre-processing unit is configured to: Performing cross-validation on the multiple modal data to obtain a cross-validation result; Determining abnormal data of the multiple modal data based on the cross-validation results; Deleting the abnormal data from the multimodal data to obtain third preprocessed data; The data fusion unit is further configured to adaptively determine target modality data to be fused in the third preprocessed data based on the reference information.

8. The system according to any one of claims 1 to 4, characterized in that: The alarm unit is used to: Processing the abnormal information to determine an evaluation parameter, wherein the evaluation parameter includes at least one of a risk diffusion speed, a potential loss level, or a handling urgency; determining the abnormality level using the evaluation parameters; The alarm information is determined based on one of the abnormality level, the occurrence time of the abnormality information, the occurrence location of the abnormality information, or the affected area of ​​the abnormality information. The alarm information includes an alarm mode and an alarm area.

9. The system according to any one of claims 1 to 4, characterized in that: The system further includes a storage unit, wherein the storage unit is configured to: The multimodal data is formatted and the formatted multimodal data is stored.

10. The system according to any one of claims 1 to 4, characterized in that: The system further includes a storage unit, wherein the storage unit is configured to: The abnormal modal data corresponding to the abnormal information is encoded to obtain a content identifier corresponding to the abnormal modal data, the content identifier is associated with the reference modal data to obtain associated data, and the associated data is stored. The reference modal data is data with a different data modality from the abnormal modal data and a same generation time.

Images