A ship intelligent video monitoring system

Abstract

The application discloses a kind of ship intelligent video monitoring systems, belong to computer technical field.System includes: data acquisition unit, for obtaining the multiple modal data of ship;Data fusion unit is used to determine reference information, based on reference information in multiple modal data self-adaptation determines the target modal data to be fused, target modal data is fused, and multiple modal fusion data are obtained;Data processing unit is used to determine abnormal information based on multiple modal fusion data, pre-constructed ship safety rule base and abnormal detection model, and abnormal information includes ship safety equipment abnormal information, ship personnel action abnormal information, sensor abnormal information and ship driving trajectory abnormal information;Alarm unit is used to process abnormal information, and the abnormal level of abnormal information is obtained, and alarm information is determined using abnormal level and abnormal information.The application can improve the safety and standardization in the process of ship navigation.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a ship intelligent video surveillance system. Background Technology

[0002] With the rapid development of the shipping industry, the number of ships is constantly increasing, and the navigation environment is becoming increasingly complex. Ships navigating in this complex marine environment face a variety of potential safety threats, such as severe weather, maritime collisions, equipment failures, human error, and the transport of dangerous goods. These factors can all lead to serious safety accidents, causing not only huge economic losses but also posing a significant threat to the marine ecosystem and human lives. Therefore, how to effectively ensure the safety of ship navigation and achieve real-time, accurate, and comprehensive ship status monitoring has become a crucial issue that urgently needs to be addressed. Summary of the Invention

[0003] This application provides a ship intelligent video monitoring system, which can improve the safety and standardization of ship navigation. The technical solution is as follows:

[0004] On the one hand, this application provides a ship intelligent video surveillance system, the system comprising:

[0005] The data acquisition unit is used to acquire multiple modal data of the ship, including at least two of the following: position modal data, monitoring modal data, environmental modal data, or sensor modal data.

[0006] A data fusion unit is used to determine reference information, adaptively determine target modal data to be fused from the multiple modal data based on the reference information, and fuse the target modal data to obtain multimodal fused data. The reference information includes at least one of environmental risk level, confidence level of the multiple modal data, or mission requirements of different stages of ship navigation.

[0007] The data processing unit is used to determine abnormal information based on the multimodal fusion data, the pre-built ship safety rule base and the anomaly detection model. The abnormal information includes abnormal information of ship safety equipment, abnormal information of ship personnel actions, abnormal information of sensors and abnormal information of ship navigation trajectory.

[0008] An alarm unit is used to process the abnormal information, obtain the abnormal level of the abnormal information, and determine alarm information using the abnormal level and the abnormal information.

[0009] On the other hand, this application provides a method for intelligent video surveillance of ships, the method comprising:

[0010] Acquire multiple modal data of the ship, wherein the multiple modal data includes at least two of the following: position modal data, monitoring modal data, environmental modal data, or sensor modal data;

[0011] Determine reference information, adaptively determine target modal data to be fused from the multiple modal data based on the reference information, fuse the target modal data to obtain multimodal fused data, wherein the reference information includes at least one of environmental risk level, confidence level of the multiple modal data, or mission requirements of different stages of ship navigation;

[0012] Based on the multimodal fusion data, the pre-built ship safety rule base and the anomaly detection model, the anomaly information is determined. The anomaly information includes anomaly information of ship safety equipment, anomaly information of ship personnel actions, anomaly information of sensors, and anomaly information of ship navigation trajectory.

[0013] The abnormal information is processed to obtain the abnormal level of the abnormal information, and the alarm information is determined using the abnormal level and the abnormal information.

[0014] On the other hand, embodiments of the present invention provide an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the ship intelligent video monitoring method as described in any of the above embodiments.

[0015] On the other hand, embodiments of the present invention provide a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the ship intelligent video monitoring method as described in any of the above embodiments.

[0016] The technical solution provided in this application brings at least the following beneficial effects:

[0017] This application adaptively selects target modal data to be fused based on reference information such as environmental risk levels, confidence levels of multiple modal data, and task requirements at different stages of ship navigation. This makes the fused data more targeted and effective, avoids interference from irrelevant or low-value data, improves data fusion quality, reduces the amount of fused data, and increases data fusion efficiency. Furthermore, the multimodal fused data includes location, monitoring, environmental, and sensor data, enabling it to more comprehensively and deeply reflect the ship's operational status. Anomaly information is identified using a pre-built ship safety rule base and anomaly detection model, and cross-validation of the multimodal data improves the accuracy of anomaly identification. Anomaly information is processed to obtain anomaly levels, and alarm information is determined using anomaly levels and anomaly information, achieving hierarchical management of anomalies. Alarms are issued by combining anomaly levels and specific anomaly information, making alarm information more accurate and targeted, facilitating managers to quickly understand the situation and take appropriate measures, thus improving the safety and standardization of ship navigation. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of a ship intelligent video surveillance system provided in an embodiment of this application;

[0019] Figure 2 This is a schematic diagram of a ship intelligent video surveillance method provided in an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of the structure of a computer system for a terminal device or server provided in an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0022] It should be noted that the terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0023] Figure 1 This is a schematic diagram of a ship intelligent video surveillance system provided in an embodiment of this application. Figure 1As shown, the ship intelligent video monitoring system includes a data acquisition unit 101, a data fusion unit 102, a data processing unit 103, and an alarm unit 104.

[0024] In an exemplary embodiment of this application, the data acquisition unit 101 is used to acquire multiple modal data of the ship, including at least two of the following: position modal data, monitoring modal data, environmental modal data, or sensor modal data.

[0025] Among them, position modal data refers to information used to describe the specific location and related motion state of a ship in geographic space. It can be obtained by the Automatic Identification System (AIS) or the Beidou positioning system, which can acquire data such as the ship's latitude and longitude, speed, heading, and the distance to surrounding ships and obstacles.

[0026] Monitoring modal data refers to data from video surveillance equipment installed in various key areas of a ship. This data records real-time scenes inside and outside the ship in the form of images and videos, providing a clear picture of personnel activities, equipment operating status, cargo loading and unloading, and changes in the surrounding environment. For example, monitoring modal data includes video data from cameras monitoring areas such as the deck, engine room, and bridge in real time, as well as abnormal sounds detected by microphone arrays (such as collisions or explosions).

[0027] Environmental modal data refers to various physical and meteorological parameters of the waters and surrounding environment in which a ship is located. Environmental modal data includes wind speed, visibility, and wave height measured by meteorological sensors; and water depth, water flow velocity, and other data measured by hydrological sensors.

[0028] Sensor modal data refers to data collected in real time by various sensors installed on a ship, which is related to the operating status and system performance of ship equipment. Sensor modal data includes smoke, temperature, and gas concentration (such as CO2) measured by in-cabin sensors, and engine speed, oil pressure, and electrical system status measured by equipment status sensors.

[0029] In an exemplary embodiment of this application, the data fusion unit 102 is used to determine reference information, adaptively determine the target modal data to be fused from multiple modal data based on the reference information, and fuse the target modal data to obtain multimodal fused data. The reference information includes at least one of environmental risk level, confidence level of multiple modal data, or mission requirements of different stages of ship navigation.

[0030] The environmental risk level in the reference information can be calculated using formula (1):

[0031] Formula (1)

[0032] In the above formula (1), This is the environmental risk level parameter at time t. N is the number of types of environmental parameters, such as wind speed, wave height, visibility, salt spray concentration, etc. It is the weight coefficient of the i-th type of environmental parameter, representing the degree of contribution of the parameter to the overall risk, which can be obtained through expert evaluation, historical data analysis or machine learning training. It is a vector of the i-th type of environmental parameters at time t. For example, if the environmental parameter is wind speed, then... It is a vector containing wind speed values. It is a vector of environmental parameters under safe conditions. Environmental parameters under safe conditions are determined by ship design standards, historical safety data or industry specifications. It is Euclidean distance. It is the variance of the i-th type of environmental parameter. It is a chaos weighting factor. It is the maximum Lyapunov index, used to quantify the degree of chaos in a ship's trajectory and predict sudden risks (such as trajectory instability).

[0033] After determining the environmental risk level parameters, multiple level thresholds can be set. The environmental risk level parameters are then compared with the level thresholds to obtain the environmental risk level. For example, the environmental risk level can be divided into high-risk, medium-risk, and low-risk levels.

[0034] The confidence levels of the multiple modal data in the reference information can be calculated using formula (2):

[0035] Formula (2)

[0036] In the above formula (2), It is the confidence score of the i-th modality data. It is a normalization function that converts the result into a probability distribution. It is the total number of time steps. It is a bidirectional long short-term memory network used to process time series data. It is a spatial attention module that processes the spatial distribution characteristics of sensors. Data for the i-th mode at time t. This represents element-wise multiplication (Hadamard product). It is a time decay function. It is a time interval.

[0037] In the exemplary embodiments of this application, the different stages of ship navigation include, but are not limited to, the departure stage, the berthing stage, the navigation stage, and the night or foggy navigation stage. Different navigation stages correspond to different task requirements. For example, the task requirements for the departure stage are obstacle detection, personnel safety, and equipment monitoring. These task requirements can be mapped to a multi-dimensional vector, i.e., X = [obstacle detection, personnel safety, equipment monitoring]. For example, T = [0.6, 0.3, 0.1]. For example, the task requirements for the berthing stage are centimeter-level positioning, cable control, and personnel standardization. These task requirements can be mapped to a multi-dimensional vector, i.e., X = [centimeter-level positioning, cable control, personnel standardization]. For example, T = [0.8, 0.1, 0.1].

[0038] In an exemplary embodiment of this application, the process of adaptively determining the target modality data to be fused from multiple modality data based on reference information includes steps A1 to A5.

[0039] In step A1, the data level of each modal data is determined based on the environmental risk level and the mission requirements of different stages of ship navigation.

[0040] For example, after determining the multidimensional vectors corresponding to the environmental risk level and the task requirements at different stages of ship navigation, the data level of different modal data is determined by the formula of the dynamic scoring model (i.e., formula (3)):

[0041] Formula (3)

[0042] In the above formula (3), It is the score of the i-th modal data; A balancing factor between environmental risk level and task requirements at different stages is used to adjust the weight of environmental risk and task requirements in the scoring. It is the environmental risk level; This is the effectiveness score of the i-th modality in historical high-risk scenarios. This score may be based on the performance of this modality in similar high-risk environments in historical data. For example, the effectiveness score of radar in typhoons is 0.9, indicating that radar performs well and has high effectiveness in typhoon conditions in historical data. It is the priority of task requirements, i.e., a multi-dimensional vector; It is the fit score of the i-th modality data to the current task.

[0043] In step A2, modal data with a data level greater than or equal to the level threshold are identified as first modal data.

[0044] For example, after determining the data level, the data is compared with a level threshold, and modal data with a level greater than or equal to the level threshold are selected as the first modal data. The level threshold can be determined based on the actual situation of the ship, and this application does not impose any restrictions on it.

[0045] In step A3, the confidence level of the first modal data is determined, and modal data whose confidence level of the first modal data is greater than or equal to the confidence level threshold are determined as target modal data.

[0046] For example, after determining the first modal data, the confidence level of the first modal data can be further determined. The confidence level of the first modal data can be determined based on formula (2), and the relevant explanation of formula (2) is provided, which will not be repeated here. The confidence threshold is adaptively adjusted according to the environmental risk level and the mission requirements of the navigation phase. In high-risk environments or critical navigation phases, the confidence threshold can be appropriately increased to ensure that only the most reliable data is selected; in low-risk environments or regular navigation phases, the confidence threshold can be appropriately decreased to obtain more information. The first modal data is further filtered using the confidence threshold to obtain the first modal data with a confidence level greater than or equal to the confidence threshold, i.e., the target modal data.

[0047] In step A4, the fusion weights of the target modal data are determined based on data level, confidence level, and reference weights, with the reference weights determined based on historical navigation data of the ship system.

[0048] For example, the reference weights can be determined based on the ship's historical navigation data. For instance, if AIS signals are frequently lost in a certain sea area in historical navigation data, their weights will be automatically reduced. The fusion weight of the target modal data = k1 × data level + k2 × confidence level + k3 × reference weight, where k1, k2, and k3 are correction parameters that can be set based on actual conditions.

[0049] In step A5, the target modal data is fused based on the fusion weights to obtain multimodal fused data.

[0050] In one embodiment, all target modal data are time-synchronized and spatially aligned to ensure data consistency. Then, a weighted average is calculated for each data point to obtain multimodal fused data. It should be noted that the target modal data fusion method described in this application is illustrative; other methods such as Kalman filtering fusion and neural network fusion can also be used to fuse target modal data, and this application does not impose any limitations on these methods.

[0051] This application determines data levels based on environmental risk levels and the mission requirements of the ship's navigation phase, filters out the most valuable modal data, and fuses it using confidence levels and reference weights. This effectively reduces the impact of noise and errors, improving the accuracy and reliability of multimodal fusion data. Furthermore, since environmental risk levels and mission requirements of the navigation phase are dynamically changing, the data filtering and fusion strategies are adaptively adjusted according to these changes, giving the system stronger adaptability.

[0052] In an exemplary embodiment of this application, the ship intelligent video surveillance system further includes a data preprocessing unit, which performs image segmentation and target object recognition processing on the video format modal data when the modal data is in video format, to obtain first preprocessed data; and a data fusion unit adaptively determines the target modal data to be fused from the first preprocessed data based on reference information.

[0053] For example, when the modal data format is video, keyframes are first selected from the video, and image segmentation is performed on the keyframes. Image segmentation includes semantic segmentation, instance segmentation, and pixel-level segmentation. Semantic segmentation can use the DeepLabV3+ model, trained on a ship-specific dataset, to obtain pixel-level classification labels (such as ocean, sky, ship, shoreline). Instance segmentation can use Mask R-CNN to separate multiple crew or equipment targets. Pixel-level segmentation can use the U-Net model to label hazardous material leakage areas. Then, based on the segmentation results and the target recognition model, target recognition is performed to identify personnel behavior, navigation equipment, marine obstacles, etc., to obtain the first preprocessed data. The target recognition model can be YOLOv8, CRNN, etc. It should be noted that the process by which the data fusion unit adaptively determines the target modal data to be fused from the first preprocessed data based on reference information is similar to the process by which the data fusion unit adaptively determines the target modal data to be fused from multiple modal data based on reference information, and will not be elaborated here.

[0054] This application adds image segmentation and target recognition preprocessing steps to video modal data, which can effectively filter out key target information such as ship personnel and equipment in the monitoring data, reduce interference from irrelevant information, and make the fused data more focused on elements related to ship prevention and control, thereby improving the data processing efficiency of the entire system.

[0055] In an exemplary embodiment of this application, the data preprocessing unit can also perform integrity detection on the target object in the modal data of the video format to obtain the detection result. The target object includes ship safety equipment and ship personnel. If the detection result indicates that the target object is occluded, supplementary features are determined. The target object is completed based on the supplementary features to obtain the second preprocessed data. The data fusion unit adaptively determines the target modal data to be fused in the second preprocessed data based on reference information.

[0056] Specifically, based on the first preprocessed data, a suitable integrity detection model, such as the YOLO-OCC model, is selected to perform target integrity detection. If there is no occlusion, the first preprocessed data is used; if occlusion exists, an occlusion threshold can be set to determine whether it is partial or severe occlusion. For example, when the visibility of key features is greater than 30%, it is considered partial occlusion, and less than or equal to 30%, it is considered severe occlusion. For partial occlusion, repair methods can be used to repair the occluded target, such as using the EdgeConnect network to restore the occluded texture, or using a Point Completion Network (PCN) to generate the missing parts of the 3D point cloud. For severe occlusion, other data can be used for supplementation. For example, for security equipment occlusion, cross-camera collaboration can be used to perform multi-view geometric consistency verification. After supplementing the features of the occluded target object, the second preprocessed data is obtained. It should be noted that the process by which the data fusion unit adaptively determines the target modal data to be fused in the second preprocessed data based on reference information is similar to the process by which the data fusion unit adaptively determines the target modal data to be fused in multiple modal data based on reference information, and will not be elaborated here.

[0057] This application can promptly detect occlusion issues in the data by performing integrity detection on the target object, and take corresponding measures to supplement the data according to different degrees of occlusion. This can effectively improve the integrity and quality of the target modal data to be fused. In the subsequent process, when using the target modal data to be fused for fusion and identifying abnormal information, it can improve the accuracy and reliability of abnormal information judgment and reduce the false alarm rate of abnormal information.

[0058] In an exemplary embodiment of this application, the data preprocessing unit may further perform cross-validation on multiple modal data to obtain cross-validation results; identify anomalous data in the multiple modal data based on the cross-validation results; delete the anomalous data in the multimodal data to obtain third preprocessed data; and the data fusion unit adaptively determines the target modal data to be fused in the third preprocessed data based on reference information.

[0059] For example, multiple modal data can be grouped according to certain rules, such as by time series or data source. For each group of data, the performance of different modal data under the same conditions can be compared. For example, within the same time period, the position data, speed data, and heading data of ships can be compared.

[0060] Cross-validation of multiple modalities is performed to check for obvious conflicts or inconsistencies between different modalities, yielding the cross-validation results. For example, location data may show a ship stationary within the port, but speed data may show a higher speed, indicating potential anomalies. Inconsistencies can be determined by setting thresholds or rules; for instance, if speed data exceeds a certain reasonable range and does not match location data, it is considered an anomaly.

[0061] In one embodiment, a reasonable threshold range is set for each modality of data based on cross-validation results and statistical characteristics of historical data. For example, for ship speed data, a normal speed range is set based on the ship type and navigation environment. When speed data exceeds this range, it is identified as abnormal data.

[0062] In another embodiment, the data's changing patterns are observed to identify data that deviates from normal patterns. For example, ship position data typically exhibits continuous and smooth changes; sudden jumps or discontinuous changes may indicate anomalous data. Data mining algorithms, such as cluster analysis and time series analysis, can be used to identify these patterns.

[0063] A comprehensive assessment of the data is conducted by combining multiple judgment methods. For example, it considers not only whether the data exceeds a threshold, but also factors such as the data's trend and its correlation with other modal data. If a data point does not exceed a threshold, but its trend is significantly different from other data points, and its correlation with other modal data is also abnormal, then it can also be judged as abnormal data.

[0064] Choose an appropriate correlation index based on the type and characteristics of the modal data. For continuous data, commonly used correlation indices include the Pearson correlation coefficient and the Spearman correlation coefficient. For example, the Pearson correlation coefficient can be used to measure the linear correlation between ship position and speed data.

[0065] Optionally, data preprocessing is required before calculating correlations to ensure data quality and consistency. This includes tasks such as handling missing values, outliers, and normalization.

[0066] Using a selected correlation index, calculate the correlation coefficient between different modalities of data. The correlation coefficient typically ranges from -1 to 1; the closer the absolute value is to 1, the stronger the correlation, and the closer the absolute value is to 0, the weaker the correlation. For example, if the Pearson correlation coefficient between location data and velocity data is close to 1, it indicates a strong positive correlation between them.

[0067] A simple linear regression model is built based on correlation to predict the value of one modality of data based on another modality of data. For example, using location data to predict velocity data, if the actual velocity data deviates significantly from the predicted value, an anomaly may exist, and thus it is identified as outlier data.

[0068] After identifying the outlier data, it is removed from the multimodal data to obtain the third preprocessed data. It should be noted that the process by which the data fusion unit adaptively determines the target modality data to be fused from the third preprocessed data based on reference information is similar to the process by which the data fusion unit adaptively determines the target modality data to be fused from multiple modal data based on reference information, and will not be elaborated upon here.

[0069] This application identifies anomalous data through cross-validation and multiple judgment methods, avoiding the limitations of single-modal data detection and enabling a more comprehensive and accurate identification of anomalies in the data. Calculating the correlation between different modalities and establishing a linear regression model reveals the inherent connections between the data, providing a basis for further data analysis and decision-making. For ship monitoring systems, comprehensive analysis of multiple modalities allows for a more complete understanding of the ship's operational status and timely detection of potential safety hazards.

[0070] In an exemplary embodiment of this application, the data processing unit 103 is used to determine abnormal information based on multimodal fusion data, a pre-built ship safety rule base and anomaly detection model. The abnormal information includes abnormal information of ship safety equipment, abnormal information of ship personnel actions, abnormal information of sensors and abnormal information of ship travel trajectory.

[0071] For example, the pre-built ship safety rule base includes, but is not limited to, equipment inspection rules, personnel behavior rules, navigation rules, and sensor verification rules. For instance, equipment inspection rules require that the number of life jackets be greater than or equal to the number of crew members; if the number of life jackets is less than the number of crew members, abnormal information regarding ship safety equipment is identified. Personnel behavior rules, such as prohibiting smoking in the hold of a hazardous materials ship, can identify abnormal information regarding personnel actions based on hotspots detected by infrared cameras and excessively high CO2 levels detected by smoke sensors in multimodal fusion data. Navigation rules, such as a deviation of the course from the planned route of less than or equal to 5 degrees, can determine whether the current course angle is within the rule range through trajectory information; if not, abnormal information regarding the ship's trajectory is identified. Sensor verification rules, such as a difference in position measured by radar and AIS exceeding 100m for more than 30 seconds, identify abnormal information regarding the ship's trajectory.

[0072] It should be noted that the rules in the pre-built ship safety rule library can be either static or dynamic. For example, during nighttime navigation, the safety equipment detection threshold corresponding to the equipment inspection rule is reduced from 0.8 to 0.7, thus lowering the missed detection rate.

[0073] In an exemplary embodiment of this application, the process of determining abnormal information may include: performing anomaly detection on multimodal fusion data using a pre-built ship safety rule base to obtain first abnormal information; performing anomaly detection on multimodal fusion data using an anomaly detection model to obtain second abnormal information and anomaly probability; and determining abnormal information based on the first and second abnormal information.

[0074] For example, any rule in the pre-built ship safety rule base can have a parameter threshold or parameter range. The multimodal fusion data is parsed to extract data fields related to each rule in the ship safety rule base. For instance, data such as the ship's current speed, lifeboat engine running time, life raft positioning device battery level, personnel operation command records, sensor readings (such as gyroscope and accelerometer data), ship position, and heading can be extracted from the fusion data.

[0075] The extracted data is matched one by one with the rules in the ship safety rule base. If the data does not meet the conditions of a certain rule, the safety item corresponding to that rule is determined to be abnormal, and the abnormal rule number, abnormal data value, and abnormal description are recorded to form the first abnormal information. For example, if the ship's current speed exceeds the speed limit rule of a certain waterway, the first abnormal information will record "Rule number: XX, Abnormal item: Ship speed, Abnormal value: Current speed A knots (speed limit B knots)".

[0076] The multimodal fusion data to be detected is input into a trained anomaly detection model, which outputs reconstructed data and error probabilities. Based on a preset reconstruction error threshold, it is determined whether the input data is anomalous. If the error probability exceeds the preset threshold, the data is identified as anomalous, and its features and probability are recorded to obtain secondary anomaly information.

[0077] Align and correlate the first and second anomaly information to identify possible corresponding anomalies. For example, if the first anomaly information records "low battery power of the life raft positioning device," while the second anomaly information records "abnormal data from a certain safety equipment sensor," and further analysis reveals that this sensor is related to the battery detection of the life raft positioning device, then these two anomaly information pieces are correlated.

[0078] The correlated anomaly information is integrated, and duplicate anomaly descriptions are removed to form a comprehensive anomaly information list. For the same anomaly, if the two detection methods provide different detailed information, these detailed information are merged. For different anomaly situations, if the anomaly probability corresponding to the second anomaly information is low, auxiliary information (visual inspection, sending messages to the corresponding crew members to determine if an anomaly exists, etc.) is obtained to determine whether it is an anomaly information. If the first anomaly information and the second anomaly information are different, both are identified as anomaly information.

[0079] Then, based on the predefined anomaly types (anomaly information of ship safety equipment, anomaly information of ship personnel actions, anomaly information of sensors, and anomaly information of ship navigation trajectory), the integrated anomaly information is classified to obtain the final anomaly information.

[0080] The proposed ship safety rule base, based on well-defined industry standards, can quickly detect anomalies conforming to known rules, exhibiting high interpretability and reliability. Meanwhile, the anomaly detection model, by learning the feature distribution of large amounts of data, can uncover potential anomaly patterns that are difficult to describe using rules. The combined use of these two technologies complements each other, improving the accuracy of anomaly detection, reducing missed and false detections, ensuring comprehensive anomaly detection coverage, and enhancing safety and reliability during navigation.

[0081] In an exemplary embodiment of this application, the ship's historical maintenance information and equipment parameter information can also be obtained through the data acquisition unit (not shown in the figure); the data processing unit 103 can determine abnormal information based on the historical maintenance information, equipment parameter information, multimodal fusion data, pre-built ship safety rule base and abnormal detection model.

[0082] For example, the data acquisition unit interfaces with the ship's maintenance management system, equipment management platform, and paper maintenance record digitization system to obtain the ship's historical maintenance information and equipment parameter information. For instance, it obtains structured data such as ship periodic maintenance records, equipment repair records, and parts replacement records from the maintenance management system through an application programming interface (API); or, for paper maintenance records, it uses OCR (Optical Character Recognition) technology to scan and recognize the data, converting the image data into processable text data to obtain historical maintenance information and equipment parameter information.

[0083] The process of identifying anomaly information using a pre-built ship safety rule base and anomaly detection model has been explained above and will not be repeated here. Further filtering of anomalies using historical maintenance information and equipment parameter information yields the filtered anomaly information.

[0084] The historical maintenance information and equipment parameter information of the vessel used in this application are used to judge abnormal information, which helps to more accurately identify potential problems such as aging and failure of ship safety equipment and abnormal equipment performance. It can also uncover potential safety hazards of the vessel more deeply and enrich the function and accuracy of the ship's prevention and control system.

[0085] In an exemplary embodiment of this application, multimodal fusion data can also be processed based on photoelectric attenuation factor to obtain abnormal information, wherein the light attenuation factor and the electrical signal attenuation factor corresponding to the multimodal fusion data are also considered.

[0086] In one embodiment, processing multimodal fusion data based on photoelectric attenuation factors to obtain anomaly information includes: calculating the illumination attenuation factor of the multimodal fusion data using a nonlinear illumination attenuation model; calculating the electrical signal attenuation factor of the multimodal fusion data using a frequency domain response model; compensating the multimodal fusion data using the illumination attenuation factor and the electrical signal attenuation factor respectively to obtain compensated multimodal fusion data; calculating the comprehensive weight of the compensated multimodal fusion data based on the illumination attenuation factor, the electrical signal attenuation factor, the confidence level of the multimodal fusion data, and the environmental risk level; generating an attenuation-resistant multimodal fusion feature matrix based on the comprehensive weight and the compensated multimodal fusion data; and inputting the attenuation-resistant multimodal fusion feature matrix into a pre-trained 3D convolutional anomaly detection network to obtain anomaly information.

[0087] Among them, the comprehensive weight of the compensated multimodal fusion data The calculation formula is as follows:

[0088] Formula (4)

[0089] In the above formula (4), Indicates the confidence level of multimodal fusion data. Indicates the environmental risk level. Indicates the light attenuation factor. Indicates the attenuation factor of the electrical signal. This represents the sum of the light attenuation factor and the electrical signal attenuation factor.

[0090] This embodiment specifically compensates for data attenuation and improves data quality. Specifically, it calculates the light attenuation factor and electrical signal attenuation factor using a nonlinear light attenuation model and a frequency domain response model, respectively, effectively addressing two core attenuation problems in ship monitoring. Light attenuation: Ships experience complex and variable lighting conditions under different navigation environments (e.g., nighttime, heavy rain, tunnels, bridges). The nonlinear model can more accurately fit the attenuation law of light intensity with distance and medium (e.g., fog, seawater splash), avoiding problems such as blurred monitoring videos and feature loss due to insufficient or overexposed lighting. Electrical signal attenuation: Sensor signals (e.g., cameras, radar) are susceptible to electromagnetic interference and noise pollution during transmission. The frequency domain response model analyzes the attenuation characteristics of signals in the frequency domain (e.g., high-frequency signals attenuate faster), effectively filtering noise, restoring signal integrity, and improving the reliability of sensor data. Compensating for the original multimodal data (e.g., video frames, sensor signals) based on the attenuation factor directly corrects distortion problems during data acquisition, providing clearer and more accurate input data for subsequent anomaly detection.

[0091] Furthermore, this embodiment employs dynamic weighted fusion, adapting to different environments and data confidence levels. Data confidence reflects the reliability of the multimodal data itself (e.g., whether the camera focal length is normal, sensor calibration status), avoiding errors introduced by fusing low-confidence data. Environmental risk levels dynamically adjust weights based on the ship's navigation stage (e.g., docking, complex sea conditions), increasing the weight of key data (e.g., ship position, personnel movements) in high-risk scenarios to ensure that anomaly detection priorities match actual needs. The attenuation factor directly correlates with the degree of data damage; the higher the attenuation, the lower the weight, avoiding over-reliance on damaged data and achieving an adaptive fusion strategy of "more weight for superior data, less weight for inferior data." This mechanism enables the system to dynamically balance the contributions of each modality in complex environments (e.g., severe weather, areas with strong electromagnetic interference), improving the robustness of multimodal fusion.

[0092] Furthermore, this embodiment constructs an anti-attenuation feature matrix to enhance the generalization ability of the anomaly detection model. By weighting and fusing the compensated data through comprehensive weights, an anti-attenuation multimodal fusion feature matrix is ​​generated. This matrix integrates spatiotemporal dimension information (such as the temporal continuity of video frame sequences and the spatial distribution of sensor data), and suppresses the effects of noise and distortion through attenuation compensation and dynamic weighting. When input into a 3D convolutional anomaly detection network, the anti-attenuation feature matrix can more clearly present anomaly patterns (such as the spatiotemporal trajectory of abnormal human movements and abrupt changes in sensor data), reducing feature ambiguity or misjudgment caused by data attenuation, and improving the model's sensitivity and detection accuracy for subtle anomalies (such as early equipment failures and personnel violations).

[0093] Furthermore, this embodiment enhances the security and reliability of the ship monitoring system. Addressing common issues in ship monitoring such as lighting variations (e.g., day-night cycles, severe weather) and electrical signal interference (e.g., complex marine electromagnetic environments), this solution improves the availability of multimodal data from the source through data compensation and dynamic weighting, thereby reducing the false negative and false positive rates of anomaly detection. By combining the weighted calculation of environmental risk levels and data confidence levels, the system can more accurately focus on key data in high-risk scenarios, meeting the stringent requirements of real-time performance and accuracy for ship safety monitoring, and providing more reliable technical support for ship navigation safety, equipment maintenance, and personnel management.

[0094] In an exemplary embodiment of this application, the alarm unit 104 is used to process abnormal information, obtain the abnormality level of the abnormal information, and determine alarm information using the abnormality level and the abnormal information.

[0095] For example, the process of determining alarm information includes, but is not limited to, steps B1 to B3.

[0096] Step B1: Process the abnormal information and determine the assessment parameters, which include at least one of the following: risk spread rate, potential loss level, or urgency of response.

[0097] Risk diffusion rates include, but are not limited to, hazardous chemical spill rates and time to collision (TCPA). For example, based on data from hazardous chemical spill sensors, combined with the physical properties of the hazardous chemical (such as density and viscosity) and the size and shape of the leak, a fluid dynamics model can be used to calculate the spill rate. For instance, for a pinhole leak, the spill rate can be calculated using Bernoulli's equation, taking into account the effects of changes in pressure, temperature, and other factors during the spill.

[0098] For example, the Automatic Identification System (AIS) can be used to obtain information such as the position, speed, and heading of the ship and surrounding vessels. Geometric algorithms can then be used to calculate the relative motion trajectory and collision time of the two ships. For instance, vector calculation methods can be employed to calculate the rate of change of distance between the two ships in the collision direction based on their position and velocity vectors, thereby obtaining the TCPA (Time of Collision).

[0099] Potential loss levels include, but are not limited to, the risk of personal injury or death, loss of cargo value, and the degree of environmental pollution. For example, by combining ship personnel distribution data (such as the number of crew members in each area, passenger distribution, etc.), the location and type of the anomaly, a risk assessment model (such as a Monte Carlo simulation-based personal injury risk model) can be used to predict possible personal injury or death. For instance, the impact of toxic gas diffusion on personnel under different leakage scenarios can be simulated, the probability of injury or death can be calculated based on the exposure time and concentration, and this probability can be converted into a personal injury or death risk level.

[0100] For example, detailed information about the ship's cargo (such as cargo type, quantity, value, etc.) is obtained, and the potential damage to the cargo caused by anomalies (such as cargo damage caused by fire, flooding, collision, etc.) is analyzed. Based on the extent of cargo damage and market value, the amount of cargo value loss is estimated and classified into different loss levels.

[0101] For example, considering the type of hazardous chemical, the amount leaked, the location of the leak, and the surrounding marine environment (such as water currents, tides, and ecologically sensitive areas), environmental diffusion models (such as oil spill diffusion models and chemical substance diffusion models) are used to predict the diffusion range and concentration distribution of pollutants in the ocean. Based on the prediction results and in conjunction with environmental protection standards, the degree of environmental pollution is assessed.

[0102] The urgency of response can be used to determine the remaining safe response time window. For example, based on the type and development trend of the anomaly, combined with the ship's emergency response capabilities (such as the availability of firefighting equipment and the response time of emergency personnel), the upper limit of the time from the discovery of the anomaly to the necessity of taking effective measures can be determined; this is the remaining safe response time window. For instance, in the case of a ship fire, based on the fire spread rate and the distribution of combustibles, combined with the firefighting capabilities of the ship's firefighting equipment, it can be calculated how long it must take to carry out firefighting operations to prevent the fire from getting out of control.

[0103] Step B2: Determine the anomaly level using the evaluation parameters.

[0104] In one embodiment, determining the anomaly level using evaluation parameters may include the following:

[0105] A hierarchical model was constructed: the risk assessment problem was decomposed into a target layer (determining the anomaly level), a criterion layer (risk spread rate, potential loss level, and urgency of response), and a solution layer (different anomaly scenarios). Ship safety experts and risk management experts were invited to conduct pairwise comparisons of the parameters in the criterion layer based on the degree of influence of each assessment parameter on the anomaly level, constructing a judgment matrix. For example, if experts believed that the risk spread rate had a slightly greater impact on the anomaly level than the potential loss level, then the corresponding position in the judgment matrix would be assigned a value of 3.

[0106] The largest eigenvalue and corresponding eigenvector of the judgment matrix are calculated using the eigenvalue method. After normalizing the eigenvector, the weight vectors of each evaluation parameter are obtained. Then, a consistency check is performed to ensure that the consistency ratio (CR) of the judgment matrix is ​​less than 0.1, so as to guarantee the rationality of the weight allocation.

[0107] The comprehensive risk value is calculated based on the quantified values ​​and weight vectors of each assessment parameter. For example, the comprehensive risk value = 0.6 × risk spread rate quantified value + 0.3 × potential loss level quantified value + 0.1 × urgency of response quantified value. Based on the magnitude of the comprehensive risk value, anomalies are classified into different levels, such as Level 1 (low risk), Level 2 (low to medium risk), Level 3 (medium risk), Level 4 (medium to high risk), and Level 5 (high risk).

[0108] In another embodiment, determining the anomaly level using assessment parameters may include the following: utilizing historical anomaly data, including assessment parameters (risk spread rate, potential loss level, urgency of response, etc.) and corresponding anomaly level labels. Feature extraction and selection are performed on this data to construct a multimodal feature vector. For example, in addition to assessment parameters, features such as sensor data trends (e.g., the rate of change of parameters such as temperature and pressure) and crew location distribution can also be considered.

[0109] We selected a random forest classifier as the machine learning model and trained it using a prepared dataset. We then adjusted the model's hyperparameters (such as the number of trees and maximum depth) using methods like cross-validation to improve its classification accuracy and generalization ability.

[0110] The evaluation parameters and multimodal features of the anomaly to be evaluated are input into a trained random forest classifier, and the model outputs the corresponding risk level (Level 1-5).

[0111] Optionally, the alarm unit 104 can also adjust the anomaly level based on the time and environment in which the anomaly occurred. This adjustment includes time-based adjustment and environmental-based adjustment.

[0112] For time-related adjustments: Current time and environmental information are obtained through the ship's clock system and weather sensors. When it is determined to be nighttime (e.g., based on sunrise and sunset times) or in severe weather (e.g., wind speeds reaching a certain level, visibility below a certain threshold, wave height exceeding a certain value, etc.), the level of specific types of anomalies is adjusted. For example, for the risk of falls, a risk threshold that might be set to Level 2 under normal conditions would automatically be reduced to Level 1 at night or in severe weather. This is because at night or in severe weather conditions, visibility is limited and movement is restricted, increasing the likelihood of falls and requiring earlier warnings.

[0113] For environmental factor correction: Utilizing the ship's navigation system and Geographic Information System (GIS), it is determined whether the ship is currently in sensitive sea areas, such as ecological protection zones, military restricted areas, or busy waterways. In sensitive sea areas, the tolerance for ship track deviation is reduced by 50%. For example, in ordinary sea areas, the permissible track deviation range is ±100 meters, while in sensitive sea areas it is adjusted to ±50 meters. Once the ship's track deviation exceeds the adjusted tolerance, its anomaly level will be increased accordingly, allowing for timely measures to avoid impacting sensitive areas.

[0114] Step B3: Determine the alarm information based on one of the following: the level of the anomaly, the time of occurrence of the anomaly, the location of the anomaly, or the area affected by the anomaly. The alarm information includes the alarm method and the alarm area.

[0115] For example, a mapping relationship is established between the revised anomaly level and the alarm format. For instance, for Level 1 anomalies, a slight alert is sent via the system interface; for Level 2 anomalies, an audible alert is added; for Level 3 anomalies, relevant personnel are notified via SMS; for Level 4 anomalies, an email is sent and the ship's broadcast system is activated; and for Level 5 anomalies, in addition to the above methods, an emergency distress signal is sent to the nearest maritime rescue center and ship management department.

[0116] Furthermore, considering the responsibilities and permissions of different personnel, personalized alarm methods are set up for personnel in different positions. For example, for the captain and watch officer, in addition to the regular alarm methods, real-time push notifications are also sent via handheld terminals to ensure that they can receive important alarm information as soon as possible.

[0117] For example, the alarm zone is determined based on the location of the anomaly and its potential impact. For instance, if the anomaly is a fire in a certain area of ​​the ship, the alarm zone should cover the fire area and the surrounding compartments, passageways, and equipment areas that may be affected.

[0118] Then, the scope of impact is optimized by considering environmental factors. For example, the alarm zone is optimized by considering factors such as the ship's structural layout, ventilation system, and personnel evacuation routes. For instance, when determining the fire alarm zone, the warning area downwind is appropriately expanded based on wind direction and ventilation direction to ensure that personnel can be evacuated to a safe area in a timely manner. Simultaneously, when anomalies occur in sensitive sea areas, the alarm zone should also include surrounding sea areas that may affect the sensitive area, in order to alert nearby vessels to take evasive action.

[0119] This application comprehensively considers multiple assessment parameters, such as the speed of risk spread, the level of potential loss, and the urgency of response, enabling a more comprehensive and accurate assessment of the severity of anomalies and avoiding inaccurate alarms caused by relying on a single factor. Furthermore, by adjusting the anomaly level based on the time and environment of the anomaly occurrence, it can adapt to changes in risk under different circumstances, ensuring timely alarms. Determining different alarm methods based on the adjusted anomaly level allows relevant personnel to take appropriate actions according to the urgency of the alarm, improving the targeting of emergency response. For example, for high-risk anomalies, using multiple alarm methods to simultaneously notify relevant personnel and management departments can quickly mobilize resources for handling, reducing accident losses. Determining the alarm area based on the anomaly's location and impact range, and optimizing it in conjunction with environmental factors, allows alarm information to be more accurately delivered to potentially affected areas and personnel, avoiding unnecessary panic and waste of resources.

[0120] In an exemplary embodiment of this application, the ship intelligent video surveillance system further includes a storage unit (not shown in the figure), which is used to format multimodal data and store the formatted multimodal data.

[0121] For example, the storage unit first formats the multimodal data to ensure that the data can be stored and retrieved efficiently. The main operations include stamping all data with a unified timestamp (UTC time, with millisecond precision) to solve the problem of clock asynchrony among multiple devices; unifying the spatial coordinates of the data, such as converting location data from BeiDou positioning system, AIS, radar, etc., into the WGS-84 standard coordinate system; and then storing the data in layers. The raw data of the most recent 72 hours can be stored by solid-state drives (SSDs), the compressed structured data within a certain period, such as data within 30 days, can be stored with the database, and data with longer time periods can be stored using Alibaba Cloud and blockchain.

[0122] This application improves data availability by formatting multimodal data through storage units, and constructs a data management system that is "real-time efficient, medium-term available, and long-term reliable" through unified spatiotemporal benchmarks and hierarchical storage, thereby optimizing storage efficiency.

[0123] In an exemplary embodiment of this application, the storage unit is further configured to encode the abnormal modal data corresponding to the abnormal information to obtain the content identifier corresponding to the abnormal modal data, associate the content identifier with the reference modal data to obtain associated data, and store the associated data. The reference modal data is data with a different data modality than the abnormal modal data and generated at the same time.

[0124] For example, after the data processing unit identifies abnormal information, the storage unit first encodes the abnormal modal data corresponding to this abnormal information. For instance, if the abnormal information is discovered by monitoring modal data (such as abnormal operational behavior of crew members captured by a camera), then the monitoring modal data becomes abnormal modal data. Encoding process transforms the abnormal modal data into a unique content identifier using a specific algorithm or rule. This content identifier can be a string of numbers, letters, or a combination thereof, used to uniquely identify the content characteristics of the abnormal modal data. For example, for video data (monitoring modal data) of a specific abnormal operational behavior of a crew member, a specific identifier is generated through an encoding algorithm. This identifier reflects the key characteristic information of the operational behavior. The encoding algorithm can be a hash algorithm, which can convert input data of arbitrary length into a fixed-length hash value through a specific hash function. This hash value can then serve as the content identifier.

[0125] Reference modal data is data with a different modality than anomalous modal data but generated at the same time. For example, if the anomalous modal data is monitoring modal data, then the reference modal data might be location modal data (such as the ship's latitude, longitude, and speed), environmental modal data (such as wind speed and direction), or sensor modal data (such as engine oil pressure and speed) acquired at the same time. The storage unit associates the generated content identifiers with the corresponding reference modal data to form associated data. This association is based on time synchronization, ensuring that the anomalous modal data and other related modal data correspond in the time dimension, facilitating subsequent comprehensive analysis. For example, the content identifier representing abnormal crew member behavior can be associated with reference modal data such as the ship's position, environmental conditions, and equipment operating parameters at the same time.

[0126] The storage unit stores the obtained associated data. Storage methods can include database storage, where the associated data is stored in a specific structure and format for easy subsequent querying, retrieval, and analysis; or cloud storage and blockchain storage can be used.

[0127] This application associates and stores the content identifiers of anomalous modal data with reference modal data, enabling rapid retrieval of multiple modal data related to the anomaly during subsequent analysis of anomaly information, facilitating comprehensive analysis of anomaly information. It organically combines time-related data from different modalities to form a complete dataset of anomalous events, ensuring data integrity and systematicity. For anomalous events during ship operation, the storage of associated data provides good traceability, and the introduction of content identifiers makes data retrieval more convenient.

[0128] This application adaptively selects target modal data to be fused based on reference information such as environmental risk levels, confidence levels of multiple modal data, and task requirements at different stages of ship navigation. This makes the fused data more targeted and effective, avoids interference from irrelevant or low-value data, improves data fusion quality, reduces the amount of fused data, and increases data fusion efficiency. Furthermore, the multimodal fused data includes location, monitoring, environmental, and sensor data, enabling it to more comprehensively and deeply reflect the ship's operational status. Anomaly information is identified using a pre-built ship safety rule base and anomaly detection model, and cross-validation of the multimodal data improves the accuracy of anomaly identification. Anomaly information is processed to obtain anomaly levels, and alarm information is determined using anomaly levels and anomaly information, achieving hierarchical management of anomalies. Alarms are issued by combining anomaly levels and specific anomaly information, making alarm information more accurate and targeted, facilitating managers to quickly understand the situation and take appropriate measures, thus improving the safety and standardization of ship navigation.

[0129] Based on the above Figure 1 The system shown in this application provides a method for intelligent video surveillance of ships. Figure 2 This is a flowchart of a ship intelligent video monitoring method provided in an embodiment of this application. The method may include steps 201 to 204.

[0130] Step 201: Acquire multiple modal data of the ship, including at least two of the following: position modal data, monitoring modal data, environmental modal data, or sensor modal data.

[0131] Step 202: Determine reference information, adaptively determine the target modal data to be fused from multiple modal data based on the reference information, and fuse the target modal data to obtain multimodal fused data. The reference information includes at least one of the following: environmental risk level, confidence level of multiple modal data, or mission requirements of different stages of ship navigation.

[0132] Step 203: Based on multimodal fusion data, a pre-built ship safety rule base and anomaly detection model, determine the abnormal information, including abnormal information of ship safety equipment, abnormal information of ship personnel actions, abnormal information of sensors and abnormal information of ship navigation trajectory.

[0133] Step 204: Process the abnormal information to obtain the abnormality level, and use the abnormality level and abnormal information to determine the alarm information.

[0134] It should be noted that step 201 has been described in detail in the relevant content of the data acquisition unit 101, step 202 has been described in detail in the relevant content of the data fusion unit 102, step 203 has been described in detail in the relevant content of the data processing unit 103, and step 204 has been described in detail in the relevant content of the alarm unit 104, and will not be repeated here.

[0135] This application adaptively selects target modal data to be fused based on reference information such as environmental risk levels, confidence levels of multiple modal data, and task requirements at different stages of ship navigation. This makes the fused data more targeted and effective, avoids interference from irrelevant or low-value data, improves data fusion quality, reduces the amount of fused data, and increases data fusion efficiency. Furthermore, the multimodal fused data includes location, monitoring, environmental, and sensor data, enabling it to more comprehensively and deeply reflect the ship's operational status. Anomaly information is identified using a pre-built ship safety rule base and anomaly detection model, and cross-validation of the multimodal data improves the accuracy of anomaly identification. Anomaly information is processed to obtain anomaly levels, and alarm information is determined using anomaly levels and anomaly information, achieving hierarchical management of anomalies. Alarms are issued by combining anomaly levels and specific anomaly information, making alarm information more accurate and targeted, facilitating managers to quickly understand the situation and take appropriate measures, thus improving the safety and standardization of ship navigation.

[0136] Figure 3 This is a schematic diagram of the structure of a client provided in an embodiment of this application. In exemplary embodiments of this application, such as... Figure 3 As shown, the client may include at least one processor 410, at least one network interface 420, a user interface 430, a bus 440, and a memory 450. The various components in the client are coupled together via the bus 440. It can be understood that the bus 440 is used to implement communication between these components. In addition to a data bus, the bus 440 also includes a power bus, a control bus, and a status signal bus.

[0137] For example, the processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0138] User interface 430 may include one or more output devices 431, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0139] Memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. Memory 450 may optionally include one or more storage devices physically located away from processor 410. Memory 450 may include volatile memory or non-volatile memory, or both. Non-volatile memory may be read-only memory (ROM), and volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory. In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures, or subsets or supersets thereof. The memory 450 may include an operating system, system programs for handling various basic system services and performing hardware-related tasks, such as a framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; the memory 450 may include a network communication module for reaching other defined devices via one or more (wired or wireless) network interfaces 420, for example, network interfaces 420 include: Bluetooth, WiFi, and Universal Serial Bus (USB); the memory 450 may include a presentation module for enabling the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 associated with the user interface 430 (e.g., a display screen, a speaker, etc.).

[0140] In some exemplary embodiments, the intelligent video surveillance device for ships provided in this application can be implemented in software. Figure 1The image shows a ship intelligent video monitoring device stored in memory 450. This device can be software in the form of programs and plug-ins, including one or more of the following software processing units (modules): data acquisition unit 101, data fusion unit 102, data processing unit 103, and alarm unit 104. These modules are logically integrated and can therefore be arbitrarily combined or further divided according to the functions they implement. The functions and roles of each module have been described in detail above and will not be repeated here.

[0141] In other exemplary embodiments, the intelligent video surveillance device for ships provided in this application can be implemented in hardware. For example, the intelligent video surveillance device for ships provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the intelligent video surveillance method for ships provided in the embodiments of this application. For example, the processor in the form of a hardware decoding processor can employ one or more application-specific integrated circuits (ASICs), digital signal processing (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components. The execution entity of the intelligent video surveillance method for ships provided in this application is mainly a client, specifically, the client can implement it by running the various computer programs described above.

[0142] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A ship intelligent monitoring video system, characterized in that, The system includes: The data acquisition unit is used to acquire multiple modal data of the ship, including at least one of position modal data, monitoring modal data, environmental modal data, or sensor modal data; A data preprocessing unit is configured to perform integrity detection on target objects in video format modal data to obtain detection results, wherein the target objects include ship safety equipment and ship personnel; if the detection results indicate that the target objects are occluded, determine supplementary features; and complete the target objects based on the supplementary features to obtain second preprocessed data, wherein when the visibility of the key features of the target objects is greater than or equal to the occlusion threshold, the supplementary features are determined based on the texture features or point cloud data of the occluded target objects; and when the visibility of the key features of the target objects is less than the occlusion threshold, the supplementary features are determined based on cross-camera data from multiple perspectives. A data fusion unit is used to determine reference information, which includes environmental risk level, confidence level of the multiple modal data, and mission requirements at different stages of ship navigation. Based on the environmental risk level and mission requirements at different stages of ship navigation, the data level of each modal data is determined. Each modal data includes the position modal data, the second preprocessed data, the environmental modal data, and the sensor modal data. The second preprocessed data is obtained by preprocessing the monitoring modal data. Modal data whose data level is greater than or equal to the level threshold are identified as first modal data; Determine the confidence level of the first modal data, and identify the modal data whose confidence level of the first modal data is greater than or equal to the confidence level threshold as the target modal data; The fusion weight of the target modal data is determined based on the data level, the confidence level, and the reference weight, wherein the reference weight is determined based on the historical navigation data of the ship system. The target modal data is fused based on the fusion weights to obtain multimodal fused data; The data processing unit is used to determine abnormal information based on the multimodal fusion data, the pre-built ship safety rule base and the anomaly detection model. The abnormal information includes abnormal information of ship safety equipment, abnormal information of ship personnel actions, abnormal information of sensors and abnormal information of ship navigation trajectory. An alarm unit is used to process the abnormal information, obtain the abnormal level of the abnormal information, and determine alarm information using the abnormal level and the abnormal information.

2. The system according to claim 1, characterized in that, The data processing unit is used for The pre-built ship safety rule base is used to perform anomaly detection on the multimodal fusion data to obtain the first anomaly information; The anomaly detection model is used to detect anomalies in the multimodal fusion data to obtain second anomaly information and anomaly probability. The abnormal information is determined based on the first abnormal information and the second abnormal information.

3. The system according to claim 1, characterized in that, The system also includes a data acquisition unit. The data acquisition unit is used to acquire the ship's historical maintenance information and equipment parameter information; The data processing unit is used to determine abnormal information based on the historical maintenance information, the equipment parameter information, the multimodal fusion data, the pre-built ship safety rule base, and the abnormality detection model.

4. The system according to any one of claims 1 to 3, characterized in that, The data preprocessing unit is further configured to perform image segmentation and target object recognition processing on the modal data in video format when the modal data is in video format, to obtain the first preprocessed data; The data fusion unit is further configured to adaptively determine the target modal data to be fused in the first preprocessed data based on the reference information.

5. The system according to any one of claims 1 to 3, characterized in that, The data preprocessing unit is also used for: Cross-validation was performed on the multiple modal data to obtain the cross-validation results; The cross-validation results identify anomalous data in the multiple modalities. The abnormal data is deleted from the multiple modal data to obtain the third preprocessed data; The data fusion unit is also used to adaptively determine the target modal data to be fused in the third preprocessed data based on the reference information.

6. The system according to any one of claims 1 to 3, characterized in that, The alarm unit is used for The abnormal information is processed to determine assessment parameters, which include at least one of risk spread rate, potential loss level, or urgency of response; The anomaly level is determined using the evaluation parameters. The alarm information is determined based on one of the anomaly level, the time of occurrence of the anomaly information, the location of occurrence of the anomaly information, or the area of ​​influence of the anomaly information. The alarm information includes the alarm method and the alarm area.

7. The system according to any one of claims 1 to 3, characterized in that, The system further includes a storage unit, the storage unit being used for: The various modal data are formatted and then stored.

8. The system according to any one of claims 1 to 3, characterized in that, The system further includes a storage unit, the storage unit being used for: The abnormal modal data corresponding to the abnormal information is encoded to obtain the content identifier corresponding to the abnormal modal data. The content identifier is associated with the reference modal data to obtain associated data. The associated data is stored. The reference modal data is data with a different data modality than the abnormal modal data and generated at the same time.

Images