Ship identity multi-modal verification method and system based on computer vision and deep learning

Abstract

The present application relates to the field of marine ship identification, and more particularly to a ship identity multi-modal verification method and system based on computer vision and deep learning, which acquires multi-modal data of a ship, wherein the multi-modal data includes visible light image data, laser radar point cloud data and ship identity information; according to the visible light image data, the visual features of the ship are extracted; the laser radar point cloud data is dynamically spatially calibrated; according to the calibration result, the three-dimensional structure features of the ship are extracted; the visual features, the three-dimensional structure features and the ship identity information are fused and input into a preset verification model, and the verification result is output through the verification model, wherein the verification result includes normal ships and abnormal ships. Different types of data provide multi-dimensional understanding of the ship, enhancing the accuracy and robustness of ship identification.

Description

Technical Field

[0001] This application relates to the field of maritime vessel identification technology, and in particular to a multimodal verification method and system for vessel identity based on computer vision and deep learning. Background Technology

[0002] Ship identification is the process of effectively identifying and verifying ships through various technological means, aiming to ensure the legality, safety, and reliability of vessels. It can be used for port management, maritime traffic monitoring, security precautions, and marine environmental protection, among other applications.

[0003] Existing technologies often rely on a single type of data for ship identification. For example, many methods depend on lidar for 3D spatial localization and obstacle detection, or on visible light images to identify the ship's appearance and shape. Environmental interference (such as weather factors, sea surface reflection, and the influence of other ships) poses a significant challenge to ship identification. While single-type data analysis can be effective, it's difficult to clearly identify the target ship's true behavior in complex environments with interference. For instance, misjudgments might identify a normally navigating ship as exhibiting abnormal behavior, or vice versa. Secondly, when existing technologies use multimodal data for identity verification, they encounter data conflict issues. Different sensors may vary in accuracy and stability. For example, lidar data may be more accurate than camera or radar data. However, if these sensor data conflict in the same context, existing technologies may struggle to determine which data source is more reliable. While simple voting mechanisms or discarding conflicting data are often used to handle these contradictions, this doesn't fully solve the problem, especially when data conflicts involve different data qualities, sensor performance, or external factors. This can easily lead to incorrect decisions or a lack of sufficient interpretability.

[0004] Therefore, existing technologies have shortcomings and need to be improved. Summary of the Invention

[0005] To address one or more problems in the existing technology, the main objective of this application is to provide a method and system for multimodal verification of ship identity based on computer vision and deep learning.

[0006] To achieve the aforementioned objectives, this application proposes a multimodal verification method for ship identity based on computer vision and deep learning, the method comprising:

[0007] Acquire multimodal data of the ship, wherein the multimodal data includes visible light image data, lidar point cloud data, and ship identification information;

[0008] Based on the visible light image data, extract the visual features of the ship;

[0009] Dynamic spatial calibration is performed on the lidar point cloud data;

[0010] Based on the calibration results, the three-dimensional structural features of the ship are extracted;

[0011] The visual features, three-dimensional structural features, and ship identity information are fused and input into a preset verification model. The verification model outputs the verification results, which include normal ships and abnormal ships.

[0012] This application also provides a ship identity multimodal verification system based on computer vision and deep learning, including:

[0013] The acquisition module is used to acquire multimodal data of the ship, wherein the multimodal data includes visible light image data, lidar point cloud data and ship identification information;

[0014] The first extraction module is used to extract the visual features of the ship based on the visible light image data;

[0015] The calibration module is used to perform dynamic spatial calibration on the lidar point cloud data;

[0016] The second extraction module is used to extract the three-dimensional structural features of the ship based on the calibration results;

[0017] The fusion module is used to fuse the visual features, three-dimensional structural features and ship identity information, and input them into a preset verification model. The verification model outputs the verification results, which include normal ships and abnormal ships.

[0018] This application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods described above.

[0019] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0020] The ship identity multimodal verification method and system based on computer vision and deep learning in this application acquires multimodal ship data, including visible light image data, lidar point cloud data, and ship identity information, enabling comprehensive and accurate capture of various ship information. Different types of data provide a multi-dimensional understanding of the ship, enhancing the accuracy and robustness of ship identification. By extracting visual features from visible light image data, the ship's appearance characteristics can be accurately identified, and detailed behavioral and state analysis can be provided using the rich visual information in the image data. The extraction of visual features provides a reliable foundation for ship behavior recognition. Dynamic spatial calibration of lidar point cloud data effectively solves the errors in point cloud data caused by environmental changes, ship movement, or sensor position variations, thereby improving the accuracy of ship 3D structural feature extraction. Dynamic calibration optimizes the 3D feature extraction process, making the identification of the ship's spatial position and shape more accurate. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a multimodal verification method for ship identity based on computer vision and deep learning according to an embodiment of this application.

[0022] Figure 2 This is a flowchart illustrating a multimodal verification method for ship identity based on computer vision and deep learning according to an embodiment of this application.

[0023] Figure 3 This is a schematic block diagram of a ship identity multimodal verification system based on computer vision and deep learning according to an embodiment of this application;

[0024] Figure 4 This is a schematic block diagram of the structure of a computer device according to an embodiment of this application.

[0025] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0027] Reference Figure 1 This application provides a multimodal verification method for ship identity based on computer vision and deep learning, the method comprising:

[0028] S1. Acquire multimodal data of the ship, wherein the multimodal data includes visible light image data, lidar point cloud data and ship identification information;

[0029] S2. Extract the visual features of the ship based on the visible light image data;

[0030] S3. Perform dynamic spatial calibration on the lidar point cloud data;

[0031] S4. Based on the calibration results, extract the three-dimensional structural features of the ship;

[0032] S5. The visual features, three-dimensional structural features and ship identity information are fused and input into a preset verification model. The verification results are output through the verification model, wherein the verification results include normal ships and abnormal ships.

[0033] As described in steps S1-S2 above, visible light cameras capture the ship's external features, such as shape, color, and size, to identify the ship's visual information. LiDAR (Light Detection and Ranging) is used to acquire the ship's three-dimensional spatial information; point cloud data provides detailed information about the ship's surface structure, overcoming the limitations of traditional image data in harsh environments. Ship identification information may include AIS (Automatic Identification System) data or other pre-registered identification data, providing important information such as the ship's navigation history and type. By fusing multiple data sources, the system can identify ships from different angles, overcoming the limitations of a single data source. For example, LiDAR can still effectively provide three-dimensional spatial data in poor lighting conditions, while visible light images can supplement detailed information about the ship during the day in good weather. Ship identification information supports the verification of the ship's true identity. Visible light image data is two-dimensional image data obtained through standard camera equipment. This image data contains visual features such as the ship's color, shape, size, and markings. In the context of deep learning, convolutional neural networks (CNNs) or other computer vision methods can automatically extract these features from the images. These features help identify the ship's type, size, markings, and status. By extracting visual features from visible light images, the system can effectively identify the appearance and shape of ships under good lighting conditions. This approach compensates for the lack of detailed information provided by lidar, and is particularly advantageous in identifying external features and ship shapes.

[0034] As described in steps S3-S5 above, lidar point cloud data provides three-dimensional spatial information about the ship. However, due to the movement, tilt, or other dynamic factors of the lidar system, the point cloud data may contain spatial errors. Therefore, dynamic spatial calibration can correct the point cloud data and ensure its accuracy in space. Dynamic calibration typically eliminates errors caused by ship motion by aligning multiple point cloud data sets or adjusting them based on other known reference objects (such as GPS positioning, inertial measurement units, IMUs, etc.). Through spatial calibration, lidar data can more accurately reflect the actual spatial structure of the ship. This step ensures the reliability of the point cloud data, thus providing accurate spatial information for subsequent ship three-dimensional feature extraction and ship identification, avoiding the influence of errors introduced by dynamic factors on the identification results. After spatial calibration, lidar point cloud data can more accurately reflect the three-dimensional morphology of the ship. By further analyzing the calibrated point cloud data, the three-dimensional structural features of the ship, such as hull outline, size, position, and relative angles, can be extracted. These three-dimensional features provide global information on the ship's morphology and are key to distinguishing different ship types and morphologies. Extracting the three-dimensional structural features of ships helps the system clearly identify their geometric features in complex environments (such as fog, haze, and nighttime), especially when two-dimensional images are insufficient for identification. The application of three-dimensional features can improve the accuracy and robustness of identification. Fusion of different types of data (visual features, three-dimensional structural features, and identity information) is key to improving the accuracy of ship identification systems. Deep learning methods (such as multimodal learning) allow the fusion of data from different modalities to extract more information and improve recognition accuracy. Pre-defined verification models (such as deep neural networks), based on training data, can analyze and combine these features to detect abnormal ships. Visual features come from the morphological and color information of visible light images. Three-dimensional structural features come from the ship's geometric shape provided by three-dimensional data from LiDAR. Ship identity information provides background information about the ship (such as type, navigation history, etc.), helping to verify whether the ship belongs to the normal category. Through the fusion of multimodal data, the system can more comprehensively analyze the ship's identity and behavior, thereby reducing the risk of false positives and false negatives. Based on the results output by the verification model, the system can accurately distinguish between normal and abnormal ships, providing reliable ship identity verification and anomaly detection capabilities.

[0035] Reference Figure 2 In one embodiment, the step of performing dynamic spatial calibration on the lidar point cloud data includes:

[0036] S31. The six-degree-of-freedom motion parameters within the scanning cycle are acquired in real time through the inertial measurement unit carried by the ship.

[0037] S32. Based on motion parameters, establish a motion compensation model for the lidar point cloud;

[0038] S33. Input the initial point cloud data into the motion compensation model, resample the initial point cloud data through the motion compensation model, and output the resampled result to obtain the motion-compensated point cloud data.

[0039] S34. Convert the motion-compensated point cloud data to the ship's main coordinate system to complete the dynamic spatial calibration.

[0040] As described above, an inertial measurement unit (IMU) is a common sensor capable of measuring the acceleration and angular velocity of an object. For ships, the IMU can provide real-time six-degree-of-freedom motion parameters through accelerometers and gyroscopes in three axes: acceleration in three translational directions (along the X, Y, and Z axes) and angular velocity in three rotational directions (around the X, Y, and Z axes). These six-degree-of-freedom motion parameters accurately reflect the ship's motion state during the scanning cycle, including dynamic changes such as position, velocity, and tilt. By acquiring the ship's six-degree-of-freedom motion parameters in real time, necessary motion information can be provided for subsequent LiDAR point cloud data calibration. The motion parameters provided by the IMU can effectively compensate for the ship's motion deviations in dynamic environments, ensuring the accuracy of the point cloud data. Without this motion data, the point cloud data may suffer spatial errors due to factors such as ship swaying and tilting. The motion compensation model is a mathematical model established based on the six-degree-of-freedom motion parameters provided by the IMU, aiming to eliminate point cloud data errors caused by the ship's motion during the scanning process. Specifically, the model predicts and adjusts the spatial position of each lidar point cloud based on the ship's speed, acceleration, and angular velocity, ensuring that the point cloud data accurately reflects the ship's actual position and attitude. This compensation model is typically based on kinematic theory and can correct the point cloud data through coordinate transformations. Establishing a motion compensation model effectively eliminates errors caused by ship motion in the lidar point cloud data, improving its spatial accuracy. Thus, even when the ship is in motion, the point cloud data collected by the lidar can still match the actual spatial position, avoiding data distortion caused by motion errors. In this step, the initial point cloud data acquired by the lidar is input into the motion compensation model. The motion compensation model resamples the point cloud data based on the ship's motion parameters, that is, by performing motion compensation on the position of each point to make it conform to the ship's true spatial position. The resampling process may include translation, rotation, and other transformations of the point cloud's spatial coordinates to compensate for motion deviations generated during the scanning process. Through this process, the compensated point cloud data more accurately reflects the ship's actual three-dimensional spatial distribution. The motion-compensated point cloud data obtained through resampling effectively corrects errors caused by ship motion, making the point cloud data more accurate in spatial positioning. The resampling process ensures high data precision, making subsequent processing and analysis (such as ship identification and 3D modeling) more reliable. This process transforms the motion-compensated point cloud data from the lidar coordinate system to the ship's coordinate system. LiDAR is typically installed at a location on the ship, acquiring point cloud data relative to the lidar coordinate system, while the ship's coordinate system is defined with the ship as the reference frame. In this step, coordinate transformation algorithms (such as rotation matrices, affine transformations, etc.) are used to transform the compensated point cloud data into the ship's coordinate system, ensuring that the point cloud data remains consistent with the ship's actual motion and attitude.Converting point cloud data to the ship's main coordinate system more accurately reflects the ship's position and attitude in the actual environment. This calibration aligns the point cloud data with the ship's own reference frame, providing more precise data support for subsequent tasks such as ship identification, path planning, and spatial analysis.

[0041] In one embodiment, prior to the step of fusing the visual features, three-dimensional structural features, and ship identification information, the method further includes:

[0042] The visual features and three-dimensional structural features are acquired, and the visual features and three-dimensional structural features are input into a preset spatial model. The current behavior features of the ship are generated through the spatial model.

[0043] The historical behavioral characteristics of the vessel are obtained, and the vessel trajectory parameters for a preset time period are predicted by combining the historical behavioral data with the current behavioral data.

[0044] Based on the predicted ship trajectory parameters, the actual trajectory parameters of the ship at the predicted time are obtained in real time, and the deviation coefficient between the actual trajectory parameters and the predicted trajectory parameters is analyzed.

[0045] The deviation coefficient is quantified to generate a confidence score for the ship;

[0046] The confidence score of the vessel is analyzed. When the confidence score is greater than a preset confidence score threshold, the vessel is judged to be acting credibly.

[0047] Based on the premise that the vessel is behaving reliably, the visual features, three-dimensional structural features, and vessel identity information are fused together.

[0048] As mentioned above, visual features and 3D structural features are acquired from the ship's external environment and used to describe information such as the ship's appearance, position, shape, and motion. Visual features can be obtained through cameras, visual sensors, or image processing technologies, while 3D structural features are acquired through LiDAR or other 3D modeling technologies. The preset spatial model is a mathematical or computational model used to comprehensively analyze these features and generate current behavioral characteristics. The spatial model may be based on principles of physics, machine learning, or statistics, effectively combining this multimodal information to extract the ship's behavioral patterns, such as its direction of travel, speed, and heading. By inputting visual features and 3D structural features into the spatial model, a feature set that comprehensively reflects the ship's behavior can be generated. These features help describe the ship's motion state, heading, speed, and other dynamic information. Historical behavioral features reflect the ship's behavioral patterns over a past period, helping to understand the ship's behavioral trends and habits. Combining historical data and current behavioral data, predictive models (such as regression analysis, time series analysis, machine learning, etc.) can be used to predict the ship's trajectory over a future period. The preset time is the target time point predicted by the model, and the trajectory parameters include position, speed, and heading, reflecting the ship's future motion trajectory. By combining historical and current behavioral data for prediction, the future trajectory of a ship can be accurately predicted, providing a foundation for subsequent deviation analysis and behavior assessment. This is achieved by monitoring the ship's actual trajectory in real time and comparing it with the predicted trajectory. Real-time acquired trajectory parameters (such as position and speed) are typically provided by positioning systems (such as GPS and inertial navigation systems). Analyzing the deviation coefficient (i.e., error or difference) between the actual and predicted trajectories is to assess the degree of abnormality in the ship's behavior. A large deviation coefficient indicates a significant difference between the ship's actual and predicted behavior, potentially suggesting an abnormal trajectory or a deviation from the expected course. Calculating the deviation coefficient effectively assesses the stability and reliability of the ship's behavior. A small deviation coefficient indicates that the ship's behavior is consistent with expectations; a large deviation coefficient suggests potential abnormal behavior. This provides a quantitative basis for subsequent confidence scoring and behavior assessment, helping to distinguish between normal and abnormal behavior. The quantification process of the deviation coefficient typically involves standardization and weighting based on the magnitude of the difference, ultimately resulting in a confidence score. The confidence score is usually calculated using statistical or machine learning models and measures the credibility of the ship's behavior. If the deviation coefficient between the ship's actual behavior and the predicted trajectory is small, it indicates that the ship's behavior is relatively reliable, and the confidence score is high; conversely, if the deviation coefficient is large, the confidence score is low. The confidence score provides a basis for subsequent decision-making and helps determine whether the ship's behavior is credible. A higher confidence score indicates that the ship's behavior is relatively stable and normal, while a lower confidence score may indicate that the ship is exhibiting abnormal behavior.Quantifying confidence scores provides an operational standard for automating the assessment of ship behavior. By comparing a ship's confidence score with a preset threshold, it's determined whether the ship's behavior is deemed trustworthy. The preset confidence score threshold, set based on experience, historical data, or system requirements, distinguishes between normal and abnormal behavior. If a ship's confidence score exceeds this threshold, its behavior is considered consistent with expectations and thus trustworthy; conversely, a low confidence score indicates untrustworthy behavior, potentially requiring further intervention or monitoring. When a ship's behavior is deemed trustworthy, the system fuses its visual features, 3D structural features, and identity information. Fusion refers to combining this information to form a comprehensive ship identification feature set. This step can utilize machine learning algorithms, weighted averaging, Bayesian inference, and other methods to fuse these features, resulting in a more comprehensive and accurate ship profile. This enables a multimodal ship identification method. By fusing visual features, 3D structural features, and ship identity information, a more comprehensive and accurate description of ship behavior can be obtained. This enables the system to gain a more comprehensive understanding of the ship's status, optimize the ship identification and tracking process, and improve the accuracy and reliability of ship behavior monitoring.

[0049] In one embodiment, after the step of quantifying the deviation coefficient and generating a confidence score for the ship, the method further includes:

[0050] When the confidence score is less than or equal to a preset confidence score threshold, the current vessel is automatically marked as a suspicious target;

[0051] Based on the judgment results, environmental parameters of the sea area where the ship is located are collected in real time.

[0052] Analyze whether the conditions for interference are met based on the environmental parameters.

[0053] When the environmental parameters meet the interference conditions, the interference pattern on the behavioral characteristics is analyzed based on the environmental parameters, and the ship is re-verified as an abnormal ship based on the interference pattern.

[0054] As mentioned above, in ship behavior analysis, the confidence score is a numerical value used to quantify the degree of anomalousness in ship behavior. It reflects the analysis system's level of confidence in whether a ship is abnormal. This threshold is a value pre-set by the system designer; when a ship's confidence score is lower than or equal to this threshold, the system considers the ship's behavior to have high uncertainty or anomalousness. Automatic labeling based on confidence scores aims to quickly identify potentially anomalous ships, avoiding manual intervention and over-reliance on manual analysis. This allows for rapid screening of potentially problematic targets for further analysis, improving the efficiency of identifying anomalous ship behavior and reducing false alarms and missed detections. In practical applications, it enables rapid identification of suspicious targets, reduces the workload of operators, and allows for timely further action. Environmental factors, such as weather conditions, ocean currents, visibility, wind speed, and waves, can interfere with ship behavior. Real-time collection of these parameters provides more background information for subsequent judgments on whether ship behavior is abnormal. The external environment is a significant interfering factor in ship behavior assessment, affecting the ship's trajectory, speed, etc., and must be considered to ensure the accuracy of the behavior analysis results. By acquiring environmental parameters in real time, we can more accurately understand ship behavior, eliminate behavioral anomalies caused by external factors, and thus avoid misjudgments. It enables us to cope with changing environmental factors; ship behavior may vary in different sea environments, and real-time data collection allows for flexible adjustments to analysis strategies. The environmental interference condition step is to determine whether the ship is affected by environmental factors, thereby influencing its behavioral characteristics. If environmental conditions (such as strong winds and large waves) can cause changes in ship behavior, these conditions are considered to meet the interference conditions. Environmental factors may cause changes in ship speed or deviation from course, which does not necessarily indicate ship anomalies but may be normal phenomena caused by environmental influences. Therefore, analyzing whether interference conditions are met can effectively distinguish between truly abnormal behavior and natural fluctuations caused by environmental factors. Judging solely based on ship behavioral characteristics may mistakenly classify ships affected by environmental interference as abnormal. By analyzing environmental parameters, we can further verify whether the ship is truly abnormal. Interference analysis is only performed when environmental conditions meet the interference requirements, reducing the interference of irrelevant factors and making the determination of abnormal ship behavior more targeted. When environmental conditions meet the criteria for disturbance, the system needs to analyze the specific impact patterns of these environmental factors on ship behavior. For example, strong winds may cause a ship to deviate from its course, and waves may cause a decrease in ship speed. By analyzing these disturbance patterns, the system can simulate and predict changes in ship behavior under different environmental conditions. Analysis of historical data can reveal behavioral characteristic deviations under different environmental conditions and establish corresponding disturbance patterns. These patterns can serve as standards for determining whether a ship is being disturbed. By understanding disturbance patterns, the system can accurately distinguish between changes in ship behavior caused by the external environment and genuine abnormal behavior, thereby avoiding misjudgments.Multiple interference modes are established for different sea areas and environmental conditions, enabling the system to operate normally in various complex environments and exhibiting better adaptability. After analyzing the interference modes, it is necessary to reassess whether the vessel is abnormal. By revising the judgment of the vessel's behavior based on the interference modes, anomalies caused by environmental factors are eliminated, thus making a more accurate judgment. If the deviation of the vessel's behavior is caused by the interference mode, then the vessel should not be judged as abnormal; if the interference mode does not conform to expectations, it indicates that the vessel is abnormal. Through re-verification, the impact of environmental interference can be effectively reduced, ensuring more accurate judgment of vessel anomalies.

[0055] In one embodiment, before the step of inputting the reflectivity parameter and deformation offset parameter into the deviation value analysis model, the method further includes:

[0056] Real-time acquisition of ambient light intensity;

[0057] When the light intensity is greater than or equal to a preset light intensity threshold, the polarizing filter is activated.

[0058] Based on the light intensity and the state of the polarizing filter, the calibrated reflectivity parameters are calculated and then input into the deviation analysis model.

[0059] As mentioned above, ambient light intensity refers to the intensity of light illuminating a target object. In practical applications, light intensity has a significant impact on the reflectivity of an object's surface, especially in optical measurements and image recognition. Changes in light intensity can cause fluctuations in reflectivity parameters, thus affecting the accuracy of subsequent analyses. Therefore, real-time acquisition of light intensity data helps to dynamically adjust the input parameters in the deviation analysis model. A polarizing filter can control the polarization state of light reflected from the object's surface, effectively filtering out stray light and unnecessary reflected light caused by changes in the illumination angle or irregularities in the reflective surface. Enabling a polarizing filter can eliminate these interferences, especially in strong light environments, effectively improving the accuracy of reflectivity measurements. When the light intensity reaches a certain threshold, the scattering of reflected light may increase, affecting measurement accuracy; therefore, a polarizing filter is used to improve measurement accuracy. The calibrated reflectivity parameter is the reflectivity data obtained after adjusting for light intensity and the polarizing filter. Since the properties of reflected light differ under different light intensities, the reflectivity parameter needs to be adjusted according to the light intensity and the filter's state. By dynamically calculating the calibrated reflectance, errors caused by environmental changes can be eliminated, ensuring data accuracy. Specifically, when the light intensity is high, polarizing filters can effectively reduce strong light interference, helping the system obtain more accurate reflectance values. The calibrated reflectance parameters are then input into the deviation analysis model for further analysis of the target object's deviation. By introducing the calibrated reflectance data, the model can calculate the deviation value based on the actual object's reflection characteristics, thereby determining the magnitude of the deviation. The deviation analysis model's role here is to help the system assess the target object's deformation, damage, or irregularity based on changes in the reflectance parameters. By inputting the calibrated reflectance parameters, the deviation analysis model can provide more accurate deviation detection results, especially under different lighting conditions. The calibrated reflectance data effectively improves the accuracy of deviation calculations, ensuring the reliability of the final results. This reduces deviation detection errors caused by changes in lighting, improving the system's robustness and accuracy.

[0060] In one embodiment, the step of analyzing interference patterns on behavioral characteristics based on the environmental parameters and re-verifying whether the vessel meets the criteria for an abnormal vessel based on the interference patterns includes:

[0061] The collected environmental parameters are subjected to interference pattern identification, wherein the interference patterns include sensing interference and mobile interference;

[0062] When the interference mode is perceived interference, the confidence score is reduced by a preset ratio;

[0063] Based on the adjusted confidence score threshold, the confidence score of the vessel is re-analyzed. When the confidence score is greater than the adjusted confidence score threshold, the vessel is judged to be believable.

[0064] When the confidence score is less than or equal to the adjusted confidence score threshold, the vessel is determined to be an abnormal vessel.

[0065] As described above, by analyzing environmental parameters, the system identifies interference patterns in the sea area where the ship is located that may affect its behavior. Interference patterns can be divided into perceived interference and moving interference. Perceived interference refers to environmental factors (such as low visibility, poor weather, etc.) affecting sensor data acquisition, causing a deviation between the ship's actual behavior and the behavior monitored by the system. Moving interference refers to external factors such as ocean currents, wind speed, and wave height causing changes in the ship's physical motion, such as affecting its course and speed. Identifying interference patterns allows for an accurate understanding of the impact of environmental factors on ship behavior. Perceived interference and moving interference are analyzed from both data acquisition and ship dynamics perspectives, providing precise basis for subsequent processing. Separately identifying these two types of interference helps to accurately determine the nature of the interference, thereby enabling targeted measures. By identifying different types of interference, the system can better distinguish between behavioral changes caused by external environmental factors and actual abnormal behavior. Handling perceived interference and moving interference separately allows the system to flexibly respond to different types of interference, making it more targeted. Perceived interference can cause deviations between the data acquired by sensors and the actual situation, thus affecting the confidence score of ship behavior. If perceptual interference is detected, the system will lower the confidence score to reduce the confidence level in identifying abnormal ship behavior, thus avoiding misclassifying perceptual interference-induced anomalies as genuine abnormal behavior. The reduction ratio is based on a preset value set by the system to ensure effective mitigation of the impact of perceptual interference on the confidence score. The ratio should be set to correct sensor errors without excessively affecting normal judgment. Perceptual interference may lead to erroneous data, directly affecting the ship's confidence score; therefore, the score needs to be reduced according to the interference intensity. Controlling the scoring range avoids incorrectly labeling ship behavior as abnormal, ensuring that the adjustment of the confidence score does not lead to overcorrection, resulting in missed or false alarms. By reducing the confidence score under perceptual interference conditions, misjudging the ship as abnormal due to sensor malfunction or environmental factors can be avoided. By correcting the bias caused by perceptual interference, the system's judgment of ship behavior becomes more stable and reliable. When perceptual interference is detected, the system will adjust the confidence score according to a preset ratio, and then reassess the ship's confidence score using a new threshold. The confidence score threshold is the standard used to determine whether the ship is normal; the adjusted threshold can more accurately reflect the actual situation after the interference. Dynamically adjusting the confidence score threshold allows the system to assess ships more flexibly and adapt to different environmental factors. After addressing interference, reapplying the new threshold ensures the accuracy of the judgment. Reanalyzing based on the adjusted confidence score threshold reduces the impact of interference on ship behavior judgments, making the results more realistic and reliable. The system automatically adjusts the confidence score threshold according to environmental changes, improving performance in complex environments. When a ship's confidence score exceeds the adjusted threshold, it indicates that after interference correction, the ship's behavior is credible and does not exhibit abnormal behavior.A confidence score greater than the adjusted threshold indicates that the ship's behavior conforms to the system's preset normal range, thus classifying the ship's behavior as trustworthy. This judgment step ensures that ship behavior is considered trustworthy only when the corrected confidence score exceeds a reasonable range. This step, through strict threshold control, avoids erroneous judgments due to incorrect interference patterns. Strict control of the confidence score threshold reduces misjudgments caused by interference patterns, ensuring accurate identification of trustworthy behavior. Increasing the rigor of the judgment step ensures that ship behavior is only considered trustworthy under reliable conditions, thereby improving accuracy. If the ship's confidence score is less than or equal to the adjusted threshold, it indicates that the ship's behavior exceeds the normal range and may be abnormal. In this case, the system will classify the ship as abnormal. If the score does not reach the adjusted threshold, the ship can be considered to be behaving abnormally under the influence of interference, and the system will mark it as an abnormal ship. Judging abnormalities based on confidence scores ensures that anomaly judgments are only made when ship behavior deviates from the normal range. By adjusting the confidence score threshold to adapt to different interference environments, the system can flexibly respond to different types of interference, avoiding misjudgments or omissions. It can accurately determine whether a ship is abnormal, avoiding the influence of interference factors on the judgment. By dynamically adjusting the confidence score threshold, the accuracy of identifying abnormal behavior is improved, and the probability of false alarms and false negatives is reduced.

[0066] In one embodiment, the step of analyzing the interference categories on behavioral characteristics based on the environmental parameters and re-verifying whether the vessel meets the criteria for an abnormal vessel based on the interference categories further includes:

[0067] When the interference mode is mobile interference, the actual trajectory parameters are corrected by a preset trajectory compensation model;

[0068] Based on the corrected actual trajectory parameters, the ship's confidence score is regenerated;

[0069] When the confidence score of the regenerated ship is greater than the preset confidence score threshold, the ship is judged to be acting credibly.

[0070] When the confidence score of a regenerated vessel is less than or equal to a preset confidence score threshold, the vessel is determined to be an abnormal vessel.

[0071] As mentioned above, environmental parameters include factors such as the weather, ocean conditions, and maritime traffic. These factors affect the ship's behavior and may cause changes in its trajectory, speed, and heading. By analyzing these environmental parameters, the type of disturbance the ship is experiencing can be determined, such as "moving disturbance" or "static disturbance." This analysis helps to understand whether the ship's behavior is affected by external factors, rather than by ship malfunctions or human error. By identifying the type of disturbance, the system can clarify the external causes of changes in ship behavior. Based on the analysis of the disturbance type, the system reassesses whether the ship's behavior is abnormal. Considering the disturbance factors, the system no longer relies solely on preliminary data of ship behavior but comprehensively considers the influence of the external environment. For example, in the case of moving disturbance, the ship may deviate from its normal course due to factors such as waves and wind speed; this behavior itself may not indicate a malfunction or abnormality. Re-verification ensures that the system can appropriately adjust for disturbances when assessing whether the ship is abnormal. It avoids misjudging behavioral deviations caused by external factors in complex environments as abnormal. This step improves the accuracy of behavior verification, making the anomaly determination more consistent with reality. When a ship is detected to be subject to "moving disturbances," such as wind or ocean currents causing changes in its trajectory, a pre-defined trajectory compensation model comes into play. This model identifies the characteristics and effects of the disturbance, calculates its specific impact on the ship's trajectory, and corrects the actual trajectory parameters. For example, if wind causes the ship to deviate from its course, the model will correct the ship's trajectory data based on wind intensity, direction, and other parameters, restoring it to a state closer to normal. The corrected trajectory parameters more realistically reflect the ship's actual behavior under environmental disturbances. Through this correction, the system can effectively reduce the impact of environmental disturbances on the ship's trajectory assessment, thereby improving the accuracy of ship behavior. After correcting the actual trajectory parameters, the new trajectory data is used to recalculate the ship's confidence score. The confidence score is an important indicator for measuring whether ship behavior is normal, and typically considers factors such as the ship's heading, speed, and deviation from the expected trajectory. By inputting the corrected trajectory data into the confidence score algorithm, the system can assess whether the ship's behavior is normal. The corrected trajectory data makes the confidence score more accurately reflect the ship's true behavior in the current environment, thus ensuring that the assessment of ship behavior is closer to reality. When the regenerated confidence score exceeds a preset confidence score threshold, it indicates that the ship's behavior is within the range of its disturbance-corrected trajectory and conforms to expected normal behavior. This threshold is typically set based on historical data of ship behavior and safety standards. When the confidence score is higher than this threshold, it means that the ship's behavior is within an acceptable range. The system can determine whether the ship's behavior meets predetermined standards. If the confidence score is high, it indicates that the ship's behavior is consistent with normal behavior, and the ship can be considered trustworthy. This helps reduce false alarms and ensures the safety and efficiency of the ship.When the regenerated confidence score is lower than or equal to the preset confidence score threshold, it indicates that the ship's behavior deviates significantly from normal expectations and may be abnormal. This abnormality may be caused by ship malfunction, abnormal operation, or other unforeseen factors. By comparing the score with the preset threshold, the system can accurately determine whether an abnormality exists.

[0072] In one embodiment, before the step of fusing the visual features, three-dimensional structural features, and ship identification information, the method further includes:

[0073] When the perceptual interference is present, the fusion weight of the visual features is reduced, and the fusion weight of the three-dimensional structural features is increased.

[0074] Based on the adjustment of fusion weights, the visual features, three-dimensional structural features, and ship identity information are fused.

[0075] As mentioned above, perceptual interference refers to external factors that affect the system's perception process, such as noise, occlusion, and adverse weather conditions. These interferences can reduce the accuracy of visual features. In the presence of perceptual interference, the reliability of visual features (such as image data, color, and shape) is affected. Therefore, to reduce misjudgments, the system reduces the weight of visual features in the fusion process. Simultaneously, three-dimensional structural features (such as the spatial layout, size, and shape of a ship) are generally less affected by visual interference; therefore, their weight should be increased in the case of perceptual interference. Through this adjustment, the system can rely more heavily on three-dimensional structural features to determine ship behavior, ensuring more stable and reliable fusion results. By reducing the fusion weight of visual features and increasing the fusion weight of three-dimensional structural features, the system can maintain high recognition accuracy in interfering environments. This weight adjustment avoids misleading effects caused by perceptual interference, improves the system's performance in complex environments, and ensures accurate ship behavior analysis, especially under strong interference. During fusion, the weights of visual features and three-dimensional structural features need to be adjusted according to the current environment (whether perceptual interference exists) to determine the contribution of each feature to the final fusion result. After adjustments, the system analyzes the ship's behavior based on these weights, generating a comprehensive identification result. This approach allows for flexible adaptation to environmental changes and ensures that even under significant perceptual interference, the system can still rely heavily on relatively stable 3D structural features and ship identification information to enhance overall judgment accuracy. By adjusting the weights of feature fusion, the system maintains high stability and accuracy even under perceptual interference. In particular, the increased weighting of 3D structural features compensates for the deficiencies of visual features under interference. Ultimately, by integrating 3D structure, visual features, and ship identification information, the system can more comprehensively evaluate ship behavior, ensuring the accuracy and reliability of ship identification.

[0076] In another embodiment, when conflicting data from LiDAR, visible light, and AIS occur simultaneously, traditional methods rely on simple voting or discarding data, resulting in insufficient decision-making basis. This method, however, combines data from different types of sensors to automatically assess the reliability of the data source and employs a more intelligent handling mechanism when data conflicts arise, rather than simply relying on simple voting or discarding conflicting data. In this implementation, considering the differences in accuracy and stability among different sensors (such as LiDAR, cameras, and radar), the system dynamically evaluates sensor performance and prioritizes data sources with higher reliability. This method effectively reduces data conflicts caused by inconsistent sensor quality or environmental interference, reduces false positives, and improves the accuracy of identity verification. Specifically, by analyzing the quality of data from various sensors through a fusion algorithm, the system can adaptively adjust the data weights to ensure that the most representative and valid information is selected when conflicts occur among multiple data sources. Furthermore, an error detection mechanism is employed to identify and exclude low-quality data in real time, ensuring a high degree of interpretability and accuracy throughout the entire identity verification process.

[0077] In one feasible embodiment, ports, as important logistics hubs, often experience environmental impacts from ship transportation, loading and unloading, and warehousing activities, such as exhaust emissions, noise pollution, and wastewater discharge. Therefore, port management departments need to conduct real-time monitoring and environmental management to ensure that port environmental standards comply with relevant regulations. Identity verification of incoming vessels at the port can be achieved by linking the vessel's AIS (Automatic Identification System) data with emission monitoring data to confirm the vessel's identity and obtain its emission data. If the vessel has a history of exceeding pollution emission standards, the system will increase the warning level. Pollutant concentration exceedance detection and early warning: The system compares the real-time detected emission data with port environmental standards. When the concentration of a certain pollutant exceeds the standard, an early warning notification is immediately triggered. This warning information will include the vessel ID, the pollutant exceeding the standard, and the pollution concentration, and will be simultaneously sent to the terminal devices of port management personnel.

[0078] Reference Figure 3 This application also provides a ship identity multimodal verification system based on computer vision and deep learning, including:

[0079] Acquisition module 1 is used to acquire multimodal data of the ship, wherein the multimodal data includes visible light image data, lidar point cloud data and ship identification information;

[0080] The first extraction module 2 is used to extract the visual features of the ship based on the visible light image data;

[0081] Calibration module 3 is used to perform dynamic spatial calibration on the lidar point cloud data;

[0082] The second extraction module 4 is used to extract the three-dimensional structural features of the ship based on the calibration results;

[0083] The fusion module 5 is used to fuse the visual features, three-dimensional structural features and ship identity information, and input them into a preset verification model. The verification model outputs the verification results, which include normal ships and abnormal ships.

[0084] As described above, it is understood that each component of the ship identity multimodal verification system based on computer vision and deep learning proposed in this application can realize the function of any of the ship identity multimodal verification methods based on computer vision and deep learning described above, and the specific structure will not be repeated.

[0085] Reference Figure 4 This application also provides a computer device, which may be a server, and its internal structure may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores monitoring data and other data. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a multimodal verification method for ship identity based on computer vision and deep learning.

[0086] The processor described above executes the multimodal ship identity verification method based on computer vision and deep learning, comprising: acquiring multimodal data of the ship, wherein the multimodal data includes visible light image data, lidar point cloud data, and ship identity information; extracting visual features of the ship based on the visible light image data; performing dynamic spatial calibration on the lidar point cloud data; extracting three-dimensional structural features of the ship based on the calibration results; fusing the visual features, three-dimensional structural features, and ship identity information, and inputting them into a preset verification model; and outputting the verification results through the verification model, wherein the verification results include normal ships and abnormal ships.

[0087] One embodiment of this application also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a multimodal verification method for ship identity based on computer vision and deep learning, including the following steps: acquiring multimodal data of the ship, wherein the multimodal data includes visible light image data, lidar point cloud data, and ship identity information; extracting visual features of the ship based on the visible light image data; performing dynamic spatial calibration on the lidar point cloud data; extracting three-dimensional structural features of the ship based on the calibration results; fusing the visual features, three-dimensional structural features, and ship identity information, and inputting them into a preset verification model; and outputting verification results through the verification model, wherein the verification results include normal ships and abnormal ships.

[0088] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media provided in this application and in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0089] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0090] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A multimodal verification method for ship identity based on computer vision and deep learning, characterized in that, The method includes: Acquire multimodal data of the ship, wherein the multimodal data includes visible light image data, lidar point cloud data, and ship identification information; Based on the visible light image data, extract the visual features of the ship; Dynamic spatial calibration is performed on the lidar point cloud data; Based on the calibration results, the three-dimensional structural features of the ship are extracted; The visual features, three-dimensional structural features, and ship identity information are fused and input into a preset verification model. The verification model outputs the verification results, which include normal ships and abnormal ships. Before the step of fusing the visual features, three-dimensional structural features, and ship identification information, the method further includes: The visual features and three-dimensional structural features are acquired, and the visual features and three-dimensional structural features are input into a preset spatial model. The current behavior features of the ship are generated through the spatial model. The historical behavioral characteristics of the vessel are obtained, and the vessel trajectory parameters for a preset time period are predicted by combining the historical behavioral data with the current behavioral data. Based on the predicted ship trajectory parameters, the actual trajectory parameters of the ship at the predicted time are obtained in real time, and the deviation coefficient between the actual trajectory parameters and the predicted trajectory parameters is analyzed. The deviation coefficient is quantified to generate a confidence score for the ship; The confidence score of the vessel is analyzed. When the confidence score is greater than a preset confidence score threshold, the vessel is judged to be acting credibly. Based on the premise that the vessel is behaving reliably, the visual features, three-dimensional structural features, and vessel identity information are fused together. After the step of quantifying the deviation coefficient and generating a confidence score for the ship, the method further includes: When the confidence score is less than or equal to a preset confidence score threshold, the current vessel is automatically marked as a suspicious target; Based on the judgment results, environmental parameters of the sea area where the ship is located are collected in real time. Analyze whether the conditions for interference are met based on the environmental parameters. When the environmental parameters meet the interference conditions, the interference pattern on the behavioral characteristics is analyzed based on the environmental parameters, and the ship is re-verified as an abnormal ship based on the interference pattern. The steps of analyzing the interference patterns on behavioral characteristics based on the environmental parameters and re-verifying whether the vessel meets the criteria for an abnormal vessel based on the interference patterns include: The collected environmental parameters are subjected to interference pattern identification, wherein the interference patterns include sensing interference and mobile interference; When the interference mode is perceived interference, the confidence score is reduced by a preset ratio; Based on the adjusted confidence score threshold, the confidence score of the vessel is re-analyzed. When the confidence score is greater than the adjusted confidence score threshold, the vessel is judged to be believable. When the confidence score is less than or equal to the adjusted confidence score threshold, the vessel is determined to be an abnormal vessel.

2. The multimodal verification method for ship identity based on computer vision and deep learning according to claim 1, characterized in that, The step of performing dynamic spatial calibration on the lidar point cloud data includes: The six-degree-of-freedom motion parameters within the scanning cycle are acquired in real time by the inertial measurement unit on board the ship. Based on motion parameters, a motion compensation model for lidar point clouds is established. The initial point cloud data is input into the motion compensation model, the initial point cloud data is resampled by the motion compensation model, and the resampled result is output to obtain the motion-compensated point cloud data. The motion-compensated point cloud data is converted to the ship's main coordinate system to complete dynamic spatial calibration.

3. The multimodal verification method for ship identity based on computer vision and deep learning according to claim 1, characterized in that, The step of analyzing the interference categories on behavioral characteristics based on the environmental parameters and re-verifying whether the vessel meets the criteria for an abnormal vessel based on the interference categories further includes: When the interference mode is mobile interference, the actual trajectory parameters are corrected by a preset trajectory compensation model; Based on the corrected actual trajectory parameters, the ship's confidence score is regenerated; When the confidence score of the regenerated ship is greater than the preset confidence score threshold, the ship is judged to be acting credibly. When the confidence score of a regenerated vessel is less than or equal to a preset confidence score threshold, the vessel is determined to be an abnormal vessel.

4. The multimodal verification method for ship identity based on computer vision and deep learning according to claim 1, characterized in that, Before the step of fusing the visual features, three-dimensional structural features, and ship identification information, the method further includes: When the perceptual interference is present, the fusion weight of the visual features is reduced, and the fusion weight of the three-dimensional structural features is increased. Based on the adjustment of fusion weights, the visual features, three-dimensional structural features, and ship identity information are fused.

5. A multimodal verification system for ship identity based on computer vision and deep learning, used in the method described in any one of claims 1-4, characterized in that, include: The acquisition module is used to acquire multimodal data of the ship, wherein the multimodal data includes visible light image data, lidar point cloud data and ship identification information; The first extraction module is used to extract the visual features of the ship based on the visible light image data; The calibration module is used to perform dynamic spatial calibration on the lidar point cloud data; The second extraction module is used to extract the three-dimensional structural features of the ship based on the calibration results; The fusion module is used to fuse the visual features, three-dimensional structural features and ship identity information, and input them into a preset verification model. The verification model outputs the verification results, which include normal ships and abnormal ships.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images