Ship identity multi-modal verification method and system based on computer vision and deep learning

Abstract

The invention relates to the field of navigation ship identification, in particular to a ship identity multi-modal verification method and system based on computer vision and deep learning, and the method comprises the steps: obtaining multi-modal data of a ship, the multi-modal data comprising visible light image data, laser radar point cloud data and ship identity information; extracting visual features of the ship according to the visible light image data; performing dynamic space calibration on the laser radar point cloud data; according to a calibration result, extracting three-dimensional structural features of the ship; the visual features, the three-dimensional structure features and the ship identity information are fused and input into a preset verification model, verification results are output through the verification model, and the verification results comprise normal ships and abnormal ships. Different types of data provide multi-dimensional understanding of the ship, and the precision and robustness of ship identification are enhanced.

Description

Technical Field

[0001] This application relates to the technical field of marine vessel identification, and particularly to a multi-modal verification method and system for vessel identity based on computer vision and deep learning. Background Art

[0002] Vessel identity identification is a process of effectively identifying and verifying vessels through different technical means, aiming to ensure the legality, safety, and reliability of vessels. It can be used for port management, maritime traffic monitoring, security prevention, marine environmental protection, etc.

[0003] Existing technologies rely on a single type of data for vessel identification in many cases. For example, many methods rely on lidar for three-dimensional spatial positioning and obstacle detection of vessels, or rely on visible light images to identify the appearance and shape of vessels. Due to environmental interference (such as weather factors, sea surface reflection, the influence of other vessels, etc.), it poses a great challenge to vessel identification. Using a single type of data for vessel analysis and identification, but when there is interference in the environment, it is difficult to clearly identify the true behavior of the target vessel from the complex environment. For example, misjudgment may lead to identifying a normally sailing vessel as having abnormal behavior, or misjudging a vessel with abnormal behavior as normal. Secondly, when existing technologies use multi-modal data for vessel identity verification, they encounter the problem of data conflict. Since different sensors may have differences in accuracy and stability. For example, lidar data may be more accurate than camera or radar data, but if the data from these sensors conflicts in the same scenario, existing technologies may not be able to effectively determine which data source is more credible. Usually, these conflicting data are processed by a simple voting mechanism or discarding the conflicting data, but this does not completely solve the problem, especially when there are different data qualities, sensor performances, or external factors behind the data conflict, it is easy to lead to wrong decisions or lack of sufficient interpretability.

[0004] Therefore, there are defects in the existing technologies and improvements are needed. Summary of the Invention

[0005] In order to solve one or several problems in the existing technologies, the main purpose of this application is to provide a multi-modal verification method and system for vessel identity based on computer vision and deep learning.

[0006] To achieve the above-mentioned invention purpose, this application proposes a multi-modal verification method for vessel identity based on computer vision and deep learning, and the method includes:

[0007] Obtain multi-modal data of a vessel, where the multi-modal data includes visible light image data, lidar point cloud data, and vessel identity information;

[0008] Extract the visual features of the ship based on the visible light image data;

[0009] Perform dynamic spatial calibration on the lidar point cloud data;

[0010] Extract the three-dimensional structural features of the ship according to the calibration result;

[0011] Fuse the visual features, three-dimensional structural features, and ship identity information, and input them into a preset verification model, and output the verification result through the verification model, where the verification result includes normal ships and abnormal ships.

[0012] An embodiment of the present application further provides a multimodal verification system for ship identity based on computer vision and deep learning, including:

[0013] An acquisition module for acquiring multimodal data of the ship, where the multimodal data includes visible light image data, lidar point cloud data, and ship identity information;

[0014] A first extraction module for extracting the visual features of the ship according to the visible light image data;

[0015] A calibration module for performing dynamic spatial calibration on the lidar point cloud data;

[0016] A second extraction module for extracting the three-dimensional structural features of the ship according to the calibration result;

[0017] A fusion module for fusing the visual features, three-dimensional structural features, and ship identity information, and inputting them into a preset verification model, and outputting the verification result through the verification model, where the verification result includes normal ships and abnormal ships.

[0018] The present application further provides a computer device, including a memory and a processor, where the memory stores a computer program, and the processor implements the steps of the method described in any one of the above when executing the computer program.

[0019] The present application further provides a computer-readable storage medium, on which a computer program is stored, and the computer program implements the steps of the method described in any one of the above when executed by a processor.

[0020] The multi-modal verification method and system for ship identity based on computer vision and deep learning according to the embodiments of the present application can comprehensively and accurately capture various information of ships by obtaining multi-modal data of ships, including visible light image data, lidar point cloud data, and ship identity information. Different types of data provide a multi-dimensional understanding of ships, enhancing the accuracy and robustness of ship identification. By extracting the visual features of ships from the visible light image data, the appearance features of ships can be accurately identified, and detailed behavior and status analysis can be provided using the rich visual information in the image data. The extraction of visual features provides a reliable basis for ship behavior recognition. By performing dynamic spatial calibration on the lidar point cloud data, the error of the point cloud data caused by factors such as environmental changes, ship movements, or sensor position changes can be effectively solved, thereby improving the accuracy of ship three-dimensional structure feature extraction. The dynamic calibration optimizes the three-dimensional feature extraction process, making the recognition of the ship's spatial position and shape more accurate. Description of the Drawings

[0021] Figure 1 is a schematic flowchart of a multi-modal verification method for ship identity based on computer vision and deep learning according to an embodiment of the present application;

[0022] Figure 2 is a schematic flowchart of a multi-modal verification method for ship identity based on computer vision and deep learning according to an embodiment of the present application;

[0023] Figure 3 is a schematic block diagram of the structure of a multi-modal verification system for ship identity based on computer vision and deep learning according to an embodiment of the present application;

[0024] Figure 4 is a schematic block diagram of the structure of a computer device according to an embodiment of the present application.

[0025] The realization, functional features, and advantages of the purpose of the present application will be further described with reference to the embodiments and the accompanying drawings. Detailed Embodiments

[0026] In order to make the purpose, technical solutions, and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0027] Refer to Figure 1 , in the embodiments of the present application, a multi-modal verification method for ship identity based on computer vision and deep learning is provided, and the method includes:

[0028] S1. Obtain multi-modal data of the ship, where the multi-modal data includes visible light image data, lidar point cloud data, and ship identity information;

[0029] S2. Extract the visual features of the ship according to the visible light image data;

[0030] S3. Perform dynamic spatial calibration on the lidar point cloud data;

[0031] S4. Extract the three-dimensional structure features of the ship according to the calibration result;

[0032] S5. Integrate the visual features, three-dimensional structure features and ship identity information, and input them into a preset verification model. Output the verification result through the verification model, where the verification result includes normal ships and abnormal ships.

[0033] As described in steps S1 - S2 above, use a visible light camera to capture the appearance features of the ship, such as shape, color, size, etc., for identifying the visual information of the ship. Obtain the three-dimensional spatial information of the ship through lidar (LiDAR). The point cloud data can provide the fine structure of the ship's surface, overcoming the deficiencies of traditional image data in harsh environments. The ship identity information can include AIS (Automatic Identification System) data or other pre-registered identity data, which can provide important information about the ship's navigation history, ship type, etc. Through the integration of multiple data sources, the system can identify the ship from different perspectives, overcoming the deficiencies of a single data source. For example, lidar can still effectively provide three-dimensional spatial data in poor lighting conditions, while visible light images can supplement the detailed information of the ship under good weather conditions during the day. The ship identity information provides support for verifying the true identity of the ship. The visible light image data is two-dimensional image data obtained through a standard imaging device. These image data contain visual features of the ship, such as color, shape, size, markings, etc. In the context of deep learning, using convolutional neural networks (CNNs) or other computer vision methods, these features can be automatically extracted from the images. These features help to identify the type, size, markings and status of the ship, etc. By extracting the visual features of the visible light images, the system can effectively identify the appearance and shape of the ship under good lighting conditions. This method makes up for the detailed information that lidar cannot provide, especially having advantages in identifying appearance features and the shape of the ship.

[0034] As described in the above steps S3 - S5, the lidar point cloud data provides three - dimensional spatial information of the ship. However, due to the movement, tilt or other dynamic factors of the lidar system, there may be spatial errors in the point cloud data. Therefore, through dynamic spatial calibration, the point cloud data can be corrected to ensure its accuracy in space. Dynamic calibration usually aligns multiple point cloud data or adjusts based on other known reference objects (such as GPS positioning, inertial measurement unit IMU, etc.) to eliminate the errors caused by ship movement. Through spatial calibration, the lidar data can more accurately reflect the actual spatial structure of the ship. This step ensures the reliability of the point cloud data, thus providing accurate spatial information for subsequent three - dimensional feature extraction and ship identification of the ship, and avoiding the influence of errors introduced by dynamic factors on the identification results. After spatial calibration, the lidar point cloud data can more accurately reflect the three - dimensional shape of the ship. By further analyzing the calibrated point cloud data, three - dimensional structural features of the ship can be extracted, such as hull contour, size, position, relative angle, etc. These three - dimensional features can provide global information about the ship's shape and are the key to distinguishing different ship types and shapes. Extracting the three - dimensional structural features of the ship helps the system to clearly identify the geometric features of the ship in complex environments (such as haze, night, etc.). Especially in cases where two - dimensional images are difficult to identify, the application of three - dimensional features can improve the accuracy and robustness of identification. Fusing different types of data (visual features, three - dimensional structural features, identity information) is the key to improving the accuracy of the ship identification system. Deep learning methods (such as multi - modal learning) allow the fusion of different modal data to obtain more information and improve the identification accuracy. The preset verification model (such as a deep neural network) can analyze and combine these features based on the training set data to detect abnormal ships. Visual features come from the shape and color information of visible light images. Three - dimensional structural features come from the three - dimensional geometry of the ship provided by lidar data. Ship identity information provides background information about the ship (such as type, navigation history, etc.) and helps to verify whether the ship belongs to the normal category. Through the fusion of multi - modal data, the system can more comprehensively analyze the identity and behavior of the ship, thus reducing the risks of mis - identification and missed - identification. Through the results output by the verification model, the system can accurately distinguish normal ships from abnormal ships and provide reliable ship identity verification and abnormal detection capabilities.

[0035] Referring to Figure 2 , in one embodiment, the steps of performing dynamic spatial calibration on the lidar point cloud data include:

[0036] S31. Real - time obtain six - degree - of - freedom motion parameters within the scanning period through the inertial measurement unit carried by the ship;

[0037] S32. Based on the motion parameters, establish a motion compensation model for the lidar point cloud;

[0038] S33. Input the initial point cloud data into the motion compensation model, resample the initial point cloud data through the motion compensation model, and output the resampled result to obtain the motion-compensated point cloud data;

[0039] S34. Convert the motion-compensated point cloud data to the ship body coordinate system to complete dynamic spatial calibration.

[0040] As described in the above steps, an Inertial Measurement Unit (IMU) is a common sensor that can measure the acceleration and angular velocity of an object. For a ship, the IMU can provide the six-degree-of-freedom motion parameters of the ship in real time through accelerometers and gyro sensors in three axes, namely: the accelerations in three translational directions (along the X, Y, and Z axes) and the angular velocities in three rotational directions (around the X, Y, and Z axes). These six-degree-of-freedom motion parameters can accurately reflect the motion state of the ship during the scanning period, including dynamic changes such as the position, speed, and tilt of the ship. By obtaining the six-degree-of-freedom motion parameters of the ship in real time, it is possible to provide the necessary motion information for the subsequent calibration of lidar point cloud data. The motion parameters provided by the IMU can effectively compensate for the motion deviation of the ship in a dynamic environment and ensure the accuracy of the point cloud data. Without these motion data, the point cloud data may have spatial errors due to factors such as the shaking and tilting of the ship. The motion compensation model is a mathematical model established based on the six-degree-of-freedom motion parameters provided by the IMU, aiming to eliminate the point cloud data errors caused by the ship's motion during the scanning process. Specifically, the model will predict and adjust the spatial position of each lidar point cloud according to the speed, acceleration, and angular velocity of the ship, so that the point cloud data can accurately reflect the actual position and attitude of the ship. This compensation model is usually based on kinematic theory and can correct the point cloud data through coordinate transformation and other methods. After establishing the motion compensation model, the errors caused by the ship's motion to the lidar point cloud data can be effectively eliminated, and the spatial accuracy of the point cloud data can be improved. In this way, even when the ship is in motion, the point cloud data collected by the lidar can conform to the actual spatial position, avoiding data distortion caused by motion errors. In this step, the initial point cloud data obtained by the lidar is input into the motion compensation model. The motion compensation model resamples the point cloud data according to the motion parameters of the ship, that is, by performing motion compensation on the position of each point to make it conform to the true spatial position of the ship. The resampling process may include transformation operations such as translation and rotation of the spatial coordinates of the point cloud to compensate for the motion deviation generated during the ship's scanning process. Through this process, the compensated point cloud data more accurately reflects the actual three-dimensional spatial distribution of the ship. The motion-compensated point cloud data obtained through resampling can effectively correct the errors caused by the ship's motion, making the point cloud data more accurate in terms of spatial position. The resampling process ensures the high precision of the data, making subsequent processing and analysis (such as ship identification and 3D modeling) more reliable. This process transforms the motion-compensated point cloud data from the lidar coordinate system to the ship's body coordinate system. The lidar is usually installed at a certain position on the ship, and the obtained point cloud data is relative to the lidar coordinate system, while the ship's body coordinate system is defined with the ship as the reference frame. In this step, through coordinate transformation algorithms (such as rotation matrices, affine transformations, etc.), the compensated point cloud data is transformed into the ship's body coordinate system to ensure that the point cloud data is consistent with the actual motion and attitude of the ship.After converting the point cloud data to the ship's body coordinate system, it can more accurately reflect the position and attitude of the ship in the actual environment. This calibration makes the point cloud data consistent with the ship's own reference frame, providing more accurate data support for subsequent tasks such as ship identification, path planning, and spatial analysis.

[0041] In one embodiment, before the step of fusing the visual features, three-dimensional structure features, and ship identity information, the method further includes:

[0042] Obtain the visual features and three-dimensional structure features, input the visual features and three-dimensional structure features into a preset spatial model, and generate the current behavior features of the ship through the spatial model;

[0043] Obtain the historical behavior features of the ship, combine the historical behavior data and the current behavior data, and predict the ship trajectory parameters of the ship in a preset time period;

[0044] Based on the predicted ship trajectory parameters, obtain the actual trajectory parameters of the ship at the predicted time in real time, and analyze the deviation coefficient between the actual trajectory parameters and the predicted trajectory parameters;

[0045] Quantify the deviation coefficient to generate a confidence score for the ship;

[0046] Analyze the confidence score of the ship. When the confidence score is greater than a preset confidence score threshold, it is determined that the ship has a credible behavior;

[0047] Based on the ship having a credible behavior, fuse the visual features, three-dimensional structure features, and ship identity information.

[0048] As described above, visual features and three-dimensional structural features are obtained from the external environment of the ship and are used to describe information such as the appearance, position, shape, and movement of the ship. Visual features can be obtained through cameras, visual sensors, or image processing techniques, while three-dimensional structural features are obtained through lidar (LiDAR) or other three-dimensional modeling techniques. The preset spatial model is a mathematical or computational model used to comprehensively analyze these features and generate current behavior features. The spatial model may be based on principles of physics, machine learning, or statistics and can effectively combine this multi-modal information to extract the behavior patterns of the ship, such as the ship's driving direction, speed, heading, etc. By inputting visual features and three-dimensional structural features into the spatial model, a feature set that comprehensively reflects the ship's behavior can be generated, and these features help to describe dynamic information such as the ship's motion state, heading, and speed. Historical behavior features reflect the behavior patterns of the ship over a certain period in the past, and these features help to understand the ship's behavior trends and habits. By combining historical data and current behavior data, through prediction models (such as regression analysis, time series analysis, machine learning, etc.), the trajectory of the ship over a period of time in the future can be predicted. The preset time is the target time point for model prediction, and the trajectory parameters include position, speed, heading, etc., which can reflect the future movement trajectory of the ship. By combining historical behavior data with current behavior data for prediction, the future movement trajectory of the ship can be accurately predicted, providing a basis for subsequent deviation analysis and behavior determination. By continuously monitoring the actual trajectory of the ship and comparing it with the predicted trajectory. The actual trajectory parameters (such as position, speed, etc.) obtained in real time are usually provided by positioning systems (such as GPS, inertial navigation systems, etc.). Analyzing the deviation coefficient (i.e., error or difference) between the actual trajectory and the predicted trajectory is to evaluate the degree of abnormality of the ship's behavior. If the deviation coefficient is large, it indicates that there is a significant difference between the actual behavior and the predicted behavior of the ship, which may mean that the ship's trajectory has deviated, or the ship's behavior has deviated from the expected route. By calculating the deviation coefficient, the stability and reliability of the ship's behavior can be effectively evaluated. If the deviation coefficient is small, it indicates that the ship's behavior is consistent with the expectation; while a large deviation coefficient can indicate the possible existence of abnormal behavior. This provides a quantitative basis for subsequent confidence scoring and behavior judgment, helping to distinguish normal behavior from abnormal behavior. The quantification process of the deviation coefficient usually includes standardization and weighting according to the size of the difference, and finally a score representing the confidence level is obtained. The confidence score is usually calculated through statistical or machine learning models and can measure the credibility of the ship's behavior. If the deviation coefficient between the actual behavior of the ship and the predicted trajectory is small, it indicates that the ship's behavior is relatively reliable and the confidence score is high; conversely, if the deviation coefficient is large, the confidence score is low. The confidence score can provide a basis for subsequent decision-making and help to determine whether the ship's behavior is credible. A high confidence score indicates that the ship's behavior is relatively stable and normal, while a low confidence score may mean that there are abnormal behaviors in the ship.The quantization of the confidence score provides an actionable criterion, which helps to make an automated judgment of the ship's behavior. By comparing the confidence score of the ship with a preset threshold, it is decided whether to judge the ship's behavior as credible. The preset confidence score threshold is set according to experience, historical data or system requirements, and is used to distinguish normal behavior from abnormal behavior. If the confidence score of the ship is greater than this threshold, it means that the ship's behavior is consistent with the expectation, and it is judged as a credible behavior; if the confidence score is low, the ship's behavior is considered not credible and may require further intervention or monitoring. When the ship is judged as a credible behavior, the system will fuse the visual features, three-dimensional structure features and identity information of the ship. Fusion means integrating these information together to form a comprehensive set of ship recognition features. This step can use machine learning algorithms, weighted average, Bayesian inference and other methods to fuse these features to obtain a more comprehensive and accurate ship feature, so as to realize the multi-modal recognition method of ship identity. By fusing visual features, three-dimensional structure features and ship identity information, a more comprehensive and accurate description of the ship's behavior can be obtained. This enables the system to understand the ship's state more comprehensively, optimize the ship's recognition and tracking process, and improve the accuracy and reliability of ship behavior monitoring.

[0049] In one embodiment, after the step of quantifying the deviation coefficient to generate the confidence score of the ship, the method further includes:

[0050] When the confidence score is less than or equal to the preset confidence score threshold, automatically mark the current ship as a suspicious target;

[0051] Based on the judgment result, collect the environmental parameters of the sea area where the ship is located in real time;

[0052] Analyze whether the interference conditions are met according to the environmental parameters;

[0053] When the environmental parameters meet the interference conditions, analyze the interference mode on the behavior characteristics according to the environmental parameters, and re-verify whether the ship meets the abnormal ship according to the interference mode.

[0054] As described above, in ship behavior analysis, the confidence score is a numerical value used to quantify the degree of abnormality of ship behavior. It reflects the level of trust of the analysis system in whether the ship is abnormal. This threshold is a value set in advance by the system designer. When the confidence score of a ship is lower than or equal to this threshold, the system considers that the behavior of the ship has a high degree of uncertainty or abnormality. Automatic marking based on the confidence score is to quickly identify potential abnormal ships, avoiding manual intervention and over-reliance on manual analysis. This can quickly screen out targets that may have problems for further subsequent analysis. It improves the efficiency of identifying abnormal ship behaviors, reducing missed reports and false alarms. In practical applications, it can quickly judge suspicious targets, reducing the workload of operators and enabling timely further actions. Environmental factors, such as weather conditions, sea currents, visibility, wind speed, and sea waves, may interfere with the behavior of ships. Real-time collection of these parameters can provide more background information for subsequent judgment of whether ship behavior is abnormal. In the judgment of ship behavior, the external environment is an important interfering factor, affecting the ship's movement trajectory, speed, etc., and must be taken into account to ensure the accuracy of the behavior analysis results. By obtaining environmental parameters in real time, the ship behavior can be understood more accurately, excluding behavior abnormalities caused by external factors, thus avoiding misjudgment. It can cope with variable environmental factors. In different sea areas, the performance of ship behavior may vary. Real-time data collection can flexibly adjust the analysis strategy. This step of environmental interference conditions is to judge whether the ship is affected by environmental factors, thus affecting its behavior characteristics. If environmental conditions (such as strong winds, large waves, etc.) can cause changes in the ship's behavior, it is considered that these conditions meet the interference conditions. Environmental factors may cause phenomena such as speed changes and course deviations of ships, which do not necessarily mean that the ship is abnormal, but may be normal phenomena caused by environmental impacts. Therefore, analyzing whether the interference conditions are met can effectively distinguish true abnormal behaviors from natural fluctuations caused by environmental factors. If only judging based on the ship's behavior characteristics, ships affected by environmental interference may be misjudged as abnormal. By analyzing environmental parameters, it can be further verified whether the ship is truly abnormal. Only when the environmental conditions meet the interference requirements is interference analysis carried out, reducing the interference of irrelevant factors and making the determination of ship abnormal behavior more targeted. When it is detected that the environmental conditions meet the interference conditions, the system needs to analyze the specific influence mode of this environmental factor on the ship behavior. For example, strong winds may cause the ship's course to deviate, and sea waves may cause the ship's speed to slow down, etc. By analyzing these interference modes, the system can simulate and predict the changes in ship behavior under different environmental conditions. Through the analysis of historical data, the behavior characteristic deviations under different environmental conditions can be obtained, and corresponding interference modes can be established. These modes can be used as criteria for judging whether the ship is interfered. By understanding the interference modes, the system can accurately distinguish the changes in ship behavior caused by the external environment from true abnormal behaviors, thus avoiding misjudgment.Establish multiple interference patterns for different sea areas and different environmental conditions, enabling the system to work properly under various complex environments and having better adaptability. After analyzing the interference patterns, it is necessary to re-judge whether the ship is abnormal. By correcting the behavior judgment of the ship according to the interference pattern and excluding the abnormalities caused by environmental factors, a more accurate judgment can be made. If the deviation of the ship's behavior is caused by the interference pattern, then the ship should not be judged as an abnormal ship; if the interference pattern does not meet the expectations, it indicates that the ship has abnormalities. Through re-verification, the influence of environmental interference can be effectively reduced, ensuring that the determination of ship abnormalities is more accurate.

[0055] In one embodiment, before the step of inputting the reflectivity parameter and the deformation offset parameter into the deviation value analysis model, the method further includes:

[0056] Collect the light intensity of the environment in real time;

[0057] When the light intensity is greater than or equal to a preset light intensity threshold, enable the polarized light filter;

[0058] Calculate the calibrated reflectivity parameter according to the light intensity and the state of the polarized light filter, and input the calibrated reflectivity parameter into the deviation value analysis model.

[0059] As described above, the ambient light intensity refers to the light intensity irradiating on the target object. In practical applications, the light intensity has an important impact on the reflection characteristics of the object surface, especially in optical measurement and image recognition. The change in light intensity may cause fluctuations in the reflectivity parameters, thus affecting the accuracy of subsequent analysis. Therefore, real-time acquisition of light intensity data helps to dynamically adjust the input parameters in the deviation analysis model. The polarizing filter can control the polarization state of the light reflected from the object surface, thus effectively filtering out stray light and unnecessary reflected light caused by changes in the light angle or irregularity of the reflecting surface. Enabling the polarizing filter can eliminate these interferences, especially in strong light environments, and can effectively improve the accuracy of reflectivity measurement. When the light intensity reaches a certain threshold, the scattering of the reflected light may increase, thus affecting the measurement accuracy. Therefore, the polarizing filter is enabled to improve the measurement accuracy. The calibrated reflectivity parameter is the reflectivity data obtained after adjusting for light intensity and polarizing filter. Since the properties of the reflected light are different under different light intensities, it is necessary to adjust the reflectivity parameter according to the light intensity and the state of the filter. By dynamically calculating the calibrated reflectivity, the error caused by environmental changes can be eliminated to ensure the accuracy of the data. Specifically, when the light intensity is high, the polarizing filter can effectively reduce the strong light interference and help the system obtain a more accurate reflectivity value. The calibrated reflectivity parameter will be passed as input data into the deviation value analysis model for further analysis of the deviation of the target object. By introducing the calibrated reflectivity data, the model can calculate the deviation value according to the real reflection characteristics of the object, so as to judge the size of the deviation. The role of the deviation analysis model here is to help the system evaluate the deformation, damage or irregularity of the target object according to the change of the reflectivity parameter. By inputting the calibrated reflectivity parameter, the deviation value analysis model can provide a more accurate deviation detection result, especially under different light conditions. The calibrated reflectivity data will effectively improve the accuracy of the deviation value calculation and ensure the reliability of the final result. This can reduce the deviation detection error caused by light changes and improve the robustness and accuracy of the system.

[0060] In one embodiment, the steps of analyzing the interference pattern of the behavior characteristics according to the environmental parameters and re-verifying whether the ship meets the abnormal ship according to the interference pattern include:

[0061] Identify the interference pattern of the collected environmental parameters, where the interference pattern includes perception interference and movement interference;

[0062] When the interference pattern is perception interference, lower the confidence score by a preset ratio;

[0063] Based on the adjusted confidence score threshold, re-analyze the confidence score of the ship. When the confidence score is greater than the adjusted confidence score threshold, it is determined that the ship has a credible behavior;

[0064] When the confidence score is less than or equal to the adjusted confidence score threshold, the ship is determined to be an abnormal ship.

[0065] As described above, by analyzing environmental parameters, the system identifies interference patterns in the sea area where the ship is located that may affect the ship's behavior. The interference patterns can be divided into perception interference and movement interference. Perception interference refers to environmental factors (such as low visibility, bad weather, etc.) that affect the data collection of sensors, resulting in a deviation between the actual behavior of the ship and the behavior monitored by the system. Movement interference refers to external factors such as sea currents, wind speeds, and wave heights that cause changes in the physical movement of the ship, such as affecting the course and speed. Identifying interference patterns can accurately understand the impact of environmental factors on the ship's behavior. Perception interference and movement interference are analyzed from the aspects of data collection and ship dynamics respectively, providing a precise basis for subsequent processing. Separately identifying these two interference patterns helps to accurately judge the nature of the interference, so as to take targeted measures. By identifying different types of interference, the system can better distinguish behavior changes caused by external environmental factors from actual abnormal behaviors. Separately processing perception interference and movement interference enables the system to flexibly respond to different types of interference with stronger pertinence. Perception interference will cause a deviation between the data obtained by the sensor and the actual situation, thus affecting the confidence score of the ship's behavior. If perception interference is detected, the system will reduce the confidence score to lower the trust in the abnormal behavior of the ship, avoiding misjudging the abnormality caused by perception interference as a real abnormal behavior. The reduction ratio is based on a preset value set by the system to ensure that the impact of perception interference on the confidence score can be effectively reduced. The setting of the ratio should be able to correct the error of the sensor without overly affecting normal judgment. Perception interference may lead to incorrect data, directly affecting the confidence score of the ship, so the score needs to be reduced according to the interference intensity. Controlling the score range can avoid mislabeling the ship's behavior as abnormal and ensure that the adjustment of the confidence score will not lead to overcorrection, resulting in missed reports or false alarms. By reducing the confidence score in the case of perception interference, misjudging the ship as abnormal due to sensor failure or environmental factors can be avoided. By correcting the deviation caused by perception interference, the system's judgment of the ship's behavior becomes more stable and reliable. When perception interference is identified, the system will adjust the confidence score according to the preset ratio and then re-evaluate the ship's confidence score using the new threshold. The confidence score threshold is the standard for judging whether the ship is normal, and the adjusted threshold can more accurately reflect the actual situation after being affected by interference. Dynamically adjusting the confidence score threshold enables the system to evaluate the ship more flexibly and adapt to different environmental factors. After targeted processing of the interference, using the new threshold for judgment again can ensure the accuracy of the judgment result. By re-analyzing according to the adjusted confidence score threshold, the impact of interference on the judgment of the ship's behavior can be reduced, making the judgment result more real and reliable. The system can automatically adjust the confidence score threshold according to environmental changes, improving its performance in complex environments. When the ship's confidence score exceeds the adjusted threshold, it indicates that after interference correction, the ship's behavior is credible and does not show abnormal behavior.A confidence score greater than the adjusted threshold means that the ship's behavior conforms to the normal range preset by the system. Therefore, it can be determined that the ship's behavior is credible. By setting this judgment step, it can be ensured that the ship's behavior is considered credible only when the corrected confidence score exceeds the reasonable range. This step avoids misjudgment caused by incorrect interference patterns through strict threshold control. By strictly controlling the confidence score threshold, misjudgment caused by interference patterns is reduced, ensuring that credible behaviors are accurately identified. Increasing the strictness of the judgment step makes it possible to determine that the ship's behavior is credible only under reliable circumstances, thereby improving the accuracy. If the confidence score of the ship is less than or equal to the adjusted threshold, it indicates that the ship's behavior is outside the normal range and may be abnormal behavior. At this time, the system will determine the ship as an abnormal ship. If the score does not reach the adjusted threshold, it can be considered that the ship behaves abnormally under the influence of interference, and the system will mark it as an abnormal ship. Judging abnormalities based on the confidence score ensures that abnormal judgments are made only when the ship's behavior deviates from the normal range. By adjusting the confidence score threshold to adapt to different interference environments, the system can flexibly respond to different types of interference, avoiding misjudgment or missed judgment. It can accurately judge whether the ship is abnormal and avoid the influence of interference factors on the judgment. By dynamically adjusting the confidence score threshold, the recognition accuracy of abnormal behaviors is improved, and the probability of false alarms and missed alarms is reduced.

[0066] In one embodiment, the step of analyzing the interference category of the behavior characteristics according to the environmental parameters and re-verifying whether the ship meets the abnormal ship according to the interference category further includes:

[0067] When the interference pattern is mobile interference, correct the actual trajectory parameters through a preset trajectory compensation model;

[0068] Based on the corrected actual trajectory parameters, regenerate the confidence score of the ship;

[0069] When the regenerated confidence score of the ship is greater than the preset confidence score threshold, determine that the ship's behavior is credible;

[0070] When the regenerated confidence score of the ship is less than or equal to the preset confidence score threshold, determine that the ship is an abnormal ship.

[0071] As described above, the environmental parameters include factors such as the weather, ocean conditions, and maritime traffic situation where the ship is located. These factors can affect the behavior of the ship and may cause changes in characteristics such as the ship's trajectory, speed, and heading. By analyzing these environmental parameters, the type of interference suffered by the ship can be determined, such as "moving interference" or "static interference". This analysis process helps to understand whether the ship's behavior is affected by external factors rather than the ship's own faults or human errors. By identifying the type of interference, the system can clarify the external reasons for the changes in the ship's behavior. Based on the analysis of the type of interference, the system will re-evaluate whether the ship's behavior is abnormal. When considering the interference factors, the system no longer simply relies on the initial data of the ship's behavior but will comprehensively consider the influence of the external environment. For example, in the case of moving interference, the ship may deviate from its normal course due to factors such as waves and wind speed, and this behavior itself may not indicate that there is a fault or abnormality with the ship. Re-verification ensures that the system can make appropriate adjustments for interference when evaluating whether the ship is abnormal. It avoids misjudging the behavior deviation caused by external factors as abnormal in a complex environment. This step improves the accuracy of behavior verification and makes the abnormal determination more in line with the actual situation. When it is detected that the ship is subject to "moving interference", such as factors like wind force and ocean current causing changes in the ship's trajectory, the preset trajectory compensation model will come into play. This model can calculate the specific impact of the interference on the ship's trajectory by identifying the characteristics and effects of the interference and correct the actual trajectory parameters. For example, if the wind force causes the ship to deviate from its course, the model will correct the ship's trajectory data according to parameters such as wind force intensity and direction to make it return to a state closer to the normal situation. The corrected trajectory parameters more realistically reflect the actual behavior of the ship under environmental interference. Through this correction, the system can effectively reduce the impact of environmental interference on the ship's trajectory assessment, thereby improving the accuracy of the ship's behavior. After correcting the actual trajectory parameters, the new trajectory data will be used to recalculate the confidence score of the ship. The confidence score is an important indicator for measuring whether the ship's behavior is normal, and the factors usually considered include the ship's heading, speed, deviation of the trajectory from the expected trajectory, etc. By inputting the corrected trajectory data into the confidence score algorithm, the system can evaluate whether the ship is behaving normally. The corrected trajectory data makes the confidence score more accurately reflect the true behavior of the ship in the current environment, thus ensuring that the assessment of the ship's behavior is closer to the actual situation. When the newly generated confidence score exceeds the preset confidence score threshold, it indicates that the ship's behavior is within the trajectory range after interference correction and meets the expectations of normal behavior. This threshold is usually set based on the historical data of the ship's behavior and safety standards. When the confidence score is higher than this threshold, it means that the ship's behavior is within the acceptable range. The system can judge whether the ship's behavior meets the predetermined standards. If the confidence score is high, indicating that the ship's behavior is consistent with normal behavior, then the ship can be considered credible. This helps to reduce false alarms and ensure the safety and efficiency of the ship.When the confidence score of the re-generated one is lower than or equal to the preset confidence score threshold, it indicates that the behavior of the ship deviates significantly from the normal expectation and there may be an anomaly. This anomaly may be caused by ship failures, abnormal operations, or other unforeseeable factors. By comparing with the preset threshold, the system can accurately determine whether there is an anomaly.

[0072] In one embodiment, before the step of fusing the visual features, three-dimensional structure features, and ship identity information, the method further includes:

[0073] When there is such a perception interference, reduce the fusion weight of the visual features and increase the fusion weight of the three-dimensional structure features;

[0074] Based on the adjustment of the fusion weights, fuse the visual features, three-dimensional structure features, and ship identity information.

[0075] As described above, perception interference refers to external factors that affect the system's perception process, such as noise, occlusion, and bad weather. These interferences will lead to a decrease in the accuracy of visual features. When there is perception interference, the reliability of visual features (such as image data, color, shape, etc.) will be affected. Therefore, in order to reduce misjudgment, the system will reduce the weight of visual features in the fusion process. At the same time, three-dimensional structure features (such as the spatial layout, size, and shape of the ship) are usually not easily affected by visual interference. Therefore, in the case of perception interference, their weights should be increased. Through this adjustment, the system can rely more on three-dimensional structure features to judge ship behavior and ensure a more stable and reliable fusion result. By reducing the fusion weight of visual features and increasing the fusion weight of three-dimensional structure features, the system can maintain a high recognition accuracy in an interference environment. This weight adjustment can avoid being misled by perception interference, improve the system's performance in a complex environment, and ensure that the ship behavior analysis is still accurate, especially in the case of strong interference. When performing fusion, the visual features and three-dimensional structure features need to be adjusted according to the current environment (whether there is perception interference), so as to determine the contribution of each feature to the final fusion result. After adjustment, the system will analyze the behavior of the ship according to these weights and generate a comprehensive recognition result. This setting can flexibly adapt to environmental changes and ensure that in the case of large perception interference, the system can still fully rely on relatively stable three-dimensional structure features and ship identity information to enhance the overall judgment accuracy. By adjusting the weights of feature fusion, the system can maintain high stability and accuracy during perception interference. Especially the increased weight of three-dimensional structure features can make up for the deficiencies of visual features in the case of interference. Finally, by integrating three-dimensional structure, visual features, and ship identity information, the system can more comprehensively evaluate ship behavior and ensure the accuracy and reliability of ship recognition.

[0076] In another embodiment, when there are contradictions in data from lidar, visible light, AIS, etc. simultaneously, traditional methods use simple voting or discard data, resulting in insufficient decision-making basis. However, this method combines data from different types of sensors, automatically evaluates the credibility of data sources, and adopts a more intelligent processing mechanism when there are data conflicts, rather than simply relying on a simple voting mechanism or discarding conflicting data. In this implementation, considering the accuracy differences and their stabilities of different sensors (such as lidar, cameras, and radars), by dynamically evaluating the sensor performance, data sources with higher reliability are preferentially selected. This method can effectively reduce data conflicts caused by inconsistent sensor quality or environmental interference, reduce misjudgments, and improve the accuracy of identity verification. Specifically, by analyzing the quality of data from each sensor through a fusion algorithm, the system can adaptively adjust the weights of the data to ensure that when there are conflicts between multiple data sources, the most representative and valid information is selected. In addition, by adopting an error detection mechanism, low-quality data can be identified and excluded in real time, ensuring that the entire identity verification process has high interpretability and accuracy.

[0077] In a feasible embodiment, as an important logistics hub, activities such as ship transportation, loading and unloading, and warehousing in ports often have certain impacts on the environment, such as exhaust emissions, noise pollution, wastewater emissions, etc. Therefore, port management departments need to conduct real-time monitoring and environmental management to ensure that the environmental protection standards of ports meet the requirements of relevant regulations. When verifying the identity of incoming ships at the port, the identity of the ship can be confirmed and its emission data can be obtained by associating the AIS (Automatic Identification System) data of the ship with the emission monitoring data. If the ship has a record of exceeding the pollution emission standard in the past, the system will increase the warning level. Detection and early warning of excessive pollutant concentration: The system compares the real-time detected emission data with the port environmental protection standards. When the concentration of a certain pollutant exceeds the standard, an early warning notice is immediately triggered. This early warning information will include information such as the ship ID, the exceeded pollutant, and the pollution concentration, and will be sent to the terminal devices of port management personnel at the same time.

[0078] Referring to Figure 3 , an embodiment of the present application also provides a multi-modal verification system for ship identity based on computer vision and deep learning, including:

[0079] An acquisition module 1 for acquiring multi-modal data of a ship, where the multi-modal data includes visible light image data, lidar point cloud data, and ship identity information;

[0080] A first extraction module 2 for extracting visual features of the ship according to the visible light image data;

[0081] A calibration module 3 for dynamically calibrating the spatial coordinates of the lidar point cloud data;

[0082] The second extraction module 4 is used to extract the three-dimensional structural features of the ship according to the calibration result;

[0083] The fusion module 5 is used to fuse the visual features, three-dimensional structural features and ship identity information, and input them into a preset verification model, and output the verification result through the verification model, where the verification result includes normal ships and abnormal ships.

[0084] As described above, it can be understood that each component of the ship identity multi-modal verification system based on computer vision and deep learning proposed in this application can implement the functions of any one of the above-mentioned ship identity multi-modal verification methods based on computer vision and deep learning, and the specific structure will not be elaborated.

[0085] Refer to Figure 4 , in the embodiment of the present application, a computer device is further provided. The computer device may be a server, and its internal structure may be as Figure 4 shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer design is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store data such as monitoring data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a ship identity multi-modal verification method based on computer vision and deep learning.

[0086] The above-mentioned processor executes the above-mentioned ship identity multi-modal verification method based on computer vision and deep learning, including: obtaining multi-modal data of the ship, where the multi-modal data includes visible light image data, lidar point cloud data and ship identity information; extracting visual features of the ship according to the visible light image data; performing dynamic spatial calibration on the lidar point cloud data; extracting three-dimensional structural features of the ship according to the calibration result; fusing the visual features, three-dimensional structural features and ship identity information, and inputting them into a preset verification model, and outputting the verification result through the verification model, where the verification result includes normal ships and abnormal ships.

[0087] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, a multimodal verification method for ship identity based on computer vision and deep learning is implemented, including the steps of: obtaining multimodal data of a ship, where the multimodal data includes visible light image data, lidar point cloud data, and ship identity information; extracting visual features of the ship according to the visible light image data; performing dynamic spatial calibration on the lidar point cloud data; extracting three-dimensional structural features of the ship according to the calibration result; fusing the visual features, three-dimensional structural features, and ship identity information, and inputting the fused data into a preset verification model, and outputting a verification result through the verification model, where the verification result includes a normal ship and an abnormal ship.

[0088] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium provided in the present application and used in the embodiments can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0089] It should be noted that in this article, the term "including", "comprising", or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, device, article, or method including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such a process, device, article, or method. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, device, article, or method including that element.

[0090] The above are only the preferred embodiments of the present application, and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present application.

Claims

1. A multi-modal verification method for ship identity based on computer vision and deep learning, characterized in that, The method includes: Obtaining multimodal data of a ship, where the multimodal data includes visible light image data, lidar point cloud data, and ship identity information; Extracting visual features of the ship according to the visible light image data; Performing dynamic spatial calibration on the lidar point cloud data; Extracting three-dimensional structural features of the ship according to the calibration result; Fusing the visual features, three-dimensional structural features, and ship identity information, and inputting them into a preset verification model, and outputting a verification result through the verification model, where the verification result includes normal ships and abnormal ships.

2. The multi-modal verification method for ship identity based on computer vision and deep learning according to claim 1, characterized in that The step of performing dynamic spatial calibration on the lidar point cloud data includes: Obtaining six-degree-of-freedom motion parameters within a scanning period in real time through an inertial measurement unit carried by the ship; Based on the motion parameters, establishing a motion compensation model for the lidar point cloud; Inputting the initial point cloud data into the motion compensation model, resampling the initial point cloud data through the motion compensation model, and outputting a resampled result to obtain motion-compensated point cloud data; Converting the motion-compensated point cloud data to the ship's main body coordinate system to complete dynamic spatial calibration.

3. The multimodal verification method for ship identity based on computer vision and deep learning according to claim 1, wherein Before the step of fusing the visual features, three-dimensional structural features, and ship identity information, the method further includes: Obtaining the visual features and three-dimensional structural features, inputting the visual features and three-dimensional structural features into a preset spatial model, and generating the current behavior features of the ship through the spatial model; Obtaining the historical behavior features of the ship, combining historical behavior data and current behavior data, and predicting the ship trajectory parameters of the ship within a preset time period; Based on the predicted ship trajectory parameters, obtaining the actual trajectory parameters of the ship at the predicted time in real time, and analyzing the deviation coefficient between the actual trajectory parameters and the predicted trajectory parameters; Quantifying the deviation coefficient to generate a confidence score for the ship; Analyzing the confidence score of the ship, and when the confidence score is greater than a preset confidence score threshold, determining that the ship has a credible behavior; Based on the ship having a credible behavior, fusing the visual features, three-dimensional structural features, and ship identity information.

4. The multimodal verification method for ship identity based on computer vision and deep learning according to claim 3, characterized in that, After the step of quantifying the deviation coefficient to generate a confidence score for the ship, the method further includes: When the confidence score is less than or equal to the preset confidence score threshold, automatically marking the current ship as a suspicious target; Based on the judgment result, collecting environmental parameters of the sea area where the ship is located in real time; Analyzing whether the environmental parameters meet the interference conditions according to the environmental parameters; When the environmental parameters meet the interference conditions, analyzing the interference mode on the behavior features according to the environmental parameters, and re-verifying whether the ship meets the abnormal ship according to the interference mode.

5. The multimodal verification method for ship identity based on computer vision and deep learning according to claim 4, characterized in that, The step of analyzing the interference mode on the behavior features according to the environmental parameters and re-verifying whether the ship meets the abnormal ship according to the interference mode includes: Performing interference mode recognition on the collected environmental parameters, where the interference mode includes perception interference and movement interference; When the interference mode is perception interference, reducing the confidence score by a preset ratio; Based on the adjusted confidence score threshold, re-analyze the confidence score of the ship. When the confidence score is greater than the adjusted confidence score threshold, it is determined that the ship has a credible behavior. When the confidence score is less than or equal to the adjusted confidence score threshold, the ship is determined to be an abnormal ship.

6. The multimodal verification method for ship identity based on computer vision and deep learning according to claim 5, wherein The step of analyzing the interference category of the behavior characteristics according to the environmental parameters and re-verifying whether the ship meets the abnormal ship according to the interference category further includes: When the interference mode is mobile interference, correct the actual trajectory parameters through a preset trajectory compensation model. Based on the corrected actual trajectory parameters, regenerate the confidence score of the ship. When the regenerated confidence score of the ship is greater than the preset confidence score threshold, it is determined that the ship has a credible behavior. When the regenerated confidence score of the ship is less than or equal to the preset confidence score threshold, the ship is determined to be an abnormal ship.

7. The multimodal verification method for ship identity based on computer vision and deep learning according to claim 5, wherein Before the step of fusing the visual feature, the three-dimensional structure feature and the ship identity information, the method further includes: When there is the perception interference, reduce the fusion weight of the visual feature and increase the fusion weight of the three-dimensional structure feature. Based on the adjustment of the fusion weights, fuse the visual feature, the three-dimensional structure feature and the ship identity information.

8. A multi-modal verification system for ship identity based on computer vision and deep learning, characterized in that, including: An acquisition module, configured to acquire multi-modal data of a ship, where the multi-modal data includes visible light image data, lidar point cloud data, and ship identity information. A first extraction module, configured to extract the visual feature of the ship according to the visible light image data. A calibration module, configured to perform dynamic spatial calibration on the lidar point cloud data. A second extraction module, configured to extract the three-dimensional structure feature of the ship according to the calibration result. A fusion module, configured to fuse the visual feature, the three-dimensional structure feature and the ship identity information, and input the fused data into a preset verification model, and output a verification result through the verification model, where the verification result includes a normal ship and an abnormal ship.

9. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images