Stable detection method applied to water surface target under ship swing condition
By fusing information from radar and image sensors, using inertial measurement units to correct data, and combining Kalman filtering and deep neural networks, the problem of target detection instability caused by ship swaying was solved, achieving stable and accurate target detection and tracking on ships.
Patent Information
- Application Number
- CN202511000196.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-21
- Publication Date
- 2025-11-14
AI Technical Summary
The rolling motion of a vessel at sea causes changes in the sensor's observation angle, leading to target loss and decreased detection accuracy. Existing hardware and software solutions are either costly or ineffective.
By fusing information from radar and image sensors, using an inertial measurement unit to acquire sway state information, performing data correction and feature fusion, using a deep neural network for target detection, and combining adaptive and extended Kalman filtering to remove noise, stable target tracking is achieved.
Stable and accurate detection and tracking of maritime targets was achieved under ship rolling conditions, reducing reliance on expensive hardware and improving detection accuracy and robustness.
Smart Images
Figure QLYQS_8
Abstract
Description
Technical Field
[0001] This invention relates to the field of marine target detection technology, specifically to a stable detection method for surface targets under ship rolling conditions. Background Technology
[0002] With the rapid development of marine development, maritime safety monitoring, and marine scientific research, vessels, as important offshore operating platforms, are being used more and more widely. Vessels are typically equipped with various sensors, such as radar and photoelectric sensors (including visible light cameras and infrared thermal imagers), for the detection and identification of maritime targets.
[0003] However, when ships sail at sea, they inevitably experience six degrees of freedom (translation along the x, y, and z axes and rotation about the x, y, and z axes) swaying motion due to the complex marine environment factors such as waves and winds. This swaying motion causes the sensor's observation angle to change constantly, leading to numerous problems when the sensor detects targets at sea.
[0004] On the one hand, the phenomenon of momentary target loss by sensors occurs frequently. For example, when a vessel rolls significantly, a target that was originally within the sensor's field of view may instantly move out of the field of view, causing the sensor to be unable to continuously track the target and thus lose target information. This is extremely disadvantageous for tasks that require real-time monitoring of target dynamics, such as maritime safety surveillance and tracking illegal vessels. Once a target is lost, subsequent recapture not only requires a significant amount of time and resources, but also, during the period of target loss, the target may exhibit various unpredictable behaviors, increasing the risk and uncertainty of the mission.
[0005] On the other hand, the change in sensor perspective caused by the swaying of the boat leads to instability in the target's position and attitude information within the sensor image. Taking images acquired by a photoelectric sensor as an example, the target's position in the image may jump rapidly, and its shape may be distorted due to changes in perspective. This poses a significant challenge to image-based target detection algorithms. Traditional target detection algorithms typically assume that the target's features in the image are relatively stable. However, when the boat is swaying, the detection accuracy and reliability of these algorithms decrease significantly, easily leading to false positives and false negatives.
[0006] Currently, some research and solutions have been developed to address the impact of ship rolling on maritime target detection. Some solutions employ hardware to stabilize the sensor platform, such as using a gyro-stabilized platform. This platform uses a gyroscope to measure the ship's rolling angle in real time and utilizes actuators like motors to adjust the sensor platform in the opposite direction, maintaining a stable observation attitude as much as possible. However, this method has certain limitations. First, gyro-stabilized platforms are expensive, increasing the overall cost and maintenance expenses of the ship. Second, due to the limited response speed of mechanical structures, the stabilization platform cannot fully compensate for the effects of rapid and violent rolling, still leading to instability in sensor observations.
[0007] Some solutions rely on software algorithms, such as filtering sensor-acquired data to remove noise and interference caused by swaying. However, these filtering algorithms are often based on simple linear models, which struggle to accurately describe the complex six-degree-of-freedom swaying motion of a vessel, and are ineffective at handling complex changes in target position and attitude caused by swaying. Furthermore, some methods stabilize target detection by performing feature matching and tracking on multiple consecutive frames, but when the target moves rapidly and the vessel sways violently simultaneously, the accuracy and real-time performance of feature matching are severely affected, leading to unsatisfactory detection results.
[0008] In summary, existing solutions all have certain shortcomings in addressing the problem of stabilizing maritime targets under ship rolling conditions. Summary of the Invention
[0009] To overcome the shortcomings of the prior art, the present invention provides a stable detection method for surface targets under ship rolling conditions, which can effectively overcome the influence of ship rolling on the detection of maritime targets without relying on expensive hardware equipment, and achieve stable and accurate detection of maritime targets.
[0010] To achieve the above objectives, the present invention is implemented through the following technical solution: A method for stable detection of surface targets under ship rolling conditions includes the following steps: The radar and image information of the target are obtained from the radar and image sensors on the ship. The ship's rolling state information is obtained from the inertial measurement unit on the ship. Synchronize radar information, image information, and sway status information in time; To overcome the impact of ship rolling on sensor observations and improve the accuracy and stability of sensor data, radar and image information are corrected based on the rolling state information to obtain radar correction information and image correction information. Based on radar correction information and image correction information, radar data features and image data features are extracted, and the vectors corresponding to radar data features and image data features are concatenated into a fused feature vector. The fused feature vectors are input into a pre-trained deep neural network model to obtain detection information, which includes the target category, the confidence level of the corresponding target category, and the target location. The motion state of the detected target is tracked based on the detection information.
[0011] Furthermore, the stability detection method for surface targets under ship rolling conditions in this application includes the following process for acquiring radar information about the target: The radar continuously scans at a preset scanning frequency to acquire initial radar data, which includes the target's range and azimuth. In order to effectively remove measurement noise and improve the accuracy of range and azimuth information, an adaptive Kalman filter is used on the initial radar data to obtain the radar information of the target. The adaptive Kalman filtering process for the initial radar data includes: Let the state vector be ,in, They represent The target's distance, azimuth, radial velocity, and rate of change of azimuth at any given time; Initialize state vector and the corresponding covariance matrix covariance matrix This reflects the uncertainty in the initial state estimation; Predict the state vector at the next time step and the corresponding prediction covariance matrix , where the state transition matrix , Let Q be the time interval, and Q be the preset process noise covariance matrix. Update state vector Among them, the measured value , and The measured range and azimuth of the target at time k are the initial radar data, and the Kalman gain is... , covariance matrix R is the preset measurement noise covariance matrix.
[0012] Furthermore, the stability detection method for surface targets under ship rolling conditions in this application includes the following process for acquiring target image information: The image sensor acquires images of the target at a preset frame rate; By iterating through each pixel in the image and calculating its gray value according to a preset gray value formula, the image is converted into a single-channel grayscale image to reduce the amount of data for subsequent processing. For each pixel in the image, a window centered on it is selected, the pixel values within the window are sorted, and the median value is taken as the new value of that pixel to form the image information. This can effectively remove noise from the image and make the image smoother.
[0013] Furthermore, the stability detection method for surface targets under ship rolling conditions in this application includes the following process for obtaining ship rolling state information: The inertial measurement unit (IMU) is used to measure and acquire the vessel's state data in real time at a preset detection frequency. The vessel's state data includes the vessel's acceleration. and angular velocity ; Extended Kalman filtering is used to obtain the ship's rolling state information from the ship's state data; The process of applying extended Kalman filtering to the vessel's state data includes: Let the state vector of the boat be... ,in, , , , They represent The position, attitude angle, and rate of change of the vessel at any given moment; Initialize the state vector of the boat and the corresponding covariance matrix ; Predict the state vector at the next time step and the corresponding covariance matrix ,in, The pre-defined nonlinear state transition function, The state transition matrix is obtained by linearizing the nonlinear state transition function; The pre-defined second process noise covariance matrix describes the noise characteristics of the system process; Update state vector and the corresponding covariance matrix ,in, Corresponding to the vessel's status data, The second measurement noise covariance matrix is preset. ,in The measurement matrix is obtained by linearizing the measurement function. This is the preset measurement function.
[0014] Furthermore, in a stability detection method for surface targets under ship rolling conditions, the radar correction information in this application includes: True azimuth of the target ,in The target azimuth angle is part of radar information. , These are the roll and pitch angles of the vessel, which are information related to its swaying state. Target true distance ,in, This represents the real-time vertical displacement between the radar antenna and the target. The target distance measured by radar is part of radar information.
[0015] Furthermore, in a method for stabilizing surface targets under swaying conditions, the image correction information includes: New image pixel location , in, (x,y) represents the corresponding pixel position in the image before correction. and This is the default value; The grayscale value of the new pixel ,in, , , , Let x be the grayscale value of the four pixels surrounding the pixel at position (x, y). , .
[0016] Furthermore, in this application, a method for stabilizing surface targets under ship rolling conditions includes radar data features such as target distance data, target velocity data, and radar cross-section; and image data features such as shape, texture, and color data in the image.
[0017] Among them, the data corresponding to the target distance and the target velocity are used to obtain the target position, while the radar cross section (RCS) is used as a radar data feature to determine the target material and serves as the basis for outputting the target category and confidence level. This effectively removes interference from sea clutter, and in scenarios where image sensors fail due to environmental factors (such as dense fog), the RCS can serve as the primary basis for determining the target category, improving environmental adaptability. Image data features are used as the basis for determining the target category and confidence level.
[0018] Furthermore, the stability detection method for surface targets under ship rolling conditions in this application performs Kalman filtering on the target position to track the motion state of the detected target. During the filtering process, the target's state vector is set... , and These represent the target's position and velocity at time k, respectively, with the measured value corresponding to the target's position.
[0019] Furthermore, the stable detection method for surface targets under swaying conditions described in this application, during motion tracking, determines that the target is lost when the confidence level of the corresponding target category falls below a preset threshold, and performs re-detection. During re-detection, the search range of the radar and / or image sensor is pre-defined, with the center of the search range being the position of the target at the last moment before it was determined to be lost. This ensures continuous and stable tracking of the target and improves the robustness of the entire detection system.
[0020] As can be seen from the above technical solution, the present invention has the following beneficial effects: This invention provides a stable detection method for surface targets under tumbling conditions, enabling stable and accurate detection and tracking of maritime targets even under these conditions. It integrates radar and image data features, fully utilizing the advantages of different sensors to learn the complex characteristics of targets under tumbling conditions, thereby improving the accuracy and reliability of target detection. It achieves stable detection of maritime targets through software algorithms without relying on expensive hardware stabilization equipment, reducing vessel costs and complexity. It is particularly suitable for unmanned surface vessel (USV) control. Detailed Implementation
[0021] A method for stable detection of surface targets under ship rolling conditions includes the following steps: The radar and image information of the target are obtained from the radar and image sensors on the ship. The swaying state information of the boat is obtained from the inertial measurement unit located in the center of mass region of the boat; Synchronize radar information, image information, and sway status information in time; To overcome the impact of ship rolling on sensor observations and improve the accuracy and stability of sensor data, radar and image information are corrected based on the rolling state information to obtain radar correction information and image correction information. Based on radar correction information and image correction information, radar data features and image data features are extracted, and the vectors corresponding to radar data features and image data features are concatenated into a fused feature vector. The fused feature vectors are input into a pre-trained deep neural network (DNN) model to obtain detection information, which includes the target category, the confidence level of the corresponding target category, and the target location. The deep neural network (such as Faster R-CNN) model is trained on marine target data when the boat is swaying, and learns the correspondence between different data features and detection information. The motion state of the detected target is tracked based on the detection information.
[0022] Based on the above method, the stable detection method for surface targets under ship rolling conditions proposed in this application can achieve stable and accurate detection and tracking of maritime targets under ship rolling conditions. Furthermore, it integrates radar data features and image data features, fully utilizing the advantages of different sensors and learning the complex characteristics of targets under rolling conditions, thereby improving the accuracy and reliability of target detection. It enables stable detection of maritime targets through software algorithms without relying on expensive hardware stabilization equipment, reducing ship costs and complexity.
[0023] In this embodiment, radar information, image information, and sway status information are synchronized using a hardware clock or software timestamp. For example, timestamps are recorded when the radar and image sensors acquire data, and the same timestamps are also recorded when the inertial measurement unit measures attitude information. In subsequent processing, the attitude information and sensor data at the same moment are matched based on the timestamps to ensure that the attitude information used when correcting the sensor data corresponds to the sensor data at that moment.
[0024] Furthermore, in this embodiment, the process of acquiring radar information of the target includes: The radar continuously scans at a preset scanning frequency to acquire initial radar data, which includes the target's range and azimuth. In order to effectively remove measurement noise and improve the accuracy of range and azimuth information, an adaptive Kalman filter is used on the initial radar data to obtain the radar information of the target. The adaptive Kalman filtering process for the initial radar data includes: Let the state vector be ,in, They represent The target's distance, azimuth, radial velocity, and rate of change of azimuth at any given time; Initialize state vector and the corresponding covariance matrix covariance matrix This reflects the uncertainty in the initial state estimation; Predict the state vector at the next time step and the corresponding prediction covariance matrix , where the state transition matrix , For time intervals, specifically The scanning interval corresponding to the radar operation, Q is a preset process noise covariance matrix that describes the noise characteristics of the system process and can be determined by experimental or theoretical analysis. Update state vector Among them, the measured value , and The measured range and azimuth of the target at time k are the initial radar data, and the Kalman gain is... , covariance matrix R is a preset measurement noise covariance matrix, which reflects the uncertainty of radar measurement and can be estimated based on the radar's performance parameters.
[0025] Furthermore, in this embodiment, the process of acquiring the image information of the target includes: Image sensors (such as visible light cameras or infrared thermal imagers) acquire images of the target at a preset frame rate; By iterating through each pixel in the image and calculating its grayscale value according to a preset grayscale formula, the image is converted into a single-channel grayscale image to reduce the amount of data for subsequent processing. The specific grayscale formula is: [grayscale value formula would be inserted here]. ,in , , These are the red, green, and blue channel values of the image, respectively. For each pixel in the image, a 3×3 window centered on it is selected, the pixel values within the window are sorted, and the median value is taken as the new value of the pixel to form the image information. This can effectively remove noise from the image and make the image smoother.
[0026] Furthermore, in this embodiment, the process of obtaining the boat's rolling state information includes: The inertial measurement unit (IMU) is used to measure and acquire the vessel's state data in real time at a preset detection frequency. The vessel's state data includes the vessel's acceleration. and angular velocity ; Extended Kalman filtering is used to obtain the ship's rolling state information from the ship's state data; The process of applying extended Kalman filtering to the vessel's state data includes: Let the state vector of the boat be... ,in, , , , They represent The position, attitude angle, and rate of change of the vessel at any given moment; Initialize the state vector of the boat and the corresponding covariance matrix ; Predict the state vector at the next time step and the corresponding covariance matrix ,in, The pre-defined nonlinear state transition function, The state transition matrix is obtained by linearizing the nonlinear state transition function; The pre-defined second process noise covariance matrix describes the noise characteristics of the system process; Update state vector and the corresponding covariance matrix ,in, Corresponding to the vessel's status data, The second measurement noise covariance matrix is preset. ,in The measurement matrix is obtained by linearizing the measurement function. This is the preset measurement function.
[0027] Based on this, taking roll angle estimation as an example: assuming the estimated roll angle at the current moment is... Based on the roll angular velocity measured by the IMU Predict the roll angle at the next moment ,in To detect the interval time, the roll angle value measured by the IMU is combined. The Kalman gain is calculated using the EKF update formula, and then the roll angle estimate is updated. .
[0028] Furthermore, in this embodiment, the radar correction information includes: True azimuth of the target ,in The target azimuth angle is part of radar information. , These are the roll and pitch angles of the vessel, which are information related to its swaying state. Target true distance ,in, This represents the real-time vertical displacement between the radar antenna and the target. The target distance measured by radar is part of radar information.
[0029] In one embodiment, specifically, the real-time vertical displacement between the radar antenna and the target. Where A is the radar antenna installation height. For antennas and boats.
[0030] Furthermore, in this embodiment, the image correction information includes: New image pixel location , in, (x,y) represents the corresponding pixel position in the image before correction. and This is the default value; The grayscale value of the new pixel ,in, , , , Let x be the grayscale value of the four pixels surrounding the pixel at position (x, y). , .
[0031] Furthermore, in this embodiment, the radar data features include data corresponding to the target distance, data corresponding to the target velocity, and radar cross section; the image data features include data corresponding to the shape, texture, and color in the image.
[0032] Among them, the data corresponding to the target distance and the data corresponding to the target velocity are used to obtain the target position. Specifically, the target position is taken as a two-dimensional plane using bounding box coordinates. express,( , ( These represent the coordinates of the target's diagonal points. The radar cross section (RCS), as a radar data feature used to determine the target's material, serves as the basis for outputting the target category and confidence level. It effectively removes interference from sea clutter. Furthermore, in scenarios where image sensors are affected by environmental factors (such as dense fog), the RCS can serve as the primary basis for determining the target category, improving environmental adaptability. Image data features are used as the basis for determining the target category and confidence level.
[0033] Specifically, the process of extracting image data features based on image correction information involves inputting the image correction information into a pre-defined convolutional neural network, such as VGG16, and extracting features such as shape, texture, and color from the image through multiple convolutional Ci and pooling Pi operations. : , ,in, For the input image, For convolution kernel, For bias.
[0034] Furthermore, in this embodiment, Kalman filtering is applied to the target position to track the motion state of the detected target. Specifically, the Kalman filtering process refers to the adaptive Kalman filtering process applied to the initial radar data described above. During the filtering process, the target's state vector is set... , and These represent the target's position and velocity at time k, respectively. The measured value of the target corresponds to the target's position, which can be represented by the midpoint of the boundary coordinates.
[0035] Furthermore, in this embodiment, during motion tracking, if the confidence level of the corresponding target category falls below a preset threshold, the target is determined to be lost, and re-detection is performed. To improve detection efficiency, the search range of the radar and / or image sensor is pre-defined during re-detection. The center of the search range is the last position of the target before it was determined to be lost. For example, if the search range is a circular area with a radius of 10 pixels, the center of the circle is the center of the search range. This ensures continuous and stable tracking of the target and improves the robustness of the entire detection system.
[0036] The technical principles of the present invention have been described above with reference to specific embodiments. These descriptions are merely for explaining the principles of the invention and should not be construed as limiting the scope of protection of the invention in any way. Based on this explanation, those skilled in the art can conceive of other specific embodiments of the invention without creative effort, and these embodiments will all fall within the scope of protection of the present invention.
Claims
1. A method for stable detection of surface targets under ship rolling conditions, characterized in that: Includes the following steps: The radar and image information of the target are obtained from the radar and image sensors on the ship. The ship's rolling state information is obtained from the inertial measurement unit on the ship. Synchronize radar information, image information, and sway status information in time; The radar information and image information are corrected based on the swing state information to obtain radar correction information and image correction information. Based on radar correction information and image correction information, radar data features and image data features are extracted, and the vectors corresponding to radar data features and image data features are concatenated into a fused feature vector. The fused feature vectors are input into a pre-trained deep neural network model to obtain detection information, which includes the target category, the confidence level of the corresponding target category, and the target location. The motion state of the detected target is tracked based on the detection information.
2. The method for stable detection of surface targets under ship rolling conditions according to claim 1, characterized in that: The process of acquiring radar information about a target includes: The radar continuously scans at a preset scanning frequency to acquire initial radar data, which includes the target's range and azimuth. Adaptive Kalman filtering is applied to the initial radar data to obtain the target's radar information; The adaptive Kalman filtering process for the initial radar data includes: Let the state vector be ,in, They represent The target's distance, azimuth, radial velocity, and rate of change of azimuth at any given time; Initialize state vector and the corresponding covariance matrix ; Predict the state vector at the next time step and the corresponding prediction covariance matrix , where the state transition matrix , Let Q be the time interval, and Q be the preset process noise covariance matrix. Update state vector Among them, the measured value , and The measured range and azimuth of the target at time k are the initial radar data, and the Kalman gain is... , covariance matrix R is the preset measurement noise covariance matrix.
3. The method for stable detection of surface targets under ship rolling conditions according to claim 1, characterized in that: The process of acquiring image information of a target includes: The image sensor acquires images of the target at a preset frame rate; By iterating through each pixel in the image and calculating its gray value according to a preset gray value formula, the image is converted into a single-channel grayscale image. For each pixel in the image, a window centered on it is selected, the pixel values within the window are sorted, and the median value is taken as the new value of that pixel to form the image information.
4. The method for stable detection of surface targets under ship rolling conditions according to claim 1, characterized in that: The process of obtaining information about the rolling state of a vessel includes: The inertial measurement unit (IMU) is used to measure and acquire the vessel's state data in real time at a preset detection frequency. The vessel's state data includes the vessel's acceleration. and angular velocity ; Extended Kalman filtering is used to obtain the ship's rolling state information from the ship's state data; The process of applying extended Kalman filtering to the vessel's state data includes: Let the state vector of the boat be... ,in, , , , They represent The position, attitude angle, and rate of change of the vessel at any given moment; Initialize the state vector of the boat and the corresponding covariance matrix ; Predict the state vector at the next time step and the corresponding covariance matrix ,in, For the preset nonlinear state transition function, The state transition matrix is obtained by linearizing the nonlinear state transition function; The pre-defined second process noise covariance matrix describes the noise characteristics of the system process; Update state vector and the corresponding covariance matrix ,in, Corresponding to the vessel's status data, The second measurement noise covariance matrix is preset. ,in The measurement matrix is obtained by linearizing the measurement function. This is the preset measurement function.
5. The method for stable detection of surface targets under ship rolling conditions according to claim 1, characterized in that: Radar correction information includes: True azimuth of the target ,in The target azimuth angle is part of radar information. , These are the roll and pitch angles of the vessel, which are information related to its swaying state. Target true distance ,in, This represents the real-time vertical displacement between the radar antenna and the target. The target distance measured by radar is part of radar information.
6. The method for stable detection of surface targets under ship rolling conditions according to claim 1, characterized in that: Image correction information includes: New image pixel location , in, (x,y) represents the corresponding pixel position in the image before correction. and This is the default value; The grayscale value of the new pixel ,in, , , , Let x be the grayscale value of the four pixels surrounding the pixel at position (x, y). , .
7. The method for stable detection of surface targets under ship rolling conditions according to claim 1, characterized in that: Radar data features include data corresponding to target distance, data corresponding to target velocity, and radar cross section; image data features include data corresponding to shape, texture, and color in the image.
8. The method for stable detection of surface targets under ship rolling conditions according to claim 1, characterized in that: Kalman filtering is applied to the target position to track the motion state of the detected target. During the filtering process, the target's state vector is set... , and These represent the target's position and velocity at time k, respectively, with the measured value corresponding to the target's position.
9. A method for stable detection of surface targets under ship rolling conditions according to claim 1, characterized in that: During motion tracking, if the confidence level of the corresponding target category is lower than a preset threshold, the target is judged to be lost and re-detected. During the re-detection process, the search range of the radar and / or image sensor is pre-defined, and the center of the search range is the position of the target at the last moment before it was judged to be lost.